Modeling Semantic Cognition as Logical Dimensionality Reduction

Modeling Semantic Cognition as Logical Dimensionality Reduction Yarden Katz, Noah D. Goodman, Kristian Kersting, Charles Kemp, Joshua B. Tenenbaum {yarden, ndg, kersting, ckemp, jbt}@mit.edu Department of Brain and Cognitive Sciences & Computer Science and Artificial Intelligence Laboratory Massachusetts Institute of Technology Cambridge, MA 02139 Abstract perspective. While these models can handle noise and un- certainty, they also have important limitations, complemen- tary to those of symbolic approaches. They do not naturally display the systematicity and compositionality that character- ize people’s intuitive theories and common-sense reasoning (Fodor & Pylyshyn, 1988). This limitation shows up most clearly when making inferences about novel entities that are only sparsely observed. Suppose that we encounter two new kinds of organisms, tufas and kibos, and we are told that kibos have some novel property (e.g., they have omulums). If we then learn that tufas are kibos, it is likely that tufas also have omulums. The CQ model makes this prediction, via property inheritance down through the taxonomy of categories. The RM network, however, does not automatically yield this in- ference, if it is trained on the two facts is a(tufa, kibo) and has a(kibo, omulum). Because the concept ‘kibo’ plays dif- ferent roles in these two propositions—it is the object of is a and the subject of has a—it is represented in different popu- lations of units, and the effects of training on these two facts appear in non-overlapping sets of weights. The network does not equate ‘kibo’ in the first proposition with ‘kibo’ in the second proposition, and so fails to draw the obvious infer- ence. This example does not imply that it would be impossible to design a connectionist semantic model whose inferences did respect basic principles of systematicity and compositional- ity. Our point is only that the connectionist approach does not naturally capture these aspects of human inference, which are no less essential than the statistical capacities it does cap- ture well. While it is possible that either the connectionist approach or the structured symbolic approach could be ex- tended in some way that makes for a satisfactory solution, our aim here is to explore new alternatives for modeling abstract semantic knowledge. We describe an approach that combines valuable capaci- ties of both traditional paradigms: an ability to represent ab- stract knowledge respecting systematicity and composition- ality, and hence to make appropriate inferences from sparse data in novel situations; and an ability to learn from noisy data and generalize in graded fashion based on the statistics of observed data. Our approach is based on a hierarchical Bayesian model over logical representations. This is similar in spirit to proposals in inductive logic programming (Mug- gleton & De Raedt, 1994), although these are not typically formalized explicitly as hierarchical Bayesian models (for a review of this approach see, Tenenbaum, Griffiths, and Kemp (2006)). A close relative of our approach is (Conklin & Wit- Semantic knowledge is often expressed in the form of intuitive theories, which organize, predict and explain our observations of the world. How are these powerful knowledge structures represented and acquired? We present a framework, logical dimensionality reduction, that treats theories as compressive probabilistic models, attempting to express observed data as a sample from the logical consequences of the theory’s under- lying laws and a small number of core facts. By performing Bayesian learning and inference on these models we combine important features of more familiar connectionist and symbolic approaches to semantic cognition: an ability to handle graded, uncertain inferences, together with systematicity and compo- sitionality that support appropriate inferences from sparse ob- servations in novel contexts. Problems of semantic cognition A person’s store of common-sense knowledge about the world is vast, but it is more than just a vast collection of facts. In many domains, the mind organizes what it knows into large-scale systems, with structure at multiple levels of abstraction. These abstract systems of representation—called schemas, or theories—are crucial for our ability to make in- ferences that go beyond the sparse and noisy data of percep- tual experience. How are such powerful knowledge representations ac- quired, structured, and used? These problems lie at the heart of semantic cognition. Early proposals focused on symbolic structures, as in the semantic networks of Collins and Quillian (1969) (hereafter CQ) for organizing categories of objects and their properties. Categories are placed in a tree-structured taxonomy, with properties located at nodes of the tree and assumed to inherit down to all categories below them. Each property needs to be stored only at the highest node of the tree where it holds generically, leading to a compact encoding of objects’ properties and the ability to project known properties to novel objects. Later work emphasized the limitations of symbolic repre- sentations in handling noisy data or exceptions, and in ac- counting for graded effects of similarity on people’s inductive judgments. Symbolic approaches were also criticized for not providing a working account of how their abstract represen- tations could be learned from experience. An alternative approach to modeling abstract semantic knowledge using connectionist networks emerged in the 1980’s. Hinton proposed a connectionist network that could learn abstract systems of kinship relations (Hinton, 1986), while RM explored a similar architecture for learning about categories and properties (Rogers & McClelland, 2004)— essentially the same problem that CQ treated from a symbolic