A Model of Concept Generalization and Feature Representation in Hierarchies

A Model of Concept Generalization and Feature Representation in Hierarchies Timothy N. Rubin (trubin@uci.edu) Department of Cognitive Sciences, University of California, Irvine Matthew D. Zeigenfuze (mzeigenf@msu.edu) Department of Psychology, Michigan State University Mark Steyvers (msteyver@uci.edu) Department of Cognitive Sciences, University of California, Irvine Abstract We present a novel modeling framework for representing cat- egory exemplars and features. This approach treats each cat- egory exemplar as a probability distribution over a hierarchi- cally structured graph. The model jointly learns the mixture of each exemplar across categories in the graph, and a fea- ture representation for each node in the graph, including nodes for which no data is directly observed. We apply this model to two distinct types of data: (1) Animal by Feature matri- ces from the Leuven Natural Concept Database, and (2) docu- ments from Wikipedia. We demonstrate that this model is use- ful for learning feature representations for nodes in the graph that are not assigned any data (i.e. for generalization to new categories). Additionally this model improves the specificity of feature representations for the nodes with observed data by explaining away more general features to parent nodes within the graph. Furthermore, we illustrate that this model is useful for understanding additional psychological aspects of concept representation, such as typicality ratings. Keywords: Concepts; Category Learning; Graphical Models; Hierarchical Models; Bayesian; Generalization Suppose you were presented with an unfamiliar concept, blorg. Despite having no experience with blorgs, if you were told a blorg is an animal you would know that it eats and that it breathes. If you were told a blorg is a mammal you would know that it has hair and births live young, as well as that it eats and breathes, since mammals are also animals. Finally, if you were told that a blorg is a dog you would know that it can bark, as well as that it possesses all of the features of animals and mammals. Clearly, both the categories to which an exem- plar belongs and the hierarchy in which those categories re- side carry considerable information about the features of that exemplar. Alhough some work has been done looking at how people associate features to a particular category (e.g., Kemp & Tenenbaum, 2009; Austerweil & Griffiths, 2009; Zeigen- fuse, 2010), it is unclear how people learn to associate fea- tures to levels in a category hierarchy, particularly when they must generalize to categories in the hierarchy for which there is no observed data. In this paper, we present a rational model of how people jointly learn to associate features with a particular level within a hierarchy, and to learn distributed representations of exem- plars across this hierarchy. This model begins with feature representations of exemplars of categories, and the category structure of the domain to which the exemplars belong. It learns the features associated with each category within the structure (even for categories for which there are no exem- plars), as well as a distributed representation for each exem- plars across multiple levels of abstraction within the hierar- chy. This approach differs fundamentally from many other approaches to modeling hierarchical relationships between categories in that learns a distributed representation for each exemplar for a category. Specifically, an underlying assump- tion behind many such approaches is that an exemplar’s fea- tures are inherently tied to only the category to which the exemplar belongs. This assumption underlies many classi- cal approaches to modeling hierarchical relationships, such as hierarchical clustering methods (e.g., Shepard, 1980), as well more recent advances which learn the basic structural form of these relationships (of which a hierarchy is just one possi- bility), in addition to the graph itself (Kemp & Tenenbaum, 2008). One notable exception is the approach to distributed representations of semantic memory proposed by Collins & Quillian (1969). Although Collins & Quillian (1969) did not address the problem of learning their proposed representa- tions, and the topic of their paper is somewhat different from our own, the underlying approach of the model we present is very much in the spirit of their work. We apply our model to two highly distinct datasets. First, using Animal by Feature matrices from the Leuven Natural Concept Database (de Deyne et al., 2008), we learn featu- ral representations of animals at the species (e.g., DOGS ), animal-category (e.g., MAMMALS ), and domain (i.e., A N - IMALS ) levels of abstraction (despite there being no data directly assigned at either the animal-category or domain level). We then show that the representation of exemplars as probability distributions across the hierarchy naturally cap- tures psychological phenomena such as an animal’s perceived “typicality” for a category, which has been shown to be a fun- damental property of category representation (e.g., Rosch & Mervis, 1975). We additionally apply our model to docu- ments from a subset of the Wikipedia category structure, in order to demonstrate that our approach is applicable to noisy, real-world data, represented within a more convoluted hier- archical structure that spans multiple domains with a wide range of category specificity. A Mixture Model for Representing Exemplar Features over Graph Hierarchies In this section, we present a model for learning feature rep- resentations for categories using a framework related to the Topic Model (Blei et al., 2003). The Topic Model was origi- nally presented as an unsupervised learning method for find- ing low-dimensional representations of text corpora. In psy- chology, the topic model has been used to explain a number of phenomena in semantic representation (Griffiths et al., 2007).