The Role of Attributional and Distributional Information in Semantic Representation

The Role of Attributional and Distributional Information in Semantic Representation Mark Andrews (mark@gatsby.ucl.ac.uk) Gatsby Computational Neuroscience Unit, University College London London, WC1N 3AR Gabriella Vigliocco (g.vigliocco@ucl.ac.uk) David Vinson (d.vinson@ucl.ac.uk) Department of Psychology, University College London London, WC1E 6BT that two words appear in similar texts might imply that these words refer to similar things in the world. It is, however, reasonable to assume that these sources are distinct, or that one source is not entirely explained by or dependent upon the other. For example it can be ar- gued that attributional information is more important in learning words referring to concrete entities and ac- tions, for which the properties of the thing or event in the world can be experienced via the senses, whereas distri- butional information may be more important in learning abstract words, those that we learn primarily via lin- guistic context. Thus both types of information can be exploited in order to learn word meanings. In previous studies, the contributions of either source alone, inde- pendent of the other, have been studied. In particular, within cognitive psychology, researchers have primarily focused on the development of models of meaning repre- sentation based on attributional information (Collins & Quillian, 1969; Hinton & Shallice, 1991; McRae, Sa, & Seidenberg, 1997; McClelland & Rogers, 2003; Minsky, 1975; Smith, Shoben, & Rips, 1974; Vigliocco, Vinson, Lewis, & Garrett, 2004), whereas recently within compu- tational linguistics, machine learning and related areas in cognitive science, researchers have primarily focused on distributional information alone (Burgess & Lund, 1997; Dagan, Lee, & Pereira, 1997; Griffiths & Steyvers, 2003; Hofmann, 1999b; Landauer & Dumais, 1997; Sch¨ utze, 1992). In this paper, we describe a model that uses both sources in combination as the basis of semantic repre- sentation. Abstract In recent studies of semantic representation, two distinct sources of information from which we can learn word meanings have been described. We refer to these as at- tributional and distributional information sources. At- tributional information describes the attributes or fea- tures associated with referents of words, and is acquired from our interactions with the world. Distributional in- formation describes the distribution of words across dif- ferent linguistic contexts, and is acquired through our use of language. While previous work has concentrated on the role of one source, to the exclusion of the other, in this paper we study the role of both sources in combi- nation. We describe a general framework based on prob- abilistic generative models for modelling both sources of information, and how they can be integrated to learn se- mantic representation. We provide examples comparing the learned structures of each of three models: attri- butional information alone, distributional information alone, and both sources combined. Keywords: Probabilistic Models; Semantic Represen- tation; Information Integration. Introduction We can distinguish between two distinct sources that provide us with information about the meanings of words. The first is what we will call attributional in- formation. This describes physical, emotional and con- ceptual (or otherwise nonlinguistic) attributes associated with the referents of words. These are features or at- tributes associated with words that are based upon or built up from our interactions in the environment, and our knowledge of the objects and relationships in the world. A second source of information about word mean- ings is what we will call distributional information. This describes the distribution of words across different lin- guistic or textual contexts. This type of information has been memorably summarized by the phrase “you shall know a word by the company it keeps”, (Firth, 1957). In other words, the type of linguistic contexts in which a word occurs can provide clues as to what that word might mean. While these two sources of information are not un- correlated, neither are they identical, and it is plausi- ble to assume that they are both utilized when learning the meaning of words. The two sources are correlated because words that refer to similar things or events in the world are likely to appear in similar linguistic con- texts. The reverse case could also be argued. Knowing Generative Models of Semantic Representations Probabilistic generative models provide a general means by which to model semantic representation. This ap- proach has already been pursued both within machine learning (Blei & Ng, 2003; Hofmann, 1999b, 1999a; Teh, Jordan, Beal, & Blei, 2004) and within cognitive science (Griffiths & Steyvers, 2003, 2002). Generative models describe the data of interest in terms of explicit proba- bilistic relations and variables. The nature of these rela- tionships and variables are inferred or learned from data. This general class of models can be used to model the role of both attributional and distributional information. Indeed, many of the previous models of semantic repre- sentation, mentioned above, can be described as implic- itly falling within this general class. In this paper, we