Construction in Semantic Memory: Generating Perceptual Representations With Global Lexical Similarity

Construction in Semantic Memory: Generating Perceptual Representations With Global Lexical Similarity Brendan T. Johns (johns4@indiana.edu) Michael N. Jones (jonesmn@indiana.edu) Department of Psychological and Brain Sciences, Indiana University 1101 E. Tenth Street, Bloomington, In., 47405 Perceptual symbol systems theory (PSS; Barsalou, 1999), one of the cornerstones of the grounded cognition movement (Barsalou, 2008), has been proposed as a competitor to distributional models as an explanatory theory for the emergence of lexical semantic structure in memory. The basis of PSS is the dismissal of amodal symbols as the central component underlying human mental representation. Rather, the PSS approach proposes that the symbols used in reasoning, memory, language, and learning are grounded in sensory modalities. In the realm of lexical semantics, PSS proposes that the mental representation of a word is based on the perceptual states that underlie experiences with the word‘s physical referent (Barsalou, 1999). Across many experiences with words, the underlying neural states tend to stabilize and create an accurate perceptual representation of a word that is grounded across sensory areas in the cortex. There is considerable evidence, across both behavioral and neuroimaging experiments, that the perceptual associates of words do play a central role in language processing (for a review see Barsalou, 2008). Although distributional models and PSS are often discussed as competing theories, the two are certainly not mutually exclusive. PSS is unable to make claims about the meanings of words that have no physical manifestation—it is fairly limited to concrete nouns and action verbs (although these are the most commonly used experimental stimuli). Further, PSS is silent regarding the simple observation that humans are quite capable of forming sophisticated lexical representations when they have been given nothing to ground those representations in. This is the situation in which distributional models excel—inferring the meaning of words in the absence of perceptual information. However, distributional models certainly fail when given tests that stress the use of perceptual information—the situation in which PSS excels. Hence, the two theories should not be viewed as competitors, but rather as complimentary (see Riordan & Jones, 2010). What is needed is research into how humans might integrate the two types of information to make full use of both the structure of language and the perceptual environment. Here we explore whether a central component of PSS, perceptual simulation, may be integrated with a distributional model to infer perceptual information for words that have never been ―perceived‖ by the model based on global lexical similarity to words that have been perceived. Further, we test the model‘s ability to infer the Abstract The literature currently contains a dichotomy in explaining how humans learn lexical semantic representations for words. Theories generally propose either that lexical semantics are learned through perceptual experience, or through exposure to regularities in language. We propose here a model to integrate these two information sources. The model uses the global structure of memory to exploit the redundancy between language and perception in order to generate perceptual representations for words with which the model has no perceptual experience. We test the model on a variety of different datasets from grounded cognition experiments. Keywords: Semantic memory; co-occurrence models; LSA. Introduction Modern computational models of lexical semantics (e.g., latent semantic analysis (LSA); Landauer & Dumais, 1997) infer representations for words by observing distributional regularities across a large corpus of text. Although their specific learning mechanisms may differ considerably, all members of this class of model rely on statistical information in text to infer semantic structure. Distributional models have seen considerable success at accounting for an impressive array of behavioral data in tasks involving semantic cognition. Since their beginning, however, distributional models have been heavily criticized for their exclusive reliance on linguistic information (e.g., Perfetti, 1998), essentially making them models of learning meaning ―by listening to the radio‖ (McClelland). More recently, empirical research has demonstrated that distributional models fail to account for a variety of semantic phenomena in the realm of embodied cognition (e.g., Glenberg & Robertson, 2000). This failure is not a great surprise given that distributional models do not receive perceptual input, and they actually perform surprisingly well on many tasks believed to require perceptual learning due to the amount of perceptual information redundantly coded in both language and the environment (for a review, see Riordan & Jones, 2010). Distributional models do not argue that perceptual information is unimportant to semantic learning. Perceptual information is still statistical information; what is required is a mechanism by which these two sources of information may be integrated. Attempts to integrate linguistic and perceptual information in a unified distributional model are now emerging (e.g., Andrews, Vigliocco, & Vinson, 2009; Jones & Recchia, 2010). However, there is little connection in these models to existing theories of modal perceptual symbol learning.