Mental Simulation in Spatial Language Processing

Mental Simulation in Spatial Language Processing Benjamin Bergen (Bergen@Hawaii.Edu) Department of Linguistics, 569 Moore Hall, 1890 East-West Rd. Honolulu, HI 96822, USA Recently, imagery has begun to surface in the study of how meaning is accessed during language processing. If imagery is of critical importance in other cognitive operations that involve conceptual categories and performing inferences about them, then there's good reason to think, as suggested by a number of researchers (Lakoff 1987, Langacker 1987), that it may well play a part in linguistic communication as well. Indeed, three scholarly traditions have recently converged on the notion that language understanding critically engages mental imagery, or mental simulation. Theoretical work in the cognitive linguistics tradition has long emphasized the importance of access to embodied representations of the world in the use and representation of language (e.g. Lakoff 1987). Research in cognitive psychology from an embodied perspective has similarly repeatedly pointed out the importance of low-level perceptual and motor processes in higher level cognitive functions like language (Barsalou 1999a, Glenberg & Robertson 2000). And research on mental models in narrative comprehension have emphasized the role of detailed perceptual and motor knowledge in the construction of mental representations of scenes described through language (Zwaan 1999). The product of this convergence of views is a number of lines of empirical and theoretical work arguing that understanding language leads to the automatic and unconscious activation of mental imagery corresponding to the content of the utterance. This imagery may be motor or perceptual in nature, and may lead to interference with actually performing actions or perceiving percepts in the world simultaneously in some experimental setups, or may lead to the facilitatory priming of such behaviors in others, as discussed below. This paper focuses on visual imagery, providing empirical evidence that language processing drives detailed perceptual images of described entities and their attributes. It advances the study of language-induced mental simulation in three ways. First, it demonstrates that mental imagery can be evoked by either subject nouns or main verbs in sentence stimuli. Second, it shows that while literal language about motion results in spatial imagery, the same is not true for metaphorical language, which implies that it isn't just lexical associations but rather the construction of a model of the whole sentence's meaning that drives simulation. And third, it shows that spatial imagery is specific to the direction of motion - up or down - and not just the axis of motion, as previously demonstrated (Richardson et al. 2003). On the basis of these results, it argues for a view of lexical and sentential meaning in which words pair phonological form with specifications for imagery to be performed, and larger Abstract There is mounting evidence that language comprehension involves the activation of mental simulations (Barsalou 1999a) of the content of utterances (Barsalou 1999b, Stanfield and Zwaan 2001, Zwaan et al. 2002, Richardson et al. 2003, Bergen et al. 2003, Narayanan et al. 2004, Bergen et al. 2004). These simulations can have motor or perceptual content. Three main questions about the process remain unexplored, however. First, are lexical associations with perception or motion sufficient to yield mental simulation, or is the integration of lexical semantics into larger structures, like sentences, necessary? Second, what linguistic elements, e.g. verbs, nouns, etc., can trigger mental simulations? And third, how detailed are the visual simulations being performed? This paper presents findings pertaining to each of these questions, using a visual object categorization task that investigated whether up- or down-related language selectively interferes with visual processing in the same part of the visual field (following Richardson et al 2003). Specifically, it finds that either subject nouns or main verbs can trigger visual imagery, but only when used in literal sentences about real space - metaphorical language does not yield significant effects. This imagery is detailed as to the part of the visual field where the described scene would take place. Keywords: Mental simulation, visual imagery, sentence processing, embodiment, metaphor Introduction thought is impossible without an image. -Aristotle, On Memory and Recollection From the earliest records forward, and with few exceptions, western reflections on the mind have assigned a critical place to imagery, the internally driven mental creation or recreation of world experiences. While certain paradigms of the second half of the twentieth century, notably symbolic cognitive science and artificial intelligence, have de- emphasized mental imagery, the past several decades have seen a steady increase in both the intensity of scientific investigation on imagery and the range of cognitive phenomena it has been shown to take part in. The recurrent finding from empirical studies using a variety of methods is that humans automatically and unconsciously engage perceptual and motor imagery when performing a broad range of cognitive tasks, like recall (Nyberg 2001) and categorization (Barsalou 1999a). The benefit of conscripting imagery for these tasks is clear - imagery provides a modality-specific, continuous representation, well suited for comparing with perceptual input or performing inference.