Simulating Realism in Language Comprehension

Simulating Realism in Language Comprehension Kevin J. Holmes (kevin.holmes@emory.edu) Phillip Wolff (pwolff@emory.edu) Department of Psychology, Emory University 36 Eagle Row, Atlanta, GA 30322 Abstract Do mental simulations of perceptual information look more like high-resolution photographs or sketchy line drawings? In the domain of language comprehension, considerable evidence suggests that words and sentences trigger simulations, but it is unclear to what extent they resemble perceptual reality. We explored the possibility that different types of language may be associated with simulations at different levels of realism. In Experiment 1, participants judged whether an object depicted in a photograph or a line drawing had been mentioned in a preceding sentence. Object recognition was faster for photographs after sentences containing adjectives than sentences containing spatial terms, and this difference was greater than for line drawings. In Experiment 2, recognition performance for color drawings was intermediate to that of photographs and line drawings, pointing to a continuum of realism from schematic to photorealistic. The results suggest that perceptual simulation is not monolithic in nature, and that language, by eliciting simulations capturing different levels of realism, may induce different ways of conceptualizing the world. Keywords: realism; mental comprehension; word meaning. simulation; language Introduction A growing body of evidence suggests that the process of language comprehension spontaneously triggers representations much like those derived through direct perceptual experience. A number of studies have demonstrated that processing words or sentences activates perceptual information such as shape, orientation, and motion (see Zwaan & Madden, 2005, for a review), suggesting that language comprehenders spontaneously construct perceptual simulations to represent the meaning of linguistic expressions (Barsalou, 1999; Zwaan, 2004). Although such simulations are assumed to bear an analog relationship to the objects and events they represent, little is known about the precise nature of their perceptual properties. In particular, it is unclear whether simulations capture the full richness of perceptual reality, as opposed to a coarser or more schematic level of detail. In this research, we investigate the possibility that simulations can vary in realism, ranging from virtually photorealistic to highly schematic, depending on the type of language processed. To the extent that realistic and schematic perceptual forms afford different mental operations, a simulation’s level of realism may provide important clues as to how meaning is represented from linguistic input and how such representations are further used in thinking and reasoning. Although research on simulation in language comprehension has not directly examined realism, a schematic level of representation has typically been assumed. It has been suggested that because simulations reflect partial reactivations of previously experienced perceptual states (Barsalou, 1999) and depend on limited attentional resources (Zwaan, Stanfield, & Yaxley, 2002), they must be relatively schematic in nature. This assumption converges with the cognitive linguistic theories of Talmy (1983) and Langacker (1987), which hold that semantic knowledge draws on image schemas that capture only select components of experience (e.g., paths, spatial relations) and omit irrelevant information. In line with this view, studies examining the psychological reality of image schemas have tended to use highly impoverished materials, consistent with the possibility that the underlying representations lack fine detail. Richardson et al. (2004), for example, used simple geometric shapes and arrows to investigate spatial aspects of verb representations. Similarly, work on perceptual simulation by Zwaan and colleagues (e.g., Stanfield & Zwaan, 2001; Zwaan et al., 2002) has focused largely on properties (e.g., shape, orientation) that can be readily depicted in line drawings (but see Yaxley & Zwaan, 2007, for findings using more realistic materials). The assumption that limited attention renders simulations schematic suggests that a simulation’s level of realism may depend on how deeply the meaning of a linguistic expression is processed. On this view, processing language deeply results in relatively rich simulations. Recent work by Holmes and Wolff (2010) raises an interesting alternative possibility. Rather than being tied to depth of processing, the simulation of realism may depend on the type of language processed, with certain types of language promoting more realistic simulations than others. This possibility was supported by a series of experiments on the perception of implied motion in realistic and schematic scenes. Holmes and Wolff found that when an object’s support was suddenly removed (e.g., a pedestal beneath a potted plant disappeared), people appeared to simulate the effect of gravity, showing insensitivity to downward changes in the object’s position. Downward motion was much more likely to be simulated, however, when people viewed scenes that resembled line drawings than when they viewed scenes that resembled photographs. Holmes and Wolff also observed that linguistic processing of the scenes influenced subsequent simulation. After writing a verbal description of the scenes, participants also simulated motion in response to the photorealistic scenes. Further, the magnitude of the simulation effect was positively correlated