Motion in Vision and Language: Seeing Visual Motion can influence Processing of Motion Verbs
2013-01-01

Motion in Vision and Language: Seeing Visual Motion can influence Processing of Motion Verbs Carolin Dudschig (carolin.dudschig@uni-tuebingen.de) Department of Psychology, Schleichstr. 4 72076 Tubingen, Germany Jan Souman (jansouman@hotmail.com) Department of Psychology, Schleichstr. 4 72076 Tubingen, Germany Barbara Kaup (barbara.kaup@uni-tuebingen.de) Department of Psychology, Schleichstr. 4 72076 Tubingen, Germany Abstract In contrast to symbolic models of language understanding, embodied models of language comprehension suggest that language is closely connected with visual and motor processing. In the current study we show that motion words, such as rise or fall, are processed faster if displayed against a background of compatible motion (e.g., upward vs. downward random dot motion with 60% motion coherence). However, this interaction between semantic processing and visual processing only occurred if the word and the motion display were presented simultaneously. If the visual motion display was short-lived and occurred 100 or 200 ms after word-onset, no interactions between language and visual motion were found. We suggest that only in situations that do not allow ignoring or strategically suppressing the visual motion display, supra-threshold visual motion can affect language comprehension. Keywords: Language processing; motion verbs; vision; visual motion processing; embodiment; grounding. Introduction Embodied models of language understanding propose a close connection between language and perceptuomotor processes in the brain (e.g., Barsalou, 1999). Recently, compelling evidence supported the close association between language and other cognitive functions (e.g., Zwaan, Stanfield & Yaxley, 2002). In the motor domain converging evidence suggests that language facilitates compatible motor actions (e.g., Glenberg & Kaschak, 2002) and that language comprehension involves cortical motor areas that are also involved in performing the described actions (e.g., Hauk, Jonsrude, & Pulvermuller, 2004). For example, Glenberg and Kaschak showed that processing sentences such as “Close the drawer” can interfere with motoric responses incompatible with the motion implied in the sentence (e.g., arm movement towards my body). Similar effects have been reported in studies using motion verbs (e.g., rise, climb) or nouns implicitly implying a location (e.g., bird vs. shoe), whereby upward verbs and nouns facilitate upward arm movements (Dudschig, Lachmair, de la Vega, De Filippis, & Kaup, 2012a; Lachmair, Dudschig, De Filippis, de la Vega & Kaup, 2011). In contrast to the effects of language on motor processing, in the perceptual domain there is rather mixed evidence regarding the relation between language and visual processing. In particular, evidence regarding the influence of non-linguistic factors on language processing is rare. This direction of cause is particularly important, as these findings would suggest that mechanisms underlying non-linguistic processes are required and recruited during language processing. Studies in the visual domain typically investigate the influence of language on perceptual detection or discrimination tasks. For example, it has been shown that words referring to entities with a typical location (e.g., hat vs. shoe) can influence visual target perception in upper or lower screen locations (e.g., Dudschig, Lachmair, de la Vega, De Filippis, & Kaup, 2012b; Estes, Verges & Barsalou, 2008). Similar results have been reported for valence words (e.g., Meier & Robinson, 2004) and religious concepts (e.g., Chasteen, Burdzy & Pratt, 2010). Additionally, there have been studies demonstrating that visual simulation can also occur during sentence processing and subsequently affect visual discrimination performance (Bergen, Lindsay, Matlock & Narayanan, 2007). Recently, it has been shown that not only visual discrimination performance but also eye-movements can be affected by words referring to entities in the upper or lower field of vision (Dudschig, Souman, Lachmair, de la Vega, & Kaup, 2013). More specifically, upward saccades are faster following words referring to entities in the upper visual field (e.g., bird) and in contrast, downward saccades are faster following words referring to entities in the lower visual field (e.g., shoe). Importantly, the relation between language and visual processing was also reported in the other causal direction: Perceiving visual motion patterns can affect language processing. For example, Kaschak, Madden, Therriault, Yaxley, Aveyard, Blanchard and Zwaan (2005) first reported the effects of visual motion perception on language comprehension. In their study, participants viewed visual motion patterns (e.g., upward vs. downward moving