Meaning in Words, Gestures, and Mental Images

Meaning in Words, Gestures, and Mental Images Paolo Bernardis (paolobernardis@units.it) Department of Psychology, University of Trieste Via S. Anastasio, 12. 34134 Trieste Italy Nicoletta Caramelli (nicoletta.caramelli@unibo.it) Department of Psychology, University of Bologna Viale Berti Pichat, 5. 40127 Bologna Italy Abstract In this study, we provide evidence for a cross-modal interaction between the meaning of pantomimes and words when the visuo-spatial and perceptual information of these last is enhanced. We recorded behavioral and electrophysiological responses with a cross-modal repetition priming. Pantomimes of objects and actions were used to prime visually presented nouns and verbs with an image formation task. The behavioral results showed that the image formation times of words primed by a preceding gesture were faster in the matching meaning condition than in the mismatching one. Electrophysiological results confirmed the interaction between gesture and word meanings showing a N400 localized all over the scalp with a peak on the left anterior hemisphere. Overall, these results support the idea of a tight interplay between the meaning of pantomimes and words when perceptual information is enhanced in words at both the behavioral and neurophysiological levels. Keywords: Image formation; Gestures; Words; Semantic priming; Event-related potentials. Introduction Recently, researchers’ interest has focused on the facilitation effect of iconic and co-verbal gestures on a number of cognitive tasks. Considerable evidence has been collected showing that the function of the gesture system cannot be reduced only to a mere support to the verbal system, providing lexical access in a tip-of-the-tongue state, nor simply to an aid in understanding language (see Butterworth & Hadar, 1989; Hadar & Butterworth, 1997; Krauss, Chen, & Chawla, 1996). Instead, it has been convincingly shown that gesture is deeply rooted in overall cognition, as it helps to highlight stages of learning (Alibali, & DiRusso, 1999; Church & Goldin-Meadow, 1986; Perry, & Elder, 1996; Pine, Lufkin, & Messer, 2004), problem solving strategies (Alibali, Bassok, Solomon, Syc, & Goldin-Meadow, 1999), how attention is directed (Goodwin, 2000) and memory improvement (Feyereisen, Developing hints by McNeill’s (1992) that gesture reflects the imagistic mental representation activated at the moment of speaking, Kita (2000) advanced the Information Packaging Hypothesis. In this perspective, the specific characteristics of the two systems complement each other. The visuo-spatial and holistic character of gesture and the segmented and linear character of language are combined in conveying complex meaning. Gesture helps speakers package spatial information into units appropriate for verbalization. In this view, the gesture system has been shown to cooperate with the language one in the conceptual planning of the message to be verbalized (Alibali, Kita, & Young, 2000; Hostetter, Alibali, & Kita 2006). This tight interaction between the gesture and language systems was also found to characterize language comprehension and evidence thus far collected suggests that speech and gesture establish a highly integrated system of communication (Beattie & Shovelton, 1999; Cassell, McNeill, & McCullough, 1999; Clark, 1996; Goldin- Meadow, Wein, & Chang, 1992; Kelly, Barr, Church, & Lynch, 1999; Kelly & Church, 1998; Krauss, 1998; Krauss, Morrel-Samuels, & Colasante, 1991; McNeill, 1992). This conclusion attained on behavioural grounds has been confirmed and complemented with converging electrophysiological data. The full understanding of a message in the brain is the result of a qualitatively similar elaboration of integrated types of information in a dynamic large scale neural network. This integration process involves information from world-knowledge, co-speech gestures, pictures, speaker’s identity derived from voice characteristics and information from a preceding discourse (for a review, see Willems & Hagoort, 2007). The Event Related Potentials (ERPs) technique has been largely used to investigate the processing of meaning. It is widely agreed that one particular component, the N400, is a general index of semantic integration that is yielded by verbal and pictorial stimuli presented both uni-modally and cross-modally (e.g. Willelms et al. 2008). The N400 distribution over the scalp can change depending on the type of the stimuli. Verbal stimuli usually produce a parietal distribution (Kutas and Hillyard, 1980, 1984) and pictorial stimuli a frontal N400 distribution (West and Holcomb's 2002; Ganis & Kutas, 2003). Hamm, Johnson and Kirk (2002), instead, found that with word-picture pairs the d- N400 was localized in the parietal areas (the d-N400, or difference wave, is the result of a subtraction between semantically congruent and incongruent ERPs). In this framework, a previous study aimed at showing the interactions between iconic gestures (pantomimes) and visually presented words showed an interference effect between pantomimes and words with matching meanings instead of the expected priming effect at the behavioral level (Bernardis, Salillas & Caramelli, 2008). On the contrary, the pattern of the electrophysiological activation clearly showed