Speed of Processing Effects on Spoken Idiom Comprehension

Real-time Integration Of Gesture And Speech During Reference Resolution Ellen Campana (ecampana@bcs.rochester.edu) Department of Brain and Cognitive Sciences, University of Rochester Rochester, NY 14627, USA Laura Silverman (lauras@psych.rochester.edu) Department of Clinical and Social Sciences in Psychology, University of Rochester Rochester, NY 14627, USA Michael K. Tanenhaus (mtan@bcs.rochester.edu) Department of Brain and Cognitive Sciences, University of Rochester Rochester, NY 14627, USA Loisa Bennetto (bennetto@psych.rochester.edu) Department of Clinical and Social Sciences in Psychology, University of Rochester Rochester, NY 14627, USA Stephanie Packard (sp002m@mail.rochester.edu) Department of Brain and Cognitive Sciences, University of Rochester Rochester, NY 14627, USA immediately integrated with the content of the co-occurring speech (McNeill, Cassell, and McCullough, 1994). One factor that may have contributed to the lack of consensus is that gesture researchers have tended to focus on answering the question: is gesture communicative? (see Kendon, 1994 for a review). In doing so, they may have confounded two questions that could be considered independently: 1) do speakers intend to communicate using gesture in natural interaction? and 2) do comprehenders benefit from gesture when it naturally co-occurs with speech? For instance, some results suggesting a lack of intentionality have been taken as evidence that listeners’ do not use gesture during comprehension (Krauss et al., 1994). In the current study, we focus only on comprehenders. Our goal is to investigate how and when comprehenders make use of naturally co-occurring gesture and speech (leaving aside for the moment the question of whether speakers intend for gesture to be communicative). There is a growing body of evidence that listeners integrate many forms of extra- linguistic information when comprehending co-occurring speech, including embodiment constraints (Chambers, Tanenhaus, and Magnuson, 2004) gaze (Hanna and Brennan, in prep), and disfluency information (Ferreira, Lau and Bailey, in press; Arnold, Tanenhaus, Altmann, and Fagnano, 2004). Listeners take advantage of this information even when it is not intended by the speakers to be communicative. We would therefore expect listeners to take account of gestures as well. Much of this recent work has been done using the Visual World Paradigm (Tanenhaus et al. 1995; Cooper, 1974). Here, we apply this methodology to determine whether or not comprehenders benefit from gesture that co- occurs with speech Abstract There is some disagreement among researchers about the role of gesture in comprehension; whether it is ignored, processed separately from speech, used only when speakers are having difficulty, or immediately integrated with the content of the co- occurring speech. The present experiment provides evidence in support of immediate integration. In our experiment participants watched videos of a woman describing simple shapes on a display in which the video was surrounded by four potential referents: the target, a speech competitor, a gesture competitor, and an unrelated foil. The task was to “click on the shape that the speaker was describing”. In half of the videos the speaker used a natural combination of speech and gesture. In the other half, the speaker’s hands remained in her lap. Reaction time and eye-movement data from this experiment provide a strong demonstration that as an utterance unfolds, listeners immediately integrate information from naturally co- occurring speech and gesture. Background Non-deictic manual gestures are ubiquitous in language production. People produce gesture even when their interlocuters cannot see them (Alibali, Heath and Myers, 2001) and can experience difficulty speaking when their hands are restricted (Graham and Heywood, 1975). There is some consensus that the production system allows different aspects of a single message to be expressed simultaneously over, and distributed between, linguistic utterances and gestures (Alibali, Kita and Young., 2000) However, researchers disagree about the role of gesture in comprehension; whether it is ignored (Krauss, Dushay, Chen and Rauscher., 1995), processed separately from language (Goldin-Meadow and Singer, 2003), used only when speakers are having difficulty (Rauscher, Krauss and Chen, 1996), or