Verb-action versus role relations congruence effects: Evidence from ERPs in picture-sentence verification

Verb-action versus role relations congruence effects: Evidence from ERPs in picture-sentence verification Pia Knoeferle (knoeferl@cit-ec.uni-bielefeld.de) Cognitive Interaction Technology Excellence Cluster Bielefeld University, 33615 Bielefeld, Germany Thomas P. Urbach (turbach@cogsci.ucsd.edu) Department of Cognitive Science University of California San Diego La Jolla, USA Marta Kutas (kutas@cogsci.ucsd.edu) Department of Cognitive Science University of California San Diego La Jolla, USA Abstract Comprehenders can rapidly use both their linguistic knowl- edge and different kinds of information in visual context dur- ing language comprehension. Little is known, however, about the relative time courses and mechanisms by which different kinds of visual information influence language comprehen- sion. We recorded event-related brain potentials (ERPs) as participants read a subject-verb-object sentence and verified whether or not it matched different (verb-action versus role re- lations) aspects of a recently viewed picture. When the verb- action did not match the depicted action, we replicated larger N400s (300-500ms) over centro-parietal scalp to the verb (300- 500 ms) relative to the responses for matches. In contrast, ERP effects to role-relation mismatches (a person depicted as under- going an action but described as performing it) qualitatively differed from and occurred prior to the verb-action congru- ence N400. Our findings implicate at least two temporally dis- tinct mechanisms governing picture-sentence verification pro- cesses. Keywords: sentence-picture verification; visual context ef- fects; event-related brain potentials; Introduction Information in visual context can rapidly influence online language comprehension and ambiguity resolution (e.g., Alt- mann, 2004; Knoeferle, Habets, Crocker, & M¨unte, 2008; Tanenhaus, Spivey-Knowlton, Eberhard, & Sedivy, 1995). Recently, researchers also have begun to examine picture- sentence congruence processes even when sentences are structurally unambiguous and do not necessarily globally match visual context (e.g., Knoeferle, Urbach, & Kutas, 2009; Vissers, Kolk, van de Meerendonk, & Chwilla, 2008; Wassenaar & Hagoort, 2007). The motivation for these stud- ies is that determining the correspondence between what is said and how things are, appears to play a central part in nat- ural language processing: Positive verification may be read- ily inferred from, e.g., expressions of agreement (So I heard) while failures to verify may be inferred from corrections and expressions of disbelief, requests for clarification and the like (e.g., Well no, actually what happened was ., Are you sure? ). Psycholinguistic research on verification processes is by no means a recent endeavour: Just and Carpenter (1971) found that participants’ verification latencies were shorter when the color of the dots on an image (red vs. black) matched than mismatched the color adjective in a corresponding sentence (henceforth ”congruence effects”, see also, e.g., Clark & Chase, 1972). To account for these findings and a range of others, Carpenter and Just (1975) developed the Constituent Comparison Model (CCM) of sentence-picture comparison processes. The model operates via a serial mechanism that incrementally compares representations of sentence ([AFF, (RED, DOTS)]) and picture (“black dots”) constituents. The comparison proceeds from inner to outer representations (in this case right to left). When a mismatch is found (here for the inner representations), it is indexed “-”; the truth value is changed to “false”, and the comparison process is reini- tialized, resulting in one extra comparison step (and hence longer response times) relative to a match (e.g., “red dots”). The output of that comparator process is the truth value of the comparison and response time values. This verification model does not specify the time course of constructing the representations of verbal information as the sentences are read, and it is unclear to which extent the constituent-wise comparator mechanism implies incremental comprehension processes. Recent event-related brain poten- tial (ERP) research, however, suggests that congruence pro- cessing is incremental, i.e., ongoing during (not merely after) word-by-word sentence processing, and furthermore can be systematically related to end-of-sentence verification times (Knoeferle et al., 2009). This was evidenced by finding (a) re- liable congruence effects in ERPs as soon as a word (e.g., the verb) that mismatched aspects of a preceding visual context (e.g., a depicted action) was encountered; (b) reliable con- gruence effects in verification time response latencies; and (c) reliable correlations between a participant’s ERP and ver- ification time congruence effects. The incremental ERP congruence effects at the verb are, in principle, compatible with the CCM comparator mecha- nism. One may question, however, to what extent the CCM can account for verification processes during (rather than af- ter) the sentence. Specifically, it is unclear whether all aspects of picture-sentence mismatch processing are adequately ac- counted for by a single comparator mechanism as suggested by the CCM.