Scan Patterns on Visual Scenes predict Sentence Production

Scan Patterns on Visual Scenes predict Sentence Production Moreno I. Coco (M.I.Coco@sms.ed.ac.uk) and Frank Keller (keller@inf.ed.ac.uk) School of Informatics, University of Edinburgh 10 Crichton Street, Edinburgh EH8 9AB, UK Abstract A range of cognitive modalities are involved in everyday tasks, which raises the question to which extend these modalities are coordinated. In this paper, we focus on two particular aspects of this coordination: linguistic structure and visual attention during sentence production, based on the hypothesis that sim- ilar scan patterns are associated with similar sentences. We tested this hypothesis using a dataset from an eye-tracking ex- periment in which participants had to describe photo-realistic scenes. We paired each sentence produced with the corre- sponding scan pattern, and computed a range of similarity measures for both modalities. Correlation and mixed model analyses confirmed that trials involving similar scan patterns also involve similar sentences productions. This was true for all pairs of linguistic and scan pattern similarity measures we investigated. The result holds both before and during sentence production, and for within-scene and between-scene analyses. Keywords: scan patterns; sentence production; eye-tracking; sentence similarity. Introduction Everyday tasks demand the coordination of a range of cogni- tive modalities. If the task is to make tea, for example, then motor actions (e.g., arm-hand movement) and visual atten- tion (e.g., looking at the pot) have to be coordinated (Land, 2006). This implies that if two different persons perform the same task, they will do so in a similar way. It follows that the sequence of scan patterns, i.e., eye fixations across spa- tial locations in temporal order (Noton and Stark, 1971) as well as the sequence of motor actions, will be similar across participants (Land, 2006). In this paper, we investigate whether a similar evidence of cross-modal coordination can be found when vision and lan- guage have to be coordinated. In particular, we focus on the similarity between scan patterns and linguistic structures in a language generation task. In the visual cognition literature, similarity of scan patterns has not received much attention, mainly because of the high variability across participants (Henderson, 2003). There are some results, however, that point toward a range of visual fac- tors that can trigger similarity. Often, these factors are related to the task (Castelhano et al., 2009) and to the degree of cross- modal interactivity required to perform it. In tasks with a low level of interactivity, i.e. free viewing, visual attention is mainly guided by exogenous factors like saliency (Itti and Koch, 2000): a measure of visual promi- nence based on low-level features (color, intensity and ori- entation). A free viewing task does not require visual atten- tion to interact with knowledge-based (top-down) informa- tion. The low interactivity of free viewing means the visual responses are driven by exogenous visual mechanisms while minimizing the need for cross-modal coordination. Different patterns of visual attention emerge in other vi- sual tasks, such as memorization or imagery (Humphrey and Underwood, 2008), where participants are asked to memo- rize images in preparation for a recall phase. In the recall phase, despite the absence of the original stimuli (preventing bottom-up effects), scan patterns on a blank screen were more similar across participants within the same scene than across different ones (Humphrey and Underwood, 2008). In this case, the task requires an endogenous control of visual atten- tion through top-down knowledge, i.e., scene layout, contex- tual information, and even semantic relations between objects (Hwang et al., 2009). Thus, exogenous bottom-up effects are overridden by endogenous contextual guidance effects. These results, consistent with similar findings from visual search studies (Yang and Zelinsky, 2009), suggest that in tasks requiring endogenous control, categorical and seman- tic information is activated. It seems plausible that this en- dogenous access to categorical information is activated dur- ing daily actions (Land, 2006); e.g., categorical knowledge about the tea pot (i.e., it has a handle to grasp) is necessary to allow cross-modal coordination between visual attention and motor action. It is important to notice that this information has a direct link with language processing. Categorical information, in fact, is typically expressed verbally, and drives linguistic tasks such as scene description. It seems likely that the same mech- anism, based on categorical information, which allows coor- dination between motor-action and visual attention might also underlie the coordination between language processing and visual attention. Previous research has looked at the interaction between vision and language principally using the visual world paradigm (VWP, Tanenhaus et al. 1995), an eye-tracking paradigm which has demonstrated clear links between the processing of certain linguistic constructions and the access to visual contextual information (Knoeferle and Crocker, 2006). Research in this field suggests a tightly coupled re- lation between vision and language, but previous works has largely focused on specific psycholinguistic phenom- ena (e.g., attachment ambiguity), rather than uncovering the shared mechanisms by which this interaction takes place. We explain this coupled relation assuming a categorical interface which coordinates the cross-modal, visual and linguistic, in- teraction. In this paper, we investigate the extent to which visual and