An Embodied Real-Time Model of Language-Guided Incremental Visual Search

An Embodied Real-Time Model of Language-Guided Incremental Visual Search Matthias Scheutz and Evan Krause and Sepideh Sadeghi {matthias.scheutz,evan.krause,sepideh.sadeghi}@tufts.edu Human-Robot Interaction Laboratory, Department of Computer Science, Tufts University Medford, MA 02155, USA Abstract A recent body of work has demonstrated that the incremen- tal presentations of linguistic search cues can speed up visual processing in conjunctive visual search. In this paper, we in- vestigate different processing configurations using a real-time embodied computational model and demonstrate that, differ- ent from previous hypotheses, the same incremental process- ing configuration can explain all experimental conditions. Keywords: Incremental interactive processing, embodied real-time model, natural language and vision interaction Introduction A large body of work in cognitive science has demonstrated that much of human information processing is interactive and incremental: “interactive” means that information is ex- changed and shared among multiple processes; “incremen- tal” means that the received information is integrated as it be- comes available. Hence, interactive incremental processing modules can incorporate information from other modules as constraints in their own processing and thus potentially finish their processing sooner. A well-studied case of such interactive incremental pro- cessing is the interaction between visual and natural language processes. Converging evidence from studies using, in par- ticular, the “visual words paradigm” demonstrate that vision and natural language processing in humans are highly interac- tive and incremental, being able to utilize constraints from the other modality to reduce processing effort and improve pro- cessing performance (Eberhard, Spivey-Knowlton, Sedivy, & Tanenhaus, 1995). For example, a visual search process at- tempting to find a target object in a visual scene such as a particular pen on a cluttered desk can be modulated through natural language instructions that provide additional informa- tion about the object (e.g., “small black”), leading to a more targeted, faster search (Spivey, Tyler, Eberhard, & Tanenhaus, 2001; Krause, Cantrell, Potapova, Zillich, & Scheutz, 2013). Conversely, visual processing of a scene can influence natu- ral language processing by helping to disambiguate otherwise ambiguous referential phrases such as the syntactic ambigu- ity due to different possible prepositional attachments in “put the black pen on the book on the table” where the black pen could be put either on the table or on the book that is on the ta- ble (Eberhard et al., 1995; Scheutz, Eberhard, & Andronache, 2004; Brick & Scheutz, 2007). While various theoretically motivated hypotheses have been proposed about an underlying processing architecture that could enable such incremental natural language and vi- sion interaction and information integration, only a few com- putational models actually demonstrate possible computa- tional mechanisms (Scheutz et al., 2004; Hamker, 2004; Brick & Scheutz, 2007; Chiu & Spivey, 2012; Krause et al., 2013). However, computational models are often necessary to show that conclusions drawn about the processing archi- tecture based on experimental evidence or theory alone might not be warranted. In this paper, we present an embodied real-time model of interactive incremental vision and natural language process- ing that can explain previous experimental findings in a novel way by showing that divergent results found in different ex- perimental conditions by Spivey et al. (2001) might not be due to differences in processing configurations (such as serial vs. parallel), but rather the specific effects of these experi- mental manipulations on the same processing configuration. We start by reviewing some of the empirical evidence for the hypothesis that natural language can incrementally con- strain vision processing and describe, in particular, the exper- iments in (Spivey et al., 2001) which we use for our model simulations. Next, we introduce the model architecture and provide a more detailed description of its vision system which is critical for the replication of the human data. We then spec- ify the simulation setup, which used the same human stimuli as Spivey et al. (2001), and report the results from extensive simulation experiments with different configurations of the processing system. The analysis of the simulation data con- firms many of the expected properties, but also shows that the same configuration can explain different experimental con- ditions that have been assumed to be the result of different processing configurations. This point is further elaborated in the subsequent discussion section and summarized in the con- clusion which also points to future work. Background and Motivation It has long been hypothesized that early stages of bottom- up visual processing are highly parallel as single-feature vi- sual search is not affected by the number of co-present dis- tractors, while later stages must include a “serial bottleneck” since conjunctive visual search (assumed to tap into later pro- cessing stages) takes longer as the number of distractors in- creases (Wolfe, 2007). While various stimuli properties can affect search speed in conjunctive search, Spivey and col- leagues demonstrated in a series of experiments that the in- cremental presentation of linguistic search cues can reduce the effect of distractors in the visual search process (Spivey et al., 2001; Reali, Spivey, Tyler, & Terranova, 2006; Chiu & Spivey, 2011, 2012). They hypothesized that the incremen- tal presentation of search cues (which is natural in spoken language) enforced a serialization of the search process, al- lowing search results based on the first cue to be utilized in