Autonomous Neural Dynamics to Test Hypotheses in a Model of Spatial Language

Autonomous Neural Dynamics to Test Hypotheses in a Model of Spatial Language Mathis Richter (mathis.richter@ini.rub.de) Jonas Lins (jonas.lins@ini.rub.de) Sebastian Schneegans (sebastian.schneegans@ini.rub.de) Yulia Sandamirskaya (yulia.sandamirskaya@ini.rub.de) Gregor Sch¨oner (gregor.schoener@ini.rub.de) Institut f¨ur Neuroinformatik, Ruhr-Universit¨at Bochum, 44870 Bochum, Germany Abstract Resolving relational spatial phrases requires that a coherent mapping emerges between a visual scene and a triad of two objects and a relational term. We present a theoretical ac- count that solves this problem based on neural principles. A neural dynamic architecture represents perceptual information in activation fields that make detection and selection deci- sions through neural interaction. Activation nodes and their connectivity to the perceptual fields represent concepts. Dy- namic instabilities enable the autonomous sequential organi- zation of the processing steps needed to resolve relational spa- tial phrases. These include bringing visual objects into the at- tentional foreground, performing spatial transformations, and making matching decisions. We demonstrate how the neural architecture may autonomously test different hypotheses to re- solve relational spatial phrases. We discuss how this neural process account relates to existing theoretical perspectives and how to move beyond the entry point sketched here. Keywords: spatial language; sequence generation; autonomy; hypothesis testing; neural dynamics; Dynamic Field Theory Introduction Language enables humans to communicate about shared en- vironments. For instance, I may use language to direct your attention to an object in a visual scene. When several simi- lar objects are visible such as in Fig. 1a, using object iden- tity (“cup”) or feature (“red”) alone is not sufficient. A rela- tional spatial phrase, for example “the red cup to the left of the green cup”, resolves ambiguity in such situations. Even in the scene in Fig. 1b, in which no object can be singled out by feature reference, this phrase uniquely specifies one of them. A typical relational phrase like the one above consists of a (a) (b) Fig. 1: Visual scenarios affording the use of spatial language. target (the red cup) and a reference (the green cup), relative to which a relational term (to the left) is applied. Interpret- ing such a phrase may require that different pairs of objects be examined. Psychophysical evidence from visual search tasks suggests that this happens in sequence rather than in parallel (Logan, 1994). Selecting the reference and target ob- ject of such a pair also appears to happen sequentially. This is suggested by characteristic shifts of attention found using EEG measurements (Franconeri, Scimeca, Roth, Helseth, & Kahn, 2012), eye-tracking (Burigo & Knoeferle, 2011), and behavioral cuing (Roth & Franconeri, 2012). The processing steps involved in interpreting a relational spatial phrase include binding each object to its role, cen- tering the reference frame on the reference object, mapping the spatial term onto this reference frame, and assessing the match of the target object with the spatial term (Logan & Sadler, 1996). While such discrete processing steps appear natural in information processing terms, they require an ex- planation in neural systems. At the population level that is relevant to behavior, neural activity evolves continuously in time. The flow of activation is determined by the structure of neural networks. Flexibility is thus an achievement in neural processing, not a given. In previous work we have provided the basis for realizing some of these processing steps in ac- cordance with neural principles (Lipinski, Schneegans, San- damirskaya, Spencer, & Sch¨oner, 2012). This work is based on the framework of Dynamic Field Theory (DFT; Schnee- gans & Sch¨oner, 2008), in which activation peaks are units of representation. The model addresses the attentive selection of target and reference objects and proposes a neural archi- tecture that transforms the location of the target object into a frame centered on the reference object. Spatial terms are en- coded relative to that frame as patterned neural connections. While the neural processes of bringing objects into the at- tentional foreground and activating spatial terms unfold au- tonomously, the sequential order of these different operations is controlled through signals from outside the system. In this paper we provide a fully autonomous neural dy- namic architecture that generates sequences of processing steps to interpret and generate relational spatial language. Within the framework of DFT, we take inspiration from ear- lier work on the autonomous generation of behavioral se- quences (Sandamirskaya & Sch¨oner, 2010; Richter, San- damirskaya, & Sch¨oner, 2012). The key idea is that elemen- tary processing steps are characterized by certain aspects that can be implemented in a neural system: The neural represen- tation of an intention drives activation in those neural struc- tures that are relevant for executing the processing step. The resulting changes in activation states are detected through a condition of satisfaction, which indicates the successful com-