Putting Geometry and Function Together — Towards a Psychologically-Plausible Computational Model for Spatial Language Comprehension

Putting Geometry and Function Together — Towards a Psychologically-Plausible Computational Model for Spatial Language Comprehension Kenny R. Coventry 1 (kcoventry@plymouth.ac.uk), Angelo Cangelosi 2 (angelo@soc.plymouth.ac.uk), Dan Joyce 2 (danj@soc.plymouth.ac.uk) and Lynn V. Richards 1 (lynnr@soc.plym.ac.uk) Centre for Thinking and Language, Department of Psychology & 2 Centre for Neural and Adaptive Systems, School of Computing, University of Plymouth, Drake Circus, Plymouth PL4 8AA, United Kingdom Describing the position of objects in space necessitates a mapping between the spatial representation(s), computed by the visual system, and the language processing system. However, it turns out that spatial description is influenced not only by where objects are in space, but also by the functions that objects afford, and the functional relations between objects. For example, the preposition at in the woman is at her desk indicates not only that the woman is in close proximity to the desk (a topological-geometric relation), but that she is likely to be working there (an extra- geometric functional relation). Indeed, there is much empirical work showing that meaning of spatial prepositions across a range of languages involves the instantiation of both geometric and extra-geometric factors (e.g., Carlson- Radvansky & Radvansky, 1996; Coventry, Prat-Sala & Richards, 2001). However, how geometric and extra- geometric constraints combine is an open question. Regier and Carlson (2001) present a computational account, the attentional vector sum (AVS) model, which grounds the preposition above in a mechanism analogous to population vector codes in the neural model of Georgopolous et al (1986). However, Regier and Carlson deal only with geometric computations over the visual scene. We present a new computational model which attempts to deal with the spatial prepositions in, on, over, under, above and below and extends processing of the visual scene to include functional factors parasitic upon object knowledge. One possibility is that object knowledge can be used as a means of weighting parts of geometric processing, as is suggested by Regier, Carlson and Corrigan (in press). In contrast, Coventry and Garrod (in press) suggest that separate geometric and extra-geometric processes are operational in parallel, and come together in a situation model. Our approach introduces cognitive-functional constraints by extending Ullman s (1984) notion of visual routines to include operations on dynamic rather than static visual input (cf. Cavanagh et al, 2001). We use neuropsychologically-inspired implementations of connectionist models (cf. Regier, 1996). Based on evidence of motion and spatial-frequency processing in areas V1-V4, the MT, and interactions from regions implicated in object- recognition, such as the IT cortex (Edelman, 1999), we construct a model which might account for extra-geometric and geometric factors in one computational system. Developmental accounts of an infant s understanding of concepts such as geometry (spatial relations), dynamics (e.g. gravity, containment and object constancy), and object individuation and identification constrain the training of relevant parts of our model. To give an example, the containing part of a mug is usually taken to be the part the liquid is poured into, and not the semi-circular handle. By watching interactions between mugs and liquids, we induce a dynamic visual routine , and a representation of the object over time. These routines and representations can then be deployed in future processing, for example, to generalize to similar objects in the absence of functional interactions. Initial results show that the computational model performs similarly to reference data, obtained from new experimental data on spatial preposition comprehension tasks. Acknowledgments This research was funded by EPSRC grant number GR/N38145, awarded to the first two authors. References Carlson-Radvansky, L. A. & Radvansky, G. A. (1996). The influence of functional relations on spatial term selection. Psychological Science, 7(1), 56-60. Cavanagh, P., Labianca, A.T. , & Thornton, I.M. (2001) Attention-based Visual Routines: Sprites, Cognition, Coventry, K. R. & Garrod, S. C. (in press). Saying, seeing and acting. The psychological semantics of spatial prepositions. Taylor Francis: Psychology Press. Coventry, K. R. , Prat-Sala, M. & Richards, L. V. (2001). The interplay between geometry and function in the comprehension of over , under , above and below . Journal of Memory and Language . 44, 376- Edelman, S. (1999). Representation and Recognition in Vision. The MIT Press. Regier, T. (1996). The Human Semantic Potential. Spatial language and constrained connectionism. Cambridge, MA: MIT Press. Regier, T. & Carlson, L. A. (2001). Grounding spatial language in perception: An empirical and computational investigation. Journal of Experimental Psychology: General, 130(2), 273-298. Regier, T. , Carlson, L. A. & Corrigan, B. (in press). Attention in spatial language: Bridging geometry and function. To appear in L. A. Carlson & E. van der Zee (Eds.), Functional features in language and space: Insights from perception, categorization and development. Oxford University Press. Ullman, S. (1984) Visual Routines, Cognition, 18, 97-159.