Automatic Categorization of Spatial Prepositions

Automatic Categorization of Spatial Prepositions Kate Lockwood (kate@cs.northwestern.edu) Ken Forbus (forbus@northwestern.edu) Daniel T. Halstead (halstead@cs.northwestern.edu) Jeffrey Usher (usher@cs.northwestern.edu) Qualitative Reasoning Group, Northwestern University, 2145 Sheridan Rd Evanston, IL 60208 USA Abstract Learning spatial prepositions is an important problem in spatial cognition. We describe a model for learning how to classify visual scenes according to what spatial preposition they depict. We use SEQL, an existing model of analogical generalization, to construct relational descriptions from stimuli input as hand-drawn sketches. We show that this model can distinguish between in, on, above, below, and left, after being trained on simple sketches exemplifying each preposition. Introduction Spatial reasoning is a skill central to many human tasks, as is being able to communicate about space. One way we share spatial information is through the use of prepositions to describe relationships between entities in the world. These utterances involve at minimum two objects: a reference object (the ground) and a located object (the figure) as well as the preposition that describes their relationship. The set of spatial prepositions in English is quite small when compared with other word categories; however computationally modeling the assignment of preposition labels to visual scenes remains a difficult and important problem. Many recent psychological studies have focused on understanding which properties of the figure and ground objects play a role in the assignment of spatial prepositions. Some of the properties studied are extracted directly from the spatial arrangement of objects and surface features. Spatial language has garnered so much attention since it is considered to be an important organizing structure for conceptual information (Talmy, 1983). Studies have also shown that children learn how to use spatial language through interactions with objects in the world and without negative evidence. In this paper, we automatically categorize simple two- dimensional geometric sketches based on the preposition that would best describe them. Sketching is particularly suited to studying this domain as our understanding of spatial terms is grounded in perception. Perceptual features can be automatically computed using sketching systems, thus removing a source of tailorability in modeling. For these experiments, we used sKEA (Forbus, Ferguson & Usher, 2001), the first open-domain sketching system. sKEA sidesteps traditional recognition problems by allowing users to conceptually label the glyphs in a sketch. We use this conceptual information along with visual properties of the ink itself to focus on understanding the relationships in the sketch. The possibilities for conceptual labels are limited only by the underlying database (currently a subset of the Cyc database containing over 35,000 concepts). In addition to the conceptual label, users can give each glyph a name to reference it by. Basic qualitative spatial relationships are extracted from the ink in the sketch (Forbus, Tomai & Usher, 2003). In sKEA, the frame of reference is also specified by allowing the user to select the view of the sketch (i.e., “looking from side”, “looking from another object”). We previously used sKEA as input into SpaceCase, a Bayesian model that assigned prepositions to individual sketches (Lockwood, Forbus, & Usher, 2005). In that model, update rules fired based on properties in the sketch such as animacy of the ground and figure objects. In that work, the rules were motivated by results from psychological studies indicating what properties of scenes were important for preposition assignment. In the experiments described here, we use sKEA to automatically compute a set of spatial relationships from sketches. These relationships are suggested by, and consistent with, those features which have been shown to influence spatial preposition judgments with human subjects. Analogical generalization is used to automatically create groupings based on the features we have extracted. The generalizations created group the sketches together based on the relationship (in, on, above, below, and left) between the two objects. Analogical Generalization We use SEQL (Skorstad, Gentner, & Medin, 1988; Kuehne, Forbus, Gentner, & Quinn, 2000) as our model of categorization. SEQL is a computer model of category learning that is based on Gentner’s (1983) structure- mapping theory of analogy and similarity. In SEQL categories are created through a process of successive comparison with incoming exemplars. The comparisons are carried out with SME, the Structure-Mapping Engine (Falkenhainer, Forbus & Gentner, 1986; Forbus, Ferguson & Gentner, 1994). For each category, a set of generalizations and exemplars is maintained. Each new