Perceived Affordances as a Substrate for Linguistic Concepts

Perceived Affordances as a Substrate for Linguistic Concepts Peter Gorniak (pgorniak@media.mit.edu) Deb Roy (dkroy@media.mit.edu) MIT Media Laboratory Cambridge, MA 02139 USA Abstract results show that this implementation can accurately predict how human players respond to spoken commands issued by their human partners, demonstrating the viability of our ap- proach for modeling the interpretation of context-dependent language on the basis of perceived affordances. We propose the use of perceived affordances to computation- ally model linguistic concepts in situated language use. Per- ceived affordances are mental structures that capture the rela- tionship between the language user and the embedding situa- tion. We employ computational models of human perceived affordances to understand situated language: language that dy- namically depends on the current physical environment and the goals and plans of communication partners. To support this theory of situated language understanding, we describe an implemented system that understands verbal commands situ- ated in a virtual gaming environment. The implementation uses probabilistic hierarchical plan recognition to generate per- ceived affordances. We have evaluated the system on its ability to correctly interpret free-form spontaneous verbal commands recorded from unrehearsed game play between human players, and find that it is able to “step into the shoes” of human players and correctly respond to a broad range of verbal commands in which linguistic meaning depends on social and physical con- text. Related Work There exist a number of competing or complementary theo- ries of concepts. Laurence and Margolis as well as Prinz give overviews of the proposals and the debates surrounding them (Laurence and Margolis, 1999; Prinz, 2002). Many theories of concepts adopt a fundamental bias: that the use of concepts by an actual language using system, and thus their connection to the world, is secondary to their internal structure and for- mal properties. This bias is exhibited most strongly by the traditional definitional theories. Where they acknowledge a link to the world at all, for example in the form of observa- tion sentences (Carnap, 1932), the link is portrayed as simple and passive. In fact, the only function usually considered is that of categorization, a feature shared with Prototype The- ory (Rosch, 1975). Closest in spirit to our proposal here is perhaps Theory-Theory (Carey, 1985), but only in so far as it acknowledges the importance of viewing concepts as embed- ded in a structure used to reason about particular domains. While Theory-Theory emphasizes internal mental theories, however, it leaves unspecified their connection to the world except, again, in the case of categorization where it explains a human tendency towards essentialism. All these theories thus largely deal with internal mental structures and at most tackle the problem of categorization using these mental structures. Categorization is only one possible use of concepts, however, and arguably not the most basic one. Concepts are intimately tied to the goals of the conceptualizer and thus a theory of concepts should not only explain categorization behaviour, but more importantly cover intentional aspects of concepts such as the predictions they imply, and generally their use in achieving goals. In a similar vein, the computational theories and systems addressing the problem of how a word comes to be about the world, treat this problem as one of adding sensory categoriza- tion functionality to a definitional theory of concepts (Harnad, 1990). There are a number of systems that have been built ac- cording to this paradigm, including Regier (1996)’s work on learning words for spatial relations, Plunkett et al. (1992)’s image labelling investigations, Yu, Ballard, and Aslins model of word learning Yu et al. (2003), as well as some of our own previous models of word grounding (Roy et al., 2002; Roy, 2002). These systems often address additional aspects of this type of grounding, such as how categories are learned by the language user (Regier, 1996; Roy, 2003) and how concepts grounded in this way can be combined (Gorniak and Roy, 2004). We have previously presented work on understanding situated language using plan recognition (Gorniak and Roy, 2005a,b), but that work separated the linguistic understanding from the plan recognition, whereas they are tightly coupled Introduction We have recently introduced the theory of Affordance-Based Concepts (ABCs) (Gorniak, 2005; Gorniak and Roy, 2006). This theory has at its core the intentional link between lan- guage users and the world treating predicted interactions as the basic building block for conceptual representation. Other theories often limit themselves to specifying the structure of concepts as opposed to how concepts come to be about the world. To tightly couple the internal structure of concepts with their intentional and functional use, the theory proposes that each element of a concept must make a prediction about the world, thus crossing over from the mind to the world. Ev- ery concept thus becomes both a property of the language us- ing system, and of its relation to the embedding world. These structural elements are called perceived affordances, yielding a theory of Affordance-Based Concepts. In this paper, we introduce a computational model that employs plan recognition as a mechanism for finding and ranking the perceived affordances of a person engaged in co-operative tasks. Situated language interpretation is mod- eled as a process of filtering perceived affordances. In effect, the complete meaning of linguistic expressions is only un- derstood when words are meshed with the situation in which they are used. To evaluate the model, we describe an im- plementation of the model that interprets situated language collected from people playing a multiplayer computer game. This implementation is based on a probabilistic, hierarchical plan recognizer in the form of an Earley parser, To understand the game players’ language, a syntactic parser filters the pre- dictions (perceived affordances) this plan recognizer makes at the time of an utterance. Language understanding thus is cast as a filtering process on perceived affordances, which are ele- ments that naturally span the boundary between the language user’s mind (by virtue of being subjective predictions tak- ing into account the language user’s goals) and the world (by virtue of taking into account the structure of the world). Our