The Semantic Pictionary Project

The Semantic Pictionary Project Brent Kievit-Kylar (bkievitk@indiana.edu) Cognitive Science Program Indiana University, Bloomington, Indiana USA Abstract Here we describe the Semantic Pictionary Project—a set of online games and tools designed to collect large amounts of structured data about the object characteristics and perceptual properties of word referents. The project hinges on the use of encoding-decoding games and a set of creation tools to capture data using online crowdsourcing. We describe the architecture of the basic tools behind the games, the structure of the resulting data, and how this information may be integrated into existing statistical semantic models. We also describe two validations using data collected from one of the tools (2D Geon Pictionary) demonstrating typicality effects in the metrics of raw Geon objects created by subjects, and unique variance in the predictions of word pair metrics over currently used linguistic and property data. Keywords: Geon; natural language processing; crowdsourcing; semantic space models; embodied cognition. Introduction Humans learn about the meanings of words and larger discourse units from repeated experience with both linguistic and perceptual information. However, current models of lexical semantics focus only on learning from linguistic structure using statistical abstraction algorithms. Part of the problem is the lack of realistic structured data containing information about the perceptual structure of word referents. Text is plentiful, but usable object structure data currently are not. Given the ideological movement towards models of embodied lexical representation, model development is currently being held up due to a lack of structured human data containing configural object and property information about concrete word referents. Here, we describe the NSF-funded Semantic Pictionary Project, an online approach to the problem of data capture that makes use of the paradigms of crowdsourcing and online gaming to gather data containing the perceptual structure of word referents. The online games may be played at www.SemanticPictionary.org. Statistical semantic models (SSMs; e.g., Landauer & Dumais, 1997) have recently been attacked as implausible cognitive models because they learn from only linguistic information and are not grounded in perception and action, contrary to the literature on embodied cognition, and this limits their ability to account for human behavior on semantic tasks (for a review of the debate, see de Vega, Graesser, & Glenberg, 2008). The inadequacy of SSMs as cognitive models punctuates the current movement in cognitive science towards models of embodied cognition. There is a growing body of both behavioral and Michael N. Jones (jonesmn@indiana.edu) Department of Psychological and Brain Sciences Indiana University, Bloomington, Indiana USA neuroimaging research demonstrating that when humans process words (in isolation or in context) they automatically activate sensorimotor information about the perceptual features of the word’s referent, how it is commonly used, and physical contexts in which it has been experienced (for a review, see Riordan & Jones, 2010). A large number of behavioral experiments also demonstrate convincing evidence that sensorimotor experience becomes an inseparable part of a word’s lexical representation, including information about object features (color, shape, motion, etc.). Perceptual information is an inherent part of the semantic organization of the human lexicon, but much of this information cannot be learned from statistics in a text corpus—it must be learned from multisensory experience. Perceptually grounded SSMs are now emerging in the cognitive science literature (e.g., Andrews, Vigliocco, & Vinson, 2009; Recchia & Jones, 2010; Steyvers, 2010). As a proxy for sensorimotor perception, these new integrative models use norms of human-generated properties (e.g., McRae et al., 2005). These norms are collected by asking hundreds of subjects to produce the physical properties (internal and external parts), appearance, sounds, smells, tastes, functional properties, categorical membership, etc. for concrete nouns and event verbs based on multisensory experience. A property vector for a word is then created by aggregating across subjects. For example, the property will have a high probability for dog and cow, but a low probability for centipede, and a zero probability for strawberry. However, is a highly salient property of strawberry and not for dog. The development of perceptually grounded SSMs is currently being held up by a lack of data. The overall goal of the Semantic Pictionary Project is to collect large amounts of object and property data online using a combination of crowdsourcing and our new encoding-decoding games, and to make the large datasets available to researchers to develop superior grounded semantic models. The Semantic Pictionary Paradigm The Semantic Pictionary paradigm is a two-stage task with self-correction built in. In the first stage, subjects are presented with a target word selected from the high- concreteness/early-AoA nouns from the MRC Psycholinguistic Database. Subjects are then provided a tool to make a representation of this noun, with the goal of making a depiction such that another subject could guess what word is being represented (similar to the popular Pictionary game). That representation is then given to a