Towards a Dataset for Human Computer Communication via Grounded Language Acquisition
Venue: AAAI Workshop: Symbiotic Cognitive Systems

The Natural Language Processing, Artificial Intelligence, and Robotics fields have made significant progress towards developing robust component technologies (speech recognition/synthesis, machine translation, image recognition) advanced inference mechanisms that accommodate uncertainty and noise; and autonomous driving systems that operate seamlessly on our roads. In spite of this, we do not yet know how to talk to the machines we build or have them speak to us in natural language; how to make them smarter via simple, natural language instructions; how to understand what they are about to do; or how to work with them collaboratively towards accomplishing some joint goal. In this paper, we discuss our work towards building a dataset that enables an empirical approach to studying the relation between natural language, actions, and plans; and introduce a problem formulation that allows us to take meaningful steps towards addressing the open problems listed above. Context for our work Establishing a bidirectional connection between natural language and actions goals in the physical world requires mechanisms that will enable the joint acquisition of language and learning of planning policies; learning of mappings between natural language utterances and action sequences (understanding) and learning of mappings between action sequences and natural language (generation) Several research areas have addressed aspects of our requirements. In natural language understanding and language grounding, NLP researchers (Zettlemoyer and Collins 2005; Kwiatkowski et al. 2013; Reddy, Lapata, and Steedman 2014; Berant et al. 2013; Roth et al. 2014) have produced systems for extracting logical representations from text and for linking concepts and entity mentions to relations and entities in large scale knowledge bases or Wikipedia entries. In this body of work, the main focus is on establishing a relation between language, objects, and properties of objects. Within robotics, there is a long history of work on HumanRobot Interaction (Klingspor, Demiris, and Kaiser 1997; Goodrich and Schultz 2007; Mavridis 2015) which take commands and generate responses from a fixed vocabulary Copyright c (c) 2016, Association for the Advancement of Artificial Intelligence (www.aaai.org) All rights reserved. and grammar. Planning advances that increase robot autonomy do not correlate to free er form language or higher level abstractions as should be the case when acquiring language. Progress on language has largely focused on grounding visual attributes (Kollar, Krishnamurthy, and Strimel 2013; Matuszek et al. 2014) and on learning spatial relations and actions for small vocabularies with hard coded abstract concepts (Steels and Vogt 1997; Roy 2002; Guadarrama et al. 2013) Language is sometimes grounded into simple actions (Yu and Siskind 2013) but the data, while multimodal, is formulaic, the vocabularies are small, and the grammar is constrained. Recently the connection between less formulaic language and simple actions has been explored successfully in the context of simulated worlds (Branavan et al. 2009; Goldwasser and Roth 2011; Branavan, Silver, and Barzilay 2011; Artzi and Zettlemoyer 2013) and videos (Malmaud et al. 2015; Venugopalan et al. 2015) To our knowledge, there is no body of work on understanding the relation between natural language and complex actions and goals or from sequences of actions to natural language. As observed by Klingspor (1997) there is a big gap between the formulaic interactions that are typical of stateof the art human robot communications and human human interactions, which are more abstract. Proposed Problem Solution Sequences Data An intuitive characterization We have built a large corpus of Problem Solution Sequences (PSS)  A PSS consists of a sequence of images/frames that encode what a robot sees as it goes about accomplishing a goal. To instantiate our framework, we focus first on PSSs specific to a simple world that has blocks stacked on a table and a robot that can visually inspect the table and manipulate the blocks on it. In Figure 1, each image in the PSS corresponds to an intermediate block configuration that the robot goes through as it modifies the initial state block configuration into the final one. The PSS makes explicit the natural language instructions that a human may give to a robot in order to transform the configuration in an Imagei into the configuration in an Imagej the two configurations may correspond to one robot action (for adjacent states in the sequence) or to a sequence of robot actions (for non adjacent states) To account for lan