CoDraw: Collaborative Drawing as a Testbed for Grounded Goal-driven Communication

1 INTRODUCTION . Building agents that can interact with humans in natural language while perceiving and taking actions in their environments is one of the fundamental goals in artificial intelligence . One of the required components , language understanding , has traditionally been studied in isolation and with tasks aimed at imitating human behavior ( e.g . language modeling Bengio et al . ( 2003 ) ; Mikolov et al . ( 2010 ) , machine translation Bahdanau et al . ( 2014 ) ; Sutskever et al . ( 2014 ) , etc . ) by learning from large text-only corpora . To incorporate both vision and action , it is important to have the language grounded ( Harnad , 1990 ; Barsalou , 1999 ) , where words like cat are connected to visual percepts and words like move relate to actions taken in an environment . Additionally , judging language understanding purely based on the ability to mimic human utterances has limitations : there are many ways to express roughly the same meaning , and conveying the correct information is often more important than the particular choice of words . An alternative approach , which has recently gained increased prominence , is to train and evaluate language generation capabilities in an interactive setting , where the focus is on successfully communicating information that an agent must share in order to achieve its goals . a. Teller View 1 . Drawer : Ready 1 . You : There ’ s a girl on the left side . SEND Target Image Chat Box 2 . Drawer : What does the girl look like ? 2 . You : She is holding a beach ball in the arm that is up . 1 . Teller : There ’ s a girl on the left side . 2 . You : What does the girl look like ? 2 . Teller : She is holding a beach ball in the arm that is up . Drag & Drop waiting for Drawer ’ s message… 1 . You : Ready a. Teller View Drawing Canvas b. Drawer View 1 . Drawer : Ready 1 . You : There ’ s a girl on the left side . SEND Target Image Chat Box 2 . Drawer : What does the girl look like ? 2 . You : She is holding a beach ball in the arm that is up . 1 . Teller : There ’ s a girl on the left side . 2 . You : What does the girl look like ? SEND Chat Box Done 2 . Teller : She is holding a beach ball in the arm that is up . Drag & Drop waiting for Drawer ’ s message… 1 . You : Ready In this paper , we propose the Collaborative Drawing ( CoDraw ) task , which combines grounded language understanding and learning effective goal-driven communication into a single , unified testbed . This task involves perception , communication , and actions in a partially observable virtual environment . As shown in Figure 1 , our game is grounded in a virtual world constructed by clip art objects ( Zitnick et al. , 2013 ; Zitnick & Parikh , 2013 ) . Two players , Teller and Drawer , play the game . The Teller sees an abstract scene made from clip art objects in a semantically meaningful configuration , while the Drawer sees a drawing canvas that is initially empty . The goal of the game is to have both players communicate so that the Drawer can reconstruct the image of the Teller , without ever seeing it . Our task requires effective communication because the two players can not see each other ’ s scenes . The Teller has to describe the scene in sufficient detail for the Drawer to reconstruct it , which will require rich grounded language . Moreover , the Drawer will need to carry out a series of actions from a rich action space to position , orient , and resize all of the clip art pieces required for the reconstruction . Note that such actions are only made possible through clip art pieces : they can represent semantically meaningful configurations of a visual scene that are easy to manipulate , in contrast to low-level pixel-based image representations . The performance of a pair of agents is judged based on the quality of reconstructed scenes . We consider high-quality reconstructions as a signal that communication has been successful . As we develop models and protocols for CoDraw , we found it critical to train the Teller and the Drawer separately on disjoint subsets of the training data . Otherwise , the two machine agents may conspire to successfully achieve the goal while communicating using a shared “ codebook ” that bears little resemblance to natural language . We call this separate-training , joint-evaluation protocol crosstalk , which prevents learning of mutually agreed upon codebooks , while still checking for goal completion at test time . We highlight crosstalk as one of our contributions , and believe it can be generally applicable to other related tasks ( Sukhbaatar et al. , 2016 ; Foerster et al. , 2016 ; de Vries et al. , 2016 ; Das et al. , 2017b ; Lewis et al. , 2017 ) . Summary of Contributions • We propose a novel CoDraw task , which is a game designed to facilitate the learning and evaluation of effective natural language communication in a grounded context . • We collect a CoDraw dataset of ∼10K variable-length dialogs consisting of ∼138K messages with the drawing history at each step of the dialog . • We define a scene similarity metric , which allows us to automatically evaluate the effectiveness of the communication at the end and at intermediate states . • We propose a cross-talk training and evaluation protocol that prevents agents from potentially learning joint uninterpretable codebooks , rendering them ineffective at communicating with humans . • We evaluate several Drawer and Teller models automatically as well as by pairing them with humans , and show that long-term planning and context reasoning in the conversation are key challenges of the CoDraw task . 2 RELATED WORK . Language and Vision . The proposed CoDraw game is related to several well-known language and vision tasks that study grounded language understanding ( Karpathy & Fei-Fei , 2015 ; Donahue et al. , 2015 ; de Vries et al. , 2016 ) . For instance , compared to image captioning ( Vinyals et al. , 2017 ; Xu et al. , 2015 ; Chen & Zitnick , 2015 ; Lu et al. , 2017 ) , visual question answering ( Antol et al. , 2015 ; Zhang et al. , 2016 ; Goyal et al. , 2016 ; Gao et al. , 2015 ; Krishna et al. , 2017 ; Malinowski & Fritz , 2014 ; Ren et al. , 2015 ; Tapaswi et al. , 2016 ; Yu et al. , 2015 ; Zhu et al. , 2016 ) and recent embodied extensions ( Das et al. , 2018 ) , CoDraw involves multiple rounds of interactions between two agents . Both agents hold their own partially observable states and need to build a mental model for each other to collaborate . Compared to work on navigation ( Vogel & Jurafsky , 2010 ; Anderson et al. , 2018 ; Fried et al. , 2018b ) where an agent must follow instructions to move itself in a static environment , CoDraw involves moving and manipulating multiple clip art pieces , which must jointly form a semantically meaningful scene . Compared to visual dialog ( Das et al. , 2017a ; b ; Strub et al. , 2017 ; Mostafazadeh et al. , 2017 ) tasks , agents need to additionally cooperate to change the environment with actions ( e.g. , move around pieces ) . Thus , the agents have to possess the ability to adapt and hold a dialog about novel scenes that will be constructed as a consequence of their dialog . In addition , we also want to highlight that CoDraw has a well-defined communication goal , which facilitates objective measurement of success and enables end-to-end goal-driven learning . End-to-end Goal-Driven Dialog . Traditional goal-driven agents are often based on ‘ slot filling ’ ( Lemon et al. , 2006 ; Wang & Lemon , 2013 ; Yu et al. , 2015 ) , in which the structure of the dialog is pre-specified but the individual slots are replaced by relevant information . Recently , endto-end neural models are also proposed for goal-driven dialog ( Bordes et al. , 2017 ; Li et al. , 2017a ; b ; He et al. , 2017 ) , as well as goal-free dialog or ‘ chit-chat ’ ( Shang et al. , 2015 ; Sordoni et al. , 2015 ; Vinyals & Le , 2015 ; Li et al. , 2016 ; Dodge et al. , 2016 ) . Unlike CoDraw , in these approaches , symbols in the dialog are not grounded into visual objects . Language Grounded in Environments . Learning language games to change the environment has been studied recently ( Wang et al. , 2016 ; 2017 ) . The agent can change the environment using the grounded natural language . However , agents do not have the need to cooperate . Other work on grounded instruction following relies on datasets of pre-generated action sequences annotated with human descriptions , rather than using a single end goal ( Long et al. , 2016 ) . Speaker models for these tasks are only evaluated based on their ability to describe an action sequence that is given to them ( Fried et al. , 2018a ) , whereas Teller models for CoDraw also need to select a desired action sequence in a goal-driven manner . Language grounding has also been studied for robot navigation , manipulation , and environment mapping ( Tellex et al. , 2011 ; Mei et al. , 2015 ; Daniele et al. , 2016 ) . However , these works manually pair each command with robot actions and lack end-to-end training ( Tellex et al. , 2011 ) , dialog ( Mei et al. , 2015 ; Daniele et al. , 2016 ) , or both ( Walter et al. , 2014 ) . Emergent Communication . Building on the seminal works by Lewis ( 1969 ; 1975 ) , a number of recent works study cooperative games between agents where communication protocols emerge as a consequence of training the agents to accomplish shared goals ( Sukhbaatar et al. , 2016 ; Foerster et al. , 2016 ) . These methods have typically been applied to learn to communicate small amounts of information , rather than the complete , semantically meaningful scenes used in the CoDraw task . In addition , the learned communication protocols are usually not natural ( Kottur et al. , 2017 ) or interpretable . On the other hand , since our goal is to develop agents that can assist and communicate with humans , we must pre-train our agents on human communication and use techniques that can cope with the greater linguistic variety and richness of meaning present in natural language . 3 CODRAW TASK AND DATASET . In this section , we first detail our task , then present the CoDraw dataset , and finally propose a Scene Similarity Metric which allows automatic evaluation of the reconstructed and original scene . 3.1 TASK . Abstract Scenes . To enable people to easily draw semantically rich scenes on a canvas , we leverage the Abstract Scenes dataset of ( Zitnick et al. , 2013 ; Zitnick & Parikh , 2013 ) . This dataset consists of 10,020 semantically consistent scenes created by human annotators . An example scene is shown in the left portion of Figure 1 . Most scenes contain 6 objects ( min 6 , max 17 , mean 6.67 ) . These scenes depict children playing in a park , and are made from a library of 58 clip arts , including a boy ( Mike ) and a girl ( Jenny ) in one of 7 poses and 5 expressions , and various other objects including trees , toys , hats , animals , food , etc . An abstract scene is created by dragging and dropping multiple clip art objects to any ( x , y ) position on the canvas . Also , for each clip art , different spatial transformations can be applied , including sizes ( Small , Normal , Large ) , and two orientations ( facing left or right ) . The clip art serve simultaneously as a high-level visual representation and as a mechanism by which rich drawing actions can be carried out . Interface . We built a drag-and-drop interface based on the Visual Dialog chat interface ( Das et al. , 2017a ) ( see Figures 4 and 5 in Appendix A for screen shots of the interface ) . The interface allows real-time interaction between two people . During the conversation , the Teller describes the scene and answers any questions from the Drawer on the chat interface , while Drawer “ draws ” or reconstructs the scene based on the Teller ’ s descriptions and instructions . Each side is only allowed to send one message at a time , and must wait for a reply before continuing . The maximum length of a single message is capped at 140 characters : this prevents excessively verbose descriptions and gives the Drawer more chances to participate in the dialog by encouraging the Teller to pause more frequently . Both participants were asked to submit the task when they are both confident that Drawer has accurately reconstructed the scene of Teller . Our dataset , as well as this infrastructure for live chat with live drawing , will be made publicly available . Additional Interaction . We did not allow Teller to continuously observe Drawer ’ s canvas to make sure that the natural language focused on the high-level semantics of the scene rather than instructions calling for the execution of low-level clip art manipulation actions , but we hypothesize that direct visual feedback may be necessary to get the all the details right . For this , we give one chance for the Teller to look at the Drawer ’ s canvas using a ‘ peek ’ button in the interface . Communication is only allowed after the peek window is closed .

Summary: This paper presents CoDraw, a grounded and goal-driven dialogue environment for collaborative drawing. The authors argue convincingly that an interactive and grounded evaluation environment helps us better measure how well NLG/NLU agents actually understand and use their language — rather than evaluating against arbitrary ground-truth examples of what humans say, we can evaluate the objective end-to-end performance of a system in a well-specified nonlinguistic task. They collect a novel dataset in this grounded and goal-driven communication paradigm, define a success metric for the collaborative drawing task, and present models for maximizing that metric.