CoDraw: Collaborative Drawing as a Testbed for Grounded Goal-driven Communication

1 INTRODUCTION . Building agents that can interact with humans in natural language while perceiving and taking actions in their environments is one of the fundamental goals in artificial intelligence . One of the required components , language understanding , has traditionally been studied in isolation and with tasks aimed at imitating human behavior ( e.g . language modeling Bengio et al . ( 2003 ) ; Mikolov et al . ( 2010 ) , machine translation Bahdanau et al . ( 2014 ) ; Sutskever et al . ( 2014 ) , etc . ) by learning from large text-only corpora . To incorporate both vision and action , it is important to have the language grounded ( Harnad , 1990 ; Barsalou , 1999 ) , where words like cat are connected to visual percepts and words like move relate to actions taken in an environment . Additionally , judging language understanding purely based on the ability to mimic human utterances has limitations : there are many ways to express roughly the same meaning , and conveying the correct information is often more important than the particular choice of words . An alternative approach , which has recently gained increased prominence , is to train and evaluate language generation capabilities in an interactive setting , where the focus is on successfully communicating information that an agent must share in order to achieve its goals . a. Teller View 1 . Drawer : Ready 1 . You : There ’ s a girl on the left side . SEND Target Image Chat Box 2 . Drawer : What does the girl look like ? 2 . You : She is holding a beach ball in the arm that is up . 1 . Teller : There ’ s a girl on the left side . 2 . You : What does the girl look like ? 2 . Teller : She is holding a beach ball in the arm that is up . Drag & Drop waiting for Drawer ’ s message… 1 . You : Ready a. Teller View Drawing Canvas b. Drawer View 1 . Drawer : Ready 1 . You : There ’ s a girl on the left side . SEND Target Image Chat Box 2 . Drawer : What does the girl look like ? 2 . You : She is holding a beach ball in the arm that is up . 1 . Teller : There ’ s a girl on the left side . 2 . You : What does the girl look like ? SEND Chat Box Done 2 . Teller : She is holding a beach ball in the arm that is up . Drag & Drop waiting for Drawer ’ s message… 1 . You : Ready In this paper , we propose the Collaborative Drawing ( CoDraw ) task , which combines grounded language understanding and learning effective goal-driven communication into a single , unified testbed . This task involves perception , communication , and actions in a partially observable virtual environment . As shown in Figure 1 , our game is grounded in a virtual world constructed by clip art objects ( Zitnick et al. , 2013 ; Zitnick & Parikh , 2013 ) . Two players , Teller and Drawer , play the game . The Teller sees an abstract scene made from clip art objects in a semantically meaningful configuration , while the Drawer sees a drawing canvas that is initially empty . The goal of the game is to have both players communicate so that the Drawer can reconstruct the image of the Teller , without ever seeing it . Our task requires effective communication because the two players can not see each other ’ s scenes . The Teller has to describe the scene in sufficient detail for the Drawer to reconstruct it , which will require rich grounded language . Moreover , the Drawer will need to carry out a series of actions from a rich action space to position , orient , and resize all of the clip art pieces required for the reconstruction . Note that such actions are only made possible through clip art pieces : they can represent semantically meaningful configurations of a visual scene that are easy to manipulate , in contrast to low-level pixel-based image representations . The performance of a pair of agents is judged based on the quality of reconstructed scenes . We consider high-quality reconstructions as a signal that communication has been successful . As we develop models and protocols for CoDraw , we found it critical to train the Teller and the Drawer separately on disjoint subsets of the training data . Otherwise , the two machine agents may conspire to successfully achieve the goal while communicating using a shared “ codebook ” that bears little resemblance to natural language . We call this separate-training , joint-evaluation protocol crosstalk , which prevents learning of mutually agreed upon codebooks , while still checking for goal completion at test time . We highlight crosstalk as one of our contributions , and believe it can be generally applicable to other related tasks ( Sukhbaatar et al. , 2016 ; Foerster et al. , 2016 ; de Vries et al. , 2016 ; Das et al. , 2017b ; Lewis et al. , 2017 ) . Summary of Contributions • We propose a novel CoDraw task , which is a game designed to facilitate the learning and evaluation of effective natural language communication in a grounded context . • We collect a CoDraw dataset of ∼10K variable-length dialogs consisting of ∼138K messages with the drawing history at each step of the dialog . • We define a scene similarity metric , which allows us to automatically evaluate the effectiveness of the communication at the end and at intermediate states . • We propose a cross-talk training and evaluation protocol that prevents agents from potentially learning joint uninterpretable codebooks , rendering them ineffective at communicating with humans . • We evaluate several Drawer and Teller models automatically as well as by pairing them with humans , and show that long-term planning and context reasoning in the conversation are key challenges of the CoDraw task . 2 RELATED WORK . Language and Vision . The proposed CoDraw game is related to several well-known language and vision tasks that study grounded language understanding ( Karpathy & Fei-Fei , 2015 ; Donahue et al. , 2015 ; de Vries et al. , 2016 ) . For instance , compared to image captioning ( Vinyals et al. , 2017 ; Xu et al. , 2015 ; Chen & Zitnick , 2015 ; Lu et al. , 2017 ) , visual question answering ( Antol et al. , 2015 ; Zhang et al. , 2016 ; Goyal et al. , 2016 ; Gao et al. , 2015 ; Krishna et al. , 2017 ; Malinowski & Fritz , 2014 ; Ren et al. , 2015 ; Tapaswi et al. , 2016 ; Yu et al. , 2015 ; Zhu et al. , 2016 ) and recent embodied extensions ( Das et al. , 2018 ) , CoDraw involves multiple rounds of interactions between two agents . Both agents hold their own partially observable states and need to build a mental model for each other to collaborate . Compared to work on navigation ( Vogel & Jurafsky , 2010 ; Anderson et al. , 2018 ; Fried et al. , 2018b ) where an agent must follow instructions to move itself in a static environment , CoDraw involves moving and manipulating multiple clip art pieces , which must jointly form a semantically meaningful scene . Compared to visual dialog ( Das et al. , 2017a ; b ; Strub et al. , 2017 ; Mostafazadeh et al. , 2017 ) tasks , agents need to additionally cooperate to change the environment with actions ( e.g. , move around pieces ) . Thus , the agents have to possess the ability to adapt and hold a dialog about novel scenes that will be constructed as a consequence of their dialog . In addition , we also want to highlight that CoDraw has a well-defined communication goal , which facilitates objective measurement of success and enables end-to-end goal-driven learning . End-to-end Goal-Driven Dialog . Traditional goal-driven agents are often based on ‘ slot filling ’ ( Lemon et al. , 2006 ; Wang & Lemon , 2013 ; Yu et al. , 2015 ) , in which the structure of the dialog is pre-specified but the individual slots are replaced by relevant information . Recently , endto-end neural models are also proposed for goal-driven dialog ( Bordes et al. , 2017 ; Li et al. , 2017a ; b ; He et al. , 2017 ) , as well as goal-free dialog or ‘ chit-chat ’ ( Shang et al. , 2015 ; Sordoni et al. , 2015 ; Vinyals & Le , 2015 ; Li et al. , 2016 ; Dodge et al. , 2016 ) . Unlike CoDraw , in these approaches , symbols in the dialog are not grounded into visual objects . Language Grounded in Environments . Learning language games to change the environment has been studied recently ( Wang et al. , 2016 ; 2017 ) . The agent can change the environment using the grounded natural language . However , agents do not have the need to cooperate . Other work on grounded instruction following relies on datasets of pre-generated action sequences annotated with human descriptions , rather than using a single end goal ( Long et al. , 2016 ) . Speaker models for these tasks are only evaluated based on their ability to describe an action sequence that is given to them ( Fried et al. , 2018a ) , whereas Teller models for CoDraw also need to select a desired action sequence in a goal-driven manner . Language grounding has also been studied for robot navigation , manipulation , and environment mapping ( Tellex et al. , 2011 ; Mei et al. , 2015 ; Daniele et al. , 2016 ) . However , these works manually pair each command with robot actions and lack end-to-end training ( Tellex et al. , 2011 ) , dialog ( Mei et al. , 2015 ; Daniele et al. , 2016 ) , or both ( Walter et al. , 2014 ) . Emergent Communication . Building on the seminal works by Lewis ( 1969 ; 1975 ) , a number of recent works study cooperative games between agents where communication protocols emerge as a consequence of training the agents to accomplish shared goals ( Sukhbaatar et al. , 2016 ; Foerster et al. , 2016 ) . These methods have typically been applied to learn to communicate small amounts of information , rather than the complete , semantically meaningful scenes used in the CoDraw task . In addition , the learned communication protocols are usually not natural ( Kottur et al. , 2017 ) or interpretable . On the other hand , since our goal is to develop agents that can assist and communicate with humans , we must pre-train our agents on human communication and use techniques that can cope with the greater linguistic variety and richness of meaning present in natural language . 3 CODRAW TASK AND DATASET . In this section , we first detail our task , then present the CoDraw dataset , and finally propose a Scene Similarity Metric which allows automatic evaluation of the reconstructed and original scene . 3.1 TASK . Abstract Scenes . To enable people to easily draw semantically rich scenes on a canvas , we leverage the Abstract Scenes dataset of ( Zitnick et al. , 2013 ; Zitnick & Parikh , 2013 ) . This dataset consists of 10,020 semantically consistent scenes created by human annotators . An example scene is shown in the left portion of Figure 1 . Most scenes contain 6 objects ( min 6 , max 17 , mean 6.67 ) . These scenes depict children playing in a park , and are made from a library of 58 clip arts , including a boy ( Mike ) and a girl ( Jenny ) in one of 7 poses and 5 expressions , and various other objects including trees , toys , hats , animals , food , etc . An abstract scene is created by dragging and dropping multiple clip art objects to any ( x , y ) position on the canvas . Also , for each clip art , different spatial transformations can be applied , including sizes ( Small , Normal , Large ) , and two orientations ( facing left or right ) . The clip art serve simultaneously as a high-level visual representation and as a mechanism by which rich drawing actions can be carried out . Interface . We built a drag-and-drop interface based on the Visual Dialog chat interface ( Das et al. , 2017a ) ( see Figures 4 and 5 in Appendix A for screen shots of the interface ) . The interface allows real-time interaction between two people . During the conversation , the Teller describes the scene and answers any questions from the Drawer on the chat interface , while Drawer “ draws ” or reconstructs the scene based on the Teller ’ s descriptions and instructions . Each side is only allowed to send one message at a time , and must wait for a reply before continuing . The maximum length of a single message is capped at 140 characters : this prevents excessively verbose descriptions and gives the Drawer more chances to participate in the dialog by encouraging the Teller to pause more frequently . Both participants were asked to submit the task when they are both confident that Drawer has accurately reconstructed the scene of Teller . Our dataset , as well as this infrastructure for live chat with live drawing , will be made publicly available . Additional Interaction . We did not allow Teller to continuously observe Drawer ’ s canvas to make sure that the natural language focused on the high-level semantics of the scene rather than instructions calling for the execution of low-level clip art manipulation actions , but we hypothesize that direct visual feedback may be necessary to get the all the details right . For this , we give one chance for the Teller to look at the Drawer ’ s canvas using a ‘ peek ’ button in the interface . Communication is only allowed after the peek window is closed .

Summary: In this paper a new task namely CoDraw is introduced. In CoDraw, there is a teller who describes a scene and a drawer who tries to select clip art component and place them on a canvas to draw the description. The drawing environment contains simple objects and a fixed background scene all in cartoon style. The describing language thus does not have sophisticated components and phrases. A metric based on the presence of the components in the original image and the generated image is coined to compute similarity which is used in learning and evaluation.  Authors mention that in order to gain better performance they needed to train the teller and drawer separately on disjoint subsets of the training data which they call it a cross talk.