Talk The Walk: Navigating Grids in New York City through Grounded Dialogue

We introduce “ Talk The Walk ” , the first large-scale dialogue dataset grounded in action and perception . The task involves two agents ( a “ guide ” and a “ tourist ” ) that communicate via natural language in order to achieve a common goal : having the tourist navigate to a given target location . The task and dataset , which are described in detail , are challenging and their full solution is an open problem that we pose to the community . We ( i ) focus on the task of tourist localization and develop the novel Masked Attention for Spatial Convolutions ( MASC ) mechanism that allows for grounding tourist utterances into the guide ’ s map , ( ii ) show it yields significant improvements for both emergent and natural language communication , and ( iii ) using this method , we establish non-trivial baselines on the full task . 1 INTRODUCTION . As artificial intelligence plays an ever more prominent role in everyday human lives , it becomes increasingly important to enable machines to communicate via natural language—not only with humans , but also with each other . Learning algorithms for natural language understanding , such as in machine translation and reading comprehension , have progressed at an unprecedented rate in recent years , but still rely on static , large-scale , text-only datasets that lack crucial aspects of how humans understand and produce natural language . Namely , humans develop language capabilities by being embodied in an environment which they can perceive , manipulate and move around in ; and by interacting with other humans . Hence , we argue that we should incorporate all three fundamental aspects of human language acquisition—perception , action and interactive communication—and develop a task and dataset to that effect . We introduce the Talk the Walk dataset , where the aim is for two agents , a “ guide ” and a “ tourist ” , to interact with each other via natural language in order to achieve a common goal : having the tourist navigate towards the correct location . The guide has access to a map and knows the target location , but does not know where the tourist is ; the tourist has a 360-degree view of the world , but knows neither the target location on the map nor the way to it . The agents need to work together through communication in order to successfully solve the task . An example of the task is given in Figure 1 . Grounded language learning has ( re- ) gained traction in the AI community , and much attention is currently devoted to virtual embodiment—the development of multi-agent communication tasks in virtual environments—which has been argued to be a viable strategy for acquiring natural language semantics Kiela et al . ( 2016 ) . Various related tasks have recently been introduced , but in each case with some limitations . Although visually grounded dialogue tasks de Vries et al . ( 2016 ) ; Das et al . ( 2016 ) comprise perceptual grounding and multi-agent interaction , their agents are passive observers and do not act in the environment . By contrast , instruction-following tasks , such as VNL Anderson et al . ( 2017 ) , involve action and perception but lack natural language interaction with other agents . Furthermore , some of these works use simulated environments Das et al . ( 2017a ) and/or templated language Hermann et al . ( 2017 ) , which arguably oversimplifies real perception or natural language , respectively . See Table 1 for a comparison . Talk The Walk is the first task to bring all three aspects together : perception for the tourist observing the world , action for the tourist to navigate through the environment , and interactive dialogue for the tourist and guide to work towards their common goal . To collect grounded dialogues , we constructed a virtual 2D grid environment by manually capturing 360-views of several neighbor- hoods in New York City ( NYC ) 1 . As the main focus of our task is on interactive dialogue , we limit the difficulty of the control problem by having the tourist navigating a 2D grid via discrete actions ( turning left , turning right and moving forward ) . Our street view environment was integrated into ParlAI ( Miller et al. , 2017 ) and used to collect a large-scale dataset on Mechanical Turk involving human perception , action and communication . We argue that for artificial agents to solve this challenging problem , some fundamental architecture designs are missing , and our hope is that this task motivates their innovation . To that end , we focus on the task of localization and develop the novel Masked Attention for Spatial Convolutions ( MASC ) mechanism . To model the interaction between language and action , this architecture repeatedly conditions the spatial dimensions of a convolution on the communicated message sequence . This work makes the following contributions : 1 ) We present the first large scale dialogue dataset grounded in action and perception ; 2 ) We introduce the MASC architecture for localization and show it yields improvements for both emergent and natural language ; 4 ) Using localization models , we establish initial baselines on the full task ; 5 ) We show that our best model exceeds human performance under the assumption of “ perfect perception ” and with a learned emergent communication protocol , and sets a non-trivial baseline with natural language . 2 TALK THE WALK . We create a perceptual environment by manually capturing several neighborhoods of New York City ( NYC ) with a 360 camera2 . Most parts of the city are grid-like and uniform , which makes it wellsuited for obtaining a 2D grid . For Talk The Walk , we capture parts of Hell ’ s Kitchen , East Village , the Financial District , Williamsburg and the Upper East Side—see Figure 5 in Appendix 13 for their respective locations within NYC . For each neighborhood , we choose an approximately 5x5 grid and capture a 360 view on all four corners of each intersection , leading to a grid-size of roughly 10x10 per neighborhood . The tourist ’ s location is given as a tuple ( x , y , o ) , where x , y are the coordinates and o signifies the orientation ( north , east , south or west ) . The tourist can take three actions : turn left , turn right and go forward . For moving forward , we add ( 0 , 1 ) , ( 1 , 0 ) , ( 0 , −1 ) , ( −1 , 0 ) to the x , y coordinates for the respective orientations . Upon a turning action , the orientation is updated by o = ( o+d ) mod 4 where d = −1 for left and d = 1 for right . If the tourist moves outside the grid , we issue a warning that they can not go in that direction and do not update the location . Moreover , tourists are shown different types of transitions : a short transition for actions that bring the tourist to a different corner of the same intersection ; and a longer transition for actions that bring them to a new intersection . The guide observes a map that corresponds to the tourist ’ s environment . We exploit the fact that urban areas like NYC are full of local businesses , and overlay the map with these landmarks as localization points for our task . Specifically , we manually annotate each corner of the intersection with a set of landmarks Λx , y = { l0 , . . . , lK } , each coming from one of the following categories : 1We avoided using existing street view resources due to licensing issues . 2A 360fly 4K camera . • Bar • Playfield • Bank • Hotel • Shop • Subway • Coffee Shop • Restaurant • Theater The right-side of Figure 1 illustrates how the map is presented . Note that within-intersection transitions have a smaller grid distance than transitions to new intersections . To ensure that the localization task is not too easy , we do not include street names in the overhead map and keep the landmark categories coarse . That is , the dialogue is driven by uncertainty in the tourist ’ s current location and the properties of the target location : if the exact location and orientation of the tourist were known , it would suffice to communicate a sequence of actions . 2.1 TASK . For the Talk The Walk task , we randomly choose one of the five neighborhoods , and subsample a 4x4 grid ( one block with four complete intersections ) from the entire grid . We specify the boundaries of the grid by the top-left and bottom-right corners ( xmin , ymin , xmax , ymax ) . Next , we construct the overhead map of the environment , i.e . { Λx′ , y′ } with xmin ≤ x′ ≤ xmax and ymin ≤ y′ ≤ ymax . We subsequently sample a start location and orientation ( x , y , o ) and a target location ( x , y ) tgt at random3 . The shared goal of the two agents is to navigate the tourist to the target location ( x , y ) tgt , which is only known to the guide . The tourist perceives a “ street view ” planar projection Sx , y , o of the 360 image at location ( x , y ) and can simultaneously chat with the guide and navigate through the environment . The guide ’ s role consists of reading the tourist description of the environment , building a “ mental map ” of their current position and providing instructions for navigating towards the target location . Whenever the guide believes that the tourist has reached the target location , they instruct the system to evaluate the tourist ’ s location . The task ends when the evaluation is successful—i.e. , when ( x , y ) = ( x , y ) tgt—or otherwise continues until a total of three failed attempts . The additional attempts are meant to ease the task for humans , as we found that they otherwise often fail at the task but still end up close to the target location , e.g. , at the wrong corner of the correct intersection . 2.2 DATA COLLECTION . We crowd-sourced the collection of the dataset on Amazon Mechanical Turk ( MTurk ) . We use the MTurk interface of ParlAI ( Miller et al. , 2017 ) to render 360 images via WebGL and dynamically display neighborhood maps with an HTML5 canvas . Detailed task instructions , which were also given to our workers before they started their task , are shown in Appendix 14 . We paired Turkers at random and let them alternate between the tourist and guide role across different HITs . 2.3 DATASET STATISTICS . The Talk The Walk dataset consists of over 10k successful dialogues—see Table 11 in the appendix for the dataset statistics split by neighborhood . Turkers successfully completed 76.74 % of all finished tasks ( we use this statistic as the human success rate ) . More than six hundred participants successfully completed at least one Talk The Walk HIT . Although the Visual Dialog ( Das et al. , 2016 ) and GuessWhat ( de Vries et al. , 2016 ) datasets are larger , the collected Talk The Walk dialogs are significantly longer . On average , Turkers needed more than 62 acts ( i.e utterances and actions ) before they successfully completed the task , whereas Visual Dialog requires 20 acts . The majority of acts comprise the tourist ’ s actions , with on average more than 44 actions per dialogue . The guide produces roughly 9 utterances per dialogue , slightly more than the tourist ’ s 8 utterances . Turkers use diverse discourse , with a vocabulary size of more than 10K ( calculated over all successful dia- 3Note that we do not include the orientation in the target , as we found in early experiments that this led to an unnatural task for humans . Similarly , we explored bigger grid sizes but found these to be too difficult for most annotators . logues ) . An example from the dataset is shown in Appendix 13 . The dataset is available at [ URL ANONYMIZED ] .

Summary: The paper introduces a new task called "Talk the Walk", where a tourist and a guide has to communicate in natural language to reach a common goal. It also introduces strong baselines for the task. The descriptions are thorough and clear. My only worry is that the task is too hard and has too many complexities to be a stand alone task.  Future work will probably focus on sub-parts of the task.