The guide has access to a map and knows the target location, but does not know where the tourist is; the tourist has a 360-degree view of the world, but knows neither the target location on the map nor the way to it. The agents need to work together through communication in order to successfully solve the task. An example of the task is given in Figure FIGREF3 . Grounded language learning has (re-)gained traction in the AI community, and much attention is currently devoted to virtual embodiment—the development of multi-agent communication tasks in virtual environments—which has been argued to be a viable strategy for acquiring natural language semantics  . Various related tasks have recently been introduced, but in each case with some limitations. Although visually grounded dialogue tasks  ,  comprise perceptual grounding and multi-agent interaction, their agents are passive observers and do not act in the environment. By contrast, instruction-following tasks, such as VNL  , involve action and perception but lack natural language interaction with other agents. Furthermore, some of these works use simulated environments  and/or templated language  , which arguably oversimplifies real perception or natural language, respectively. See Table TABREF15 for a comparison.

paragraph4 0.1ex plus0.1ex minus.1ex-1em Emergent language A tourist, starting from a random location, takes INLINEFORM0 random actions INLINEFORM1 to reach target location INLINEFORM2 . Every location in the environment has a corresponding set of landmarks INLINEFORM3 for each of the INLINEFORM4 coordinates. As the tourist navigates, the agent perceives INLINEFORM5 state-observations INLINEFORM6 where each observation INLINEFORM7 consists of a set of INLINEFORM8 landmark symbols INLINEFORM9 . Given the observations INLINEFORM10 and actions INLINEFORM11 , the tourist generates a message INLINEFORM12 which is communicated to the other agent. The objective of the guide is to predict the location INLINEFORM13 from the tourist's message INLINEFORM14 . paragraph4 0.1ex plus0.1ex minus.1ex-1em Natural language In contrast to our emergent communication experiments, we do not take random actions but instead extract actions, observations, and messages from the dataset. Specifically, we consider each tourist utterance (i.e.

Talk the Walk: Navigating New York City through Grounded Dialogue	We introduce"Talk The Walk", the first large-scale dialogue dataset grounded in action and perception. The task involves two agents (a"guide"and a"tourist") that communicate via natural language in order to achieve a common goal: having the tourist navigate to a given target location. The task and dataset, which are described in detail, are challenging and their full solution is an open problem that we pose to the community. We (i) focus on the task of tourist localization and develop the novel Masked Attention for Spatial Convolutions (MASC) mechanism that allows for grounding tourist utterances into the guide's map, (ii) show it yields significant improvements for both emergent and natural language communication, and (iii) using this method, we establish non-trivial baselines on the full task.

Even with these simplifications, the localization-based baseline comes with its own set of challenges. As we show in Section SECREF34 , the task requires communication about a short (random) path—i.e., not only a sequence of observations but also actions—in order to achieve high localization accuracy. This means that the guide needs to decode observations from multiple time steps, as well as understand their 2D spatial arrangement as communicated via the sequence of actions. Thus, in order to get to a good understanding of the task, we thoroughly examine whether the agents can learn a communication protocol that simultaneously grounds observations and actions into the guide's map. In doing so, we thoroughly study the role of the communication channel in the localization task, by investigating increasingly constrained forms of communication: from differentiable continuous vectors to emergent discrete symbols to the full complexity of natural language. The full navigation baseline hinges on a localization model from random trajectories. While we can sample random actions in the emergent communication setup, this is not possible for the natural language setup because the messages are coupled to the trajectories of the human annotators. This leads to slightly different problem setups, as described below.

Analysis of Localization Task	paragraph4 0.1ex plus0.1ex minus.1ex-1em Task is not too easy The upper-bound on localization performance in Table TABREF32 suggest that communicating a single landmark observation is not sufficient for accurate localization of the tourist ( INLINEFORM0 35% accuracy). This is an important result for the full navigation task because the need for two-way communication disappears if localization is too easy; if the guide knows the exact location of the tourist it suffices to communicate a list of instructions, which is then executed by the tourist. The uncertainty in the tourist's location is what drives the dialogue between the two agents. paragraph4 0.1ex plus0.1ex minus.1ex-1em Importance of actions We observe that the upperbound for only communicating observations plateaus around 57% (even for INLINEFORM0 actions), whereas it exceeds 90% when we also take actions into account. This implies that, at least for random walks, it is essential to communicate a trajectory, including observations and actions, in order to achieve high localization accuracy.

Introduction	As artificial intelligence plays an ever more prominent role in everyday human lives, it becomes increasingly important to enable machines to communicate via natural language—not only with humans, but also with each other. Learning algorithms for natural language understanding, such as in machine translation and reading comprehension, have progressed at an unprecedented rate in recent years, but still rely on static, large-scale, text-only datasets that lack crucial aspects of how humans understand and produce natural language. Namely, humans develop language capabilities by being embodied in an environment which they can perceive, manipulate and move around in; and by interacting with other humans. Hence, we argue that we should incorporate all three fundamental aspects of human language acquisition—perception, action and interactive communication—and develop a task and dataset to that effect. We introduce the Talk the Walk dataset, where the aim is for two agents, a “guide” and a “tourist”, to interact with each other via natural language in order to achieve a common goal: having the tourist navigate towards the correct location.

Grounding also plays a crucial role in the emergent research of multi-agent communication, where, agents communicate (in natural language or otherwise) in order to solve a task, with respect to their shared environment  ,  ,  ,  ,  ,  ,  .

Experiments	We investigate the difficulty of the proposed task by establishing initial baselines. The final Talk The Walk task is challenging and encompasses several important sub-tasks, ranging from landmark recognition to tourist localization and natural language instruction-giving. Arguably the most important sub-task is localization: without such capabilities the guide can not tell whether the tourist reached the target location. In this work, we establish a minimal baseline for Talk The Walk by utilizing agents trained for localization. Specifically, we let trained tourist models undertake random walks, using the following protocol: at each step, the tourist communicates its observations and actions to the guide, who predicts the tourist's location. If the guide predicts that the tourist is at target, we evaluate its location. If successful, the task ends, otherwise we continue until there have been three wrong evaluations. The protocol is given as pseudo-code in Appendix SECREF12 .

None	0pt0.03.03 * 0pt0.030.03 * 0pt0.030.03 We introduce “Talk The Walk”, the first large-scale dialogue dataset grounded in action and perception. The task involves two agents (a “guide” and a “tourist”) that communicate via natural language in order to achieve a common goal: having the tourist navigate to a given target location. The task and dataset, which are described in detail, are challenging and their full solution is an open problem that we pose to the community. We (i) focus on the task of tourist localization and develop the novel Masked Attention for Spatial Convolutions (MASC) mechanism that allows for grounding tourist utterances into the guide's map, (ii) show it yields significant improvements for both emergent and natural language communication, and (iii) using this method, we establish non-trivial baselines on the full task.

Tourist: crossing the street now Tourist: ACTION:FORWARD ACTION:FORWARD ACTION:TURNLEFT Tourist: there is an I love new york shop across the street on the left from me now Tourist: ACTION:TURNRIGHT ACTION:FORWARD Guide: ok. I'll see if it's right. Guide: EVALUATE_LOCATION Guide: It's not right. Tourist: What should I be on the look for? Tourist: ACTION:TURNRIGHT ACTION:TURNRIGHT ACTION:TURNRIGHT Guide: There should be shops on two corners but you need to be on one of the corners  without the shop. Guide: Try the other corner. Tourist: this intersection has 2 shop corners and a bank corner Guide: yes. that's what I see on the map. Tourist: should I go to the bank corner? or one of the shop corners?  or the blank corner (perhaps a hotel) Tourist: ACTION:TURNLEFT ACTION:TURNLEFT ACTION:TURNRIGHT ACTION:TURNRIGHT Guide: Go to the one near the hotel. The map says the hotel is a little  further down but it might be a little off.

Related Work	The Talk the Walk task and dataset facilitate future research on various important subfields of artificial intelligence, including grounded language learning, goal-oriented dialogue research and situated navigation. Here, we describe related previous work in these areas. paragraph4 0.1ex plus0.1ex minus.1ex-1em Related tasks There has been a long line of work involving related tasks. Early work on task-oriented dialogue dates back to the early 90s with the introduction of the Map Task  and Maze Game  corpora. Recent efforts have led to larger-scale goal-oriented dialogue datasets, for instance to aid research on visually-grounded dialogue  ,  , knowledge-base-grounded discourse  or negotiation tasks  . At the same time, there has been a big push to develop environments for embodied AI, many of which involve agents following natural language instructions with respect to an environment  ,  ,  ,  ,  ,  , following-up on early work in this area  ,  . An early example of navigation using neural networks is  , who propose an online learning approach for robot navigation.

Recently, there has been increased interest in using end-to-end trainable neural networks for learning to navigate indoor scenes  ,  or large cities  ,  , but, unlike our work, without multi-agent communication. Also the task of localization (without multi-agent communication) has recently been studied  ,  . paragraph4 0.1ex plus0.1ex minus.1ex-1em Grounded language learning Grounded language learning is motivated by the observation that humans learn language embodied (grounded) in sensorimotor experience of the physical world  ,  . On the one hand, work in multi-modal semantics has shown that grounding can lead to practical improvements on various natural language understanding tasks  ,  . In robotics, researchers dissatisfied with purely symbolic accounts of meaning attempted to build robotic systems with the aim of grounding meaning in physical experience of the world  ,  . Recently, grounding has also been applied to the learning of sentence representations  , image captioning  ,  , visual question answering  ,  , visual reasoning  ,  , and grounded machine translation  ,  .

Emergent Language Localization	We first report the results for tourist localization with emergent language in Table TABREF32 . paragraph4 0.1ex plus0.1ex minus.1ex-1em MASC improves performance The MASC architecture significantly improves performance compared to models that do not include this mechanism. For instance, for INLINEFORM0 action, MASC already achieves 56.09 % on the test set and this further increases to 69.85% for INLINEFORM1 . On the other hand, no-MASC models hit a plateau at 43%. In Appendix SECREF11 , we analyze learned MASC values, and show that communicated actions are often mapped to corresponding state-transitions. paragraph4 0.1ex plus0.1ex minus.1ex-1em Continuous vs discrete We observe similar performance for continuous and discrete emergent communication models, implying that a discrete communication channel is not a limiting factor for localization performance.

Conclusion	We introduced the Talk The Walk task and dataset, which consists of crowd-sourced dialogues in which two human annotators collaborate to navigate to target locations in the virtual streets of NYC. For the important localization sub-task, we proposed MASC—a novel grounding mechanism to learn state-transition from the tourist's message—and showed that it improves localization performance for emergent and natural language. We use the localization model to provide baseline numbers on the Talk The Walk task, in order to facilitate future research.

Talk The Walk is the first task to bring all three aspects together: perception for the tourist observing the world, action for the tourist to navigate through the environment, and interactive dialogue for the tourist and guide to work towards their common goal. To collect grounded dialogues, we constructed a virtual 2D grid environment by manually capturing 360-views of several neighborhoods in New York City (NYC). As the main focus of our task is on interactive dialogue, we limit the difficulty of the control problem by having the tourist navigating a 2D grid via discrete actions (turning left, turning right and moving forward). Our street view environment was integrated into ParlAI  and used to collect a large-scale dataset on Mechanical Turk involving human perception, action and communication. We argue that for artificial agents to solve this challenging problem, some fundamental architecture designs are missing, and our hope is that this task motivates their innovation. To that end, we focus on the task of localization and develop the novel Masked Attention for Spatial Convolutions (MASC) mechanism. To model the interaction between language and action, this architecture repeatedly conditions the spatial dimensions of a convolution on the communicated message sequence.

Visualizing MASC predictions	Figure FIGREF46 shows the MASC values for a learned model with emergent discrete communications and INLINEFORM0 actions. Specifically, we look at the predicted MASC values for different action sequences taken by the tourist. We observe that the first action is always mapped to the correct state-transition, but that the second and third MASC values do not always correspond to right state-transitions.

Dataset Details	paragraph4 0.1ex plus0.1ex minus.1ex-1em Dataset split We split the full dataset by assigning entire 4x4 grids (independent of the target location) to the train, valid or test set. Specifically, we design the split such that the valid set contains at least one intersection (out of four) is not part of the train set. For the test set, all four intersections are novel. See our source code, available at URL ANONYMIZED, for more details on how this split is realized. paragraph4 0.1ex plus0.1ex minus.1ex-1em Example Tourist: ACTION:TURNRIGHT ACTION:TURNRIGHT Guide: Hello, what are you near? Tourist: ACTION:TURNLEFT ACTION:TURNLEFT ACTION:TURNLEFT Tourist: Hello, in front of me is a Brooks Brothers Tourist: ACTION:TURNLEFT ACTION:FORWARD ACTION:TURNLEFT ACTION:TURNLEFT Guide: Is that a shop or restaurant? Tourist: ACTION:TURNLEFT Tourist: It is a clothing shop.

Upon a turning action, the orientation is updated by INLINEFORM8 where INLINEFORM9 for left and INLINEFORM10 for right. If the tourist moves outside the grid, we issue a warning that they cannot go in that direction and do not update the location. Moreover, tourists are shown different types of transitions: a short transition for actions that bring the tourist to a different corner of the same intersection; and a longer transition for actions that bring them to a new intersection. The guide observes a map that corresponds to the tourist's environment. We exploit the fact that urban areas like NYC are full of local businesses, and overlay the map with these landmarks as localization points for our task. Specifically, we manually annotate each corner of the intersection with a set of landmarks INLINEFORM0 , each coming from one of the following categories:  Bar Playfield Bank Hotel Shop Subway Coffee Shop Restaurant Theater  The right-side of Figure FIGREF3 illustrates how the map is presented. Note that within-intersection transitions have a smaller grid distance than transitions to new intersections. To ensure that the localization task is not too easy, we do not include street names in the overhead map and keep the landmark categories coarse.

Implementation Details	For the emergent communication models, we use an embedding size INLINEFORM0 . The natural language experiments use 128-dimensional word embeddings and a bidirectional RNN with 256 units. In all experiments, we train the guide with a cross entropy loss using the ADAM optimizer with default hyper-parameters  . We perform early stopping on the validation accuracy, and report the corresponding train, valid and test accuracy. We optimize the localization models with continuous, discrete and natural language communication channels for 200, 200, and 25 epochs, respectively. To facilitate further research on Talk The Walk, we make our code base for reproducing experiments publicly available at https://github.com/facebookresearch/talkthewalk.

Task	For the Talk The Walk task, we randomly choose one of the five neighborhoods, and subsample a 4x4 grid (one block with four complete intersections) from the entire grid. We specify the boundaries of the grid by the top-left and bottom-right corners INLINEFORM0 . Next, we construct the overhead map of the environment, i.e. INLINEFORM1 with INLINEFORM2 and INLINEFORM3 . We subsequently sample a start location and orientation INLINEFORM4 and a target location INLINEFORM5 at random. The shared goal of the two agents is to navigate the tourist to the target location INLINEFORM0 , which is only known to the guide. The tourist perceives a “street view” planar projection INLINEFORM1 of the 360 image at location INLINEFORM2 and can simultaneously chat with the guide and navigate through the environment. The guide's role consists of reading the tourist description of the environment, building a “mental map” of their current position and providing instructions for navigating towards the target location. Whenever the guide believes that the tourist has reached the target location, they instruct the system to evaluate the tourist's location.

Results and Discussion	In this section, we describe the findings of various experiments. First, we analyze how much information needs to be communicated for accurate localization in the Talk The Walk environment, and find that a short random path (including actions) is necessary. Next, for emergent language, we show that the MASC architecture can achieve very high localization accuracy, significantly outperforming the baseline that does not include this mechanism. We then turn our attention to the natural language experiments, and find that localization from human utterances is much harder, reaching an accuracy level that is below communicating a single landmark observation. We show that generated utterances from a conditional language model leads to significantly better localization performance, by successfully grounding the utterance on a single landmark observation (but not yet on multiple observations and actions). Finally, we show performance of the localization baseline on the full task, which can be used for future comparisons to this work.

Interestingly, we observe that the supervised model (with greedy and beam-search decoding) as well as the policy gradient model leads to an improvement of more than 10 accuracy points over the human utterances. However, their level of accuracy is slightly below the baseline of communicating a single observation, indicating that these models only learn to ground utterances in a single landmark observation. paragraph4 0.1ex plus0.1ex minus.1ex-1em Better grounding of generated utterances We analyze natural language samples in Table TABREF38 , and confirm that, unlike human utterances, the generated utterances are talking about the observed landmarks. This observation explains why the generated utterances obtain higher localization accuracy. The current language models are most successful when conditioned on a single landmark observation; We show in Appendix UID43 that performance quickly deteriorates when the model is conditioned on more observations, suggesting that it can not produce natural language utterances about multiple time steps.

Dataset Statistics	The Talk The Walk dataset consists of over 10k successful dialogues—see Table FIGREF66 in the appendix for the dataset statistics split by neighborhood. Turkers successfully completed INLINEFORM0 of all finished tasks (we use this statistic as the human success rate). More than six hundred participants successfully completed at least one Talk The Walk HIT. Although the Visual Dialog  and GuessWhat  datasets are larger, the collected Talk The Walk dialogs are significantly longer. On average, Turkers needed more than 62 acts (i.e utterances and actions) before they successfully completed the task, whereas Visual Dialog requires 20 acts. The majority of acts comprise the tourist's actions, with on average more than 44 actions per dialogue. The guide produces roughly 9 utterances per dialogue, slightly more than the tourist's 8 utterances. Turkers use diverse discourse, with a vocabulary size of more than 10K (calculated over all successful dialogues). An example from the dataset is shown in Appendix SECREF14 . The dataset is available at https://github.com/facebookresearch/talkthewalk.

Guide: then go straight Tourist: That would be going back the way I came Guide: yeah. I was looking at the wrong bank Tourist: I'll notify when I am back at the brooks brothers, and the bank. Tourist: ACTION:TURNRIGHT Guide: make a right when the bank is on your left Tourist: ACTION:FORWARD ACTION:FORWARD ACTION:TURNRIGHT Tourist: Making the right at the bank. Tourist: ACTION:FORWARD ACTION:FORWARD Tourist: I can't go that way. Tourist: ACTION:TURNLEFT Tourist: Bank is ahead of me on the right Tourist: ACTION:FORWARD ACTION:FORWARD ACTION:TURNLEFT Guide: turn around on that intersection Tourist: I can only go to the left or back the way I just came. Tourist: ACTION:TURNLEFT Guide: you're in the right place. do you see shops on the corners? Guide: If you're on the corner with the bank, cross the street Tourist: I'm back where I started by the shop and the bank. Tourist: ACTION:TURNRIGHT Guide: on the same side of the street?

Evaluation on Full Setup	We provide pseudo-code for evaluation of localization models on the full task in Algorithm SECREF12 , as well as results for all emergent communication models in Table TABREF55 .  INLINEFORM0 INLINEFORM1   INLINEFORM0 take new action INLINEFORM1 INLINEFORM2  Performance evaluation of location prediction model on full Talk The Walk setup

Talk The Walk	We create a perceptual environment by manually capturing several neighborhoods of New York City (NYC) with a 360 camera. Most parts of the city are grid-like and uniform, which makes it well-suited for obtaining a 2D grid. For Talk The Walk, we capture parts of Hell's Kitchen, East Village, the Financial District, Williamsburg and the Upper East Side—see Figure FIGREF66 in Appendix SECREF14 for their respective locations within NYC. For each neighborhood, we choose an approximately 5x5 grid and capture a 360 view on all four corners of each intersection, leading to a grid-size of roughly 10x10 per neighborhood. The tourist's location is given as a tuple INLINEFORM0 , where INLINEFORM1 are the coordinates and INLINEFORM2 signifies the orientation (north, east, south or west). The tourist can take three actions: turn left, turn right and go forward. For moving forward, we add INLINEFORM3 , INLINEFORM4 , INLINEFORM5 , INLINEFORM6 to the INLINEFORM7 coordinates for the respective orientations.

Landmark Classification	While the guide has access to the landmark labels, the tourist needs to recognize these landmarks from raw perceptual information. In this section, we study landmark classification as a supervised learning problem to investigate the difficulty of perceptual grounding in Talk The Walk. The Talk The Walk dataset contains a total of 307 different landmarks divided among nine classes, see Figure FIGREF62 for how they are distributed. The class distribution is fairly imbalanced, with shops and restaurants as the most frequent landmarks and relatively few play fields and theaters. We treat landmark recognition as a multi-label classification problem as there can be multiple landmarks on a corner. For the task of landmark classification, we extract the relevant views of the 360 image from which a landmark is visible. Because landmarks are labeled to be on a specific corner of an intersection, we assume that they are visible from one of the orientations facing away from the intersection. For example, for a landmark on the northwest corner of an intersection, we extract views from both the north and west direction. The orientation-specific views are obtained by a planar projection of the full 360-image with a small field of view (60 degrees) to limit distortions.

Natural Language Localization	We report the results of tourist localization with natural language in Table TABREF36 . We compare accuracy of the guide model (with MASC) trained on utterances from (i) humans, (ii) a supervised model with various decoding strategies, and (iii) a policy gradient model optimized with respect to the loss of a frozen, pre-trained guide model on human utterances. paragraph4 0.1ex plus0.1ex minus.1ex-1em Human utterances Compared to emergent language, localization from human utterances is much harder, achieving only INLINEFORM0 on the test set. Here, we report localization from a single utterance, but in Appendix SECREF45 we show that including up to five dialogue utterances only improves performance to INLINEFORM1 . We also show that MASC outperform no-MASC models for natural language communication. paragraph4 0.1ex plus0.1ex minus.1ex-1em Generated utterances We also investigate generated tourist utterances from conditional language models.

This work makes the following contributions: 1) We present the first large scale dialogue dataset grounded in action and perception; 2) We introduce the MASC architecture for localization and show it yields improvements for both emergent and natural language; 4) Using localization models, we establish initial baselines on the full task; 5) We show that our best model exceeds human performance under the assumption of “perfect perception” and with a learned emergent communication protocol, and sets a non-trivial baseline with natural language.

Additional Natural Language Experiments	First, we investigate the sensitivity of tourist generation models to the trajectory length, finding that the model conditioned on a single observation (i.e. INLINEFORM0 ) achieves best performance. In the next subsection, we further analyze localization models from human utterances by investigating MASC and no-MASC models with increasing dialogue context.

at any point in the dialogue), obtain the current tourist location as target location INLINEFORM0 , the utterance itself as message INLINEFORM1 , and the sequence of observations and actions that took place between the current and previous tourist utterance as INLINEFORM2 and INLINEFORM3 , respectively. Similar to the emergent language setting, the guide's objective is to predict the target location INLINEFORM4 models from the tourist message INLINEFORM5 . We conduct experiments with INLINEFORM6 taken from the dataset and with INLINEFORM7 generated from the extracted observations INLINEFORM8 and actions INLINEFORM9 .

Localization from Human Utterances	We conduct an ablation study for MASC on natural language with varying dialogue context. Specifically, we compare localization accuracy of MASC and no-MASC models trained on the last [1, 3, 5] utterances of the dialogue (including guide utterances). We report these results in Table TABREF41 . In all cases, MASC outperforms the no-MASC models by several accuracy points. We also observe that mean predicted INLINEFORM0 (over the test set) increases from 1 to 2 when more dialogue context is included.

paragraph4 0.1ex plus0.1ex minus.1ex-1em Effect of path length There is a strong negative correlation between task success and the conditioned trajectory length. We observe that the full task performance quickly deteriorates for both human and random trajectories. This suggests that the tourist generation model can not produce natural language utterances that describe multiple observations and actions. Although it is possible that the guide model can not process such utterances, this is not very likely because the MASC architectures handles such messages successfully for emergent communication. We report localization performance of tourist utterances generated by beam search decoding of varying beam size in Table TABREF40 . We find that performance decreases from 29.05% to 20.87% accuracy on the test set when we increase the beam-size from one to eight.

INLINEFORM2 , and train it with a mean squared error loss. paragraph4 0.1ex plus0.1ex minus.1ex-1em Natural Language Because observations and actions are of variable-length, we use an LSTM encoder over the sequence of observations embeddings INLINEFORM0 , and extract its last hidden state INLINEFORM1 . We use a separate LSTM encoder for action embeddings INLINEFORM2 , and concatenate both INLINEFORM3 and INLINEFORM4 to the input of the LSTM decoder at each time step: DISPLAYFORM0   where INLINEFORM0 a look-up table, taking input tokens INLINEFORM1 . We train with teacher-forcing, i.e. we optimize the cross-entropy loss: INLINEFORM2 . At test time, we explore the following decoding strategies: greedy, sampling and a beam-search. We also fine-tune a trained tourist model (starting from a pre-trained model) with policy gradients in order to minimize the guide's prediction loss.

once for each action), and then aggregate the map embeddings by a sum over positionally-gated embeddings: INLINEFORM1 . We score locations by taking the dot-product of the observation embedding INLINEFORM2 , which contains information about the sequence of observed landmarks by the tourist, and the map. We compute a distribution over the locations of the map INLINEFORM3 by taking a softmax over the computed scores: DISPLAYFORM0  paragraph4 0.1ex plus0.1ex minus.1ex-1em Predicting T While emergent communication models use a fixed length trasjectory INLINEFORM0 , natural language messages may differ in the number of communicated observations and actions. Hence, we predict INLINEFORM1 from the communicated message. Specifically, we use a softmax regression layer over the last hidden state INLINEFORM2 of the RNN, and subsequently sample INLINEFORM3 from the resulting multinomial distribution: DISPLAYFORM0  We jointly train the INLINEFORM0 -prediction model via REINFORCE, with the guide's loss as reward function and a mean-reward baseline.

Similar to the continuous communication model, we use a linear layer over action message INLINEFORM2 to obtain action embedding INLINEFORM3 for time step INLINEFORM4 . paragraph4 0.1ex plus0.1ex minus.1ex-1em Natural Language The message INLINEFORM0 contains information about observations and actions, so we use a recurrent neural network with attention mechanism to extract the relevant observation and action embeddings. Specifically, we encode the message INLINEFORM1 , consisting of INLINEFORM2 tokens INLINEFORM3 taken from vocabulary INLINEFORM4 , with a bidirectional LSTM: DISPLAYFORM0   where INLINEFORM0 is the word embedding look-up table. We obtain observation embedding INLINEFORM1 through an attention mechanism over the hidden states INLINEFORM2 : DISPLAYFORM0  where INLINEFORM0 is a learned control embedding who is updated through a linear transformation of the previous control and observation embedding: INLINEFORM1 . We use the same mechanism to extract the action embedding INLINEFORM2 from the hidden states. For the observation embedding, we obtain the final representation by summing positionally gated embeddings, i.e., INLINEFORM3 .

The task ends when the evaluation is successful—i.e., when INLINEFORM3 —or otherwise continues until a total of three failed attempts. The additional attempts are meant to ease the task for humans, as we found that they otherwise often fail at the task but still end up close to the target location, e.g., at the wrong corner of the correct intersection.

Data Collection	We crowd-sourced the collection of the dataset on Amazon Mechanical Turk (MTurk). We use the MTurk interface of ParlAI  to render 360 images via WebGL and dynamically display neighborhood maps with an HTML5 canvas. Detailed task instructions, which were also given to our workers before they started their task, are shown in Appendix SECREF15 . We paired Turkers at random and let them alternate between the tourist and guide role across different HITs.

Tourist: It's a big hotel it's possible. Tourist: ACTION:FORWARD ACTION:TURNLEFT ACTION:FORWARD ACTION:TURNRIGHT Tourist: I'm on the hotel corner Guide: EVALUATE_LOCATION

We propose to learn such state-transitions from the tourist message through a differentiable attention-mask over the spatial dimensions of a 3x3 convolution. paragraph4 0.1ex plus0.1ex minus.1ex-1em MASC We linearly project each predicted action embedding INLINEFORM0 to a 9-dimensional vector INLINEFORM1 , normalize it by a softmax and subsequently reshape the vector into a 3x3 mask INLINEFORM2 : DISPLAYFORM0   We learn a 3x3 convolutional kernel INLINEFORM0 , with INLINEFORM1 features, and apply the mask INLINEFORM2 to the spatial dimensions of the convolution by first broadcasting its values along the feature dimensions, i.e. INLINEFORM3 , and subsequently taking the Hadamard product: INLINEFORM4 . For each action step INLINEFORM5 , we then apply a 2D convolution with masked weight INLINEFORM6 to obtain a new map embedding INLINEFORM7 , where we zero-pad the input to maintain identical spatial dimensions. paragraph4 0.1ex plus0.1ex minus.1ex-1em Prediction model We repeat the MASC operation INLINEFORM0 times (i.e.

Tourist Localization	The designed navigation protocol relies on a trained localization model that predicts the tourist's location from a communicated message. Before we formalize this localization sub-task in Section UID21 , we further introduce two simplifying assumptions—perfect perception and orientation-agnosticism—so as to overcome some of the difficulties we encountered in preliminary experiments. paragraph4 0.1ex plus0.1ex minus.1ex-1em Perfect Perception Early experiments revealed that perceptual grounding of landmarks is difficult: we set up a landmark classification problem, on which models with extracted CNN  or text recognition features  barely outperform a random baseline—see Appendix SECREF13 for full details. This finding implies that localization models from image input are limited by their ability to recognize landmarks, and, as a result, would not generalize to unseen environments. To ensure that perception is not the limiting factor when investigating the landmark-grounding and action-grounding capabilities of localization models, we assume “perfect perception”: in lieu of the 360 image view, the tourist is given the landmarks at its current location. More formally, each state observation INLINEFORM0 now equals the set of landmarks at the INLINEFORM1 -location, i.e. INLINEFORM2 .

If the INLINEFORM3 -location does not have any visible landmarks, we return a single “empty corner” symbol. We stress that our findings—including a novel architecture for grounding actions into an overhead map, see Section UID28 —should carry over to settings without the perfect perception assumption. paragraph4 0.1ex plus0.1ex minus.1ex-1em Orientation-agnosticism We opt to ignore the tourist's orientation, which simplifies the set of actions to [Left, Right, Up, Down], corresponding to adding [(-1, 0), (1, 0), (0, 1), (0, -1)] to the current INLINEFORM0 coordinates, respectively. Note that actions are now coupled to an orientation on the map—e.g. up is equal to going north—and this implicitly assumes that the tourist has access to a compass. This also affects perception, since the tourist now has access to views from all orientations: in conjunction with “perfect perception”, implying that only landmarks at the current corner are given, whereas landmarks from different corners (e.g. across the street) are not visible.

Model	This section outlines the tourist and guide architectures. We first describe how the tourist produces messages for the various communication channels across which the messages are sent. We subsequently describe how these messages are processed by the guide, and introduce the novel Masked Attention for Spatial Convolutions (MASC) mechanism that allows for grounding into the 2D overhead map in order to predict the tourist's location.

The Tourist	For each of the communication channels, we outline the procedure for generating a message INLINEFORM0 . Given a set of state observations INLINEFORM1 , we represent each observation by summing the INLINEFORM2 -dimensional embeddings of the observed landmarks, i.e. for INLINEFORM3 , INLINEFORM4 , where INLINEFORM5 is the landmark embedding lookup table. In addition, we embed action INLINEFORM6 into a INLINEFORM7 -dimensional embedding INLINEFORM8 via a look-up table INLINEFORM9 . We experiment with three types of communication channel. paragraph4 0.1ex plus0.1ex minus.1ex-1em Continuous vectors The tourist has access to observations of several time steps, whose order is important for accurate localization. Because summing embeddings is order-invariant, we introduce a sum over positionally-gated embeddings, which, conditioned on time step INLINEFORM0 , pushes embedding information into the appropriate dimensions. More specifically, we generate an observation message INLINEFORM1 , where INLINEFORM2 is a learned gating vector for time step INLINEFORM3 . In a similar fashion, we produce action message INLINEFORM4 and send the concatenated vectors INLINEFORM5 as message to the guide.

We can interpret continuous vector communication as a single, monolithic model because its architecture is end-to-end differentiable, enabling gradient-based optimization for training. paragraph4 0.1ex plus0.1ex minus.1ex-1em Discrete symbols Like the continuous vector communication model, with discrete communication the tourist also uses separate channels for observations and actions, as well as a sum over positionally gated embeddings to generate observation embedding INLINEFORM0 . We pass this embedding through a sigmoid and generate a message INLINEFORM1 by sampling from the resulting Bernoulli distributions:  INLINEFORM0  The action message INLINEFORM0 is produced in the same way, and we obtain the final tourist message INLINEFORM1 through concatenating the messages. The communication channel's sampling operation yields the model non-differentiable, so we use policy gradients  ,  to train the parameters INLINEFORM0 of the tourist model. That is, we estimate the gradient by INLINEFORM1   where the reward function INLINEFORM0 is the negative guide's loss (see Section SECREF25 ) and INLINEFORM1 a state-value baseline to reduce variance. We use a linear transformation over the concatenated embeddings as baseline prediction, i.e.

The Guide	Given a tourist message INLINEFORM0 describing their observations and actions, the objective of the guide is to predict the tourist's location on the map. First, we outline the procedure for extracting observation embedding INLINEFORM1 and action embeddings INLINEFORM2 from the message INLINEFORM3 for each of the types of communication. Next, we discuss the MASC mechanism that takes the observations and actions in order to ground them on the guide's map in order to predict the tourist's location. paragraph4 0.1ex plus0.1ex minus.1ex-1em Continuous For the continuous communication model, we assign the observation message to the observation embedding, i.e. INLINEFORM0 . To extract the action embedding for time step INLINEFORM1 , we apply a linear layer to the action message, i.e. INLINEFORM2 . paragraph4 0.1ex plus0.1ex minus.1ex-1em Discrete For discrete communication, we obtain observation INLINEFORM0 by applying a linear layer to the observation message, i.e. INLINEFORM1 .

We represent the guide's map as INLINEFORM0 , where in this case INLINEFORM1 , where each INLINEFORM2 -dimensional INLINEFORM3 location embedding INLINEFORM4 is computed as the sum of the guide's landmark embeddings for that location. paragraph4 0.1ex plus0.1ex minus.1ex-1em Motivation While the guide's map representation contains only local landmark information, the tourist communicates a trajectory of the map (i.e. actions and observations from multiple locations), implying that directly comparing the tourist's message with the individual landmark embeddings is probably suboptimal. Instead, we want to aggregate landmark information from surrounding locations by imputing trajectories over the map to predict locations. We propose a mechanism for translating landmark embeddings according to state transitions (left, right, up, down), which can be expressed as a 2D convolution over the map embeddings. For simplicity, let us assume that the map embedding INLINEFORM0 is 1-dimensional, then a left action can be realized through application of the following INLINEFORM1 kernel: INLINEFORM2 which effectively shifts all values of INLINEFORM3 one position to the left.

Comparisons	To better analyze the performance of the models incorporating MASC, we compare against a no-MASC baseline in our experiments, as well as a prediction upper bound. paragraph4 0.1ex plus0.1ex minus.1ex-1em No MASC We compare the proposed MASC model with a model that does not include this mechanism. Whereas MASC predicts a convolution mask from the tourist message, the “No MASC” model uses INLINEFORM0 , the ordinary convolutional kernel to convolve the map embedding INLINEFORM1 to obtain INLINEFORM2 . We also share the weights of this convolution at each time step. paragraph4 0.1ex plus0.1ex minus.1ex-1em Prediction upper-bound Because we have access to the class-conditional likelihood INLINEFORM0 , we are able to compute the Bayes error rate (or irreducible error). No model (no matter how expressive) with any amount of data can ever obtain better localization accuracy as there are multiple locations consistent with the observations and actions.

Localization-based Baseline	Table TABREF36 shows results for the best localization models on the full task, evaluated via the random walk protocol defined in Algorithm SECREF12 . paragraph4 0.1ex plus0.1ex minus.1ex-1em Comparison with human annotators Interestingly, our best localization model (continuous communication, with MASC, and INLINEFORM0 ) achieves 88.33% on the test set and thus exceed human performance of 76.74% on the full task. While emergent models appear to be stronger localizers, humans might cope with their localization uncertainty through other mechanisms (e.g. better guidance, bias towards taking particular paths, etc). The simplifying assumption of perfect perception also helps. paragraph4 0.1ex plus0.1ex minus.1ex-1em Number of actions Unsurprisingly, humans take fewer steps (roughly 15) than our best random walk model (roughly 34). Our human annotators likely used some form of guidance to navigate faster to the target.

Tourist Generation Models	After training the supervised tourist model (conditioned on observations and action from human expert trajectories), there are two ways to train an accompanying guide model. We can optimize a location prediction model on either (i) extracted human trajectories (as in the localization setup from human utterances) or (ii) on all random paths of length INLINEFORM0 (as in the full task evaluation). Here, we investigate the impact of (1) using either human or random trajectories for training the guide model, and (2) the effect of varying the path length INLINEFORM1 during the full-task evaluation. For random trajectories, guide training uses the same path length INLINEFORM2 as is used during evaluation. We use a pre-trained tourist model with greedy decoding for generating the tourist utterances. Table TABREF40 summarizes the results. paragraph4 0.1ex plus0.1ex minus.1ex-1em Human vs random trajectories We only observe small improvements for training on random trajectories. Human trajectories are thus diverse enough to generalize to random trajectories.

To cover the full field of view, we extract two images per orientation, with their horizontal focus point 30 degrees apart. Hence, we obtain eight images per 360 image with corresponding orientation INLINEFORM0 . We run the following pre-trained feature extractors over the extracted images: For the text recognition model, we use a learned look-up table INLINEFORM0 to embed the extracted text features INLINEFORM1 , and fuse all embeddings of four images through a bag of embeddings, i.e., INLINEFORM2 . We use a linear layer followed by a sigmoid to predict the probability for each class, i.e. INLINEFORM3 . We also experiment with replacing the look-up embeddings with pre-trained FastText embeddings  . For the ResNet model, we use a bag of embeddings over the four ResNet features, i.e. INLINEFORM4 , before we pass it through a linear layer to predict the class probabilities: INLINEFORM5 . We also conduct experiments where we first apply PCA to the extracted ResNet and FastText features before we feed them to the model. To account for class imbalance, we train all described models with a binary cross entropy loss weighted by the inverted class frequency.

We create a 80-20 class-conditional split of the dataset into a training and validation set. We train for 100 epochs and perform early stopping on the validation loss. The F1 scores for the described methods in Table TABREF65 . We compare to an “all positive” baseline that always predicts that the landmark class is visible and observe that all presented models struggle to outperform this baseline. Although 256-dimensional ResNet features achieve slightly better precision on the validation set, it results in much worse recall and a lower F1 score. Our results indicate that perceptual grounding is a difficult task, which easily merits a paper of its own right, and so we leave further improvements (e.g. better text recognizers) for future work.

Tourist: ACTION:TURNLEFT Guide: You need to go to the intersection in the northwest corner of the map Tourist: ACTION:TURNLEFT Tourist: There appears to be a bank behind me. Tourist: ACTION:TURNLEFT ACTION:TURNLEFT ACTION:TURNRIGHT ACTION:TURNRIGHT Guide: Ok, turn left then go straight up that road Tourist: ACTION:TURNLEFT ACTION:TURNLEFT ACTION:TURNLEFT ACTION:FORWARD ACTION:TURNRIGHT  ACTION:FORWARD ACTION:FORWARD ACTION:TURNLEFT ACTION:TURNLEFT ACTION:TURNLEFT Guide: There should be shops on two of the corners but you  need to go to the corner without a shop. Tourist: ACTION:FORWARD ACTION:FORWARD ACTION:FORWARD ACTION:TURNLEFT ACTION:TURNLEFT Guide: let me know when you get there. Tourist: on my left is Radio city Music hall Tourist: ACTION:TURNLEFT ACTION:FORWARD ACTION:TURNLEFT ACTION:TURNRIGHT ACTION:TURNRIGHT Tourist: I can't go straight any further. Guide: ok. turn so that the theater is on your right.

That is, the dialogue is driven by uncertainty in the tourist's current location and the properties of the target location: if the exact location and orientation of the tourist were known, it would suffice to communicate a sequence of actions.

Question: What language do the agents talk in?
Answer: English