Title: COMMON SENSE AND SEMANTIC-GUIDED NAVIGA-TION VIA LANGUAGE IN EMBODIED ENVIRONMENTS Anonymous

Abstract: One key element which differentiates humans from artificial agents in performing various tasks is that humans have access to common sense and semantic understanding, learnt from past experiences. In this work, we evaluate whether common sense and semantic understanding benefit an artificial agent when completing a room navigation task, wherein we ask the agent to navigate to a target room (e.g. "go to the kitchen"), in a realistic 3D environment. We leverage semantic information and patterns observed during training to build the common sense which guides the agent to reach the target. We encourage semantic understanding within the agent by introducing grounding as an auxiliary task. We train and evaluate the agent in three settings: (i) imitation learning using expert trajectories (ii) reinforcement learning using Proximal Policy Optimization and (iii) self-supervised imitation learning for fine-tuning the agent on unseen environments using auxiliary tasks. From our experiments, we observed that common sense helps the agent in long-term planning, while semantic understanding helps in short-term and local planning (such as guiding the agent when to stop). When combined, the agent generalizes better. Further, incorporating common sense and semantic understanding leads to 40% improvement in task success and 112% improvement in success per length (SPL) over the baseline during imitation learning. Moreover, initial evidence suggests that the cross-modal embeddings learnt during training capture structural and positional patterns of the environment, implying that the agent inherently learns a map of the environment. It also suggests that navigation in multi-modal tasks leads to better semantic understanding.

1. INTRODUCTION:

Humans utilize semantic information and common sense knowledge when exploring unseen environments. For instance, we do not need to ask for detailed instructions to navigate to a restroom in a new restaurant. On the other hand, artificial agents find it challenging to perform grounded language navigation tasks (e.g."go to the kitchen") or embodied question answering (e.g."what color is the car?") in realistic 3D environments and, especially in unfamiliar environments, tend to fail miserably (Tangiuchi et al., 2019) . We hypothesize that common sense and semantic understanding can benefit artificial agents in the same way that they benefit humans. For example, in the task of navigating around the house, common sense understanding can be helpful in long-term planning and setting the general course of the trajectory. When trying to go to the kitchen, it is useful to know that a dining room is likely close to the kitchen, and that a hallway is likely close to the dining room. On the other hand, semantic understanding (i.e. a deeper understanding of the layout of each room, objects that are usually in it, etc.) should help in choosing better local actions (such as "stop" when the target room is identified or "go forward" when the target room is in the vicinity of the current view). In this work, we evaluate the role of common sense and semantic understanding in embodied agents, using concept-driven navigation (RoomN av (Wu et al., 2018b) ) as a testbed.

With advancements in 3D simulated environments such as Habitat AI (Manolis Savva et al., 2019) or MatterPort3D (Chang et al., 2017) , it is possible to train agents that can interact with these multimodal environments and perform a variety of embodied tasks such as following instructions, answering questions, or navigating. Figure 1 depicts the RoomN av task, wherein an agent is put in a realistic 3D house environment and is given an instruction to navigate to a target room by performing a sequence of actions: turn left, turn right, move forward, or stop. We focus on the RoomNav task as it allows us to investigate several research questions related to grounded language understanding, such as (1) does common sense modeling help navigation? (2) does semantic understanding facilitate navigation and does navigation lead to better semantic representations? (3) can auxiliary tasks help in hard exploration problems? (4) is the agent able to approximate the layout of the environments?

We leverage the semantic information and patterns observed during training on the RoomN av task (such as next room, and sequence of rooms observed along the trajectories) to build the common sense which guides the agent to reach the target room. We enforce semantic understanding by performing two auxiliary tasks: (i) grounding during navigation (asking the agent to predict the current and nearby rooms from current view as depicted in figure 1 ) (ii) grounding after navigation (asking questions such as "did you see a bathroom on your way). The common sense module, therefore, captures sequential and structural information of rooms and is used to guide the agent in a hierarchical fashion (Kulkarni et al., 2016; Wernsdorfer & Schmid, 2014) . Inspired by Wang et al. (2018) , we leverage the idea of self-supervised imitation learning (SIL), which is fine-tuning the agent on unseen environments using a cycle-reconstruction loss obtained by reversing the original problem and using it as a critic. Unlike previous works, we perform SIL by introducing auxiliary tasks related to semantic understanding. We teach the agent how to perform semantic understanding and then use that knowledge to make the agent get familiar with unseen environment. We also address the challenge of multiple targets (for example, a house can have multiple bedrooms) by modifying the loss and reward function (section 4), a problem which has been widely ignored in previous works by filtering out such scenarios to avoid ambiguity. lastly, we showcase that crossmodal embeddings trained with semantic and common sense understanding mimic structural and positional patterns, which helps in effective planning and overall navigation tasks.

2. RELATED WORK:

Embodied Tasks: Significant research has been done around visual and video question answering (Agrawal et al., 2017; Das et al., 2016; 2017b; Le et al., 2019; de Vries et al., 2017, among others) . Recently, some work has been done around grounded language understanding (Harnad, 1999; Hermann et al., 2017) , wherein an agent interacts with and navigates through a simulated 3D environment to complete some tasks such as room navigation, finding an object (Wijmans et al., 2019; Wu et al., 2018b) , embodied question answering (Das et al., 2017a; 2018; Gordon et al., 2017; Manolis Savva et al., 2019; Mirowski et al., 2018) , and following instructions (Anderson et al., 2017; Fried et al., 2018; Shah et al., 2018; Wang et al., 2018) . Although researchers have significantly advanced the state of the art in these tasks, the fundamental question of how language and semantics facilitate navigation and how navigation helps facilitate semantic understanding is yet to be fully addressed.

Robustification through Reinforcement Learning: While Wu et al. (2018b ) used Deep Deterministic Policy Gradient (DDPG, Heess et al. (2015) ) and Asynchronous Advantage Actor Critic (A3C, Mnih et al. (2016) ) to evaluate generalizability aspects of the agents for the RoomN av task on the semantically rich House3D environment, their policies did not leverage any common sense or knowledge-grounded semantic information available in the environment.

Understanding and Common Sense: Hermann et al. ( 2017) analyzed the problem of how agents learn to interpret instructions and how they generalize in one-shot and multi-task settings through reinforcement and unsupervised learning using the DeepMind Lab framework. Their study show-cased the importance of prior semantic knowledge and curriculum learning for improved generalization and faster task completion. These experiments were performed in simple one or two room settings (Beattie et al., 2016) with few objects in the environment. On the other hand, there are some environments which are realistic such as House3D (Wu et al., 2018a ) or MatterPort3D (Chang et al., 2017) which constitutes of indoor houses with multiple rooms containing a wide variety of objects. However, limited amount of research has been done in these environments around understanding or common sense. Work done by Yang et al. (2019) is the closest research which has been done around the ideas that we propose. The authors used an external knowledge graph and fed in a Graph Convolutional Network (Kipf & Welling, 2017) to encode the knowledge which is then fed as priors for object navigation. In our work, in addition to implicitly incorporating common sense understanding, we teach the agent how to perform semantic understanding itself via auxiliary tasks, which helps the agent in two ways: (i) guiding the agent to help navigate throughout the trajectory (unlike target alone in case of prior work) and (ii) further fine-tune the agent on unseen environments through self-supervised imitation learning.

Our objective is not to build an agent which outperforms state-of-the-art in existing embodied task. The primary focus of our research is to evaluate if common sense and semantic understanding facilitate navigation and if navigation helps in better semantic understanding. We propose a variety of ways in which common sense and semantic understanding can be fed to the agent in order to perform exhaustive experiments to interpret how common sense and semantic understanding impact the navigation task. We choose RoomNav task because it involves planning and semantic understanding. RoomNav task has been addressed on Habitat environment Manolis Savva et al. (2019) previously, however, the data used is not accessible anymore. Moreover, we select MatterPort3D environment for our experiments which unlike Habitat is realistic in nature and is harder to address. To the best of our knowledge, we do not know of any other work related to RoomN av on MatterPort3D environment and therefore we could not directly compare against previous work. Input: We have four kinds of inputs (shown in green in Figure 2 ): (i) task specific instruction (e.g. "go to the kitchen") (ii) RGB values for each state, (iii) previous action, and (iv) semantic information such as room annotations or generating grounding questions. Semantic information, either for obtaining loss in semantic predictions or for generating questions for grounding, is used during training only as we are not aware of semantics on unseen environments. Following previous work on embodied navigation (Wang et al., 2018 )), we extract panoramic image features using a fixed pre-trained ResNet-152 (He et al., 2015) . The difference is that we turn the agent 90 degrees to the left and right, respectively, to get a 90-degree view for each direction (in a total of 270 degrees) at each timestep. We extract and concatenate images features and pass them through a single layer Feed Forward Neural Network to obtain the visual representation. We use RGB features only as opposed to using other sensors such as semantic masking features (Wu et al., 2018b) or depth features so that the agent will learn to perform understanding of the environment rather than memorizing segments or avoiding obstructions.

3. AGENT ARCHITECTURE:

Baseline Model: We use an LSTM baseline following the work in Das et al. (2017a) for our experiments. The components shown in black color in figure 2 depict the baseline, which consists of an LSTM navigator. The navigator receives previous action, visual representation obtained from ResNet, and target room embedding as input, and predicts one of the four possible actions at each step. Rationale behind choosing a straightforward LSTM as baseline against a more state-of-the-art architecture is to evaluate how much common sense and semantic understanding contributes in task completion and to evaluate a variety of ways in which CS and SU can be incorporated within the agent.

3.1. COMMON SENSE AND SEMANTICALLY-GROUNDED AGENT:

Our architecture consists of the following three modules as depicted in Figure 2: a. Semantically-Grounded Navigator (SGN): The navigator (shown in black in Figure 2 ) is an LSTM model which generates a state at each step that is used for multiple tasks: (i) action prediction to perform one of four possible actions at each step: go forward (0.25m), turn left (10 degrees), turn right (10 degrees), and stop (ii) semantic grounding via detecting the current room (SU CR), and (iii) semantic grounding via post navigation grounding (SU PN), that is generating a response in the final state when the agent predicts "stop". The SGN takes the RoomN av instruction at each step (which remains static throughout the task), visual representation (which changes at each step) as input. We add a linear layer for each of the three tasks, however, for action prediction we also obtain the contextual representation by attending to the current SGN state over the generic sequence of rooms (between source and target room) generated by the common sense planning module (CS RS). The attention helps the navigator move towards rooms which are closer to the target. Both the contextual representation and SGN state are used at each step for action prediction.

b. Common Sense Planning Module (CS)::

We argue that realistic house environments follow structural patterns (such as a refrigerator is usually placed in the kitchen) and sequential patterns (such as the kitchen is usually near the dining room). We incorporate this information via CS modules. For distant destinations, the agent may not be able to utilize a static instruction for route planning. Instead, we design a next room prediction module (CS Nxt) to help the agent navigate to an intermediate target along the route. Next room prediction is trained during the imitation learning phase since during the RL phase the agent might deviate from optimal trajectories (the sequence of rooms in sub-optimal trajectories will not reflect common sense). Next room prediction is a function of the current state (obtained from SGN) and the sequence of rooms between the source and the target room. To incorporate this, the room sequence common sense model (CS RS) is trained simultaneously to capture generic room sequence patterns (between source and target rooms) across the houses through sequences observed in training trajectories. However, such an intermediate target prediction model does not always have explicit information about what rooms are near the target room. Therefore, we design a backward room sequence model using the LSTM to generate sequences starting from the target room. We get the contextual representation by attending to the output of the backward room sequence and predict the next intermediate room.

c. Semantic Understanding (SU)::

To incorporate SU, apart from the two auxiliary tasks for the SGN described previously, we introduce another task for detecting the current and nearby rooms, using a separate Multi-layer Perceptron (MLP) room detection model (SU RD). The MLP takes the representation of the panoramic view from ResNet as input and generates a hidden state before performing classification over four rooms (left (L), front (F), current (C), right (R)). We also pass the hidden state of the MLP to the SGN for better action prediction, especially in cases where the panoramic image contains multiple rooms (as shown in figure 1 ). Regarding the two auxiliary tasks performed by SGN, we add separate output heads on top of SGN. We add a linear layer to perform current room detection (SU CR) using the state obtained from SGN at each step and another linear layer to produce a "yes" or "no" response to the grounded question (e.g. "did you see a bathroom on your way?"). The questions -generated by sampling from the annotations and using templates for positive and negative cases -are used to incorporate semantic understanding (SU PN) within the agent. The rationale behind having two models for current room detection (SGN and SU RD) is that we can use the latter to fine-tune SGN on unseen environments as explained in section 4.

3.2. LEARNING:

a. Training and Fine-tuning on Unseen Environments: We train the agent in three ways: (i) imitation learning with shortest path trajectories available during the training (ii) reinforcement learning, to further robustify the agent after imitation learning, using Proximal Policy Optimization (P P O) on the training environments, and (iii) self-supervised imitation learning on unseen environments, inspired by the work from Wang et al. (2018) . Self-supervision is the reason we have two room detection models (SU RD and SU CR): we use the rooms detected from SU RD as the ground truth for performing self-supervised imitation learning on unseen environments. We let SU CR get further fine-tuned on unseen environments by obtaining the loss based on SU RD's output. Further, we sample grounded questions using the rooms detected by SU RD on the trajectory to perform SU PN on unseen environments. Losses obtained from these auxiliary tasks on unseen environments update the SGN which is also used for navigation, hence continues to update the semantic understanding on newer environments similar to what humans do.

b. Loss and Rewards: During imitation learning, apart from the main action prediction task, we perform five auxiliary tasks: (i) next room detection (CS Nxt) (ii) target to source room sequence prediction (CS RS) (iii) current room detection using SU CR (iv) post navigation response generation (SU PN) and (v) current and nearby room predictions (SU RD). In total, we have six losses during imitation learning including action prediction. Equation 1 shows the sum of all the losses during imitation learning. Equation 2 corresponds to per-step loss for each task as a function of state and input. State in equation 2, corresponds to the SGN state of the LSTM once all the four inputs (target room embedding, image representation from ResNet, previous action and hidden state of room detection MLP) are passed through it, as represented in equation 3. During self-supervised imitation learning, we fine-tune the agent using two auxiliary tasks on unseen environments using the labels obtained from SU RD MLP model. Equation 4 corresponds to self-supervised imitation learning loss, with losses (L ) coming from the models. During the reinforcement learning phase, since we fine-tune the policy on the already seen environment, we freeze the CS and SU modules to avoid introducing noise. We only update the action layer and the SGN with per step reward shown in equation 5. We have a reward for agent getting closer to the target, a success reward, and a discounted future reward. We set the discount factor to be 0.99 in our experiments and success reward to be 10. All the λ's corresponds to hyper-parameters across the equations, c t correspond to contextual representation as shown in Figure 2 and z (2)

s t = SGN (action t-1 , ResN et(pan img t ), M LP (pan img), target room emb) (3) L self -supervised-IL = λ su cr * L SU CR + λ su pn * L SU P N (4) R(s t , a t ) = λ td * (d t-1 -d t ) + λ suc * (succ reward) + T t =t+1 γ t -1 r(s t , a t ) (5)

4. EXPERIMENTS AND RESULTS:

Data and Environment: We use the Habitat environment (Manolis Savva et al., 2019) with the MatterPort3D dataset for all of our tasks. Habitat's task is Point Navigation, where an agent needs to navigate from a source coordinate to a target coordinate. We adapt this task to form RoomNav by replacing the target coordinates with the corresponding 27 room types annotated in the dataset (excluding "other room"). After removing games where the target is in the same room as the source room and where the target is in the border of several rooms, we generated 53 houses (5020 games) for training, 11 houses for validation (168 games), and 15 houses (324 games) for testing. We use the same measure as Point Navigation task to define the complexity of each game by the geodisic distance of the shortest path. The average number of rooms between the source and target room is 2.41, 3.01, and 4.06, respectively for easy, medium, and hard games in the training data.

Evaluation Metrics: We use two metrics: (i) Success rate: rate of the games in which agent enters the target room. (ii) Success Per Length (SPL) which is a success metric normalized with respect to the shortest path (Wang et al., 2018) . SPL requires that the agent chooses to terminate in the target room for the game to be considered successful. In our evaluation, we also calculate non-stop SPL where we relax this requirement of termination. 

Model:



4.1. IMITATION LEARNING:

From Table 1 , it can be observed that common sense planning and semantic understanding helps in Imitation Learning experiments across the board when compared to the LSTM only baseline.

Common Sense Planning: We incorporate common sense in two ways: (i) next room guidance (CS Nxt) and (ii) generic room sequence between target and source room (CS RS) as described in section 3. As anticipated, CS modules help in long-term planning hence helps in medium and harder games more than the easy games. Next room guidance alone leads to the second best SP L (80% improvement over baseline) and hard game success rate (40% improvement over baseline). Room sequence module (CS RS) alone helps in harder games. When combined with (CS Nxt), the performance on easy game improves significantly, while performance on medium games outperforms all other settings. Although, performance on harder games degrades relatively, which implies that it might be hard for the room sequence module to learn longer patterns.

Semantic Understanding: SU is performed in three ways: (i) SGN predicting current room (SU CR) (ii) SGN performing grounding post navigation (SU PN) and (iii) MLP predicting nearby rooms (SU RD). As expected, SU generally helps in short-term planning through better semantic detection leading to higher SP L scores with least average number of steps required to complete the task. However, (SU PN) do not follow similar pattern, since grounding in case of (SU PN) is performed in terminal state, hence it directly does not impact turn-level action prediction. In fact, it performs significantly better than most settings on medium and hard games because it lets the SGN focus more on action prediction during intermediate steps, while ensuring semantic understanding at terminal state. Early stopping could also be the reason why (SU RD) and (SU CR) individually might perform worse on medium and harder games. When combined together, we get the best SPL score (0.141, 110% better than the baseline. However improvement can be seen only on easy games. SGN in these settings might tend to focus more on auxiliary task than the action prediction task. We hypothesize that hyperparameter tuning is important in combining different modules.

Common Sense Planning and Semantic Understanding: CS when combined with SU leads to best overall success (25% relative improvement overall) when SGN is trained with (CS Nxt) and (SU PN) because former helps in long-term planning and the latter leads to better semantic detection guides the agent in effective stopping. As expected, most improvement is observed on easy games (40% relative improvement over baseline), performance improvement on medium games is intermediate, while that on hard games is second best. Adding (SU CR) to (CS Nxt) or (CS Nxt) and (CS RS) degrades the performance caused by early stopping. Moreover, it can be observed that when several auxiliary tasks are introduced to the agent, the performance generally degrades as the agent tends to focus more on the auxiliary tasks than the action prediction task. After bootstrapping the policy with imitation learning, we fine-tune the policy using PPO. We compare baseline model with the best performing model (CS Nxt) on hard games. Figure 3 depicts the performance of both the models on hard games of test environment.

From our initial analysis, we observed that agent trained with common sense generalizes better on unseen environments on success and SPL metrics. Although, performance of both the models degrade initially on the success metrics and improves on SPL, the results obtained with existing RL experiments should be considered as preliminary findings.

4.3. SELF-SUPERVISED IMITATION LEARNING (SIL):

To perform self-supervised imitation learning (SIL), we either use SU CR or SU PN as the auxiliary task to fine-tune SGN by assuming the output of SU RD as the ground truth. We do not use CS Nxt model for performing SIL because performance of CS modules degrades significantly when step level auxiliary tasks are introduced, as depicted earlier. After performing SIL for the first 20 steps on unseen environments, using the SU CR for fine-tuning helps the overall accuracy by 6.5% with respect to SU CR + SU RD baseline. When SU PN is used for fine-tuning instead, we achieve an improvement of 9.1% with respect to SU PN + SU RD baseline. With more SIL steps, performance degrades due to introduction of noise, as the current room prediction from the SU RD model is not perfect/ground truth. Through SIL, we showcase that the agent can be made to update the semantic understanding through navigation.

4.4. CROSS-MODAL EMBEDDINGS:

Traditional word embeddings (e.g. Pennington et al. (2014) ), are functions of words or semantic entities appearing in similar contexts and may not capture the visual and structural properties of entities in a realistic 3D world. Therefore, we use cross-modal embeddings, which are randomly initialized Embeddings of SU CR + SU RD model after fine-tuning using SIL and then trained with common sense and semantic understanding across multiple games. We qualitatively analyze these embeddings to see if they reflect the structural and visual characteristics of the environment. Figure 4 represents the top-view of a training environment and Figure 5 visualizes the embeddings trained using the (SU CR + SU RD) model in two dimensions after aligning. We finetuned the agent (and embeddings) on the house shown in figure 4 using self-supervised imitation learning and visualize it in figure 6 , after aligning it with respect to the original map for comparison. It can be observed that fine-tuned embeddings tend to mimic the structural and positional patterns of the house. Embeddings obtained before fine-tuning mimic the average structural pattern of rooms across all the houses, such as the dining room is close to the living room. Apart from positional characteristics, the cross-modal embeddings also mimic visual properties, for example the bathroom and the TV room are usually separated before and after fine-tuning, because these rooms are visually distinct from other rooms.

Cross-modal Embeddings as an alternative to SLAM: From this observation, we can conclude that after fine-tuning the agent on a given house using SIL, embeddings tend to mimic the layout of the house. These approximate maps obtained from embeddings can be an alternative to SLAM algorithm (Durrant-Whyte & Bailey, 2006) . We leave detailed comparison to future work.

Navigation facilitates Semantic Understanding: The embeddings or semantic representations learnt by our approach contain information beyond what is captured in language, such as structural and visual characteristics, which is similar to how humans represent semantics in their mind. From the analysis performed above, we show that cross-modal navigation improves semantic understanding by capturing information from various modalities into the embeddings.

5. CONCLUSION:

The only thing which humans have when they navigate in unseen environments is common sense and semantic understanding obtained through past experiences. In this work, we investigate if this also holds in artificial agents, by incorporating common sense and making the agent semantically aware while performing a room navigation task in realistic 3D environments. We showcased that the agent generalizes better if it is taught to perform common sense and semantic understanding. We introduced semantic grounding within the navigator through multiple auxiliary tasks, and showcased that the agent can be fine-tuned further to generalize better on unseen environments through auxiliary tasks using self-supervised imitation learning. We also showed that the cross-modal embeddings obtained during training tend to capture structural and positional patterns of the houses, implying that the agent learns better planning with common sense and semantic auxiliary tasks.

:



Figures and Tables:

Figure 1: Panoramic view of the agent (left, right, and front views concatenated).

Figure 2: Common Sense and Semantic-Guided Navigation for RoomNav Task. Input to the agent is represented in green. Black components correspond to the baseline navigator model. Purple components are introduced to incorporate common sense planning while pink components are for Semantic Understanding.

t correspond to state in CS RS LSTM.L imitation-learning = λ a * L action + λ cs nxt * L CS N xt + λ cs rs * L CS RS + λ su rd * L SU RD + λ su cr * L SU CR + λ su pn * L SU P N(1) l action = l P (actiont|st) ; l CS N xt = l P (CS N xtt|st,ct) ; l CS RS = l P (CS RSt|target,zt) ; l SU RD = l P (SU RDt|ResN et(pan imgt)) ; l SU CR = l P (SU CRt|st) ; l SU P N = l P (SU P Nt|s last )

Figure 3: SPL and Success Rate vs. Number of frames used for training on hard games of test environment

Figure 4: Top view of house 17DR with dark areas as obstacles Figure 5: Embeddings of SU CR + SU RD model before fine-tuning on 17DR house Figure 6: Embeddings of SU CR + SU RD model after fine-tuning using SIL

Results on Imitation Learning and Self-supervised IL on Test Environments. Metrics: SP L -Shortest path normalized by length; non-stop SPL -Best SP L anywhere in trajectory.

Table content: <table><tr><td/><td>SPL</td><td>non-</td><td>average</td><td>succ.</td><td>easy</td><td>medium</td><td>hard</td></tr><tr><td/><td/><td>stop</td><td>steps</td><td>rate</td><td>succ.</td><td>succ.</td><td>succ.</td></tr><tr><td/><td/><td>SPL</td><td/><td/><td>rate</td><td>rate</td><td>rate</td></tr><tr><td>baseline</td><td colspan="2">0.067 0.239</td><td>119</td><td colspan="2">0.253 0.412</td><td>0.246</td><td>0.175</td></tr><tr><td>CS Nxt</td><td colspan="2">0.120 0.264</td><td>137</td><td colspan="2">0.303 0.441</td><td>0.285</td><td>0.246</td></tr><tr><td>CS RS</td><td colspan="2">0.078 0.236</td><td>114</td><td colspan="4">0.256 0.426 0.2301 0.191</td></tr><tr><td>CS Nxt + CS RS</td><td colspan="2">0.079 0.260</td><td>127</td><td colspan="2">0.306 0.485</td><td>0.300</td><td>0.214</td></tr><tr><td>SU CR</td><td colspan="2">0.085 0.227</td><td>120</td><td colspan="2">0.244 0.471</td><td>0.223</td><td>0.143</td></tr><tr><td>SU RD</td><td colspan="2">0.103 0.245</td><td>114</td><td colspan="2">0.269 0.485</td><td>0.231</td><td>0.191</td></tr><tr><td>SU PN</td><td colspan="2">0.077 0.254</td><td>154</td><td colspan="2">0.293 0.427</td><td>0.292</td><td>0.222</td></tr><tr><td colspan="3">SU PN + SU RD (SIL baseline) 0.056 0.223</td><td>158</td><td colspan="2">0.252 0.48</td><td>0.21</td><td>0.17</td></tr><tr><td colspan="3">SU CR + SU RD (SIL baseline) 0.141 0.248</td><td>118</td><td colspan="2">0.278 0.529</td><td>0.238</td><td>0.19</td></tr><tr><td>CS Nxt + SU CR</td><td colspan="2">0.094 0.259</td><td>119</td><td colspan="2">0.284 0.50</td><td>0.277</td><td>0.175</td></tr><tr><td>CS Nxt + SU PN</td><td colspan="2">0.068 0.260</td><td>146</td><td colspan="2">0.315 0.574</td><td>0.269</td><td>0.222</td></tr><tr><td>CS Nxt + CS RS + SU CR</td><td colspan="2">0.051 0.223</td><td>135</td><td colspan="2">0.253 0.441</td><td>0.215</td><td>0.191</td></tr><tr><td>SIL: SU PN + SU RD</td><td colspan="2">0.072 0.239</td><td>154</td><td colspan="2">0.275 0.529</td><td>0.246</td><td>0.167</td></tr><tr><td>SIL: SU CR + SU RD</td><td colspan="2">0.142 0.253</td><td>116</td><td colspan="2">0.296 0.559</td><td>0.262</td><td>0.191</td></tr></table>

Peer review (iclr2020):
This submission aims to understand whether common sense and semantic understanding can benefit in a room navigation task. The authors design several interesting auxiliary tasks to understand the significance of common sense and semantic understanding for room navigation. They conduct experiments to understand the importance of each auxiliary task empirically. However, the clarity and quality of writing is poor which makes it difficult to follow the manuscript at many places and evaluate its significance. Some auxiliary tasks are not defined well, several ideas are not motivated well and some paragraphs seem unnecessary. I find the experiments inconclusive due to lack of statistical significance testing, insufficient data, and inadequate hyperparameter tuning. More details in the following comments. Comments on Section 3: - The agent architecture description is not easy to follow. Terms are used without defining them. For example, acronyms CS and SU are stated in Sec3 Baseline model without defining what they stand for or what they mean, in Sec 3 Input, what are “ generating grounding questions”?. There are too many acronyms to keep track of and they are often unintuitive until you read the whole section. There are many cross-references between paragraphs which makes it very difficult to follow the description. For example, In Section 3.1 “generic sequence of rooms (between source and target room) generated by the common sense planning module (CS_RS)”. Here, what is a generic sequence of rooms? How are they generated? What is the common sense planning module? Why is it abbreviated CS_RS? - Why is Das et al. (2017a) cited for LSTM baseline? I can think of many papers before this doing reinforcement and Imitation learning using LSTMs or recurrent networks. - I do not understand the motivation behind the proposed method of self-supervised learning. The prediction of SU_RD (room detection) is used as labels for fine-tuning SU_CR (current room detection) and SU_PN (post navigation grounding) in unseen environments. Why would SU_RD generalize well on unseen environments but not SU_CR and SU_PN? - What is the need for SU_CR (current room detection) when SU_RD (room detection) already consists of current room? Why not just use SU_RD as an auxiliary task for SGN instead of SU_CR? - The motivation behind SU_PN is unclear. Why does the agent need to remember which rooms during the trajectory when it stops? How is this supposed to help in navigation? - The definition of some auxiliary tasks is not clear. It is not clear how the labels were generated for some auxiliary tasks. What are the left, front and right rooms in SU_RD? Do they need to be visible in the current image or does the agent need to predict likely rooms which are behind walls? What if there are multiple rooms in one direction? The CS_RS task is also not defined clearly. It seems from Figure 2 that the input to the CS_RS model is the sequence of rooms in the ground truth trajectory but it is not stated anywhere. What is it predicting? What are “generic room sequence patterns”? How are the labels of CS_RS, SU_RD generated? - If CS_RS is taking the sequence of rooms in the ground truth trajectory as input, the model makes a strong assumption of having access to sequence of rooms to reach the target room at test time. It also makes the comparison to the baseline unfair. If it is not taking sequence of rooms as input, what is the input to the CS_RS model? Comments on Section 4: - What is the meaning of ‘games’ in this context? I am assuming trajectories. - What hyperparameters were tuned using the validation set? - Looking at the results in Table 1, I am not sure whether the difference in the performance of different models is significant or not. I believe statistical significance tests need to be conducted especially because the difference between the performance of different models is small and the test set consists of only 324 trajectories, which are further split into easy, medium and hard sets. - The performance of several models is unintuitive in many places. For example, the performance of CS_RS lower than baseline in medium trajectories, CS_RS improves the performance when added to CS_Nxt in easy and medium but not in hard, SU_PN is better than SU_CR in medium but not in easy, however, SU_PN+CS_Nxt is better than SU_CR+CS_Nxt in easy but not in medium. SU_PN+SU_RD is worse than SU_CR+SU_RD across all settings but SU_PN is overall better than SU_CR. The authors have provided some explanation for some of the above but not for all. I am suspecting that many of the above observations are due to the variance in the results, because of small test sets and not due to the difference in auxiliary tasks. And this again raises questions about the statistical significance of all the results. - Several statements in the results indicate inadequate hyperparameter tuning such as “We hypothesize that hyperparameter tuning is important in combining different modules” and “the performance generally degrades as the agent tends to focus more on the auxiliary tasks than the action prediction task”. I believe it is essential to tune the hyperparameters for auxiliary tasks if the goal is to understand the importance of auxiliary tasks empirically. - What is the purpose of Section 4.2? What are we supposed to learn from this experiment? - It is not clear how are the embeddings visualized in Sec 4.4 and Figures 5 and 6. How are the embeddings projected in two dimensions? How are they aligned? Suggestions for improvement: - The writing can be improved significantly. I suggest first describing all the auxiliary tasks and then describing the agent architecture for training the auxiliary tasks. The description of auxiliary tasks can be improved, especially SU_RD and CS_RS. - I suggest increasing the size of the test set and performing statistical significance tests for making sure that the difference in performance numbers is not due to noise/variance. - The manuscript needs to be proof-read. There are some grammatical errors.