Learning with a Mole: Transferable latent spatial representations for navigation without reconstruction
Authors: Guillaume Bono
Leonid Antsfeld
Assem Sadek
Gianluca Monaci
Christian Wolf
Venue: ICLR 2024 poster (2024)
Keywords: Navigation
Embodied AI
Perception
Primary area: applications to robotics, autonomy, planning
TL;DR: Instead of learning to reconstruct, we cast the robotic perception task as a navigation task by a blind auxiliary agent generating a learning signal for the main agent

Abstract: Agents navigating in 3D environments require some form of memory, which should hold a compact and actionable representation of the history of observations useful for decision taking and planning. In most end-to-end learning approaches the representation is latent and usually does not have a clearly defined interpretation, whereas classical robotics addresses this with scene reconstruction resulting in some form of map, usually estimated with geometry and sensor models and/or learning. In this work we propose to learn an actionable representation of the scene independently of the targeted downstream task and without explicitly optimizing reconstruction. The learned representation is optimized by a blind auxiliary agent trained to navigate with it on multiple short sub episodes branching out from a waypoint and, most importantly, without any direct visual observation. We argue and show that the blindness property is important and forces the (trained) latent representation to be the only means for planning. With probing experiments we show that the learned representation optimizes navigability and not reconstruction. On downstream tasks we show that it is robust to changes in distribution, in particular the sim2real gap, which we evaluate with a real physical robot in a real office building, significantly improving performance.

Published as a conference paper at ICLR 2024
LEARNING WITH A MOLE: TRANSFERABLE LATENT SPATIAL REPRESENTATIONS FOR NAVIGATION WITHOUT RECONSTRUCTION
Guillaume Bono, Leonid Antsfeld, Assem Sadek, Gianluca Monaci & Christian Wolf Naver Labs Europe Meylan, France
[firstname. iastname e nnveerlabs........ com
ABSTRACT
Agents navigating in 3D environments require some form of memory, which should hold a compact and actionable representation of the history of observations useful for decision taking and planning. In most end-to-end learning approaches the representation is latent and usually does not have clearly defined interpretation, whereas classical robotics addresses this with scene reconstruction resulting in some form of map, usually estimated with geometry and sensor models and/or learning. this work we propose to learn an actionable representation of the scene independently of the targeted downstream task and without explicitly optimizing reconstruction. The learned representation is optimized by a blind auxiliary agent trained to navigate with it on multiple short sub episodes branching out from a waypoint and, most importantly, without any direct visual observation. We argue and show that the blindness property is important and forces the (trained) latent representation to be the only means for planning. With probing experiments we show that the learned representation optimizes navigability and reconstruction. On downstream tasks we show that it is robust to changes in distribution, in particular the sim2real gap, which we evaluate with a real physical robot real office building, significantly improving performance.
INTRODUCTION
Navigation in 3D environments requires agents to build actionable representations, which we define in Ghosh et al. (2019) as "aim(ing) capture those factors of variation that are important for decision making". Classically, this has been approached by integrating localization and reconstruction through SLAM (Thrun et al., 2005; Bresson al., 2017; Lluvia et al., 2021), followed by planning on these representations. On the other end of the spectrum we can find end-to-end approaches, which map raw sensor data through latent representations directly to actions and are typically trained aarge-scale in simulations from reward (Mirowski et al., 2017; Jaderberg et al., 2017) or with imitation learning (Ding et al., 2019). Even for tasks with low semantics like PointGoal, is not completely clear whether an optimal representation should be "handcrafted" or learned. While trained agents can achieve extremely high success rates of up to 99% (Wijmans et al., 2019; Partsey et al., 2022), this has been reported in simulation. Performance in real environments is far lower, and classical navigation stacks remain competitive in these settings (Sadck et al., 2022). This raises the important question of whether robust and actionable representations should be based on precise reconstruction, and argue that an excess amount of precision can potentially lead to a higher internal sim2real gap and hurt transfer, similar (but not identical) the effect of fidelity in training in simulation (Truong et al., 2022). Interestingly, research in psychology has shown that human vision has not been optimized for high-fidelity 3D reconstruction, but for the usefulness and survival (Prakash et al., 2021).
We argue that artificial agents should follow a similar strategy and we propose tailored auxiliary losses, which are based on interactions with the environment and directly target the main desired property of a latent representation: its usability for navigation. This goal is related to Cognitive Maps, spatial representations built by biological agents known to emerge from interactions (Tolman, 1948; Blodgett, 1929; Menzel, 1973), even in blind agents, biological (Lumelsky & Stepanov, 1987) or artificial ones (Wijmans, 2022; Wijmans et al., 2023).

---

Published as a conference paper at ICLR 2024
Inspired by this line of research, propose learning a latent spatial representation we call Navigability and avoid spending training signals on learning explicitly reconstruct the scene in unnecessary detail, potentially not useful or even harmful for transfer. We augment the amount of information carried by the training signal compared to reward. We consider the ability of per- forming local navigation an essential skill for robot, i.e. the capabiliiy to detect free navigable space, avoid obstacles, and find openings in closed spaces in order to leave them. We propose to learn a representation which optimizes these skills directly, prioritizing usabil- ity over fidelity, as shown in Figure 1. A representation is built by agent through sequential integration of visual observations. This representation is passed to a blind auxiliary agent, which is trained to use it as its sole information to navigate to a batch of intermediate subgoals. Optimizing over the success of the blind aux- iliary agent leads to an actionable representation and can be done independently of the downstream task.
Figure I: Without reconstruction, learn ac- tionable map-like representation computed by an agent of the visual observations collected along its trajectory -- We optimize for its usefudness: a representation estimated at point d is passed to a blind auxiliary agent trained to reach subgoals on short episodes - Solving this requires sufficient latent reconstruction of the scene, and we show that blindness of the auxiliary agent a key property. We train in simulation and transfer to a real environment.
We explore the following questions: (i) Can a latent cognitive map be learned by an agent through the communication with a blind auxiliary agent? (ii) What kind of spatial information does it contain? (iii) Can it be used for downstream tasks? (iv) Is it more transferable than end-to-end training in out-of diitrii-ittiiittoonn situations such as sim2real transfer?
2 RELATED WORK
Navigation with mapping and planning Classical methods typically require map (Burgard et al., 1998; Marder-Eppstein et al., 2010; Macenski et al., 2020) and are composed of three modules: mapping and localization using visual observations or Lidar (Thrun al., 2005; Labbé Michaud, 2019; Bresson et al., 2017: Lluvia et al., 2021). high-level planning (Konolige, 2000; Sethian, 1996) and low-level path planning (Fox et al., 1997; Rösmann et al., 2015). These methods depend on sophisticated noise filtering, temporal integration with precise odometry and loop closure. In comparison, we avoid the intermediate goal of explicit reconstruction, directly optimizing usefulness.
End-to-end training- on the other side of the spectrum we find methods which directly map sensor input to actions and trained latent representations, either flat vectorial like GRU memory, or structured variants: neural metric maps encoding occupancy (Chaplot et al., 2020b), semantics (Chaplot et al., 2020a) or fully latent metric representations (Parisotto & Salakhutdinov, 2018; Beeching et al., 2020b; Henriques & Vedaldi, 2018); 8); neural topological maps (Beeching et al., 2020a; Shah & Levine, 2022; Shah et al., 2021); transformers (Vaswani et al., 2017) adapted to navigation (Fang et al., 2019; Du et al., 2021; Chen al., 2022; Reed et al., 2022); and implicit representations (Marza al., 2023). While these methods share our goal of learning useful and actionable representations, these representations are tied the actual downstream task, whereas our proposed "Navigability" optimizes for local navigation, an important capability common to all navigation tasks.
Pre-text tasks Unsupervised learning and auxiliary tasks share a similar high-level goal with our work, they provide a richer and more direct signal for representation learning. Potential fields (Ramakrishnan et al., 2022) are trained from top down maps and contain unexplored areas and estimates of likely object positions. A similar probabilistic approach has been proposed in RECON by Shah et al. (2021). In (Marza et al., 2022), goal direction directly supervised. Our work can be seen pre-text task directly targeting the usefulness of the representation combined with an inductive bias in the form of the blind auxiliary agent.
Backpropagating through planning - in this line of work downstream objective is backpropa- gated through a differentiable planning module to learn the upstream representation. In (Weerakoon et al., 2022) and similarly in (Dashora et al., 2021), this is cost-map used by a classical planner. Neural-A* * (Yonetani et al., 2021) learns a cost-map used by a differentiable version of A Similarly, Cognitive Mapping and Planning (Gupta al., 2017) learns mapping function by backpropagating
2

---

Published as a conference paper at ICLR 2024
through Value Iteration Networks (Tamar et al., 2016). Our work shares the overall objective, the difference being that (i) we optimize Navigability as an additional pre-text task and, (ii) we introduce the blind agent inductive bias, which minimizes the reactive component of the task and strengthens the more useful temporal integration of a longer history of observations - see Section 4. Somewhat related to work, motion planning performance has been proposed learnable evaluation metric (Philion et al., 2020), and attempts have been made to leverage this kind of metric for learning representations (Philion & Fidler, 2020; Zeng et al., 2021), albeit without application to navigation.
Rim2Real transfer - transferring representations to real world has gained traction since the recent trend large-scale learning in simulated 3D environments (Höfer al., 2020; Chattopadhyay et al., 2021; Kadian et al., 2020; Anderson et al., 2018; Dey et al., 2023). Domain randomization randomizes the factors of variation during training (Peng et al., 2018; Tan et al., 2018), whereas domain adaptation transfers the model to real environments, or in both directions (Truong et al., 2021), through adversarial learning (Zhang et al., 2019), targeting dynamics (Bysenbach al., 2021) or perception (Zhu et al., 2019), or by fine-tuning to the target environment (Sadek et al., 2022). Table 6 in appendix lists some efforts. Our work targets the transfer of a representation by optimizing its usefulness instead of reconstruction.
Biological agents - like rats, have been shown to build Cognitive Maps, spatial representations emerging from interactions (Tolman, 1948), shown partially emerge even when there is no reward nor incentives (Blodgett, 1929; Tolman, 1948). Similarly, chimpanzees are capable of developing greedy search after interactions (Menzel, 1973). Blind biological agents have been shown to be able navigate (Lumelsky & Stepanov, 1987), which has recently been replicated for artificial agents (Wijmans et al., 2023). Certain biological agents have also been shown develop grid, border and place cells (Hafting et al., 2005), which have also been reproduced in artificial agents (Cueva & Wei, 2018; Banino al., 2018). There are direct connections our work, where a representation emerges from interactions of blind agent.
Goal-oriented models are learned through optimizing an objective with respect to subgoals. Hindsight Experience Replay (Andrychowicz et al., 2017) optimizes sample efficiency by reusing unsuccessful trajectories, recasting them as successful ones wrt. different goals. Chebotar et al. (2021) learn "Actionable Models" by choosing different states in trajectories replay. In Reinforcement Learning (RL), successor states provide a well founded framework for goal dependent value functions (Blier et al., 2021). Subgoals have also recently been integrated into the MDP-option framework (Lo et al., 2022). Similarly, subgoals play a major part in our work.
subgoals through hindsight
3 LEARNING NAVIGABILITY
We learn a representation useful for different visual navigation problems, and without loss of generality formalize the problem as PoinsGoal task: An agent receives RGB-D observations o, and Euclidean goal vector (GPS+ CPs+aospsss G, at each time step and must take actions a, which are typically discrete actions from given alphabet {FORWARD 25cm, TURN LEFT 10", TURN RIGHT 10 and STOP}. }. The agent sequentially builds a representation r, from the sequence of observations for hee and the previous action a; and a policy predicts distribution over actions,
[oo, Gr, r,,1, p(a:) = TT::),
where f in our case is a neural network with GRU memory (Cho et al., 2014), but which can also be modeled as self-attention over time as in (Chen al., 2021; Janner et al., 2021; Reed et al., 2022). We omit dependencies on parameters and gating mechanisms from our notations.
We do not focus on learning the main policy x, which can be trained with RL, imitation learning (II.) or other losses on a given downstream task. Instead, we address the problem of learning the representation r, through its usefulness: we opti- mize the amount of information carries about the navigability of (and towards) different parts the scene. When given to blind auxiliary agent, this information should allow the agent to navigate to any sufficiently close point, without requiring any
BR
Figure Our training data organized into "long episodes" from start S to target I navigated by the main agent æ and "short episodes" branching out at waypoints W; to subgoals gj-
3

---

Published as a conference paper at ICLR 2024
BC
a
2-2
a 4-1
a
Navigability
a
a
a
2+2,
a
2
a
a
a
aj
a
a
2,+2
π
1 2
1--1
Ft
Tí-2 2
Tt-1
9. Batch of different sub goals
Or-2 0:-1 O2
01-2 0:-1 Ot
sizzraa
Figure 3: Learning navigability: the difference between classical behavior cloning (Left), which directly learns the main target policy on downstream task, and leamnng navigability (Right), i.c. learning representation r. predicted by f through blind auxiliary policy P allowing to navigate by predicting sequences of actions for a batch of different subgoals for each time instant t. The representation is then used by the downstream policy æ.
visual observation during its own steps. We cast this as tailored variant of behavior cloning (BC) using privileged information from the simulator.
We organize the training data into episodes, for each of which we suppose the existence of optimal (shortest) paths, e.g. calculated in simulation from GT maps. We distinguish between long and short episodes, shown in Figure 2. During training. long episodes are followed by the (main) agent which integrates observations into representations r, as in eq. (I). At waypoints W; sampled regularly during the episode, representations r collected and sent to batch (of size B) of multiple blind
auxiliary agents, which branch out and are trained to navigate to a batch of subgoals {9و ij=1...B
The blind agent is governed by auxiliary policy P operating on its own recurrent GRU memory h,
hy.j = f(g,, 11ft Fe: ak-1,)); p( ( = p(hs,),
where the index j is over the subgoals of the batch, l: goes over the steps of the short episode, and the actions as.j are actions of the aux agent. The representation r collected at step the main policy remains constant over the steps of the auxiliary policy, with its GRU update function. This is illustrated in Figure 3 (f not shown, integrated into p).
We train the policy p and the function f predicting the representation r; jointly by BC minimizing the error between actions j and the GT actions āi, derived from shortest path calculations:
aarg minj,p Σ. Ex CCE( ag.y):
where LeE is the cross-entropy loss and the index & runs over all steps in the training set. Let us recall again a detail, which might slightly hidden the notation equation (4): while the loss runs over the steps k in short-trajectee trajectracct,,,ss these steps are attached to the steps & in long episodes through the visual representation He built by the encoder f. as in equation (3). The auxiliary policy p is pre-text task and not used after training. Navigation is performed by the main agent æ, which is finetuned on its own downstream objective.
Navigability vs. BC there is crucial difference to classical Behavior Cloning, which trains the main policy jointly with the representation I from expert trajectories mimicking or approximating the desired optimal policy (see Ramrakhya al. (2023) for comparisons), i.e.:
r f(o,, Gr, p(ar)= = 7 (Ix)
(5)
(f. arg min,,a EIED Cor(at,
(6)
where G2 is the (global) navigation goal and D the training data. In the case of navigation, these experts are often shortest path calculations or human controllers combined with goal selection through hindsight. It a well-known that BC is suboptimal for several reasons (Kumar et al., 2022). Amongst others, it depends on sufficient sampling of the state space in training trajectories, and it fails to adequately learn exploration in the case where single optimal solution is available to the agent due to partial observability. In contrast, our navigability loss trains the representation r, only, and can be combined with independently chosen downstream policies.
4

---

Published as a conference paper at ICLR 2024
Subgoal mining- For given compute budget, the question arises how many steps spent long vs. short episodes, as each step spent on a short episode is removing one visual observation from training p is blind. We sample waypoints each long episode and attach fixed number of subgoals to each waypoint sampled uniformly at given Fuclidean distance. Mining subgoal positions is key the success of the method: Sampled too close, they lack information. Outside of the observed are (and thus not represented in ry), the blind auxiliary agent would have to rely on regularities in environment layouts to navigate, and not on rç. We sample large initial number of subgoals and remove less informative ones, ie. those whose geodesic distance de to the waypoint is close to its Euclidean distance dæ, i.e. de <T. For these, following the compass vector would be a dø sufficient strategy, non-informative about the visual representation. Details are given in Section 5.
Implementation and downstream tasks Akin to training with PPO (Schulman et al., 2017), rollout buffer is collected with multiple parallel environment interactions, on which the Navigability loss trained. This facilitates optional batching of PPO with navigability, with both losses being separated over different environments see Section 5, and the appendix for implementation.
Training of the agent is done in two phases: first representation training, in which the main policy т, the representation F2 and the auxiliary agent are jointly trained minimizing Lae. eq. (4). This followed by fine-tuning on the downstream task with PPO. We also propose a combination of Navigability and BC losses using L = Naav + LBC. The advantages are two-fold: (i) training the main policy is not idle for environments selected for the Navigability loss, and (ii) visual observations gathered in environment steps spent in short episodes are not wasted, as they are used for training re through backpropagation through the main policy - see Section 5.
4 ON THE IMPORTANCE OF THE BLINDNESS PROPERTY
We argue that blindness of the auxiliary agent is an essential property, which we motivate by consid- ering the impact of the supervised learning objective terms of compression and vanishing gradients. Figure 4 shows main agent œ which, after having entered the scene and made a circular motion, has observed the central square-shaped obstacle and the presence of the door it came through. Our goal is to maximize the amount of information on the obstacles and navigable space extracted by the agent through its training objective. Without loss of generality out the representation estimated at t=t indicated in Figure 4. While this information is in principle present in the observation history {or Jestt there is no guarantee that will be kept in the representation r, at t=t,, as the amount of information storable the recurrent memory r, is much lower
T
single
of
Figure 4: Beh. cloning. than the information observed during the episode. Agent training leads to a learned compression mechanism, where the policy (expressed through equation (5)) compresses {or)tee, in two steps: (1) information from o, not useful at all is discarded by f before it is integrated into rt; (2) information from o, useful for a single step is integrated into rt, used by æ and then discarded by ʃ at the next update, i.e. it does not make it into r++1- Here we mean by "information content" the mutual information (MT) between rt and the observation history, i.e.
log p(Fe(Opast) p(r.)
[(r Opast) REprosase)
where Opest =
for Ir Dong et al. (2020) provide a detailed analysis of information retention and compression in RNNs in terms of MI and the information bottleneck criterion (Bialek et al., 2001).
The question therefore arises, whether the BC objective sufficient to retain information on the scene structure observed before t=t in F2 at t=t,. Without loss of generality, let single out the learning signal at t=t2, where f2 >> t1, as in Figure 4. We assume the agent predicts an action a,. which would lead to collision with the central obstacle, and receives supervision GT signal at
which avoids the collision:
Vaa. Minimizing L(ar, a;) requires learning to predict the correct action at given its "input" rt, and in this case this can happen in two different reasoning modes:
(r1) learning memory-less policy which avoids obstacles visible in its current observation, (r2) learning a policy which avoids obstacles it detects in its internal latent map, which was integrated over its history of observations.
5

---

Published as a conference paper at ICLR 2024
It is needless to say that (r2) is the desired behavior compatible with our goal stated above. However, minimizing the BC objective can be realized by both, (rl) and (r2), we argue that training will prioritize learning (r1) and neglect (12) for essentially two reasons: firstly, the compression mechanism favors (r1) which does not require holding it in an internal state for longer than one step. Secondly, multiple hops and requires backpropagation over multiple time instants, necessary for the integration of the observed sequence into usable latent map. The vanishing gradient problem will make learning (r2) harder than the short chain reasoning (rl).
reasoning (r2) happens
Let's now consider the navigability loss, illustrated in Figure 5. The main agent π integrates visual observations over its own trajectory up to waypoint t-t1. The aux agent navigates on the blue trajectory and we again consider the effect of the supervised learning signal 1=12- Minimizing L(a,, at) requires learning an agent which can predict the correct action a given its "input" re. but now, this can happen in one way only: since the agent p is blind, the BC objective cannot lead to reasoning (r1), i.e. memory-less, as it lacks the necessary visual observation to do so. To consistently predict the correct action a the representation ry collected at t=t, is necessary. i.e. (12). Making the aux agent blind has thus the double effect of resisting the compression mechanism in learning, and to force the learning signal through longer backpropagation chain, both of which help integrating relevant observations into the agent memory. Ribeiro et al. (2020) have recently shown that information retention and vanishing gradients, albeit different concepts, are related.
P
1/2
Figure 5: Navigability.
For these reasons, navigability is different from data augmentation (DA): the steps on short episodes improve the representation integrated over visual observations on long episodes, whereas classical DA would generate new samples and train them with the same loss. We see it as generating new learning signal for existing samples on long episodes using privileged information from the simulator.
An argument could be made that our stated objective, i.c. to force an agent to learn latent map, not necessary if optimizing BC does not naturally lead to it. As counter argument we claim that integrating visual information over time (r2) increases robustness compared to taking decisions from individual observations (r1), in particular in the presence of sim2real gaps. We believe that reactive reasoning (rl) will lead to more likely exploitation of spurious correlations in simulation then mapping (r2), and we will provide evidence for this claim in the sim2real experiments in Section 5.
Related work recent work (Wijmans et al., 2023) has covered experiments on representations learned by a blind agent. Compared to our work, Wijmans et al. (2023) present an interesting set of experiments on the reasoning of trained blind agent, but it does not propose no gradients flow from the probed agent to the blind one. In contrast, in our work the blind agent contributes to enrich the representation of a new navigation agent. Our situation corresponds to non-blind person which is blind-folded and needs to use the previously observed information from memory, with gradients flowing back from the blindfolded situation to the non-blind one.
method:
EXPERIMENTAL RESULTS
We train all agents in the Habitat simulator (Savva al., 2019) and the Gibson datasel (Xia et al., 2018). We follow the standard train/val split over scenes, i.c. 72 training. 14 for validation, 14 for testing, with approximately 68k, 75k and 71k episodes per scene, respectively.
Subgoals- - All standard episodes are used as long episodes during training, short episodes have been sampled additionally from the training scenes. To be comparable, evaluations are performed on the standard (long) episodes only. To produce short episodes, we sample waypoints every 3m on each long episode and attach 20 subgoals to each waypoint at a Euclidean distance E [3, 5)m. The threshold for removing uninformative subgoals is set to 7-1.5m. This leads to the creation of ~ 36M short training episodes - no short episode is used for validation or testing.
Sim2Real evaluating sim2real transfer is inherently difficult, as it would optimally require to evaluate all agent variants and ablations on a real physical robot and on a high number of episodes.
We opted for a three-way strategy: (i) Sim2Real experiments evaluate the model and policy и trained in simulation
real physical robot. It is the only form of evaluation which correctly estimates navigation performance in a real world scenario, but for practical reasons we limit it to 11 episodes in large (unseen) office environment shown in Fig. and Table 1; (ii) Evaluation in

---

Published as a conference paper at ICLR 2024
Simulation allows large-scale evaluation large number of unseen environments and episodes; (iii) Sim2NoisySim allows similar large-scale evaluation and approximates the transfer gap through artificial noise on simulation parameters. We added noise of different types, similar to Anderson et al. (2018), but with slightly different settings: Gaussian Noise of intensity 0.1 on RGB, Redwood noise with D=30 on depth, and actuator noise with intensity 0.5. Details are given in the appendix.
Metries - we report Success, which is the number of correctly terminated episodes, and SPL (Anderson et al., 2018), Success weighted by the optimality of the navigated path, SPL = ,े4 Em' S. (m)x((..,))) where 4 is the length of the agent's 's path in episode and f is the length of the GT path. For robot experiments we also use Soft-SPI. (Anderson et al., 2018), which extends SPL by modifying the definition of S,; in failed episodes it is weighted by the distance achieved towards the goal instead of being zero
Table 1: Sim2Real transfer - Avg. per- formance over 11 episodes in a real envi- ronment (Fig. 1 and map below) using the map+plan baseline and three agents trained with PPO, BC and ours, corresponding to variants (a), (c), (e) of Table 2.
we propose a two-phase strategy with a fixed compute budget of 100M env. steps:
Training setup- -
Phase 1: Pre-training takes 50M steps. We combine
and test 4 strategies: standard PPO, standard BC, Nav- igability loss and a reconstruction loss (details below), for total of 8 variants (Table 2).
Method
Success
SPL
sSPL
Phase 2: Fine-tuning is the same for all methods, done with PPO (Schulman et al., 2017) for 50M steps on the downstream PointGoal task. We reinitialize the main policy π to evaluate the quality of the learned represen- tations, the aux agent is not used. We use common
(-) Map+Plan
36.4
29.6
32.7
(a) PPO
45.5
14.7
20.0
(c) BCQ
72.7
36.3
36.3
(e) Navig.BC
81.8
45.6
44.0
reward definition (Chattopadhyay et al., 2021) as rt = R Lussess - Geo X, where R-2.5,
AGe0 is the gain in geodesic distance to the goal, and a slack cost 0.0.1 encourages efficiency,
The best checkpoint is chosen on the validation set and all results are reported on the test set.
Robot experiments are performed with Locobot robot on a large office building with thick carpets, uneven floor and large windows (Fig. and top-down map in Table 1). Results for 11 episodes, shown on the map (avg. GT geodesic length 8.9m), are reported in Table for three end- to-end trained agents (PPO, BC and our proposed agent) and for a map+plan baseline (Gupta et al., 2017) used in (Chaplot et al., 2020b;a), for total of 44 robot experiments. A detailed description of the agents can be found further below. Overall, the differences in performance are significant, with clear advantage of the proposed Navigability representation in terms of all 3 metrics, providing evidence for its superior robustness (Success) and efficiency (SPL, sSPL). Qualitatively, the three agents show significantly different behavior these real robot experiments. Our Navigability+BC variant is more efficient in reaching the goals and goes there more directly. The PPO agent shows zig-zag motion (increasing SPL and sSPL.), and, in particular, often requires turning, whose objective we conjecture is to orient and localize itself better. The BC variant struggled less with zig-zag motion but created significantly longer and more inefficient trajectories than the proposed Navigability agent.
Impact of navigability - is studied in Table 2, which compares different choices of pre-training in phase 1. For each experiment, agents were trained on 12 parallel environments. These environments where either fully used for training with BC (with choice of the proposed navigability loss or classical BC of the main policy, or both), or fully used for classical RL with PPO training with PointGoal reward, or mixed (6 for BC and 6 for PPO), indicated in columns Nr. Envs PPO and Nr. BC Envs, respectively. Agents trained on the navigability loss navigated fewer steps over the long episodes and saw fewer visual observations, only 5% 2.5M. We see that BC (b) outperforms PPO (a), and navigability loss alone (d) is competitive and comparable, outperforming these basclines when transferred noisy environments. Mixing training with PPO on the PointGoal downstream task in half of the training environments, as done in variants (f) and (g), does not provide gains.
Optimizing usage of training data As the number of environment steps is constant over all agent variants evaluated in Table 2, the advantage of the Navigability loss in terms of transferability is slightly counter-balancee by disadvantage in data usage: adding the short episodes to the training in
7

---

Published as a conference paper at ICLR 2024
Table 2: Influence of the navigability loss and short episodes: compare PPO (Schulman et al., 2017). BC=Behavior cloning and our Navigability, which constitute Phase 1 50M env steps. environments are trained in parallel, distributed over the different learning signals. In Phase 2. all of these methods are finetuned with PPO only for an additional 50M env steps. The best validation checkpoints of phase 2 are chosen and evaluated the test set. "Do S short episodes are used by the agent. [D=lng episodes, ($=short episodes.
The agent trained with navigability only has seen only 5% of the visual observations (=2.5M).
Nr. Env Nr. visual
Nr. Envs
Nr. BC Envs
Eval Sim
Eval NoisySim
Method
steps
obs seen
Do
S
x:PPO
m: a: p:
Success SPL
Success
SPL
(a) Pure PPO
50M
50M
12
89.6
71.7
74.6
55.8
(b) Pure BCD
50M
50M
_
12
92.0
79.6
76.0
61.7
(c) BCD ( (data augm.)
50M
50M
7
12
12
94.2
80.1
89.6
74.0
(d) Navig.
50M
2.5M
12
92.9
77.3
86.8
68.8
(e) Navig. + BCQ
50M
25
1.25M
V
12
12
95.5
80.3
90.9
73.3
PPO + Navig.
50M
25+
91.5
72.6
85.2
63.8
g PPO + Navig. + BCS
50M
50M
6
90.3
73.7
83.9
66.0
(h) AUX reconst. + BCO
50M
50M
12
94.9
80.4
76.7
61.2
Table 3: Impact of the hidden state of the main policy Table 4: Communicating with the Mole: Impact of transitions between long and short episodes: (c.1) the choice of connection between representation and set to zero (short episodes are data augme.): ): (c.2) set blind policy p for agent (e) of Table 2. (e.1) r. is fed as last waypoint (clear separation of short and long ep.); "observed" input to p at each step: (e.2) as initialization (c.3) always continue (maximize episode length but of p's own hidden state; (e.3) as previous, but 128 introduce sparse teleportations).
additional dimensions are added to the state of p.
Method
Eval Sim Eval NoisySim
Method
Eval Sim
Eval Noi
Suce SPL Suce
SPL
h
Success SPL
Success
SPL
(c.1) Set to zero
90.3 74.5 85.1
64.8
(e.1) As observation
512
92.8
77.4
90.9
73.3
(c.2) Set to last waypoint
88.6 74.2 81.4
63.0
(e.2) Copy
512
95.5
80.3
80.7
62.5
(c.3) Always continue
94.2 80.1 89.6
74.0
(e.3) Copy+extend
640
90.4
76.9 83.3
66.6
variant (d) has two effects: (i) a decrease in the overall length of episodes and therefore of observed information available agents; (ii) short episodes are only processed by the blind agent, and this decreases the amount of visual observations available to ~5%. Combining Navigability with classical BC in agent (e) in Table provides the best performance by large margin. This corroborates the intuition expressed in Section 3 of the better data exploitation of the hybrid variant.
Is navigability reduced to data augmentation? performance are obtained by the navigability loss, or by the contribution of additional training data in the form of short episodes, and we again recall, that the number of environment steps is constant over all experiments. Agent (c) performs classical BC on the same data, i.e. long and short episodes. It is outperformed by Navigability combined with BC, in particular when subject to the sim2noisy-sim gap, which confirms our intuition of the better transferability of the Navigability representation.
a control experiment tests whether the gains in
Continuity of the hidden state- - The main agent æ maintains a hidden state rt, updated from its previous hidden state r and the current observation o,. If this representation is a latent map. then, similar to a classical SLAM algorithm, the state update needs to take into account the agent dynamics to perform a prediction step combined with the integration of the observation. When the agent is trained through BC of the main agent on long and short episodes, as for variants (c) and (e), the main agent follows a given long episode and is interrupted by short episodes. How should r, be updated when the agent "teleports" from the terminal state of short episode back to the waypoint on the long trajectory? In Table 3 we explore several variants: setting r, to zero waypoints is clean solution but decreases the effective length of the history of observations seen by the agent. Saving Ft at waypoints and restoring it after each short episode ensures continuity and keeps the amount of observed scene intact. We lastly explore variant where the hidden state always continues, maximizing observed information, but leading to discontinuities as the agent teleported to new locations. Interestingly, this variant performs best, which indicates that data is extremely important. Note that during evaluation, only long episodes are used and no discontinuities are encountered.
Communication with the Mole in Table 4 we explore different ways of communicating the representation Ie from the main to the aux policy during Phase 1. In variant (e. receives ra as input at each step; in variants (c.2) and (e.3), the hidden GRU state h of p is initialized as r at the beginning of each short episode, and no observation (other than the subgoal g. is passed to it. Variant (e.2) is the best performing in simulation, and conjecture that this indicates likely
8

---

Published as a conference paper at ICLR 2024
Sim
NoisySim
Method
Reconstr
2D Nav.
Reconstr.
2D Nav.
loU
Succ. Sym-SPL
IoU
Succ. Sym-SPL
PPO
31,0
37.9
10.1
15.8
18.4
6.1
(c.1) BC
25.5
45.2
11.0
11.1
16.2
4.6
GT
(a) PPO
GT
(c.1) BC
(c.3) BCOS
11.9
13.2
3.5
1.1
0.6
0.3
(e) ND
19.8
66.5
14.7
18.0
64.8
13.2
Viewpoints figures the left are comparable different agents performm different trajectories. The agent is the blue dot in the center and trujectories out of 10 used for Sym- are plotted. Goul points are green when reachable (always GT) and red otherwise.
GT
(c.3) BC GT
(e) Ours
Figure 6: Probing reconstruction: we estimate how much information on reconstruction is contained in the learned representations. PPO (a) appears to capture only parts of the map, leading to failures in this navigation task. Reconstruction by (c.1) exhibits similar characteristics. The (c.3) variant performs poorly tasks, and the reconstructed map confirms that. Our approach, (e), estimates less accurately than PPO and (c.1) the shape the navigable space, but still appears to capture important geometric aspects of the environment.
these prohing
ego-centric nature of the latent visual representation r,. Providing as initialization allows it to evolve and be updated by the blind agent during navigation. This is further corroborated by the drop in performance of these variants in NoisySim, as the update of an ego-centry representation requires odometry, disturbed by noise in the evaluation, and unobserved during training. Adding 128 dimensions to the blind agent hidden state h in variant (e.3) does not seem to have an impact.
Probing the representations - of blind agent has been previously proposed by Wijmans al. (2023). We extend the procedure: for each agent, a dataset of pairs {[r,, M;) is generated by sampling rollouts. Here, rę is the hidden GRU state and M; is an ego-centric 2D metric GT map of size 95x95 calculated from the simulator using only information observed by the agent. A probe Φ trained on training scenes to predict M, = <(r:) minimizing cross- enssopy, and tested on val scenes. Results and example reconstructions on test scenes are shown in Fig. 6. We report reconstruction performance measured by IoU on unseen test scenes for variants (a), (c) and (e), with (c) declined into sub- variants (c.1) and (c.3) from Table 3. PPO outperforms the other variants pure reconstruction in noiseless setting, but this is not necessarily the goal of an actionable representation. We propose new goal-oriented metric directly measuring the usefulness of the representation in terms of navigation. For each pair (M;, M.) of predicted and GT maps, we sample N=10 reachable points (pn. I==1..N the GT map M We compute shortest paths from the agent position pa: one on the GT map M,, f and one on the predicted map M,, C, We introduce Symmetric-SPL
as Sym-SPL - Eb Σn Si, min L
where, similar to SPL, Si,n denotes success of the Ei.m
episode, but on the predicted map M, and towards Pa- Results in Figure show that representations learned with Navigability lead to better navigation performance, particular in NoisySim. While this study is speculative and it is hard do draw conclusive insights from it, these observations seem to corroborate the improved transferability of representations learned with Navigability.
Further comparison with reconstruction - we claim that unnecessar-
Table 5: Map+plan baseline.
ily accurate reconstruction sub-optimal in presence of high sim2real gap, and, additional to already discussed robot experiments (Table 1),
Method Noise/Suce. SPL
Class. no
95.7 89.9
we compare in simulation with method supervising reconstruction. The loss is identical to the probing loss described above, but used to train the representation during Phase 1, combined with BCC Corresponding method (h) in Table 2 compares unfavorably with our method (e), in particular in noisy settings, In Table 5 we also compare with a classical map+plan baseline of Gupta et al. (2017) and show that under noisy conditions our approach outperforms the classical planner.
Ours (c) no
95.5 80.3
Class.
yes 27.9 16.3
Ours (e)
yes
90.9 73.3
CONCLUSION
Inspired by Cognitive Maps in biological agents, we have introduced a new technique, which learns a latent representations from interactions of blind agent with the environment. We position it between explicit reconstruction, arguably not the desired when a high sim2real gap is present, and pure end-to-end training downstream task, which is widely argued to provide weak learning signal. In experiments on sim2real and sim2noisy-sin evaluations, we have shown that our learned representation is particularly robust to domain shifts.
9

---

Published as a conference paper at ICLR 2024
REFERENCES
Peter Anderson, Angel X. Chang, Devendra Singh Chaplot, Alexey Dosovitskiy, Saurabh Gupta, Vladlen Koltun, Jana Kosecka, Jitendra Malik, Roozbeh Mottaghi, Manolis Savva, and Amir Roshan Zamir. On evaluation of embodied navigation agents. arXiv preprint, 2018. 3,
Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, Pieter Abbeel, and Wojeiech Zaremba. Hindsight Experience Replay. In NeurIPS, 2017.
Andrea Banino, Caswell Barry, Benigno Uria, Charles Blundell, Timothy Lillicrap, Piotr Mirowski, Alexander Pritzel, Martin J Chadwick, Thomas Degris, Joseph Modayil, Greg Wayne, Hubert Soyer, Fabio Viola, Brian Zhang, Ross Goroshin, Neil Rabinowitz, Razvan Pascanu, Charlie Beattie, Stig Petersen, Amir Sadik, Stephen Gaffney, Helen King, Koray Kavukcuoglu, Demis Hassabis, Raia Hadsell, and Dharshan Kumaran. Vector-based navigation using grid-like representations in artificial agents. Nature, 557, 2018.
Edward Beeching, Jilles Dibangoye, Olivier Simonin, and Christian Wolf. Learning reason on uncertain topological maps. In ECCV, 2020a.
Edward Beeching, Christian Wolf, Jilles Dibangoye, and Olivier Simonin. EgoMap: Projective mapping and structured egocentrie memory for Deep RL. In ECML-PKDD, 2020b. 2
William Bialek, Ilya Nemenman, and Naftali Tishby. Predictability, Complexity, and Learning.
Neural Computation, 13(1 1):2499-2463, November 2001. 5
Léonard Blier, Corentin Tallec, and Yann Ollivier. Learning Successor States and Goal-Deeendent
Values: A Mathematical Viewpoint. In MwXiv:2101.07123 2021. 3
Hugh Carlton Blodgett. The effect of the introduction of reward upon the maze performance of rats. University of California Publications in Psychology, 4:133-134, 1929. 1,
Guillaume Bresson, Zayed Alsayed, Li Yu, and Sébastien Glaser. Simultaneous localization and
mapping: A survey of current trends in autonomous driving. IEEE Transactions Intelligent Vehicles, 2017. 1,2
Wolfram Burgard, Armin B Cremers, Dieter Fox, Dirk Hähnel, Gerhard Lakemeyer, Dirk Schulz, Walter Steiner, and Sebastian Thrun. The interactive museum tour-guide robot. In Aaai/iaai, pp. 11-18, 1998.
Devendra Singh Chaplot, Dhiraj Gandhi, Abhinav Gupta, and Ruslan Salakhutdinov. Object goal navigation using goal-oriented semantic exploration. In NeurIPS, 2020a. 2,
Devendra Singh Chaplot, Dhiraj Gandhi, Saurabh Gupta, Abhinav Gupta, and Ruslan Salakhutdinov. Learning explore using active neural slam. In ICLR, 2020b. 2.7
Prithvijit Chattopadhyay, Judy Hoffman, Roozbeh Mottaghi, and Aniruddha Kembhavi. Robustnav:
Towards benchmarking robustness in embodied navigation. CoRR, 2106.04531, 2021. 3, 7, 15
Yevgen Chebotar, Karol Hausman, Yao Lu, Ted Xiao, Dmitry Kalashnikov, Jake Varley, Alex Irpan, Benjamin Eysenbach, Ryan Julian, Chelsea Finn, and Sergey Levine. Actionable Models: Unsupervised Offline Reinforcement Learning of Robotic Skills. In ICML, 2021. 3
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lec, Aditya Grover, Michael Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. In NeurlPS, 2021. 3
Shizhe Chen, Pierre-Louis Guhur, Makarand Tapaswi, Cordelia Schmid, and Ivan Laptev. Think Global, Act Local: Dual-scale Graph Transformer for Vision-and- -nt--angua-LLag Navigation.
arXiv:2202. / 1742, 2022. 2
Kyunghyun Cho, Bart van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau. Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using RNN picoder-decoder for statistical machine translation. In Conference on Empirical Methods in Natural Language Processing, 2014.7
10

---

Published as a conference paper at ICLR 2024
Christopher J. Cueva and Xue-Xin Xin Wei. Emergence of grid-like representations by training recurrent neural networks to perform spatial localization. In ICLR, 2018. 3
Nitish Dashora, Daniel Shin, Dhruv Shah, Henry Leopold, David Fan, Ali
Nicholas Rhinehart, and Sergey Levine. Hybrid imitative planning with geometric and predictive costs in off-road environments. In NeurlPS Workshop on Deep-RL, 2021. 2
Sombit Dey, Assem Sadek, Gianluca Monaci, Boris Chidlovskii, and Christian Wolf. Learning whom to trust navigation: dynamically switching between classical and neural planning. IROS, 2023.
3
Yiming Ding, Carlos Florensa, Pieter Abbeel, and Mariano Phielipp. Goal-conditioned imitation learning. NeurIPS, 2019.
Zhe Dong, Deniz Oktay, Ben Poole, and Alexander A. Alemi. On Predictive Information in RNNs. 2020. 5
Heming Du, Xin Yu, and Liang Zheng. VTNet: Visual Transformer Network for Object Goal
Navigation. In ICLR, 2021. 2
Benjamin Eysenbach, Shreyas Chaudhari, Swapnil Asawa, Sergey Levine, and Ruslan Salakhutdinov. Off-dynamics reinforcement learning: Training for transfer with domain classifiers. In Intern. Conf. Learning Representations, ICLR, 2021. 3
Kuan Fang, Alexander Toshev, Li Fei-Fei, and Silvio Savarese. Scene memory transformer for embodied agents in long-horizon tasks. In CVPR, 2019.
Dieter Fox, Wolfram Burgard, and Sebastian Thrun. The dynamic window approach to collision avoidance. IEEE Roboties & Automation Magazine, 4(1 4(1):23-33, 1997.
Dibya Ghosh, Abhishek Gupta, and Sergey Levine. Learning actionable representations with goal- conditioned policies. In ICLR, 2019.
Saurabh Gupta, James Davidson, Sergey Levine, Rahul Sukthankar, and Jitendra Malik. Cognitive mapping and planning for visual nav. In CVPR, 2017. 2, 7,9
Torkel Hafting, Marianne Fyhn, Sturla Molden, May-Britt Moser, and Edvard Moser. Microstructure of spatial map in the entorhinal cortex. Nature, 436:801-6, 6-, 09 2005. doi: 10. 10.103///auue00321..
João F. Henriques and Andrea Vedaldi. Mapnet: An allocentric spatial memory for mapping environments. In CVPR, 2018.
Scbastian Höfer, Kostas E. Bekris, Ankur Handa, Juan Camilo Gamboa Higuera. Florian Golemo, Melissa Mozifian, Christopher G. Atkeson, Dieter Fox, Ken Goldberg, John Leonard, C. Karen Liu, Jan Peters, Shuran Song, Peter Welinder, and Martha White. Perspectives on sim2real transfer for robotics: A summary of the R: SS 2020 workshop. arXiv:2012.03880 030800 2020. 3
Max Jaderberg, Volodymyr Mnih, Wojeiech Marian Czarnecki, Tom Schaul, Joel Z. Leibo, David Silver, and Koray Kavukcuoglu. Reinforcement learning with unsupervised auxiliary tasks. In ICLR, 2017.
Michael Janner, Qiyang Li, and Sergey Levine. Offline reinforcement learning as big sequence modeling problem. In NeurIPS, 2021.
Abhishek Kadian, Joanne Truong, Aaron Gokaslan, Alexander Clegg, Erik Wijmans, Stefan Lee, Manolis Savva, Sonia Chernova, and Dhruv Batra. Are making real progress in simulated environments? measuring the sim2real gap in embodied visual navigation. In arXiv: 19/2.0631, 2019. 18
Abhishek Kadian, Joanne Truong, Aaron Gokaslan, Alexander Clegg, Erik Wijmans, Stefan Lee, Manolis Savva, Sonia Chernova, and Dhruv Batra. Sim2real predictivity: Does evaluation in simulation predict real-world performance? IEEE Robotics Autom. Lett., 5(4)66670-66 -6777, 2020. 3,
15
11

---

Published as a conference paper at ICLR 2024
Kurt Konolige. A gradient method for realtime robot control. In IROS, 2000.
Aviral Kumar, Joey Hong, Anikait Singh, and Sergey Levine. When Should We Prefer Offline Reinforcement Learning Over Behavioral Cloning? In ICLR, 2022. 4
Mathieu Labbé and François Michaud. RTAB-Map open-source lidar and visual simultaneous localization and mapping library for large-scale and long-term online operation. Journal of Field Robotics, 36(2):416 446, 2019.
Rosanne Liu, Joel Lehman, Piero Molino, Felipe Petroski Such, Eric Frank, Alex Sergeev, and Jason Yosinski. An intriguing failing of convolutional neural networks and the CoordConv solution. In NeurIPS, 2018. 15
Iker Lluvia, Elena Lazkano, and Ander Ansuategi. Active Mapping and Robot Exploration: Survey. Sensors, 21(7):2445, 2021. 1,2
Chunlok Lo, Gabor Mihucz, Adam White, Farzane Aminmansour, and Martha White. Goal-space planning with subgoal models. arXiv:2206.029000 029222 2022. 3
Ily Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2018. 16
Vladimir Lumelsky and Alexander A. Stepanov. Path-planning strategies for point mobile automaton moving amidst unknown obstacles of arbitrary shape. Algorithmica, 2(1):403-430, 1987. 3,
Steve Macenski, Francisco Martín, Ruffin White, and Jonatan Ginés Clavero. The marathon 2: A navigation system. In IROS, 2020. 2
Eitan Marder-Eppstein, Eric Berger, Tully Foote, Brian Gerkey, and Kurt Konolige. The office marathon: Robust navigation in indoor office environment. In ICRA, 2010. 2
Pierre Marza, L. Matignon, Simonin, and C. Wolf. Teaching Agents how to Map: Spatial Reasoning for Multi-Object Navigation. In IROS, 2022. 2
Pierre Marza, Laetitia Matignon, Olivier Simonin, and Christian Wolf. Multi-Object Navigation with dynamically learned neural implicit representations. In ICCV, 2023.
Emil W. Menzel. Chimpanzee Spatial Memory Organization. Science, 182:943-945, 1973. 1,3
Piotr Mirowski, Razvan Pascanu, Fabio Viola, Hubert Soyer, Andy Ballard, Andrea Banino, Misha Denil, Ross Goroshin, Laurent Sifre, Koray Kavukcuoglu, Dharshan Kumaran, and Raia Hadsell. Learning navigate in complex environments. In ICLR, 2017.
Emilio Parisotto and Ruslan Salakhutdinov. Neural map: Structured memory for deep reinforcement
learning. In /CLR, 2018.
Ruslan Partsey and Oleksandr Maksymets. Robust visual odometry for realistic pointgoal navigation, 2021. 15
Ruslan Partsey, Erik Wijmans, Naoki Yokoyama, Oles Dobosevych, Dhruv Batra, and Oleksandr Maksymets. mapping necessary for realistic pointgoal navigation? In CVPR, 2022.
Xue Bin Peng, Marcin Andrychowicz, Wojciech Zaremba, and Pieter Abbeel. Sim-to-real transfer of robotic control with dynamics randomization. In ICRA, 2018. 3
Jonah Philion and Sanja Fidler. Lift, splat, shoot: Encoding images from arbitrary camera rigs by
implicitly unprojecting to 3D. In ECCV. 2020. 3
Jonah Philion, Amlan Kar, and Sanja Fidler. Learning evaluate perception models using planner- centric metrics. In CVPR, 2020. 3
Chetan Prakash, Kyle D. Stephens. Donald D. Hoffman. Manish Singh, and Chris Fields. Fitness Beats Truth in the Evolution Perception. Acta Biotheoretica, 69(3):319-33 19-111, 2021. URL https: //t://inn spr.k http://.....oo...o.o////////a.a0
12

---

Published as a conference paper at ICLR 2024
Santhosh Kumar Ramakrishnan, Devendra Singh Chaplot, Ziad Al-Halah, Jitendra Malik, and Kristen Grauman. PONI: Potential functions for objectgoal navigation with interaction learning. In CVPR, 2022. 2
Ram Ramrakhya, Dhruv Batra, Erik Wijmans, and Abhishek Das. PIRLNav: Pretraining with Imitation and RL Finetuning for ObjectNav. In CVPR, 2023. 4
Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gomez Colmenarejo, Alexander Novikov, Gabriel Barth-Maron, Mai Gimenez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, Tom Eccles, Jake Bruce, Ali Razavi, Ashley Edwards, Nicolas Heess, Yutian Chen, Raia Hadsell, Oriol Vinyals, Mahyar Bordbar, and Nando de Freitas. A Generalist Agent. arXiv:2205.005 0677,, 2022. 2.
António H Ribeiro, Koen Tiels, Luis A Aguirre, and Thomas Schön. Beyond exploding and vanishing gradients: analysing RNN training using attractors and smoothness. In AISTATS, 2020. 6
Marco Rosano, Antonino Furnari, Luigi Gulino, and Giovanni Maria Farinella. On embodied visual navigation in real environments through habitat. In 2020 25th International Conference on Pattern
Recognition (ICPR), pp. 9740-9747. IEEE, 2021. 15
Christoph Rösmann, Frank Hoffmann, and Torsten Bertram. Timed-elastic-bands for time-optimal point-to-point nonlinear model predictive control. In European Control Conference (ECC), 2015.
Assem Sadek, Guillaume Bono, Boris Chidlovskii, and Christian Wolf. An in-depth experimental study of sensor usage and visual reasoning of robots navigating in real environments. ICRA, 2022. 1, 3, 15
Manolis Savva, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, Vladlen Koltun, Jitendra Malik, Devi Parikh, and Dhruv Batra. Habitat: A platform for embodied ai research. In ICCV, 2019. 6
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms.
2017. 5, 7,8
James A Sethian. fast marching level set method for monotonically advancing fronts. PNAS, 93(4): 1591-1595, 1996.
Dhruv Shah and Sergey Levine. ViKiNG: Vision-Based Kilometer-Scale Navigation with Geographic Hints. In RSS, 2022. 2
Dhruv Shah, Benjamin Eysenbach, Nicholas Rhinehart, and Sergey Levine Rapid Exploration for Open-World World Navigation with Latent Goal Models. In CoRL, 2021.
Open-
Aviv Tamar, Yi Wu, Garrett Thomas, Sergey Levine, and Pieter Abbeel. Value Iteration Networks. In NeurlPS, 2016. 3
Jie Tan, Tingnan Zhang, Erwin Coumans, Atil Iscen, Yunfei Bai, Danijar Hafner, Steven Bohez, and Vincent Vanhoucke. Sim-to-real: Learning agile locomotion for quadruped robots. In RSS, 2018.
Alex Teichman, Stephen Miller, and Sebastian Thrun. Unsupervised intrinsic calibration of depth sensors via SLAM. In RSS, 2013. 18
Sebastian Thrun, Wolfram Burgard, and Dieter Fox. Probabilistic robotics, 2005. 1,2
Edward C. Tolman. Cognitive maps in rats and men. The Psychological Review, 55(4), 1948. 1,3
Joanne Truong, Sonia Chernova, and Dhruv Batra. Bi-directional Domain Adaptation for Sim2Real Transfer Embodied Navigation Agents. IEEE Robotics and Automation Letters, 6(2), 2021.
Joanne Truong, Max Rudolph, Naoki Yokoyama, Sonia Chernova, Dhruy Batra, and Akshara Rai. Rethinking Sim2Real: Lower Fidelity Simulation Leads to Higher Sim2Real Transfer in Navigation. In CoRL, 2022.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones. Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurlPS, 2017.
13

---

Published as a conference paper at ICLR 2024
Kasun Weerakoon, Adarsh Jagan Sathyamorrthy, and Dinesh Manocha. Sim-to-real strategy for spatially aware robot navigation in uneven outdoor environments. In ICRA 2022 Workshop on Releasing Robots into the Wild:, 2022. 2
Erik Wijmans. Emergence Intelligent Navigation Behavior in Embodied Agents Massive-Scale Simulation. PhD thesis, Georgia Institute of Technology. 2022. 1, 15
Erik Wijmans, Abhishek Kadian, Ari Morcos, Stefan Lee, Irfan Essa, Devi Parikh, Manolis Savva, and Dhruv Batra. Dd-ppo: Learning near-perfect pointgoal navigators from 2.5 billion frames. In ICLR, 2019. 1, 15
Erik Wijmans, Manolis Savva, Irfan Essa, Stefan Lee, Ari S. Morcos, and Dhruv Batra. Emergence of Maps in the Memories of Blind Navigation Agents. In ICLR, 2023. 1,3,6,9 9, 15
Fei Xia, Amir R Zamir. Zhiyang He, Alexander Sax, Jitendra Malik, and Silvio Savarese. Gibson env: Real-world perception for embodied agents. In CVPR, 2018.
Ryo Yonetani, Tatsunori Taniai, Mohammadanin Barekatain, Mai Nishimura, and Asako Kanezaki. Path planning using neural a* search. In ICMI., 2021. 2
Wenyuan Zeng, Wenjie Luo, Simon Suo, Abbas Sadat, Bin Yang. Sergio Casas, and Raquel Urtasun. End-To-End Interpretable Neural Motion Planner. CVPR, 2021.
Fangyi Zhang, Jürgen Leitner, Zongyuan Ge, Michael Milford, and Peter Corke. Adversarial discriminative sim-to-real transfer of visuo-motor policies. Int. J. Robotics Res., 38(10-11), 2019.
Xinge Zhu, Jiangmiao Pang, Ceyuan Yang, Jianping Shi, and Dahua Lin. Adapting object detectors via selective cross-domain alignment. CVPR, 2019.
14

---

Published as a conference paper at ICLR 2024
Method
Steps Setup
Eval Sim SHocess SPI
Eval NoisySun
Evaal Real
Suceess
SPI.
Saccess
SPI
DDPOT (Wimaans 2019)
2.50
Giboa--+,, SE-RCINGIATIO ESTM
NIA
N/A
KIA
N/A
2.56
Gibace-2,, RosNec50 LSIM-112
94.0
N/A
NA
NIA
Emnh VaNNaaaaaaaannnnooo al, 2021)
2,5G42.4M
GiboonMPSSDerell obs., esal custom acene, RGR
99.1
91.3
N/A
N/A
97.
800
only
TuOdooNiw(PPmsee & Makaymets, 2021)
2M
Gibson (Tlah Challenge 2021)
36.0
66.0
N/A
NIA
NIA
NIA
Robust.Nar Nav (Chantoadhyya et al., 20233
75M++166
Exhausiive evaluation in sum2noisy-sim sim
Check 86.4
puper for broad study W. muttiplc params
Sadek al. (Sadek et nl, 3022)
100M
Real robot, 20 eva spiondes only, of custem envi
71.1
N/A
N/A
75.0
$3.9
Pred (Kadion al,, 2020)
500M
Triining sliding no train
36.0
N/A
NIA
61.0
500M
Truningg sliding, Training nose
N/A
NA
NIA
51.0
Ours
100M
Hybnd NavigabilityBCC followed by PPO, no train
95.5
80.3
90.9
73.3
818
45.6
noise
Table 6: Comparison with SOTA
These numbers are comparable as clear experimental protocol exists for sim2real or sim2noisy-sim experiments. The reported experiments feature extreme variations in setups, training regimes, data, number of test episodes. Several papers report large number of experiments, we reproduce only small number of variants here. Agent Ours is agent (c) in Table 2.
A APPENDIX
A.1 COMPARISON WITH SOTA
Comparisons are difficult, as no clear evaluation protocol exists, in particular when transfer and sim2real performance targeted. In Table 6 we attempt to provide a non-exhaustive review of PointGoal performance targeting transfer situations particularly. However, we need to point out that the performance numbers are not comparable, as there are very large variations in setups, training regimes, data, number of test episodes etc.
A.2 ARCHITECTURE OF THE AGENTS
An architecture diagram of the full agent is given in Figure 7. The main agent i uses a standard architecture from the literature, eg. in (Wijmans et al., 2019) and large body of follow-up work. A hidth 4-channels ResNet18 encodes the RGB-D frames into a 512D vector. A linear layer encodes the pointgoal information with integrated GPS and compass layer (a trained LUT) encodes the previous action into a 32D vector. These 3 vectors are then concatenated to form the input of GRU, which updates the 512D vector representing the internal state of the agent after each new observation. The internal state is finally fed to 2 linear layers to compute the argprobbabilities of each action and an estimate of the current state value, respectively.
64D vector. An embedding
When a waypoint is reached along the main path, the main agent stores its internal state (see Section A.5 for implementation details of the training algorithm). The auxiliary agent p (a.k.a. the "mole") starts exploring subgoals from the waypoint. Its inputs are the internal state of the main agent when it reached the waypoint, a new "sub-pointgoal" vector pointing towards the currently pursued subgoal, the previously taken action, and its own auxiliary internal state. The sub-poontggoal" and the previous action are encoded through similar linear and embedding layers as the main agent, respectively. also has its own GRU, followed by a linear layer producing the action log- bog-probabilities.
As described in the main text and ablated in Table 4, we tested a few different ways to connect the internal state of the main agent to the auxiliary one: 1. Concatenate it with the other features before feeding it as an input to the auxiliary GRU; 2. Use it to initialize the auxiliary internal state; 3. Use to initialize part of the auxiliary internal state while keeping some room for values dedicated to the subgoal exploration task.
A.3 PROBING EXPERIMENTS
A.3.1 ARCHITECTURE OF THE PROBING AGENT
The probing network's architecture is inspired by the approach proposed in (Wijmans, 2022; Wijmans et al., 2023), It processes the 512D vector r,, representing the GRU memory after pre-training (Phase in Section 5) with a 2-layer MLP with 256 hidden dimensions to produce an output vector of dimension 1600. This vector is reshaped into 3D tensor of size [64, 5, 5] and processed by a Coordinate Convolution (CoordCon)) layer (Liu et al., 2018), followed by four CoordConv- CoordUpConw (Coordinate Up-Convolution blocks. Each such block is composed of:
15

---

Published as a conference paper at ICLR 2024
Pry.Main Hid.
GRU
Main Hid.
Actor
Log- Log-prob.
(actions twds main goal)
RGB-D
ResNet18
Pt.Goal
Linear
Prv.Act.
Embedding
Aux.Emb.
Sub-Pt.Goal
Aux.Lin.
cat
Prv.Aux.Hid.
Aux. GRU
Aux.Hid.
Aux.Actor
Log-prob,
(actions twds sub-goal)
Figure 7: Architecture of the encoder the main agent and the auxiliary agent
2D Dropout layer with dropout probability 0.05;
CoordCony layer with kernel size = 3, padding 1, stride 1, that reduces the channel size by half while keeping the other dimensions intact;
CoordUpCony layer with kernel size = 3, padding = O, stride = 2, that maintains the channel dimension and doubles the spatial dimensions of the feature map: ReLU activation except for the last block where it is removed.
This stack produces a feature map of dimension [4,95, 95) that is processed by a 1 x 1-Convolution layer to create the output of size [2, 95, 95), that represents the unnormalized logits of each map pixel being navigable and navigaaaea
For each agent tested, create training dataset by running the agent subset of 150 episodes for each of the 72 scenes of the Gibson PointGoal training split for total of 10,800 episodes. We evenly sample each trajectory to obtain approximately 20 observations for each episode, so that we collect training sets of about 200,000 samples. Using the same procedure we also collect validation dataset from Gibson scenes comprising approximately 20,000 samples, and test dataset built using the Gibson Pointgoal val split also comprising 20,000 samples.
The probing network is trained with a batch size 64 using the AdamW optimizer (Loshchilov & Hutter, 2018) with learning rate 10-3 and weight decay 10-5 to minimize the cross-entropy loss Loe between groundtruth and reconstructed occupancy maps. The validation dataset used to perform early-stopping, which in probing experiments is particularly important avoid that scene structure gets its way into the parameters of the probing network.
A.3.2 ADDITIONAL PROBING EXPERIMENTS
As discussed in the main text, we visualize the probing results for four agent variants: (a), (c) and (e) from Table and Table from the main text, with (c) declined into sub-variants (c. 1) and (c.3) in Table 3. The reconstructed maps are shown in Figures 8 and 9. As reminder, (a) is standard PPO agent, while (c) variants are trained using behaviour cloning.
Figure 8 shows examples of reconstructed occupancy maps for the four agents in the Sim envi- ronment without noise. PPO estimates reasonably well the shape of the navigable space, but is somewhat conservative and captures well only parts of the map, leading frequent navigation failures. Reconstructions provided by (c. exhibit similar characteristics. The (c.3) variant performs poorly according to all quantitative probing metrics, and the reconstructed maps visually confirm that.
16

---

Published as a conference paper at ICLR 2024
Method
GT
Pred.
GT
Pred.
GT
Pred.
(a) PPO
(c.1) BC
(c.3) BC
(e) Ours
Figure 8: Additional probing results: Sim setting without noise. Three pairs of groundtruth and predicted navigable maps (in gray) for the four agents (a). (c.1). (c.3) and (e). The agent is in the center (blue dot), and of the 10 trajectories used compute the Symmetric-SPI. are plotted with light blue lines. Trajectories terminate with green goal points when the target is reachable, while target points are when not reachable, and no trajectory leads to them.
Method
GT
Pred.
GT
Pred.
GT
Pred.
(a) PPO
(c.1) BC
(c.3) BC
(e) Ours
Figure 9; Additional probing results: NoisySim setting. Three pairs of groundtruth and predicted navigable maps with shortest path trajectories for agents (a), (c. (c.3) and (e).
Teleportation seems severely compromise the representation for the probing task. The proposed approach, agent (e), estimates less accurately than PPO and variant (c.1) the shape of the navigable space, but captures reasonably well the geometry of the environment, especially in the vicinity of the agent.
Figure displays qualitative evaluations the NoisySim environment. In this regime the proposed approach outperforms alternative methods, producing predictions comparable to the noiseless setting, while other methods' predictions degrade significantly.
17

---

Published as a conference paper at ICLR 2024
While the results of the probing experiments only allow to speculate about the nature of the represen- tations learned by the different methods, these observations seem to corroborate that representations learned in the pre-training phase with Navigability are more robust and transferable than those of alternative approaches tested the paper.
A.4 DETAILS OF THE NIISYSIM BVAL ENVIRONMENT
Complementary to our evaluation experiments on the real Locobot, evaluation experiments in simulation were conducted in, both, not-n and noisy simulated environments. Our noisy environment is largely based on the noise settings of the CVPR Habitat challenge (Kadian al., 2019), more specifically Gaussian noise on the RGB image, Redwood noise on the Depth image and Proportional Controller noise on the actuators.
However, we have applied some adjustments, motivated by the different objectives of our work, which to transfer agents from simulation to the target domain with zero domain adaptation to evaluate the transferability of the learned representation. We have observed that in the Habitat implementation of the depth noise model, a depth D above a given threshold was set to zero i.c.
if (D>T)
(8)
==0,
which is the inverse behavior of the noiseless setting, ie.
(D>T)
=
We argue that this extremely strong discrepancy does not fall into the category of noise but rather to a change in the nature of the sensor, messes up transfer and does not allow sound evaluation we therefore replaced (8) by the thresholding process of the standard variant (9).
Secondly, since the original Redwood noise designed for 640 480 images (Teichman et al., 2013), applying it to arbitrarily size depth image caused the appearance of black borders?. We have extended this by setting depth on the borders to their original values.
A.5 DETAILS ON THE IMPLEMENTATION OF NAVIGABILITY
Figure 10 illustrates the implementation of the navigability loss in example showing two long episodes, the first going from A₁ to Bi, the second from A2 to B2. Each episode has one waypoint each and several sub-goals and short episodes branching out from the waypoints, shown in Figure 10a. As in PPO, learning is implemented with a rollout buffer, which stores a sub sequence of the episode. The rollout buffer of length 128 steps in implementation can hold one several long sequences, or only a part of one or more sequences.
As mentioned in the paper, and is custom in the literature, train with multiple parallel environ- ments. We strictly separate between two types of environments, and maintain separate rollout buffers for these two types:
PPO Environments - are classical environments trained with RL (the PPO variant). As classically done, the rollout buffer is filled in the collection step and used for policy updates in the update step. No short sequences are used, and the trajectories are the ones taken by the agent.
BC Environments
either trained with classical behavior cloning (BC) with our Navigability loss or with a combination these two. this case, as shown in Figure 10b, we flatten the hierarchical structure of long and short episodes into single sequence, with the agent navigating between waypoints, at each waypoint going through the different short episodes, and then resuming at the waypoint again. The hidden state of the agent stored waypoints and restored if necessary, according to the choices ablated in Table 3.
hntppa ////tt hub. com/ cow/tacenoo///eooooooooooooooooe lacencokreaee rch/ habi tat cat-ss sim/o ob /d3/1 50c 7744 7:4 00435000544117 4479884444797 762147666944
habit at Lim/ sensurs monnesssooiieeeeeeeeeee/es sttiisseeedeeeeeeeeeeeeeeeeeeeee poodela/redwood httdeedddddeeppppeeoeedddddddddd noopiinnisee iewoosieeeeeeeeeeeeeeeeee model. py#173
Satppa: //ththub.b//t///// com/ facehcokreseardch wawassoorrhhhhaaaaaa http//haabattaaeeeeeeeeaaaaaaeae tatttttttaaaaaaeee /bb////b////aa /dd/d/dd1 50c52 67 42455a644 479001 76244766633
babitat_ee sensars///eeeeee..eee.e.e.eee sesessesooo mode r/rewoood _eept hun ise modell ppy8188
18

---

Published as a conference paper at ICLR 2024
(a)
Rollout buffer
M
EnvSi Envs/agents ageents
BC PPO
Environment steps
M
(c)
repı
aux
Figure Illustration the implementation of navigability loss. (a) An example showing two long episodes, first going from A₁ to B1, second from A2 to B2. Each episode has one waypoint each several sub-goals and short episodes branching out from the waypoints. (b) As in PPO, learning is implemented with a rollout buffer, which stores sub sequenec. (c) The computation graph of agent only trained on the navigability loss. The hidden state the representation f' is collected long episodes (Green) and the hidden state of auxiliary agent p collected on short episodes (Blue).
Both types of environments are combined into single unique batch for gradient updates.
Figure 10c illustrates how the rollout buffer of Figure 10b mapped the computation graph the full agent combining the representation predictor f. main policy æ and the auxiliary policy p. when everything trained with navigability loss only. Vertical time instants of Figures 10a and 10b are aligned and correspond each other. In its purest form, ie. agent variant (d) of Table only the navigability loss is used for training. In this case, the hidden state of the representation f is collected on long episodes (Green) and the hidden state of auxiliary agent p collected on short episodes (Blue).
A.6 DETAILS ON MAPPING PREDICTED DISCRETE ACTIONS TO VELOCITY LOCOBOT ACTIONS
In the experiments involving simulation only, all actions were discrete, and as stated the main paper, were taken from the alphabet (MOVE FORWARD 25cm, TURN LEFT, TURN RI GHT and STOP). The experiments on the real robot (the Locobot) required mapping the predictions of these discrete actions to velocity commands. A widely used closed-loop strategy executes each action by closed-loop controller until the discrete action is terminated, then pauses to collect sensor readings, performs prediction by the neural agent, and repeats. This strategy leads to slow executation and delays between discrete actions.
We have opted for handcrafted 0-order hold, open-loop velocity command, which led to speed-up of a factor of around 3x-4x. We map each discrete action to an equivalent velocity command o to be sent the Kobuki driver of the LoCoBot through ROS for execution. Then, we wait for fixed waiting time Ag to collect the observations (RGB-D and pose). Here, u is either linear velocity nn or angular velocity v, depending on the action taken. We empirically, found that setting v 0.25m/sec and U. to + 60deg/sec for a fixed waiting (A, = seconds) was the best setup to execute the required discrete actions.
19


Reviews:
[{"review_id": "jfe8XqAsw7", "reviewer": "Reviewer_sYMN", "summary": "The authors address the task or learning an informative latent state representation for downstream tasks. Instead of a conventional scene reconstruction loss they opt to learn a representation that optimizes navigability by introducing a time-series behavior cloning loss for a blind auxiliary policy (with no future timestep observations or reconstructions) on generated short-term sub-goals (section 3-4, page 4).\n\nThe authors discuss the importance of the “blindness property” with a toy example in section 4 (Figures 4 and 5) where they explain that the conventional behavior cloning method may learn to adopt a memory-less latent representation whereas their approach involving future timestep predictions based on past observations encourages memory in the latent representation.\n\nThe method is evaluated in simulation and real-world experiments. They report raw navigation success rate and SPL (success weighted by the optimality of planned paths) versus variants of behavior cloning and PPO (Table 1 and 2) where they method outperforms baselines. They further examine how to continue the latent state at the end of short sub-goal episodes (Table 3) and how to set the initial latent for the auxiliary blind policy (Table 4). The authors also examine the representative ability of their latent state by training a scene map reconstruction network and doing subsequent path planning with it (Figure 6 and Table 5). Although other baselines obtain a better raw map reconstruction, their method generally outperforms others in the navigation planning score (especially in noisy environments).", "questions": "- On page 8, “Impact of navigability”, the text reads: “We see that PPO (a) outperforms classical BC (b), which corroborates known findings in the literature.” However, the results in Table 1 appear to indicate the opposite, with BC (b) having a higher success rate. Is this a typo or a misunderstanding on my part?\n\n- In Table 3, the “c.3. Always continue” variant performs best. However, in Figure 6, now c.1 outperforms c.3. Is there any intuition behind this result?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Discussion and experiments are detailed and analyze the different components of the method. Experiments test many ablations and variants of baselines to illustrate the impacts of their additions.\n- Both simulation and real-world results.\n- The work is well written and clear.", "weaknesses": "I feel the prediction of future labels without reconstruction is not a very novel architecture. I am also not sure if the improvement in results is because of the “blindness property” in the navigability loss (equation 5) or simply the addition of training data from generated sub-goals. There is an experimental discussion section addressing this concern (“Is navigability reduced to data augmentation?”, page 8) where they compare against a behavior cloning agent with added sub-goals. However, the difference in performance ((c) versus (e) in Table 1) seems relatively small (about 1% for success rate and even smaller point margins for SPL) and there is no reported confidence intervals or variance for these results over repeated seeds or trials. Thus, I am unsure how convincing these results are.\n\n__Minor Criticisms:__\n- Table 2 seemed to be discussed in detail before Table 1 in the experimental section which I found a bit disorienting for readability purposes.\n- I believe the methods relation to a mole is only explained in appendix  A.2. (The auxiliary agent ρ (a.k.a. the “mole”)) and so I was initially a bit confused by the title.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JMLPqKpT3z", "reviewer": "Reviewer_hBzC", "summary": "This work presents an approach to learn an actionable representation of the scene without optimizing for the reconstruction objective. The key idea is to use the learned representation to navigate on multiple short sub-episodes without any direct visual observations. The learned representation is optimized by this blind auxiliary agent for navigability and not reconstruction. Extensive experiments in both simulation and real world demonstrate the effectiveness of the proposed approach over several baselines, eg. BC, PPO, Map+Plan, especially in noisy settings.", "questions": "The questions are mentioned in the weaknesses above.\n\n---\nI have read the other reviews and author's response and I am retaining my rating. I think the paper will be a good addition to the conference.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The learned representation is optimized for its usability in navigation rather than reconstruction objective. This makes it suitable for any downstream task involving navigation.\n- The latent cognitive map can be learned via a blind auxiliary agent without the use of direct visual observations.\n- The paper is well written. The related work section is quite comprehensive and provides a good overview of the literature. Sec. 4 provides a good discussion on the importance of blindness and differences with data augmentation.\n- Extensive experiments (Table 1) in simulation and NoisySim settings show the benefits of Navigability when combined with policy learning approaches, eg. BC and PPO. Sim2real performance (Table 2) on a physical robot shows its effectiveness over BC, PPO & mapping+planning in real world.\n- Ablation study on the continuity of hidden state (Table 3), connection between the representation and blind policy (Table 4), and reconstruction performance (Fig. 6) provide valuable insights into the capabilities of the proposed approach.", "weaknesses": "- Sec. 4 reasoning makes sense for GRUs & LSTMs since the gating mechanism in GRUs & LSTMs explicitly encourages the specified behavior. Would this also hold for other architectures, eg. transformer? It'd be useful to provide some insights into this.\n- Sec. 4 argues that the training will prioritize learning `r1` and neglect `r2` and provides two reasons. Is there any empirical evidence for this in the context of navigation?\n- For BC(L)(S)(data augm.) baseline in Table 1, when training on the short routes, is the hidden state copied from the long trajectory or initialized separately?\n- How are the c.1 & c.3 variants in Fig. 6 different from Table 3? The c.3 variant in Fig. 6 is worse than other baselines but it works the best in Table 3. It seems like `Always continue` works well with navigability but not with BC. It'd be helpful to clarify why the performances are so different.\n- What does `sym-SPL` represent? why is the minimum of the two ratios taken?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JhDRy14AEy", "reviewer": "Reviewer_SbcL", "summary": "This work is about learning an abstract latent representation $\\mathbf{r}_t$ for navigation based on end-to-end learning, without a metric map.\nThe representation is inferred by a recurrent neural network from past RGB-D images and PointGoal-task information.\nA policy outputs actions based on this representation.\n\nThe main contribution is in how the representation is learned: instead of training it through the main policy directly, the authors use an auxiliary \"blind\" policy.\nIn training, the auxiliary policy has to navigate to randomly picked subgoals branching off of the main policy path, but it has to do so using only the last inferred representation $\\mathbf{r}_t$ from the main policy loop, with no access to new RGB-D observations.\nIt is argued that this would force the representation to be better suited for navigation in unseen environments (which is debatable, see my comments about this below).\n\nThe approach is trained and evaluated in simulation (Habitat, Gibson scenes), and sim2real transfer is studied on a small scale (11 runs on a real robot).", "questions": "- In the sim2real experiments, which BC variant was used as a baseline? With augmentation (i.e. subgoals) or without?\n- I am not sure I agree with the interpretation of the probing experiments. Sure, they indicate the reconstructed occupancy grids from the navigability representation are more optimistic, but then again can this be interpreted at all? There is no indication that the policy uses the representation in the same way, not to mention that absolute optimism w.r.t. collisions is not a desirable property per se.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The method appears sound on the whole, and most design choices are motivated.\n\n- The presentation was easy to follow, but there is some room for improvement (e.g. see comment on missing environment examples below).\n\n- The results indicate the method works in the considered environments, with success rates of ca. 90% in simulation and 80% in sim2real. To the best of my knowledge this is reasonable, compared to the prior art on end-to-end learned navigation. \n\n- Ablations of the main method aspects are included, which is appreciated.\n\n- There is some indication of improved sim2real transfer (e.g. compared to PPO), but there are also limitations of the evaluation (see weaknesses).", "weaknesses": "- The number of real-world experiments is rather low (11 episodes). While I can see how three baselines + the method itself add up in terms of workload, with only 11 runs I cannot tell if the sim2real results are statistically significant. I understand that it is unlikely this can be corrected on a short term during the review process, but it is an important point to consider for future revisions.\n- Related to the previous point, the paper is currently missing information about the complexity of the environments and the length of the evaluated navigation runs in them. Some qualitative examples in 2D would be nice, particularly of the real office environment. I am primarily concerned about this because navigation success metrics depend very much on a) the geodesic length of the navigation trajectory and b) how maze-like the layout of the scene is. Revealing the complexity of the considered navigation tasks will make the paper stronger.\n- I am not certain if the blindness property (a forefront contribution of the paper) is as useful compared to data augmentation:\n    - The results seem to support that subgoal spawning is beneficial. However, I am not convinced that the removal of observations in the auxiliary policy matters a lot. Simple behavior cloning + the subgoal data augmentation ((c) in Table 1) performs very well both in Sim and in NoisySim, better than plain navigability training ((d) in Table 1).\n    - I understand that the number of RGB-D observations with the proposed method is reduced, but this is a choice. The number of env interactions remains the same, so the comparison is on even terms.\n    - In the sim2real experiment, was the BC baseline ((c) in Table2) trained with the data augmentation regime, or regular? I believe sim2real results with BC + subgoal augmentation compared against the proposed method would have been ideal here.\n- The main text should specify the exact data fed into the networks clearly, this would make it much easier to assess the applicability of the solution. If I understand appendix A.2 correctly, the representation GRU takes both RGB-D images (not just RGB) and positional data (GPS + compass orientation), whereas the main text only talks about visual observations. This is somewhat misleading, navigation based on RGB data alone is very different from navigation with RGB-D images (from which collision information is easy to extract) with known poses.\n- The probing network is not directly related to the policy, so I am skeptical about the interpretation of the probing results at the very end of sec. 5. I think the paper would be better off by highlighting that this investigation is speculative, and should be taken with a grain of salt.\n\nOverall, I find that some of the main claims are stronger than what the experiments show (e.g. sec. 4) and that the evaluation could have been more precise. Note that I understand and can agree with the intuitive arguments in sec. 4, but these need to be supported better by the results. Still, the proposed training scheme does have some merit and the method seems to work, so I am leaning slightly positive in my assessment.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)