Learning to Move with Affordance Maps
Authors: William Qi
Ravi Teja Mullapudi
Saurabh Gupta
Deva Ramanan
Venue:  (2020)
Keywords: navigation
exploration

Abstract: The ability to autonomously explore and navigate a physical space is a fundamental requirement for virtually any mobile autonomous agent, from household robotic vacuums to autonomous vehicles. Traditional SLAM-based approaches for exploration and navigation largely focus on leveraging scene geometry, but fail to model dynamic objects (such as other agents) or semantic constraints (such as wet floors or doorways). Learning-based RL agents are an attractive alternative because they can incorporate both semantic and geometric information, but are notoriously sample inefficient, difficult to generalize to novel settings, and are difficult to interpret. In this paper, we combine the best of both worlds with a modular approach that {\em learns} a spatial representation of a scene that is trained to be effective when coupled with traditional geometric planners. Specifically, we design an agent that learns to predict a spatial affordance map that elucidates what parts of a scene are navigable through active self-supervised experience gathering. In contrast to most simulation environments that assume a static world, we evaluate our approach in the VizDoom simulator, using large-scale randomly-generated maps containing a variety of dynamic actors and hazards. We show that learned affordance maps can be used to augment traditional approaches for both exploration and navigation, providing significant improvements in performance.

Published as a conference paper at ICLR 2020
LEARNING TO MOVE WITH AFFORDANCE MAPS
William Qi' Ravi Teja Mullapudi!
Saurahh Gupta' Deva Ramanan'
Carnegie Mellon University
UICC
ABSTRACT
The ability to autonomously explore and navigate physical space is funda- mental requirement for virtually any mobile autonomous agent, from household robotic vacuums to autonomous vehicles. Traditional ELAM-based approaches for exploration and navigation largely focus on leveraging scene geometry, but fail to model dynamic objects (such as other agents) or semantic constraints (such as wet floors or doorways). Learning-based RL agents are an attractive alterna- tive because they can incorporate both semantic and geometric information, but are notoriously sample inefficient, difficult to generalize to novel settings, and are difficult interpret. In this paper, combine the best of both worlds with modular approach that learns a spatial representation of a scene that is trained to be effective when coupled with traditional geometric planners. Specifically, we design an agent that learns to predict a spatial affordance map that elucidates what parts of a scene are navigable through active self-supervised experience gathering. In contrast to most simulation environments that assume a static world, we evalu- ate our approach in the VizDoom simulator, using large-scald sandomly-generatte maps containing a variety of dynamic actors and hazards. We show that learned affordance maps can be used to augment traditional approaches for both explo- ration and navigation, providing significant improvements in performance.
INTRODUCTION
The ability to explore and navigate within a physical space is a fundamental requirement for virtually any mobile autonomous agent, from household robotic vacuums to autonomous vehicles. Traditional approaches for navigation and exploration rely on simultaneous localization and mapping (SLAM) methods to recover scene geometry, producing explicit geometric map output. Such maps can be used in conjunction with classic geometric motion planners for exploration and navigation (such as those based on graph search).
However, geometric maps fail to capture dynamic objects within an environment, such as humans, vehicles, or even other autonomous agents. In fact, such dynamic obstacles are intentionally treated as outliers to be ignored when learning a geometric map. However, autonomous agents mus/ follow navigation policy that avoids collisions with dynamic obstacles ensure safe operation. Moreover, real-world environments also offer a unique set of affordances and semantic constraints specific to each agent: a human-sized agent might fit through a particular door, but a car-sized agent may not; similarly, bicycle lane may be geometrically free of obstacles, but access is restricted to most agents. Such semantic and behavioral constraints are challenging to encode with classic SLAMM
One promising alternative is end-to-end reinforcement learning (RL) of policy for exploration and navigation. Such approaches have the potential to jointly learn an exploration/navigatoon planner together with an internal representation that captures both geometric, semantic, and dynamic con- straints. However, such techniques suffer from well-known challenges common to RI. such as high sample complexity (because reward signals tend be sparse), difficulty in generalization to novel environments (due to overfitting), and lack of interpretability.
We advocate hybrid approach that combines the best of both worlds. Rather than end-to-end learn- ing of both a spatial representation and exploration policy, we apply learning only "as needed". Specifically. we employ fff-the-sheeff planners, but augment the classic geometric map with spa- tial affordance map that encodes where the agent can safely move. Crucially, the affordance map is learned through self-supervised interaction with the environment. For example, agent can

---

Published as a conference paper at ICLR 2020
discover that spatial regions with wet-looking floors are non-navigable and that spatial regions that recently contained human-like visual signatures should be avoided with a large margin of safety. Evaluating on exploration-basee task, we demonstrate that affordance map-based approaches far more sample-efficient, generalizable, and interpretable than current RI.-based methods.
Even though we believe our problem formulation to be rather practical and common, evaluation is challenging in both the physical world and virtual simulators. It it notoriously difficult to evalu- ate real-world autonomous agents over large and diverse set of environments. Moreover, many realistic simulators for navigation and exploration assume a static environment (Wu et al., 2018) Saavaa al. 2017 Xia al. 2018). We opt for first-person game-based simulators that populate virtual worlds with dynamic actors. Specifically, we evaluate exploration and navigation policies in VizDoom (Wydmuch et al., 2018), a popular platform for RI. research. We demonstrate that affor- dance maps, when combined with classic planners, dramatically outperform traditional geometric methods by 60% and state-of-the we demonstrate that by combining active learning and affordance maps with geometry, navigation performance improves by up to 55% in the presence of hazards. However, a significant gap still remains between human and autonomous performance, indicating the difficulty of these tasks even in the relatively simple setting of a simulated world.
RI. approaches by 70% in the exploration task. Additionally,
RELATED WORK
Navigation in Classical Robotics. Navigation has classically been framed as a geometry problem decomposed into two parts: mapping and path planning. Inputs from cameras and sensors such as LiDARs are used to estimate geometric representation of the world through SLAM (or structure from motion) techniques (Thrun et al. 2005 Cadena et al. 2016). This geometric representation is used to derive map of traversability, encoding the likelihood of collision with any of the in- ferred geometry. Such a map of traversability can be used with path planning algorithms (Kavraki) al.) 1996; Canny| 1988] La Valle, 1998) to compute collision-free paths to desired goal locations. Navigation applications can be built upon these two primitives. For example, exploration of novel environments can be undertaken by sampling point goals in currently unknown space, planning paths to these point goals, and incrementally building map using the sensor measurements along the way (also known as frontier-based exploration (Yamauchi) 1997)). Such an approach for exploration has proven to be highly effective, besting even recent RL- -Lasd techniques in static environments (Chen ct al. 2019), while relying on classical planning (Fiorini & Shiller 1998).
Semanties and Learning for Navigation. Taking a purely geometric approach to navigation is very effective when the underlying problem is indeed geometric, such as when the environment is static or when traversability is determined entirely by geometry. However, an entirely geometric treat- ment can be sub-optimal in situations where semantic information can provide additional cues for navigation (such as emergency exit signs). These considerations have motivated study semantic SLAM (Kuipers& & Byun 1991), that seeks to associate semantics with maps (Bowman et al. 2017]
McCormac et al. 2017). ) speed up map building through active search (Leung et al. 2008). or factor out dynamic objects (Bescos et al. 2018).
In a similar vein, a number of recent works also investigate the use of learning to solve navigation tasks in an end-to-end manner (Zhu et al. 2017; Gupta et al. 2017 Mirowski et al. 2017 Fang jt al. 2019, Shreetha al., 2019), built upon the theory that an agent can automatically learn about semantic regularities by directly interacting with the environment. Semantics have also been used as intermediate representations to transfer between simulation and the real world (Müller et al.) 2018). While such use of learning is promising, experiments in past work have focused only on semantics associated with static maps. Instead, we investigate the role of semantics in dynamic environments, and in scenarios where the notion of affordance goes beyond simple geometric occupancy.
Another recent approach (Mirchev et al., 2018) introduces a method of learning generalized spatial representations for both exploration and navigation, employing attention-based generative model to reconstruct geometric observations. Planning for navigation occurs in belief space, in contrast to the metric cost maps (incorporating both semantics and geometry) used in our work.
Hybrid Navigation Policies. While barming-based methods leverage semantic cues, training such policies be sample inefficient. This has motivated the pursuit of hybrid policy architectures that
2

---

Published as a conference paper at ICLR 2020
combine learning with geometrie reasoning (Gupta al. 2017 Bhatti al. 2016 known robot
dynamic models (Bansal et al. 2019 Kaufmann et al. 2018 ; Müller et al., 2018). Our work also
presents hybrid approach, but investigates fusion of learned mapper with analytic path planning.
Self-Supervised Learning. Recent work in robotics has sought to employ self-supervised learn- ing (Pinto & Gupta, 2016)
alternative to end-to-end reward-based learning. [Hadsell et al. (2099) and Bruls et al. (2018) employ passive cross-modal self-supervision to learn navigability (from stereo monocular images, and from LiDAR monocular images, respectively). In con- trast, we learn through active interaction with the environment. Thus, our work is most similar to that of Gandhi al. (2017), though we learn dense traversibility predictions for long range path- planning, rather than sanre-range predictions for collision avoidance.
Navigation in Dynamic Environments. Finally, a number of other works develop specialized tech- niques for navigation in dynamic environments, by building explicit models for other agents' dy- namics (Chen et al., 2018) Kretzschmar et al., our definition of traversability beyond geometry alone, we can automatically capture the dynamics of other agents implicitly and jointly with other environmental features.
al.] 2016). contrast, by generalizing
3 APPROACH
Our goal is to build an agent that can efficiently explore and navigate within novel environments populated with other dynamic actors, while respecting the semantic constraints of the environment. The scenario we consider a mobile agent capable of executing basic movement macro-actions. The agent is equipped with RGBD camera and some form of proprioceptive feedback indicating well- being (e.g bump sensor, wheel slip, game damage). We assume that the agent is localized using noisy odometry and that depth sensing is also imperfect and noisy. At test time, the agent is initialized within a novel environment containing an unknown number of dynamic and environmental hazards. Furthermore, assume that the exact dimensions of the agent and nature of affordances provided by entities within the environment are not initially known.
Mapping (T t)
Planning
RGB Image
Affordance Map
Local Map
M,
Planning Module
Navigability Module
RGB
Map
T=t+1
MC
Depth Image
Geometric Map
Global Map
Planned Path
Figure 1: Overview of our proposed architecture for navigation. RGBD inputs are used to predict affordance maps li and transformed into egocentric navigability maps M, that incorporate both geo- metric and semantic information. In the example shown, M, is labelled as non-navigable in regions near the monster. A running estimate of the current position at each time step is maintained and used to update global, allocentric map of navigability G; that enables safe and efficient planning.
We propose modular approach to tackle this problem, adopting a classical pipeline of map build- ing and path planning. Figure shows an overview of this pipeline, which builds a navigability map using both geometric and semantic information, as opposed to traditional methods that rely on geometry alone. Our main contribution, shown in Figure 2 is method for predicting which parts of a scene are navigable by actively leveraging the feedback sensor to generate partially-labeled training examples. We then use the labeled samples to train a model which predicts a per pexepixel af- fordance map from the agent's viewpoint. At evaluation time, the outputs from the learned module are combined with geometric information from the depth sensor to build egocentric and allocentric representations that capture both semantic and geometric constraints. The fused representations
3

---

Published as a conference paper at ICLR 2020
then be used for exploration/navigation by employing traditional path planning techniques, enabling safe movement even within dynamic and hazardous environments.
3.1 NAVIGABILITY MODULE
Given a scene representation z captured by a RGBD camera, our goal is to train a module that labels each pixel with binary affordance value, describing whether the corresponding position is a valid space for the agent to occupy and forming a segmentation map of "navigability" y. We encode this understanding of the environment by training image segmentation model in supervised fashion. However, training such model requires a set of labeled training images D = [( M1 ()),,..(ar,, Mn]] where each pixel is annotated for navigability. Traditionally, obtaining such set of labels has required dense annotation by an oracle (Cordts et al.) (0166). at a cost that scales linearly with the amount of data labeled. These properties have generally limited applications to domains captured by large segmentation datasets (Lin et al.) 2014 Zhou et al. (017)) that have been curated using hundreds of hours of human annotation time. We address this problem by employing a self-supervised approach to generate partially labeled examples j, in place of oracle annotation.
Termination Condition
Labeled Navigability
+
Episode
Active/Random
Sensor
Trajectory Sampling
Exploration
Back-Projection
Initialization
Feedback
Figure 2: Overview of self-supervised labeling for navigability training pairs (ž, 1). The agent performs series walks along random or planned trajectories within the environment. Affordance information collected from each walk is back projectdd onto pixel-level labels in the agent's 's POV from previous time steps. Sampling over a variety of maps allows for the collection visually and semantically diverse set of examples that can be used to train a navigability module и. This figure illustrates the generation of negative example, with the agent contacting dynamic hazard.
Seff-Supervision. We generate labeled affordance data in self-supervised manner through con- tinuous interactive exploration by the agent; this algorithm makes use of RGBD observations x, readings from a feedback sensor and history of actions a, executed over time. In each episode, the agent is initialized at a random location and orientation within a training environment. The agent selects nearby point and attempts navigate towards it. Labeled training data is generated based on whether or not the agent is able to reach this point: every location that the agent successfully traverses during its attempt marked navigable, while undesirable locations (e.g. bumping into obstacles, loss of traction, loss in health, getting stuck) are marked as non-navigable, These locations in world space
then back-projected into previous image frames using estimated camera intrin- sics, in order to obtain partial segmentation labels (examples of which are visualized in Figure 3). Pixels for which there are no positive negative labels are marked as unknown. more detailed discussion about the real-world applicability of this approach can be found in Appendix A.4
Dense Labels. Backprojection of affordance labels produces a dense set of pixelwise labels for observations at past time steps. Importantly, even without spatio-temporal inputs, this enables the training models which incorporate safety margins to account for motion, as the future position of dynamic actors is encoded within labelled views from the past (discussed further in Appendix A.33. In contrast, most RL-based methods return only a single sparse scalar reward, which often leads to sample inefficient learning, potentially requiring millions of sampling episodes (Zhu et al.) 2017). Furthermore, our generated labels ÿ are human interpretable, forming a mid-level representation that improves interpretability of actions undertaken by the agent.
4

---

Published as a conference paper at ICLR 2020
Figure 3: Examples of samples labeled through back- pooj-projection (navigable area labeled in green, non-navigable in yellow, and unknown in purple). The first three examples show negative examples, labeled by damage from monster, impediment of movement by barrel, and damage taken from envi- ronmental hazard respectively. The fourth illustrates successful traversal between monsters and the fifth shows an example collected along a minimum cost path part of an active learning loop.
Navigability Segmentation. The collected samples 1) are used to train segmentation network such as UNet (Ronneberger et al. 2013), allowing for generalization of sampled knowledge to novel scenarios. A masked loss function Lmaak K @ L y) based on binary cross-entropy is employed to ensure that only labeled, non non-unknown points K within each example contribute to the loss. Given enough training data, the navigability module is capable of generating segmentation maps that closely approximate ground truth navigability, even in previously unseen environments.
Active Trajectory Sampling. In order to further improve sample efficiency, uncertainty to actively plan paths during sampling episodes so as to maximize label entropy along traversed trajectories. Intuitively, many semantically interesting artifacts (such environmental hazards) are rare, making it difficult to learn a visual signature. In these cases, sampling can be made more efficient by intentionally seeking out such artifacts. This can be achieved by first collecting small number (n) of samples using random walks and training a seed segmentation model. Using the seed model, we then predict an affordance map ý during the first step of each subsequent episode and use it to construct cost map for planning, with values inversely proportional to the prediction uncertainty (defined as the entropy of the predicted softmax distribution over class labels) at each position. Planning and following a minimal-cost path in this space is equivalent to maximization of label entropy, as the agent will attempt to interact most with highly uncertain areas. Once an additional n samples have been actively collected using this strategy, the model is retrained using mixture of all samples collected so far, and the sample/train loop can be repeated again. We find that active learning further increases sample efficiency, requiring fewer sampling episodes to learn visual signatures for hazards and dynamic actors (example shown in Figure rightmost.
employ model
3.2 MAP CONSTRUCTION
While some hazards only be identified using semantic information, geometry provides ef- fective and reliable means to identify navigability around large, static, obstacles such as walls. To capture both types of constraints, ric information when constructing the projected navigability cost maps M and G used for planning. As the agent moves around the environment, observed depth images are used to construct local, egocentric occupancy maps at each time step, incorporating only geometric information. By reading depth values from the center scanline of the depth image, projecting into the XY-plane, and marking the corresponding cells as non-navigable, a map of geometric obstacles M can be obtained. As the exact dimensions and locomotion capabilities of the agent are unknown, only depth values returned by the center scanline are known to be obstacles with certainty.
augment our predicted semantic maps with additional geomet-
Map Fusion. Given a pixel-wise affordance map ÿt obtained from the navigability module and local, egocentric geometric map MC. the two inputs can be combined using fusion module F(r.. M,C) to form a single local navigation cost map M, that incorporates both semantic and geometric information. To do so, the segmentation map St is first projected into the 2D plane using estimated camera intrinsics, forming an egocentric navigability map M, Cells marked as obstacles by M are also marked as impassable within Me. with remaining cells in free space assigned cost values inversely proportional to the confidence navigability provided by g.. Finally, M, is used to update global, allocentric map G, of navigability at the end of each time step.
5

---

Published as a conference paper at ICLR 2020
3.3 PLANNING
Given the global navigability map, path plamning can be tackled using classical algorithms such as A*, as all required semantic and geometric information encoded within the map itself. Addi- tionally, since both Mf and G2 are updated at every time step, dynamic hazards are treated as any other obstacle and can be avoided successfully as long as paths are re-planned at sufficiently high frequency. Our work is agnostic the choice of planning algorithm and our semantic maps can also be employed with more sophisticated planners, though for simplicity we evaluate using A#
4 EXPERIMENTS
We perform our evaluation in simulation using VizDoom, it allows for procedural generation of large, complex 3D maps that contain a variety of dynamic actors and semantic constraints in the form environmental hazards. Although prior work (Savinov al., 2018) on navigation has also relied on VizDoom, evaluation has been restricted to small set of hand designed maps without any dynamic actors or semantic constraints. We evaluate the effectiveness of incorporating learned affordance maps to tackle two difficult tasks: novel environment exploration and goal-directed navigation.
4.1 EXPERIMENTAL SETUP
We conduct our experiments within eeveddurall-generrate VizDoom maps created by the Oblige (Apted) (017) level generator, which enables construction of training and test maps containing unique, complex, and visually diverse environments. Each generated map large, containing a variety of dynamic hazards (such as monsters) and environmental hazards (such as lava pools), in addition to static obstacles (such as barrels) and areas where a geometry-affordant (such as ledges lower than sensor height, but beyond the movement capabilities of the agent). We generate a collection of 60 training and 15 test maps and further categorize the 15 test maps as either hazard-dense hazard-sparse, based concentration of hazards within the initial exploration area.
mismatch exists
Observation and Action Space. We assume that the agent's 's RGBD camera returns a regular RGB image with 60% field of view and approximately-coree clidean distance of each pixel from the camera in the XY plane (due to the 2.5DD nature of the Doom rendering engine). The feedback sensor returns scalar value corresponding to the magnitude of damage received by the agent while executing the previous action (some hazards are more danger- ous than others). The action space is limited to three motion primitives: move forward, turn right; only one action can be executed at each time step. Localization is imperfect and achieved through odometry from noisy measurements, with approximately 2% error.
depth image that records the 2D Eu-
left, and
4.2 SAMPLE-EFFICIENT EXPLORATION USING AFFORDANCE MAPS
We quantitatively evaluate exploration performance by measuring the total amount of space observed within particular environment over time, approximated by the total surface area of the constructed global map. Each episode of evaluation terminates after 2000 time steps or after receiving a total of 100 damage during exploration, whichever occurs first. Agents receive damage per time step when coming into contact with dynamic hazards and 20 damage for environmental hazards.
Frontier- Exploration. As classical, non-learning baseline, we compare against a variant of frontier-basee exploration (Yamauchi) 1997 Dornhege & Kleiner 2013). This approach relies purely on geometry, updating global map G2 at every step using the projected scanline observation MC from the current POV. A close-by goal from within the current "frontier region" is selected and a path towards it is re-planned (using A+) every steps as the map is updated. Once the selected goal has been reached the goal is determined to no longer be reachable, the process is repeated with a newly-selccted goal. Although dynamic actors can be localized using geometry alone, they treated as static obstacles in the cost map, relying frequent re- planning for collision avoidance.
RL-Based Exploration. We also compare against a state-of of fi-the-art art deep RL-based approach (Chen et al. 2019) for exploration that is trained using PPO (Schulman al. (017)) and incorporates both geometric and learned representations. We implement an augmented variant of the method proposed by (Chen et al. 2019), adding an additional depth map =.f the 3 original inputs: the current RGB

---

Published as a conference paper at ICLR 2020
observation 2 small-scale egocentric crop of G₁, and large-scale egocentric crop of G,. We identical to those proposed by the original authors,
evaluate this approach using with the only exception being the addition of penalty in the reward that is scaled by the amount of damage received at each time step. We report the mean performance obtained by the best model from each of training runs (2M samples each), with access to the full 60 map training set.
Cotordance-uugeentee Frontier Exploration. To evaluate the efficacy of our proposed repre- sentation, we augment the frontier-based approach with semantic navigability maps obtained from affordance predictions; all other components (including goal selection and path planning) are shared with the baseline. We collect approximately 100k total samples across the 60 training maps self-supervised manner and train the navigability module for 50 epochs using the collected dataset; ResNet- 18-based (He et al. 2016) UNet (Ronneherger et al. 2015) architecture is employed for segmentation. Episodic sample goals are selected randomly from within the initial visible area and simple path planning is employed, with the agent always taking a straight line directly towards the goal. Dack-projection is performed using game damage as a feedback mechanism, with the size of negative labels corresponding to the magnitude of damage received. At test time, we use estimated camera intrinsics to project output from the navigability module into the 2D plane. Additional ex- perimental details are discussed in Appendix
A.I A.I
Haare Spana Explor
Pacard Danum
Explo aalorn Perforr arte as Fandium of Thansng Samplex
Fronter
10.03
Truader
X102
2009
3000
Figure 4: Comparison of exploration performance across all evaluated approaches in hazard-dense (Left) and hazard-sparge environments (Center), plotted as a function of area observed over time. (Right) Comparison of final exploration coverage achieved by affordance-augmentte frontier ex- ploration, trained using varying amounts of self-supervised training data. All plots report mean performance measured over 5 test runs, shaded areas indicate the range of measured values, and RI. results are reported as mean performance of the best model from each of training runs.
Inside hazard- -parre environments (Figure left), agents generally don't encounter hazards within the first 2000 time steps, placing increased emphasis goal selection over hazard avoidance. In this setting, augmenting the frontier-based approach with affordance maps does not provide significant improvements, in the absence of semantic hazards, the two methods are functionally equivalent. In line with previous work (Chen et al. 2019), the PPO-based RL approach also fails to beat the frontier baseline, likely due the heavy emphasis placed exploration policy. Without taking high-level representation of the global map as input, it is difficult for a RL-based approach to plan over long time horizons, causing the agent to potentially re-visit areas it has already seen before. Finally, we note that humans are much better at both goal selection and hazard avoidance, managing explore upwards of 3x more area than the closest autonomous approach.
Successful exploration in hazard-dense environments (Figure 4 center) necessitates the ability to identify -eestriinc hazards, as well as the ability to plan paths that safely navigate around them. In this setting, augmenting the frontier-based approach with affordance maps increases per- formance by approximately 60%, which is more than 2/3 of the difference between frontier and the random baseline. Qualitatively, we observe that agents using learned affordance maps plan paths that leave a wide margin of safety around observed hazards and spend far less time stuck areas of geometry-affordane learns about agent-specific locomotion capabilities, predicting when low ceilings and tall steps may restrict movement. Although RL-based exploration out-performs the frontier baseline in this sce- nario by learning that proximity to hazards is detrimental to reward maximization, a lack of long term planning still hinders overall exploration performance.
mismatch. Through self-supervised sampling, the navigability module also
7

---

Published as a conference paper at ICLR 2020
Sample Efficiency. In order to understand the effect of training set size learned exploration, measure exploration performance with different amounts of collected samples in the hazard-dense setting, shown in Figure 4 right. After collecting few 5000 training samples, the navigability module learns to recognize dynamic hazards, allowing for paths to be planned with a margin of safety. As the number of samples collected increases, exploration performance improves well. However, as one might expect, the relative gain provided by each additional example decreases after a point. Qualitatively, we observe that 10,000 samples provides sufficient diversity to enable accurate localization of common dynamic hazards, while additional examples beyond this point help to improve detection of less commonly observed environmental hazards and aceuracy near hazard boundaries. Notably, even after training on 20 times as many samples, RL-based exploration still fails to outperform our approach in this setting, illustrating clear advantage in sample efficiency.
4.3 GOAL-DDRECTED NAVIGATION USING ACTIVE AFFORDANCE MAP LEARNING
Novigator Success Rates at Varying Damage Thresholds
stao
Bacch snd This Rosie))
Afhrrcanc (26 RE1
ac Dumaga
Figure 5: Comparison of navigation per- formance across all evaluated approaches, plotted as a function of success rate vs. maximum amount of damage permitted per trial (mean results over 5 test runs reported).
Figure 6: Examples of actively-planned trajectories that maximize label entropy along sampled loca- tions. (Left) shows predicted affordances, (Middle) shows the projected confidence map, and (Right) shows the cost map used to plan the optimal path.
In order to further demonstrate the applicability and efficacy of affordance- basee representations, we set up a series of 15 navigation trials, one for each map in the test set. Within each trial, the agent begins at fixed start point and is tasked with navigating to an end goal (specified in relative coordinates) in under 1000 time steps, while minimizing the amount of damage taken along the way. Each trial is designed to be difficult, with numerous hazards and obstacles separating the start and end points, presenting a challenge even for skilled humans. Additional experimental details are discussed in Appendix A.2
In this setting, we show that by adding semantic information obtained from affordance maps, it is possible to improve navigation performance significantly, even when employing simple geometry- based approaches that plan using A By introducing a navigability module trained on 100k col- lected samples generate cost maps for planning, we observe 45% improvement in overall nav- igation success rate, with improvements of 25% observed even when using a model trained on a dataset just one fifth the size. Even when the re-planning frequency increased 10-fold, such that observed dynamic hazards can be treated as static obstacles more accurately, the baseline still fails to beat the affordance-aud augmentted variant.
Additionally, we compare against results obtained by a PPO-based RL model, which is trained similarly to its counterpart discussed in Section 4.2 In order reduce the difficulty of planning over long time horizons, we provide the model with a sequence of waypoints (extracted from the best-performing human trajectory) as an additional input, which are used as local intermediate goals that converge towards faraway global goal. However, set of inputs, the RL-based approach still fails to beat any of the affordance- methods, echoing results observed in the exploration experiments.
observe that even with this augmented
We also explore how active learning can be used to further improve the efficiency of self self-super learning, by evaluating two additional models trained on samples collected from aatively-planned trajectories. We show that using just 40% of the data, models employing active data collection outperform those trained using random samples alone. At the 100k total sample mark, observe
8

---

Published as a conference paper at ICLR 2020
that actively sampled models out-perform their randomly sampled counterparts by more than 10%. These results, along with comparisons to the baseline, are summarized Figure f examples of actively-plannee trajectories visualized in Figure Qualitatively, observe that active trajec- tory sampling significantly improves temporal stability and prediction accuracy along hazard and obstacle boundaries (shown in Figure [). These properties enable more efficient path planning, allowing the agent to move safely with tighter margins around identified hazards.
5 DISCUSSION
We have described a learnable approach for exploration and navigation novel environments, Like RL-based policies, approach learns to exploit semantic, dynamic, and even behavioural proper- ties of the novel environment when navigating (which are difficult to capture using geometry alone). But unlike traditional RL, approach is made sample-efficient and interpretable by way of spatial affordance map, a novel representation that is igation with off-the- the-shelf planners. Though conceptually simple, we believe affordance maps open up further avenues for research and could help close the gap between human and autonomous ex- ploration performance. For example, the dynamics of moving obstacles currently captured only in an implicit fashion. A natural extension is making this explicit, either in the form of dynamic map or navigability module that makes use of spatio-temporal cues for better affordance prediction.
so as to be useful for nav-
REFERENCES
Andrew Apted. Oblige level maker.
wwchfak..cckk.kk.e sicchfak..ni--iillffll...ee content/bworice robot control www.wolssotwwarrreoooomm 2017. URL http://oblige.
cont- sourceforge.net.
Somil Bansal, Varun Tolani, Saurabh Gupta, Jitendra Malik, and Claire Tomlin. Combining optimal control and learning for visual navigation novel environments. arXiv preprint arXiv:1933.02533 2019.
Berta Bescos, José M Fácil, Javier Civera, and José Neira. Dynaslam: Tracking, mapping, and inpainting in dynamic scenes. IEEE Robotics and Automation Letters, 3(4):4076-408) 2018
Shehroze Bhatti, Alban Desmaison, Ondrej Miksik, Nantas Nardelli, N Siddharth, and Philip HS Torr. Playing doom with slam-augmented deep reinforcement learning. arXiv preprint arXiv: 161200000, 2016.
Sean Bowman, Nikolay Atanasov, Kostas Daniilidis, and George Pappas. Probabilistic data asso- ciation for semantic slam. In IEEE Int. Conf. on Robotics and Automation (ICRA), 2017. doi: http://www.dx.doi.orgg 10. 10.10//CAA.2221.788920000
Tom Bruls, Will Maddern, Akshay A Morye, and Paul Newman. Mark yourself: Road marking seg- mentation via weakly-supervised annotations from multimodal data. In 2018 IEEE International Conference Robotics and Automation (ICRA), pp. 1863-1870. IEEE, 2018.
Cesar Cadena, Luca Carlone, Henry Carrillo, Yasir Latif, Davide Scaramuzza, José Neira, Ian Reid, and John Leonard. Past, present, and future of simultancous localization and mapping: Toward the robust-perception age. IEEE Transactions on Robotics, 2016.
John Canny. The complexity of robot motion planning. MIT press, 1988.
Changan Chen, Yuejiang Liu, Sven Kreiss, and Alexandre Alahi. Crowd-robot interaction: Crowd-aware robot navigation with antention-based deep reinforcement learning. arXiv preprint 180.001800.08855, 2018.
Tao Chen, Saurabh Gupta, and Abhinav Gupta. Learning exploration policies for navigation, arXiv preprint 10..:v::90330159,, 2019.
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3213-3223, 2016.

---

Published as a conference paper at ICLR 2020
Vinayak V Dixit, Sai Chand, and Divya J Nair. Autonomous vehicles: disengagements, accidents and reaction times. PLoS one, 11(12):e016804, 2016.
Christian Dornhege and Alexander Kleiner. A frontier-viid-based approach for autonomous explo- ration in 3d. Advanced Robotics, 2766)4499-468 2013.
Kuan Fang, Alexander Toshev, Li Fei-Fei, and Silvio Savarese. Scene memory transformer for em- bodied agents in long-horizon tasks. IEEE Conference on Computer Vision and Pattern Recogni- tion (CVPR), 2019.
Paolo Fiorini and Zvi Shiller. Motion planning in dynamic environments using velocity obstacles. The International Journal of Roboties Research, 17(7):700-772, 1998.
Dhiraj Gandhi, Lerrel Pinto, and Abhinav Gupta. Learning to fly by crashing. In 2017 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 3948-3955. IEEE, 2017.
Saurabh Gupta, James Davidson, Sergey Levine, Rahul Sukthankar, and Jitendra Malik. Cognitive mapping and planning for visual navigation. In CVPR, 2017.
Raia Hadsell, Pierre Sermanet, Jan Ben, Ayse Erkan, Marco Scoffier, Koray Kavukcuoglu, Urs Muller, and Yann LeCun. Learning long-range vision for autonomous off-road driving. Journal of Field Robotics, 20-114, 2009.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Elia Kaufmann, Mathias Gehrig, Philipp Foehn, René Ranftl, Alexey Dosovitskiy, Vladlen Koltun, and Davide Scaramuzza. Beauty and the beast: Optimal methods meet learning for drone racing. 2018.
arXiv preprini
Lydia E Kavraki, Petr Svestka, J-C Latombe, and Mark H Overmars. Probabilistic roadmaps for path planning in high-dimensional configuration spaces. RA, 1996.
Henrik Kretzschmar, Markus Spies, Christoph Sprunk, and Wolfram Burgard. Socially compliant mobile robot navigation via inverse reinforcement learning. The International Journal of Robotics Research, 35(1
1307, 2016.
Benjamin Kuipers and Yung-Tai Byun. A robot exploration and mapping strategy based mantic hierarchy of spatial representations. Robotics and autonomous systems, 8(1-2):47-63, 1991.
Steven M Vallee.. Rapidly-exploring random trees: new tool for path planning. 1998.
Cindy Leung, Shoudong Huang, and Gamini Dissanayake. Active slam structured environments. In 2008 IEEE International conference on Robotics and Automation, pp. 1898-1903, IEEE, 2008
Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
John McCormac, Ankur Handa, Andrew Davison, and Stefan Leutenegger. Semanticfusion: Dense 3d semantic mapping with convolutional neural networks. In 2017 IEEE International Conference on Robotics and automation (ICRA), pp. 4628 4635. IEEE, 2017.
Atanas Mirchev, Baris Kayalibay, Patrick van der Smagt, and Justin Bayer Approximate bayesian inference in spatial environments. arXiv preprint arXiv:1805.07206, 2018.
Piotr Mirowski, Razvan Pascanu, Fabio Viola, Hubert Soyer, Andy Ballard, Andrea Banino, Misha Denil, Ross Goroshin, Laurent Sifre, Koray Kavukcuoglu, et al. Learning to navigate in complex environments. In ICLR, 2017.
Matthias Müller, Alexey Dosovitskiy, Bernard Ghanem, and Vladen Koltun. Driving policy transfer via modularity and abstraction. arXiv preprint arXiv: 0004 2018.
10

---

Published as a conference paper at ICLR 2020
Brian Paden, Michal Cáp, Sze Zheng Yong, Dmitry Yershov, and Emilio Frazzoli. A survey of motion planning and control techniques for self-driving urban vehicles. IEEE Transactions on intelligent vehicles, 1(1):33-55, 2016.
Lerrel Pinto and Abhinav Gupta. Supersizing self-supervision: Learning grasp from 50k tries and 700 robot hours. In 2016 IEEE international conference on robotics and automation (ICRA), pp. 3406-3413. IEEE, 2016.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedi- cal image segmentation. In International Conference Medical image computing and computer- assisted intervention, pp. 234-241. Springer, 2015.
Nikolay Savinov, Alexey Dosovitskiy, and Vladlen Koltun. Semi-parametric topological memory for navigation. In ICLR, 2018.
Manolis Savva, Angel X. Chang, Alexey Dosovitskiy, Thomas Funkhouser, and Vladlen Koltun. MI- NOS: Multimodal indoor simulator for navigation in complex environments. 171.:3171,,3331,, 2017.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1007.06347, 2017.
Rakesh Shrestha, Fei-Peng Tian, Wei Feng, Ping Tan, and Richard Vaughan. Learned map prediction for enhanced mobile robot exploration. In Proceedings of the IEEE International Conference on Robotics and Automation, ICRA. IEEE, 2019.
Sebastian Thrun, Wolfram Burgard, and Dieter Fox. Probabilistic robotics. MIT press, 2005.
Yi Wu, Yuxin Wu, Georgia Gkioxari, and Yuandong Tian. Building generalizable agents with realistic and rich 3d environment. arXiv preprint vrXiv:/801.022000 2018.
Marek Wydmuch, Michat Kempka, and Wojciech Jaškowski. Vizdoom competitions: Playing doom from pixels. IEEE Transactions on Games, 2018.
Fei Xia, Amir R. Zamir, Zhi- Yang He, Alexander Sax, Jitendra Malik, and Silvio Savarese. Gibson env: real-world perception for embodied agents. In Computer Vision and Pattern Recognition (CVPR), 2018 IEEE Conference on. IEEE, 2018.
Brian Yamauchi. A frontier-based approach for autonomous exploration. 1997.
Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 633-641 2017.
Yuke Zhu, Roozbeh Mottaghi, Eric Kolve, Joseph Lim, Abhinav Gupta, Li Fei-Fei, and Ali Farhadi. Target-driven visual navigation in indoor scenes using deep reinforcement learning. In 2017 IEEE international conference on robotics and automation (ICRA), pp. 3357-3364. IEEE, 2017.

---

Published as a conference paper at ICLR 2020
A APPENDIX
A.1 EXPLORATION EXPERIMENTAL DETAILS
A.1.1 ENVIRONMENTS
Each test map used to evaluate exploration performance is extremely large, containing sequences of irregularly shaped rooms connected by narrow hallways and openings. As it would require upwards of 10,000 time steps even for a skilled human to explore any of these environments, most of the space covered by the evaluated approaches is contained within the initial rooms next to the start point. As such, to clearly illustrate the challenges posed by semantic constraints, we choose to further categorize the test maps as either hazard-sparse or hazard-dense, based on the likelihood of encountering navigability-reste restrictnng hazards within the initial exploration area. Figure 7 shows a top-down visualization of the difference in hazard concentration between the two test subsets.
Figure 7: Top-down visualizations of initial exploration areas in hazard-sparse (Left) and hazard- dense (Right) test environments. Agent start position is marked in green, with environmental haz- ards marked in yellow, and initial locations of dynamic hazards marked in purple. Hazard-dense environments present significant challenge for autonomous exploration, containing high concen- tration of navigability restricting areas that must be avoided successfully.
A.1.2 BASELINES
We next provide additional details about the baselines that we compare against:
Random. In order to show that both geometry and learning-basee approaches set competitive baseline and perform well above the bar set by random exploration, evaluate policy that selects between each of the available actions with uniform probability at each time step.
2. RL-Based Exploration. To ensure fair comparison against RL-based exploration, we implement the approach proposed by (Chen et al., (019)) with identical sypperparameters, maintaining global map where each grid cell represents an also introduce new penalty term used to scale the reward by amount of damage taken in each step (set to 0.02), order encourage the agent avoid hazardous regions of the map.
game unit area the VizDoom world. We
3. Human. To demonstrate that all autonomous approaches for exploration still have significant room for improvement, we asked human volunteers to explore a sequence of novel environments using a near-identical experimental setup. In order make movement more natural, allowed participants to execute actions for rotation and forward movement concurrently, providing a slight edge locomotion compared to other approaches.
A.1.3 NAVIGABILITY MODULE TRAINING
As labeled samples y are generated by painting all pixels within radius of specified point in the simulated world as either navigable or non-navigable, we are more "certain" that pixels close by in world space share the same semantic constraints than those farther away. To ensure stability of training. we express this belief the loss by weighting each pixel's contribution by its inverse Euclidean distance to the closest such point in world space.
A.2 NAVIGATION EXPERIMENTAL. DETAILS
Each trial evaluated as part of the goal-directed navigation experiments is intentionally designed to be difficult, requiring agents to navigate through areas of high hazard concentration and around
12

---

Published as a conference paper at ICLR 2020
obstacles that present geometry- -ffortra-afoo mismatches. Indeed, none of human participants were able to complete all 15 trials successfully without taking any damage. Qualitatively, we observed that most failures in the baseline occurred when the agent attempted path through an obstacle lower than sensor height, causing the agent to become stuck as it continually tries to path through a cell registered free space in the geometric map. The second most common scenario that leads to degraded performance is failure to avoid dynamic hazards, causing agents to collide with monsters they attempt to follow nearby path the goal.
A.2.1 GEOMETRY- CHOMETRY-BASED NAVIGATION BASELINE
We implement the baseline approach for navigation using modified variant of the planning and locomotion modules employed for rontier-based exploration. Global navigability maps used for planning constructed by averaging values obtained from local maps allowing for increased temporal stability and robustness to sensor and localization noise. A simple A A*-based algorithm is then employed for planning, treating the value of each cell in the global navigability map as the cost traversing through a particular location in the environment.
multiple time steps,
In this setup, dynamic actors treated static obstacles within the map, assumption that holds true as long as re-planning is invoked at sufficiently high frequency. In order to evaluate the effect of re-planning frequency on navigation performance, also evaluate variant of the baseline approach that re- re-plans at 10x the frequency (every step instead of every 10 steps) and observe that this results in a small improvement in navigation trial success rate, largely attributed the reduction in latency required to respond to environmental dynamics.
A.2.2 AFFORDANCE- IHTORDANCE-AUGMENTER NAVIGATION
To evaluate the efficacy of active trajectory sampling, we evaluate effordance-augmeente navigation using segmentation models trained with active data gathering. The procedure follow is to first train a seed model using 20k random samples, before collecting an additional 20k samples actively and re-training generate improved version of the model. We repeat the active sample/train loop for an additional iterations, building a dataset with total size of 100k samples. Visualizations of predicted affordance maps generated by trained models after iterations 0, 2. and 4 of the sample/train loop are shown in Figure and compared to model trained using 100k randomly collected samples.
Figure 8: Comparison of affordance maps generated by models trained using datasets containing (a) 20k random samples, (b) 20k random samples + 40k active samples, (c) 20k random samples 80k active samples, and (d) 100k random samples. Employing active learning allows models to effectively identify and localize regions containing environmental hazards, feat that difficult to achieve using random samples alone.
A.3 CAPTURING DYNAMIC BEHAVIOR
In order to better understand how dynamic behavior is captured using our affordance-labeling ap- proach, we pose a scenario where dynamic actor moves from point to point B, and collides with the agent point B. In this scenario, all observations of point B (including those collected pre- collision) will be labelled as hazardous, potentially mapping to an image region near the dynamic actor rather than the actor itself. We will next deseribe approach for explicitly modeling such moving obstacles, and then justify why our current approach implicitly captures such dynamics.
Explicit Approach. In principle, our supe-vupe labeling system be modified to replace naive back projecttonn with an explicit image-based tracker (keeping all other components fixed). Essentially, labeled patches can be tracked backwards from the final timestep which they
13

---

Published as a conference paper at ICLR 2020
identified as hazardous (since those prior visual observations available sample time) to obtain their precise image coordinates when backprojecting to prior timesteps.
Implicit Approach. Even without image-based tracking, pipeline implicitly learns to generate larger safety margins for visual signatures that have been associated with dynamic behavior. Es- sentially, system learns avoid regions that are spatially close to dynamic actors (as seen in Figure 99. Such notions of semantic-specific safety margins (e.g.. autonomous systems should use larger safety margins for pedestrians roadside trash cans) are typically hand-coded in current systems, but these emerge naturally from our learning-ban approach. As we found success with an implicit encoding of dynamics, did experiment with explicit encodings, but this would certainly make for interesting future work.
Figure 9: Examples of learned margins for visual signatures associated with dynamic actors. From left to right: the first image shows a RGB view the scene, the second image shows predicted affordances ý overlaid on top of the RGB view, the third image shows the projected confidence map, and the last image shows the cost map used to plan the optimal path. From the first example, it can be seen that regions that are spatially close to dynamic actors are associated with higher traversal costs in the final cost map, akin to "margin of safety". The second example shows that static hazards/obstacles such as barrels are not associated with substantial affordance margins.
A.4 REAL-WORLL APPLICABILITY
During the sampling stage our proposed method, we employ a "trial and error" -style approach (similar to RL-based methods) that could lead potentially hazardous situations in real-word world robotics settings if deployed in certain configurations. However, we argue that this is not an un- reasonable way of collecting data and that there exist both common and practical solutions for risk mitigation that have already been widely deployed in the real-world.
Framing the current state of self-driving research within the context of our work, we can view all autonomous vehicles today as being within the sampling stage of a long-term active learning loop that ultimately aims enable L4 autonomy. Almost every of these vehicles on public roads today is equipped with one, if not multiple safety operators who are responsible for disengaging autonomy and taking over when the system fails to operate within defined bounds for safety and comfort. Moreover, each of these takeover scenarios is logged and used to improve the underlying models in future iterations of this learning loop (Dixit et al.] 2016). Indeed, in this scenario, the safety operator serves the purpose of the feedback sensor and can ultimately be removed test time, once the autonomous driving model has been deemed safe.
In less safety critical scenarios, such closed course or small-scale testing, the role of the safety driver could be replaced with some form of high- figh-freuency. high-resolution sensing such as multiple short-ramge range LIDARs. These feedback sensors the initial stages of active training, stopping the agent and providing labels whenever an undesirable state is entered. Importantly, since data from these expensive sensors not directly used input
be used to help the robot avoid collisions during
14

---

Published as a conference paper at ICLR 2020
by the model, they can be removed once satisfactory model has been trained: production-sper robots are free to employ low-cost sensing without the need for high-cost feedback sensors.
Additionally, there exist many scenarios in which feedback sensors help label examples with- out the need to experience catastrophic failures such as high-speed collisions. One example is the discrepancy between wheel speed sensor values, which can be used to detect loss of traction a wheeled robot when travelling over rough or slippery surfaces. By collecting observation-lane pairs, we could then learn affordance maps help such robot navigate over the smoothest terrain.
Finally, we would like to emphasize that in scenarios where it is difficult to obtain oracle-labelled data and "trial and error" approaches employed by necessity, have shown that proposed approach is many times more sample efficient that previous PPO-based reinforcement learning ap- proaches for mobile robotics (Chen et al. 2019) (which also suffer from the same types of problems). If collecting a sample is costly due to the burden of operational hazards, we argue that reduction in the number of samples required translates an improvement overall safety.
A.5 CODE
As the proposed system contains large number of moving parts and tunable pareee-pramettes, we plan to release modular open-source code for the described system at the following location: https://gitubb.com////////
A.6 ADDITIONAL VISUALIZATIONS
Figure 10: Examples of affordance maps I pre- dicted by the navigability module, showing curate localization of semantic constraints within the scene. (Left) contains dynamic hazards in the form of monsters and (Right) contains areas of geometry-affooddane mismatch, in the form of barrels shorter than sensor height.
Comparison of thresholded global maps constructed by frontier exploration using geometry (Left) and affordance- (Right) representations in the same environment. In this setting, semantic representations help the agent take less damage over time, allowing for more area to be explored during the episode.
Figure 11:
15


Reviews:
[{"review_id": "SkgvG0dpFr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes to learn affordance maps: a method to judge whether a certain location is accessible. This is done by distilling a series of \"trial and error\" runs and the relation of a pixel in the image/depth plane to a corrdinate into a model.\n\nI like the idea and think the paper should be accepted. The idea to use trial and error (something I prefer to self-supervision, which is used differently in many contexts, I believe) to obtain a data set for learning a model is nice and very practical.\n\nSome concerns that I think should be adressed.\n\n- The term information gain is used wrongly. The entropy of class labels is not infogain. Infogain is the expected KL of the model posterior from the model prior. Please correct this. See [1, 2].\n- *Learning* a model of the environment and using it for navigation/exploratin has also been tackled recently by [1]. I think the authors should draw connetions to that work.\n- Self-supervision has recently been proposed by Lecun as a subsitute (of sorts) for unsupervised learning. What he means is that a part of the data is used to predict another part of the data. I have no hard feelings about the term, personally preferring unsupervised, but the authors should be aware of the name clash.\n\nI wonder how the authors envision to extend this method to real scenarios. The \"trial and error\" method is clearly not viable for robotics setups, as hazards are costly. It would be nice if the authors could give there perspective on things.\n\n[1] Depeweg et al, \"Decomposition of Uncertainty in Bayesian Deep Learning for Efficient and Risk-sensitive Learning\", Proceedings of the 35th International Conference on Machine Learning\n[2] Mirchev et al, \"Approximate Bayesian Inference in Spatial Environments\" in proceedings of Robotics: Science and Systems XV.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "H1eF2cVy5r", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes an interesting, and to the best of my knowledge novel, pipeline for learning a semantic map of the environment with respect to navigability, and simultaneously uses it for further exploring the environment.\n\nThe pipeline can be summarized as follows: Navigate somewhere using some heuristic. When navigation \"works\", as well as when encountering something \"negative\", back-project that into past frames, and label the corresponding pixels as such: either positive or negative. This generates a collection of partially densely labelled images, on which a segmentation network can be learned that learns which part of the RGBD input are navigable and which should be avoided. For navigation, navigability of the current frame is predicted, and that prediction is down-projected into an \"affordance map\" that is used for navigation. One experiment confirms the usefulness of such an affordance map.\n\n\nI am marking weak reject currently because of the following concerns, which might be me just missing something. On the one hand, I am glad to see something that is not just blind \"end to end RL with exploration bonus\", sounds reasonable, and works well. On the other hand, I do have several major concerns about the method, outlined as follows:\n\n1. How can this approach work for moving obstacles? Let's say a monster walks from point A to point B, and collides with the agent at point B. Then, point B is marked as a hazard, but in the previous frames, the monster is not located at point B, and thus an image region that does not contain the monster is marked as hazard. Am I missing something here?\n2. The method does not seem practical for actual mobile robots, only for in-game or in-simulation agents. The reason being that in order to learn \"robot should not bump into baby\", the robot actually needs to bump into multiple babies in order to collect data about that hazard. To be fair, blind \"PPO+exploration bonus\" suffers from the same problem, but in this paper, the whole motivation is about mobile robots (at least that was my impression after reading it).\n\nFurthermore, I do not think I would be able to reproduce any of the experiments, as many details are missing. Will code be released?\n\n\n###### Post-rebuttal update\n\nI am happy with the author's response to my concerns, and they have included corresponding discussions in their paper. Thus, I am improving my rating to recommend acceptance of this paper to ICRL2020.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "BygsJ2jatS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents an approach for navigating and exploring in environments with dynamic and environmental hazards that combines geometric and semantic affordance information in a map used for path planning.\n\nOverall this paper is fairly well written.  Results in a VizDoom testbed show favorable performance compared to both frontier and RL baselines, and the author's approach is more sample-efﬁcient and generalizable than RL-based approaches.\n\nI wouldn't consider any particular aspect of this paper to be that novel, but it is a nice combination of leveraging active self-supervised learning to generate spatial affordance information for fusion with a geometric planner.\n\nAs humans show the best performance on the tasks, it might be worth considering learning a policy from human demonstrations through an imitation learning approach.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a framework for navigation that leverages learning spatial affordance maps (ie what parts of a scene are navigable) via a self-supervision approach in order to deal with environments with dynamics and hazards. They evaluate on procedurally generated VizDoom levels and find improvements over frontier and RL baseline agents.

Reviewers all agreed on the quality of the paper and strength of the results. Authors were highly responsive to constructive criticism and the engagement/discussion appears to have improved the paper overall. After seeing the rebuttal and revisions, I believe this paper will be a useful contribution to the field and I’m happy to recommend accept.

Decision: Accept (Poster)