SEAL: Self-supervised Embodied Active Learning using Exploration and 3D Consistency
Authors: Devendra Singh Chaplot
Murtaza Dalal
Saurabh Gupta
Jitendra Malik
Ruslan Salakhutdinov
Venue: NeurIPS 2021 Poster (2021)
Keywords: Embodied Active Learning
Embodied AI
Exploration
Navigation
Computer Vision
Self-Supervised Learning

Abstract: In this paper, we explore how we can build upon the data and models of Internet images and use them to adapt to robot vision without requiring any extra labels. We present a framework called Self-supervised Embodied Active Learning (SEAL). It utilizes perception models trained on internet images to learn an active exploration policy. The observations gathered by this exploration policy are labelled using 3D consistency and used to improve the perception model. We build and utilize 3D semantic maps to learn both action and perception in a completely self-supervised manner. The semantic map is used to compute an intrinsic motivation reward for training the exploration policy and for labelling the agent observations using spatio-temporal 3D consistency and label propagation. We demonstrate that the SEAL framework can be used to close the action-perception loop: it improves object detection and instance segmentation performance of a pretrained perception model by just moving around in training environments and the improved perception model can be used to improve Object Goal Navigation.

SEAL: Self-supervised Embodied Active Learning using Exploration and 3D Consistency
Devendra Singh Chaplot' Murtaza Dalal', Saurabh Gupta',
Jitendra Malik 14, Ruslan Salakhutdinov?,
Facebook AI Research, CCrnegie Mellon University, 'UICC, 'UC Berkeley
Project Webpage: https://deveedraccaapp httpaaah.potggg.t.bbob.o httthhbbo/poooecss/eeaae
Abstract
In this paper, we explore how we can build upon the data and models of Internet images and use them to adapt to robot vision without requiring any extra labels. We present a framework called Self-supervised Embodied Active Learning (SEAL). It utilizes perception models trained on internet images to learn an active exploration policy. The observations gathered by this exploration policy are labelled using 3D consistency and used to improve the perception model. We build and utilize 3D semantic maps to learn both action and perception in a completely self-supervised manner. The semantic map is used to compute an intrinsic motivation reward for training the exploration policy and for labelling the agent observations using 3D consistency and label propagation. We demonstrate that the SEAL framework can be used to close the retiion-peccetion loop: it improves object detection and instance segmentation performance of pretrained perception model by just moving around in training environments and the improved perception model can be used to improve Object Goal Navigation.
Introduction
Even though computer vision started out field to aid embodied agents (robots) [21], in current times it has evolved into Internet computer vision, where the focus is on training models for and from Internet data. Fueled by the successes of supervised learning, today's 's models can successfully classify, detect and segment out objects in Internet images reasonably well [19]. This raises a natural question: would we need to restart from scratch and gather similarly large-scale labeled datasets to get computer vision to work for embodied agents, or can we somehow bootstrap off the progress made in Internet computer vision?
Internet data comprises of sparse and unrelated snapshots of the world, carefully chosen by humans. On the one hand, this simplifies the problem models only need to reason about small and well-chosen subset of possible views of the world. On the other, this makes learning hard. The dog in image_ 0032 in the ImageNet dataset, is different from the dog in image_ image_0033, and there is no way to understand how the dog in image_0022 0032 will look like from another view. In contrast, an active agent, embodied in 3D environment, experiences views of 3D consistent world. Spatio-temporal continuity in views of the underlying 3D world can allow better inference at test time, and label efficient learning train time. These effects are further amplified, as the agent can actively choose the views it experiences to maximize learning. Thus, in contrast to Internet computer vision, embodiment opens up the possibility of deriving supervision from spatio-temporal continuity and interaction.
Such self-supervision for physical tasks (e.g. collisions, graspability) can be done through the use of appropriate sensors [17, 36). How to do for semantic tasks, such segmenting and detecting objects is less clear, and is the focus of our work in this paper. We build off models from Internet computer vision and show how their performance can be improved through self-supervised interaction.
"Conespondence: dchaplot@fb.com
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Phase 1: Action Learning active exploration polley
Phase 2: Perception
Training Mask-FICNN using spalio-temporal label propagation
Explaation Pallyy
Action
Sample Trgeehonos
Modiun PArei PICRE
Explosston
a0 Semands Maipping
Policy
Researd
Obsorvation
3D Semanttcc Map
3D Semantic Map
Model
30 Gomantic
Gebea
Marping
Percppton
Mocule
(Maak FIGNN)
Semantic Predictions
Figure 1: Self-supervised Embodied Active Learning, Our framework called Self-supervised Embodied Active Learning (SEAL) consists of two phases, Action, where learn active exploration policy, and Perception, where we train Perception Model on data gathered using the exploration policy and labels obtained using spatio-temporal label propagation. Both action and perception are learnt in completely self- supervised manner without requiring access to the ground-truth semantic annotations or map information.
We show that a) this can be done purely via interaction, without needing any additional supervision, b) improved models exhibit better performance in test environments as is, c) models can be improved further through small amounts of self- interaction in the test environment. We also show that improved perception can, in turn, improve the performance at interactive tasks.
We propose framework called Self-supervised Embodied Active Learning (SEAL) as shown in Figure It consists of two phases, one for learning Action, and another for learning Perception. During the Action phase, the agent learns self- seff-super exploration policy to gather observations of objects with at least one highly confident viewpoint. We propose an intrinsic motivation reward called Gainful Curiosity to train this policy. In the Perception phase, the learned exploration policy used to gather single episode of observations in an environment. We propose a method called 3DLabelProp to obtain labels for these observations in a self-supervised fashion. Both the action and perception phases involve constructing a 3D Semantic Map. The agent observations in an episode are used construct episodic 3D semantic map. The map is used to compute the Gainful Curiosity reward during the Action phase. And during the perception phase, the semantic labels in the 3D map projected the agent's 's original observations using 3DLabe/Prop to generate the supervision for training the embodied perception model.
Our experiments demonstrate that the SEAL framework can be used to close the action-perception loop. Perceptual models allow the agent to act the world and collect data that improve the perception models. Improved perception models can, in turn, improve the agent's 's policy for interacting with the world. We first use SEAL framework to improve object detection and instance segmentation performance of pretrained perception model (Mask RCNN [19]) from 34.82/32.54 AP50 scores to 40.02/36.23 AP50 scores by just moving around in training environments, without having access to any additional human annotations. By allowing the agent to cxplore the test environment for single episode, we can further improve the performance to 41. 21.23/37.29 AP50 scores. Next, we also show that this improved perception model can be used improve the performance of an embodied agent Object Goal Navigation from 44.4% to 62.7% % success rate.
2 Related Work
In this paper, we propose technique for self- self-supervised improvement of perception through active interaction, exploration, and 3D semantic mapping. Many papers tackle related problems and we survey them here.
Self-supervised Learning. A number of papers focus on the design of label-free pretext tasks to pre-train visual representations [1, 24, 57]. Most such works focus on learning a good generic feature representation without access to any semantic labels. Our work is also self-suuprrvise,, we also don't 't rely on any external source of supervision during training. However, instead of producing a generic feature representation, we produce improved object detection and segmentation models through self-supervision. Our supervision comes from the 3D self-consistency of given perception model on different views of the data. Furthermore, we learn to actively seek data to conduct this self-supervision past work that employs pre-collected datascts.

---

Domain Adaptation. Another related line of work is that of unsupervised domain adaptation [20, 47, 39]. where the focus to adapt models trained on one domain to work well on another domain without any labels, or any assumptions on the data that is available in the two domains. Our focus here is orthogonal: we specifically focus on opportunities for self sufervuueeviion available the context of embodied agents and learn policies to gather data in the target domain that eases transfer. General domain adaptation techniques could be used on top of our work.
Active Perception and Learning. Active interaction with an environment can improve performance at test time by gaining information from better views (i.e. active perception [6. 21), as well as at train time to generate more training data or to mine harder examples for external labeling (i.e. active learning [44]). Active learning and active perception have largely been thought of as two separate bodies of research. Our proposed approach tackles both these problems, and we highlight key differences from existing research.
Work in active learning focuses on the selection of data points from an unlabeled corpus for labeling by an oracle (e.g. humans). Researchers have explored the use of prediction uncertainty [45, 16], coverage [42] and meta-learning [28, 56] to learn such data selection functions. Our work departs from standard active learning in two ways: we do not assume access to pre-collected corpus of unlabeled data, nor to a labeling oracle. We learn a policy to efficiently and autonomously acquire unlabeled data, and rely on multi-view 3D consistency to derive supervision. This differentiates our work from work in active learning [37, 44], and also recent works that relax one or the other requirements: the need for oracles using spatio-temporal consistency to improve models using pre-collected unlabeled videos [38, 46, 27], or oracle supervised active learning in embodied environments [1 33].
Similarly, active perception has been studied over the last many years [6, 2]. Early methods used coommation-theorelle measures to determine the next best view [12]. More recently focus has been on learning policy in an end-to-end fashion directly improve metries of interest [25, 55, 26, 3, 54]. There also has been some work to improve segmentation and object detection using Simultaneous Localization and Mapping (SLAM) [58, 48]. The above works learn active interaction with an environment to improve performance at test time by gaining information from better views. In contrast, our focus is on active learning through interaction, i.e. learning how to explore an environment at train time to automatically generate more training data which is used to improve perception. Our proposed technique can also used to scale up active perception to effectively detect and segment objects in a large-scale 3D scene at test time (see the specialization setting in Sec 3). Rather than exploring each object one at time, we learn to explore the whole 3D scene at once. This improves recognition performance for all objects in the scene in single episode. Furthermore, unlike many prior methods, our approach is completely self-supervised.
Learning through Interaction. Robot learning and reinforcement learning focus on learning to solve interactive tasks directly through hit and trial interaction. However, there is also small body of work that seeks to improve perception models through unsupervised or self-supervised interaction. Unlike prior work [13. 35, 23] which tackle bottom-up image segmentation or learning object representations in table-top manipulation setting, we learn to improve object detection and instance segmentation models in the context mobile navigation agent. Parallel work from Fang et al. [14] also focuses on improving object detection models using active data. Unlike their work, we learn a policy for data collection, and aggregate information using 3D semantic maps.
3D Semantic Mapping. 3D mapping (reconstruction and localization) has been well studied, and is fairly mature sub-field of robotics and computer vision. We refer the readers to Fuentes-Pachest et al. [15] for a survey. Researchers have also considered the task of associating semantics with 3D maps [32, 8]. We adopt and adapt these ideas to our setting, and focus on how we could use these representations for learning active exploration policies to improve models for embodied perception.
3 Method
Our objective is to train an embodied agent to learn both action and perception by moving around in physical environment. We assume the agent is given access to a perception (object detection and instance segmentation) model, fr. such as a MaskRCNN [19] pretrained on static Internet data. The agent needs to learn a policy to move in the environment and use the experience to learn embodied perception in completely self-supervised manner without having access to the ground-trute semantic annotations map information.
3

---

RGB (1)
Semantic Predictions
Geocentric Vewel
(r...)
Aggregalion
Depth (D.)
Map
Point Cloud
Semannic scores
Point Cloud
Semanntcc
Egocantric
(c))))
Geocentric Cloudic!")
Sensor Pose Reading Us))
3D Semantic Mapping
Figure 2: 3D Semantic Mapping. The 3D Semantic Mapping module takes in sequence of RGB (T,) and
Depth (D₂) images and produces a 3D Semantic Map.
We propose a framework called Self-supervised Embodied Active Learning (SEAL) to tackle this problem. As shown Figure it consists of two phases, one for learning Action, and the other for learning Perception. During the Action phase, the agent learns self-supervised exploration policy to gather useful observations. In the Perception phase, the learned exploration policy used to gather a single episode of observations in an environment. The observations are labeled in self-superviser manner using 3D Label Propagation, which is then used for training the embodied perception model. In our framework, both the Action and Perception phases require building 3D semantic map from sequence agent observations. We first describe the 3D Semantic Mapping module (Sec 3.1) and then describe how it is utilized in both the phases train the active exploration policy (Sec 3.2) and the embodied perception model (Sec 3.3).
Generalization and Specialization. We employ this framework under two settings, Generalization and Specialization. In the Generalization setting, the agent is allowed to train for 10 million frames in the set of training environments and tested directly on set of unseen test environments. In the Specialization setting, after training, the agent also allowed to explore each test environment for single episode of ((= 300) time steps. In both settings, the agent has to learn completely in self-supervised manner, without having access to the ground-truth semantic annotations or maps in training or test environments. For the Generalization setting, the trained perception model is tested on random images in the unseen test environments. For the Specialization setting, the second phase is repeated for each test environment before testing the unseen images in the test environment.
Agent Specification. We follow the agent specification from Chaplot et al. [9]. For each time step depth observation, D2 E R and a 3-DOF pose sensor F; E R denoting the x and y coordinates of the agent and the orientation of the agent. The action space consists of 3 discrete actions: move forward (25em), turn left (30%) and turn right (30%). The agent camera height is 88cm.
i, the observation space consists of an RGB observation, I, E TR W, x,
3.1 3D Semantic Mapping
We use voxel-based representation for the semantic 3D Map. The semantic map, m, is 4D tensor of size K L W x H, where L,, W, H, denote the spatial dimensions, and K C + 1, where C' is the number of semantic object categories. Among the K channels, the first channel denotes whether the corresponding voxel (x-y-z location) is occupied or not and each of channels stores the score (between 0 and 1) of the corresponding voxel belonging to particular object category. We use a cubic voxel of size (5cm)).
Figure 2 shows an overview of the 3D Semantic Mapping module. The map is initialized with all zeros the beginning of an episode, my xx,xxX The agent always starts the center of the map facing east at the beginning of the episode, the pretrained MaskRCNN [19] (ʃp) to used to get semantic predictions from the input RGB observation, 4, The semantic predictions consist of the score for each pixel belonging to particular category obtained directly from the Mask-RCNN, there are multiple predictions for the same pixel, take the maximum score for each category across all the predictions. The depth observation,
(L/2, W/2, 0.0). At each time step

---

Sensor Pose Reading (x))
3D Semantic
Mapping
Somantic Map (rg)
Global Policy
Deterministic Local Policy
Observation (5)
Long torm goal (g)
Action (a.)
(RGBD)
Figure 3: Learning Action using Gainful Curiosity. We use a modular architecture for the Gainful Curiosity Policy. The 3D Semantic Mapping module is used to construct and update the map, m. at cach time step t. The Global Policy is used to sample long term goal (g:). A deterministic Local Policy is used plan path to the long-term goal and take low-Ievel navigational actions, The Gainful Curiosity intrinsic motivation reward computed using the 3D Semantic Map.
De, is used to compute egocentric point cloud, c, c:90. Each point in this egocentric point cloud is associated with the corresponding semantic predictions. The egocentric point cloud is converted to a geocentric point cloud, e using geometrie transformations based on the agent pose, 2t, which is then converted to geocentric voxel representation v E IR x x.H using geometric projections. In the voxel representation, the first channel among K channels represents occupancy, and the rest of the C' channels denote the maximum score for the voxel belonging to the corresponding semantic category. This voxel representation is aggregated over time using channel-wiz max pooling get the 3D Semantic Map.
3.2 Learning Action
The active exploration policy in SEAL needs to explore the environment to gather useful observations for learning perception. Intuitively, would like the agent to explore as many objects possible with a highly confident prediction for each object from at least one viewpoint. We require at least one view with high confidence the agent needs to learn in self-supervised fashion. In the next subsection, we describe how a highly confident prediction from one viewpoint can be used to label other viewpoints using 3DD consistency.
Gainful curiosity. We define an intrinsic motivation reward called Gainful Curiosity to train the active exploration policy to learn such behavior of maximizing exploration of objects with high confidence. We define 6(= 0.9) to be the score threshold for confident predictions. The Gainful Curiosity reward then defined to be the number of voxels in the 3D Semantic Map having greater than score for at least one semantic category. This reward encourages the agent to find new objects and keep looking the object from different viewpoints until it gets highly confident prediction for the object from at least one viewpoint. It also encourages the agent to not spend time exploring walls and corners as they do not belong any object category. Unlike prior formulations of intrinsic motivation, such as prediction-error curiosity [34] or temporal inconsistency- hased semantic curiosity [11] which aim to maximize uncertainty, Gainful Curiosity aims gain definitive knowledge.
We use a modular architecture for the Gainful Curiosity policy inspired by prior work on modular learning models for visual navigation [10, 9]. The architecture shown in Figure 3 consists of the 3D Semantic Mapping module to build the map as described earlier. The 3D Semantic Map (m²) is passed input to Global Policy which selects long-term goal (gt) essentially an x-y coordinate of the map. The Global Policy consists of convolutional layers followed by fully connected layers. A deterministic Local Policy, which uses the Fast Marching Method [43] for path planning, is used to navigate to the long-term goal using low-level navigation actions. The Global Policy operates coarse time scale, sampling goal every 25 local steps. It is trained to maximize the intrinsic motivation reward with reinforcement learning.
waypoint, which is
3.3 Learning Perception
The trained exploration policy used to sample trajectories in the training environments. For each trajectory, we build the 3D Semantic Map as described above. The semantic predictions from the Perception Model (fr) for the observations in this trajectory might contain both false positives (detecting an object when there is no object or predicting the wrong object category) and false
5

---

3D Label Propagation
RGB Observation
Seper-uuee Labels
3D Semantic Map
Learning Perception using 3DI ProeePProp. The agent trajectory is to used to create a semantic 3D map of the environment. The map is labelled in self-supervised manner using 3D consistency. The label for each pixel in the agent trajectory is obtained using ray-tracing in the labeled map based on the agent's pose.
Figure 4:
negatives (not detecting object when there is an object present). Consequently, the 3D Semantic Map will contain ambiguities with scores for multiple object categories for a given voxel that may or may not belong an object.
PDLabelProp. We propose a method called 3DI abeeProp to obtain self-supervised labels from the 3D Semantic map. First, to perform disambiguation, we label each voxel to belong to the category with the maximum score above s. If all categories have a score less than 3 for a voxel, we label it as not belonging to any object category. After labeling each voxel in the map, we find the set of connected voxels labeled with the same category to find object instances. We fill small holes (< 0.25m²) in object instances and remove small objects (< 0.025m²) to get the final labeled 3D semantic map. The instance label for each pixel in each observation in the trajectory is then obtained using ray-tracing in the labeled 3D map based on the agent's 's pose as shown in Figure 4. Pixel-wise instance labels are used to obtain masks and bounding boxes for each instance. Note that this labeling process is completely self-supervised and does not require any human annotation. The set of observations and self-supervised labels are used to fine-tune the pre-trained perception model.
4 Experiments
Setup. We use the Habitat simulator [40] with the Gibson dataset [50] for our experiments. The Gibson dataset consists of scenes that are 3D reconstructions of real-world environments. We use set of 30 scenes from the Gibson tiny set for our experiments whose semantic annotations are available from Armeni et al. [5]. We use split of 25 and 5 scenes for training and testing identical to prior work [9]. The list of training and test scenes is provided in the supplementary material. Following the setup in prior work [11, 9], we use 6 common indoor object categories for all our experiments: chair, couch, bed, toilet, TVV, and potted plant. We randomly sample set of 2500 images (500 per test scene) and evaluate the final perception model trained using SEAL and the baselines on object detection and instance segmentation tasks. The same test set is used for all the methods and for both Generalization and Specialization settings. We report bounding box and mask AP50 scores for object detection and instance segmentation, respectively. AP50 is the average precision with at least 50% IOU. IOU is defined to be the intersection over union of the predicted and ground-trute -trth bounding box or the segmentation mask.
4.1 Hyperparameter and Architecture Details
Perception Model. We use Mask-RCNN [19] using Feature Pyramid Networks [31] with ResNet-50 [18] backbone as the Perception model. The Mask-RCNN is pretrained on the MS-COCO dataset [30] for object detection and instance segmentation. During the Perception phase, we fine- tune the Mask-RCNN on the data gathered by the Gainful Curiosity policy and labeled using 3D Label Propagation. We use Stochastic Gradient Descent [7] with fixed learning rate of 0.0001 for N = -5000 iterations. All other hyperparameters are set to default settings in Detectron2 [49].
Action Model. In the Gainful Curiosity model, the Global Policy is the only learnable component. It a layer convolutional network (2x3D convolution layers + 3x2D convolution layers) followed by 3 fully connected layers. In addition to the 3D Semantic map, we also pass the agent orientation as a separate input, which is processed using an Embedding layer and added as an input to the fully- connected layers. The Global Policy is trained using Proximal Policy Optimization [41] algorithm

---

Table 1: Results. Performance of all the baselines compared to the proposed SEAL framework for both Generatlization and Specialization settings. report bounding box and mask AP50 scores for Object Detection and Instance Segmentation.
Generalization
Specialization
Object Detection
Instance Segmentation
Object Detection
Instance Segmentation
Method
Pretrained Mask-RCNN
34.82
32.54
34.82
32.54
Random Policy + Self--raining [52]
33.41
31.89
34.11
31.23
Random Policy + Optical Flow [22]
33.97
32.34
34.33
32.22
Frontier Exploration [53] + Self-training [52]
33.78
32,45
33.29
32.50
Frontier Exploration [53] Optical Flow [22]
35.22
31,90
34.19
32.12
Active Neural SLAM [10] Self-training [52]
34.35
31.20
34.84
32.44
Active Neural SLAM [10] Optical Flow [22]
35.85
32.22
35.90
33.12
Semantic Curiosity [11] + Self-training [52]
35.04
32.19
35.23
32.88
Semantic Curiosity [11] + Optical Flow [22]
35.61
32.57
35.71
33.29
SEAL
40.02
36.23
41.23
37.28
with 25 parallel threads, with each thread using one scene in the training set. We use time horizon of 20 steps, 12 mini-batches, and 4 epochs in each PPO update. Our PPO implementation is based on [29]. The policy is trained with the Gainful Curiosity reward which is computed by counting the the number voxels explored with s(= 0.9) score for at least one object category. We use Adam optimizer with a learning rate of 0,000025, a discount factor of 7 = 0.99, an entropy coefficient of 0.001, value loss coefficient of 0.5 for training the Global Policy.
4.2 Baselines
We are not aware of any methods directly comparable to the proposed method. We adapt prior methods as separate baselines for Action and Perception phases and compare combinations of these baselines to our SEAL framework. We implement the following Action baselines:
Random Policy. A policy taking random navigation action at each time step.
Frontier Exploration [53]. This baseline uses the classical frontier-based exploration heuristic of navigating to the nearest unexplored point to explore unseen environments.
Active Neural SLAM [10]. Similar our Gainful Curiosity policy, Active Neural SLAM is also modular map-based learning method. It builds a spatial top-down map and learns higher-level global policy select waypoints in the top-down map space maximize area coverage.
Semantic Curiosity [11]. This baseline is closest to our Gainful Curiosity policy, which trained to maximize the temporal inconsistency in object detections and segmentation in trajectory.
We implement the following Perception baselines:
Self-Training. this baseline, we train a Mask-RCNN on its own predictions using the data collected by the exploration policy. This baseline is adapted from prior work self-training which shows improvement in image classification by training on large datasets of unlabelled images [52, 51].
Optical Flow. In this baseline, we use optical flow [22] between consecutive images for label propagation. Each pixel in the current image is labeled with the maximum score prediction of the pre-trained Mask-RCNI is also used by Eitel et al. [13] for learning segmentation in an interactive manipulation setting.
associated pixels in the previous, current, and next image. Optical flow
Results
We report the performance of the proposed SEAL framework and all the baselines for both the Generalization and Specialization settings in Table 1. SEAL outperforms all the baselines by large margin, 40.02/36.2 23 vs 35.851 85/32.57 AP50 score for Generalization, and 41. 11.23337.28 vs 35, 55.90/33.22 AP50 score for Specialization. SEAL also considerably improves over the Pretrained Mask-RCNN baseline (41.23/37.25 vs 34.82/32.54 AP50 score for Specialization) while all the other baselines perform worse or comparable the Pretrained Mask-RCNN baseline. This indicates that the SEAL Framework can be used explore physical environment and improve perception in completely self-supervised manner without requiring any human annotation.
7

---

ROB Olseevatioo
30 Semantic Map
Sell-Superviens RONN- Predecions
Figure 5: Example. Figure showing 3 frames in trajectory gathered using the trained active exploration policy, TA, corresponding pretrained Mask-RCNN predictions, and the self-supervised labels obtained using the (1) The Mask-RCNN predictions contain false negatives, it fails to detect the couch and the chair, (2) Mask-RCNN still fails to detect the couch, (3) MaskRCNN detects the couch but contains false positive prediction of the coffee table chair. The 3D Semantic Map (shown the right) obtained using the MaskRCNN predictions correctly consists the couch and chair. Self-supervised labels obtained using semantic map correctly identify the objects at all the timesteps.
3D Semantic Map.
Qualitative example. Figure we show an example trajectory and corresponding trajectory obtained through SEAL. The pretrained Mask-RCNN predictions (shown in the second row) contain both false positives and false negatives and are inconsistent over time they are based on individual images. Since the MaskRCNN correctly recognizes the objects with a high score from at least one viewpoint, they are correctly labeled in the 3D semantic map. Consequently, the self-supervised labels obtained from the map using ray-tracing are accurate and consistent over time. More examples are provided the supplement.
Ablations. To understand and quantify the importance of both the Action and Perception phases in SEAL, we perform experiments with several ablations. In each ablation, we replace the Action the Perception phase in SEAL with a corresponding baseline. Results in Table 4 indicate that both the Action and Perception phases are important for the overall performance. The Perception phase is more critical as replacing it with Self-training or Optical Flow drops the performance comparable the Pretrained -CNN. As noted in prior work [52], Self Self-Training requires large amounts (millions of images) of unlabelled data be effective which is not available in our setting. The performance of Optical Flow also limited as it can only aggregate information over consecutive frames. This indicates that aggregating information using semantic 3D mapping and self-supervised labeling using ray-tracing crucial for overall performance of the framework.
5.1 Closing the Action-Perception Loop: Object Goal Navigation
We demonstrated that the SEAL framework be used to learn an active exploration policy to gather data and improve perception models, i.e. used action to improve perception. In order to close the perception-action loop i.c. use perception im- prove action, we deploy the SEALLimproved percep- tion model downstream Object Goal Navigation task. We utilize the SemExp model from Chaplot al. [9] and replace the pretrained Mask-RCNN SemExp with the SEAL-improved perception model. All the other model components and the experimental setup is identical to [9]. As shown in Table 2, SEAL leads to large improvement, the SemExp + SEAL Specialization perception model led to a success rate of 0.627 and SPI. [4] of 0.331 as compared to 0.544 Success and 0.199 SPL for the SemExp model with the pretrained
Table Object Goal Navigation. Performance of SEAL-improved perception model on the Ob- ject Goal Navigation task. SEAL (Gen.) and SEAL (Spec.) refer to SEAL perception models from Generalization and Specialization settings.
Method
Success
SPI.
SemExp [9]
0.544
0.199
SemExp SEAL (Gen.)
0.611
0.323
SemExp SEAL (Spec.)
0.627
0.331

---

Table 4: Ablation Results. Performance of all the ablations of the SEAL framework for both Generatlizatinn Specialization settings. each ablation, we replace Action Perception phase SEAL with baseline. We report bounding box and mask AP50 scores for Object Detection and Instance Segmentation.
Generalization
Specialization
Object Detection
Instance Segmentation
Object Detection
Instance
Method
Segmentation
Pretrained Mask-RCNN
34.82
32.54
34.82
32.54
SEAL w/o Action + Random Policy
35.43
31.22
35.77
31.79
SEAL w/o Action + Frontier Exploration [53]
37.39
33.49
37.99
34.55
SEAL w/o Action + Active Neural SLAM [10]
38.90
34.99
39.01
35,41
SEAL w/o Action + Semantic Curiosity [11]
38,39
35.20
39.21
35.62
SEAL w/o Perception Self-training [52]
35.35
32.47
35.88
33.20
SEAL w/o Perception Optical Flow [22]
35.65
32.49
35.92
33.44
SEAL
40.02
36.23
41.23
37.28
5.2 Extension: Weak Supervision
We demonstrated that the SEAL framework can be used to improve perception and ac- tion in a self-supervised fashion. Addition- ally, it also be used under the weak supervision setting, where few frames in each test environment are annotated by hu- mans. For each annotated frame, we can simply replace the pretrained MaskRCNN predictions in the Perception phase with human annotations with score of 1. We sample frames with the highest average entropy over voxels corresponding to all pixels in the 3D semantic map and assume they are human-an .aman--nnnttted. In Table 3, we report the performance of a Mask-RCNI naively fine-tuned with few frames (k 0, 5. 10) of labeled data as compared using the proposed SEAL framework for label propagation. Naively fine-tuning Mask-RCNN with 10 labeled examples does not improve the performance much. SEAL leads to a considerable performance improvement as labels are propagated to other observations using 3D semantic mapping.
Table 3: Weak Supervision Results. Performance of a Mask- RCNN naively fine-tune tine-tuned with few frames of labelled data as compared using the proposed SEAL framework for label propagation. report bounding box and mask AP5O scores for Object Detection and Instance Segmentation.
Fine-tuning MCNN
SEAL
Num labels
Object Detection
Instance Segmentation
Object Detection
Instance
Segmentation
34.82
32.54
41.23
37.28
34,22
31.67
41.44
37.65
10
35.14
32.52
42.63
38.48
Discussion
We presented the Self-supervised Embodied Active Learning (SEAL) framework for closing the irtion-perceptions loop. We demonstrated that the SEAL framework be used to utilize pretrained perception model to learn an active exploration policy for gathering useful observations. The observations gathered by the policy can be used improve the pretrained perception model. Our ablation experiments highlight the importance of both the Action and Perception phases. The SEAL- improved perception further be used to improve performance at Object Goal Navigation. The entire framework is completely self-supervised, the agent learns a policy and improves perception by just moving around in physical environments without having access to any human annotations.
The ability to learn a self-supervised fashion comes at cost of some limitations. The quality of the 3D semantic map (and the labels obtained from it) is dependent on the performance of the pretrained perception model. the perception model never detects an object from any viewpoint, there is no way to learn about the object without any additional supervision. Similarly, if the perception model makes wrong predictions with high scores, these errors will be amplified by label propagation.
In the weak supervision extension, we explored how these limitations can be addressed to some extent with additional human supervision. In the future, the SEAL framework can also be extended to tackle few-shot learning of new objects with few annotated examples. We studied this problem under the embodied active setting, but the self-supervised label propagation is also applicable to passive video data. We tackled static scenes in this paper. In the future, the 3D semantic map can potentially be extended to dynamic scenes with moving humans by explicitly predicting which voxels belong to static and dynamic objects.

---

Acknowledgements
Carnegie Mellon University effort supported in part by the US Army Grant W911NFI920104 and DSTA. UIUC effort is partially funded by NASA Grant 8ONSSC2IK I030. Jitendra Malik received funding support from ONR MURI 100004-14-1-0000 Ruslan Salakhutdinov would also like to acknowledge NVIDIA's GPU support.
Licenses for referenced datasets:
Gibson: http://stt /svl..vll.... stanf http://..soooaaaaa//eeee//e aar.saaereeeetteeee
10

---

References
[1] Pulkit Agrawal, Joao Carreira, and Jitendra Malik. Learning to see by moving. In Proceedings of the IEEE international conference computer vision, pages 37-45, 2015. 2
[2] John Aloimonos, Isaac Weiss, and Amit Bandyopadhyay. Active vision. International journal of computer vision, 1(4):333-356, 1988. 3
[3] Phil Ammirato, Patrick Poirson, Eunbyung Park. Jana Košecká, and Alexander C Berg. A dataset for developing and benchmarking active vision. In 2017 IEEE International Conference on Robotics and Automation (ICRA), J. pages 1378-1385. IEEE, 2017.
[4] Peter Anderson, Angel Chang, Devendra Singh Chaplot, Alexey Dosovitskiy, Saurabh Gupta, Vladlen Koltun, Jana Kosecka, Jitendra Malik. Roozbeh Mottaghi, Manolis Savva, al. On evaluation of embodied navigation agents. arXiv preprint 1807v 06757, 2018. 8
[5] Iro Armeni, Zhi- Yang He, Jun Young Gwak, Amir R Zamir, Martin Fischer, Jitendra Malik, and Silvio Savarese. 3d scene graph: A structure for unified semantics, 3d space, and camera. In Proceedings of the IEEE International Conference on Computer Vision, pages 5664-5673, 2019.
[6] Ruzena Bajesy, Active perception. Proceedings of the IEEE, 76(8):966-1055, 1988.
[7] Léon Bottou. Large-scale machine learning with stochastic gradient descent. In Proceedings of COMP- STAT"2010, pages 177-186. Springer, 2010. 6
[8] Sean I. Bowman, Nikolay Atanasov, Kostas Danilliiis, and George Pappas. Probabilistic data association for semantic slam. In 2017 IEEE international conference on robotics and automation (ICRA), pages 1722 1729. IEEE, 2017. 3
[9] Devendra Singh Chaplot, Dhiraj Gandhi, Abhinav Gupta, and Ruslan Salakhutdinov. Object goal navigation using goal-oriented semantic exploration. In Neural Information Processing Systems, 2020. 4, 5, 6,
[10] Devendra Singh Chaplot, Dhiraj Gandhi, Saurabh Gupta, Abhinav Gupta, and Ruslan Salakhutdinov. Learning to explore using active neural slam. In ICLR, 2020. 5, 7.9
[11] Devendra Singh Chaplot, Helen Jiang. Saurabh Gupta, and Abhinav Gupta. Semantic curiosity for active visual learning. In ECCV, 2020, 3, 5, 6, 7,9
[12] Benjamin Charrow.
cettmmatiion-hhooettie active perception for multi-robot teams. 2015. 3
[13] Andreas Fitel, Nico Hauff, and Wolfram Burgard. Self-supervised transfer learning for instance segmenta- tion through physical interaction. In 2019 IFEF/RSI International Conference on Intelligent Robots and Systems (IROS), pages 4020-4026. IEEE, 2019. 3,7
[14] Zhaoyuan Fang, Ayush Jain, Gabriel Sarch, Adam W Harley, and Katerina Fragkiadaki. Move sec better: Towards self supervisdd amodal object detection, arXiv preprint arXiv:2012. 00000027200007, 2020. 3
[15] Jorge Pachhec,, José Ruiz-Ascencio, and Juan Manuel Rendón- Mandha.. Visual simultaneous localization and mapping: survey. Artificial intelligence review, 43(1):55-81, 2015. 3
[16] Yarin Gal, Riashat Islam, and Zoubin Ghahramani. Deep bayesian active learning with image data. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pages 1183-1192.
JMLR. org, 2017. 3
[17] Dhiraj Gandhi, Lerrel Pinto, and Abhinav Gupta. Learning to fly by crashing. In 2017 TEEF/RS) International Conference on Intelligent Robots and Systems (IROS), pages 3948-3955. IEEE, 2017. 1
[18] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual lcarning for image recognition,
2015. 6
[19] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask R-CNN. In ICCV, pages 2961-2969.
2017. 1, 2, 4.
[20] Judy Hoffman, Eric Tzeng, Tacsung Park, Jun- Yan Zhu, Phillip Isola, Kate Sacnko, Alexci Efros, and Trevor Darrell. Cycada: Cycle-consistent adversarial domain adaptation. In International conference on machine learning, pages 1989-1998. PMLR, 2018. 3
[21] Berthold Horn. Robot vision. MIT press, 1986.
[22] Berthold KP Horn and Brian G Schunck. Determining optical flow. Artificial intelligence, 17(1-3)1185-003
1981. 7.9 9
[23] Eric Jang, Coline Devin, Vincent Vanhoucke, and Sergey Levine. Grasp2vec: Learning object representa- tions from self-supervised grasping. arXiv preprint arXiv: 1.06964, 2018. 3
[24] Dinesh Jayaraman and Kristen Grauman. Learning image representations tied ego-motion. In Proceed- ings of the IEEE International Conference on Computer Vision, pages 1413-1421, 2015. 2
[25] Dinesh Jayaraman and Kristen Grauman. Look-ahead before you leap: end-to end-to-end active recognition by forecasting the effect of motion. In European Conference on Computer Vision, pages 489 505. Springer. 2016. 3
11

---

[26] Dinesh Jayaraman and Kristen Grauman. Learning to look around: Intelligently exploring unseen environments for unknown tasks. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pages 7238-1247,10 2018.
[27]
Angjoo Kanazawa, Jason Y. Zhang. Panna Felsen, and Jitendra Malik. Learning 3d human dynamics from video. In Computer Vision and Pattern Regognition (CVPR), 2019. 3
[28] Ksenia Konyushkova, Raphael Sznitman, and Pascal Fua. Learning active learning from data, arXiv preprint /r03v: 2017. 3
[29] Ilya Kostrikov. Pytorch implementations reinforcement learning algorithms. https:// com/ kkostrikov/pytordde
aappo--ccr--aaaaa 2018.
[30] Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, 2014. 6
[31] Tsung-Ye Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2117-2125, 2017. 6
[32] John McCormac, Ankur Handa, Andrew Davison, and Stefan Leutenegger. Semanticfusion: Dense 3d semantic mapping with convolutional neural networks. In 2017 IEEE International Conference on Robotics and automation (ICRA), pages 4628-4635. IEFF, 2017.
[33] David Nilsson, Aleksis Pirinen, Erik Gärtner, and Cristian Sminchisescu. Embodied visual active learning for semantic segmentation. arXiv preprinl arXiv:2012.09500 2020. 3
[34] Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pages 16 (6,17,22 2017. 5
[35] Deepak Pathak, Yide Shentu, Dian Chen, Pulkit Agrawal. Trevor Darrell, Sergey Levine, and Jitendra Malik. Learning instance segmentation by interaction. In CVPR Workshop on Benchmarks for Deep Learning in Robotic Vision, 2018. 3
[36] Lerrel Pinto and Abhinav Gupta. Supersizing self- supervision: Learning to grasp from 50k tries and 700 robot hours. In 2016 IEEE international conference on robotics and automation (ICRA), pages 1406-3411
IEEE, 2016. 1
[37] Pengzhen Ren, Yun Xiao, Xiaojun Chang, Po- Yao Huang, Zhihui Li, Xiaojiang Chen, and Xin Wang. A survey of deep active learning. arXiv preprint arXiv:2009.00236 2020.
[38]
Chris Rockwell and David F. Fouhey. Full-body awareness from partial observations. In ECCV, 2020. 3
[39] Kate Saenko, Brian Kulis, Mario Fritz, and Trevor Darrell. Adapting visual category models new domains. In European conference on computer vision, pages 213 1--26...666 Springer, 2010. 3
[40] Manolis Savva, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, Vladlen Koltun, Jitendra Malik, et al. Habitat: A platform for embodied ai research. In ICCV, 2019. 6
[41] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint 17XX:v1777.03477 2017. 6
[42] Oran Sener and Silvio Savarese. Active learning for convolutional neural networks: A core-set approach. arXiv preprint arXiv: 1708,70880440 00489, 2017. 3
[43] James A Sethian. A fast marching level set method for monotonically advancing fronts. Proceedings of the National Academy of Sciences, 154))159 1996. 5
[44] Burr Settles. Active learning literature survey. Technical report, University of Aisconsin-Madisee Department of Computer Sciences, 2009.
[45] H Sebastian Seung, Manfred Opper, and Haim Sompolinsky, Query by committee. In Proceedings of the fifth annual workshop on Computational learning theory, pages 287-294, 1992. 3
[46] Dandan Shan, Jiaqi Geng, Michelle Shu, and David Fouhey. Understanding human hands in contact at internet scale. In CVPR, 2020. 3
[47] Fric Treng, Judy Hoffman, Kate Saenko, and Trevor Darrell. Adversarial discriminative domain adaptation. In Proceedings of the IEFF conference on computer vision and pattern recognition, pages 7167-7176,
2017. 3
[48] Kai Wang, Yimin Lin, Luowei Wang, Liming Han, Minjie Hua, Xiang Wang, Shiguo Lian, and Bill Huang.
A unified framework for mutual improvement of slam and semantic segmentation. In 2019 International Conference on Robotics and Automation (ICRA). pages 2224-223. IEEE, 2019. 3
[49] Yuxin Wu, Alexander Kirillov, Francisco Massa, Wan- Yen Lo, and Ross Girshick. Detectron2, https: 2019. 6
/ gi thub. wwtp////cbboeeeeeerr///eettettee
12

---

[50] Fei Xia, Amir R. Zamir. Zhi-Y Maig He, Alexander Sax, Jitendra Malik, and Silvio Savarese. Gibson Env: real-world perception for embodied agents. In Computer Vision and Pattern Recognition (CVPR). 2018 IEEE Conference on. IEEE, 2018. License: attp://svt stanford. edu/ balu/gibar gigggnb////2/ assets/GDS. agr sggeetttt.po pdf. 6
[51] Qizhe Xie, Minh- Thang Luong, Eduard Hovy, and Quoe V Le. Self-training with noisy student improves imagenet classification. Proceedings of IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 10687-10698, 2020.
[52] Zeeki Yalniz, Hervé Jégou, Kan Chen, Manohar Paluri, and Dhruv Mahajan. Billion-scale semi-supervised learning for image classification. arXiv preprint arXiv:/9055 00546, 2019.
9
[53] Brian Yamauchi. A frontier-based approach for autonomous exploration. In cira, volume 97, page 146, 1997. 7,9
[54] Jianwei Yang, Zhile Ren, Mingze Xu, Xinlei Chen, David Crandall, Devi Parikh, and Dhruv Batra. Emhodied amodal recognition: Learning to move to perceive objects. In Proceedings of the IEEE
International Conference on Computer Vision, pages 2040-20555 2019. 3
[55] Serena Yeung. Olga Russakovsky, Greg Mori, Li Fei-Fei. End-to-end learning action detection from frame glimpses in videos. In Proceedings of the IEEE conference on computer vision and puttern recognition, pages 2678 2687, 2016. 3
[56] Donggeun Yoo So Kweon. Learning loss active learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 93-102, 2019. 3
[57] Richard Zhang, Phillip Isola, and Alexei A Efros. Colorful image colorization. In European conference on computer vision, pages 666. Springer, 2016.
[58] Fangwei Zhong, Sheng Wang, Ziqi Zhang, and Yizhou Wang. Detect-slam: Making object detection and slam mutually beneficial. In 2018 IEEE Winter Conference Applications Computer Vision (WACV). pages 1001-1010. IEEE, 2018.
13


Reviews:
[{"review_id": "bAeF3q_4HF", "reviewer": "Reviewer_4ztR", "summary": "This paper proposes an approach to fine-tune and transfer an object detection and segmentation approach (Mask-RCNN) to a set of 3D textured scene models by label propagation and exploration. The object detector is pretrained on an annotated large-scale Internet image dataset (MS Coco). A \"perception\" component maps the class pixel labels of Mask R-CNN into the 3D scene through raycasting in the given map and classifying voxels in a volumetric 3D semantic map. A policy is trained to explore the environment for fine-tuning the Mask-RCNN detector.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- The paper proposes an interesting novel combination of 3D semantic mapping, label propagation and exploration using a goal policy network.\n- The experiments demonstrate the approach on a benchmark dataset and show good results, improving over several state of the art baselines and ablations.\n\nWeaknesses:\n- A voxel-based semantic representation is accumulated by classifying the voxels according to the highest classification score of the pixels striding the voxel. It seems the instance segmentation of Mask R-CNN is not used and individual objects cannot be distinguished. This would make the method fail in scenes where multiple instances of a class (like chairs) are close and would be detected as one instance, since Mask R-CNN would receive wrong instance segmentation labels for training. \n- The method assumes noiseless depth and pose information which is unrealistic. How does the method perform if real depth images and a localization estimator need to be used ?\n- Which optical flow method is used as a baseline? Please compare against a state-of-the-art method like PWC-Net.\n- Please specify the memory and runtime requirements at test time for the Action model.\n- It's not clear what the \"Random Policy\" is: does the agent execute a random move action, or navigate to random goal locations and take images there ? I think the latter would be clearly superior.\n- When does each baseline method stop collecting images? Is the same number of images collected per baseline? Are the comparisons between methods fair ?\n\n== Post-rebuttal rating ==\n\nThe author response well addressed most of my concerns. The paper contains limited but sufficient novelty over previous methods which perform active SLAM or simultaneously train the object detector during SLAM  by presenting an actor component which is trained for semantic exploration. I increase my score to borderline 6 \"marginally above acceptance threshold\".", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Several limitations of the work are well discussed in Sec. 6. See paper weaknesses above for unaddressed shortcomings (instance segmentation not used).\n\n", "review_text": "", "extra_scores": {"ethical_concerns": "none apparent.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "Xbbb302Vhpm", "reviewer": "Reviewer_RRT1", "summary": "The paper introduces “SEAL”, a framework for self-supervised learning of an embodied exploration policy while simultaneously refining the agent’s perception network. The method relies on building a semantic 3d map propagating confident predictions from one viewpoint to another for visual refinement. The method is validated on the Gibson dataset in terms of perceptual accuracy and on the downstream task of object goal navigation.\n", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality:\n\nPaper builds on previous work and efficiently combines previously introduced concepts (intrinsic reward for exploration, self-supervised learning of object detector, training exploration policy using RL) into a holistic framework for self-supervised learning of embodied exploration and perceptual refinement. \nThis area of research is very active and I believe the authors have cited the most relevant work. The most similar work I could find was “Move to See Better: Towards Self-Supervised Amodal Object Detection” which is currently under review and cited by the authors. The methodology of the two papers differs significantly since this paper relies on building a 3d map of the environment and learning an exploration policy.\n\nQuality:\n\nIn general the paper is technically sound and the idea interesting since self-supervised learning would enable life-long learning for robots in the field. The problem however with the paper is that it relies on ground-truth depth and position. An ablation on the effects of using predicted depth and a noisy position sensor would demonstrate the applicability of SEAL for real-world robotics beyond a simulation. If the current method requires ground-truth pose and depth then this limitation should be discussed.\n \nClarity:\n\nThe paper is generally clear and easy to read. However the result section could be improved upon to be easier to read. It took me some time to understand why you had two AP50 scores. Writing that one is for object detection and is for instance segmentation rather than implying it would improve the text.\nI would also clarify early in the text what self-supervision means in your context. I.e. given a supervised pre-trained model (MaskRCNN) you can learn an exploration policy using self-supervision. The perceptual model itself is learned using a mix of supervised and self-supervised training.\n\nSignificance:\n\nThe significance of this paper lies in the novel idea of a self-supervised learning framework for embodied exploration and navigation. As a researcher I find this concept very appealing and the results on the downstream task verifies that it could have practical usefulness. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "See my note regarding the limitations of the method since it relies on ground-truth pose and depth from the simulator.", "review_text": "", "extra_scores": {"ethical_concerns": "I could not identify any major ethical concerns.\n", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "Dk8-Qik9fFv", "reviewer": "Reviewer_HudJ", "summary": "The proposed method aims to improve the performance of a pretrained perception model in an active learning way. The whole training framework consists of the Action phase and Perception phase, which are trained with gainful curiosity reward and pseudo labels projected from that of the 3D Map, separately. The experiments conducted on detection, segmentation, and object goal navigation tasks illuminate that they can achieve a stronger perception which benefits the embodied robotic task (Object Goal Navigation) in static scenes.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength：\n- The motivation is intuitive and reasonable as well as and the results look great.\n- It is well written and easy to understand the design of gainful curiosity reward and the way to get pseudo labels.\n- The weaknesses of proposed methods are discussed and addressed by weak supervision.\n\nWeakness: \n-  It is not a new concept to employ spatial-temporal consistency to generate pseudo labels for fine-tuning the perception module. It would be better to discuss the difference to related work, e.g., [1], [2].\n-  If my understanding serves me right, the framework is carried out by two steps sequentially, instead of an interactive learning style, which seems more reasonable for me.\n-  Missing some details, like the evaluation metric\n-  Some typos or mistakes, like ‘to used to’ in Line160.\n\nDetail comments:\n-  It is not clear whether this method is applicable to all embodied robots. If it can, why not evaluate it on more tasks. If not, can it perform well on some common dataset for the segmentation task? Or does it have its own significant advantages for Object Goal Navigation? If there are some moving objects in the scene, can the method still work well?\n-  Although the proposed method outperforms the baseline methods on some metrics, like AP50, according to the video, it seems that the proposed algorithm tends to cover a larger area (even some wrong areas) compared with Mask RCNN while predicting segmentation. Can you give an explanation? \n-  The Gainful reward designing seems intuitive, do you use the number of voxels in the 3D Map (value> score threshold) directly, without any processing?\n\n\n\nReferences:\n\n[1] Zhong, Fangwei, et al. \"Detect-slam: Making object detection and slam mutually beneficial.\" WACV 2018.\n\n[2] Wang, Kai, et al. \"A unified framework for mutual improvement of slam and semantic segmentation.\" ICRA 2019.\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The weaknesses of proposed methods are discussed and addressed by weak supervision.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8"}, "extra_text": {}}, {"review_id": "68g3B2QAGJ", "reviewer": "Reviewer_9kyX", "summary": "\nThis paper presents a technique for improving a pre-trained object detection model by learning an exploration policy in a 3D environment, automatically labeling the experience collected by this policy, and then using this data to finetune the object detection model. This improves the object detection performance of the pretrained model, and using this finetuned model in an agent leads to higher success in object navigation tasks.\n\nThe exploration policy operates on a 3D voxel-based map representation. Each voxel contains one number per object category (e.g., chair, bed) denoting the confidence that an object of that type is present in that voxel. This representation is built by using a pre-trained MaskRCNN and using depth information to map back pixels in 2D space to 3D voxels. This voxel-level object confidence scores are summed over time to create the 3D (voxel-based) semantic map of the environment. To learn the exploration policy, they use a curiosity signal named Gainful Curiosity. This essentially counts the number of voxels with a confident object prediction. The policy consists of 2 levels with the high level Global policy proposing a new goal every 25 timesteps, and a low level Local policy that uses a path finding algorithm to navigate to the proposed goal. The global policy is trained to maximize the reward obtained from the Gainful Curiosity signal.\n\nThe experience collected by the exploration policy is then used to finetune the object detection model. To do this, they introduce a label propagation technique called 3DLabelProp. This essentially looks at a set of connected voxels with the same label and constructs the 2D bounding boxes/masks by mapping back from 3D map to 2D observations. Then they train the object detection model on observations labeled in this way.\n\nThey evaluate their technique on a simulated house environment and show that they can improve object detection/segmentation performance of a pre-trained MaskRCNN. They also show that this improved object detection model can then be used to achieve higher success in object goal navigation.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nThe presented technique is relatively straightforward and perhaps not very novel. However, the motivation is I think quite solid; it is obvious that the 3D spatio-temporal structure of the world provides valuable information to our perceptual systems and this works shows one simple way how this can be exploited to improve object detection/segmentation. I think overall the paper is technically sound and the evaluations look OK. It is also clearly written and easy to follow. \n\nI think the main issue with the paper is that the presented technique itself is quite simple. This is in itself not necessarily a problem of course. However, if the presented technique is simple, I'd expect much more evidence that even the technique itself is very simple, it has very wide applicability and results in quite significant improvements on a wide variety of environments. As it stands, I think the paper unfortunately fails to show this.\n\nPlease see my other comments below:\n- Does the local policy (fast marching method) know about the 3D layout of the house (e.g., walls, doors etc.)?\n- In the ablation study, what is random policy? Is this just picking random actions? Or picking goals randomly (positions from 3D map)? If it is the former, I think doing the latter (sample goals randomly and navigate to them) would be a better test of how much the gainful curiosity helps.\n- For the object navigation experiment that shows you get higher success with finetuned MaskRCNN, isn't this pretty much expected? I feel like a better comparison could be to train a policy to navigate to objects from RGB-D data (or the 3D semantic map with a standard/pre-trained MaskRCNN). Since your technique uses RGB-D data to improve the perceptual model, it seems fairer to provide the same info to the baseline as well. \n- typo, line 133, \"both the phases\" -> \"both phases\" or \"both of the phases\"", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"ethical_concerns": "No ethical concerns.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a method which, like active vision, combines perception and control in a closed loop to improve perception (and also an agent policy). It leverages data from pre-trained models, which are bootstrapped, and proposes two different settings, generalization vs. specialization, which also allows training on the test environment.

The paper received 4 expert reviews and was initially on the fence. The reviewers initially mainly raised issues on lack of novelty, an requested additional experiments.

The authors provided a response, which was highly appreciated by most reviewers and solved several issues, in particular through a fruitful discussion on novelty. Additional experiments also help the paper. Some reviewers raised the score above borderline, and 3 out of the 4 reviewers were convinced on acceptance.

The AC followed the discussion, and while he does not find the paper particularly well-written, concurs that it has merits and recommends acceptance.

Decision: Accept (Poster)