SEAL: Self-supervised Embodied Active Learning using Exploration and 3D Consistency

1 Introduction . Even though computer vision started out as a field to aid embodied agents ( robots ) [ 21 ] , in current times it has evolved into Internet computer vision , where the focus is on training models for and from Internet data . Fueled by the successes of supervised learning , today ’ s models can successfully classify , detect and segment out objects in Internet images reasonably well [ 19 ] . This raises a natural question : would we need to restart from scratch and gather similarly large-scale labeled datasets to get computer vision to work for embodied agents , or can we somehow bootstrap off the progress made in Internet computer vision ? Internet data comprises of sparse and unrelated snapshots of the world , carefully chosen by humans . On the one hand , this simplifies the problem as models only need to reason about a small and well-chosen subset of possible views of the world . On the other , this makes learning hard . The dog in image_0032 in the ImageNet dataset , is different from the dog in image_0033 , and there is no way to understand how the dog in image_0032 will look like from another view . In contrast , an active agent , embodied in a 3D environment , experiences views of a 3D consistent world . Spatio-temporal continuity in views of the underlying 3D world can allow better inference at test time , and label efficient learning at train time . These effects are further amplified , as the agent can actively choose the views it experiences to maximize learning . Thus , in contrast to Internet computer vision , embodiment opens up the possibility of deriving supervision from spatio-temporal continuity and interaction . Such self-supervision for physical tasks ( e.g . collisions , graspability ) can be done through the use of appropriate sensors [ 17 , 36 ] . How to do so for semantic tasks , such as segmenting and detecting objects is less clear , and is the focus of our work in this paper . We build off models from Internet computer vision and show how their performance can be improved through self-supervised interaction . ⇤Correspondence : dchaplot @ fb.com 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . We show that a ) this can be done purely via interaction , without needing any additional supervision , b ) improved models exhibit better performance in test environments as is , c ) models can be improved further through small amounts of self-supervised interaction in the test environment . We also show that improved perception can , in turn , improve the performance at interactive tasks . We propose a framework called Self-supervised Embodied Active Learning ( SEAL ) as shown in Figure 1 . It consists of two phases , one for learning Action , and another for learning Perception . During the Action phase , the agent learns a self-supervised exploration policy to gather observations of objects with at least one highly confident viewpoint . We propose an intrinsic motivation reward called Gainful Curiosity to train this policy . In the Perception phase , the learned exploration policy is used to gather a single episode of observations in an environment . We propose a method called 3DLabelProp to obtain labels for these observations in a self-supervised fashion . Both the action and perception phases involve constructing a 3D Semantic Map . The agent observations in an episode are used to construct an episodic 3D semantic map . The map is used to compute the Gainful Curiosity reward during the Action phase . And during the perception phase , the semantic labels in the 3D map are projected on the agent ’ s original observations using 3DLabelProp to generate the supervision for training the embodied perception model . Our experiments demonstrate that the SEAL framework can be used to close the action-perception loop . Perceptual models allow the agent to act in the world and collect data that improve the perception models . Improved perception models can , in turn , improve the agent ’ s policy for interacting with the world . We first use SEAL framework to improve object detection and instance segmentation performance of a pretrained perception model ( Mask RCNN [ 19 ] ) from 34.82/32.54 AP50 scores to 40.02/36.23 AP50 scores by just moving around in training environments , without having access to any additional human annotations . By allowing the agent to explore the test environment for a single episode , we can further improve the performance to 41.23/37.28 AP50 scores . Next , we also show that this improved perception model can be used to improve the performance of an embodied agent at Object Goal Navigation from 54.4 % to 62.7 % success rate . 2 Related Work . In this paper , we propose a technique for self-supervised improvement of perception through active interaction , exploration , and 3D semantic mapping . Many papers tackle related problems and we survey them here . Self-supervised Learning . A number of papers focus on the design of label-free pretext tasks to pre-train visual representations [ 1 , 24 , 57 ] . Most such works focus on learning a good generic feature representation without access to any semantic labels . Our work is also self-supervised , we also don ’ t rely on any external source of supervision during training . However , instead of producing a generic feature representation , we produce improved object detection and segmentation models through self-supervision . Our supervision comes from the 3D self-consistency of a given perception model on different views of the data . Furthermore , we learn to actively seek data to conduct this self-supervision on vs. past work that employs pre-collected datasets . Domain Adaptation . Another related line of work is that of unsupervised domain adaptation [ 20 , 47 , 39 ] , where the focus is to adapt models trained on one domain to work well on another domain without any labels , or any assumptions on the data that is available in the two domains . Our focus here is orthogonal : we specifically focus on opportunities for self-supervision available in the context of embodied agents and learn policies to gather data in the target domain that eases transfer . General domain adaptation techniques could be used on top of our work . Active Perception and Learning . Active interaction with an environment can improve performance at test time by gaining information from better views ( i.e . active perception [ 6 , 2 ] ) , as well as at train time to generate more training data or to mine harder examples for external labeling ( i.e . active learning [ 44 ] ) . Active learning and active perception have largely been thought of as two separate bodies of research . Our proposed approach tackles both these problems , and we highlight key differences from existing research . Work in active learning focuses on the selection of data points from an unlabeled corpus for labeling by an oracle ( e.g . humans ) . Researchers have explored the use of prediction uncertainty [ 45 , 16 ] , coverage [ 42 ] and meta-learning [ 28 , 56 ] to learn such data selection functions . Our work departs from standard active learning in two ways : we do not assume access to a pre-collected corpus of unlabeled data , nor to a labeling oracle . We learn a policy to efficiently and autonomously acquire unlabeled data , and rely on multi-view 3D consistency to derive supervision . This differentiates our work from work in active learning [ 37 , 44 ] , and also recent works that relax one or the other requirements : the need for oracles using spatio-temporal consistency to improve models using pre-collected unlabeled videos [ 38 , 46 , 27 ] , or oracle supervised active learning in embodied environments [ 11 , 33 ] . Similarly , active perception has been studied over the last many years [ 6 , 2 ] . Early methods used information-theoretic measures to determine the next best view [ 12 ] . More recently focus has been on learning a policy in an end-to-end fashion to directly improve metrics of interest [ 25 , 55 , 26 , 3 , 54 ] . There also has been some work to improve segmentation and object detection using Simultaneous Localization and Mapping ( SLAM ) [ 58 , 48 ] . The above works learn active interaction with an environment to improve performance at test time by gaining information from better views . In contrast , our focus is on active learning through interaction , i.e . learning how to explore an environment at train time to automatically generate more training data which is used to improve perception . Our proposed technique can also be used to scale up active perception to effectively detect and segment objects in a large-scale 3D scene at test time ( see the specialization setting in Sec 3 ) . Rather than exploring each object one at a time , we learn to explore the whole 3D scene at once . This improves recognition performance for all objects in the scene in a single episode . Furthermore , unlike many prior methods , our approach is completely self-supervised . Learning through Interaction . Robot learning and reinforcement learning focus on learning to solve interactive tasks directly through hit and trial interaction . However , there is also a small body of work that seeks to improve perception models through unsupervised or self-supervised interaction . Unlike prior work [ 13 , 35 , 23 ] which tackle bottom-up image segmentation or learning object representations in table-top manipulation setting , we learn to improve object detection and instance segmentation models in the context a mobile navigation agent . Parallel work from Fang et al . [ 14 ] also focuses on improving object detection models using active data . Unlike their work , we learn a policy for data collection , and aggregate information using 3D semantic maps . 3D Semantic Mapping . 3D mapping ( reconstruction and localization ) has been well studied , and is a fairly mature sub-field of robotics and computer vision . We refer the readers to Fuentes-Pacheco et al . [ 15 ] for a survey . Researchers have also considered the task of associating semantics with 3D maps [ 32 , 8 ] . We adopt and adapt these ideas to our setting , and focus on how we could use these representations for learning active exploration policies to improve models for embodied perception . 3 Method . Our objective is to train an embodied agent to learn both action and perception by moving around in a physical environment . We assume the agent is given access to a perception ( object detection and instance segmentation ) model , fP , such as a MaskRCNN [ 19 ] pretrained on static Internet data . The agent needs to learn a policy to move in the environment and use the experience to learn embodied perception in a completely self-supervised manner without having access to the ground-truth semantic annotations or map information . We propose a framework called Self-supervised Embodied Active Learning ( SEAL ) to tackle this problem . As shown in Figure 1 , it consists of two phases , one for learning Action , and the other for learning Perception . During the Action phase , the agent learns a self-supervised exploration policy to gather useful observations . In the Perception phase , the learned exploration policy is used to gather a single episode of observations in an environment . The observations are labeled in self-supervised manner using 3D Label Propagation , which is then used for training the embodied perception model . In our framework , both the Action and Perception phases require building a 3D semantic map from a sequence of agent observations . We first describe the 3D Semantic Mapping module ( Sec 3.1 ) and then describe how it is utilized in both the phases to train the active exploration policy ( Sec 3.2 ) and the embodied perception model ( Sec 3.3 ) . Generalization and Specialization . We employ this framework under two settings , Generalization and Specialization . In the Generalization setting , the agent is allowed to train for 10 million frames in the set of training environments and tested directly on a set of unseen test environments . In the Specialization setting , after training , the agent is also allowed to explore each test environment for a single episode of T ( = 300 ) time steps . In both settings , the agent has to learn completely in a self-supervised manner , without having access to the ground-truth semantic annotations or maps in training or test environments . For the Generalization setting , the trained perception model is tested on random images in the unseen test environments . For the Specialization setting , the second phase is repeated for each test environment before testing on the unseen images in the test environment . Agent Specification . We follow the agent specification from Chaplot et al . [ 9 ] . For each time step t , the observation space consists of an RGB observation , It 2 R3⇥WI⇥HI , a depth observation , Dt 2 R3⇥WI⇥HI and a 3-DOF pose sensor xt 2 R3 denoting the x and y coordinates of the agent and the orientation of the agent . The action space consists of 3 discrete actions : move forward ( 25cm ) , turn left ( 30 ) and turn right ( 30 ) . The agent camera height is 88cm .

Summary:  This paper presents a technique for improving a pre-trained object detection model by learning an exploration policy in a 3D environment, automatically labeling the experience collected by this policy, and then using this data to finetune the object detection model. This improves the object detection performance of the pretrained model, and using this finetuned model in an agent leads to higher success in object navigation tasks.  The exploration policy operates on a 3D voxel-based map representation. Each voxel contains one number per object category (e.g., chair, bed) denoting the confidence that an object of that type is present in that voxel. This representation is built by using a pre-trained MaskRCNN and using depth information to map back pixels in 2D space to 3D voxels. This voxel-level object confidence scores are summed over time to create the 3D (voxel-based) semantic map of the environment. To learn the exploration policy, they use a curiosity signal named Gainful Curiosity. This essentially counts the number of voxels with a confident object prediction. The policy consists of 2 levels with the high level Global policy proposing a new goal every 25 timesteps, and a low level Local policy that uses a path finding algorithm to navigate to the proposed goal. The global policy is trained to maximize the reward obtained from the Gainful Curiosity signal.  The experience collected by the exploration policy is then used to finetune the object detection model. To do this, they introduce a label propagation technique called 3DLabelProp. This essentially looks at a set of connected voxels with the same label and constructs the 2D bounding boxes/masks by mapping back from 3D map to 2D observations. Then they train the object detection model on observations labeled in this way.  They evaluate their technique on a simulated house environment and show that they can improve object detection/segmentation performance of a pre-trained MaskRCNN. They also show that this improved object detection model can then be used to achieve higher success in object goal navigation.