SEAL: Self-supervised Embodied Active Learning using Exploration and 3D Consistency

1 Introduction . Even though computer vision started out as a field to aid embodied agents ( robots ) [ 21 ] , in current times it has evolved into Internet computer vision , where the focus is on training models for and from Internet data . Fueled by the successes of supervised learning , today ’ s models can successfully classify , detect and segment out objects in Internet images reasonably well [ 19 ] . This raises a natural question : would we need to restart from scratch and gather similarly large-scale labeled datasets to get computer vision to work for embodied agents , or can we somehow bootstrap off the progress made in Internet computer vision ? Internet data comprises of sparse and unrelated snapshots of the world , carefully chosen by humans . On the one hand , this simplifies the problem as models only need to reason about a small and well-chosen subset of possible views of the world . On the other , this makes learning hard . The dog in image_0032 in the ImageNet dataset , is different from the dog in image_0033 , and there is no way to understand how the dog in image_0032 will look like from another view . In contrast , an active agent , embodied in a 3D environment , experiences views of a 3D consistent world . Spatio-temporal continuity in views of the underlying 3D world can allow better inference at test time , and label efficient learning at train time . These effects are further amplified , as the agent can actively choose the views it experiences to maximize learning . Thus , in contrast to Internet computer vision , embodiment opens up the possibility of deriving supervision from spatio-temporal continuity and interaction . Such self-supervision for physical tasks ( e.g . collisions , graspability ) can be done through the use of appropriate sensors [ 17 , 36 ] . How to do so for semantic tasks , such as segmenting and detecting objects is less clear , and is the focus of our work in this paper . We build off models from Internet computer vision and show how their performance can be improved through self-supervised interaction . ⇤Correspondence : dchaplot @ fb.com 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . We show that a ) this can be done purely via interaction , without needing any additional supervision , b ) improved models exhibit better performance in test environments as is , c ) models can be improved further through small amounts of self-supervised interaction in the test environment . We also show that improved perception can , in turn , improve the performance at interactive tasks . We propose a framework called Self-supervised Embodied Active Learning ( SEAL ) as shown in Figure 1 . It consists of two phases , one for learning Action , and another for learning Perception . During the Action phase , the agent learns a self-supervised exploration policy to gather observations of objects with at least one highly confident viewpoint . We propose an intrinsic motivation reward called Gainful Curiosity to train this policy . In the Perception phase , the learned exploration policy is used to gather a single episode of observations in an environment . We propose a method called 3DLabelProp to obtain labels for these observations in a self-supervised fashion . Both the action and perception phases involve constructing a 3D Semantic Map . The agent observations in an episode are used to construct an episodic 3D semantic map . The map is used to compute the Gainful Curiosity reward during the Action phase . And during the perception phase , the semantic labels in the 3D map are projected on the agent ’ s original observations using 3DLabelProp to generate the supervision for training the embodied perception model . Our experiments demonstrate that the SEAL framework can be used to close the action-perception loop . Perceptual models allow the agent to act in the world and collect data that improve the perception models . Improved perception models can , in turn , improve the agent ’ s policy for interacting with the world . We first use SEAL framework to improve object detection and instance segmentation performance of a pretrained perception model ( Mask RCNN [ 19 ] ) from 34.82/32.54 AP50 scores to 40.02/36.23 AP50 scores by just moving around in training environments , without having access to any additional human annotations . By allowing the agent to explore the test environment for a single episode , we can further improve the performance to 41.23/37.28 AP50 scores . Next , we also show that this improved perception model can be used to improve the performance of an embodied agent at Object Goal Navigation from 54.4 % to 62.7 % success rate . 2 Related Work . In this paper , we propose a technique for self-supervised improvement of perception through active interaction , exploration , and 3D semantic mapping . Many papers tackle related problems and we survey them here . Self-supervised Learning . A number of papers focus on the design of label-free pretext tasks to pre-train visual representations [ 1 , 24 , 57 ] . Most such works focus on learning a good generic feature representation without access to any semantic labels . Our work is also self-supervised , we also don ’ t rely on any external source of supervision during training . However , instead of producing a generic feature representation , we produce improved object detection and segmentation models through self-supervision . Our supervision comes from the 3D self-consistency of a given perception model on different views of the data . Furthermore , we learn to actively seek data to conduct this self-supervision on vs. past work that employs pre-collected datasets . Domain Adaptation . Another related line of work is that of unsupervised domain adaptation [ 20 , 47 , 39 ] , where the focus is to adapt models trained on one domain to work well on another domain without any labels , or any assumptions on the data that is available in the two domains . Our focus here is orthogonal : we specifically focus on opportunities for self-supervision available in the context of embodied agents and learn policies to gather data in the target domain that eases transfer . General domain adaptation techniques could be used on top of our work . Active Perception and Learning . Active interaction with an environment can improve performance at test time by gaining information from better views ( i.e . active perception [ 6 , 2 ] ) , as well as at train time to generate more training data or to mine harder examples for external labeling ( i.e . active learning [ 44 ] ) . Active learning and active perception have largely been thought of as two separate bodies of research . Our proposed approach tackles both these problems , and we highlight key differences from existing research . Work in active learning focuses on the selection of data points from an unlabeled corpus for labeling by an oracle ( e.g . humans ) . Researchers have explored the use of prediction uncertainty [ 45 , 16 ] , coverage [ 42 ] and meta-learning [ 28 , 56 ] to learn such data selection functions . Our work departs from standard active learning in two ways : we do not assume access to a pre-collected corpus of unlabeled data , nor to a labeling oracle . We learn a policy to efficiently and autonomously acquire unlabeled data , and rely on multi-view 3D consistency to derive supervision . This differentiates our work from work in active learning [ 37 , 44 ] , and also recent works that relax one or the other requirements : the need for oracles using spatio-temporal consistency to improve models using pre-collected unlabeled videos [ 38 , 46 , 27 ] , or oracle supervised active learning in embodied environments [ 11 , 33 ] . Similarly , active perception has been studied over the last many years [ 6 , 2 ] . Early methods used information-theoretic measures to determine the next best view [ 12 ] . More recently focus has been on learning a policy in an end-to-end fashion to directly improve metrics of interest [ 25 , 55 , 26 , 3 , 54 ] . There also has been some work to improve segmentation and object detection using Simultaneous Localization and Mapping ( SLAM ) [ 58 , 48 ] . The above works learn active interaction with an environment to improve performance at test time by gaining information from better views . In contrast , our focus is on active learning through interaction , i.e . learning how to explore an environment at train time to automatically generate more training data which is used to improve perception . Our proposed technique can also be used to scale up active perception to effectively detect and segment objects in a large-scale 3D scene at test time ( see the specialization setting in Sec 3 ) . Rather than exploring each object one at a time , we learn to explore the whole 3D scene at once . This improves recognition performance for all objects in the scene in a single episode . Furthermore , unlike many prior methods , our approach is completely self-supervised . Learning through Interaction . Robot learning and reinforcement learning focus on learning to solve interactive tasks directly through hit and trial interaction . However , there is also a small body of work that seeks to improve perception models through unsupervised or self-supervised interaction . Unlike prior work [ 13 , 35 , 23 ] which tackle bottom-up image segmentation or learning object representations in table-top manipulation setting , we learn to improve object detection and instance segmentation models in the context a mobile navigation agent . Parallel work from Fang et al . [ 14 ] also focuses on improving object detection models using active data . Unlike their work , we learn a policy for data collection , and aggregate information using 3D semantic maps . 3D Semantic Mapping . 3D mapping ( reconstruction and localization ) has been well studied , and is a fairly mature sub-field of robotics and computer vision . We refer the readers to Fuentes-Pacheco et al . [ 15 ] for a survey . Researchers have also considered the task of associating semantics with 3D maps [ 32 , 8 ] . We adopt and adapt these ideas to our setting , and focus on how we could use these representations for learning active exploration policies to improve models for embodied perception . 3 Method . Our objective is to train an embodied agent to learn both action and perception by moving around in a physical environment . We assume the agent is given access to a perception ( object detection and instance segmentation ) model , fP , such as a MaskRCNN [ 19 ] pretrained on static Internet data . The agent needs to learn a policy to move in the environment and use the experience to learn embodied perception in a completely self-supervised manner without having access to the ground-truth semantic annotations or map information . We propose a framework called Self-supervised Embodied Active Learning ( SEAL ) to tackle this problem . As shown in Figure 1 , it consists of two phases , one for learning Action , and the other for learning Perception . During the Action phase , the agent learns a self-supervised exploration policy to gather useful observations . In the Perception phase , the learned exploration policy is used to gather a single episode of observations in an environment . The observations are labeled in self-supervised manner using 3D Label Propagation , which is then used for training the embodied perception model . In our framework , both the Action and Perception phases require building a 3D semantic map from a sequence of agent observations . We first describe the 3D Semantic Mapping module ( Sec 3.1 ) and then describe how it is utilized in both the phases to train the active exploration policy ( Sec 3.2 ) and the embodied perception model ( Sec 3.3 ) . Generalization and Specialization . We employ this framework under two settings , Generalization and Specialization . In the Generalization setting , the agent is allowed to train for 10 million frames in the set of training environments and tested directly on a set of unseen test environments . In the Specialization setting , after training , the agent is also allowed to explore each test environment for a single episode of T ( = 300 ) time steps . In both settings , the agent has to learn completely in a self-supervised manner , without having access to the ground-truth semantic annotations or maps in training or test environments . For the Generalization setting , the trained perception model is tested on random images in the unseen test environments . For the Specialization setting , the second phase is repeated for each test environment before testing on the unseen images in the test environment . Agent Specification . We follow the agent specification from Chaplot et al . [ 9 ] . For each time step t , the observation space consists of an RGB observation , It 2 R3⇥WI⇥HI , a depth observation , Dt 2 R3⇥WI⇥HI and a 3-DOF pose sensor xt 2 R3 denoting the x and y coordinates of the agent and the orientation of the agent . The action space consists of 3 discrete actions : move forward ( 25cm ) , turn left ( 30 ) and turn right ( 30 ) . The agent camera height is 88cm .

Summary: This paper proposes an approach to fine-tune and transfer an object detection and segmentation approach (Mask-RCNN) to a set of 3D textured scene models by label propagation and exploration. The object detector is pretrained on an annotated large-scale Internet image dataset (MS Coco). A "perception" component maps the class pixel labels of Mask R-CNN into the 3D scene through raycasting in the given map and classifying voxels in a volumetric 3D semantic map. A policy is trained to explore the environment for fine-tuning the Mask-RCNN detector.