Embodied Contrastive Learning with Geometric Consistency and Behavioral Awareness for Object Navigation

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-13 09:11:42

Abstract: Object Navigation (ObjcetNav), which enables an agent to seek any instance of an object category specified by a semantic label, has shown great advances. However, current agents are built upon occlusion-prone visual observations or compressed 2D semantic maps, which hinder their embodied perception of 3D scene geometry and easily lead to ambiguous object localization and blind exploration. To address these limitations, we present an Embodied Contrastive Learning (ECL) method with Geometric Consistency (GC) and Behavioral Awareness (BA), which motivates agents to actively encode 3D scene layouts and semantic cues. Driven by our embodied exploration strategy, BA is modeled by predicting navigational actions based on multi-frame visual images, as behaviors that cause differences between adjacent visual sensations are crucial for learning correlations among continuous visions. The GC is modeled as the alignment of behavior-aware visual stimulus with 3D semantic shapes by employing unsupervised contrastive learning. The aligned behavior-aware visual features and geometric invariance priors are injected into a modular ObjectNav framework to enhance object recognition and exploration capabilities. As expected, our ECL method performs well on object detection and instance segmentation tasks. Our ObjectNav strategy outperforms state-of-the-art methods on MP3D and Gibson datasets, showing the potential of our ECL in embodied navigation.  The experimental code is available as supplementary material.

Main paper: 
 introduction: 
 Object Navigation (ObjectNav) task [3, 23] requires an agent to navigate through a previously unknown 3D scenario to find an object instance, according to a semantic label. Existing work has made great advances in visual representations [18, 19, 63, 67] , data augmentation techniques [40, 46] , and auxiliary tasks [33, 58] for pre-training. Their core ideas are fully exploiting scene layouts and semantic contexts to enhance agents' object localization or scene exploration capabilities. Some methods [18, 19, 41, 65] speculate on correlations among historical visual features for ObjectNav decision-making by emphasizing spatio-temporal awareness of visual observations. Although promising progress has been made, domestic scenes are characterized by substantial occlusion, which poses challenges for agents to accurately localize object goals and efficiently explore scenarios. Moreover, agents typically establish high-level awareness of objects by moving around and perceiving them from different angles and distances. For instance, learning about basic physical concepts for object localization, such as large and long, requires moving beyond image-based observations.Research in behavioral psychology [42, 51] has shown that many animals maintain spatial representations of their environments while navigating. Inspired by this, some other methods attempt to develop Topological Scene Representations (TSRs) [16, 17, 32, 34, 60, 63] or 2D contextual semantic maps [9, 22, 23, 45, 61] based on visual images to balance exploration and exploitation better, as shown in Fig. 1 (a) . Nodes in TSRs typically consist of abstract visual or object features [16, 34] . Edges in TSRs usually involve discrete semantic or geometric relationships (e.g., the pillows are in the bed, and the mouse is used to operate the computer) [32, 60] . However, the abundant geometric and semantic relations among objects should be a large relational space and thus difficult to model with discrete TSRs exhaustively. The 2D contextual semantic maps somewhat reconstruct the layouts and semantic patterns of the scenarios and can provide agents with compressed materials to formulate the continuous relational space [11] . However, RGB image-based semantic segmentation errors may lead to low-quality and ambiguous 2D semantic maps, which severely impair the ObjectNav performance, please see Section A of the supplementary material for more details.To alleviate the above problems, we propose an Embodied Contrastive Learning (ECL) method with Geometric Consistency (GC) and Behavioral Awareness (BA) to motivate agents to actively explore 3D scene layouts and encode semantic cues. Instead of modeling the correlations among visual features from a spatio-temporal aware perspective, we advocate inferring the relevance among visual features at the root by predicting intermediate actions from consecutive visual frames. We believe that the behaviors that lead to differences between two adjacent observations are crucial for learning the relations between two visions, as shown in Fig. 1 (b) . Our BA modeling is more concise than predicting visual features from action sequences since the observation space consisting of high-dimensional sensor inputs tends to be large and variable, while the action space is small, discrete, and relatively fixed.When searching for a specific object, humans usually take sequential actions to actively transform their Field of View (FoV), locating the target and exploring the scene by aligning their visual senses with 3D space. Inspired by this, we empower agents to reconstruct local 3D spatial structures and semantic patterns during navigation, as shown in Fig. 1 (c ). The 3D local scene priors allow agents to learn rich scene representations by immersively aligning the visual features encoded by the behavior-aware visual encoder with the 3D scene features encoded by the 3D PointCloud (PCL) encoder. As a result, the 2D scene understanding is enhanced by introducing geometric and view-invariant priors into the behavioraware 2D visual features. In a nutshell, the GC is modeled as the alignment of behavior-aware visual perceptions with 3D semantic shapes by employing unsupervised contrastive learning.Notably, to adequately mimic the situational interactions of humans with 3D scenes, the above BA and GC-based contrastive representation learning is performed in an embodied manner. In particular, we propose a curiosity and action-aware exploration policy 𝐸 2 -CL, which continually motivates agents to adopt diverse actions to discover novel visual perceptions. On the one hand, the semantic-rich visual stimulus facilitates agents to carry out more comprehensive and robust 2D-3D scene representation learning. On the other hand, the diverse action-vision data pairs collected online provide rich learning materials and feature bases for BA modeling. With the collection of novel and complicated action-vision pairs, the visual frames-based action prediction will be more challenging. Therefore, the BA modeling will be gradually enhanced in this adversarial learning process.During the experimental phase, we first validate the superiority of our ECL on generic object detection and instance segmentation tasks. Particularly, the visual encoder pre-trained by ECL is further retrained to solve these two tasks. The great performance of our approach on both tasks reflects ECL's expertise in object recognition, which will be further migrated to the ObjectNav task. In addition, the pre-trained visual and PCL encoders are integrated into a modular ObjectNav strategy, which is compared with state-of-the-art (SOTA) ObjectNav methods on Matterport3D (MP3D) [7] and Gibson [54] datasets. Concretely, our method improves the ObjectNav success rate by 1.4% ∼ 6.2% and 0.8% ∼ 3.1% on the two datasets, respectively. Sufficient ablation studies demonstrate the substantial contributions of the individual components in our method. Overall, the contributions of this paper are as follows:(1) An embodied contrastive representation learning method with BA and GC is proposed. The BA modeling helps agents take informed navigational actions based on behavior-aware visual perceptions. The GC modeling infuses agents with 3D geometric invariance priors. (2) A curiosity and action-aware exploration strategy is proposed to support embodied ECL. The diverse action-vision pairs collected online provide rich feature bases for the BA and GC modeling. (3) Sufficient comparative and ablative studies on object detection, instance segmentation, and ObjectNav tasks demonstrate the superiority of our ECL and 𝐸 2 -CL methods. 
related work: 
 (1) Spatio-Temporal Visual Modeling for Visual Navigation. Human beings can naturally navigate in new environments, which requires us to find parallels between the new observations and our past experiences. Inspired by this, visual modeling of spatiotemporal awareness [13, 18, 19, 30, 36, 41, 65, 66] can provide agents with historical contextual information for navigation. Its core concept is to implicitly encode the visual semantic clues, the relative spatial information among objects, and the temporal correlations among multiple visual frames, using recurrent neural networks [18, 19, 41] or Transformers [13, 30, 36, 65, 66] . Some improved methods [41] exploit spatio-temporal attention mechanisms to filter keyframes and intelligently focus on semantic and spatial cues that are most relevant to the navigation goal. One of our main insights is that variations in visual perceptions are the consequence of active navigational behaviors. Therefore, unlike spatio-temporal visual modeling that extrapolates dynamic correlations back from observations, we believe the modeling of behavioral patterns can help characterize those correlations at the source.(2) TSRs and Semantic Maps for Visual Navigation. TSRs have continuously shown improvements in various visual semantic tasks. Visual language navigation [14, 21, 35, 52] typically embeds panoramic visual features into the topological graph's nodes and represents the topological graph's edges as combinations of relative positions and orientations between the nodes. Although the topology retains the partial geometry of the scene, the relationships and dynamic transitions between nodes should be encoded with richer geometric and semantic cues. ObjectNav [18, 19, 64] usually injects object features detected from visual perceptions into the topological graph's nodes. The difference is that the topological graph's edges include not only geometric relative positions but also semantic relationships, e.g., a mouse is used to operate a computer. Nevertheless, some studies of continuous scene representations [11, 20] have shown that such discrete relational modeling is prone to inductive bias in spatial and semantic understanding.Since semantic maps preserve fine-grained scene layouts and semantic patterns, they can alleviate the inductive bias by providing navigation agents with richer scene representations. By projecting high-dimensional features encoded from a neural network to a topdown map, existing works [6, 26] try to generate a deep feature map, which is used for reconstructing scenes, predicting semantic maps, and navigating. The semantic map is a type of occupancy map [44] , which indicates whether a point is occupied, represents the semantic categories of objects, and provides location information for robotic navigation and exploration. By decoupling ObjectNav tasks into object localization and scene exploration subtasks, a series of 2D semantic map-based modular methods [39, 45, 61, 62] have been proposed and achieved promising performance. Although 2D contextual semantic maps somewhat reconstruct the layouts and semantic patterns of the scenarios, they lose the 3D geometric structure that is critically important for embodied navigation.Although existing work achieves impressive performance on visual navigation by introducing 3D semantic maps [8, 62] and bird's-eye views [1, 38] , how to fully mine and exploit 3D geometric features is still a challenging and open topic. To release the agents from the 2D observation space, this paper proposes a novel 2D-3D contrastive representation learning method with geometric awareness by aligning rich visual features with 3D scene priors in an embodied manner.(3) Unimodal and Multimodal Representation Learning for Visual Navigation. Unimodal representation learning has emerged with significant success in visual navigation tasks. OVRL [55] employs the concept of knowledge distillation [5] to learn navigation-friendly visual representations from pure visual images in an offline manner. In contrast, CRL [20] presents an embodied adversarial contrastive learning technique to encourage agents to actively explore novel surroundings for learning robust visual representations online. To alleviate the typically substantial occlusions in visual images, Chen et al. [11] employed offline contrastive learning to extract continuous relations among objects from 2D semantic maps. In terms of TSRs, a novel scene graph contrastive loss [50] is proposed to encourage representations that encode objects' semantics, relationships, and history. Unlike the above methods, one of our main insights is to merge the merits of different modalities to achieve richer and more robust scene representations.Multimodal representation learning gains attraction due to its ability to share modality-specific contexts. Humans naturally build spatially meaningful cognitive maps in their brains during navigation. Inspired by this, 𝐸𝑔𝑜 2 -Map [27] proposes a navigation-specific method for learning visual representations by aligning egocentric views with 2D semantic maps in a cross-modal manner. Although this idea is commendable, ObjectNav agents are born and work in 3D scenes. In this work, we experimentally prove that 3D geometric awareness is crucial for ObjectNav decision-making by comparing our method with 𝐸𝑔𝑜 2 -Map. Inspired by recent work on 2D-3D multimodal representation learning [2, 12, 29, 57] , this paper proposes a GC-based ECL technique that encourages agents to actively exploit 3D geometric and semantic priors. Our inspiration comes from Pri3D [29] , which introduces a contrastive learning method for multi-view RGB frames and 3D scene scans. Unlike Pri3D, our ECL method can organically fuse the geometric features in 3D semantic shapes with behavior-aware visual features. The features from both modalities will be utilized to enhance ObjectNav. In addition, our ECL is conducted online in an embodied fashion, with the aim of active learning through interactions with the scenarios. 
methodology 3.1 problem statement and overview: 
 Problem Statement. In an unknown environment, the ObjectNav task requires the agent to navigate to an instance of the specified target category. As initialization, the agent is located randomly without access to a pre-built environment map and is given a target category 𝑐 𝑡𝑎𝑟𝑔𝑒𝑡 ∈ {1, 2, ..., 𝐶}, where 𝐶 is the number of possible target categories. For each navigation state 𝑠 𝑡 , the agent receives noiseless onboard sensor readings, including egocentric RGB-D images {𝑜 𝑡 , 𝑑 𝑡 } and a 3-DoF pose {𝑥, 𝑦, 𝜃 } (2D position and 1D orientation) relative to the starting of the episode. Then the agent predicts its action 𝑎 𝑡 for movement in a discrete action space, consisting of 𝑀𝑜𝑣𝑒_𝐹𝑜𝑟𝑤𝑎𝑟𝑑, 𝑇𝑢𝑟𝑛_𝐿𝑒 𝑓 𝑡, 𝑇𝑢𝑟𝑛_𝑅𝑖𝑔ℎ𝑡, and 𝑆𝑡𝑜𝑝. An episode is considered successful if the agent executes 𝑆𝑡𝑜𝑝 within 1.0 m of a target object and the object can be viewed from the agent's position. Each episode has a time limit of 500 steps.Overview. Fig. 2 and Fig. 3 provide overviews of the proposed ECL method and our modular ObjectNav strategy, respectively. A curious contrastive reward R 𝐸𝑥𝑝 with action awareness is designed for the embodied exploration in ECL, which motivates agents to actively explore the scene and consistently gather novel visual images ( §3.2). We believe navigation behavior is one of the main factors affecting embodied agents' visual perceptions, the BA is modeled by using continuous visual frames to predict the intermediate actions ( §3.3). Meanwhile, a PCL-based semantic mapping method is employed to project the semantically segmented RGB images to a 3D semantic map based on the depth images, the camera parameters, and the agent's poses. This paper focuses on whether 3D scene priors can enhance visual semantic navigation, thus a contrastive loss L 𝐸𝐶𝐿 is proposed for GC modeling by aligning behavior-aware visual features with corresponding 3D scene priors ( §3.3). Finally, the visual encoder 𝐹 𝜃 and PCL encoder 𝐹 𝜑 pre-trained by ECL are migrated to downstream tasks to boost the performance of object detection, instance segmentation, and ObjectNav ( §3.4).In a modular fashion, we decompose the ObjectNav pipeline into three phases: BA and GC-based scene representation, predictionbased high-level goal selection, and deterministic low-level planning, as shown in Fig. 3 . Following existing modular approaches [9, 23, 45, 61, 62] , our ObjectNav agent uses a top-down 2D semantic map as its internal environmental representation. The ObjectNav task is decoupled into two sub-tasks: object localization and scene exploration.Learning (𝐸 2 -CL)This work advocates that agents learn to build environmental cognition by continuously interacting with their surroundings as humans do. drives agents to consistently discover novel visual perceptions and facilitates the learning of semantically enriched scene representations. Specifically, the exploration policy 𝜋 𝜃 consists of a visual encoder 𝐹 𝜃 and a Multilayer Perceptron (MLP) projection head used for predicting exploration actions, as shown in Fig. 2 (Left). For the RGB visual observations 𝑂 = {𝑜 𝑘 } 𝑘=1,2,...,𝑁 collected by the agent during exploration, different image augmentation methods are applied to each image to obtain pairs of augmented images Ô = { ô1 𝑘 , ô2 𝑘 } 𝑘=1,2,...,𝑁 . 𝑁 denotes the number of collected RGB images. Each augmented image is encoded using 𝐹 𝜃 and followed by L2 normalization, which is formulated as ẑ * 𝑘 = 𝑁𝑜𝑟𝑚𝑎𝑙𝑖𝑧𝑒 (𝐹 𝜃 ( ô * 𝑘 )). To maximize the diversity of visions, the following reward signal is used to train 𝜋 𝜃 by maximizing the similarity between the augmented images in the same pair and minimizing the similarity between the augmented images in different pairs:EQUATIONwhere Ẑ = { ẑ1 𝑘 } 𝑘=1,2,...,𝑁 , 𝜏 is a softmax temperature scaling parameter, and 𝑠𝑖𝑚(•, •) corresponds to the dot product.In practice, we find the agent may only slyly perform steering actions in place to greedily maximize the reward described in Equation (1) . To alleviate this problem, another reward signal is designed to maximize the diversity of actions:EQUATIONwhere Â = { â𝑘 } 𝑘=1,2,...,𝑁 , â𝑘 = 𝑀𝐿𝑃 (𝑎 𝑘 ), and A = {𝑎 𝑘 } 𝑘=1,2,...,𝑁 denotes the actions taken to collect visual observations. Overall, the curiosity and action awareness-driven reward R 𝐸𝑥𝑝 = R 𝑉 + 𝛼R 𝐴 is employed as the total reward signal for our 𝐸 2 -CL. 𝛼 is a weight used to balance the two rewards. By doing so, the agent is motivated to discover diverse RGB images by taking diverse actions. Notably, the exploration policy is trained to maximize the following cumulative reward by utilizing the Proximal Policy Optimization (PPO) [48] :EQUATIONIn particular, 𝜋 𝜃 is trained by optimizing the objective 𝐿(𝜃 ) = E[𝑚𝑖𝑛(𝑐 𝑡 (𝜃 )𝐴 𝑡 , 𝑐𝑙𝑖𝑝 (𝑐 𝑡 (𝜃 ), 1 -𝜖, 1 + 𝜖)𝐴 𝑡 )], where the clip ratio𝑐 𝑡 = 𝜋 𝜃 (𝑎 𝑡 |𝑠 𝑡 ) 𝜋 𝜃 𝑜𝑙𝑑 (𝑎 𝑡 |𝑠 𝑡 )and the advange 𝐴 𝑡 is computed utilizing the value function 𝑉 (𝑠 𝑡 ). During exploration, 𝜋 𝜃 is optimized by using the collected minibatches of data from PPO. Please see Section B of the supplementary material for more details. As shown in Fig. 2 , the RGB-D images {𝑜 𝑡 , 𝑑 𝑡 } and agent's poses 𝑠 𝑡 collected by 𝐸 2 -CL are used for the subsequent online ECL.Behavioral Awareness Modeling. When searching for a specific object (e.g., a key), humans usually actively transform their FoVs or move forward to localize the object instance. Inspired by this, we propose to model the correlations between behaviors and visions (the long-horizon dynamic transitions between visual frames) by predicting action sequences based on successive multi-frame visual perceptions, as shown in Fig. EQUATIONIt is worth noting that the agent's action space is small and discrete while the observation space is relatively large and variable. Therefore, our BA modeling is concise and manageable in complexity compared to the modeling technique of predicting visual observations from navigation actions. Geometric Consistency Modeling. To align the 2D behavioraware visual features with the 3D scene priors, the semantically segmented RGB images are projected into a 3D semantic map in the form of PCL based on the camera's parameters and the corresponding depth images. At time step 𝑡, the past 𝑙 + 1 frames of visual observations {𝑜 𝑖 , 𝑑 𝑖 } 𝑡 𝑖=𝑡 -𝑙 and poses {𝑠 𝑖 } 𝑡 𝑖=𝑡 -𝑙 are utilized to construct a 3D local semantic map 𝑀 𝑡 3𝐷 ← {(𝑃 𝑡 𝑝 , 𝑃 𝑡 𝑠 )} ∈ R 𝑄 𝑡 × (𝐶+3) for the consistency of 2D-3D features. Here, 𝑃 𝑡 𝑝 ∈ R 3 and 𝑃 𝑡 𝑠 ∈ R 𝐶 denote each point's position and semantic category, respectively. 𝐶 and 𝑄 denote the number of semantic categories and the number of PCL in 𝑀 𝑡 3𝐷 , respectively. In practice, the off-the-shelf models [24, 31] are employed to obtain semantic segmentations from {𝑜 𝑖 } 𝑡 𝑖=𝑡 -𝑙 and combine the depths {𝑑 𝑖 } 𝑡 𝑖=𝑡 -𝑙 to generate 3D semantic PCL. However, each semantic point may probabilistically belong to multiple different semantic categories. Therefore, we suggest employing a max-fusion mechanism [62] to merge the temporalvariant semantic PCL to ensure consistent scene semantics.Humans often efficiently localize specific object targets in an embodied manner by actively matching their visual senses with 3D scene structures. Inspired by this, a multimodal contrastive loss L 𝐸𝐶𝐿 is proposed to push the 2D and 3D features describing the same spatial patterns closer to each other while pushing the 2D and 3D features describing different spatial patterns farther away from each other. More specifically, L 𝐸𝐶𝐿 aligns the behavioraware visual features 𝑒 𝑣 with the 3D geometric-aware features 𝑒 𝑔 by constructing a common 2D-3D space for the visual encoder 𝐹 𝜃 and the PCL encoder 𝐹 𝜑 , as shown in Fig. 2 . To this end, the multimodal contrastive learning loss is as follows:EQUATIONEQUATIONL 𝐸𝐶𝐿 can not only transfer geometric information from 3D to behavior-aware 2D features but also transfer semantic details from 2D to 3D features. Overall, the visual encoder 𝐹 𝜃 and the PCL encoder 𝐹 𝜑 are trained by using the loss L 𝑉 = L 𝐶𝐸 + 𝛽L 𝐸𝐶𝐿 and the loss L 𝐸𝐶𝐿 , respectively. 𝛽 is a weight used to balance the two losses. Moreover, 𝐹 𝜃 is also trained by utilizing PPO as described in subsection 3.2.Object Detection and Instance Segmentation. Before transferring the pre-trained visual encoder 𝐹 𝜃 to the ObjectNav task, 𝐹 𝜃 is employed to solve the object detection and instance segmentation tasks to verify its expertise in terms of object recognition. During ECL, the semantically rich and novel visual samples collected by 𝐸 2 -CL are saved for retraining 𝐹 𝜃 . In addition, the semantic labels corresponding to the visual samples are extracted from the simulator and used as supervisory signals for these two tasks. Object Navigation. The pre-trained 𝐹 𝜃 and 𝐹 𝜑 are integrated into the framework illustrated in Fig. 3 to validate their contributions to ObjectNav. Following existing works [45, 61, 62] , the ObjectNav task is decoupled into an object localization sub-task and a scene exploration sub-task, which are trained using PPO. 𝐹 𝜃 and 𝐹 𝜑 are used to extract features from consecutive 𝑙 +1 frames of visual perceptions and the corresponding 3D local semantic maps, respectively. We believe the retrained 𝐹 𝜃 and 𝐹 𝜑 will deliver object-specific semantics and structural cues to the object localization sub-strategy. Moreover, the retrained 𝐹 𝜑 can provide directional guidance and 3D-level semantic relations for the exploration sub-strategy. In our work, 𝐹 𝜃 and 𝐹 𝜑 are implemented as the Mask-RCNN [24] with ResNet-50 [25] backbone and the PointNet [43] , respectively.Following existing works [23, 45, 61, 62] , an additional egocentric 2D semantic map 𝑀 𝑡 2𝐷 ∈ R 𝑈 ×𝑊 ×𝐻 is constructed for the exploration sub-strategy and the deterministic local path planning policy. 𝑊 and 𝐻 denote the height and width of 𝑀 𝑡 2𝐷 , respectively. Each element in 𝑀 𝑡 2𝐷 corresponds to a cell of size 25𝑐𝑚 2 in the physical world. Each pixel on the egocentric top-down map is labeled with the corresponding semantic category, represented with a one-hot vector with 𝑈 = 𝐶 + 2 channels where 𝐶 is the number of object categories, and the extra two channels indicate obstacles and explored regions. An encoder 𝐹 𝜗 consisting of fully convolutional networks is employed to extract the scene layout features in 𝑀 𝑡 2𝐷 . The scene layout features and the 3D geometric cues encoded by 𝐹 𝜑 are fed into the exploration sub-strategy to predict exploration goals 𝑔 𝑡 𝐸𝑥𝑝 . This paper highlights the significant contributions of ECL pretrained encoders to both scene exploration and object recognition. As shown in Fig. 3 , we employed a similar corner-guided exploration sub-strategy as in 3DAware [62] . Please see Section F of the supplementary material for more details. The target goal 𝑔 𝑡 𝑡𝑎𝑟𝑔𝑒𝑡 is the set of elements in the 2D semantic map with semantic 𝑐 𝑡𝑟𝑎𝑔𝑒𝑡 . To be specific, once the probability that part of the point clouds in the current 3D local map belongs to the target category 𝑐 𝑡𝑟𝑎𝑔𝑒𝑡 is greater than the threshold predicted by our object recognition sub-policy, the agent recognizes the object target. Then, this part of the point clouds is projected onto the 2D semantic map to localize the positions of the object target 𝑔 𝑡 𝑡𝑎𝑟𝑔𝑒𝑡 . The goals 𝑔 𝑡 𝑡𝑎𝑟𝑔𝑒𝑡 and 𝑔 𝑡 𝐸𝑥𝑝 from two sub-strategies will be consistently updated during Object-Nav. The Fast Marching Method [49] is used to plan the shortest path from the agent's current location to the goal, which is followed by the agent by taking deterministic actions. 
experiments 4.1 experimental setups: 
 Object Detection and Instance Segmentation. We perform experiments on the Habitat simulator [47] with the Gibson [54] dataset that contains photorealistic 3D reconstructions of realworld environments. Following previous works [8, 9] , we use 25 train / 5 val scenes from the Gibson tiny split for our experiments where semantic annotations are available. The agent's observation space consists of 640 × 480 RGB-D images. The agent's discrete action space consists of Move_Forward (0.25m), Turn_Left (30 • ), and Turn_Right (30 • ). Our agent performs about 819K frames of interactive learning in diverse scenarios to optimize the exploration policy 𝐸 2 -CL, the visual encoder 𝑓 𝜃 , and the PCL encoder 𝑓 𝜑 . Our ECL is carried out on four NVIDIA 3090 GPUs and takes about 72 hours. The pseudo-code, hyperparameters, and training curves of ECL are listed in Section B of the supplemental material. For downstream task retraining, our agent actively acts to collect 10K visual frames in each training scene using 𝐸 2 -CL. That is, a total of 250K visual frames are collected. Meanwhile, the corresponding semantic labels are extracted from the Habitat simulator [47] as supervision signals for retraining object detection and instance segmentation models. Following evaluation setups in previous works [8, 9] , we use six common indoor object categories for our experiments: chair, couch, bed, toilet, TV, and potted plant. Thus, 250K frames of images contain a total of about 316K objects. We consider two evaluation settings: (1) Train Split. 5K images containing 6393 objects are randomly sampled from 25 in-distribution training scenes (200 images per scene). (2) Test Split. 5K images containing 5025 objects are randomly sampled from 5 out-of-distribution test scenes (1000 images per scene).Our method is implemented by using the collected 250K frames to retrain the ResNet50 pre-trained by ECL. To highlight the contributions of our ECL to object-oriented visual representation learning, the following baselines are selected for comparative study: SimCLR [15] , CRL [20] , OVRL [55] , 𝐸𝑔𝑜 2 -MAP † [28] , Pri3D [29] , and MIT [57] . Among them, CRL [20] employs an adversarial contrastive loss for embodied visual representation learning based on only RGB images. 𝐸𝑔𝑜 2 -MAP [28] proposes a multimodal contrastive representation learning method based on visual features and 2D semantic maps. Since we do not have access to the source code of 𝐸𝑔𝑜 2 -MAP, the RGB images and the 2D semantic maps in our method are utilized to replicate the approach as much as possible.The replicated approach is named 𝐸𝑔𝑜 2 -MAP † . To ensure fair comparisons, the RGB images used to pre-train our method are saved for the pre-training of SimCLR [15] and OVRL [55] . Besides the RGB images, additional 2D semantic maps are collected for the pre-training of 𝐸𝑔𝑜 2 -MAP † . Moreover, another two fundamental baselines are set: (1) A raw ResNet50 without pre-training is trained in a supervised manner using the collected 250K frames, which is named 𝐹𝑟𝑜𝑚 𝑆𝑐𝑟𝑎𝑡𝑐ℎ.(2) The collected 250K frames are used to retrain the ImageNet-supervised pre-trained weights, which is named 𝐼𝑚𝑎𝑔𝑒𝑁 𝑒𝑡 𝑆𝑢𝑝𝑒𝑟𝑣𝑖𝑠𝑒𝑑.We report the bounding box and the mask AP scores for Object Detection (ObjDet) and Instance Segmentation (InstSeg) tasks, respectively. AP is the average precision averaged over multiple Intersection over Union (IoU) (10 IoU thresholds of .50:.05:.95) and is the primary challenge metric in the COCO dataset [37] . IOU is defined to be the intersection over union of the predicted and ground-truth bounding box or the segmentation mask.Object Navigation. We perform experiments on the MP3D [7] and Gibson [54] datasets with the Habitat simulator [47] . For Gibson, we use 25 train / 5 val scenes from the Gibson tiny split. Following existing works [62] , we consider 6 goal categories, including chair, couch, potted plant, bed, toilet, and TV. For MP3D, we use the standard split of 61 train / 11 val scenes with the Habitat ObjectNav dataset [47] , which consists of 21 goal categories. In the pre-training phase, the visual encoder 𝐹 𝜃 and the PCL encoder 𝐹 𝜑 are optimized using our ECL method in diverse Gibson and MP3D scenes, respectively. Subsequently, 𝐹 𝜃 and 𝐹 𝜑 are integrated into the framework shown in Fig. 3 for object navigation. Both our ECL and ObjectNav agents have 640 × 480 RGB-D observation spaces for constructing semantic maps. The discrete action space of our ObjectNav agent consists of Move_Forward (0.25m), Turn_Left (30 • ), Turn_Right (30 • ), and Stop. Note that, the RGB-D {𝑜 𝑡 , 𝑑 𝑡 } and pose 𝑠 𝑡 readings are noise-free from simulation. The pre-trained 2D semantic model RedNet [31] and the Mask-RCNN [24] trained with COCO dataset [37] are employed for 2D and 3D semantic mapping on MP3D and Gibson datasets, respectively. For each frame, we randomly sample 512 points for PCL-based 3D semantic construction. That is, the number of points in the 3D local semantic map is 𝑄 𝑡 = 512 × (𝑙 + 1). During training, we sample actions every 25 steps and use the PPO [48] for both object localization and scene exploration sub-policies.To evaluate all methods qualitatively, the following three metrics are reported: (1) Success Rate (SR): percentage of successful episodes, (2) SPL: success weighted by path length, which measures the efficiency of the agent over oracle path length. SPL ranges from 0 to 1 and higher SPL indicates better model performance. (3) DTS: geodesic distance of agent to the object goal at the end of the episode. In addition, we report which methods use external data (Ext. Data) to enhance ObjectNav. Besides using the 𝑅𝑎𝑛𝑑𝑜𝑚 𝑆𝑎𝑚𝑝𝑙𝑒 and classical FBE as non-learning baselines, we consider the following mainstream baselines in the ObjectNav task: (1) End-to-end strategies: DD-PPO [53] , Habitat-Web [46] , Red-Rabbit [58] , TreasureHunt [40] , OVRL [55] , and 𝐸𝑔𝑜 2 -MAP [28] . (2) Modular strategies: FBE [56] , ANS [10] , SemExp [9] , FSE [59] , L2M [22] , PONI [45] , and 3D-Aware [62] .The quantitative comparative results between our method and several baselines on the object recognition tasks are shown in Table 1 . Firstly, as expected, both self-supervised and 𝐼𝑚𝑎𝑔𝑒𝑛𝑒𝑡 𝑆𝑢𝑝𝑒𝑟𝑣𝑖𝑠𝑒𝑑 baselines have significant performance gains relative to 𝐹𝑟𝑜𝑚 𝑆𝑐𝑟𝑎𝑡𝑐ℎ. This phenomenon reflects that different types of pre-training can improve the models' performance on both tasks. Secondly, our method and Pri3D outperform 𝐼𝑚𝑎𝑔𝑒𝑛𝑒𝑡 𝑆𝑢𝑝𝑒𝑟𝑣𝑖𝑠𝑒𝑑 pre-training models on both tasks. These results reflect the potential and superiority of 2D-3D self-supervised contrastive learning, i.e., the 3D scene priors can enrich the 2D visual features. Thirdly, the advantages of embodied contrastive learning over offline self-supervised learning are revealed by comparing CRL with SimCLR (or OVRL). Notably, CRL, SimCLR, and OVRL are retrained using the same 250K visual frames. The difference is that CRL uses an adversarial contrastive loss for interactive visual pretraining, while SimCLR and OVRL are pre-trained offline using saved data.Fourthly, the experimental results of 𝐸𝑔𝑜 2 -Map † indicate that contrastive learning between 2D visual features and 2D semantic maps yields no significant performance improvement. In contrast, the exchange of information between 2D visual features and 3D geometric cues facilitates the accuracy of object recognition tasks. In particular, Pri3D improves the AP metrics by 3.91 (Train Split) Our ECL-enhanced ObjectNav policy is evaluated on the MP3D(val) and Gibson(val) datasets. As shown in Table 2 , our method improves the SR and SPL metrics by 12.1%∼26.8% and 7.5%∼12.9% compared to the learning-based (DD-PPO) and classical frontierbased (FBE) baselines, respectively. In the comparative studies with SOTA methods (OVRL, 𝐸𝑔𝑜 2 -MAP, and 3D-Aware † ), our policy improves the SR and SPL metrics by 1.4% ∼ 6.2% and 1.1% ∼ 7.3% compared to the end-to-end and modular approaches, respectively. Moreover, our scheme achieves the best DTS metric on MP3D(val). Notably, Habitat-Web achieved the best SR metric on MP3D (Val) thanks to the use of a large amount of external data. Nevertheless, our scheme competes strongly with the external data enhanced methods (TreasureHunt, Habitat-Web, and 𝐸𝑔𝑜 2 -MAP).As an end-to-end approach, OVRL uses a knowledge distillationlike self-supervised learning method to pre-train the visual encoder. In contrast, 𝐸𝑔𝑜 2 -MAP further introduces additional contrastive pre-training based on 2D visual features and 2D scene priors. However, OVRL and 𝐸𝑔𝑜 2 -MAP perform less well than our method. The superiority of our approach is partly attributed to modeling BA and GC in an embodied manner, and partly to a modular strategy design based on semantic maps. More significantly, our approach likewise outperforms the modular and semantic map-based Object-Nav schemes (SemExp, L2M, and PONI). In particular, our approach outperforms 3D-Aware † , which also attempts to exploit the 3D scene priors. Such experimental results reflect the contributions of our ECL-based pre-trained point cloud encoder and visual encoder for scenario exploration and object recognition tasks. Moreover, our BA and GC-based embodied pretraining provides new ways for active scene perception and object recognition. As shown in Table 3 , our method's performance on Gibson(val) is similar to that on MP3D(val). These experimental results reflect our scheme works universally on different datasets. Quantitatively, our method improves the SR and SPL metrics by 0.8% ∼ 3.1% and 0.7% ∼ 5.7% compared to the SOTA methods (FSE, PONI, and 3D-Aware † ), respectively. Our method employs an enhanced cornerguided exploration policy, which is distinct from that of SemExp, FSE, and 3D-Aware † . This is one of the reasons why we achieve good ObjectNav performance. Qualitatively, a specific ObjectNav example on the Gibson dataset is shown in Fig. 4 . The agent initially recognizes the couch as a chair due to a semantic segmentation error. Although the segmentation error has been corrected to some extent, point clouds with the couch semantic in the map are still intermingled with point clouds with chair semantics. Luckily, the features provided by our pre-trained visual encoder motivate the agent to make correct recognition and navigate to a chair in another room. Please see Section H of the supplementary material for more ObjectNav examples.In addition, we evaluate our method's generalization ability across datasets to verify whether it can handle unseen scenarios, as detailed in Section E of the supplementary material. The experimental results show that ECL pre-training across datasets likewise enhances the ObjectNav performance. As expected, the behavioraware visual features that fuse 2D-3D cross-modal scene priors can be generalized to novel and unseen scenes.We conduct ablation studies on specific components in the ECL, and the results are shown in Table 4 . Specifically, we employ combinations of different components to implement ECL and evaluate the pre-trained models on two object recognition tasks. We first ablate BA L 𝐶𝐸 and GC L 𝐸𝐶𝐿 while retaining the exploration policy 𝐸 2 -CL (R 𝑉 and R 𝐴 ) (line 1 ∼ line 3). The results reflect that both BA and GC enhance our method's performance. The best results are achieved when both are used at the same time. In addition, we retain BA and GC and ablate the different components of 𝐸 2 -CL. Line 5 of Table 4 indicates that a randomized wandering exploration method is used. The results show that both R 𝑉 and R 𝐴 enhance our method. Notably, Line 3 of Table 4 shows that better AP metrics can also be achieved by using only L 𝐶𝐸 (without L 𝐸𝐶𝐿 ), which suggests that action awareness facilitates the agent's active movement and perception in the scenarios. The comparison with the randomized wandering exploration also demonstrates the superiority of our 𝐸 2 -CL. Please see Section C of the supplementary material for more comparative studies on exploration strategies. As shown in Table 5 , the contributions of 𝑓 𝜃 and 𝑓 𝜑 are ablated by adopting or not adopting the pre-trained models to initialize the ObjectNav policy. The results show the enhanced ObjectNav performance by using our BA and GC-based 2D-3D multimodal ECL. Both pre-trained 𝑓 𝜃 and 𝑓 𝜑 can boost the SR and SPL metrics. Additional parametric studies and computational cost evaluation are available in Section D and Section G of the supplemental material, respectively. 
conclusion: 
 This paper focuses on ObjectNav and proposes a novel embodied contrastive representation learning method based on BA and GC, named ECL. In addition, we equip ECL with an exploration strategy 𝐸 2 -CL based on a curiosity and action awareness-driven contrastive reward. Unlike previous self-supervised learning methods based on RGB images, our approach performs an organic cross-modal fusion of semantically rich 2D visual patterns and 3D geometric structural features. In addition, our embodied BA and GC modeling outperforms existing purely vision-based self-supervised learning approaches and offline 2D-3D contrastive representation learning techniques on object recognition tasks. By integrating the pretrained point cloud encoder and visual encoder into a modular ObjectNav framework, our policy achieves the best performance on the MP3D and Gibson datasets. These improvements are attributed to the adequate scene representation capability and excellent object recognition potential of our pre-trained models. In particular, the ablation studies also indicate the effective contributions of the individual components in our method. In the future, more efficient embodied learning paradigms and 3D feature extraction methods need to be further investigated and discussed.  
 
 Appendix:  


Decision: This paper proposes a contrastive learning method to address the issue of the heterogeneous gap between 2D and 3D feature in current embodied navigation methods. In addition, two modules (Geometric Consistency and Behavior Awareness) are proposed to alleviate the mentioned issue. The experiments show the effectiveness in performance compared to previous methods.

All three reviewers agree on the acceptance of this paper. Concerns such as innovation and differentiation are addressed by the author's rebuttal. AC agrees with reviewers that the paper should be accepted. Please revise the paper accordingly based on comments (especially comments from reviewer D7ft).

Reviews: review: paper_summary: paper_summary: The authors have studied the problem of object navigation. The goal is to address previous limitations of 3D scene perception. The authors therefore propose an Embodied Contrastive Learning (ECL) method with Geometric Consistency (GC) and Behavioral Awareness (BA). The proposed ObjectNav strategy outperforms previous state-of-the-art methods on MP3D and Gibson datasets. The proposed ECL method also works well on the object detection and instance segmentation tasks.
strength_weakness: strength_weakness: 1. The paper is well organised.

2. Extensive experiments and ablation studies have demonstrated the effectiveness of the proposed ECL and GC.
limitations: limitations: Comparison with previous works. There have been lots of works on object navigation. The authors have demonstrated the difference of their work with previous ones in related work. But it would be helpful if there could be a table to summarise the difference more clearly. I may consider to raise my score if my concerns are properly addressed.
paper_summary: paper_summary: The paper addresses limitations in Object Navigation (ObjectNav) by introducing an Embodied Contrastive Learning (ECL) method with Geometric Consistency (GC) and Behavioral Awareness (BA). Current ObjectNav agents struggle with occlusion-prone visual observations or compressed 2D semantic maps, leading to ambiguous object localization and blind exploration. To overcome these challenges, the ECL method motivates agents to actively encode 3D scene layouts and semantic cues. Behavioral Awareness (BA) is modeled by predicting navigational actions based on multi-frame visual images, emphasizing behaviors that cause differences between adjacent visual sensations to learn correlations among continuous visions.

Geometric Consistency (GC) is achieved by aligning behavior-aware visual stimuli with 3D semantic shapes using unsupervised contrastive learning. This alignment, along with geometric invariance priors, enhances the agent's ability to recognize and explore objects. These features are integrated into a modular ObjectNav framework, significantly improving object recognition and exploration capabilities. The ECL method demonstrates superior performance in object detection and instance segmentation tasks and outperforms state-of-the-art methods on the MP3D and Gibson datasets, showcasing its potential in advancing embodied navigation.
strength_weakness: strength_weakness: 1. This paper is well-written, and the figures are clear to show the methods;
2. The proposed method shows great improvments and the experiment is well-supported.
3. The method is meaningful, and can be useful for the future paper.
limitations: limitations: The novelty seems a little limited, because contrastive learning method is common used in the object navigation task, I want the authors comment on that.
paper_summary: paper_summary: This paper introduces the problem of the perception of 2D semantic graphs hindering 3D scene perception and object localization in ObjcetNav.To address this, this paper proposes an Embodied Contrastive Learning (ECL) method with Geometric Consistency (GC) and Behavioral Awareness (BA) is proposed. Furthermore, they propose a curious contrastive reward R𝐸𝑥𝑝 with action awareness, which promotes embodied exploration and encourages agents to gather novel visual images. The authors believe that predicting navigational actions based on multi-frame visual images is crucial for learning correlations among continuous visions. They propose GC to align behavior-aware visual stimuli with 3D semantic shapes through unsupervised contrastive learning. A large number of experimental results prove the effectiveness of the method proposed in the paper.
strength_weakness: strength_weakness: The paper introduces several novel modules:
Geometric Consistency (GC)  aligns 2D observational perception with 3D scene features.
Behavioral Awareness (BA) utilizes continuous visual frames to predict intermediate actions.
A curiosity and action-aware exploration policy 𝐸2 -CL, which continually motivates agents to adopt diverse actions to discover novel visual perceptions.
The approach of aligning 2D observations to 3D features is novel and represents a significant advancement in the field of embodied navigation and perception.
The paper demonstrates thorough experimental work, with results being comprehensive and well-presented.
limitations: limitations: 1. Innovation and Differentiation:
While the alignment of 2D and 3D features indeed presents novelty, the modules for action prediction and curiosity-driven exploration are commonly found in reinforcement learning algorithms. 
It is essential to emphasize the innovative aspects of the two proposed modules and highlight the distinctions from traditional action prediction and exploration problems in reinforcement learning.

2. Engineering Complexity and Accessibility:
This work leans more towards being an engineering project rather than a research endeavor.
The paper introduces numerous modules, modeling additional information such as semantic map and 3D point cloud data into the visual navigation task. While this engineering effort is commendable, the extensive complexity of the work may limit its accessibility to the research community.

Please elaborate on the contributions of this work to the core research problems in visual navigation, as well as its contributions to the visual navigation community.