Leveraging Locality to Boost Sample Efficiency in Robotic Manipulation

Authors: tong  zhang
yingdong  hu
jiacheng  you
yang  gao
Venue: robot-learning_org_CoRL_2024_Conference
Published: 2024-06-07 12:06:07

Abstract: Given the high cost of collecting robotic data in the real world, sample efficiency is a consistently compelling pursuit in robotics. In this paper, we introduce SGRv2, an imitation learning framework that enhances sample efficiency through improved visual and action representations. Central to the design of SGRv2 is the incorporation of a critical inductive bias—$\textit{action locality}$, which posits that robot's actions are predominantly influenced by the target object and its interactions with the local environment. Extensive experiments in both simulated and real-world settings demonstrate that action locality is essential for boosting sample efficiency. SGRv2 excels in RLBench tasks with keyframe control using merely 5 demonstrations and surpasses the RVT baseline in 23 of 26 tasks. Furthermore, when evaluated on ManiSkill2 and MimicGen using dense control, SGRv2's success rate is 2.54 times that of SGR. In real-world environments, with only eight demonstrations, SGRv2 can perform a variety of tasks at a markedly higher success rate compared to baseline models.

Main paper: 
 introduction: 
 The creation of a versatile, general-purpose robot has long captivated the robotics community. Recent advances in imitation learning (IL) [1, 2, 3] have enabled robots to exhibit increasingly complex manipulation skills in unstructured environments. However, prevailing imitation learning techniques frequently require an abundance of high-quality demonstrations, the acquisition of which incurs substantial costs. This contrasts markedly with disciplines such as computer vision (CV) and natural language processing (NLP), wherein vast repositories of internet data are readily available for utilization. In this paper, we investigate methods to boost sample efficiency in robotic manipulation by developing improved visual and action representations.In machine learning, introducing inductive bias is a standard strategy to enhance sample efficiency. For instance, CNNs [4, 5] inherently embed spatial hierarchies and translation equivariance in each layer, while RNNs [6] and LSTMs [7] incorporate temporal dependencies in their architecture. In the realm of robotic manipulation, a critical inductive bias is action locality, which posits that a robot's actions are predominantly determined by the target object and its relationship with the surrounding local environment. However, previous studies on representation learning for robotic manipulation have not effectively leveraged this bias. Typically, these studies [8, 9, 10, 11] aim to develop a global representation that encapsulates the entire scene, which is then directly employed to predict robot actions. These approaches have demonstrated notably low sample efficiency. As depicted in Figure 1 , reducing the number of demonstrations from 100 to 5 leads to a substantial decrease in the performance of previous works such as SGR [10] , which seeks to capture both semantic and geometric information in a global vector. To effectively utilize the inductive bias of action locality, we introduce SGRv2, a systematic framework of visuomotor policy that considers both visual and action representations. As shown in Figure 2 , SGRv2 builds on the foundation of SGR but integrates action locality throughout the entire framework. SGRv2 demonstrates exceptional sample efficiency and consistently outperforms its predecessor across various demonstration sizes, achieving remarkable results with as few as 5 demonstrations, compared to SGR's performance with 100 demonstrations. The key algorithmic designs that lead to this achievement include: (i) an encoder-decoder architecture for extracting point-wise features, (ii) a strategy for predicting the relative target position to ensure translation equivariance, (iii) the application of point-wise weights to highlight critical local regions, and (iv) dense supervision to enhance learning efficiency.We conduct an extensive evaluation of SGRv2 through behavior cloning across three benchmarks: RLBench [12] , where keyframe control is utilized, and ManiSkill2 [13] and MimicGen [14] , where dense control is applied (refer to Section 3.1 for an discussion on keyframe versus dense control). SGRv2 significantly surpasses both SGR [10] and PointNeXt [15] across these benchmarks and consistently outperforms baselines, including R3M [9] , PerAct [16] , and RVT [17] on RLBench. To confirm the necessity of our locality design, we conduct a series of ablation studies. Additionally, real-world experiments with Franka Emika Panda robot demonstrate SGRv2's capability to complete complex long-horizon tasks across 10 sub-tasks, validating its effectiveness. Further experiments on real-world generalization underscore SGRv2's remarkable ability to generalize. 
related work: 
 Semantic Representation Learning for Robotics. Numerous studies have focused on learning visual representations from in-domain data, tailored specifically to the relevant environment and task [18, 19, 20, 21, 22, 23, 24] . However, the efficacy of these methods is limited by the availability of robot data. Consequently, various efforts have been made to pre-train on large-scale out-ofdomain data for visuo-motor control [25, 26, 27, 8, 28, 29, 30, 31, 32] . Notably, R3M [9] has demonstrated that models pre-trained on human video data can serve as a frozen perception module, facilitating downstream policy learning. Nonetheless, these approaches predominantly prioritize the pre-training of 2D visual representations, thus overlooking critical 3D geometric information, which is essential for enhancing spatial manipulation skills in robotics.3D Representation Learning for Robotics. Recent works have increasingly explored the 3D representations in robotics. Studies such as C2F-ARM [33] , PerAct [16] , and GNFactor [34] employ voxelized observations to derive representations. However, the creation and reasoning over voxels entail a high computational burden. In contrast, RVT [17] and some researches [35, 36, 37] leverage projection techniques to generate multi-view images, extracting representation in 2D spaces and thereby reducing computational demands. Nevertheless, these methods do not incorporate 3D geometry in the process of representation extraction, consequently limiting their capacity for 3D spatial reasoning. Point-based models, such as PointNet++ [38] and PointNeXt [15] , efficiently conserve computational resources while directly processing 3D information. These models serve as the foundation for numerous robotics studies [39, 40, 41, 42, 43, 44, 45, 46, 47] . Specifically, SGR [10] utilizes point-based models to extract 3D geometric information and employs 2D foundational models for semantic understanding, integrating both to enrich representations for downstream tasks. However, the approach of SGR to extract actions from a global vector does not effectively harness locality information, thereby leading to suboptimal sample efficiency.Incorporating Inductive Biases into Robot Learning. Incorporating inductive biases is an essential strategy for enhancing sample efficiency in neural network designs. Several studies have sought to improve sample efficiency by designing networks that adhere to equivariance properties. For example, Act3D [45] employs a translation-equivariant neural network architecture, while USEEK [48] utilizes an SE(3)-equivariant keypoint detector for one-shot imitation learning. However, these approaches are based on global equivariance [49] , which may not be effective when there are relative movements between the object and the environment, which is prevalent in robot manipulation. To tackle this issue, NDFs [50] and EquivAct [51] segment the manipulated object before utilizing the equivariant models. However, given their reliance on a well-segmented point cloud, these methods are ineffective when the object is required to interact with its surroundings. Some studies integrate locality into their models. L-NDF [52] incorporates locality through voxel partitioning, EDFs [49] and RiEMann [53] achieves both equivariance and locality via local message-passing mechanisms in SE(3)-Transformers [54] . Nonetheless, these methods require some hyperparameters to specify a proper size of receptive field, which limits their flexibility and extensibility. It also incurs a tradeoff between locality and expressiveness. Another line of work, such as Transporter [55] , PerAct [16] , and RVT [17] , integrates locality by modeling action as the maximizer of scores on a predefined grid of locations. However, these designs render a voluminous action representation and suffer from quantization artifacts [56] . In contrast, our approach not only satisfies translation equivariance without relying on highly non-local centroid subtraction [50, 57] but also adaptively determines the local scope required for the task through a learnable weight, without sacrificing expressiveness or introducing a grid. 
method: 
 In this section, we present a detailed description of the methodologies employed in SGRv2. Initially, we present an overview of SGR, keyframe and dense control, along with the problem formulation, as outlined in Section 3.1. Subsequently, we explore how to leverage the inductive bias of locality to enhance sample efficiency in robotic manipulation learning, as discussed in Section 3.2. Finally, we describe the training approaches under different control modes, detailed in Section 3.3.Semantic-Geometric Representation (SGR) [10] . SGR is composed of the semantic branch, geometric branch and fusion network. In the semantic branch, the RGB part of RGB-D images are fed into a CLIP [58] image encoder, and the resulting semantic features are then back-projected onto the 3D points. The geometric branch and fusion network split a PointNeXt encoder into two stages, with semantic features injected at the interface between them. SGR models the action by solely relying on the global features extracted by the PointNeXt [15] encoder, without utilizing locality. Refer to Appendix B.2 for more details.Keyframe and Dense Control. In the field of robotic manipulation, keyframe control [33, 16, 59] and dense control [13, 60] are two prevalent control modes. Keyframe control outputs a few sparse target poses, which are then executed through a motion planner. It exhibits reduced compounding errors [59] , and enhanced suitability for visuomotor tasks that leverage visual priors [33, 16, 17] at the cost of inferior flexibility and expressiveness. Dense control, on the other hand, generates a dense sequence with hundreds of actions to control the robot directly. It is applicable across a wide range of robotic scenarios but faces significant challenges with compounding errors. Given that the two models are complementary, it is highly compelling to construct a framework that supports both simultaneously. However, previous research has predominantly focused on a single control mode [16, 17, 61] . In contrast, our framework can support both modes seamlessly.Problem Formulation. We frame our task as a vision-based robot manipulation problem. At each timestep, the robot receives an observation O comprising single or multi-view RGB-D images {I k } K k=1 and proprioceptive data z. For keyframe control, following the setup in PerAct [16] , an action consists of the position, rotation, gripper open state, and collision indicator: a keyframe = {a pos , a rot , a open , a collide }. For dense actions, as illustrated in ManiSkill2 [13] and robosuite [60] , an action consists of the delta position, delta rotation, and gripper open state [13, 60, 14] : a dense = {a ∆pos , a ∆rot , a open }. We assume we are given N expert demonstration trajectories D = {τ i } N i=1 . Each trajectory τ i is a sequence of observation-action pairs (o 1 , a 1 , . . . , a T -1 , o T ). The robot is then trained using the Behavioral Cloning (BC) algorithm with these demonstrations.To develop a sample-efficient framework for robotic manipulation that is effective in both keyframe and dense control scenarios, we capitalize on the inductive bias that actions exhibit locality properties and build our locality aware action modeling on the top of SGR. In SGRv2, we achieve locality through 4 primary designs: (1) an encoder-decoder architecture, (2) a strategy for predicting pointwise relative position formulation with (3) a learned weight, and (4) a dense supervision strategy. Refer to Figure 2 for an overview of our designs, Table 3 for ablation studies, and Appendix B.1 for architecture details.Encoder-Decoder Architecture. In contrast to the encoder-only architecture used by SGR [10] , we employ the encoder-decoder architecture of PointNeXt [15] , which is a U-Net like architecture that excels in dense prediction tasks (e.g. segmentation). This architecture can yield a feature enriched with both global and local information for each point, namely f i ∈ R C for the i-th point, where C is the dimension of the feature. Note that as designed in PointNeXt [15] , the output features are solely dependent on relative coordinates, ensuring that the point-wise features f i remain invariant to translational transformations of the input coordinates. This point-wise features serve as the cornerstone of our locality aware action modeling.Relative Position Predictions. With the point-wise features, we can predict an action at each point.Our key insight is that the end-effector usually moves towards a target close to a specific object within each execution stage. Thus, it is natural to predict the displacement of the target relative to each point. Driven by this insight, for keyframe, we represent the position component of a keyframe action a pos by p i +∆p(f i ) for the i-th point, where p i and f i are the coordinate and point-wise feature of the i-th point respectively, and ∆p is a Multilayer Perceptrons (MLP). For dense control, we can predict the delta position component a ∆pos of a dense action by modeling its direction (towards a target) and magnitude separately. Concretely, we predict the direction a∆pos ||a∆pos||2 by pi+∆p(fi) ||pi+∆p(fi)||2 and the magnitude ||a ∆pos || 2 by m(f i ), where m is another MLP. In dense control, given that we employ the end-effector coordinate frame for the point cloud input2 , p i + ∆p(f i ) can be interpreted as the target position relative to the end-effector. Consequently, pi+∆p(fi) ||pi+∆p(fi)||2 can represent the direction of movement. The utilization of relative position leverages the locality information and achieves translation equivariance without depending on the extensive non-local centroid subtraction [50, 57] . This greatly aids in enhancing sample efficiency.For other action components, we directly predict rotation by r(f i ), gripper open state by o(f i ) and collision indicator by c(f i ), where r, o, c are MLPs.Weighted Average Actions. After obtaining the point-wise action predictions, we need to integrate these predictions into an aggregated action prediction. We adopt a simple yet effective strategy, namely weighted average. For each component, including position (which is broken down into direction and magnitude in dense control), rotation, gripper open state, and collision indicators, we employ a learned weight w * (f i ) for each point. Here, w * denotes separate MLPs (softmax is applied for normalization) for each component. The motivation behind this design is that only a few regions within the point cloud are crucial for accomplishing the task. For instance, in tasks such as picking up a cube, points located on the cube itself are more informative than those on the surrounding table. By learning these weights, we enable the aggregated prediction to concentrate on the most predictive local regions, thereby enhancing both the overall accuracy and sample efficiency.Dense Supervision. To enhance the learning efficiency of local features, we adopt a dense supervision strategy. This approach integrates both aggregated action predictions and point-wise action predictions into the loss function, expressed as L * = L aggregated * + L points * . To compute L aggregated * and L points * , we adopt the same loss formulation with the same ground-truth labels. Dense supervision provides feedback for all points, enabling models to learn local features more efficiently.Smoothness Regularization for Dense Control. As illustrated in point-wise relative predictions, we predict the direction and the magnitude of the delta position component a ∆pos separately. Thus, the position loss is accordingly decomposed into L dir and L mag , i.e. L pos = L dir + L mag . We predict the direction by pi+∆p(fi) ||pi+∆p(fi)||2 , which poses an underdetermined problem and allows ∆p to output spuriously large values. To mitigate this issue, we incorporate a smoothness regularization lossL reg = 1 N 2 i,j ∥(p i + ∆p(f i )) -(p j + ∆p(f j ))∥ 22 . This loss enforces the consistency between p i + ∆p(f i ) and p j + ∆p(f j ) for any two points i and j.Overview of Losses. In keyframe control, following SGR [10] , position is represented by a continuous 3D vector: a pos ∈ R 3 . For rotations, the ground-truth action is represented as a one-hot vector per rotation axis with R rotation bins: a rot ∈ R (360/R)×3 (R = 5 degrees in our implementation). Open and collide actions are binary one-hot vectors: a open ∈ R 2 , a collide ∈ R 2 . In this way, our loss objective is as follows:EQUATIONwhere L pos is L1 loss, and L rot , L open , and L collide are cross-entropy losses.In dense control, following ManiSkill2 [13] and robosuite [60] , both delta position and delta rotation (in the form of axis-angle coordinates) are represented by continuous 3D vectors:a ∆pos ∈ R 3 , a ∆rot ∈ R 3, while open actions are still binary one-hot vectors: a open ∈ R 2 . Our loss objective is:EQUATIONwhere L 
experiments: 
 Our experiments are designed to answer the following questions: (1) How does SGRv2 perform when locality is incorporated into designs, especially in data-limited scenarios, compared to various 2D and 3D representations? (2) Can SGRv2 consistently demonstrate advantages across different control modes? (3) What are the contributions of the key components of SGRv2's locality design to its overall performance? (4) How does SGRv2 perform in real-robot tasks, and does it possess the ability to generalize in the real world?Environment and Tasks. The simulation experiments are conducted on 3 robot learning benchmarks: RLBench [12] , ManiSkill2 [13] , and MimicGen [14] . RLBench is a large-scale benchmark designed for vision-guided manipulation. Following previous works [33, 16, 17] we use keyframe control on RLBench. ManiSkill2 is a comprehensive benchmark for manipulation skills, enhancing diversity with object-level variations. MimicGen generates large-scale robot learning datasets from human demonstrations. On ManiSkill2 and MimicGen, following prior works [61, 14] , we use dense control. On RLBench, we use 4 RGB-D cameras positioned at the front, left shoulder, right shoulder, and wrist of a Franka Emika Panda, while on ManiSkill2 and MimicGen, we use a frontview and a wrist-view RGB-D camera. We use 26 RLBench tasks with 5 demonstrations per task, and 4 ManiSkill2 tasks and 7 MimicGen tasks with 50 demonstrations per task (except for PickSing-leYCB, where 50 demonstrations per object are used). On MimicGen, we use D 1 initial distribution, which presents a broader and more challenging range. See Appendix A for more details.Evaluation. The evaluation approach is designed to minimize variance in our results. On RLBench, we train an agent for 20,000 iterations and save checkpoints every 800 iterations, while in ManiSkill2 and MimicGen, we train for 100,000 iterations and save checkpoints every 4,000 iterations. Then we evaluate the last 5 checkpoints for 50 episodes and get the average success rates 3 . Finally, we conduct the experiments with 3 seeds and report the average results.Baselines. We compare SGRv2 against the following baselines: (1) R3M [9] is a 2D visual representation designed for robotic manipulation, which is pre-trained on large-scale human video datasets.For fair comparisons, we utilize frozen R3M to process RGB images, employ a separate 2D CNN to process depth images, and subsequently fuse the two resulting features. (2) PointNeXt [15] is an enhanced version of the classic PointNet++ architecture for point cloud processing. We employ the encoder of PointNeXt to obtain the 3D representation. (3) PerAct [16] is a 3D representation that voxelizes the workspace and utilizes a Perceiver Transformer [62] to process voxelized observations. ( 4) SGR [10] is a representation that integrates both high-level 2D semantic understanding and low-level 3D spatial reasoning. ( 5) RVT [17] is a 3D representation that utilizes a multi-view transformer to predict actions and integrates these predictions into a 3D space through back-projection from multiple viewpoints. RLBench Performance. Figure 1 depicts the performance of SGR and SGRv2 across differing number of demonstrations. Initially, as the availability of data decreases, we note a significant decline in SGR's performance compared to SGRv2, highlighting the difficulties in maintaining model performances without the inductive bias towards locality awareness. Additionally, Table 1 provides a comparison of success rates obtained with only 5 demonstrations using various representations. We observe that the absence of 3D geometric information, as demonstrated by R3M using a 2D representation, results in markedly low performance, highlighting the critical role of 3D priors in robotic manipulation under data-constrained scenarios. Finally, SGRv2 is demonstrated to be the superior representation, achieving 53.2% average success rate with merely 5 demonstrations and significantly outperforming the most competitive baseline, RVT, with an average improvement factor of 1.32× and achieving enhanced performance in 23 out of 26 tasks. The results underscore the advantages of incorporating 3D geometry and well-designed locality to enhance sample efficiency.ManiSkill2 and MimicGen Performance. To evaluate the performance of SGRv2 in dense control scenarios, we conduct comparisons with several baselines on ManiSkill2 and MimicGen. Our experimental results, summarized in Table 2 , demonstrate that SGRv2 significantly outperforms the baselines. In particular, thanks to our tailored approach for dense control, SGRv2 exhibits superior performance in tasks where the object's location consistently aligns with the direction of the delta actions, such as Pick Cube and Stack Three. These findings confirm that SGRv2 acts as a sample-efficient, universal representation adept at handling both keyframe and dense control scenarios. See Appendix D for results on different numbers of demonstrations and additional baselines. Ablations. As shown in Table 3 , we conduct ablations to assess the locality design choices of SGRv2.(1) Decoder architecture is the cornerstone of our locality designs. Omitting the decoder from the SGRv2 would prevent the application of other locality designs, forcing us to rely solely on the global representation from the encoder. This would lead to a significant decrease in performance. (2) Predicting absolute positions, results in markedly poorer performance. This underscores that relative position predictions are the key insight of the locality design. (3) Substituting point-wise weights with uniform weights reduces performance, confirming the role of point-wise weighting in focusing on predictive local regions. (4) Eliminating dense supervision leads to a decline in overall performance, illustrating that dense supervision enhances the model's learning efficacy.Emergent Capabilities. In our study, we visualize the point-wise weights of SGRv2 detailed in Section 3.2. As depicted in Figure 3 , we sequentially visualize point clouds with RGB and pointwise weights. Surprisingly, the results consistently demonstrate an alignment between points with higher weights (in red) and the object affordances, which denote the functional areas of objects. This observation highlights the capability of SGRv2 to precisely identify and emphasize critical local regions on objects. Motivated by this, we conduct experiments on visual distractors in Appendix E. To evaluate the effectiveness of SGRv2 in a real-robot setting, we conduct experiments using the Franka Emika Panda across two long-horizon tasks (a total of 10 sub-tasks), and a generalization task. Refer to Appendix C for more details on robot setup, task designs and failure cases discussions.Long-horizon Tasks. We collect 8 demonstrations per task and train multi-task agents. Each subtask is tested across 10 episodes. We present comparative results between PerAct, RVT, and SGRv2 in Figure 4 , where SGRv2 demonstrates a substantial performance advantage. A common issue with PerAct is that it tends to bias towards occupied voxels, frequently causing collisions. For RVT, typical failures stem from large position errors along the camera's viewing direction, potentially caused from the inaccuracies in the virtual images rendered orthogonal to the real camera's perspective. Generalization Task. We assess the generalization capability of SGRv2 by employing 6 cups of different colors. Each scene involves one target cup and two distractor cups of different colors. We collect 5 demonstrations for each of the 4 colors and test the model on both 4 seen and 2 unseen scenarios. Each color is tested across 10 episodes. As indicated in the right table, compared with PerAct and RVT, SGRv2 exhibits the ability to generalize across color variations. The results show SGRv2 without the semantic branch achieves zero performance on unseen colors, underlining the critical role of semantic awareness in enhancing generalization. 
discussion: 
 Limitations. Our models are currently trained using a vanilla BC objective. A promising direction involves integrating the Diffusion Policy [2] , which excels in dealing with multimodal and heterogeneous trajectories, with our locality framework to further enhance performance in real world. Additionally, due to our focus on sample efficiency, the evaluations on generalization are currently insufficient. We are eager to expand this work to include a broader range of generalization aspects, such as object shapes, camera positions, and background textures.We present SGRv2, a systematic framework of visuomotor policy that considers both visual and action representations. Built upon the SGR, SGRv2 integrates action locality across its entire framework. Through comprehensive evaluations across a diverse range of tasks in multiple simulated and real-world environments with limited data availability, SGRv2 exhibits superior performance and outstanding sample efficiency. Our simulation experiments are conducted on 3 robot learning benchmarks: RLBench [12] , Man-iSkill2 [13] , and MimicGen [14] . See Figure 5 for an overview of the simulation tasks. In these simulations, all cameras have a resolution of 128 × 128. In the following, we will provide a detailed examination of tasks from the three benchmarks.We utilize 26 RLBench tasks, including 8 tasks used in SGR [10] and 18 tasks used in PerAct [16] and RVT [17] . For tasks with multiple variations, we use the first variation. In RLBench, we use 5 demonstrations per task, unless specified otherwise. Given that SGR and PerAct provide detailed descriptions of these RLBench tasks, we omit these details here for simplicity.We utilize 4 ManiSkill2 tasks, each described in detail as follows. ( 1 of size H ×W and corresponding depth images of the same size from multiple camera views. Point clouds are generated from these depth images using known camera extrinsics and intrinsics. A crucial aspect is the alignment of the RGB images with the point clouds, ensuring a precise one-to-one correspondence between elements in the two data forms. For keyframe control, the point cloud is represented in the robot's base frame. In contrast, for dense control-inspired by FrameMiner [44] -the point cloud is transformed into the end-effector frame to simplify computation and enhance performance.For keyframe control, the model additionally receives proprioceptive data z, which includes four scalar values: gripper open state, left finger joint position, right finger joint position, and action sequence timestep. In dense control, proprioceptive data is not utilized. Additionally, following SGR [10] , if a task comes with language instruction S, this also forms part of the model's input.Other Details. Following SGR [10] , we use CLIP-ResNet-50 as the image encoder for the semantic branch. For the 3D encoder-decoder, we employ PointNeXt-XL. The output from the encoderdecoder is a point-wise feature, denoted as f raw i ∈ R C for the i-th point, where the feature dimension C is 64. We apply a linear layer followed by a ReLU activation to produce a processed point-wise feature f i , increasing the feature dimension to 256. We then predict the relative position ∆p(f i ), magnitude m(f i ) (for dense control), rotation r(f i ), gripper open state o(f i ), and collision indicator c(f i ) (for keyframe control) of the i-th point, where ∆p, m, r, o, c are 3-layer MLPs. Note that when representing the ground-truth actions as one-hot vectors-such as rotation, gripper open state, and collision indicators in keyframe control-the action predictions correspond to the output probabilities following the softmax layer. Finally, for each action component, we assign a learned weight w * (f i ) to each point, where w * represents separate 3-layer MLPs with softmax normalization across the points dimension.SGRv2 is built upon SGR [10] , which we briefly introduced in Section 3.1. Here, we provide a detailed description of SGR's three components: semantic branch, geometric branch, and fusion network.Semantic Branch. Using a collection of RGB images {I k } K k=1 from K calibrated cameras, they initially apply a frozen pre-trained 2D model G, such as CLIP's visual encoder, to extract multi-view image features {G(I k )} K k=1 . When a language instruction S accompanies a task, they utilize a pretrained language model H, like CLIP's language encoder, to generate the language features H(S). They align these image features G(I k ) with the language features H(S) using a visual grounding module, producing {M k } K k=1 . Subsequently, they rescale the visual or aligned feature maps to the dimensions of the original images through bilinear interpolation and reduce their channels by 1 × 1 convolution, generating a set of features {F k } K k=1 , where each F k ∈ R H×W ×C1 . These high-level semantic features are then back-projected into 3D space to form point-wise features for the point cloud, expressed asF sem ∈ R N ×C1 , where N = K × H × W . Geometric Branch. They construct the initial point cloud coordinates P = {p i } N i=1 ∈ R N ×3and RGB features F c ∈ R N ×3 using multi-view RGB-D images and camera parameters (i.e., camera intrinsics and extrinsics). Optionally, they append a D-dimensional vector, derived from robot proprioceptive data z via a linear layer, to each point feature. They then process the point cloud coordinates P and features F c through a hierarchical PointNeXt encoder, extracting compact geometric coordinates P ′ ∈ R M ×3 and features F ′ c ∈ R M ×C2 (M < N ). Fusion Network. To merge the two complementary branches, they first subsample the point-wise semantic features F sem using the same point subsampling procedure as in the geometric branch, resulting in F ′ sem ∈ R M ×C1 . They then perform a channel-wise concatenation of the semantic and geometric features to form C1+C2) . Finally, the fused features are processed through several set abstraction blocks [38, 15] , enabling a cohesive modeling of the cross-modal interaction between 2D semantics and 3D geometric information.F fuse = Concat(F ′ sem , F ′ c ) ∈ R M ×(Losses. As illustrated in Section 3.3, in keyframe control, our loss objective is as follows:EQUATIONwhere L pos is L1 loss, and L rot , L open , and L collide are cross-entropy losses. In our experiments, we set α 1 = 300 and α 2 = α 3 = α 4 = 1.In dense control, our loss objective is:EQUATIONwhere L dir , L mag and L rot are MSE losses, L open is cross-entropy loss, and L reg is smoothness regularization loss. In our experiments, we set β 1 = 10, β 2 = β 3 = 1 and β 4 = 0.3.(1) Translation and rotation perturbations: in keyframe control, the training samples is augmented with ±0.125 m translation perturbations and ±45 • yaw rotation perturbations. (2) Color drop is to randomly replace colors with zero values. This technique serves as a powerful augmentation for PointNeXt [15] , leading to significant enhancements in the performance of tasks where color information is available. (3) Feature drop: Color drop randomly replaces colors with zero values, which results in both the RGB and semantic features becoming constant.However, there are certain tasks where colors play a crucial role, and disregarding color information in these tasks would make them unsolvable. To address this issue, we propose feature drop. Specifically, this involves randomly replacing the semantic features with zero values, while keeping the RGB values unchanged. (4) Point resampling is a widely used technique in point cloud data processing that adjusts the density of the point cloud. It involves selecting a subset of points from the original dataset to create a new dataset with a modified density. Firstly, we filter out points outside the workspace. Then in keyframe control, we resample 4096 points from the point cloud using farthest point sampling (FPS), while in dense control, we resample 1200 points using the same method.(5) Demo augmentation [33] [16] , used in keyframe control, captures transitions from intermediate points along a trajectory to keyframe states, rather than from the initial state to the keyframe state. This approach significantly increases the volume of the training data.Hyperparameters. The configuration of hyperparameters applied in our studies are shown in Table 4. For each task, the experiments are conducted on a single NVIDIA GeForce RTX 3090 GPU. For our real-robot experiments, we use a Franka Emika Panda manipulator equipped with a parallel gripper. We utilize keyframe control, and the motion planning is executed through MoveIt 4 . Perception is achieved through an Intel RealSense L515 camera, positioned in front of the scene. The camera generates RGB-D images with a resolution of 1280×720. We leverage the realsense-ros 5 to align depth images with color images. The extrinsic calibration between the camera frame and robot base frame is carried out using the MoveIt calibration package.When preprocessing the RGB-D images, we resize the 1280 × 720 images to 256 × 256 using nearest-neighbor interpolation. We choose this interpolation method instead of others, like bilinear interpolation, because the latter can introduce artifacts into the depth map, resulting in a noisy point cloud. Following these steps enables us to process RGB-D images as we do in our simulation experiments. It is essential to adjust the camera's intrinsic parameters appropriately after resizing the images. We train SGRv2 for 40,000 training steps and use the final checkpoint for evaluation. Make Coffee (as shown in Figure 4 Bottom Left) is to make pour-over coffee. This task is composed of 4 sub-tasks, each described in detail as follows: (1) Turn on coffee machine: Press the button on the coffee machine to activate it. Move Color Cup to Target (as shown in Figure 6 ) is to select the target color cup from three cups and move it to the white area. The target color is indicated through language instructions. We have 6 cups of different colors: white, red, yellow, orange, black, and green. Each scenario involves one target cup and two distractor cups of different colors. We collect five demonstrations for each of the first 4 colors and test the model on both 4 seen and 2 unseen scenarios.In the real-robot experiments, the primary failure cases of SGRv2 include: (1) For smaller objects like lollipops, the model sometimes struggles to detect them, especially when they are farther away from the camera. (2) For tasks that are sensitive to grasping position or angle, such as grasping markers or funnels, slight deviations can lead to unsuccessful attempts or cause the object to slip.(3) In keyframe control, where a few sparse target poses are output and then executed through a motion planner, failures can occur due to collisions during the motion planning phase.Our primary objective of Section 4 is to demonstrate the sample efficiency of SGRv2. This is why we initially choose to evaluate our method using only 5% of the data employed in the baselines' original settings. However, to make our experiments more complete, we conduct additional experiments to provide a more comprehensive evaluation on more different numbers of demonstrations and more baselines. Note that all our results are the average success rates of the last 5 checkpoints.Avg. RLBench with 100 Demonstrations. We test SGRv2 and the baseline methods (RVT, SGR, PerAct, and PointNeXt) using 100 demonstrations. Refer to Table 5 for the results.RVT with Varying Demonstrations. We test RVT (the most competitive baseline in RLBench) on 26 RLBench tasks, with demonstration numbers ranging from 100 to 5. Table 6 shows the average results of 26 tasks compared with results of SGRv2.MimicGen with 1000 Demonstrations. We evaluate SGRv2 against SGR, PointNeXt, 2D BC-RNN (used in MimicGen [14] and robomimic [64] ), and 2D BC (used in robomimic [64] ). The latter two baselines are trained and tested using the official codes of MimicGen6 and robomimic 7 , but with different evaluation metrics-we report the average results of the last 5 checkpoints instead of the maximum of 30 checkpoints, as explained in Section 4.1. Table 7 shows the results.More Baselines in MimicGen with 50 Demonstrations. In addition to the baselines included in the Table 2 , we evaluated SGRv2 against the 2D BC-RNN (used in MimicGen [14] and robomimic [64] ), 2D BC (used in robomimic [64] ), and R3M [9] baselines with 50 demonstrations. Table 8 shows the results.MimicGen with 200 Demonstrations. We further compared SGRv2 against 2D BC-RNN (used in MimicGen [14] and robomimic [64] ) and 2D BC (used in robomimic [64] ) using 200 demonstrations. Table 9 shows the results.The results from these extended experiments show that SGRv2 consistently outperforms the baselines across all settings. While it's noteworthy that BC-RNN achieves comparable performance to SGRv2 when trained on 1000 demonstrations, it falls short when the number of demonstrations is reduced to 50 or 200. This highlights the superior sample efficiency of SGRv2. Additionally, we As illustrated in Section 4.2, the predicted per-point weights of SGRv2 effectively focus on locations that align well with object affordances. This raises the question: does this emergent capability make the model robust to distractors in the scene that it has never seen before? In Section 4.3, we present experiments involving distractor cups of different colors; nonetheless, it remains to be seen how SGRv2 performs in the presence of completely unseen objects and whether it will disregard them.To address this question, we conduct additional experiments where we randomly introduce taskirrelevant objects (such as YCB [63] objects, basketballs, etc.) as visual distractors into the RL-Bench environments for 3 tasks (meat off grill, phone on base, and push buttons). Refer to Figure 7 for a visualization. We then test the SGRv2 and RVT models, which were previously trained on data without distractors, in these modified environments. The results, as shown in Table 10, indicate that SGRv2 is more robust to these distractors compared to RVT. This suggests that SGRv2 can effectively focus on relevant areas even in the presence of unseen objects, demonstrating robustness to visual distractors. For our simulation experiments using the SGRv2 on RLBench with 5 demonstrations (mentioned in Table 1 ) and on ManiSkill2 and MimicGen with 50 demonstrations (mentioned in Table 2 ), we employed 3 random seeds to ensure the reliability of our results. In the main body of the paper, we present averaged results for clarity. Here we include both the mean and standard deviation derived from our simulation results. The results for RLBench are shown in Table 11 , and the results for ManiSkill2 and MimicGen are presented in Table 12 .We also report the ablations mentioned in Table 3 for each task in Table 13 .Motivated by FrameMiner[44], in dense control, we transform the point coordinates into the end-effector frame, positioning the end-effector at the origin of coordinates for easier computation and performance benefit.MimicGen[14] reports maximum results across different checkpoints, while we provide average results, offering a more robust and realistic measure of model performance.https://moveit.ros.orghttps://github.com/IntelRealSense/realsense-roshttps://github.com/NVlabs/mimicgenhttps://github.com/ARISE-Initiative/robomimic 
 
 Appendix:  


Decision: The reviewers found some strengths in this submission, but also clearly articulated some questions for the rebuttal phase. 
Thank you for your detailed responses.

Reviews: review: title: title: Review
paper_summary: paper_summary: This paper proposes SGRv2, an imitation learning method that is sample efficient in the number of demonstrations used. The method builds on prior work (SGR), with several improvements added. The method is shown to improve over other baselines when trained on limited quantities of demonstration data, and is also demonstrated on real hardware.
main_review: main_review: **Strengths**

- The paper is well-written and the method description and experiments are easy to follow.
- The proposed changes show a clear advantage over the baseline SGR method
- Real robot experiments are provided.

**Weaknesses**

- The experiment scope is limited and does not paint a complete picture of the performance of the proposed method compared to other baselines. The experiment setup seems carefully designed to make the proposed method outperform other baselines. For example, in RLBench, all methods are evaluated on 5 demonstrations, and in MimicGen and ManiSkill2, methods are evaluated on 50 demos. It would make more sense to evaluate all methods on more settings (e.g. different numbers of demonstrations) to present a more complete understanding of the advantages of the proposed approach.

- In some cases, comparison against more baselines would be highly valuable. For example, the MimicGen paper uses visuomotor policies without depth, and has a complete set of results for using 1000 demonstrations on all tasks. Comparing all methods in that setting would be useful. Comparing similar visuomotor policies against the proposed method on 50 demonstrations would also be useful, to contextualize the performance in this setting. Some of the MimicGen numbers reported in this paper appear low. For example, Threading has 6.7 success on 50 demos, but in the MimicGen paper, Threading has 19.3 on the 10 source human demos using a visuomotor policy. Similarly, Coffee has 27.9 success rate on 50 demos, but in the MimicGen paper, Coffee has 74 on the 10 source human demos. There are other suitable baselines for closed-loop control as well in this setting, such as Diffusion Policy (https://diffusion-policy.cs.columbia.edu/) and 3D Diffusion Policy (https://arxiv.org/abs/2403.03954) (for a comparison against a closed-loop policy that uses depth information).  

- The overall performances reported in the paper are low. This is reasonable given the limited number of demonstrations, but this makes the significance of the presented experiment settings limited, especially since the evaluated benchmarks provide ways to procedurally generate more demonstrations for each task. 

**Additional Comments**

- Typo in Table 2 caption (says "buttom")
- In Figure 2, should add more details on method, e.g. what the semantic branch, geometric branch, etc. entails
questions: questions: See the points raised above.
limitations: limitations: 3
review_summary: review_summary: More thorough evaluations are needed to assess the quality of the proposed method.
title: title: Official Review of Paper 618 by Reviewer AWsF
paper_summary: paper_summary: This paper presents a visuomotor policy that incorporates novel visual and action representations for sample efficient imitation learning. The proposed method, SGRv2, builds upon its predecessor SGR but introduces the use of local per-point features. Instead of using global features to predict actions, SGRv2 uses an encoder-decoder architecture to extract per-point features from point cloud inputs. The method then predicts actions relative the end-effector at each point and aggregates these per-point predictions with a learned a weighted average.
main_review: main_review: Strengths:

* This work tackles the important problem of improving the sample efficiency of point cloud based imitation learning methods.
* The paper is well-written and easy to follow. The figure assists gaining good understanding of the paper.
* Although the idea of leveraging locality via aggregating per-point predictions is not new, the paper shows a novel integration of such idea with prior work and good performance with comprehensive experiments.
* Simulation results are convincing. Ablation results are thoughtfully designed and nicely executed.
* Real robot experiments show that proposed method can achieve good performance even with less than 10 demonstrations.

Weakness and questions:

* What is the training and inference speed of the method?
* What do the $w_*(f_i)$ MLPs take as input? Is there any direct supervision on these weights? How are the per-point predictions aggregated for rotation, gripper open state, and collision indicator?
* Since RVT is the second best performing method in Table 1, it would be nice to include this baseline in ManiSkill2 and MimicGen experiements too unless there is a reason that prevents such a comparison. It would be also nice to add a curve for RVT in the left plot of Figure 1 if possible.
* If the predicted per-point weights of the model can focus on places with that align well with object affordances, is the method robust to distractors in the scene that it has never scene before? In the "Generalization Task" section, the authors showed experiments with distractor cups, but what if there are objects that are completely unseen? Will the method ignore them?
* On the website of the paper, it would be good to add visualizations of demonstrations as well as the distribution of initial poses for objects in these demonstrations. This will facilitate better understanding of the real robot setup.
* It would be great to discuss about failure cases of the proposed method in real robot experiments.

Minor correction:

* Table 2 caption: "buttom" -> "bottom"
questions: questions: See above.
limitations: limitations: 3
review_summary: review_summary: This work presents an interesting method to improve sample efficiency of visuomotor policies for imitation learning. The method is not ground-breaking but are well-designed and accompanied with convincing results. Some additional comparisons and analysis can make this paper stronger.
title: title: Straight-forward method exploiting inductive bias in robotics to improve performance and lower data requirement.
paper_summary: paper_summary: SGRv2 is an imitation learning framework that try to leverage the locality in robot actions to improve sample efficiency. The framework uses dense point-wise feature and predicts actions using the weighted average from such dense features. Another design choice is to encode the action in a relative coordination frame. With these inductive biases SGRv2 achieves higher task success rate using less training data.
main_review: main_review: Pros 
- Very well-written manuscript, easy to follow, nice organization. 
- Using pixel-wise dense prediction has been shown effective in many 3D vision tasks so it’s worth trying that in robotics as well. The emergent property on object affordances looks interesting.
- Experimental result looks strong comparing to the baselines.

Cons
- I suggest giving a clearer definition early in the paper, of what locality and sample efficiency the authors are looking for. E.g. training efficiency/fewer data, locality as in action representation/object affordances.
- Figure 3 is bad visually, could be improved. Is that a chair? It would be helpful to list the task there too.
- In Sec 4.3, SGRv2 w/o sem - if the semantic branch is removed, the input to the network is only the point cloud? If so how does it recognize the color for seen/unseen scenarios, or is it just overfit to the trained action sequence. And if the color image is part of the input, there is no reason that it will generalize to unseen color tasks.
questions: questions: - Does the proposed method improve generalization to unseen object configurations, such as different object position/orientation/view point? For example in RLBench the train/test variance is very small. I imagine the relative encoding and use of 3D/point cloud would help a bit on that.
- For the real setup, long-horizon tasks, are the testing setups identical to the training ones?
- Are the models trained per-task or is there one model for all the tasks, with 5 demonstrations each?
limitations: limitations: 3
review_summary: review_summary: Overall I think it’s a nice paper and recommend accepting. The method used here is quite straight-forward and demonstrates strong empirical result.