Leveraging Locality to Boost Sample Efficiency in Robotic Manipulation
Authors: Tong Zhang
Yingdong Hu
Jiacheng You
Yang Gao
Venue: CoRL 2024 (2024)
Keywords: Robotic Manipulation
Sample Efficiency

Abstract: Given the high cost of collecting robotic data in the real world, sample efficiency is a consistently compelling pursuit in robotics. In this paper, we introduce SGRv2, an imitation learning framework that enhances sample efficiency through improved visual and action representations. Central to the design of SGRv2 is the incorporation of a critical inductive bias—$\textit{action locality}$, which posits that robot's actions are predominantly influenced by the target object and its interactions with the local environment. Extensive experiments in both simulated and real-world settings demonstrate that action locality is essential for boosting sample efficiency. SGRv2 excels in RLBench tasks with keyframe control using merely 5 demonstrations and surpasses the RVT baseline in 23 of 26 tasks. Furthermore, when evaluated on ManiSkill2 and MimicGen using dense control, SGRv2's success rate is 2.54 times that of SGR. In real-world environments, with only eight demonstrations, SGRv2 can perform a variety of tasks at a markedly higher success rate compared to baseline models.

Leveraging Locality to Boost Sample Efficiency in Robotic Manipulation
Tong Zhang!.2. Yingdong Hu
Jiacheng You'
Yang 1..3+
"Tsngguua University Shanghai Qi Zhi Institute
3 SSangghai Artificial Intelligence Laboratory (zhangton20, huyd21 yjc23) @mails. tsinghua. edu. en, gaoyangi sinnggiis@mail... tsinghua. edu.
Abstract: Given the high cost of collecting robotic data in the real world, sam- ple efficiency is consistently compelling pursuit in robotics. In this paper, we introduce SGRv2, an imitation learning framework that enhances sample effi- ciency through improved visual and action representations. Central to the de- sign of SGRv2 is the incorporation of a critical inductive bias ciass-ction locality, which posits that robot's actions are predominantly influenced by the target object and its interactions with the local environment. Extensive experiments in both simulated and real-world settings demonstrate that action locality is essential for boosting sample efficiency. SGRv2 excels in RLBench tasks with keyframe con- trol using merely 5 demonstrations and surpasses the RVT baseline in 23 of 26 tasks. Furthermore, when evaluated on ManiSkil12 and MimicGen using dense control, SGRv2's success rate is 2.54 times that SGR. In real-world environ- ments, with only eight demonstrations, SGRv2 can perform a variety of tasks at a markedly higher success rate compared to baseline models. Project website: sgrv2-robot, www.rrooo..ggttii... github.iiii. io.
Keywords: Robotic Manipulation, Sample Efficiency
Introduction
The creation of a versatile, general-purpose robot has long captivated the robotics community. Re- cent advances in imitation learning (IL) [1, 2, 3] have enabled robots exhibit increasingly complex manipulation skills in unstructured environments. However, prevailing imitation learning techniques frequently require an abundance of high-quality demonstrations, the acquisition of which incurs sub- stantial costs. This contrasts markedly with disciplines such as computer vision (CV) and natural language processing (NLP), wherein vast repositories of internet data are readily available for uti- lization. In this paper, we investigate methods to boost sample efficiency in robotic manipulation by developing improved visual and action representations.
In machine learning, introducing inductive bias is a standard strategy to enhance sample efficiency. For instance, CNNs [4, 5] inherently embed spatial hierarchies and translation equivariance in each layer, while RNNs [6] and LSTMs [7] incorporate temporal dependencies in their architecture. In the realm of robotie manipulation, a critical inductive bias is action locality, which posits that robot's actions are predominantly determined by the target object and its relationship with the surrounding local environment. However, previous studies representation learning for robotic manipulation have not effectively leveraged this bias. Typically, these studies [8, 9, 10, 11] aim to develop a global representation that encapsulates the entire scene, which is then directly employed to predict robot actions. These approaches have demonstrated notably low sample efficiency. As depicted in Figure 1, reducing the number demonstrations from 100 to 5 leads to a substantial decrease in the performance of previous works such as SGR [10], which seeks to capture both semantic and geometric information in global vector.
"Corresoonding author
8th Conference Robot Learning (CoRL. 2024), Munich, Germany.

---

RLBench
Maniskill2
MimicGen
RLBench
26 Lasks
35.6
65
60
25.0
Avg. Success Rate (96)
55
10.1%
RBMC
PointNors
22.3
PerAct
Sciv2
Pointhultt
14.9
SGRv2
Polntkaatt
SORV
50-
scn
966
45
SGR
40
35
24.2%
decrease
30
100
50
20
10
Number of Demonstratons
Figure Left: Sample efficiency of SGRv2. We evaluate SGR and SGRv2 on 26 RLBench tasks, with demonstration numbers ranging from 100 5. Results indicate that, owing to the locality of SGRv2, it exhibits exceptional sample efficiency. with its success rate declining by only about 10%. Top Right: Overview of simulation results. We test SGRv2 benchmarks, consistently outperforming the baselines. Bottom Right: Tasks of 3 simulation benchmarks.
To effectively utilize the inductive bias of action locality, we introduce SGRv2, a systematic frame- work of visuomotor policy that considers both visual and action representations. As shown in Fig- ure 2, SGRv2 builds the foundation of SGR but integrates action locality throughout the tire framework. SGRv2 demonstrates exceptional sample efficiency and consistently outperforms its predecessor across various demonstration sizes, achieving remarkable results with as few as demonstrations, compared to SGR's 's performance with 100 demonstrations. The key algorithmic designs that lead to this achievement include: (i) an pecoder-decodee architecture for extracting point-wise features, (ii) strategy for predicting the relative target position to ensure translation cquivariance, (iii) the application of point-wise weights to highlight critical local regions, and (iv) dense supervision enhance learning efficiency.
We conduct an extensive evaluation of SGRv2 through behavior cloning across three benchmarks: RLBench [12], where keyframe control is utilized, and ManiSkil12 [13] and MimicGen [14], where dense control is applied (refer to Section 3.1 for an discussion on keyframe versus dense control). SGRv2 significantly surpasses both SGR [10] and PointNeXt [15] across these benchmarks and consistently outperforms baselines, including R3M [9], PerAct [16], and RVT [17] RLBench. To confirm the necessity of our locality design, conduct series of ablation studies. Additionally, real-world experiments with Franka Emika Panda robot demonstrate SGRv2's capability to complete complex long-horizon tasks across 10 sub-tasks, validating its effectiveness. Further experiments on real-world generalization underscore SGRv2's remarkable ability to generalize.
2 Related Work
Semantic Representation Learning for Robotics. Numerous studies have focused on learning visual representations from in-domain data, tailored specifically to the relevant environment and task [18, 19, 20, 21, 22, 23, 24). However, the efficacy of these methods is limited by the availability of robot data. Consequently, various efforts have been made to pre-train on large-scale out-of- domain data for visuo-motor control [25, 26, 27, 8, 28, 29, 30, 31, 32]. Notably, R3M [9] has demonstrated that models pre-trained on human video data can serve as frozen perception module, facilitating downstream policy learning. Nonetheless, these approaches predominantly prioritize the pre-training of 2D visual representations, thus overlooking critical 3D geometric information, which is essential for enhancing spatial manipulation skills in robotics.
3D Representation Learning for Robotics. Recent works have increasingly explored the 3D rep- resentations in robotics. Studies such as C2F-ARM [33], PerAct [16), and GNFactor [34] employ voxelized observations to derive representations. However, the creation and reasoning over voxels entail high computational burden. In contrast, RVT [17] and some researches [35, 36, 37] lever- age projection techniques to generate multi-view images, extracting representation in 2D spaces

---

and thereby reducing computational demands. Nevertheless, these methods do not mcorporate 3D geometry in the process of representation extraction, consequently limiting their capacity for 31) spatial reasoning. Point-based models, such as PointNet++ [38] and PointNeXt [15], efficiently conserve computational resources while directly processing 3D information. These models serve as the foundation for numerous robotics studies [39, 40, 41, 42, 43, 44, 45, 46, 47J. Specifically, SGR [10] utilizes point-based models to extract 3D geometric information and employs 2D founda- tional models for semantic understanding, integrating both to enrich representations for downstream tasks. However, the approach of SGR to extract actions from a global vector does not effectively harness locality information, thereby leading to suboptimal sample efficiency.
Incorporating Inductive Biases into Robot Learning. Incorporating inductive biases an essen- tial strategy for enhancing sample efficiency in neural network designs. Several studies have sought to improve sample efficiency by designing networks that adhere to equivariance properties. For example, Act3D [45] employs translation-equivarian neural network architecture, while USEEK keypoint detector for one-shot imitation learning. However, these
[48] utilizes an SE(3) approaches are based on global equivariance [49], which may not be effective when there are rela- tive movements between the object and the environment, which is prevalent in robot manipulation. To tackle this issue, NDFs [50] and EquivAct [51] segment the manipulated object before utiliz- ing the equivariant models. However, given their reliance on a well-segmented point cloud, these methods are ineffective when the object is required to interact with its surroundings.
Some studies integrate locality into their models. L-NDF [52] incorporates locality through voxel partitioning, EDFs [49] and RiEMann [53] achieves both equivariance and locality via local messagg--pssing mechanisms in Traas-Trasssssmrrs [54]. Nonetheless, these methods require some hyperparameters to specify a proper size of receptive field, which limits their flexibility and extensibility. It also incurs tradeoff between locality and expressiveness. Another line of work, such Transporter [55], PerAct [16], and RVT [17], integrates locality by modeling action as the maximizer of scores on a predefined grid of locations. However, these designs render voluminous action representation and suffer from quantization artifacts [56]. In contrast, our approach not only satisfies translation equivariance without relying on highly non-local centroid subtraction [50, 57] but also adaptively determines the local scope required for the task through a learnable weight, without sacrificing expressiveness or introducing a grid.
3 Method
In this section, we present a detailed description of the methodologies employed in SGRv2. Initially, we present an overview of SGR, keyframe and dense control, along with the problem formulation, as outlined in Section 3.1. Subsequently, we explore how to leverage the inductive bias of locality to enhance sample efficiency in robotic manipulation learning, as discussed in Section 3.2. Finally, we describe the training approaches under different control modes, detailed in Section 3.3.
3.1 Background
Cemantic-Geometri Representation (SGR) [10). SGR is composed of the semantic branch, geometric branch and fusion network. In the semantic branch, the RGB part of RGB-D images are fed into CLIP [58] image encoder, and the resulting semantic features are then back-projected onto the 3D points. The geometrie branch and fusion network split a PointNeXt encoder into two stages, with semantic features injected at the interface between them. SGR models the action by solely relying the global features extracted by the PointNeXt [15] encoder, without utilizing locality. Refer to Appendix B.2 for more details.
Keyframe and Dense Control. In the field of robotic manipulation, keyframe control [33, 16, 59] and dense control [13, 60) are two prevalent control modes. Keyframe control outputs a few sparse target poses, which are then executed through a motion planner. exhibits reduced compounding
errors [59], and enhanced suitability for visuomotor tasks that leverage visual priors [33, 16, 17] at the cost of inferior flexibility and expressiveness. Dense control, on the other hand, generates
3

---

reletiva prediction
targei posilion
Encoder (SGR)
high weight
low wo git
Semantic
Opos
20 Images
Ftaion Network
Decoder
Arot
Oopen
Geometric
Branch
Feature Map Point-wise Prediction
Figure 2: SGRv2 Architecture. Built upon SGR, SGRv2 integrates locality into its framework through four primary designs: an encoder-decoder architecture point-wise features, strategy for predicting relative target position, a weighted average for focusing on critical local regions, and a dense supervision strategy (not shown in the figure). This illustration specifically represents the water plants task. For simplicity in the visualization, we omit the proprioceptive data that is concatenated with the RGB of the point cloud before being fed into the geometric branch.
dense sequence with hundreds of actions to control the robot directly. It is applicable across wide range of robotic scenarios but faces significant challenges with compounding errors. Given that the two models are complementary, is highly compelling construct a framework that supports both simultaneously. However, previous research has predominantly focused on single control mode [16, 17, 61]. contrast, our framework can support both modes seamlessly.
Problem Formulation. We frame our task as a vision-based robot manipulation problem. At each timestep, the robot receives an observation O comprising single or multi-view RGB-D im- ages { and proprioceptive data 2. For keyframe control, following the setup in PerAct [16], an action consists of the position, rotation, gripper open state, and collision indicator: keyfrume {apos aro, Gopen Acollide ). For dense actions, illustrated in ManiSkil12 [13] and robosuite[60J, an action consists of the delta position, delta rotation, and gripper open state [13, 60, 14]: dense {aspos: G Gopen We assume Each trajectory is a sequence of observation-action pairs (01, .1..... dr -1, or). The robot is then trained using the Behavioral Cloning (BC) algorithm with these demonstrations.
given N expert demonstration trajectories D = {T))
3.2 Locality Aware Action Modeling
To develop sample-efficient framework for robotic manipulation that is effective in both keyframe and dense control scenarios, capitalize on the inductive bias that actions exhibit locality proper- ties and build our locality aware action modeling on the top of SGR. In SGRv2, we achieve locality through 4 primary designs: (1) an encoder-decoder architecture, (2) strategy for predicting point- wise relative position formulation with (3) a learned weight, and (4) a dense supervision strategy. Refer to Figure for an overview our designs, Table for ablation studies, and Appendix B. for architecture details.
Encoder-Decodee Architecture. In contrast to the encoder-only architecture used by SGR [10), we employ the encoder-decoder architecture of PointNeXt [15], which is a U-Net like architecture that excels in dense prediction tasks (e.g. segmentation). This architecture can yield feature enriched with both global and local information for each point, namely fa E RCC for the i-th point, where C the dimension of the feature. Note that as designed in PointNeXt [15), the output features are solely dependent on relative coordinates, ensuring that the point-wise features fi remain invariant translational transformations of the input coordinates. This point-wise features as the cornerstone of our locality aware action modeling.
Relative Position Predictions. With the point-wise features, we can predict an action at each point. Our key insight is that the end-effector usually moves towards a target close to a specific object within each execution stage. Thus, it natural to predict the displacement of the target relative to each point. Driven by this insight, for keyframe, we represent the position component of keyframe action apos by pa Ap(fi) for the i-th point, where p; and f. the coordinate and point-wise feature of the i-th point respectively, and Ap is Multilayer Perceptrons (MLP). For dense control, we can

---

predict the delta position component &pps of a dense action by modeling its direction (towards
"Ape by put Ap(f.) &pp:+l and the magnitude | || by m(f:), where m is another MLP. In dense control, given that we employ the end-effector coordinate frame for the point cloud input", P; Ap(fi) can be interpreted as the p.+Ap(())) target position relative to the end-effector. Consequently, Ayp++l)[JJ]l can represent the direction of movement. The utilization of relative position leverages the locality information and achieves translation equivariance without depending on the extensive non-local centroid subtraction [50, 57]. This greatly aids in enhancing sample efficiency.
target) and magnitude separately. Coneretely, we predict the direction ||алииllz
For other action components, we directly predict rotation by r(fi,, gripper open state by o(f²) and collision indicator by c(fi), where r, o, c are MLPs.
Weighted Average Actions. After obtaining the point-wise action predictions, need to integrate these predictions into aggregated action prediction. We adopt simple yet effective strategy, namely weighted average. For each component, including position (which is broken down into direction and magnitude in dense control), rotation, gripper open state, and collision indicators, we employ learned weight w. (fi) for each point. Here, w, denotes separate MLPs (softmax is applied for normalization) for each component. The motivation behind this design that only a few regions within the point cloud are crucial for accomplishing the task. For instance, in tasks such as picking up a cube, points located on the cube itself are more informative than those on the surrounding table. By learning these weights, enable the aggregated prediction to concentrate on the most predictive local regions, thereby enhancing both the overall accuracy and sample efficiency.
Dense Supervision. To enhance the learning efficiency of local features, we adopt a dense super- vision strategy. This approach integrates both aggregated action predictions and point-wise action predictions into the loss function, expressed as L. Chggregated + C.oints To compute Cageregated and Epoints we adopt the same loss formulation with the same ground-truth labels. Dense supervision provides feedback for all points, enabling models to learn local features more efficiently.
3.3 Training
Smoothness Regularization for Dense Control. As illustrated in point-wise relative predictions, we predict the direction and the magnitude of the delta position component OAApos separately. Thus, the position loss is accordingly decomposed into Lai and Lmag, i.e. Cpos Ldir + Lmag. We predict the direction by [pp+-p((f))l>> which poses an underdetermined problem and allows Ap to output spuriously large values. To mitigate this issue, we incorporate a smoothness regularization loss Creg No Eis (p + Ap(ff)) (p, + Ap(f,)) This loss enforces the consistency between p: + Ap(f.) and Pj + Ap(f,) for any two points and j.
p.+Ap(ff)
Overview of Losses. In keyframe control, following SGR [10], position is represented by a contin- uous 3D vector: apos E T For rotations, the ground-truth action is represented as a one-hot vector per rotation axis with R rotation bins: ano E (360/ R2) (R = 5 degrees in our implementation). R2, Gcallide E R2, In this way, our loss
Open and collide actions are binary one-hot vectors: Copon objective is as follows:
Lieyframe (x1 Cpos + ar2Cm + Or3 CrpCopen Ceallide
where Cpos is LI loss, and Lrots Lopen- and Leollide cross-entropy losses.
In dense control, following ManiSkil12 [13) and robosuite [60], both delta position and delta rotation (in the form of axis-angle coordinates) are represented by continuous 3D vectors: aAppss E IR3 Arro E 3 while open actions are still binary one-hot vectors: Oopen E IR²2. Our loss objective is:
Ldense ( + Emag) + Br + 8yLapen + Ba
where Lair- Lmng and Lior are MSE losses, Lopen is cross-entropy loss, and Lreg is smoothness regu- larization loss. For more training details, refer Appendix B.3.
Mooivated by FrameMiner [44], in dense control, we transform the point coordinates into the end- eefffeecoo frame, positioning the end-effee and-effector the origin of coordinates for easier computation and performance benefit.
5

---

Avg. Success
Open
Water
Tailer
Phone
Take Out
Open Drawer
Slide
Sweep Dusipen
Mear Off
Method
Rank
Microwave
Door
Plants
Seat Up
Base
Books
Umbbrela
Fridge
Block
Grill
R3M
36.4
2.9
15.5
0.0
0.0
0.4
PointNeXi
25.3
3,4
7.1
60.9
5.6
49.9
46.4
57.5
37.5
9.2
21.7
59.5
42.0
59.9
PerAct
22.3
4.1
4.3
59.6
28.5
69.3
0.0
25.1
75.9
3.1
56.4
47.5
2,8
85.9
SGR
23.6
4.1
55.3
24.9
30.7
47.2
29.3
36.3
7.1
31,9
72.0
43.6
52.7
RVT
40.4
2.2
18.3
71.2
34.8
47.6
62.3
46.5
85.3
24,0
75.1
85.1
19.6
90.5
SGRv2 (ours)
53.2
89.6
84.1
81.3
61.5
Tum
Put In
Close
Drag
Stack
Screw
Pur In
Place
Par In
Soit
Posh
Insert
Stack
Place
Method
Tap
Drawer
Stick
Blocks
Rulb
Safe
Cuphoard
Shape
Buttons
Peg
Cups
Cups
R3M
26.1
0.0
0.3
0.0
0.0
0.3
0.4
0.0
0.0
6.8
0.0
0.0
0.0
PointNeXI
48.7
17.1
36.0
18.5
1.9
4.1
12.1
31.5
3,3
0.4
22,0
0.1
4.4
0.4
PerAct
8.0
0.1
0.5
10.3
1.7
4.4
0.9
8.7
0.4
0.4
83.1
1.9
0.1
0.7
14.4
0.9
169
12.0
RVT
38.4
19.6
25.2
45.7
8.8
24.0
30.7
92.7
5.6
1.6
90.4
4.0
3.1
1.2
SGRv2 (ours)
87.9
75.9
94.9
17.5
24.1
55.6
53.1
20.3
93.2
21.3
Table 1: Performance on RLBench with 5 Demonstrations. All numbers represent percentage success rates averaged over 3 seeds. See Appendix F for standard deviation. SGRv2 outperforms the most competitive baseline RVT on 23/26 tasks, with average improvement of 1.32x.
4
Experiments
Our experiments designed answer the following questions: (1) How does SGRv2 perform when locality is incorporated into designs, especially in data-limited scenarios, compared to various 2D and 3D representations?
(2) Can SGRv2 consistently demonstrate advantages across different control modes? (3) What are the contributions of the key components of SGRv2's locality design to its overall performance? (4) How does SGRv2 perform in real-robot tasks, and does it possess the ability to generalize in the real world?
4.1 Simulation Setup
Environment and Tasks. The simulation experiments are conducted on 3 robot learning bench- marks: RLBench [12]. ManiSkil12 [13], and MimicGen [14], RLBench is a large-scale benchmark designed for vision-guided manipulation. Following previous works [33, 16, 17]
keyframe control on RLBench. ManiSkill2 is comprehensive benchmark for manipulation skills, enhancing diversity with object-level variations. MimicGen generates large-scale robot learning datasets from human demonstrations. On ManiSkil12 and MimicGen, following prior works [61, 14], we use dense control. On RLBench, we use 4 RGB-D cameras positioned at the front, left shoulder, right shoulder, and wrist of a Franka Emika Panda, while on ManiSkil12 and MimicGen, we use front- view and a wrist-view RGB-D camera. We use 26 RLBench tasks with 5 demonstrations per task, and ManiSkill2 tasks and MimicGen tasks with 50 demonstrations per task (except for PickSing- ICYCB where 50 demonstrations per object used). On MimicGen, we use D1 initial distribution, which presents a broader and more challenging range. See Appendix A for more details.
The evaluation approach is designed to minimize variance in our results. On RLBench, we train an agent for 20,000 iterations and save checkpoints every 800 iterations, while in ManiSkill2 and MimicGen, we train for 100,000 iterations and save checkpoints every 4,000 iterations. Then evaluate the last 5 checkpoints for 50 episodes and get the average success rates 3. Finally, conduct the experiments with 3 seeds and report the average results.
Evaluation.
We compare SGRv2 against the following baselines: (1) R3M [9] is a 2D visual represen- tation designed for robotic manipulation, which is pre-trained on large-scale human video datasets. For fair comparisons, we utilize frozen R3M to process RGB images, employ a separate 2D CNN to process depth images, and subsequently fuse the two resulting features.
Baselines.
(2) PointNeXt [15] is an enhanced version of the classic PointNet++ architecture for point cloud processing. We employ the encoder of PointNeXt to obtain the 3D representation.
(3) PerAct [16] is 3D representation that voxelizes the workspace and utilizes a Perceiver Transformer [62] to process voxelized observa- tions. (4) SGR [10] is a representation that integrates both high-level 2D semantic understanding and low-level 3D spatial reasoning. (5) RVT [17] a 3D representation that utilizes a multi-view trans- former to predict actions and integrates these predictions into 3D space through hack-projection from multiple viewpoints.
MimicGen [14] reports maximum results across different checkpoints, while we provide average results, offering more robust and realistic measure of model performance.

---

Method
Avg. Success
Avg. Rank
LifiCube
PickCube
StackCube
PickSingleYCB
PointNeXt
16.8
2.5
50.8
4.7
10.6
1.1
SGR
14.9
2.5
26.9
12.2
3.5
17.0
SGRv2 (ours)
55.8
1.0
80,5
72.9
27.7
42.2
Method
Avg. Success
Avg. Rank |
Stack
StackThree
Square
Threading
Coffee
HanmerCleamup MugCleanup
PoinNeXt
13.6
2.9
56.1
3.7
0.9
3.6
12.0
11.7
7.1
SGR
14.2
2.0
50.8
5.6
1.3
4.0
14.1
14.1
9.7
SGRv2 (ours)
26.0
1.0
81.2
37,9
2,8
6.7
27.9
16.1
9.7
Table 2: Performance ManiSkil12 (top) and MimicGen (bottom) with 50 Demonstrations. We report
success rales averaged over 3 seeds. See Appendix F for standard deviation. We observe that SGRv2 consis- tently outperforms baselines like SGR and PointNeXt.
4.2 Simulation Results
RLBench Performance. Figure depicts the performance of SGR and SGRv2 across differing number of demonstrations. Initially, as the availability of data decreases, we note significant de- cline in SGR's performance compared to SGRv2, highlighting the difficulties in maintaining model performances without the inductive bias towards locality awareness. Additionally, Table provides a comparison of success rates obtained with only 5 demonstrations using various representations. We observe that the absence of 3D geometric information, as demonstrated by R3M using 2D representation, results in markedly low performance, highlighting the critical role of 3D priors in robotic manipulation under data-constrained scenarios. Finally, SGRv2 is demonstrated to be the superior representation, achieving 53.2% average success rate with merely 5 demonstrations and significantly outperforming the most competitive baseline, RVT, with an average improvement fac- tor of 1.32x and achieving enhanced performance in 23 out of 26 tasks. The results underscore the advantages of incorporating 3D geometry and well-designed locality to enhance sample efficiency.
ManiSkill2 and MimicGen Performance. To evaluate the performance of SGRy2 in dense con- trol scenarios, conduct comparisons with several baselines ManiSkill2 and MinicGen. Our experimental results, summarized in Table 2, demonstrate that SGRv2 significantly outperforms the baselines. In particular, thanks to our tailored approach for dense control, SGRv2 exhibits supe- rior performance in tasks where the object's location consistently aligns with the direction of the delta actions, such as Pick Cube and Stack Three. These findings confirm that SGRv2 acts as a sample-efficient, universal representation adept at handling both keyframe and dense control scenar- ios. See Appendix D for results different numbers of demonstrations and additional baselines.
Ablations. As shown in Table 3, we conduct abla- tions to assess the locality design choices of SGRv2. (1) Decoder architecture is the cornerstone of our locality designs. Omitting the decoder from the SGRv2 would prevent the application of other local- ity designs, forcing
Method
Avg. success
SGRv2
53.2
SGRv2 w/o decoder
21.3
SGRv2 w/ absolute pos prediction
21.0
SGRv2 uniform point weight
44.2
rely solely the global representation from the encoder. This would lead a significant decrease in performance. (2) Predict- ing absolute positions, results in markedly poorer performance. This underscores that relative position predictions are the key insight of the locality design.
SGRv2 w/o dense supervision
40.1
Table 3: Ablations. Average success rate of 26 RLBench tasks with demonstrations after ablat- ing key components of locality design.
weights with uni- form weights reduces performance, confirming the role of point-wise weighting in focusing on pre- dictive local regions. (4) Eliminating dense supervision leads to a decline in overall performance, illustrating that dense supervision enhances the model's learning efficacy.
(3) Substituting point-wise
Emergent Capabilities. In our study, we visualize the point-wise weights of SGRv2 detailed in Section 3.2. As depicted in Figure 3, we sequentially visualize point clouds with RGB and point- wise weights. Surprisingly, the results consistently demonstrate an alignment between points with higher weights (in red) and the object affordances, which denote the functional areas of objects. This observation highlights the capability of SGRv2 to precisely identify and emphasize critical local regions on objects. Motivated by this, we conduct experiments on visual distractors in Appendix E.
7

---

Figure 3: Emergent Capabilities. We visualize the point-specific weights and find that the points with high
weights (in red) align with the object's affordances. Left: toilet seat Right: open mi crowave.
Task
Sub-task
PerAct
RVT
SGRv2
Put trash in trash can
50
50
80
socks box
60
Tidy Up Table
Put marker in pen holder
10
10
30
Open drawer
20
40
60
Put lollipop drawer
10
Close drawer
40
60
Turn on coffee machine
100
100
100
Make Coffee
Put funnel onto carafe
0
20
Pour powder funnel
Pour water
10
30
70
Avg. Success Rate
30
41
63
Figure 4: Left: Real-robot long morgioo tasks. Right: Success rate (%) of multi-task agents real-robot tasks. We collect demonstrations and evaluate 10 episodes for each task.
4.3 Real-Robot Results
To evaluate the effectiveness SGRv2 real-robot setting, we conduct experiments using the Franka Emika Panda across two long-horizon tasks (a total of 10 sub-tasks), and generalization task. Refer Appendix C for more details on robot setup, task designs and failure cases discussions.
Long-horizon Tasks. We collect 8 demonstrations per task and train multi-task agents. Each sub- task is tested across 10 episodes. We present comparative results between PerAct, RVT, and SGRv2 in Figure 4, where SGRv2 demonstrates a substantial performance advantage. A common issue with PerAct is that tends to bias towards occupied voxels, frequently causing collisions. For RVT, typi- cal failures stem from large position errors along the camera's 's viewing direction, potentially caused from the inaccuracies in the virtual images rendered orthogonal to the real camera's perspective.
Generalization Task. We assess the generalization capability of SGRv2 by employing 6 cups of different colors. Each scene involves one target
Method
Seen Unseen
PerAct
100
10
cup and two distractor cups of different colors. We collect 5 demonstra-
RVT
100
tions for each of the 4 colors and test the model on both 4 seen and unseen scenarios. Each color is tested across 10 episodes. As indicated in the right table, compared with PerAct and RVT, SGRv2 exhibits the ability to generalize across color variations. The results show SGRv2 without the semantic branch achieves zero performance on unseen colors, underlining the critical role of semantic awareness in enhancing generalization.
SGRv2 w/o
100
SGRv2
100
5 Discussion
Limitations. Our models are currently trained using a vanilla BC objective. A promising direction involves integrating the Diffusion Policy [2], which excels in dealing with multimodal and hetero- geneous trajectories, with our locality framework to further enhance performance real world. Additionally, due to our focus on sample efficiency, the evaluations on generalization are currently insufficient. We are eager to expand this work to include broader range of generalization aspects, such object shapes, camera positions, and background textures.
Conclusion. We present SGRv2, a systematic framework of visuomotor policy that considers both visual and action representations. Built upon the SGR, SGRv2 integrates action locality across its entire framework. Through comprehensive evaluations across diverse range tasks in multi- ple simulated and real-world environments with limited data availability, SGRv2 exhibits superior performance and outstanding sample efficiency.

---

Acknowledgments
We thank Haoxu Huang and Fanqi Lin for their assistance in conducting real-robot experiments. This work is supported by the Ministry of Science and Technology of the People's Republic of China, the 2030 Innovation Megaprojects "Program on New Generation Artificial Intelligence" (Grant No. 2021AAA0150000), This work is also supported by the National Key R&D Program of China (202ZZD0161100),
References
[1] B. D. Argall, S. Chernova, M. Veloso, and B. Browning. A survey of robot learning from demonstration. Robotics and autonomous systems, 57(5):469-488) 2009.
[2] C. Chi, Feng. Du, Z. Xu, E. Cousineau, Burchfiel, and S. Song. Diffusion policy: Visuomotor policy learning via action diffusion. arXiv preprint arXiv:2303.04137, 2023.
[3] T. Z. Zhao, V. Kumar, S. Levine, and C. Finn. Learning fine the-grained bimanual manipulation with low-cost hardware. arXiv preprint arXiv:2304. 13705, 2023.
[4] Y. LeCun, B. Boser, S. Denker, D. Henderson, R. Howard, W. Hubbard, and D. Jackel. Backpropagation applied to handwritten zip code recognition. Neural computation, 1(4):541- 551, 1989.
[5] Z. Liu, H. Mao, C.-Y. Wu, C. Feichtenhofer, T. Darrell, and S. Xie. convnet for the 2020s. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11976-11986, 2022.
[6] J. L. Elman. Finding structure in time. Cognitive science, 14(2):179-2) 111121, 1990.
[7] A. Graves and Graves. Long short-term memory. Supervised sequence labelling with recurrent neural networks, pages 37-45, 2012.
[8] Y. J. Ma, S. Sodhani, D. Jayaraman, O. Bastani, V. Kumar, and A. Zhang. Vip: Towards universal visual reward and representation via value-implicit pre-training. arXiv preprint arXiv:2210.0003 2022.
[9] S. Nair, A. Rajeswaran, V. Kumar, C. Finn, and A. Gupta. R3m: A universal visual represen- tation for robot manipulation. arXiv preprint arXiv:2203. 2022.
[10] Zhang, Y. Hu, H. Cui, Zhao, and Y. Gao. universal semantic gemamtic-goometric representation for robotic manipulation. In Conference on Robot Learning, pages 3342-3363, PMLR, 2023.
[11] Y. Ze, G. Zhang, K. Zhang, C. Hu, M. Wang, and H. Xu. 3d diffusion policy. arXiv preprint arXiv:2403.03954 2024.
[12] James, Z. Ma. Arrojo, and A. Davison. Rlbench: The robot learning benchmark & learning environment. IEEE Robotics and Automation Letters, 5(2):3019- 2020.
[13] J. Gu, F. Xiang, X. Li, Z. Ling, X. Liu, T. Mu, Y. Tang, S. Tao, X. Wei, Y. Yao, et al. Maniskil12: unified benchmark for generalizable manipulation skills. arXiv preprint arXiv:2302.04659, 2023.
[14] A. Mandlekar, S. Nasiriany, B. Wen, I. Akinola, Y. Narang, L. Fan, Y. Zhu, and D. Fox. Mimicgen: A data generation system for scalable robot learning using human demonstrations. arXiv preprint arXiv:2310.17596 2023.
[15] G. Qian, Y. Li, H. Peng. Mai, H. A. A. K. Hammoud, M. Elhoseiny, and B. Ghanem. Pointnext: Revisiting pointnet++ with improved training and scaling strategies. arXiy preprint arXiv:2206.0460 2022.

---

[16] M. Shridhar, L. Manuelli, and D. Fox. Perceiver-actor: A multi-task transformer for robotic manipulation. arXiv preprint arXiv:220900545|, 2022.
[17] A. Goyal, J. Xu, Y. Guo, V. Blukis, Y.WW. Chao, and Fox. Rvt: Robotic view transformer for 3d object manipulation. In Conference on Robot Learning, pages 694-710. PMLR, 2023,
[18] M. Laskin, A. Srinivas, and P. Abbeel. Curl: Contrastive unsupervised representations for reinforcement learning, In International Conference on Machine Learning, pages 5639-5650. PMLR, 2020.
[19] M. Laskin, K. Lee, A. Stooke, Pinto, P. Abbeel, and A. Srinivas. Reinforcement learning with augmented data. Advances in neural information processing systems, 33: 33:19884-19885, 2020.
[20] I. Kostrikov, D. Yarats, and R. Fergus. Image augmentation all you need: Regularizing deep reinforcement learning from pixels. arXiv preprint arXiv:2004.13649 2020.
[21] C. Gelada, S. Kumar, J. Buckman, O. Nachum, and M. G. Bellemare. Deepmdp: Learning
continuous latent space models for representation learning. In International Conference Machine Learning, pages 2170-2179, PMLR, 2019.
[22] D. Hafner, T. Lillicrap, J. Ba, and M. Norouzi. Dream to control: Learning behaviors by latent imagination. arXiv preprint arXiv: 1rXiv:9912.01003, 2019.
[23] R. Jonschkowski and O. Brock. Learning state representations with robotic priors, Autonomous Robots, 39:407-428, 2015.
[24] A. Zhang, R. McAllister, R. Calandra, Y. Gal, and S. Levine. Learning invariant representttos for reinforcement learning without reconstruction. arXiv preprint arXiv:2006. 10742, 2020.
[25] I. Radosavovic, Xiao, S. James, Abbeel, Malik, and T. Darrell. Real-world robot learning with masked visual pre- vre-training. arXiv preprint arXiv:2210.03100 2022.
[26] A. Majumdar, K. Yadav, S. Amaud, Y. J. Ma, C. Chen, S. Silwal, A. Jain, V.-P. Berges, Abbeel, J. Malik, et al. Where are we in the search for an artificial visual cortex for embodied intelligence? arXiv preprint
23V:23333182400 2023.
[27] Y. Hu, R. Wang, L. E. Li, and Y. Gao. For pre-trained vision models in motor control, not all policy learning methods are created equal. arXiv preprint arXiv:2304.04591, 2023.
[28] R. Shah and V. Kumar. Rrl: Resnet as representation for reinforcement learning, arXiv preprint arXiv:2107.03380, 2021.
[29] C. Wen, X. Lin, So, K. Chen, Q. Dou, Y. Gao, and P. Abbeel. Any-point trajectory modeling for policy learning. arXiv preprint arXiv:2401.000 200400000555,,, 2023.
[30] Z. Yuan, Z. Xue, B. Yuan, X. Wang. Y. Wu, Y. Gao, and H. Xu. Pre-trained image encoder for generalizable visual reinforcement learning. arXiv preprint arXiv:2212.088860 2022.
[31] Xiao, I. Radosavovic, T. Darrell, and J. Malik. Masked visual pre-training for motor control. arXiv preprini arXiv:2203.0617 2022.
[32] S. Parisi, A. Rajeswaran, S. Purushwalkam, and A. Gupta. The unsurprising effectiveness of pre-trained vision models for control. In International Conference on Machine Learning, pages 77359-17371. PMLR, 2022.
[33] S. James, Wada, T. Laidlow, and A. Davison. Coarse-to-fine q-attention: Efficient learning for visual robotic manipulation via discretisation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13739-13748, 2022.
10

---

[34] Y. Ze, G. Yan, Y.-H. Wu, A. Macaluso, Y. Ge, J. Ye, N. Hansen, L. E. Li, and X. Wang. Gnfactor: Multi-task real robot learning with generalizable neural feature fields. In Conference on Robot Learning, pages 284 PMLR, 2023.
[35] H. Liu, L. Lee, K. Lee, and P. Abbeel. Instruction followiing agents with jointly pre- pre-trained rision-language models. arXiv preprint arXiv:2210.134311 2022.
[36] P.-.. Guhur, S. Chen, R. Gareia, M. Tapaswi, I. Laptev, and C. Schmid. Instruction-driven
history-aware policies for robotic manipulations. arXiv preprint arXiv:2209.04899, 2022.
[37] Y. Seo, J. Kim, S. James, K. Lee, J. Shin, and P. Abbeel. Multi-view masked world models for visual robotic manipulation. arXiv preprint arXiv:2302.02400 2023.
[38] C. R. Qi, L. Yi, H. Su, and L. J. Guibas. Pointnet++: Deep hierarchical feature learning on point sets metric space. Advances neural information processing systems, 30, 2017.
[39] H.-.. Fang, C. Wang, M. Gou, and Lu. Graspnet-Ibillice A large-scale benchmark for general object grasping. In Proceedings the IEEE/CVF conference computer vision and pullern recognition, pages 11444-11455 2020.
[40] M. Sundermeyer, A. Mousavian, R. Triebel, and D. Fox. Contact-grann graspnet: Efficient 6-dof grasp generation in cluttered scenes. In 202/ IEEE International Conference on Robotics and Automation (ICRA), pages 13444. IEEE, 2021.
[41] W. Huang. I. Mordatch, Abbeel, and D. Pathak. Generalization in dexterous manipulation geometry-aware multi-tase learning. arXiv preprint arXiv:2111.03062, 2021.
[42] Y.-H. Wu, J. Wang, and X. Wang. Learning generalizable dexterous manipulation from human grasp affordance. arXiv preprint arXiv:2204.02320, 2022.
[43] Y. Qin, B. Huang, Z.-H. Yin, H. Su, and X. Wang. Dexpoint: Generalizable point cloud rein-
forcement learning for sim-to-real dexterous manipulation. arXiv preprint S.ii::2211.09423. 2022.
[44] M. Liu, X. Li, Z. Ling, Y. Li, and H. Su. Frame mining: free lunch for learning robotic manipulation from 3d point clouds. arXiv preprint arXiv:2210.07442, 2022.
[45] T. Gervet, Z. Xian, N. Gkanatsios, and K. Fragkiadaki. Act3d: 3d feature field transformers for multi-task robotic manipulation. In 7th Annual Conference on Robot Learning, 2023.
[46] S. Chen, R. Garcia, I. Laptev, and C. Schmid. Sugar: Pre-training 3d visual representations for robotics. arXiv preprint arXiv:2404.0149/ 2024.
[47] T.-W. Ke, N. Gkanatsios, and K. Fragkiadaki. 3d diffuser actor: Policy diffusion with 3d scene representations. arXiv preprint 10X8v:2402.10885, 2024.
[48] Z. Xue, Z. Yuan, J. Wang, X. Wang. Y. Gao, and Н. Xu. Useek: Unsupervised (3)- equivariant 3d keypoints for generalizable manipulation. In 2023 IEEE International Con- ference on Robotics and Automation (ICRA), pages 1715-1722. IEEE, 2023.
[49] H. Ryu, H.-i Lee, J.-H. Lee, and J. Choi. Equivariant descriptor fields: Se (3)-equivariant energy-based models for end-to-end visual robotic manipulation learning. arXiv preprint 0:Xiv:2206.08321, 2022.
[50] A. Simeonov, Y. Du, A. Tagliasacchi, B. Tenenbaum, Rodriguez, P. Agrawal, and V. Sitz- mann. Neural descriptor fields: Se (3)-equivariant object representations for manipulation. In 2022 International Conference Robotics and Automation (ICRA), pages 6394-64000 IEEE, 2022.

---

[51] J. Yang. C. Deng, J. Wu, R. Antonova, L. Guibas, and J. Bohg. Equivact: Sim equuiuvrar visuomotor policies beyond rigid object manipulation. arXiv preprint (00o::331...0300000 2023.
[52] E. Chun, Y. Du, A. Simeonov, T. Lozano-Perez, and L.. Kaelbling. Local neural descriptor fields: Locally conditioned object representations for manipulation. In 2023 IEEE Interna- tional Conference on Robotics and Automation (ICRA), pages 1830-1836. IEEE, 2023.
[53] C. Gao, Xue, S. Deng, T. Liang, S. Yang. L. Shao, and H. Xu. Riemann: Near real- time (3)-equivariant robot manipulation without point cloud segmentation. arXiv preprint arXiv:2403.19466 2024.
[54] Fuchs, D. Worrall, V. Fischer, and M. Welling. Se (3)-transformers: 3d roto-translation equivariant attention networks. Advances in neural information processing systems, 33:1970- 1981, 2020.
[55] A. Zeng, P. Florence, J. Tompson, S. Welker, J. Chien, M. Attarian, T. Armstrong, I. Krasin, D. Duong, V. Sindhwani, et al. Transporter networks: Rearranging the visual world for robotic manipulation. In Conference on Robot Learning, pages 726-747. PMLR, 2021.
[56] C. R. Qi, H. Su, K. Mo, and L. J. Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 652-660, 2017.
[57] A. Simeonov, Y. Du, Y--.. Lin, A. R. Garcia, L. P. Kaclbling, T. Jozano-Pérez. and P. Agrawal. Se equuivvrrt relational rearrangement with neural descriptor fields. In Conference on Robot Learning, pages 335-846. PMLR, 2023.
[58] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, al. Learning transferable visual models from natural language supervi- sion. In International conference machine learning, pages 8748 8763. PMLR, 2021.
[59] L. X. Shi, Sharma, T. Z. Zhao, and C. Finn. Waypoint-based imitation learning for robotic manipulation. In Conference on Robot Learning, pages 2209. PMLR, 2023.
[60] Y. Zhu, J. Wong, A. Mandlekar, R. Martin-Martín, A. Joshi, S. Nasiriany, and Y. Zhu. ro- bosuite: A modular simulation framework and benchmark for robot learning. arXiv preprint arXiv:2009. 10009,.222,..... 2020.
[61] N. Hansen, H. Su, and X. Wang. Td-mpe2: Scalable, robust world models for continuous control. arXiv preprint arXiv:2310.16825 2023.
[62] A. Jaegle, S. Borgeaud, J.-B. Alayrac, C. Doersch, C. Ionescu, D. Ding. S. Koppula, D. Zoran, A. Brock, E. Shelhamer, et al. Perceiver io: A general architecture for structured inputs & outputs. arXiv preprint arXiv:2107. Stii:2107.147999 14795, 2021.
[63] B. Calli, A. Singh, A. Walsman, Srinivasa, Abbeel, and A. M. Dollar. The ycb object and model set: Towards common benchmarks for manipulation research. In 2015 international conference on advanced robotics (ICAR), pages 510-517. IEEF, 2015.
[64] A. Mandlekar, D. Xu, J. Wong, S. Nasiriany, C. Wang, R. Kulkarni, L. Fei-Fei, S. Savarese, Y. Zhu, and R. Martin-Martin. What matters in learning from offline human demonstrations for robot manipulation. arXiv preprint arXiv: 03298, 2021.
12

---

A
Simulation Task Details
(5)
Open Microowave
Open Door
Water Plants
Tollet Seat Up
Phone On Base
Put Books
[12)
Take Out Umbrella
Fridge
Open Drawer
Slide Block
Sweep Dustpan
Meat Off Grill
114
(16)
[18]
Drag Blocks
[19]
200]
211
[22]
23
24]
Put In Safe
Place Wine
Place In Cupboard
Sort Shape
Push Buttons
Insert Peg
Stack Cups
Place
Lift Cube
Pick Cube
Stack Cube
Pick Singe YCB (002)
(31)
(32)
(33)
(34)
(35)
(36)
Singo (004)
Singo [005)
Singe (006)
Singe YCS (007)
Singe (008)
Singe (010)
(37)
(38)
(39)
(40)
(41)
(42)
Pick Singe YCB (011)
Stack
Stack Three
Square
Threading
Coffee
(43)
Hammer Cleanup
Mug Cleanup
Figure 5: Simulation Tasks. Our simulation experiments encompass 26 tasks (1-26) from RL- Bench, 4 tasks (27-37, where 30-37 are 8 different YCB [63] objects of task Pick Single YCB) from ManiSkil12, and 7 tasks (38-44) from MimicGen.
Our simulation experiments conducted 3 robot learning benchmarks: RLBench [12], Man- iSkil12 [13]. and MimicGen [14]. See Figure 5 for overview of the simulation tasks. In these
13

---

simulations, all cameras have resolution of 128 x 128. In the following, we will provide a detailed examination tasks from the three benchmarks.
A.1 RLBench Tasks
We utilize 26 RLBench tasks, including tasks used in SGR [10] and tasks used in PerAct [16] and RVT [17]. For tasks with multiple variations, we use the first variation. In RLBench, we use 5 demonstrations per task, unless specified otherwise. Given that SGR and PerAct provide detailed descriptions of these RLBench tasks, we omit these details here for simplicity.
A.2 ManiSkill2 Tasks
We utilize 4 ManiSkil12 tasks, each described in detail as follows. (1) Lift Cube: Pick up a red cube and lift it to specified height. (2) Pick Cube: Pick up a red cube and move it to a target position. (3) Stack Cube: Pick up a red cube and place onto a green cube. (4) Pick Single YCB: Pick up a YCB [63] object and move
the target position. In our experiments, we use 8 YCB objects (excluding
those that are too difficult pick up): 002 master chef can, 004 sugar box, 005 tomato soup can, 008 108_pudding_bor 010 010-potted siatteddmmtt_camm 01 I Jananna For the first three tasks, we utilize 50 demonstrations per task, while for the last one (Pick Single YCB), we employ 50 demonstrations per YCB object.
006 maussard_bottle 007 J07_juna_fish_com
A.3 MimicGen Tasks
We utilize 7 MimicGen tasks with 50 demonstrations per task, all employing the initial distribution follows: (1) Stack:
D1, which presents broader and more challenging range. The details
green one. (2) Stack Three: Similar to Stack, but with additional step of stacking a blue block on the red one. (3) Square: Pick up square nut and place it a peg. (4) Threading: Pick up needle and thread it through a hole in tripod. (5) Coffee: Pick up a coffee pod, insert it into the coffee machine, and close the machine hinge. (6) Hammer Cleanup: Open a
Stack red block
drawer, pick up a hammer, place it back into the drawer, and close the drawer. (7) Mug Cleanup: Similar to Hammer Cleanup, but with mug.
B
SGRv2 Details
B.1
Architecture Details
Input Data. The SGRv2 model takes as input RGB images {4x k=1 size H x W and correspond- ing depth images of the same size from multiple camera views. Point clouds are generated from these depth images using known camera extrinsics and intrinsics. A crucial aspect is the alignment of the RGB images with the point clouds, ensuring a precise one-to-one correspondence between elements in the two data forms. For keyframe control, the point cloud is represented in the robot's base frame. In contrast, for dense control inspired by FrameMiner [44] the point cloud is transformed into the end-effector frame to simplify computation and enhance performance.
For keyframe control, the model additionally receives proprioceptive data s, which includes four scalar values: gripper open state, left finger joint position, right finger joint position, and action sequence timestep. dense control, proprioceptive data is not utilized. Additionally, following SGR [10], if a task comes with language instruction S, this also forms part of the model's input.
Other Details. Following SGR [10], we use CLIP-ResNet-50 as the image encoder for the seman- tie branch. For the 3D encoder-decoder, we employ XtL The output from the encoder- decoder is a wiset-wise feature, denoted as f!a E R for the i-th point, where the feature dimension is 64. We apply a linear layer followed by ReLU activation to produce processed point-wine wise feature ft. increasing the feature dimension to 256. We then predict the relative position Ap(ft), magnitude mff.) (for dense control), rotation r(fe), gripper open state off.), and collision indicator c(f+) (for keyframe control) of the i-th point, where Ap, m, r, O, are 3-layer ML.ss. Note that when
14

---

representing the ground-truth actions as one-hot vectors-stee as rotation, gripper open state, and collision indicators in keyframe control-the action predictions correspond to the output probabil- ities following the softmax layer. Finally, for each action component, we assign a learned weight w. (fi) to each point, where w, represents separate 3-layer MLPs with softmax normalization across the points dimension.
B.2 SGR Details
SGRv2 is built upon SGR [10], which we briefly introduced in Section 3.1. Here, provide detailed description of SGR's three components: semantic branch, geometric branch, and fusion network.
Semantic Branch. Using collection of RGB images { from K calibrated cameras, they initially apply a frozen pre-trained 2D model G, such as CLIP's visual encoder, to extract multi-view p
image features (T(Tx) ) ) k=1- When language instruction S accompanies task, they utilize pre- trained language model H. like CLIP's language encoder, to generate the language features H(S). They align these image features ((xx) with the language features H(S) using visual grounding module, producing {M&] Subsequently, they rescale the visual or aligned feature maps to the dimensions of the original images through bilinear interpolation and reduce their channels by 1 x convolution, generating set of features {Fis 1 where each F3 E j xCi, These high-level semantic features are then back-projected into 3D space to form point-wise features for the point cloud, expressed as Feem E NC1, where N = x H x W.
Geometric Branch. They construct the initial point cloud coordinates P == (p:) RR and RGB features F. E IR Nx3 using multi-view RGB-D images and camera parameters (i.e., cam- era intrinsics and extrinsics). Optionally, they append a D-dimensional vector, derived from robot proprioceptive data via linear layer, to each point feature. They then process the point cloud coor- dinates P and features F, through a hierarchical PointNeXt encoder, extracting compact geometric coordinates P' R M3 and features F xxx (M N).
Fusion Network. To merge the complementary branches, they first subsample the point-wise semantic features Feem using the same point subsampling procedure in the geometric branch, resulting in F E MxCI They then perform a channel -wise concatenation of the semantic and geometric features to form Fruse = Concat sem F() E TRMx(CI+CC Finally, the fused features are processed through several set abstraction blocks [38, 15), enabling a cohesive modeling of the cross-modal interaction between 2D semantics and 3D geometric information.
B.3 Training Details
Losses. As illustrated in Section 3.3, in keyframe control, our loss objective is follows:
Lkeytrams Cp + az Cro + Copon + a4 Coollide
(3)
where Lpss L1 loss, and Luo, Lopens and Leallide are cross-entropy losses. In our experiments, we set 0j 300 and a2 ag 04
In dense control, our loss objective is:
Laense
(L Emag) + B2 + 33 + B,
(4)
where Lair. Langg and Lis are MSE losses, Lopen is cross-entropy loss, and freg is smoothness regu- larization loss. In our experiments, we set 31 10, 32 B3 = and 34 = 0.3.
Data Augmentation. (1) Translation and rotation perturbations: in keyframe control, the train- ing samples is augmented with 10.125 m translation perturbations and +45e yaw rotation pertur- bations. (2) Color drop is to randomly replace colors with zero values. This technique serves as powerful augmentation for PointNeXt [15], leading to significant enhancements in the performance of tasks where color information is available. (3) Feature drop: Color drop randomly replaces colors with zero values, which results in both the RGB and semantic features becoming constant.
15

---

However, there are certain tasks where colors play crucial role, and disregarding color informa- tion in these tasks would make them unsolvable. To address this issue, we propose feature drop- Specifically, this involves randomly replacing the semantic features with zero values, while keeping the RGB values unchanged. (4) Point resampling is a widely used technique in point cloud data processing that adjusts the density of the point cloud. It involves selecting subset of points from the original dataset create new dataset with modified density. Firstly, filter out points outside the workspace. Then in keyframe control, we resample 4096 points from the point cloud using far- thest point sampling (FPS), while in dense control, we resample 1200 points using the same method. (5) Demo augmentation [33] [16]. used in keyframe control, captures transitions from intermediate points along a trajectory to keyframe states, rather than from the initial state to the keyframe state. This approach significantly increases the volume the training data.
Hyperparameters. The configuration of hyperparameters applied in studies shown in Ta- ble 4. For each task, the experiments are conducted on a single NVIDIA GeForce RTX 3090 GPU.
Table 4: Hyper-parameters used in our simulation experiments.
Config
Keyframe Control
Dense Control
Training iterations
20, 000
100,
Leraning rate
0.003
0.0003
Batch size
16
16
Optimizer
Adam Cosine
Adam W Cosine
Lr Scheduler
Warmup step
200
Weight decay
10 10-6
1 x 10-6
Color drop
0.4
0 0.4
Feature drop
Number of input points
4096
1200
B.4 Training and Inference Speed
We test the training and inference speed of SGRv2 on NVIDIA 3090 GPU. For keyframe control, training takes approximately 5 hours for 20k steps, with an inference speed of 10 FPS. For dense control, training requires around 7 hours for 100k steps, with an inference speed of 30 FPS.
C Real-Robot Details
C.1 Real-Robot Setup
For our real-robot experiments, we use a Franka Emika Panda manipulator equipped with a parallel gripper. We utilize keyframe control, and the motion planning is executed through MoveIt Per- ception is achieved through an Intel RealSense L515 camera, positioned in front of the scene. The camera generates RGB-D images with resolution of x280x720. We leverage the realsense-ros to align depth images with color images. The extrinsic calibration between the camera frame and robot base frame is carried out using the MoveIt calibration package.
When preprocessing the RGB-D images, we resize the 1280 x 720 images to 256 x 256 using neighot- interpolation. We choose this interpolation method instead of others, like bilinear interpolation, because the latter can introduce artifacts into the depth map, resulting in noisy point cloud. Following these steps enables us to process RGB-D images as we do in our simulation experiments. It is essential to adjust the camera's 's intrinsic parameters appropriately after resizing the images. We train SGRv2 for 40,000 training steps and use the final checkpoint for evaluation.
16

---

Figure 6: Reaal-robo generalization task.
C.2 Real-Robot Tasks
Our real-robot experiments involve three tasks: Tidy Up the Table, Make Coffee, and Move Color Cup to Target. The first two are long-horizon tasks, while the last is generalization task. We provide details of the task design as follows.
Tidy Up the Table (as shown in Figure 4 Top Left) is to place the clutter on the table in its appropri- ate locations. The task consists of sub-tasks, each detailed as follows: (1) Put trash in trash can: Pick up the trash and place it in the trash can. (2) Put socks in box: Pick up the socks and place them in the box. (3) Put marker
pen holder: Pick up the marker and place it in the pen holder. (4) Open drawer: Grasp the drawer handle and pull it open. (5) Put lollipop in drawer: Pick up the lollipop and place it into the drawer. (6) Close drawer: Push the drawer closed.
Make Coffee (as shown in Figure 4 Bottom Left) is to make pour-over coffee. This task composed of sub-tasks, each described in detail as follows: (1) Turn on coffee machine: Press the button on the coffee machine activate it. (2) Put funnel onto carafe: Pick up the funnel and place onto the carafe. (3) Pour powder into funnel: Pick up the powder holder and pour the powder into the funnel. (4) Pour water: Pick up the kettle and pour the water onto the powder in the funnel.
Move Color Cup to Target (as shown in Figure 6) is to select the target color cup from three cups and move to the white area. The target color indicated through language instructions. We have cups of different colors: white, red, yellow, orange, black, and green. Each scenario involves one target cup and two distractor cups of different colors. We collect five demonstrations for each of the first 4 colors and test the model on both 4 seen and unseen scenarios.
C.3 Discussions of Failure Cases
In the real-robot experiments, the primary failure cases of SGRv2 include: (1) For smaller objects like lollipops, the model sometimes struggles to detect them, especially when they are farther away from the camera. (2) For tasks that are sensitive to grasping position or angle, such as grasping markers or funnels, slight deviations can lead unsuccessful attempts or cause the object slip. (3) In keyframe control, where a few sparse target poses output and then executed through motion planner, failures can occur due to collisions during the motion planning phase.
D Additional Results
Our primary objective of Section 4 is to demonstrate the sample efficiency of SGRv2. This is why we initially choose to evaluate our method using only 5% of the data employed in the baselines' original settings. However, to make our experiments more complete, we conduct additional experiments to provide a more comprehensive evaluation on more different numbers of demonstrations and more baselines. Note that all our results are the average success rates of the last 5 checkpoints.
"ncrpe://noveit.roo. http:p//gttthaooo/eeeellaaaaaaar
17

---

Avg. Success
Open Microwave
Open
Water Plants
Toilet Seat Up
Phone On Base
Put Books
Take Out
Open
Method
Door
Umbrella
Fridge
PointNeXt
33.1
13.6
61.6
14.8
64.4
57.2
48.0
83.6
16.4
PerAct
36.7
9.2
78.0
12.0
83.6
0.0
18.8
91.6
14.4
SGR
47.8
46.0
76.8
24.4
59.6
82.8
92.0
90.0
26.4
RVT
52.1
19.2
79.2
11.2
62.0
78.4
63.6
97.2
18.4
SGRv2 (ours)
63.3
68.4
86.0
17.6
69.2
85.6
69.2
95.6
19.2
Open Drawer
Slide
Sweep To
Meat Off
Turn
Put In
Close
Drag Stick
Stack
Method
Block
Dustpan
Grill
Tаp
Drawer
Jar
Blocks
PointNeXt
63.6
83.6
52.4
0.0
84.8
35.6
0.0
8.8
PerAct
89.8
97.3
32.9
98.2
5.5
4.4
23.2
75.3
43.4
SGR
75,6
89.2
63.2
93.6
94.8
22.8
36.4
80.8
0.0
RVT
70.0
71.2
18.0
92.0
73.6
84.4
35.2
100.0
18.8
SGRv2 (ours)
92.8
94.4
64.4
97.6
95.2
80.8
32.4
94.8
52.0
Screw Bulb
Put
Place
Sort Shape
Push
Insert
Stack
Place
Method
Safe
Wine
Cupboard
Buttons
Peg
Cups
Cups
PointNeXt
21.6
7.2
13.6
18.0
2.8
100.0
1.2
6.0
0.0
PerAct
18.2
7.9
39.7
7.9
2.2
82.0
8.9
7.7
1.2
SGR
17.6
27.6
35.6
12.4
2.8
84.8
2.0
6.0
0.8
RVT
43.2
67.2
92.0
17.6
6.4
100.0
12.8
22.8
0.4
SGRv2 (ours)
68.4
59.2
68.0
50.4
6.4
99.2
8.0
70.4
0.8
Table 5: Performance on RLBench with 100 demonstrations.
#Demonstrations
100
50
20
10
5
RVT
52.1
46.3
43.3
42.3
40.4
SGRv2 (ours)
63.3
62.4
61.9
56.0
53.2
Table 6: Average performance of 26 RLBench tasks with varying number of demonstrations.
RLBench with 100 Demonstrations. We test SGRv2 and the baseline methods (RVT, SGR, PerAct, and PointNeXt) using 100 demonstrations. Refer Table for the results.
RVT with Varying Demonstrations. We test RVT (the most competitive baseline in RLBench) on 26 RLBench tasks, with demonstration numbers ranging from 100 to 5. Table shows the average results of 26 tasks compared with results of SGRv2.
MinicGen with 1000 Demonstrations. We evaluate SGRv2 against SGR, PointNeXt, 2D BC- RNN (used in MimicGen [14] and robomimic [64]), and 2D BC (used in robomimic [64]). The latter two baselines are trained and tested using the official codes of MimicGen and robomimic 7, but with different evaluation metrics-we report the average results of the last 5 checkpoints instead of the maximum of 30 checkpoints, as explained in Section 4.1. Table 7 shows the results.
More Baselines in MimicGen with 50 Demonstrations. In addition the baselines included in the Table 2, evaluated SGRv2 against the 2D BC-RNN (used in MimicGen [14] and robomimic [64]), 2D BC (used in robomimic [64]), and R3M [9] baselines with 50 demonstrations. Table 8 shows the results.
MimicGen with 200 Demonstrations. We further compared SGRv2 against 2D BC-RNN (used in MimicGen [14] and robomimic [64]) and 2D BC (used in robomimic [64]) using 200 demonstra- tions. Table shows the results.
The results from these extended experiments show that SGRv2 consistently outperforms the base- lines across all settings. While it's noteworthy that BC-RNN achieves comparable performance to SGRv2 when trained on 1000 demonstrations, falls short when the number of demonstrations is reduced to 50 or 200. This highlights the superior sample efficiency of SGRv2. Additionally, we
18

---

Method
Avg. Success
Stack
StackThree
Square
Threading
Coffee
HammerCleanup MugCleanup
SGR
42.1
84,4
54.0
26.4
11.6
41.6
38.4
38.4
PointNeXt
42.3
90.4
72.4
12.4
12.8
36.4
33.6
38.0
2D BC
32.3
84.4
35.6
13.2
6.8
26.8
2D BC-RNN
96.0
56.8
34.8
82.8
46.0
SGRv2 (ours)
66.2
96.4
84.2
56.4
56.0
86.0
46.2
38.4
Table 7: Performance on MimicGen with 1000 demonstrations.
Method
Avg. Success
Stack
StackThree
Square
Threading
Coffee
HammerCleanup
MugCleanup
R3M
5.3
34.5
0.3
0.0
0.5
1.2
0.3
0.0
2D BC
10.6
31.2
3.6
0.4
4.4
22.8
8.0
3.6
2D BC-RNN
10.0
30.0
3.2
0.0
0.8
24.0
4.0
8.0
SGRv2 (ours)
26.0
81.2
37.9
2.8
6.7
27.9
16.1
9.7
Table Performance of additional baselines on MimicGen with 50 demonstrations.
recognize that SGRv2 and methods like BC-RNN, which model temporal dependencies, are com- plementary. Integrating temporal dependencies into SGRv2 presents promising avenue for future research.
E Robustness to Visual Distractors
As illustrated Section 4.2, the predicted per-point weights of SGRv2 effectively focus on locations that align well with object affordances. This raises the question: does this emergent capability make model robust to distractors the scene that it has never seen before? In Section 4.3, we present experiments involving distractor cups of different colors; nonetheless, it remains be seen how SGRv2 performs in the presence of completely unseen objects and whether it will disregard them.
To address this question, we conduct additional experiments where we randomly introduce task- irrelevant objects (such as YCB [63] objects, basketballs, etc.) as visual distractors into the RL- Bench environments for 3 tasks (meat off grill. phone on base, and push buttons). Refer Figure 7 for visualization. We then test the SGRv2 and RVT models, which were previously trained on data without distractors, in these modified environments. The results, shown in Ta- ble 10, indicate that SGRv2 is more robust these distractors compared to RVT. This suggests that SGRv2 can effectively focus relevant areas even in the presence of unseen objects, demonstrating robustness to visual distractors.
Figure RLBench tasks with visual distractors.
Detailed Results
For our simulation experiments using the SGRv2 on RLBench with 5 demonstrations (mentioned Table 1) and on ManiSkil12 and MimicGen with 50 demonstrations (mentioned in Table 2), we employed random seeds ensure the reliability of our results. In the main body of the paper, present averaged results for clarity. Here include both the mean and standard deviation derived from our simulation results. The results for RLBench are shown in Table 11, and the results for ManiSkil12 and MimicGen are presented in Table 12.
also report the ablations mentioned Table for each task Table 13.
19

---

Method
Avg.
Stack
StacKThree
Square
Threading
Coffee
HammerCleanup MugClcanup
2D BC
21.9
62.8
23.6
4.8
24.8
8.4
2D BC-RNN
41.1
84.0
51.6
15.2
16.8
69.6
22.4
28.4
SGRv2 (ours)
55.8
95.2
80.4
32.4
42.2
74.4
38.0
28.2
Table 9: Performance on MimicGen with 200 demonstrations.
Meat Off Grill
Phone On Base
Push Buttons
RVT on w/o distractors RVT on env w/ distractors
90.5
62.3
90.4
65.0
2.5
67.5
SGRv2 w/o distractors
96.5
84.1
93.2
SGRv2 on env w/ distractors
92.4
80.4
81.7
Table 10: Performance evaluation in environments with and without visual distractors.
Avg.
Open
Open
Water Plants
Toilet
Phone
Put
Take Out
Open
Method
Success
Microwave
Door
Seat Up
On Base
Books
Umbrella
Fridge
R3M
4.7
0.9 0.6
36.4 37
+.7 3.7
15.5 2.1
0.0 0.0
0.9
5.2 8.7
114
PointNeXt
25.3
6.3
60.9
5.6 4.6
49.9 = 14.7
46.4 4,9
57.5 +.2
37.5
9.2 400
PerAct
22.3
4.3
59.6 16.0
28.
=1.3
0.0
+4.
3.1 750
SGR
23.6
6.2
55.3
=4.
30,7
47.2
292
7.1
RVT
40.4
=8.
71.2
34.8 :33
4.7
6.
4
10.9
85,3
24.0
SGRv2 (ours)
53,2
=
76.8 8.0
38.0 1.7
89.6
84.1 4.5
63.7 11.8
74.5
13.2 3.4
Open
Slide
Sweep To Dustpan
Meat Off Grill
Turn
Put In
Close Jar
Drag Stick
Stack
Method
Drawer
Block
Tap
Drawer
Blocks
R3M
0.0
24.0
0.4
0.1
26.1
0.0
0.0
0.3
0.0
PointNeXt
21.7 20.4
59.5
22.11
42.0 34.7
59.9
48.7 13.4 17.1 27.8
36.0
18.5
PerAct
56.4
47.5 24.3
2.8
0.4
85.9
1.0
0.1
0.6
10.3
1.7
SGR
31.9 62
72.0
43.6
8.4
52.7
34.4
8.3
13.3
64.4
11.4
RVT
-
85.1
19.6
90.5
38.4
19.6
25.2
45.7
10.9
SGRv2 (ours)
3.1
100.0
61.5
96.5
87.9
75.9
25.6
94.9
0.6
17.5
Screw Bulb
Put In
Place
Put
Sort Shape
Push Buttons
Insert
Stack Cups
Place
Method
Safe
Wine
Cupboard
Peg
Cups
R3M
0.0 0.0
0.3
0.4
0.4
0.0
0.0 0.0
6.8 3.7
0.0 = 0.0
0.0 0.0
0.0 = 0.0
PointNeXt
4.1 1.5
12.1 142
31,5
4.5
3.6
0.4 104
22.0
38.1
0.1 1.2
4.4 388
0.4 1.4
PerAct
4.2
0.9 0.9
1.7
0.4
0.4 104
83.1 5.3
1.9
0.1 02
0.7 06
SGR
0.9 0.8
16.9 2.2
24.7 58
0.1
0.1 0.2
12.0
14
0.1 0.2
0.0 0.0
1.1 0.9
RVT
24.0
30.7 249
92.7
=06
5.6
1.6
90.4
4,0
3.1
1.2 0.7
SGRv2 (ours)
24.1 0.6
55.6 8.0
53.1 774
20.3 992
1.96
93.2
5.3
4.1 1.4
21.3 118
1.7
Table 11: RLBench results (%) on 5 demonstrations with mean and standard deviation.
Method
Avg. Success 1 Avg. Rank J.
LifiCube
PickCube
StackCube
PickSingleYCB
PointNeXt
16.8
2.5
50.8 15.2
4.7 0.4
10.6 4.3
1.1 + 0.1
SGR
14.9
2.5
26.9 4.0
12.2 3.1
3.5 2.2
17.0 0.2
SGRv2 (ours)
55.8
1.0
80.5 = 7.3
72.9 4.1
27.7 #
42.2 = 2.3
Method
Avg. Success
Avg.
Stack
StackThree
Square
Threading
Coffee
HammerCleanup
MugCleanup
PointNeXi
2.9
1.
11.7
7.1 +09
SGR
14.2
2.0
50.8 477
5.6+17
1.3 0.5
40-08
14.1+20
14.1
9.7 224
SGRv2 (ours)
26.0
1.0
81.2
37.9
6.7
27.9.70
16.1
9.7
Table 12: ManiSkil12 and MimieGen results (%) on 50 demonstrations with mean and standard
deviation.
20

---

Avg.
Open
Open Door
Water Plants
Toilet Seat Up
Phone On Base
Put Books
Take Out
Open
Method
Success
Microwave
Umbrella
Fridge
SGRv2
53.2
27.2
76.8
38.0
89.6
84.1
63.7
74.5
13.2
SGRv2 w/o decoder
21.3
4.4
68.4
12.4
38.8
32.8
27.2
35.6
6.8
SGRv2 w/ absolute pos prediction
21.0
8.0
$7.6
21.2
17.2
14.8
21.2
44.4
SGRv2 w/ uniform point weight
21.6
82.4
28.8
32.0
68.0
14.0
SGRv2 dense supervision
40.1
6.4
59.2
6.8
54.4
78.4
60.8
68.0
6.0
Open
Slide
Sweep
Mear Off
Turn
Put
Close
Drag
Stack
Method
Drawer
Block
Dustpan
Grill
Tap
Drawer
Jar
Stick
Blocks
SGRv2
81,3
100.0
61.5
96.5
87.9
94.9
17.5
SGRv2 w/o decoder
14.0
78.8
15.6
40.8
46.8
4.0
18.8
0.8
0.0
SGRv2 w/ absolute prediction
20.4
99.2
50.8
10.8
68.8
4.0
6.4
54.8
1.2
SGRv2 w/ uniform point weight
92.8
100.0
72.8
90.8
67.6
74.0
43.6
97.6
1.6
SGRv2 w/o dense supervision
75.2
92.8
19.2
72.0
84.0
42.0
28.8
59.6
43.2
Screw
Put In
Place
Put In
Sort
Push
Insert
Stack
Place
Method
Bulb
Safe
Wine
Cupboard
Shape
Buttons
Peg
Cups
Cups
SGRv2
24.1
55.6
53.1
20.3
1.9
93.2
4.1
21.3
1.6
SGRv2 w/o decoder
3.2
16.4
27.6
0.0
0.4
0.8
1.6
SGRv2 w/ absolute pos prediction
0.4
1.6
3.2
0.0
0.0
35.2
1.6
0.0
0.0
SGRv2 w/ uniform point weight
1.6
32.8
50.4
1.2
0.8
64.4
0.0
5.2
0.8
SGRv2 w/o dense supervision
0.0
30.4
52.0
0.4
0.0
100.0
0.0
0.0
2.4
Table 13: Ablations results (%) for SGRv2 on RLBench with metries for each task.
21


Reviews:
[{"review_id": "9rHSUMRtZN", "reviewer": "Reviewer_LbA4", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 2, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposes SGRv2, an imitation learning method that is sample efficient in the number of demonstrations used. The method builds on prior work (SGR), with several improvements added. The method is shown to improve over other baselines when trained on limited quantities of demonstration data, and is also demonstrated on real hardware.", "summary_of_recommendation": "More thorough evaluations are needed to assess the quality of the proposed method.", "technical_quality": 2, "clarity_of_presentation": 3, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "See the points raised above.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8Ywgq5b6w3", "reviewer": "Reviewer_AWsF", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper presents a visuomotor policy that incorporates novel visual and action representations for sample efficient imitation learning. The proposed method, SGRv2, builds upon its predecessor SGR but introduces the use of local per-point features. Instead of using global features to predict actions, SGRv2 uses an encoder-decoder architecture to extract per-point features from point cloud inputs. The method then predicts actions relative the end-effector at each point and aggregates these per-point predictions with a learned a weighted average.", "summary_of_recommendation": "This work presents an interesting method to improve sample efficiency of visuomotor policies for imitation learning. The method is not ground-breaking but are well-designed and accompanied with convincing results. Some additional comparisons and analysis can make this paper stronger.", "technical_quality": 4, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "See above.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mCQkzkbTBy", "reviewer": "Reviewer_tgDw", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 4, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "SGRv2 is an imitation learning framework that try to leverage the locality in robot actions to improve sample efficiency. The framework uses dense point-wise feature and predicts actions using the weighted average from such dense features. Another design choice is to encode the action in a relative coordination frame. With these inductive biases SGRv2 achieves higher task success rate using less training data.", "summary_of_recommendation": "Overall I think it’s a nice paper and recommend accepting. The method used here is quite straight-forward and demonstrates strong empirical result.", "technical_quality": 4, "clarity_of_presentation": 5, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "- Does the proposed method improve generalization to unseen object configurations, such as different object position/orientation/view point? For example in RLBench the train/test variance is very small. I imagine the relative encoding and use of 3D/point cloud would help a bit on that.\n- For the real setup, long-horizon tasks, are the testing setups identical to the training ones?\n- Are the models trained per-task or is there one model for all the tasks, with 5 demonstrations each?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The reviewers found some strengths in this submission, but also clearly articulated some questions for the rebuttal phase. 
Thank you for your detailed responses.
Decision: Accept