A Universal Semantic-Geometric Representation for Robotic Manipulation
Authors: Tong Zhang
Yingdong Hu
Hanchen Cui
Hang Zhao
Yang Gao
Venue: CoRL 2023 Poster (2023)
Keywords: Representation Learning
Robotic Manipulation

Abstract: Robots rely heavily on sensors, especially RGB and depth cameras, to perceive and interact with the world. RGB cameras record 2D images with rich semantic information while missing precise spatial information. On the other side, depth cameras offer critical 3D geometry data but capture limited semantics. Therefore, integrating both modalities is crucial for learning representations for robotic perception and control. However, current research predominantly focuses on only one of these modalities, neglecting the benefits of incorporating both. To this end, we present $\textbf{Semantic-Geometric Representation} (\textbf{SGR})$, a universal perception module for robotics that leverages the rich semantic information of large-scale pre-trained 2D models and inherits the merits of 3D spatial reasoning. Our experiments demonstrate that SGR empowers the agent to successfully complete a diverse range of simulated and real-world robotic manipulation tasks, outperforming state-of-the-art methods significantly in both single-task and multi-task settings. Furthermore, SGR possesses the capability to generalize to novel semantic attributes, setting it apart from the other methods. Project website: https://semantic-geometric-representation.github.io.

A Universal Semantic-Geometric Representation for Robotic Manipulation
Tong Zhang!.2.5.
1233+ Yingdong Hu2.2,3+ Hanchen Cui
Hang Zhao 1,.3 Yang Gao 1.2.f
Tsinghua University
2 Shanghai Artificial Intelligence Laboratory 3 Shanghai Qi Zhi Institute (zhangton20, huyd21 (rmaiismmaall @mails. tsinghua. edu. en, hanchen. cui@sjtu. edu. en, (hangzhao, g taingguasinggaa.... edu. cn
Abstract: Robots rely heavily on sensors, especially RGB and depth cameras, to perceive and interact with the world. RGB cameras record 2D images with rich semantic information while missing precise spatial information. On the other side, depth cameras offer critical 3D geometry data capture limited seman- ties. Therefore, integrating both modalities crucial for learning representations for robotic perception and control. However, current research predominantly fo- cuses only one of these modalities, neglecting the benefits of incorporating both. To this end, we present Semantic-Geometrie Representation (SGR), a universal perception module for robotics that leverages the rich semantic infor- mation of large-scale pre-trained 2D models and inherits the merits 3D spatial reasoning. Our experiments demonstrate that SGR empowers the agent to success- fully complete diverse range of simulated and real-world robotic manipulation tasks, outperforming state-of-the-arr methods significantly in both single-task and multi-task settings. Furthermore, SGR possesses the capability generalize novel semantic attributes, setting it apart from the other methods. Project website: semantic- gewmetric-reppeeeeeeeeeeeeeeeeee
www.oo..gittiii github.io io.
Keywords: Representation Learning, Robotic Manipulation
Introduction
In the field of robotics, sensors play indispensable role in enabling robots to perceive and interact with the world [1]. Among the various types of sensors, RGB cameras and depth cameras stand out for their accessibility and affordability [2]. RGB cameras capture high-resolution 2D images that are rich in semantic information, providing a high-level understanding of the scene [3, 4J. However, hard to use them to reason about complex spatial relationships. On the other hand, depth cameras provide 3D geometry information, which crucial for accurate fine-grained manipulation [5, 6, 7]. Nevertheless, they have limited semantic understanding. As shown in Figure it is of utmost importance in robot learning to utilize these two complementary modalities to develop general- purpose robots that can learn a diverse set of manipulation skills. In this paper, we ask the following question: in the new era of pre-trained large vision models, how can we learn representation for robots that integrates both semantic understanding and 3D spatial reasoning?
A growing body of research is focused on utilizing pre-trained 2D vision models learn visual representations that possess general knowledge of world [8, 9, 10, 11. 12, 13]. This high- level semantic information is invaluable for generalizable robotic control. However, the current emphasis has largely been on pre-training on huge volumes of 2D images [14, 15], despite the fact that our world is intrinsically 3D. When confronted with the immense complexity real-world environments, a robot with only 2D prior knowledge may face challenges such as partial occlusion
"qqual contribution Corrreponndig author
7th Conference Robot Learning (CoRL. 2023), Atlanta, USA.

---

Large-scale 2D Images inftrmation
Robotic Manipulation
Clouds Accurate goometic information
Semantic-Geometric Representation
Figure 1: Semantic-Geometric Representation (SGR): Leveraging semantic information from massive 2D images and geometric information from 3D point clouds, we present a representation model SGR that enables the robots solve range of simulated and real-world manipulation tasks.
and geometric shape comprehension [16, 17]. Therefore, it is essential to develop models that can handle 3D visual data to enable robots to perform spatial reasoning effectively.
One natural and intuitive solution to develop pre-trained 3D vision models. However, despite some attempts in the vision community [18, 19, 20, 21], current models are not yet sufficiently transferable to robotic manipulation tasks due to two main reasons. (i) The expensive data acquisition and labor-intensive annotation process result in lack of large-scale 3D datasets [22, 23). (ii) 3D point clouds with sparse structural patterns do not provide diversified semantics compared to colorful 2D images [24], thereby constraining the generalization capacity of 3D pre-training.
To address the challenges associated with using 2D or 3D vision models independently, we propose Geomeric Representation (SGR), hybrid perception module for robotics that captures both high-level semantics and low-level spatial patterns. As illustrated in Figure 2, we first utilize large vision foundation model [25, 26, 27], pre-trained massive amounts of internet data, to encode semantic feature maps from 2D images. Secondly, the context-rich 2D feature vectors are back-pre pack-projected into 3D space and combined with the point cloud features that are extracted from point clouds using point-based network [28, 29]. These fused features are fed into a number of set abstraction (SA) blocks [30], which jointly model the cross-modal interaction between 2D semantics and 3D geometry information. Finally, based on the output representations from the SA blocks, we predict the robotic action to execute.
We learn control policy through behavior cloning [31] on broad spectrum of robotic manipulation tasks from the Robot Learning Benchmark (RLBench) [32]. In both single-task and multi-task learning scenarios, our SGR consistently outperforms a wide range of 2D and 31) representations, such as R3M [12], CLIP [26], and PERACT [33], by a significant margin. This striking performance demonstrates the effectiveness of integrating semantic and geometric representations synergistically. Furthermore, SGR stands the approach capable of generalizing to previously unseen semantic attributes, including colors and shapes. To further validate its effectiveness, Franka Emika Panda robot, training multi-task agent across 8 real-world tasks. Remarkably, SGR significantly surpasses a strong baseline (P (PERACT) on tasks that involve visual distractors. All these findings provide compelling evidence that SGR possesses the potential to serve as a universal representation for general-purge robot learning.
assess SGR using
2 Related Work
2D Representation Learning for Robotics. wealth of prior approaches learn visual representa- tions from in-domain data taken directly from the target environment and task. These techniques encompass broad spectrum, ranging from data augmentation [34, 35, 36] and forward dynamics modeling [37, 38] to the utilization of task-specific information [39, 40]. However, recent studies

---

shift their focus towards acquiring more generalized representations. They achieve this by employ- ing vision models pre-trained on large-scale out-of-domain data as the cornertoones for visuo-motor control [11, 41]. RRL [42], PIE-G [43], and MVP [10] demonstrating the effectiveness of supervised or self-supervised pre-trained representations for RL agents. PVR [9] and R3M [12] find that vision models pre-trained on real-world data enable data-efficient behavior cloning. VIP [44] proposes a pre-trained representation capable of producing dense reward signals. Hu et al. [8] conducts the first thorough empirical evaluation of pre-trained visual representations across different downstream policy learning methods. However, these approaches lack an explicit consideration of 3D geometry, which limits their capacity to develop highly accurate spatial manipulation skills in robotics.
3D Representation Learning for Robotics. The 3D perception module for robotics can be classi- fied into the following three types: (1) Projection- models: Many approaches utilize multi-view images, which represent the projections of 3D environment onto various image planes, direct inputs for robots [45, 46, 47]. However, significant drawback of these methods is the loss of geo- metric information during the projection stage. (2) Voxel-basee models: Another way to represent 3D environment is voxelization, which extends the concept of 2D pixels [48, 49]. Recent works like C2FARM [50] and PERACT [33] employ a voxelized observation and action space for 6-DoF ma- nipulation. Nevertheless, these models underutilize the sparsity of point sets 3D and suffer from massive computational and memory costs. (3) Point-based models: In contrast to the previous two types, point- based models, such as PointNet [28), PointNet++ [30], directly and efficiently process point clouds. A plethora of approaches in robotics use PointNet PointNet++ as a visual feature encoder [51, 52, 53, 54, 55, 56]. In this paper, we utilize point- bast- models to extract precise 31) geometric information from point clouds due to their effectiveness and efficiency.
Fuse 2D and 3D information. There rich body of literature dedicated to the study of sensor fusion, which aims to leverage the complementary nature of 2D and 3D information. In the field of autonomous driving, numerous previous works utilize camera-captured images and LiDAR point clouds to achieve accurate and robust 3D object detection [57, 58, 59, 60, 61]. Similarly, in the field of robotics, several prior studies explore the combination of color and depth information from RGB-D input to estimate object poses [62, 63, 64). One notable approach, called DenseFusion [62], processes the two data sources individually and employs a fusion network to extract pixel-wise dense feature embeddings. However, one limitation of DenseFusion is its exclusive training on data from the target domain, leading failure in generalizing novel objects. In contrast, our approach leverages visual representations pre-trained on large-scale images sourced from the web, empowering our robot to generalize to unseen attributes. This level of generalization is unattainable for domain-specifie method.
3 Method
In this section, we cover the different components of our approach. We start with a brief introduc- tion of CLIP [26] and PointNeXt [28] in Section 3.1. Next, in Section 3.2, we present Semantic- Geometric Representation (SGR), method that leverages the benefits of both pre-trained 2D models and 3D networks. Lastly, in Section 3.3, we demonstrate the integration of SGR into behavior- cloning agent for 6-DoF manipulation tasks.
3.1 Background
Contrastive Language-Image Pre-training (CLIP) [26] has emerged as a simple yet powerful methodology for learning representations by pre-training on extensive image-text pairs from the web. Initially, CLIP employs separate modality-specific models to embed images and text. Subse- quently, these vectors are projected into a shared embedding space and normalized. The InfoNCE loss [65] is calculated based on these final embeddings, utilizing corresponding images and cap- tions as positive pairs and treating all non-matching images and captions as negative pairs. The learned visual representation not only exhibits impressive semantic discriminative power and zero- shot transferability [26], but also demonstrates exceptional generalization performance in robotic
3

---

(Optional) Instruction: "Water the plant"
Encoder
Semantic Branch
Image Encoder
Visuo Grundrrg
Otn Untampre
Cheirel Cfe Dommart
Pusion Network
Subsampah
atrans
Abetraaton
images mulliple cameras
Concat
Arot
MUP
Propriocaptivee data
aupen
Gcollide
MUP
Suboumple
Goouing
Rodacion
ML
point
Figure 2: Architecture. The semantic branch generates point-wise semantic features from the multi- view images. Simultaneously, the geometric branch extracts geometric features from point cloud. Finally, we fuse the two kinds of complementary features and predict the action to execute.
manipulation and navigation tasks [66, 67]. Although we utilize CLIP extract semantic features, our perception module is also compatible with wide range of pre-trained 2D vision models [68, 13].
PointNet, PointNet++, and PointNeXt. PointNet [28] is pioneering and classical model that di- rectly operates on unordered point clouds, capable of learning global point cloud features through shared multi-layer perceptrons (MLPs) and a max-pooling operation. PointNet++ [30] extends PointNet by utilizing a hierarchical neural network to handle local and global features in a more efficient manner. PointNeXt [29] recent study that revisits PointNet++ with improved training and scaling strategies enhance its performance to the state-of-the-art level. Because of the sim- plicity, efficiency, and effectiveness of PointNeXt, we use PointNeXt in our experiments to enable robots to obtain accurate 3D spatial information.
3.2 Semantic-Goometit Representation
Our objective is to design universal perception module for robotics that incorporates various sen- sory inputs, such as RGB images, point clouds, and proprioception data (e.g., gripper open state). This module generates a Semanti--Gemeeric Representation (SGR) capable of capturing both high-level semantics and low-level spatial patterns. The resulting representations are utilized to pre- dict robotic actions for manipulation tasks. Moreover, for language-condittonee tasks, our module can process natural language and ground semantic concepts present in the images. The module comprises three key components: (1) semantic branch, (2) geometric branch, and (3) fusion network. Figure 2 provides overview of our design.
Semantic branch. Given a set of RGB images {40, obtained from K calibrated cameras, we first utilize a frozen pre-trained 2D model & (c.g., the visual encoder CLIP) to extract multi-view image features {G(I,)) K If an language instruction trained language model H. (e.g., the language encoder of CLIP) obtain the language encoding H(S). Following MaskCLIP [69] and PROGRAMPORT [70], we use visual grounding module to align each feature map G(T,) with the language encoding H(S)), resulting in aligned feature maps (M) For details the visual grounding module, please refer to Appendix B.I. Next, upsample the visual feature maps { (or aligned feature maps {M, to the same size as the input images by bilinear interpolation and downsample the channels via x convolution, resulting in a image size. These encoded 2D features, which encompass rich high-level semantics, projected back into 3D space, yielding point-wise semantic features for the point cloud. We formulate them as F3D E TR. XC, where N = K x H W.
provided for task, we employ pre-
features denoted as ( where F; IR #x and H. W represent the

---

Geometric branch. The raw point cloud, denoted as P E RNx3, is generated by utilizing multi- view depth images and known camera parameters (i.e., camera intrinsics and extrinsics). Addition- ally, we incorporate robot proprioceptive information 2 by applying a linear layer, which produces a D-dimensional vector that is attached to all the points. This augmented point cloud is represented as P' E (3++))+ Subsequently, we employ a point-based network (e.g.. a hierarchical PointNeXt) to process the point cloud P'. This processing step yields downsampled point geometric features, denoted as G E MR where M < N. While these geometric features effectively capture local 3D structure and spatial characteristics, they lack the understanding of semantic information.
Fusion network. To integrate the two complementary branch, we begin by processing the point-wise semantic features F3D using the same point subsampling procedure as the geomet- ric branch. The subsampled semantic features are denoted as F&b RRMxC, Next, we fuse the semantic features and geometric features through channel-wise concatenation, expressed as PR Concat( suh. G) IR- +C2) Finally, the fused features undergo several set abstrac- tion blocks [30], enabling a cohesive modeling of the cross-modal interaction between 2D seman- ties and 3D geometric information. The resulting global feature represents our Semmntic-Geomerric Representation, which utilized for action prediction. SGR back back-projects 2D feature maps to 3D and fuses them with point cloud features directly in 3D space. This process fully utilizes the seman- tic information from pre-trained models and leads better alignment with geometric information.
3.3 Robot Learning Framework
Behavior Cloning. Hu et al. [8] conduct large-scale benchmarking study of pre-trained vision models using different policy learning methods. Their investigation show that evaluation based on reinforcement learning (RL) is highly variable and therefore unreliable, and further advocate for using more robust methods like behavior cloning (BC) [31]. Heeding this advice, we investigate the learning of 6-DoF manipulation tasks using BC. Inspired by PERACT [33], we formulate BC training as task of predicting the "next best action". At each timestep, the robot receives an obser- vation O comprising multi-view RGB images (4+)==1- organised point cloud P, and proprioceptive data 2. Leveraging the perception module introduced in Section 3.2, the robot extracts the SGR representation and predicts the next best action in terms of target translation, rotation, and gripper state. Subsequently, this action is executed through a motion planner, and the process repeats until termination. Following James et al. [71, 50], all predicted robot actions are keyframe actions that capture bottleneck end-effector poses (i.e., joint velocities close to zero and unchanged gripper open state). This keyframe discovery strategy allows for the circumvention of direct predictions of long sequences of noisy actions, thus enabling more efficient policy learning. The robot is trained through supervised learning with input-keyframe action tuples from dataset of expert demonstrations.
Training Details. Following the setup in PERACT [33], an action a consists of the 6-DoF' pose (translation and rotation), gripper open state, and whether the motion-planner used collision avoid- ance to reach an intermediate pose: = {Orrans: Gyot: Gopens acollide) ). For rotations, the ground-truth action is represented as one-hot vector per rotation axis with R rotation bins: Oro E (3600/ R)
(R = 5 degrees in our implementation). Open and collide actions binary one-hot vectors: aopen IR² acollide E R², The exception is translation, which is represented by continuous 3D
vector: Gurans E 3R33. In this way, our loss objective is as follows:
Coral Xrnns + A2Cror + AaCopen A4
where Lirass is L1 loss for translation regression, and Lro. Lopens and Leutlide are cross-entropy losses for corresponding classification. Since these two kinds of losses have different scales, in exper- iments, we set \ = 300 and \ A3 A = 1. We use Adam [72] to optimize the model.
We utilize several data augmentation methods [29] to enhance the robustness and generalization of the model. During training. the translation is perturbed with +50. m, and colors are randomly replaced with zero values, which are called translation perturbations and color drop respectively. Additionally, adopt the point resampling strategy in both training and evaluation. For more details, refer to Appendix B.2,
5

---

Method
open microwave
open
water plants
toilet seat
phone on
put books
take out
open fridge
Average
door
base
umbrella
R3M
7.0
24.2
1.2
59.7
0.0
15.3
28.3
4.7
17.6
CLIP
9.1
68.4
4.1
40.1
7.3
38.0
66.7
7.4
30.2
PointNeXIyLIP
9.4
60.0
10.4
63.0
35.6
41.8
49.8
15.3
35.7
PointNeXlus
17.0
63.8
28.2
52.7
65.9
53.1
95.1
7.3
47.9
PERACT
26.9
47.8
41.7
67.1
0.0
63.7
94.4
10.1
44.0
SGR (Ours) 52.6
80.2
40.2
80.1
81.1
84.8
94.7
34.8
68.6
Table 1: Single- Task Test Results. All numbers represent percentage success rates averaged over seeds. See Appendix E for standard deviation. On average, our SGR outperforms PointNeXtr.ss the most competitive baseline, with an improvement of 1.43x.
Experiments
Our experiments are designed to answer the following questions: (1) To what extent does SGR outperform methods exclusively concentrating on semantic or geometric information individually? (2) it feasible to train multi-task model capable of handling all tasks, and how well does perform? (3) Can SGR exhibit generalization capabilities previously unseen semantic attributes such colors and shapes? (4) How well does SGR perform real-world tasks?
4.1 Simulation Setup
Environment and Tasks. Our simulated experiments are conducted on RLBench [32], a large-scale benchmark and learning environment designed for vision- guidddd manipulation research. The visual observations are obtained from four RGB-D cameras positioned at the front, left shoulder, right shoulder, and wrist of Franka Emika Panda robot. All cameras have a resolution of 128 128 and noiseless. We use 8 tasks with 100 demonstrations per task for training. For more details about individual tasks and language-conditioned tasks, please Appendix A.
Evaluation. We employ a meticulous evaluation protocol to minimize variance in our results. Specifically, for every task, we train an agent for 40,000 iterations and save checkpoints every 1,000 iterations. Then we evaluate the last 20 checkpoints for 25 episodes and mark the top best-performing checkpoints. Lastly, evaluate the top checkpoints for 100 episodes and get the average success rates
test performances. The experiments are conducted with 3 seeds.
Baselines. To verify the effectiveness of SGR, we compare with the following baselines: 2D representations: We evaluate CLIP [26] and R3M [12]. Notably, R3M is pre-trained on diverse human video datasets and showcases remarkable performance on robotic manipulation tasks. For fair comparisons, we utilize frozen CLIP or R3M to process RGB images, employ a separate 2D CNN process depth images, and subsequently fuse the two resulting features together [73].
3D representations: We examine PERACT [33], PointNeXtras [29], and PointNeXtuur [74]. PER- ACT is voxel-based model that utilizes a Perceiver Transformer [75] to process voxelized ob- servations. As for the point-based models, we consider both a Breniing-roo--ereee (PointNeXtuas) and a frozen XointNeXt that has been pre-trained by ULIP [74] (PointNeXtyLIv) ULIP a recent 3D representation learning method that aligns features from images, text, and point clouds in the same space, following the philosophy of CLIP. Importantly, irrespective of whether 2D 3D representations are employed, all the baselines utilize the same input modalities (RGB-D im- ages) and the same multi-view (4 cameras) observations to ensure fair and meaningful comparisons.
PointNeXt
4.2 Simulation Results
Single-Task Performance. Table presents a comprehensive comparison of success rates achieved through single-task imitation learning using different representations. Initially, we observe a gen- eral underperformance of 2D representations compared their 3D counterparts. This finding em- phasizes the critical role of 3D geometric information in accomplishing fine-grained manipulation tasks. Among the 2D representations, CLIP demonstrates superior performance compared to R3M, suggesting that pre-trained R3M exhibits less transfcrability to the RLBench environment. Regard-

---

CLIP PointNeXIL.ss
PERACT
SGR (Ours)
100
80
82
(96)
64.
05.3
60
36.7
36
20
72.
19.3
21.3
ogen microwave
open
Water
phone on base
put bookss
open fridge
Average
plants
seat up
Figure 3: Multi-Task Test Results, We report the success rates averaged over 3 seeds. We observe that across 8 tasks SGR consistently outperforms baselines like CLIP, PointNeXtLis. and PERACT.
ing 3D representations, Learning-from-Scraate PointNeXt (PointNeXtt.s)) outperforms its frozen counterpart pre-trained using ULIP (PointNeXtuurr)) We hypothesize that this discrepancy arises due to the current focus of 3D pre- training methods on learning from small object-level datasets [22], resulting in representations that are less suitable for scene-level environments typically encountered in robotic manipulation tasks. Furthermore, PointNeXIr.ss achieves comparable average results to PERACT while significantly reducing both computational and memory costs. Ultimately, our SGR proves to be the most effective representation, outperforming the second-best method by over 20% on average and yielding superior performance in 6 out of 8 tasks. These results highlight the benefits of capturing both 2D semantics and 3D spatial patterns for robotic manipulation. See more results about ablations and single- view comparisons in Appendix G and Appendix H, respectively.
Multi- Task Performance. In practical scenarios, our objective is ensure that robots acquire broad range of skills, rather than focusing single skill alone. With this goal in mind, train single agent on all 8 tasks, while also providing language instructions for each task to enhance the robot's 's ability to differentiate between them. For detailed information on the training of this Lunguage-condttoons behavior-cloning agent, please refer to the Appendix B.2. We compare our method, SGR, with three strong baselines and summarize the results in Figure SGR demonstrates exceptional performance in multi-task scenarios, achieving an average accuracy of 61.4%. This outcome underscores the advantageous synergistic effect achieved by considering both semantic and geometric information, even in diverse multi-task settings. However, it is surprising to note that the performance of all multi-task models declines to varying extents when compared to their single- task counterparts. Notably, this phenomenon has also been observed in PERACT [33] and other recent studies [76, 77]. While multi-task training holds great promise for knowledge sharing among related tasks and facilitating more efficient learning, presents numerous optimization challenges in practice [78, 79]. We hypothesize that employing advanced optimization algorithms [80, 81, 82] may help alleviate the negative transfer effects. We leave this exploration future work.
Generalizing to Novel Attributes. A remarkable feature of human manipulation skill is that, once humans have learned to manipulate a category of objects, they are able to manipulate even unseen objects, regardless of their variations in color and shape. To evaluate whether an agent exhibits a similar level of generalization, we introduce three tasks: 1) reaching to a colored ball (reach target), 2) sliding block onto a colored square target (slide block), and 3) reaching specified shape (reach shape). Notably, the reach target and slide block tasks examine the agent's is ability generalize across different colors, with training conducted 10 colors and testing on 10 unseen colors. The reach shape task assesses the ability to generalize across various shapes, where training encompasses 3 shapes while testing involves unseen shapes. Language instructions are employed to specify the desired color or shape. See Appendix A for more task details.
Figure 4 demonstrates that SGR achieves significantly higher performance when confronted with unseen colors shapes, highlighting its remarkable generalization ability. This exceptional capac- ity primarily stems from the semantic branch of SGR (a pre-trained CLIPP), which has undergone
7

---

reach
target Unseen
slide block
reach shape
Method
Soen
Seen
Unseen
Seen
Unseen
R3M
20.2
5.5
11.0
3.0
78.1
6.7
CLIP
42,0
17.7
49.0
16.3
95.5
0.2
PointNeXtuup
45.9
6.7
PointNeXt..s
99.7
4.0
96.5
6.2
99.8
188
DenseFusion
71.8
9.5
26.5
16.5
99.3
3.8
spene
Unsezn
SGR (Ours) 94.0
36.8
97.7
43.0
99.9
27.7
Figure 4: Left: For the slide block task, we train the agent on 10 colors and evaluate it 10
unseen colors. Please see Appendix A for reach target and reach shape tasks. Right: Our SGR is the only method that demonstrates the ability to generalize across various colors and shapes. PointNeXtuLiP only accepts point coordinate input, thus the first two tasks are not applicable to it.
extensive exposure to a wide range of semantic attributes during its pre-training process on millions of image-caption pairs. In contrast, if we initialize the semantic branch randomly and train it from scratch, the resulting model would resemble DenseFusion [62]. As depicted in Figure 4, DenseFu- sion exhibits limited generalization ability. However, crucial to recognize that relying solely on semantic information proves inadequate. Even prominent pre-trained models such CLIP and R3M encounter challenges in achieving satisfactory performance the seen split of certain tasks, largely due to their inability to capture precise 3D spatial information.
4.3 Real-Robot Results
Finally, we evaluate the performance of SGR on real-world tasks, We design set of manipulation tasks using a Franka Emika Panda. See our website for qualitative results that showcase the diversity of our tasks. We collect 8 demonstrations for each task and train multi-task agent on all 8 tasks. Further details regarding the real robot setup can be found in Appendix D. In Table 2, we present the success rates and compare SGR with PERACT, one of the strong baselines we evaluated in simulation. SGR achieves an impressive average success rate of 59,, while PerAct attains 48%. However, it is important to note that our method is not flawless. The most common failure scenario involves predicting grasp poses that already in proximity to the object but lack sufficient precision. because the distribution of human demonstrations is inherently multi-modal, and our translation ac- tion (which outputs a point estimate) tends to learn the average of this distribution. To address this limitation, future research could explore integrating SGR into methods that possess multi-modal modeling capabilities [83, 84].
Success out of Trials
|GR
PERACT
Hiiting Ball Putting Marker in Drawer
70%
30%
70%
60%
Moving Cup to Goal
70%
30%
Picking Red Block Putling Banana in Put Putting Apple
60%
60%
60%
50%
Opening Drawer
50%
30%
Pressing Handsan
40%
40%
Average
599
48%
Table 2: Success rates of multi- task agents on real-world tasks.
We speculate that this occurs
Discussion
Limitations. Although we uncover the significance of harnessing both semantic and geometric in- formation for robotic tasks, our current approach relies on modality-specific networks extract these two complementary types of information and subsequently combines them using a straight- forward method. However, we are thrilled by the prospect of employing unified architecture, such as multimodal Transformer [85], which would allow us to encode and fuse various modalities in a more elegant manner. This unified approach would also enable us to learn semantic-goomerric representations during a pre-training phase, utilizing a wide range of data from real-world.
Representation (SGR), perception module for robotics that integrates high-level semantic understanding and low-level spatial reasoning. Through extensive evaluations in both single-task and multi-task learning scenarios, we observe remarkable performance improvements when employing SGR. Importantly, SGR exhibits an impressive ability to generalize to novel attributes, a pivotal factor in achieving general general-purpose purpose manipulation skills.
Conclusion. We present
8

---

Acknowledgments
This work is supported by the Ministry of Science and Technology of the People's Republic of China, the 2030 Innovation Megaprojects "Program New Generation Artificial Intelligence" (Grant No. 2021AAA0150000), This work is also supported by the National Key R&D Program of China (2022Z0016170)).
References
[1] K. Fu, R. C. Gonzalez, and C. G. Lee. Robotics, IEEE Computer Society Press, 1983.
[2] Realsense.
https://ww: http::////wwwnnnlll.m.mm cww/ http///coneenttww/ss//////eeeeee htmm,, 2023.
[3] S. Dasari, F. Ebert, S. Tian, S. Nair, B. Bucher, K. Schmeckpeper, S. Singh, S. Levine, and
C. Finn. Robonet: Large-scale multi-robot learning. arXiv preprint arXiv:1910.11215, 2019.
[4] F. Ebert, Y. Yang, K. Schmeckpeper, B. Bucher, G. Georgakis, K. Daniilidis, C. Finn, and
S. Levine. Bridge data: Boosting generalization of robotic skills with cross-domain datasets. arXiv preprint arXiv:2109.133 13396, 2021.
[5] T. Schmidt, K. Hertkorn, R. Newcombe, Z. Marton, M. Suppa, and D. Fox. Depth-based track- ing with physical constraints for robot manipulation. In 2015 IEEE International Conference on Robotics and Automation (ICRA), pages 119-126. IEFE, 2015.
[6] D. Morrison, Corke, and J. Leitner. Learning robust, real-time, reactive robotic grasping. The International journal of robotics research, 39(2-3):183-200) 2020.
[7] Y. Cong, R. Chen, B. Ma, H. Liu, D. Hou, and C. Yang. A comprehensive study of 3-d vision- based robot manipulation. IEEE Transactions on Cybernetics, 2021.
[8] Y. Hu, R. Wang, L. E. Li, and Y. Gao. For pre-trained vision models in motor control, not all policy learning methods are created equal. arXiv preprint arXiv:2304.0455 04591, 2023.
[9] S. Parisi, A. Rajeswaran, S. Purushwalkam, and A. Gupta. The unsurprising effectiveness of pre-trained vision models for control. In International Conference on Machine Learning, pages 77359-17371. PMLR, 2022.
[10] T. Xiao, I. Radosavovic, T. Darrell, and J. Malik, Masked visual pre- pre-training for motor control. arXiv preprint arXiv:2203.061/ 2022.
[11] I. Radosavovic, Xiao, S. James, Abbecl, J. Malik, and T. Darrell. Real-world robot learning with masked visual pre-training. arXiv preprint arXiv:22 109, 2022.
[12] S. Nair, A. Rajeswaran, V. Kumar, C. Finn, and A. Gupta. R3m: A universal visual represen- tation for robot manipulation. arXiv preprint 2rXiv:2203.1260) 2022.
[13] S. Karamcheti, S. Nair, A. S. Chen, T. Kollar, C. Finn, D. Sadigh, and P. Liang. Language- driven representation learning for robotics. arXiv preprint 2rx.v:2302. 2023.
[14] O. Russakovsky. J. Deng, H. Su, Krause, Satheesh, S. Ma. Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, et Imagenet large scale visual recognition challenge. Interna- tional journal of computer vision, 115(3):11-252, 2015.
[15] C. Schuhmann, R. Vencu, R. Beaumont, R. Kaczmarezyk, C. Mullis, Katta, T. Coombes, Jitsev, and A. Komatsuzaki. Laion-400m: Open dataset of clip-filtered 400 million image- text pairs. arXiv preprint arXiv:2111.0211 2021.
[16] R. Cheng, Agarwal, and K. Fragkiadaki. Reinforcement learning of active vision for manip- ulating objects under occlusions. In Conference Robot Learning, pages 422 431. PMLR, 2018.

---

[17] T. Chen, J. Xu, and P. Agrawal. A system for general in-hand object re-orientation. In Confer- ence on Robot Learning, pages 297-307. PMLR, 2022.
[18) S. Xie, J. Gu, D. Guo, C. R. Qi, L. Guibas, and O. Litany. Pointcontrast: Unsupervised pre- training for 3d point cloud understanding. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part III 16, pages 574-591. Springer, 2020.
[19] X. Yu, L. Tang, Y. Rao, T. Huang, J. Zhou, and J. Lu. Point-bert: Pre-training 3d point cloud transformers with masked point modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19313-19322, 2022.
[20] Y. Pang, W. Wang, F. E. Tay, W. Liu, Y. Tian, and L. Yuan. Masked autoencoders for point cloud self-supervvsed learning. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part II, pages 604-621. Springer, 2022.
[21] R. Zhang, Z. Guo, P. Gao, R. Fang, B. Zhao, D. Wang, Y. Qiao, and H. Li. Point-m2ae: multi-scale masked autoencoders for hierarchical point cloud pre-training. arXiv preprint arXiv:2205. 14401, 2022.
[22] A. X. Chang, T. Funkhouser, L. Guibas, P. Hanrahan, Q. Huang, Z. Li, S. Savarese, M. Savva, S. Song, H. Su, et al. Shapenet: An information-rich 3d model repository. arXiv preprint
arXiv:1512.0300 2015.
[23] M. A. Uy. Q.-H. Pham, B.-S. Hua, T. Nguyen, and S.-K. Yeung. Revisiting point cloud classi- fication: A new benchmark dataset and classification model on real-world data. In Proceedings of the IEEE/CVF international conference on computer vision, pages 1588-1597, 2019.
[24] R. Zhang, L. Wang, Y. Qiao, P. Gao, and H. Li. Learning 3d representations from 2d pre- trained models via image-to-point masked autoencoders. arXiv preprint arXiv:22/2.06785. 2022.
[25] R. Bommasani, D. A. Hudson, E. Adeli, R. Altman, S. Arora, S. von Arx, M. S. Bernstein, J. Bohg, A. Bosselut, E. Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint 20Xi::2008.0255:: 2021.
[26] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning, pages 8748-877 8763. PMLR, 2021.
[27] L. Yuan, D. Chen, Y.-L. Chen, N. Codella, X. Dai, J. Gao, H. Hu, X. Huang, B. Li, C. Ci et Florence: A new foundation model for computer vision. arXiv preprint 21X1:22111.11432, 2021.
[28] C. R. Qi, H. Su, K. Mo, and L. J. Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 652-660, 2017.
[29] G. Qian, Y. Li, H. Peng. J. Mai, H. A. A. K. Hammoud, M. Elhoseiny, and B. Ghanem. Pointnext: Revisiting pointnet++ with improved training and scaling strategies. arXiv preprint arXiv:2206.04l 04670, 2022.
[30] C. R. Qi, L. Yi, H. Su, and L. J. Guibas. Pointnet++: Deep hierarchical feature learning on point sets in metric space. Advances in neural information processing systems, 30, 2017.
[31] D. A. Pomerleau. Alvinn: An autonomous land vehicle in neural network. Advances in neural information processing systems. 1, 1988.
[32] S. James, Z. Ma. D. R. Arrojo, and A. J. Davison. Rlbench: The robot learning benchmark & learning environment. IEEE Robotics and Automation Letters, 5(2):3019-3026, 2020.
10

---

[33] M. Shridhar, L. Manuelli, and D. Fox. Perceiver-actor: A multi-task transformer for robotic manipulation. arXiv preprint arXiv:2209.0545|, 2022.
[34] M. Laskin, A. Srinivas, and P. Abbeel. Curl: Contrastive unsupervised representations for reinforcement learning. In International Conference on Machine Learning, pages 5639-5650. PMLR, 2020.
[35] M. Laskin, K. Lee, A. Stooke, L. Pinto, P. Abbeel, and A. Srinivas. Reinforcement learning with augmented data. Advances in neural information processing systems, 33: 13:19884-19895, 2020.
[36] I. Kostrikov, D. Yarats, and R. Fergus. Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. arXiv preprint arXiv:2004. 13649, 2020.
[37] C. Gelada, S. Kumar, J. Buckman, O. Nachum, and M. G. Bellemare. Deepmdp: Learning continuous latent space models for representation learning. In International Conference on Machine Learning, pages 2170-2179. PMLR, 2019.
[38] D. Hafner, T. Lillicrap, J. Ba, and M. Norouzi. Dream to control: Learning behaviors by latent imagination. arXiv preprint arXiv: 1rXiv:/9/2.011 19/.00603, 2019.
[39] R. Jo.schkoowkki and O. Brock. Learning state representations with robotic priors. Autonomous Robots, 39:407-428, 2015.
[40] A. Zhang, R. McAllister, R. Calandra, Y. Gal, and S. Levine. Learning invariant representations for reinforcement learning without reconstruction. arXiv preprint arXiv:2006. 10742, 2020.
[41] A. Majumdar, K. Yadav, S. Armad, Y. J. Ma, C. Chen, S. Silwal, A. Jain, V.-P. Berges, P. Abbeel, J. Malik, et al. Where are we in the search for an artificial visual cortex for embodied intelligence? arXiv preprint ar30v: 1804. 2023.
[42] R. Shah and V. Kumar. Rrl: Resnet as representation for reinforcement learning. arXiv preprint arXiv:2107.03380. 2021.
[43] Z. Yuan, Z. Xue, B. Yuan, X. Wang, Y. Wu, Y. Gao, and H. Xu. Pre-trained image encoder for generalizable visual reinforcement learning, arXiv preprint (2Xiv:2212.08860 2022.
[44] Y. J. Ma, S. Sodhani, D. Jayaraman, O. Bastani, V. Kumar, and A. Zhang. Vip: Towards universal visual reward and representation via value-implicir pre-training arXiv preprint arXiv:2210.0000
2022.
[45] H. Liu, L. Lee, K. Lee, and P. Abbeel. Instruction-followingg agents with jointly pre-trained vision-language models. arXiv preprint (10t:::22101143311 2022.
[46] P.-L. Guhur, S. Chen, R. Garcia, M. Tapaswi, I. Laptev, and C. Schmid. Instruction-driven history-aware policies for robotic manipulations. arXiv preprint arXiv:2209.04899 2022.
[47] Y. Seo, J. Kim, S. James, K. Lee, Shin, and Abbeel. Multi-view masked world models for visual robotic manipulation. arXiv preprint arXiv:2302.02408, 2023.
[48] D. Maturana and Scherer. Voxnet: A 3d convolutional neural network for real-time object recognition. In 2015 JEEE/RSJ international conference on intelligent robots and systems (IROS), pages 922-92222299 928. IEEE, 2015.
[49] S. Song, Yu, A. Zeng, A. Chang, M. Savva, and T. Funkhouser. Semantic scene com- pletion from single depth image. In Proceedings of the IEEE conference computer vision and pattern recognition, pages
1754, 2017.
[50] S. James, K. Wada, T. Laidlow, and A. Davison. Coarse-to-fine q-attention: Efficient learning for visual robotic manipulation via discretisation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13739-13748, 2022.

---

[51] H.-S. Fang, C. Wang, M. Gou, and C. Lu. Craspnet-lbillion A large-scale benchmark for general object grasping. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pages 11444 1453, 2020.
[52] M. Sundermeyer, A. Mousavian, R. Triebel, and D. Fox. Contact-graspner Efficient 6-dof grasp generation in cluttered scenes, In 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 14438-13444 IEEE, 2021.
[53] W. Huang. I. Mordatch, Abbeel, and D. Pathak. Generalization in dexterous manipulation via geometry-aware multi-task learning. arXiv preprini arXiv:2111.0300 2021.
[54] Y.-H. Wu, J. Wang, and X. Wang. Leamning generalizable dexterous manipulation from human grasp affordance. arXiv preprint arXiv:2204.02320, 2022.
[55] Y. Qin, B. Huang, Z.-H. Yin, H. Su, and X. Wang. Dexpoint: Generalizable point cloud rein- forcement learning for sim-to-real dexterous manipulation. arXiv preprint arXiv:2211.0944 2022.
[56] M. Liu, X. Li, Z. Ling, Y. Li, and H. Su. Frame mining: free lunch for learning robotic manipulation from 3d point clouds. arXiv preprint arXiv:2210.04442, 2022.
[57] D. Xu, D. Anguelov, and A. Jain. Pointfusion: Deep sensor fusion for 3d bounding box estimation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 244 253, 2018.
[58] Z. Wang, W. Zhan, and M. Tomizuka. Fusing bird's eye view lidar point cloud and front view camera image for 3d object detection. 2018 IEEE intelligent vehicles symposium (IV), pages 1-6. IEEE, 2018.
[59] M. Liang B. Yang, S. Wang, and R. Urtasun. Deep continuous fusion for multi-sensor 3d object detection. In Proceedings of the European conference on computer vision (ECCV), pages 641-656, 2018.
[60] M. Liang B. Yang. Chen, R. Hu, and R. Urtasun. Multi-task multi-sensor fusion for 3d object detection. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7345-7353, 2019.
[61] S. Vora, A. H. Lang, B. Helou, and O. Beijbom. Pointpainting: Sequential fusion for 3d object detection. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 4604-4612, 2020.
[62] C. Wang. D. Xu, Y. Zhu, R. Martin, C. Lu, L. Fei-Fei, and S. Savarese. Densefusion: 6d object pose estimation by iterative dense fusion. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3343-3352, 2019.
[63] G. Zhou, Y. Yan, D. Wang, and Q. Chen. A novel depth and color feature fusion framework for 6d object pose estimation. IEEE Transactions on Multimedia, 23:1630-1639, 2020.
[64] C. Wang, Martfn-Martin, D. Xu, Lv, Lu, L. Fei-Fei, Savarese, and Y. Zhu. 6- pack: Category-level 6d pose tracker with anchor-based based keypoints. 2020 IEEE International Conference Robotics and Automation (ICRA), pages 10059-10066.. IEEE, 2020.
[65] A. d. Oord, Y. and O. Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv: 1807:03800033743 2018
[66] M. Shridhar, L. Manuelli, and D. Fox. Cliport: What and where pathways for robotic manipu- lation. In Conference Robot Learning, pages 894-906. PMLR, 2022.
12

---

[67] A. Khandelwal, L. Weihs, R. Mottaghi, and A. Kembhavi. Simple but effective: Clip embed- dings for embodied ai. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages
2022.
[68] C. Jia, Y. Yang. Y. Xia, Y.-T. Chen, Z. Parekh, H. Pham, Le, Y.-H. Sung, Z. Li, and T. Duerig- Scaling up visual and risionllanuuae representation learning with noisy text supervision. In International Conference Machine Learning, pages 4904-4916. PMLR, 2021.
[69] C. Zhou, C. C. Loy, and B. Dai. Extract free dense labels from clip. In Computer Vision- ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXVIII, pages 696-712. Springer, 2022.
[70] R. Wang, J. Mao, J. Hsu, H. Zhao, J. Wu, and Y. Gao. Programmatically grounded, composi- tionally generalizable robotic manipulation. arXiv preprint arXiv:2304. /3826, 2023.
[71] S. James and A. J. Davison. Q-attention: Enabling efficient learning for vision-based robotic manipulation. IEEE Roboties and Automation Letters, (12,1 2022.
[72] I. Loshchilov and F. Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05001, 2017.
[73] Y. Hu, R. Wang, K. Zhang, and Y. Gao. Semantic-aware fine-grained correspondence. In European Conference on Computer Vision, pages 97-115. Springer, 2022.
[74] L. Xue, M. Gao, C. Xing. R. Martin-Martin Wu, C. Xiong, R. Xu, J. C. Niebles, and Savarese. Ulip: Learning unified representation of language. image and point cloud for 3d understanding. arXiv preprint arXiv:22/2.05171, 2022.
[75] A. Jaegle, S. Borgeaud, J.-B. Alayrac, C. Doersch, C. Ionescu, D. Ding. S. Koppula, D. Zoran, A. Brock, E. Shelhamer, et al. Perceiver io: A general architecture for structured inputs & outputs. arXiv preprint arXiv:2107. 14795, 2021.
[76] E. Parisotto, J. L. Ba, and R. Salakhutdinov. Actor-mimic: Deep multitask and transfer rein- forcement learning. arXiv preprint arXiv:/51100344,, 2015.
[77] M. Crawshaw. Multi-task learning with deep neural networks: A survey, arXiv preprint 20X0::0009009796. 2020.
[78] M. Hessel, H. Soyer, L. Espeholt, W. Czarnecki, S. Schmitt, and H. van Hasselt. Multi-task deep reinforcement learning with popart. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 3796-3803, 2019.
[79] A. Kendall, Y. Gal, and R. Cipolla. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7482-7491, 2018.
[80] T. Yu, S. Kumar, A. Gupta, S. Levine, K. Hausman, and C. Finn. Gradient surgery for multi- task learning, Advances in Neural Information Processing Systems, 33:5824-5836, 2020.
[81] S. Wu, H. R. Zhang, and C. Ré. Understanding and improving information transfer in multi- task learning. arXiv preprint arXiv:2005. 2020.
[82] Liu, S. James, A. J. Davison, and E. Johns. Auto-lambda: Disentangling dynamic task relationships. arXiv preprint arXiv:2202.0309/, 2022.
[83] N. M. Shafiullah, Cui, A. A. Altanzaya, and L. Pinto. Behavior transformers: Cloning
modes with stone. Advances in neural information processing systems, 01222955-22966 2022.
13

---

[84] T. Pearce, T. Rashid, A. Kanervisto, D. Bignell, M. Sun, R. Georgescu, S. V. Macua, S. Z. Tan, I. Momennejad, K. Hofmann, et al. Imitating human behaviour with diffusion models. arXiv preprint arXiv:2301. 2023.
[85] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A.N. Gomez, L. Kaiser, and I. Polo- sukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[86] L.. Manuelli, W. Gao, P. Florence, and R. Tedrake. kpam: Keypoint affordances for category- level robotic manipulation. In The International Symposium Robotics Research, pages 132- 157. Springer, 2019.
[87] B. Wen, W. Lian, K. Bekris, and S. Schaal. You only demonstrate once: Category-level ma- nipulation from single visual demonstration. arXiv preprint arXiv: 2022.
14

---

A
Task Details
Setup. Our simulation experiments are conducted on Robot Learning Benchmark (RLBench) envi- ronment [32].
(a)
(c)
"open
"open the door"
"water plant"
"lift tollet seat up"
(g)
"put the phone on the base"
"put 1 books bookshell"
"take umbrella out of stand"
"open fridge"
(k)
"the gray target"
azure square"
triangle"
Figure 5: RLBench Tasks. Our simulation experiments encompass 11 tasks from RLBench. Tasks (a)-(h) serve as the basis for single-task and multi-task learning, while tasks (i)-(k) are employed to evaluate generalization performance.
For single-task and multi-task learning, we use 8 tasks that closely mirror real-world scenarios, for example, our tasks incorporate objects like microwaves, plants, phones, books, and toilets that align more closely with the reality of domestic robot environments. Our aim is leverage the powerful semantic understanding and spatial reasoning capabilities of SGR to facilitate the manipulation of diverse and complex array of objects in real-world situations.
For the generalization part, we conduct experiments on 3 tasks characterized by a substantial amount of visual variations. Among the tasks, two focus on color generalization, while the remaining one centered around shape generalization. The color variations include 20 instances:colors (red, maroon, lime, green, blue, navy, yellow, cyan, magenta, silver, gray, orange, olive, purple, teal, azure, violet, rose, black, white). The shape variations include 5 instances: shapes {cube, cylinder, triangle, star, moon].
As depicted Figure 5, we utilize diverse set of simulation tasks. following sections, we will provide detailed examination of each task, and for those tasks that have been modified compared with the original RLBench implementation', we will discuss the specifies of the modifications in detail.
15

---

A.1
Open Microwave
Filename: open microwave.pi
Task:
Open the microwave on the table.
Modified: No.
Objects: microwave.
A.2 Open Door
Filename: open door py
Task:
Grip the handle and push the door open.
Modified: No.
Objects: 1 door.
A.3 Water Plants
Filename: water-pll pnter-plants.pp
Task: Pick up the watering can by its handle and water the plant.
Modified: No.
Objects: watering can and I plant.
A.4 Toilet Seat Up
Filename: toilet seat py
Task: Grip the edge of the toilet and lift up an upright position.
Modified: No.
Objects: toilet with lid closed.
A.5 Phone on Base
Filename: phone on base. py
Task: Grasp the phone and put it on the base.
Modified: No.
Objects: phone and phone base.
A.6 Put Books
Filename: put_ books bookshelf py
Task:
Pick up a book and place them on the top shelf.
Modified: No. The original task has 3 variations, here we only use the first variation.
Objects: bookshelf and 3 books.
A.7 Take out Umbrella
Filename: take wmbbrella..uttooo.mmrellll...... pp
Task: Grasp the umbrella by its handle, lift it up and out of the stand.
16

---

Modified:
No.
Objects: umbrella and umbrella stand.
A.8 Open Fridge
Filename: open.fridge.py
Task: Grip the handle and slide the fridge door open.
Modified: No.
Objects: fridge with door closed.
A.9 Reach Target
Filename: reach_target. py
Reach the language-instructed colored ball. There are 3 balls, distinguishable solely by their colors, which are randomly distributed in space. The 3 colors are sampled from full set of 20 color instances, thus offering 20 variations based on the target color.
Task:
Yes. The sizes of the balls are enlarged so that they are distinguishable in the RGB-D input.
Modified:
Objects: balls.
A.10
Slide Block
Filename:
Slide the block to the language-instructed colored square targets. There are 3 distractor target of different colors. The 4 targets are placed symmetrically and the colors are sampled from the full set of 20 color instances, which makes the 20 variations.
Task:
Modified: Yes. The original slide block ttuttttttaaaaa to_tttgetttttttee py task contains only target. Three other targets are added.
Objects: block and 4 colored target squares.
A.11 Reach Shape
Filename:
reach_ -yy
Reach the language-instructed shape of the block. There always 1 target shape and distractor shapes, which are from the 5 shape instances and randomly placed the table. The different target shapes make the 5 variations.
Task:
Modified: Yes, newly added task.
Objects: 5 shapes.
B
SGR Details
B.1
Architecture Details
Input Data. The SGR model takes as input RGB images [ of size x W and corresponding depth images of the same size from multiple camera views. Then point cloud, represented in the robot's base frame, derived from depth images with known camera extrinsics and intrinsics. A key detail here is that the RGB images and the organised point cloud are aligned, ensuring a one- to-one correspondence between the points across the two data forms. In the simulation setting, have H W 128 and K 4, corresponding to four camera positions: front, left shoulder, right
17

---

Share Subsample Process
Debalk Explanation
water the plant Insrrction(OOttonal)
Text Encoder
Concatwrate
Subsample
Grouping
Roductionn
Frozen
MUP
Size Upsample Channal Dounsmple
Grounding
Backspoojector
Subsample
Atrans
Set Abstraction 2
Set Abstrrction 3
Set Abstraction 4
20 images from multiple
Gyat
MLP
Proprioception
dopen
Abstraction
Abstraction
MLP
Gcollide
[N/6600]
cloud
(N2244]
INA122
Figure Architecture Details.
shoulder, and wrist. For real robot experiments, the setup is H W 256 and K 1, denoting single front-facing camera.
The model also receives proprioceptive data z, comprising four scalar values: gripper open state, left finger joint position, right finger joint position, and action sequence timestep. Additionally, task comes with language instruction this also forms part of the model's input.
Visual Grounding Module. The fundamental role of visual grounding module is to localize visual areas that correspond to objects possessing property as dictated by language. To equip our manipulation model with a deep comprehension of variety of object properties and spatial relationships, including those not previously seen in the manipulation demonstration data, we take inspiration from MaskCLIP [69] and PROgRAMPORT [70].
It's worth noting that vanilla FLIP-ResNet-50 employs attention pooling to merge visual features from various spatial locations into one feature vector, and then uses linear layer F as follows:
(mbg(AvgPPol (X)) Embg (X,)"
fglobal F
Σ softmax
Emb, (Xx)
where X₃ is the input feature at spatial location i, Embg,,,. are linear embedding layers and C' is some fixed scalar.
Following MaskCLIP [69] and PROGRAMPORT [70], establish two C, and Ci, which are initialized with the weights derived from Emb, and F respectively. Applying these layers in sequence enables the visual feature to be projected into the same space as the language feature. Ultimately, our grounding module formulates dense mask encompassing image segments corresponding to the language S in the following manner:
convolution layers,
falign Σ ((TIE((H(S)) @ ((C)(())) E RI' xw x1
where H(S) RD, (X)) E Rl' xw XD (H', w', D represent the height, width, and di- mension of the pre-aligned feature respectively). H is the CLIP language encoder and TILE serves as spatial broadcast operator that expands the language embedding vector to match the spatial di- mensions of the visual features. The symbol @ indicates an element-wise multiplication, while the summation is performed over the feature dimension D. Additionally, we concatenate this feature with the pre-aligned image feature to generate our aligned feature maps:
M Concat( (c,, g² xw' x(1+D)
(4)
18

---

Other Details. For the image encoder, we employ CLIP-ResNet To maintain consistency with the resolution predominantly used in the pre-training of vision models, we upsample the input image size from 128 x 128 to 256 x 256 in simulation experiments. The CLIP text feature exhibits a dimensionality of 1024, and the CLIP image encoder generates a feature map of channels. For the 3D encoder, we utilize PointNeXt-SS. Detailed information regarding the number of channels and sub-points in each layer of the PointNeXt network can be seen in Figure 6.
with 2048
B.2 Training Details
Data Augmentation. To further improve the performance and robustness of our model, we incorpo- rate a variety of data augmentation techniques. (1) Translation perturbations: during training, the translation is perturbed with 10.125 m. We also experimented with rotation perturbations but found they did not contribute to performance improvement. (2) Color drop is to randomly replace colors with zero values. This technique serves a powerful augmentation for PointNeXt, leading to sig- nificant enhancements in the performance of tasks where color information is available. Qian et al. [29] suggest that by implementing color drop, the network is encouraged to pay more attention to the geometric relationships between points, leading to a boost in overall performance. (3) Point resam- pling is a common technique utilized in the processing of point cloud data, often applied to adjust the density of the point cloud. The technique involves selecting a subset of points from the original data, thereby creating new dataset with altered density. In our simulation experiment, we resample 4096 points from the original point cloud. (4) Keyframe discovery and demo augmentation [50] [33] are to convert every data point within demonstration into a task that involves predicting the subsequent next-best keyframe action. Keyframes are extracted by heuristic method: an action is a keyframe if the joint velocities are close to zero and the gripper open state is unchanged. This keyframe discovery strategy bypasses the need to predict long sequences of noisy actions, making policy learning more efficient.
The configuration of hyperparameters applied in studies can be found in Table 3. The single-task learning and generalization experiments are trained on single NVIDIA GeForce RTX 3090 GPU, while the multi-task learning is trained on 8 GPUs. Since color plays a crucial role in some tasks and disregarding color information would render these tasks unsolv- able, such as reach target and slide block, the color drop is not utilized in the generalization experiments.
Hyperparameters.
Table 3: Hyper-parameters used in our simulation experiments.
Config
Single-task
Multi-task
Generalization
Training iterations
40, 000
40, 000
40, 000
Leraning rate
0.0003
0.001
0.0003
Batch size
32
256
32
Optimizer
AdamW
AdamW
AdamW
Weight decay
10-6
10-6
x 10-6
Color drop
0.2
0.4
0
Instruction provided
No
Yes
Yes
Number of input points
4096
4096
4096
C Evaluation Workflow
Simulation. In pursuit of the reliability and stability of our results, our evaluation follows such sub- sequent steps. (1) Train the agent on the train set for 40,000 training iterations and save checkpoints every 1,000 iterations.
(2) Evaluate the last 20 checkpoints for 25 episodes. checkpoints for 100 episodes, and use the average performance of the 3 checkpoints as the results. (4) Repeat the above steps for random seeds.
(3) Evaluate the best
19

---

Method
open nicrowave
open door
planta
seat
on base
phone
put books
unbrella
out
open fridge
Average
7.0
1.2
59.7
16.5
4.7 3.8
17.6
CLIP
9.1 10.8
68.4 8.8
4.1 1.8
40.1
20.8
7.3 1 2.5
38.0
16.3
66.7 8:2.2
7.4
2.8
30.2
9.4
60.0
10.4
1.6
63.0
7.9
35.6
1.6.100 4.8
41.8
41.11.1
19.8
15.3 15.3112
35.7
Poius
17.0= 5.1
63.8
28.2
= 2.2
52.7
14.
65.9 19.9
58.1 16.0
95.1330
7.3 4.7
47.9
PERACT
26.9 1 4.3
47.8.1 41.1
4.11
67.1
4.2
0.0 0.0
63.7 4 3.5
94.4 1:141:1.. 1.3
10.1 3.9
44.0
(Ours)
8..2
8.3
80.1
91.7
34.8
Table 4: Single-task results with mean and standard deviation (%).
Method
open nicrovave
open door
kater plants
toilet sext up
phone on baze
pat books
uabrella
take out
Eridge
open
Average
CLIP
6.9
78.0 2.0
6.7.6...
10.4
38.0
15.3
9.3:2:5.0
21.3
Pois
22.7 1: 1.2
80.0 = 11.1
3.1
32.0 = 20.0
69.3 1: 12.2
63.3 1 13.3
72.0 = 10.0
18.0 1: 2.0
47.3
PERACT
22.7 2.3
58.7= 114
8.0 10.6
61.7 12.9
0.0 0.0
17.3 4 18.1
96.0
1.0
37.7
SGR (Ours)
38.7 15.0
84.0
20.7
67.3 10.3
88.0
26.7
Table 5: Multi-task results with mean and standard deviation (%)
Real-Robot. For the real-robot experiments, we train the agent for 40,000 iterations and simply
choose the last checkpoint for evaluation, since it's expensive to evaluate more checkpoints in the real robot. We evaluate 10 episodes per task. During the evaluation of trained agent, the agent keeps acting until achieving the goal reaching the maximum episode length.
D Real-Robot Setup
For our real-robot experiments, we use a Franka Emika Panda manipulator equipped with a parallel gripper. Perception is achieved through an Intel RealSense D415 camera, positioned in front of the scene. The camera generates RGB-D images with a resolution of 1280 x 720. We leverage the realsense-ros to align depth images with color images. The extrinsic calibration between the camera frame and robot base frame is carried out using the easy easy_handeye package".
When preprocessing the RGB-D images, we initially crop the 1280 x 720 images to a 720 x 720 frame, and then resize them to 256 x 256 using nearest-neighbor interpolation. This interpolation is preferred over others, such as bilinear interpolation, as the latter can cause non-existent points in the depth map, leading to a noisy point cloud. Following these steps, we can process RGB-D images in the same manner as our simulation experiments. It's important to note that the camera's 's intrinsics must be adjusted accordingly after the images are cropped and resized. We train SGR for 40,000 training steps with 64 demonstrations in total and use the final checkpoint for inference.
E Additional Results
For all simulation experiments, we use 3 random seeds to ensure the reliability of our results. While we present averaged results in the main body of the paper for clarity, we provide more comprehensive results in this section. Here, we report both the mean and standard deviation derived from our simulation results to offer a complete view of our experimental outcomes. Table 4, Table 5, and Table 6 show the results of single-task learning, multi-task learning, and generalization, respectively.
F Computational Efficiency
SGR stands out significantly in computational efficiency, boasting impressive gains in both memory and time efficiency. In the single-task setting, SGR converges within 7 hours using a batch size of 32 and consumes mere 17GB of GPU memory. This training can be efficiently executed on a single NVIDIA RTX 3090 GPU. In comparison, the voxel-based method, PerAct, can only accommodate a batch size of on the same GPU and requires over 16 hours to converge.
'https://github wtt/ waa.tha..co//EE.CC///aaaaaa..aee
20

---

reach
target
slide block
reach shape
Method
Seen
Unseen
Seen
Unseen
Seen
Unseen
R3M
20.2 6.2
5.5 + 0.9
11.0
3.0 + 2.8
78.1 + 24.5
6.7 + 8.2
CLIP
42.0 27.0
17.7 17.7.:44.
49.0 14.3
16.31:28
9
0.2 0.3
PointNeXtuLi
45.9
6.7 4.7
PointNeXtix
99.7
4.0
96
6.2
99.8=0.4
1.8 1.6
DenseFusion
71.8
9.5
26.3
16.5
9..
3.8
SGR (Ours)
91.02.
16
9..7
4.0
99.9
27.7 4.0
Table 6: Generalization results with mean and standard deviation (%).
Method
open nicrowave
open door
vater plants
toilet seat
phone base
put books
Laice out
opea fridge
Average
umbrel14
52.6
80.2
SGR w/ RGBs added for points
37.0
71.8
28.1 1
14.9
77.2
61.7
88. 1
1.7
51.7
SGR w/o semantic brunch (PoiniNeXaus)
17.0
63.8
28.2
52.7
65.9
53.1
95.1
7.3
47.9
SGR geometric branch (CLIP)
9.1
68.1
10.1
7.3
38.0
66.7
744
30.2
SGR LIS (DenseFusion)
20.4
87.5
31.5
80.2
64.2
63.9
75.7
25.0
56.0
Table 7: Ablation experiments on single-task setting.
The noticeable disparity in efficiency stems mainly from SGR's design. Its semantic branch uses frozen pre-trained 2D model, while the geometric branch adopts point-based models. These point- based models exploit the sparsity of 3D point sets and inherently demand fewer computational re- sources than their voxel-based counterparts.
G Ablation Experiments
In order to delve deeper into the understanding and evaluation of the proposed method, we conduct a series of ablation experiments. The results are presented in Table 7.
Adding RGB colors for each point in the geometric branch. We conduct experiments with adding RGB colors for each point in the geometric branch. Notably, this addition led to reduction in performance. We hypothesize that this decline in performance could be attributed to the lack of pre- training on colored point clouds, which might render the geometric branch susceptible to overfitting certain spurious RGB features.
Deleting of the two branches. Indeed, certain baseline models outlined in Section 4.1 be interpreted deleting of the two branches in SGR model. In particular, PointNeXtuus and CLIP are designed to reflect scenarios where only the geometric semantic branch is retained, respectively. Additionally, to ensure fair comparison, we make sure all ablation models utilize the same input modalities (RGB-D images). the case PointNeXtLis, we add RGB colors into point cloud. For the CLIP model, we employ a separate 2D CNN to process depth images and subsequently fuse the CLIP features and depth features together.
Switching out the pre-trained encoder with one trained from scratch. This aligns with the im- plementation of DenseFusion in Section 4.2. The results presented in Figure 4 demonstrate that replacing a pre-trained encoder with one that trained from scratch will significantly reduce its generalization capabilities. Furthermore, to offer a more comprehensive perspective, we have sup- plemented with experimental results Table for DenseFusion under the single-task setting. The results also demonstrate the significant role of the pre-trained model within SGR.
H Single-view Setup
In Section 4.1, all simulation experiments are conducted under a multi-view setup with 4 cameras. To more comprehensively demonstrate the advantages of SGR, we undertake further experiments compare the performance of SGR. CLIP, and R3M under single-view setup, utilizing the front camera. These experiments executed in single-task setting and the results presented in Table 8.
21

---

Method
open microwave
open door
water planta
toilet aeat
phone on base
put books
take out
open fridge
Average
umbrella
R3M
21.1
57.0
19.8
34.3
5.2
50.9
51.4
12.4
31.5
CLIP
18.2
48.6
25.5
61.3
20.8
27.9
28.2
7.0
29.7
SGR (Ours)
31.2
81.3
23.1
65.0
70.5
64.1
88.0
18.9
55.3
Table Single-view (the front camera) performance under single-task setup.
We note that, compared with the multi-view setting in Table 1, SGR's performance experiences reduction (from 68.6% to 55.3%). This decline is largely attributed to the fact that a single-view camera provides limited information, which in turn weakens the geometric understanding capabil- ity of SGR's geometric branch. Meanwhile, CLIP maintains consistent performance (30.2% vs. 29.7%), and interestingly, R3M even witnesses an improvement (from 17.6% 31.5%). We hy- pothesize that this phenomenon occurs because both CLIP and R3M were originally pre-trained in single-view setup. As a result, the multi-view setting might not provide additional benefits for their 2D representations and might even introduce some interference.
Nonetheless, it's important to note that even in the single-view setting, owing the spatial rela- tionship captured by the geometric branch, SGR still stands out and exhibits superior performance compared to CLIP and R3M.
I Additional Related Work
Generalization in Robotics. Generalization is pivotal foundation for the extensive application of robots in the real world. Some previous works [86] [87] have tackled challenging category-level manipulation tasks and demonstrated robust generalization in real-world scenarios. However, both works heavily rely on human priors, including manual specification of semantic keypoints [86] and the use of NUNOCS Net [87]. Such dependencies, coupled with their extensive and intricate ma- nipulation pipelines, can pose scalability challenges in diverse tasks and environments. Contrarily, our approach focuses on learning simple and scalable representation for visuo-motor control. This end-to-end approach is not only more efficient but also facilitates ease of implementation and de- ployment.
22


Reviews:
[{"review_id": "kTpLaTa17I", "reviewer": "Reviewer_ebNV", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper presents a robot perception framework that processes 2D images and point cloud information in two separate branches. The 2D sensor images are processed in a semantic branch equipped with pre-trained visual encoders so that the model could extract relevant semantic information in the scene. The raw point cloud is processed in the geometric branch with a point-based network so that the encoded representation in this branch captures the 3D structure of the scene. The outputs of the two branches are integrated into a fusion network to produce a global feature, which is then used to predict the next robot action.\n\nThe paper presents results on using the proposed framework in behavior cloning in 8 simulation environments and 8 real robot tasks. The results show that the proposed method outperforms various baselines that use different 2D representations or 3D representations.", "summary_of_recommendation": "Overall, this is a nice paper with an interesting idea. However, because of the existing issues in the experiments section, my internal rating of this paper is borderline. I would be happy to adjust my rating if the authors make their experimental results more convincing.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "* Justify design choices in the proposed method according to the clarification questions listed above.\n* Add additional simulation experiments on RLBench tasks shared with the PerAct paper.\n* Add PerAct baseline in Figure 4.\n* Add ablation experiments if the authors also believe they add value to the paper.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Strengths.**\n\n* The paper tackles an important problem of constructing a universal visual representation for robot manipulation. The proposed method is well-motivated and clearly presented.\n* The idea of making use of both 2D and 3D representation learning methods for the purpose of robot manipulation seems to be an under-explored area.\n* The authors conducted extensive experiments and showed quantitative results including comparisons with baselines in both sim and real.\n\n**Clarification questions with the proposed method.**\n\n* What is the benefit of processing 2D images and 3D point clouds in two branches when one could design a framework that makes use of pre-trained vision models (eg. CLIP) while processing both 2D images and 3D point clouds at the same time? For example, one could take a PerAct-like approach to use a transformer-based architecture, process the 2D images with CLIP-based visual encoders and feed the encoded representations into the transformer as additional inputs.\n* In the semantic branch, it seems that only RGB images from multiple views are used as inputs. Could the model benefit from processing the depth images from multiple views too?\n* In the geometric branch, why is only the raw point cloud loaded, when the RGB colors of the points could be easily added for each point?\n\n**Issues with experiments.** Although the authors performed a decent range of experiments on both simulation and real robot platforms, several problems exist in the details of those experiments and need to be addressed.\n\n* The authors performed sim experiments on 8 RLBench tasks, but it seems that those tasks do not overlap the 18 RLBench tasks PerAct evaluated on. Could the authors pick a few tasks evaluated in PerAct and show at least some single-task quantitative results on them?\n* In the \"generalizing to novel attributes\" experiments presented in Figure 4, the PerAct baseline is missing. Is there a particular reason results on this baseline are not presented? It would be nice if the authors could run and present results on PerAct in this table.\n* It seems that the quantitative results of the real robot experiments are not very good. There indeed seems to be a gap between the performance of SGR and PerAct. However, out of the 8 tasks evaluated, SGR only outperforms PerAct in 4 of them.\n* The paper lacks ablation experiments that study the extent to which parts of the method contribute to the overall performance. Some experiments that the authors could consider performing include (1) deleting one of the two branches in the network architecture while keeping other components of the framework; (2) switching out the pre-trained encoder with one that is trained from scratch.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "Wdn7HIe0Jk", "reviewer": "Reviewer_RMKE", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper presents Semantic-Geometric Representation (SGR), a perception module for robots that integrates rich semantic information from pre-trained 2D models with 3D spatial reasoning. The experiments demonstrate SGR's effectiveness in enabling robots to perform diverse simulated and real-world manipulation tasks, outperforming state-of-the-art methods in single-task and multi-task scenarios. The paper also highlights SGR's unique capability to generalize to novel semantic attributes.", "summary_of_recommendation": "The paper presents a compelling approach to integrating semantic and geometric representations, showing promising results in generalizing to previously unseen semantic attributes. The paper is well-written, and the flow enhances its readability. However, there are some aspects that need addressing, as mentioned earlier in the review section. Considering the interesting approach presented in the paper and its potential contributions to the field, I recommend a weak accept. However, addressing the above-mentioned points in a revised version would significantly strengthen the paper's overall quality.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "Overall, the paper was interesting to read. However, there are a few questions which I hope the authors could address:\n* The paper mentions that SGR achieves significantly higher performance when exposed to unseen colors or shapes. What happens in scenarios where a different shaped object requires a slightly different manipulation strategy? Does the method show signs of being able to adjust to different shapes in terms of the manipulation strategy?\n* How is success measured? The authors mention that the real-robot experiments involve success out of 10 trials. However, the average success rate of SGR as compared to per act is only 11% higher, which is equivalent to one additional success. Because real-robot experiments are difficult to reproduce in general, and is prone to noise, such an increase isn't convincing.\n* What is the runtime / computation resources needed by the method?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Overall, the paper presented an interesting approach to integrate semantic and geometric representations, and claims that it stands as the only approach capable of generalizing to previously unseen semantic attributes. The paper is well written and the flow makes the paper easy to understand. However, there are a few points which should be addressed:\n\n* The paper mentions that SGR achieves significantly higher performance when exposed to unseen colors or shapes. What happens in scenarios where a different shaped object requires a slightly different manipulation strategy? Does the method show signs of being able to adjust to different shapes in terms of the manipulation strategy?\n* How is success measured? The authors mention that the real-robot experiments involve success out of 10 trials. However, the average success rate of SGR as compared to per act is only 11% higher, which is equivalent to one additional success. Because real-robot experiments are difficult to reproduce in general, and is prone to noise, such an increase isn't convincing.\n* What is the runtime / computation resources needed by the method?", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "V8f6VKJQ5VN", "reviewer": "Reviewer_B5ZP", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This work presents Semantic-Geometric Representation (SGR), a universal perception module for robotics that leverages the rich semantic information of large-scale pre-trained 2D models and inherits the merits of 3D spatial reasoning. To do so, a network is designed to fuse the information from semantic and geometric features from 3D an 2D modalities respectively.\n", "summary_of_recommendation": "Overall, the work is potentially suitable for publication but needs some more clarifications given the current state. Addressing the above concerns is highly encouraged.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "see above", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths:\n- Despite some clarifications missing as mentioned below, the paper overall is well-written and easy to follow\n\n- The idea of fusing semantic and geometric information is reasonable. The pipeline is appropriately designed.\n\n- The experiments in both simulation and real world have been conducted. The results are encouraging and outperforms the baselines by a large margin.\n\n\nWeakness and questions:\n- In experiments, the proposed method leverages 4 cameras for observation. However, for comparison methods (both 2D and 3D), it is not clearly mentioned whether the same multi-view observation has been provided for fair comparison. Also some of the methods were not pretrained using multi-view images and thus cannot take full advantage of the multi-view. To solely validate the proposed fusion technique, single-view setup comparison would be helpful.\n\n- In intro, the statement of “SGR stands as the only approach capable of generalizing to previously unseen semantic attributes, including colors and shapes” seems over-claimed. In fact, multiple prior works [A-D] have tackled much more challenging category-level manipulation tasks in the real world. Discussion or comparison to these line of works is encouraged to be added into the paper.\n\n- It is mentioned “robot proprioceptive information is incorporated”. However, this is not reflected in fig2. It is suggested to ensure the consistency.\n\n- It would be more interesting to show side-by-side comparison to the baselines in the qualitative results or in the video demos.\n\nOverall, the work is potentially suitable for publication but needs some more clarifications given the current state. Addressing the above concerns is highly encouraged.\n\nReferences:\n    A) Manuelli, L., Gao, W., Florence, P. and Tedrake, R., 2019, October. kpam: Keypoint affordances for category-level robotic manipulation. In The International Symposium of Robotics Research (pp. 132-157). Cham: Springer International Publishing.\n    B) Wen, B., Lian, W., Bekris, K. and Schaal, S., 2022. You only demonstrate once: Category-level manipulation from single visual demonstration. RSS 2022", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "SBliB6Vobm", "reviewer": "Reviewer_e9U4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposed a way to fuse semantic information with RGB data and spatial information with depth data to generate better representation for robot manipulation.", "summary_of_recommendation": "Overall I found the paper of very good quality, the idea is innovative and supported with both simulation & real robotics experiments, thusI recommend accepting the paper.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "I in general in favor of accepting the paper therefore I don’t have further questions for rebuttal.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper is very clear and follows a logical flow making it a pleasure to read. The idea of combining RGB information with depth is natural and the way this paper adopt is innovative.\n\nThe literature review is comprehensive. The reviewer especially likes the style of splitting into different category and at end of each highlighting the difference between this and aforementioned works.\n\nThe architecture image in Figure 2 is informative and clearly shows the framework.\n\nThe new representation (SGR) is then used with behavior cloning (BC) and compared with multiple SOTA and shows significant improvement in most of the tasks.\n", "extra_scores": {"impact": "2: Unsure, difficult to assess."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper introduces a new method on learning semantic-geometric representation for robot manipulation.

After the rebuttal, reviewers still have diverge opinions on the paper. The remaining concerns of the reviewers are
- Experimental evaluation of the proposed method.
- Some claim of the paper is too strong.

Overall, the paper introduces a novel method which improves over previous state-of-the-art approaches.
Therefore, I recommend acceptance for the paper.
Due to the remaining concerns of the reviewers, I recommend the paper as a poster presentation.
Decision: Accept (Poster)