GenDP: 3D Semantic Fields for Category-Level Generalizable Diffusion Policy
Authors: Yixuan Wang
Guang Yin
Binghao Huang
Tarik Kelestemur
Jiuguang Wang
Yunzhu Li
Venue: CoRL 2024 (2024)
Keywords: Semantic Fields
Category-Level Generalization
Imitation Learning
Diffusion Models

Abstract: Diffusion-based policies have shown remarkable capability in executing complex robotic manipulation tasks but lack explicit characterization of geometry and semantics, which often limits their ability to generalize to unseen objects and layouts. To enhance the generalization capabilities of Diffusion Policy, we introduce a novel framework that incorporates explicit spatial and semantic information via 3D semantic fields. We generate 3D descriptor fields from multi-view RGBD observations with large foundational vision models, then compare these descriptor fields against reference descriptors to obtain semantic fields. The proposed method explicitly considers geometry and semantics, enabling strong generalization capabilities in tasks requiring category-level generalization, resolving geometric ambiguities, and attention to subtle geometric details. We evaluate our method across eight tasks involving articulated objects and instances with varying shapes and textures from multiple object categories. Our method demonstrates its effectiveness by increasing Diffusion Policy's average success rate on \textit{unseen} instances from 20\% to 93\%. Additionally, we provide a detailed analysis and visualization to interpret the sources of performance gain and explain how our method can generalize to novel instances. Project page: https://robopil.github.io/GenDP/

GenDP: 3D Semantic Fields for Category-Level Generalizable Diffusion Policy
Yixuan Wang', Guang Yin², Binghao Huang', Tarik Kelestemur", Jinguung Wang', Yunzhu Li¹
1Columbia University, University of Illinois, Urbana-Champaign, 3oston Dynamics Al Institute
Abstract: Diffusion-basee policies have shown remarkable capability in execut- ing complex robotic manipulation tasks but lack explicit characterization of geom- etry and semantics, which often limits their ability to generalize to unseen objects and layouts. enhance the generalization capabilities of Diffusion Policy, we introduce a novel framework that incorporates explicit spatial and semantic infor- mation via 3D semantic fields. We generate 3D descriptor fields from multi-view RGBD observations with large foundational vision models, then compare these descriptor fields against reference descriptors to obtain semantic fields. The pro- posed method explicitly considers geometry and semantics, enabling strong gen- eralization capabilities in tasks requiring category-level generalization, resolving geometric ambiguities, and attention to subtle geometric details. We evaluate our method across eight tasks involving articulated objects and instances with varying shapes and textures from multiple object categories. Our method demonstrates its effectiveness by increasing Diffusion Policy's ss average success rate unseen instances from 20% to 93%. Additionally, we provide detailed analysis and visu- alization to interpret the sources of performance gain and explain how our method can generalize to novel instances. Project page
Keywords: Semantic Fields, Category-Level Generalization, Imitation Learning. Diffusion Models
Heatmap
Predicted Actions
Execution
Use Knifé
Handle
Soda Tab
Flip Can
Open Pen
Cap
(a) Daily Tasks Examples Figure 1: Generalizable Diffusion Policy using Semantic Fields. Our approach introduces diffusion policy capable of generalizing to new instances within a category by utilizing 3D semantic fields. These fields distinguish semantically meaningful parts of objects in 3D space, as illustrated in the heatmap example. Panel (a) on the left showcases daily task examples where semantic understanding is crucial, while panel (b) the right demonstrates our method's ability to highlight semantically meaningful parts, such a knife handle, and how our predicted policy accomplishes these tasks using the 3D semantic fields.
(b) Our Imitation Learning Framework
8th Conference Robot Learning (CoRL. 2024), Munich, Germany.

---

Introduction
Diffusion-basee policies have recently shown promising results in imitation learning for real robot deployment in complex robotic manipulation tasks [1, 2], because they can express multimodal action distribution, output high-dimensional actions, and train stably. However, existing end-to- end diffusion policy frameworks do not generalize to novel instances due to their brittleness to environment variances, such as object appearances and background changes. Our work proposes diffusion policy framework capable of category-level generalization and robust to environmental changes using 3D semantic fields.
Previous work has explored the use of point clouds as policy inputs to help generalization [2, 3], but the reliance on geometric information is insufficient for fine-grained scene understanding and generalization. For instance, as illustrated in Figure I (b), a marker's 's head and tail are geometrically ambiguous despite their functional and semantic differences. In this work, we augment geometric information with semantics generalize novel instances, resolve geometric ambiguity, and attend to subtle geometric details.
An ideal representation should not only extract geometric information from raw observation but also retain semantic information for better category-leve generalization. In this work, we introduce a diffusion policy framework that uses a scene representation in the form of 3D semantic fields. The 3D semantic fields are defined as the similarity fields, where the similarity score higher for points closer one object part, such as knife handles. Note that the 3D semantic fields have multiple channels, shown in Figure 2, where the first channel corresponds to the shoe tongue, and the last one corresponds to the shoe cuff. Our framework consists of three main modules: a 3D descriptor fields encoder, a semantic fields constructor, and an action policy. The 3D descriptor encoder takes in multi-view RGBD observations and outputs a point cloud with high-dimensional descriptors using large foundational vision models like DINOv2 [4]. These descriptors are then fed into the semantic fields constructor and converted into low-dimensional semantic fields. Finally, the policy takes in the semantic fields along with the point cloud as inputs and predicts actions.
The proposed framework offers three benefits: (1) Category-level generalization: As semantic fields contain both 3D and semantic information, it guides our policy to focus on semantically meaningful parts essential for task completion, allowing generalization across instances within a category. (2) Resolve geometric ambiguity: Geometric information can be ambiguous. For ex- ample, the knife blade and knife handle are geometrically similar despite functional and semantical differences. Our semantic fields can localize space semantically close to parts important for task completion, such as knife blades, disambiguate vague geometric information. (3) A Attention to subtle semantic details: Semantic details might be lost due to real-world observation noise, such as toothbrush head and soda can tab. Some tasks are impossible to accomplish without sufficient de- tails, such as spreading toothpaste on a toothbrush and flipping a lying soda can upwards, as shown in Figure 1. Because our semantic fields highlight semantically distinct regions, our method can pay attention to nuanced semantic details for task completion.
We systematically evaluate our method across eight tasks. Our task settings involve novel instances, ambiguous object geometry, and require attention to subtle semantic details. Compared to baseline methods, which frequently fail to generalize to novel instances due to a lack of both 3D and semantic information in their representation, our approach demonstrates significant advantages in category- level generalization. It also leads to substantial performance improvements in scenarios containing geometric ambiguity and subtle details. Through comprehensive experiments, we show an improve- ment in Diffusion Policy's average success rate on unseen instances from 20% to 93%. We also provide a detailed analysis of how well and why method can generalize novel instances.
In this work, our contributions are threefold: (1) We propose a diffusion policy framework that uses 3D semantic fields, which leverages visual foundation models to encode the environment's geomet- ric and semantic information. (2) Our design allows category-level generalization, differentiating geometric ambiguity, and attending to geometric details. (3) We conduct comprehensive experi- ments and improve the original diffusion policy's 's average success rate on unseen instances from 20% 93%. We also provide detailed analysis of how well and why our method can generalize to novel instances.

---

(a) Policy Rollout
Point Cloud
Features
Diffusion Policy
(b) Network Archilecture
PointNct+
i.. Multi-View RGBD
(iv). Predicted Actions
(ii), Descriptor Field
Reference Image Reference Feats (iii). Feature Channel
(ii). Fenture Channel N
Figure 2: Method Overview. The top row (a) shows sequence of real policy rollouts in the aligning shoe task. We first take in multi-view RGBD observations (i), then extract the 3D descriptor field, with each point possessing corresponding high-dimensional descriptor (ii) [5]. We select reference features from reference images. By computing the cosine similarity between the descriptor field and 2D reference semantic features, we could obtain several semantic fields (ii). These semantic fields, concatenated with the point cloud, are then input into PointNet++ and the diffusion policy to output predicted actions (iv).
2 Related Works
Diffusion Models for Robotics. Beginning with random samples, diffusion models operate by iterative denoising to reveal the underlying distribution. They are widely used in robotics, including navigation [6], manipulation [1, 2, 7-14], control [15], and planning [16-18]. Among these works, Diffusion Policy and 3D Diffusion Policy are closely related to our work [1, 2].
Diffusion Policy produces visuomotor policy under the action diffusion model, which can model the multi-modal distribution of demonstration actions in the real world. However, it relies on multi- view RGB observations, which are brittle to environment changes, including novel object instances, camera viewpoints, and background changes, and fail to generalize to novel instances. Instead of RGB observations, our work enables category-level generalization using 3D semantic fields.
3D Diffusion Policy approaches this problem using point clouds as inputs [2]. While this proven to be sample-efficient, point clouds without semantic information be insufficient for many ma- nipulation tasks. In this work, we augment geometric information with semantic information using foundational vision models.
Geometric Representation in Robotic Manipulation. One typical geometric representation used in robotics is point clouds [3, 19-26], which enables the policy focus on geometry and facilitates generalization across different environments. Yet, this often results in the loss of critical RGB data, limiting the understanding of object's semantic properties. Unlike these approaches, our approach retains both semantic and geometric information, enhancing downstream decision-making
Another line of works uses keypoints as the geometric representation robotic manipulation [27- 34). They have shown impressive generalization capabilities and high efficiency due to their low dimension. However, the resulting sparsity loses geometric details and limits the range of tasks that can be effectively addressed. In contrast, framework employs representation with rich geometric information and enables broader range of tasks.

---

Semantic Representation in Robotic Manipulation. Semantic reasoning is an ongoing field in robotics [35]. One common semantic representation learning frame is directly learning object- centric or scene-centric embeddings from RGB observations [36-44]. However, these representa- tions are hard to generalize to new environments with different backgrounds and lighting conditions. In contrast, we use 3D semantic representation, which is generalizable across diverse environments and instances. Another class of semantic representation extracts functional affordance informa- tion from observations [43, 45-52J. This line of work focuses on tasks that can be accomplished using motion primitives such as grasping, picking, and placing, while our diffusion-based policy is more flexible in action representation and tasks.
recent line of research used neural implicit models such as Occupancy Networks [53] or NeRFs [54] to encode semantic features [5, 9, 25, 55-61]. GeFF is the closest one to our work. It operates by utilizing FeatureNeRF [62], which distills NeRF model from foundational models. Although the distillation process allows for building NeRF upon sparse input views, it reduces the generalization capability. In contrast, we extract 3D descriptor fields without distillation, which inherits generalization capabilities from foundational vision models. In addition, GeFF uses prede- fined open- pps--uushe action sequences to solve manipulation tasks. Our method differs by deploy- ing an imitation learning policy that could accomplish more complicated tasks.
Method
3.1 Problem Statement
We define our system as a Markov Decision Process (MDP) consisting of state 8 E S and action a E A. The system transition is defined through the dynamics model St41 = f(st, We as- (s{)), where и is the learned policy. The goal is to minimize the difference between P Pa and ground truth action probability Pal (a(()s)), given
sume predicted actions follow the probability distribution a, ~ PPrea(a)ss)
set of human demonstrations D (ro, T1,-.. TN), ). where represents trajectory comprising {so, do, 81, ...... ar). Here, the state consists of sequence of multi-view RGBD observations, and the action is a sequence of robot states.
3.2 3D Descriptor Fields
We use an off-the-shelf large foundational vision model, DINOv2 [4], obtain semantic features from multi-view RGBD observations. Given its ability extract consistent semantie features from the RGB images across context and instance variances, we selected it as the backbone network.
We provide pseudocode for building 3D descriptor fields in Algorithm 1. We denote single-view RGBD observation as O, = (I, R2), with E {1,2, N} representing the camera index, consist- ing of an RGB image I, E R x x3 and a depth image R4 E H We first use DINOv2 to extract dense 2D feature maps W; corresponding to the RGB image I4 [4]. For an arbitrary 3D point p, project it onto the image space find its corresponding pixel location u, and the distance to camera rg- We then interpolate to derive features S, from the feature map and the depth r from R,,
The depth difference Ar; = r r reflects how distant p is from the surface. When is closer to the surface in view i, greater weight is given to ft. We fuse features from multiple viewpoints by applying a weighted sum, thus obtaining the descriptor f corresponding to p. practice, we follow the implementation details in Doiields [5] to extract point cloud P E kxx3 and associated features F e IR KxF, where K is the point cloud's size.
3.3 3D Sematic Fields
First, we obtain several 2D images containing the task-relevant object category and extract feature maps. Then we select features from each feature map and take the average to get a set of reference descriptors Fiet IR MFF, Each descriptor represents a part of the object, such as the shoe head.
We denote the set of reference descriptors Fier with M descriptors and descriptor fields F with K descriptors such that F = {F, E i E {1, K}} and Fist = {Frt., E R E {1....,M]] M}}. The semantic fields E IR KxM are defined as the similarity between the descriptor fields and reference descriptors:
FF.FFL |[

---

By computing the similarity scores, we convert oigh-dimensional descriptor fields into M- dimensional semantic fields, where is typically less than 5. We then concatenate semantic fields C and raw point cloud P together, which are then inputted into the policy.
3,4 Policy Learning
Algorithm 3D Descriptor Fields Computation
In this work, we model our policy Denoising Diffusion Probabilistic Models (DDPMs), similar to Diffu-
Infer feature map W; from single RGB image Z 2: procedure EVALUATE())
Project p to camera and compute projected pixel and distance to camera ri
sion Policy [1, 63]. Instead of re- gressing the action directly, we train a noise predictor network
Obtain interpolated features fi Wi[u,]
5: Obtain depth r = R. [[,
Compute depth difference Ar; = r4
= s,k),
(2)
7: Fuse features F, h(fi
that takes in noisy actions a², current observations 8, and denoising iterations and predicts the noise ch. During training, we randomly choose denoising step k and sample noise c² added to the unmodified sample a.. Our training loss is the difference between e and predicted noise:
L = MSELoss(E* 2²).
(3)
During the inference time, our policy starts from random actions a and denoises for K steps to obtain the final action predictions. At each step, the action is updated following
k- = aa "ea(a", k) + (0
(4)
where a, 7, and hyperparameters. In practice, input 3D semantic fields into PointNer++ and obtain visual features. Then feed visual features into diffusion policy, similar to [1].
Truining Objects
Testing Objects
(a) Robot Setup
(b) Objects
Figure 3: Real Experiment Setup. (a) We use four RealSense cameras to capture RGBD observations and ALOHA robots to execute policy. (b) We test on diverse set of objects, including shoes, soda cans, marker pens, knives, spoons, toothbrushes, and toothpaste, with diverse geometry and appearance.
4 Experiments
In this section, we evaluate our method on eight diverse tasks. We aim to answer the following three questions through experiments. 1) How does the performance of our method compare that of the stateoof-theaa imitation learning methods? 2) How well configurations and instances? 3) What enabled our method to generalize novel instances?
method generalize different
4.1 Setup
For experiments in simulation, we use SAPIEN [64] to build the environments and evaluate the policies. For the real-world experiments, we use the ALOHA robot and four RealSense cameras for real-world data collection and testing [65]. Figure shows our real-world setup and objects used.
We evaluate method eight tasks, shown in Figure 4. We test in the simulation Hang Mug and Insert Pencil tasks. In the real world, we evaluate our method's ability from different per- spectives, such as insignificant geometric details, category generalization, and geometric ambiguity. We collected 200 demonstration episodes for the Hang Mug task, 100 episodes for Insert Pencil task, and 60 episodes for all real-world tasks. More details are listed the supplementary material. For each task, we train one policy to accomplish that single task.
We test our method various object categories, including shoes, toothbrushes, soda cans, and knives, among others, which present significant challenges for category-level generalization due to
5

---

factors such as large geometric variances, ambiguous geometric information, and nuanced geometric details. More detailed descriptions of the tasks can be found in the supplementary material.
Total (Seen)
Total (Unseen)
Hang Mug (Seen)
Pencil Insertion (Seen)
Ours
100
Success Rate
Own witho Sereetios
50
0
0
(a) Total
(b) Seen Instances
Hang Mug (Unseen)
Pencil (Unseen)
Can (Unseen)
Use Toothbrush (Unseen)
100
100
100
100
Success Rate
75-
75-
50
50
(i Simulation
(ii) Geometry Details
Collect Knife (Unseen)
Open Pen (Unseen)
Align Shoes (Unseen)
Use Spoon (Unseeu)
100
100
100
100
Success Rate
75-
15
(iii) Geometry Ambiguity
(iv) Category Generalization
(c) Unseen Instances
Figure 4: Success Rate. Our method was evaluated across eight tasks. (a) The aggregated quantitative results show that method has similar performance as diffusion policy for instances, but method outperforms all baselines unseen instances. (b) For seen instances, diffusion policy our method have similar performances on two simulation tasks. (c) Diffusion policy performance degrades significantly on unseen instances. In addition, our method outperforms all other haselines, which underscores our policy's s capability to attend to geometric details, distinguish geometric ambiguities, and generalize to novel instances.
4.2 Comparison with Baselines
We compare our method with three baselines: 1) Ours without Semanties: input raw point cloud without semantic fields, 2) vanilla Diffusion Policy (DP), and 3) DP with RGBD: vanilla DP but with depth observations. For all baselines, utilize four cameras to obtain observations and input them into the policy. We evaluate different policies by the success rate. The quantitative result is summarized in Figure 4 with detailed numbers the supplementary material.
Our method shows similar performance with the original diffusion policy in the simulation on seen instances. However, when generalizing unseen instances, diffusion policy performance degrades significantly, because it takes RGB images as input, which is brittle to environmental factors like object appearances and geometries. Therefore, the diffusion policy predicts undesirable actions and fails to accomplish the task. In contrast, our framework encodes explicit 3D and semantic information about the scene, which is robust to object appearance and geometry variances, which helps the policy to generalize to novel instances.
In addition, our method consistently outperforms ours without semantics, whether for seen instances unseen instances. The benefits from 3D semantic fields threefold. First, they help attend to geometric details, such as the can tab, to place the can on the table upright, while ours without semantics cannot when solely relying on geometry information. Second, 3D semantic fields help to distinguish geometric ambiguity. For instance, when grasping the knife, it is hard to tell the knife directions from ambiguous geometry, which might lead to unsafe actions like grasping the knife blade. Our 3D semantic fields encode semantie information like knife handles to distinguish geometric ambiguities and show higher success rate Figure 4 (c)(ii). Third, Figure (c)(iv demonstrates 3D semantic fields could help policy focus semantically meaningful parts to complete tasks and generalize to novel instances.

---

Demo Example
Init Config
DP
DPw w/ RGBD
Ours wi Raw PC
Ours
Collect Knife
Unsafe Behavior
Grasping Failure
Pick Can
Early Failure
Grasping Failure
Wrong
Open Pen
Wrong Direction
Early Failure
Wrong Direction
Training
Testing in Unseen Objects
Figure 5: Policy Rollout in Real World. The figure illustrates the policy rollout results in the real world. On the left, the blue block displays the demonstration examples and corresponding training instances. On the right, the orange block presents policy rollout results. From left to right, they are, respectively, initial configurations, diffusion policy, diffusion policy with RGBD, ours without semantics, common failure modes. Early failure and grasping failure could happen when the novel instance is presented. Diffussin policy may also lead to unsafe behavior when encountering novel instances. Ours without semantics might identify wrong directions due to geometric ambiguity and nuanced geometric details.
our method. We summarize four
We found that Diffusion Policy with RGBD is not effective our method. Directly adding depth observation to the diffusion policy will make the input space even larger. Therefore, it would require more demonstrations to have sufficient training data coverage, which can make it less data efficient.
Figure shows real-world policy rollout results. We notice different failure modes for baseline methods. For diffusion policy and diffusion policy with RGBD, when novel instance is presented, they may stop early and fail to make progress. Due their reliance 2D observations, they cannot generalize to unseen instances with varying appearances and geometries. Ours without semantics often fail to differentiate geometric ambiguity, like knife handles, while our method could identify the right part for manipulation. Our method also focuses on geometric details, like the soda can tab to accomplish the can flipping task, while ours without semantics places the soda can upside down.
4.3 Generalization to Novel Configurations and Instances
We also analyze how well our method can generalize to novel configurations and instances. Figure illustrates the predicted actions different policies under the same observations. For the seen instances, as shown the leftmost column, we can see that the predicted trajectories for all policies in the first example point towards the knife handle correctly. However, when the knife direction changes, our method without semanties fails to respond accordingly, while the diffusion policy and method predict actions accordingly. This demonstrates that our method semantic difference between knife handle and knife blade, which results in safe robot behaviors.
understand the
When the novel instance is presented with similar configurations, our method will predict a suc- cessful trajectory guiding toward the knife handle, while diffusion policy will predict nonsmooth trajectory, which shows that it is brittle to environmental factors and lacks category- generaliza- tion capabilities. Although our method without semantics predicts smooth trajectory, it approaches the knife blade, which is unsafe.
4.4 Generalization Analysis
We visualize the 3D semantic fields in Figure by overlaying them with the raw point cloud. These semantic fields benefit the policy for two primary reasons: 1) Highlighted points are semantically meaningful and crucial for task completion. For example, to collect books on the shelf upright, the
7

---

robot must recognize the book titles. Geometric information alone is inadequate for task completion. 2) Semantic fields are consistent across different instances. The mug example shows that the activa- tion on the handle is consistent across mugs with various appearances and poses. The category-level consistency enables our method to achieve category-level generalization. also present t-SNE analysis in the supplementary material.
Scene
Scene
Novel Object
Third Person View
4.5 Extension to Other IL Methods
We also extended 3D) semantic fields and showed benefits to other imitation learning (IL) methods, such as Action Chunking with Trans- former (ACT) [65]. The detailed results are presented in the supplementary material.
Difiusion Policy
Conclusion
In this work, we proposed a diffusion policy framework leveraging 3D semantic representa- tions, with policies modeled as DDPMs con- ditioned on the semantic fields alongside point clouds. We conduct comprehensive physical experiments to show that our framework can generalize to novel instances within the cate- gory, resolve geometric ambiguity, and resolve nuanced geometric details. Compared with the vanilla diffusion policy with 20% success rate, our method can reach 93% success rate on un- seen instances, demonstrating the effectiveness of our method.
DP wl
RGAD
Ours wio Semanties
Ours
Figure 6: Predicted Action Comparisons. We pare different policies' responses under the same state, Diffusion policy's 's predicted actions change with knife direction accordingly but not smooth for novel in- stances. Due to geometric ambiguity, method with- our semanties fails distinguish knife directions, and predicts similar actions even under different knife rections, while our method consistently recognizes the knife handle position and predicts correct actions.
Limitation. For each task, we select set of fixed reference features to construct semantic features. However, for long-horizon and fine- grained tasks, such making coffee and as- sembling, the policy's attention needs to adapt as tasks progress. Constructing task-specific semantic fields that can adaptively select the abstraction level, can future direction. In addition, our method could become more interpretable and potentially more efficient we can incorporate the geometric properties in more explicit fashion.
Shoe
Book
Mug
RGB Observation Raw Point Cloud Figure 7: 3D Semantic Fields Visualization. We visualize 3D semantic fields in the scene with multiple instances from the category. 3D semantic fields exhibit similar patterns across different instances. such highlighted book titles all books. Furthermore, these highlighted areas represent semantically meaningful parts and are important for various tasks, such as shoclaces, shoe heads, book stands, mug handles, and so on.
Similarity Field Similarity Field 2
Similarity Field 3

---

Acknowledgement
This work is partially supported by Sony Group Corporation. The opinions and conclusions ex- pressed in this article are solely those of the authors and do not reflect those of any Sony entity.
References
[1] C. Chi, S. Feng, Du, Z. Xu, E. Cousineau, B. Burchfiel, and S. Song, Diffusion policy: Visuomotor policy learning via action diffusion. In Proceedings of Robotics: Science and Systems (RSS), 2023.
[2] Y. Ze, G. Zhang, K. Zhang, C. Hu, M. Wang, and H. Xu. 3d diffusion policy: Generalizable visuomotor policy learning via simple 3d representations. arXiv preprint arXiv:2403.03954, 2024.
[3] Y. Zhu, 7. Jiang, P. Stone, and Y. Zhu. Learning generalizable manipulation policies with object-centric 3d representations. In 7th Annual Conference on Robot Learning, 2023.
[4] M. Oquab, T. Darcet, T. Moutakanni, H. V. Vo, M. Szafraniec, V. Khalidov, P. Fernandez, D. Haziza, F. Massa, A. El-Nouby, R. Howes, P.-Y. Huang, H. Xu, V. Sharma, S.-W. Li, W. Galuba, M. Rabbat, M. Assran, N. Ballas, G. Synnaeve, I. Misra, H. Jegou, Mairal, Labatut, A. Joulin, and P. Bojanowski. Dinov2: Learning robust visual features without supervision, 2023.
[5] Y. Wang, Li, M. Zhang, K. Driggs-Campbell Wu, L. Fei-Fei, and Y. Li. D9fields: Dy- namic 3d descriptor fields for zero-shot generalizable robotic manipulation. arXiv preprint arXivv2309.100 16118, 2023.
[6] A. Sridhar, Shah, C. Glossop, and S. Levine. NoMaD: Goal Masked Diffusion Policies for Navigation and Exploration. arXiv pre-print, 2023. URL 07896.
[7] Octo Model Team, D. Ghosh, H. Walke, K. Pertsch, K. Black, O. Mees, S. Dasari, J. Hejna, C. Xu, J. Luo, T. Kreiman, Y. Tan, D. Sadigh, C. Finn, and S. Levine. Octo: An open-source generalist robot policy, https: http:///cooommoeeee .iiis.ggttubb. io, 2023.
[8] T. Pearce, T. Rashid, A. Kanervisto, D. Bignell, M. Sun, R. Georgescu, S. V. Macua, S. Z. Tan, I. Momennejad, K. Hofmann, et al. Imitating human behaviour with diffusion models. arXiv preprint arXiv:2301. 2023.
[9] T.WW. Ke, N. Gkanatsios, and K. Fragkiadaki. 3d diffuser actor: Policy diffusion with 3d scene representations. Arxiv, 2024.
[10] M. Reuss, M. Li, X. Jia, and R. Lioutikov. Goal conditioned imitation learning using score- based diffusion policies. In Robotics: Science and Systems, 2023.
[11] J. Urain, N. Funk, J. Peters, and G. Chalvatzaki. Seff)ddiifsssoofieeld:: Learning smooth cost functions for joint grasp and motion optimization through diffusion. 2023 IEEE In- ternational Conference on Roboties and Automation (ICRA), pages 5923-5930, 2023. doi: A.1109//CRA48881..00 10161569...
[12] W. Liu, Y. Du, T. Hermans, Chernova, and C. Paxton. Structdiffusion: Language-guide creation of physically-valid structures using unseen objects. In RSS 2023, 2023.
[13] H. R. D. M.-m. R. Shelving, Stacking. Sornet: Spatial object-centric representations for sequential manipulation. In 7th Annual Conference on Robot Learning, 2023.
[14] Z. Xian, N. Gkanatsios, T. Gervet, T.-W. Ke, and K. Fragkiadaki. Chaineddiffuser: Unifying trajectory diffusion and keypose prediction for robotic manipulation. In 7th Annual Conference on Robot Learning, 2023.

---

[15] T. Yoneda, L. Sun, G. Yang, B. C. Stadie, and M. R. Walter. To the noise and back: Diffusion for shared autonomy. In Robotics: Science and Systems XIX, Daegu, Republic of Korea, July 10-14, 2023, 2023.
[16] Z. Wang. T. Oba, T. Yoneda, R. Shen, M. Walter, and B. C. Stadie. Cold diffusion on the replay buffer: Learning to plan from known good states. 7th Annual Conference on Robot Learning, 2023. URL https: openreview net. forum?? did-nrrr_o22222222
[17] K. Saha, V. Mandadi, Reddy, A. Srikanth, A. Agarwal, Sen, A. Singh, and M. Krishna. Edmp: Ensemble-of-coots-gule diffusion for motion planning. 2023.
[18] M. Janner, Y. Du, J. Tenenbaum, and Levine. Planning with diffusion for flexible behavior synthesis. In International Conference Machine Learning, 2022.
[19] A. Goyal, J. Xu, Y. Guo, V. Blukis, Y.W. Chao, and Fox. Rv:: Robotic view transformer for 3d object manipulation. arXiv preprint arXiv:2306.14896 2023.
[20] James, K. Wada, T. Laidlow, and A. Davison. Coarse-to-fine q-attention: Efficient learning for visual robotic manipulation via discretisation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13739-13748, 2022.
[21] T. Gervet, Z. Xian, N. Gkanatsios, and K. Fragkiadaki. Act3d: 3d feature field transformers for multi-task robotic manipulation. In Conference on Robot Learning, pages 3949-3965, PMLR, 2023.
[22] Y. Qin, B. Huang, Z.-H. Yin, H. Su, and X. Wang. Dexpoint: Generalizable point cloud rein- forcement learning for sim-to-real dexterous manipulation. Conference on Robot Learning, pages 594 605. PMLR, 2023.
[23] D. Bauer, Patten, and M. Vineze. Reagent: Point cloud registration using imitation and reinforcement learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14586-14594, 2021.
[24] T. Chen, M. Tippur, S.
Wu, Kumar, E. Adelson, and Agrawal. Visual dexterity: In- hand reorientation of novel and complex object shapes. Science Roboties, 884))eade9244, 2023. 1..11:0...1///oobboato http://////cmoooiccccee22222 URL https: /wwww//www//////////////. www.scciecc..co 10.1126/scirobtiicss
[25] A. Mousavian, C. Eppner, and D. Fox. 6-dof graspnet: Variational grasp generation for object manipulation. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 2901-2910, 2019.
[26] Y. Yuan, H. Che, Qin, B. Huang, Z.H. Yin, K.-W. Lee, Y. Wu, S.-C. Lim, and X. Wang. Robot synesthesia: In-hand manipulation with visuotactile sensing. arXiv preprint arXiv:2312.01883, 2023.
[27] Y. Li, Wu, Tedrake, J. B. Tenenbaum, and A. Torralba. Learning particle dynamics for manipulating rigid bodies, deformable objects, and fluids. In ICLR, 2019.
[28] L. Manuelli, Y. Li, P. Florence, and Tedrake. Keypoints into the future: Self-supervised cor- respondence in model-based reinforcement learning. arXiv preprint arXiv:2009.05088. 2020.
[29] H. Shi, H. Xu, Z. Huang, Y. Li, and J. Wu. Robocraft: Learning see, simulate, and shape elasto-plasti objects with graph networks. arXiv preprint wwXiv:2205.02909 2022.
[30] H. Shi, H. Xu, S. Clarke, Y. Li, and J. Wu. Robocook: Long-horizon elasto- object manipulation with diverse tools. arXiv preprint
2023.
10

---

[31] Y. Wang, Y. Li, K. Driggs-Campbell. L. Fei-Fei, and J. Wu. Dynamic-Resolutinn Model Learning for Object Pile Manipulation. In Proceedings of Robotics: Science and Systems, Daegu, Republic of Korea, July 2023. doi:10. doi:10.15607/RSS. Sttp///SS..02.XXXXXXX
[32] H. Chen, Y. Niu, K. Hong, S. Liu, Y. Wang, Y. Li, and K. R. Driggs-Campbell. Predicting object interactions with behavior primitives: An application in stowing tasks. In 7th Annual Conference on Robot Learning, 2023. URL https: /oppereeeeeeeeeeoeeeeeeeeee net forum?ir ?id= VH6WIPF4S].
[33] M. Minderer, C. Sun, R. Villegas, Cole, K. Murphy, and Lee. Unsupervised learning of object structure and dynamics from videos. Advances Neural Information Processing Systems, 32, 2019.
[34] N. D. Palo and Johns. Dinobot: Robot manipulation via retrieval and alignment with vision foundation models. In IEEE International Conference on Robotics and Automation (ICRA), 2024.
[35] W. Liu, A. Daruna, M. Patel, K. Ramachandruni, and S. Chernova. A survey of semantic reasoning frameworks for robotic systems. Robotics and Autonomous Systems, 159:104294, 2023.
[36] E. Jang, A. Irpan, M. Khansari, D. Kappler, F. Ebert, C. Lynch, S. Levine, and C. Finn. Be-z: Zero-shot task generalization with robotic imitation learning. In Conference Robot Learn- ing, pages 991-1002. PMLR, 2022.
[37] N. M. Shafiullah, Z. Cui, A. A. Altanzaya, and L. Pinto. Behavior transformers: Cloning k modes with one stone. Advances in neural information processing systems, 35:22955-22968 2022.
[38] I. Radosavovic, B. Shi, L. Fu, K. Goldberg, T. Darrell, and J. Malik. Robot learning with sensorimotor pre-training, arXiv preprint arXiv:2306. 10007, 2023.
[39] A. Padalkar, A. Pooley, A. Jain, A. Bewley, A. Herzog, A. Irpan, A. Khazatsky, A. Rai, A. Singh, A. Brohan, et al. Open x-embodiment: Robotic learning datasets and rt-x mod- els. arXiv preprint arXiv:2310.0888 2023.
[40] W. Yan, A. Vangipuram, P. Abbeel, and L. Pinto. Learning predictive representations for deformable objects using contrastive estimation. In Conference on Robot Learning, pages 564 574, PMLR, 2021.
[41] Y. Jiang, A. Gupta, Z. Zhang, G. Wang, Y. Dou, Y. Chen, L. Fei-Fei, A. Anandkumar, Y. Zhu, and L. Fan. Vima: General robot manipulation with multimodal prompts. In Fortieth Interna- tional Conference on Machine Learning, 2023.
[42] W. Yuan, C. Paxton, K. Desingh, and D. Fox. Sornet: Spatial object-centric representations for sequential manipulation. 5th Annual Conference on Robot Learning, pages 148-157. PMLR, 2021.
[43] X. Chen, K. Zheng, Z. Zeng, C. Kisailus, S. Basu, J. Cooney, J. Pavlasek, and O. C. Jenkins. Manipulation-orientn -rrientt object perception in clutter through affordance coordinate frames. In 2022 IEEE-RAS 21st International Conference on Humanoid Robots (Humanoids), pages I86- 193. IEEE, 2022.
[44] C. Ku, C. Winge, R. Diaz, W. Yuan, and K. Desingh. Evaluating robustness of visual rep- resentations for object assembly task requiring spatio-geometrical reasoning. arXiv preprint a7Xiv:3310.09943 2023.
[45] D. Paulius, Y. Huang, R. Milton, W. D. Buchanan, J. Sam, and Y. Sun. Functional object- oriented network for manipulation learning. In 2016 IEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pages 2655-2662. IEEE, 2016.

---

[46] M. Kokic, J. A. Stork, J. A. Haustein, and D. Kragic. Affordance detection for task mask-specific grasping using deep learning. 2017 IEEE-RAS 17th International Conference on Humanoid Robotics (Humanoids), pages 91-98. IEEE, 2017.
[47] R. Wu and Y. Zhao. Vat-mart: Learning visual action trajectory proposals for manipulating 3d articulated objects. In International Conference on Learning Representations (ICLR), 2022, 2022.
[48] Y. Zhao, R. Wu, Z. Chen, Y. Zhang, Q. Fan, K. Mo, and H. Dong. Dualafford: Learning collaborative visual affordance for dual-gripper manipulation. In International Conference on Learning Representations, 2023.
[49] Y. Wang, R. Wu, K. Mo, J. Ke, Q. Fan, L. J. Guibas, and H. Dong, Adaafford: Learning to adapt manipulation affordance for 3d articulated objects via few-shot interactions. In European conference on computer vision, pages 90-107. Springer, 2022.
[50] R. Wu, C. Ning, and H. Dong. Learning foresightful dense visual affordance for deformable object manipulation. In IEEE International Conference on Computer Vision (ICCV), 2023.
[51] B. Wen, W. Lian, K. Bekris, and S. Schaal. You only demonstrate once: Category-level ma- nipulation from single visual demonstration. Robotics: Science and Systems 2022, 2022.
[52] N. Di Palo and E. Johns. On the effectiveness of retrieval, alignment, and replay in manipula- tion. IEEE Robotics and Automation Letters, 2024.
[53] L. Mescheder, M. Oechsle, M. Niemeyer, S. Nowozin, and A. Geiger. Occupancy networks: Learning 3d reconstruction in function space. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 4460-4470, 2019.
[54] B. Mildenhall, Srinivasan, M. Tancik, T. Barron, R. Ramamoorthi, and Ng. Nerf: Representing scenes neural radiance fields for view synthesis. Communications of the ACM, 65(1):99-106, 2021.
[55] W. Shen, G. Yang, A. Yu, J. Wong, Kaelbling, and Isola. Distilled feature fields enable few-shot songuagg--uideed manipulation. arXiv preprint arXiv:2308.07931, 2023.
[56] A. Simeonov, Y. Du, A. Tagliasacchi, J. B. Tenenbaum, Rodriguez, P. Agrawal, and V. Sitz- mann. Neural descriptor fields: Se (3)-equivariant object representations for manipulation. In 2022 International Conference Robotics and Automation (ICRA), pages 6394-6400. IEEE, 2022.
[57] Y.-C. Lin, Florence, A. Zeng, T. Barron, Y. Du, W.-C. Ma. Simeonov, A. Garcia, and P. Isola. Mira: Mental imagery for robotic affordances. In Conference on Robot Learning, pages 1916-1927. PMLR, 2023.
[58] Y. Ze, G. Yan, Y.-H Wu, A. Macaluso, Ge, J. Ye, N. Hansen, L. E. Li, and X. Wang. Gnfactor: Multi-task real robot learning with generalizable neural feature fields. In Conference on Robot Learning, pages 284-301. PMLR, 2023.
[59] Kerr, C. M. Kim, K. Goldberg, A. Kanazawa, and M. Tancik. Lerf: Language embedded radiance fields. In International Conference Computer Vision (ICCV), 2023.
[60] A. Rashid, Sharma, C. M. Kim, Kerr, L. Y. Chen, A. Kanazawa, and K. Goldberg. Language embedded radiance fields for zero-shot task-oriented grasping, In 7th Annual Conference Robot Learning, 2023. URL https: http///ppnneeeeeeeeeeeeee net forum?id= k-Fg8JDQmc.
[61] R.-Z. Qiu, Y. Hu, G. Yang. Y. Song, Y. Fu, Ye, J. Mu, R. Yang, N. Atanasov, Scherer, and X. Wang. Learning generalizable feature fields for mobile manipulation. arXiv preprint a7X1v:2403.07563, 2024.
12

---

[62] J. Ye, N. Wang, and X. Wang. Featurenerf: Learning generalizable nerfs by distilling founda- tion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8962-8973, 2023.
[63] J. Ho. A. Jain, and P. Abbeel. Denoising diffusion probabilistic models. arXiv preprint arxiv:2006. / 1239, 2020.
[64] Xiang, Y. Qin, K. Mo, Y. Xia, H. Zhu, F. Liu, M. Liu, H. Jiang, Y. Yuan, H. Wang, L. Yi, A. X. Chang, L. Guibas, and H. Su. SAPIEN: A simulated part-based interactive environ- ment. In The IEEE Conference Computer Vision and Pattern Recognition (CVPR), June 2020.
[65] Z. Zhao, V. Kumar, Levine, and Finn. Learning fine- fine-grained bimanual manipulation with low-cost hardware. In Proceedings of Robotics: Science and Systems (RSS), 2023,
13


Reviews:
[{"review_id": "7xe8J8E6cx", "reviewer": "Reviewer_m8cy", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The authors aim to solve a behavioral cloning problem with diffusion policies. To make the model for generalizable, they adapt the perception pipeline with a 3D semantic field. They show that adding the following increases the generalization capabilities wrt. the baselines.", "summary_of_recommendation": "The paper introduces an interesting approach to improve the generalization of diffusion policies, by using 3D semantic fields as perception. The work is interesting. Nevertheless, both sematic fields and diffusion policies are not novel in the community and the authors do not propose any geometrically meaningful approach to exploit the semantic fields.Thus, it is a weak accept for me.", "technical_quality": 3, "clarity_of_presentation": 5, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 2, "questions_for_rebuttal": "- How do you select the images for the reference descriptors? Is it automatized? Could the reference descriptors represent background elements that are not relevant for solving the task?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Xd4NFpVCVe", "reviewer": "Reviewer_WbYD", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 1, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "Imitation Learning via Diffusion Policy using a multi-view semantically segmented pointcloid (\"3D semantic field\") as input instead of single view image inputs", "summary_of_recommendation": "No clear novelty, just the use of a different input representation to a standard Diffusion Policy Imitation Learning.", "technical_quality": 2, "clarity_of_presentation": 4, "potential_impact": 2, "robotics_focus": 4, "quality_of_limitations": 1, "questions_for_rebuttal": "1. What is the main novelty or contribution of the paper? How is the 3D semantic fields you show different from other works in literature that augment pointclouds with additional descriptors?\n2. Please elaborate on how the approach disambiguates geometric information (which is not that it uses semantic information)\n3. Please comment on the issue of baselines as mentioned in the review\n4. In the explanation of attending to subtle geometric details, you highlight that the semantics helps in understanding where to grasp, however, this does not explain how the \"geometric details\" help the process, but rather it's the semantics that helps the learning.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MKUhMaciC1", "reviewer": "Reviewer_ruvW", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "Generalizable Diffusion policy with information about the complete 3D point cloud and semantics", "summary_of_recommendation": "The proposed method is a strong advancement towards generalizable manipulation. However further clarifications are required.", "technical_quality": 4, "clarity_of_presentation": 4, "potential_impact": 4, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "1. Can you provide reference images and features for tasks other than shoe? Why do you need reference features for your problem statement? What happens if you only merge scene point cloud and DINOv2 feature cloud?\n\n2. Does the performance change if we use different reference (images I suppose from D3fields)? How do you choose them?\n\n3. Can you clarify figure 7 further? How are you generating different similarity fields for different parts of the object?\n\nThe above clarifications can help in understanding the capabilities of the proposed approach.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: Strengths:
+ Paper is written clearly.
+ GenDP achieves compelling results with unseen objects. 
+ Results include ablations and evaluations that showcase the generalization aspects of GenDP.

Weaknesses:
- The approach is not entirely novel. Several prior manipulation works use 3D semantic fields as noted by all the reviewers.
- Details on how reference descriptors are obtained is missing.
- Diffusion Policy baseline should use multi-view input instead of single-view RGB-D for a fairer comparison.
- GenDP needs to be better positioned among existing works like 3D Diffuser Actor, 3D Diffusion Policies, etc.

Post rebuttal:  
The author responses clarified some of the concerns regarding novelty. Additional experiments ablated different encoders.
Decision: Accept