GenDP: 3D Semantic Fields for Category-Level Generalizable Diffusion Policy

Authors: yixuan  wang
guang  yin
binghao  huang
tarik  kelestemur
jiuguang  wang
yunzhu  li
Venue: robot-learning_org_CoRL_2024_Conference
Published: 2024-06-07 08:22:42

Abstract: Diffusion-based policies have shown remarkable capability in executing complex robotic manipulation tasks but lack explicit characterization of geometry and semantics, which often limits their ability to generalize to unseen objects and layouts. To enhance the generalization capabilities of Diffusion Policy, we introduce a novel framework that incorporates explicit spatial and semantic information via 3D semantic fields. We generate 3D descriptor fields from multi-view RGBD observations with large foundational vision models, then compare these descriptor fields against reference descriptors to obtain semantic fields. The proposed method explicitly considers geometry and semantics, enabling strong generalization capabilities in tasks requiring category-level generalization, resolving geometric ambiguities, and attention to subtle geometric details. We evaluate our method across eight tasks involving articulated objects and instances with varying shapes and textures from multiple object categories. Our method demonstrates its effectiveness by increasing Diffusion Policy's average success rate on \textit{unseen} instances from 20\% to 93\%. Additionally, we provide a detailed analysis and visualization to interpret the sources of performance gain and explain how our method can generalize to novel instances. Project page: https://robopil.github.io/GenDP/

Main paper: 
 introduction: 
 Diffusion-based policies have recently shown promising results in imitation learning for real robot deployment in complex robotic manipulation tasks [1, 2] , because they can express multimodal action distribution, output high-dimensional actions, and train stably. However, existing end-toend diffusion policy frameworks do not generalize to novel instances due to their brittleness to environment variances, such as object appearances and background changes. Our work proposes a diffusion policy framework capable of category-level generalization and robust to environmental changes using 3D semantic fields. Previous work has explored the use of point clouds as policy inputs to help generalization [2, 3] , but the reliance on geometric information is insufficient for fine-grained scene understanding and generalization. For instance, as illustrated in Figure 1 (b), a marker's head and tail are geometrically ambiguous despite their functional and semantic differences. In this work, we augment geometric information with semantics to generalize to novel instances, resolve geometric ambiguity, and attend to subtle geometric details. An ideal representation should not only extract geometric information from raw observation but also retain semantic information for better category-level generalization. In this work, we introduce a diffusion policy framework that uses a scene representation in the form of 3D semantic fields. The 3D semantic fields are defined as the similarity fields, where the similarity score is higher for points closer to one object part, such as knife handles. Note that the 3D semantic fields have multiple channels, as shown in Figure 2 , where the first channel corresponds to the shoe tongue, and the last one corresponds to the shoe cuff. Our framework consists of three main modules: a 3D descriptor fields encoder, a semantic fields constructor, and an action policy. The 3D descriptor encoder takes in multi-view RGBD observations and outputs a point cloud with high-dimensional descriptors using large foundational vision models like DINOv2 [4] . These descriptors are then fed into the semantic fields constructor and converted into low-dimensional semantic fields. Finally, the policy takes in the semantic fields along with the point cloud as inputs and predicts actions. The proposed framework offers three benefits: (1) Category-level generalization: As semantic fields contain both 3D and semantic information, it guides our policy to focus on semantically meaningful parts essential for task completion, allowing generalization across instances within a category. (2) Resolve geometric ambiguity: Geometric information can be ambiguous. For example, the knife blade and knife handle are geometrically similar despite functional and semantical differences. Our semantic fields can localize space semantically close to parts important for task completion, such as knife blades, to disambiguate vague geometric information. (3) Attention to subtle semantic details: Semantic details might be lost due to real-world observation noise, such as toothbrush head and soda can tab. Some tasks are impossible to accomplish without sufficient details, such as spreading toothpaste on a toothbrush and flipping a lying soda can upwards, as shown in Figure 1 . Because our semantic fields highlight semantically distinct regions, our method can pay attention to nuanced semantic details for task completion. We systematically evaluate our method across eight tasks. Our task settings involve novel instances, ambiguous object geometry, and require attention to subtle semantic details. Compared to baseline methods, which frequently fail to generalize to novel instances due to a lack of both 3D and semantic information in their representation, our approach demonstrates significant advantages in categorylevel generalization. It also leads to substantial performance improvements in scenarios containing geometric ambiguity and subtle details. Through comprehensive experiments, we show an improvement in Diffusion Policy's average success rate on unseen instances from 20% to 93%. We also provide a detailed analysis of how well and why our method can generalize to novel instances. In this work, our contributions are threefold: (1) We propose a diffusion policy framework that uses 3D semantic fields, which leverages visual foundation models to encode the environment's geometric and semantic information. (2) Our design allows category-level generalization, differentiating geometric ambiguity, and attending to geometric details. (3) We conduct comprehensive experiments and improve the original diffusion policy's average success rate on unseen instances from 20% to 93%. We also provide a detailed analysis of how well and why our method can generalize to novel instances. task. We first take in multi-view RGBD observations (i), then extract the 3D descriptor field, with each point possessing a corresponding high-dimensional descriptor (ii) [5] . We then select reference features from 2D reference images. By computing the cosine similarity between the descriptor field and 2D reference semantic features, we could obtain several semantic fields (iii). These semantic fields, concatenated with the point cloud, are then input into PointNet++ and the diffusion policy to output predicted actions (iv). 
related works: 
 Diffusion Models for Robotics. Beginning with random samples, diffusion models operate by iterative denoising to reveal the underlying distribution. They are widely used in robotics, including navigation [6] , manipulation [1, 2, [7] [8] [9] [10] [11] [12] [13] [14] , control [15] , and planning [16] [17] [18] . Among these works, Diffusion Policy and 3D Diffusion Policy are closely related to our work [1, 2] . Diffusion Policy produces a visuomotor policy under the action diffusion model, which can model the multi-modal distribution of demonstration actions in the real world. However, it relies on multiview RGB observations, which are brittle to environment changes, including novel object instances, camera viewpoints, and background changes, and fail to generalize to novel instances. Instead of RGB observations, our work enables category-level generalization using 3D semantic fields. 3D Diffusion Policy approaches this problem using point clouds as inputs [2] . While this is proven to be sample-efficient, point clouds without semantic information can be insufficient for many manipulation tasks. In this work, we augment geometric information with semantic information using foundational vision models.Geometric Representation in Robotic Manipulation. One typical geometric representation used in robotics is point clouds [3, [19] [20] [21] [22] [23] [24] [25] [26] , which enables the policy to focus on geometry and facilitates generalization across different environments. Yet, this often results in the loss of critical RGB data, limiting the understanding of an object's semantic properties. Unlike these approaches, our approach retains both semantic and geometric information, enhancing downstream decision-making. Another line of works uses keypoints as the geometric representation in robotic manipulation [27] [28] [29] [30] [31] [32] [33] [34] . They have shown impressive generalization capabilities and high efficiency due to their low dimension. However, the resulting sparsity loses geometric details and limits the range of tasks that can be effectively addressed. In contrast, our framework employs a representation with rich geometric information and enables a broader range of tasks.Semantic Representation in Robotic Manipulation. Semantic reasoning is an ongoing field in robotics [35] . One common semantic representation learning frame is directly learning objectcentric or scene-centric embeddings from RGB observations [36] [37] [38] [39] [40] [41] [42] [43] [44] . However, these representations are hard to generalize to new environments with different backgrounds and lighting conditions. In contrast, we use 3D semantic representation, which is generalizable across diverse environments and instances. Another class of semantic representation extracts functional or affordance information from observations [43, [45] [46] [47] [48] [49] [50] [51] [52] . This line of work focuses on tasks that can be accomplished using motion primitives such as grasping, picking, and placing, while our diffusion-based policy is more flexible in action representation and tasks. A recent line of research used neural implicit models such as Occupancy Networks [53] or NeRFs [54] to encode semantic features [5, 9, 25, [55] [56] [57] [58] [59] [60] [61] . GeFF is the closest one to our work. It operates by utilizing FeatureNeRF [62] , which distills a NeRF model from foundational models. Although the distillation process allows for building NeRF upon sparse input views, it reduces the generalization capability. In contrast, we extract 3D descriptor fields without distillation, which inherits generalization capabilities from foundational vision models. In addition, GeFF uses predefined open-push-close action sequences to solve manipulation tasks. Our method differs by deploying an imitation learning policy that could accomplish more complicated tasks. 
problem statement: 
 We define our system as a Markov Decision Process (MDP) consisting of state s ∈ S and action a ∈ A. The system transition is defined through the dynamics model s t+1 = f (s t , a t ). We assume predicted actions follow the probability distribution a t ∼ P pred (a|s t ) = π(s t ), where π is the learned policy. The goal is to minimize the difference between P pred (a|s t ) and ground truth action probability P gt (a|s t ), given as a set of human demonstrations D = {τ 0 , τ 1 , ..., τ N }, where τ i represents a trajectory comprising {s 0 , a 0 , s 1 , ..., a T }. Here, the state s consists of a sequence of multi-view RGBD observations, and the action a is a sequence of robot states.We use an off-the-shelf large foundational vision model, DINOv2 [4] , to obtain semantic features from multi-view RGBD observations. Given its ability to extract consistent semantic features from the RGB images across context and instance variances, we selected it as the backbone network. We provide pseudocode for building 3D descriptor fields in Algorithm 1. We denote single-view RGBD observation as o i = (I i , R i ), with i ∈ {1, 2, ..., N } representing the camera index, consisting of an RGB image I i ∈ R H×W ×3 and a depth image R i ∈ R H×W . We first use DINOv2 to extract dense 2D feature maps W i corresponding to the RGB image I i [4] . For an arbitrary 3D point p, we project it onto the image space to find its corresponding pixel location u i and the distance to camera r i . We then interpolate to derive features f i from the feature map and the depth r ′ i from R i . The depth difference ∆r i = r ′ i -r i reflects how distant p is from the surface. When p is closer to the surface in view i, greater weight is given to f i . We fuse features from multiple viewpoints by applying a weighted sum, thus obtaining the descriptor f corresponding to p. In practice, we follow the implementation details in D 3 Fields [5] to extract point cloud P ∈ R K×3 and associated features F ∈ R K×F , where K is the point cloud's size.First, we obtain several 2D images containing the task-relevant object category and extract feature maps. Then we select features from each feature map and take the average to get a set of reference descriptors F ref ∈ R M ×F . Each descriptor represents a part of the object, such as the shoe head. We denote the set of reference descriptors F ref with M descriptors and descriptor fields F with K descriptors such thatF = {F i ∈ R F |i ∈ {1, ..., K}} and F ref = {F ref,j ∈ R F |j ∈ {1, ..., M }}.The semantic fields C ∈ R K×M are defined as the similarity between the descriptor fields and reference descriptors:EQUATION)By computing the similarity scores, we convert high-dimensional descriptor fields into Mdimensional semantic fields, where M is typically less than 5. We then concatenate semantic fields C and raw point cloud P together, which are then inputted into the policy. Project p to camera i and compute projected pixel u i and distance to camera r i 4:Obtain interpolated featuresf i = W i [u i ] 5: Obtain depth r ′ i = R i [u i ] 6:Compute depth difference ∆r i = r ′ i -r i 7:Fuse featuresF j = h(f i , ∆r i ), i ∈ {1, 2, ..., N }In this work, we model our policy as Denoising Diffusion Probabilistic Models (DDPMs), similar to Diffusion Policy [1, 63] . Instead of regressing the action directly, we train a noise predictor networkEQUATIONthat takes in noisy actions a k , current observations s, and denoising iterations k and predicts the noise ϵ k . During training, we randomly choose a denoising step k and sample noise ϵ k added to the unmodified sample a 0 . Our training loss is the difference between ϵ k and predicted noise:L = MSELoss(ϵ k , ϵ k ).(3) During the inference time, our policy starts from random actions a K and denoises for K steps to obtain the final action predictions. At each step, the action is updated followingEQUATIONwhere α, γ, and σ are hyperparameters. In practice, we input 3D semantic fields into PointNet++ and obtain visual features. Then we feed visual features into diffusion policy, similar to [1] .  
experiments: 
 In this section, we evaluate our method on eight diverse tasks. We aim to answer the following three questions through experiments. 1) How does the performance of our method compare to that of the state-of-the-art imitation learning methods? 2) How well can our method generalize to different configurations and instances? 3) What enabled our method to generalize to novel instances?For experiments in simulation, we use SAPIEN [64] to build the environments and evaluate the policies. For the real-world experiments, we use the ALOHA robot and four RealSense cameras for real-world data collection and testing [65] . Figure 3 shows our real-world setup and objects used. We evaluate our method on eight tasks, as shown in Figure 4 . We test in the simulation on Hang Mug and Insert Pencil tasks. In the real world, we evaluate our method's ability from different perspectives, such as insignificant geometric details, category generalization, and geometric ambiguity. We collected 200 demonstration episodes for the Hang Mug task, 100 episodes for Insert Pencil task, and 60 episodes for all real-world tasks. More details are listed in the supplementary material.For each task, we train one policy to accomplish that single task. We test our method on various object categories, including shoes, toothbrushes, soda cans, and knives, among others, which present significant challenges for category-level generalization due to factors such as large geometric variances, ambiguous geometric information, and nuanced geometric details. More detailed descriptions of the tasks can be found in the supplementary material. results show that our method has similar performance as diffusion policy for seen instances, but our method outperforms all baselines on unseen instances. (b) For the seen instances, diffusion policy and our method have similar performances on two simulation tasks. (c) Diffusion policy performance degrades significantly on unseen instances. In addition, our method outperforms all other baselines, which underscores our policy's capability to attend to geometric details, distinguish geometric ambiguities, and generalize to novel instances.We compare our method with three baselines: 1) Ours without Semantics: input raw point cloud without semantic fields, 2) vanilla Diffusion Policy (DP), and 3) DP with RGBD: vanilla DP but with depth observations. For all baselines, we utilize four cameras to obtain observations and input them into the policy. We evaluate different policies by the success rate. The quantitative result is summarized in Figure 4 with detailed numbers in the supplementary material. Our method shows similar performance with the original diffusion policy in the simulation on seen instances. However, when generalizing to unseen instances, diffusion policy performance degrades significantly, because it takes RGB images as input, which is brittle to environmental factors like object appearances and geometries. Therefore, the diffusion policy predicts undesirable actions and fails to accomplish the task. In contrast, our framework encodes explicit 3D and semantic information about the scene, which is robust to object appearance and geometry variances, which helps the policy to generalize to novel instances. In addition, our method consistently outperforms ours without semantics, whether for seen instances or unseen instances. The benefits from 3D semantic fields are threefold. First, they help to attend to geometric details, such as the can tab, to place the can on the table upright, while ours without semantics cannot when solely relying on geometry information. Second, 3D semantic fields help to distinguish geometric ambiguity. For instance, when grasping the knife, it is hard to tell the knife directions from ambiguous geometry, which might lead to unsafe actions like grasping the knife blade. Our 3D semantic fields encode semantic information like knife handles to distinguish geometric ambiguities and show a higher success rate in Figure 4 (c)(iii). Third, Figure 4 (c)(iv) demonstrates our 3D semantic fields could help policy focus on semantically meaningful parts to complete tasks and generalize to novel instances. We found that Diffusion Policy with RGBD is not as effective as our method. Directly adding depth observation to the diffusion policy will make the input space even larger. Therefore, it would require more demonstrations to have sufficient training data coverage, which can make it less data efficient.Figure 5 shows real-world policy rollout results. We notice different failure modes for baseline methods. For diffusion policy and diffusion policy with RGBD, when a novel instance is presented, they may stop early and fail to make progress. Due to their reliance on 2D observations, they cannot generalize to unseen instances with varying appearances and geometries. Ours without semantics often fail to differentiate geometric ambiguity, like knife handles, while our method could identify the right part for manipulation. Our method also focuses on geometric details, like the soda can tab to accomplish the can flipping task, while ours without semantics places the soda can upside down.We also analyze how well our method can generalize to novel configurations and instances. Figure 6 illustrates the predicted actions of different policies under the same observations. For the seen instances, as shown in the leftmost column, we can see that the predicted trajectories for all policies in the first example point towards the knife handle correctly. However, when the knife direction changes, our method without semantics fails to respond accordingly, while the diffusion policy and our method predict actions accordingly. This demonstrates that our method can understand the semantic difference between knife handle and knife blade, which results in safe robot behaviors.When the novel instance is presented with similar configurations, our method will predict a successful trajectory guiding toward the knife handle, while diffusion policy will predict a nonsmooth trajectory, which shows that it is brittle to environmental factors and lacks category-level generalization capabilities. Although our method without semantics predicts a smooth trajectory, it approaches the knife blade, which is unsafe.We visualize the 3D semantic fields in Figure 7 by overlaying them with the raw point cloud. These semantic fields benefit the policy for two primary reasons: 1) Highlighted points are semantically meaningful and crucial for task completion. For example, to collect books on the shelf upright, the robot must recognize the book titles. Geometric information alone is inadequate for task completion.2) Semantic fields are consistent across different instances. The mug example shows that the activation on the handle is consistent across mugs with various appearances and poses. The category-level consistency enables our method to achieve category-level generalization. We also present t-SNE analysis in the supplementary material. We also extended our 3D semantic fields and showed benefits to other imitation learning (IL) methods, such as Action Chunking with Transformer (ACT) [65] . The detailed results are presented in the supplementary material. 
conclusion: 
 In this work, we proposed a diffusion policy framework leveraging 3D semantic representations, with policies modeled as DDPMs conditioned on the semantic fields alongside point clouds. We conduct comprehensive physical experiments to show that our framework can generalize to novel instances within the category, resolve geometric ambiguity, and resolve nuanced geometric details. Compared with the vanilla diffusion policy with 20% success rate, our method can reach 93% success rate on unseen instances, demonstrating the effectiveness of our method.Limitation. For each task, we select a set of fixed reference features to construct semantic features. However, for long-horizon and finegrained tasks, such as making coffee and assembling, the policy's attention needs to adapt as tasks progress. Constructing task-specific semantic fields that can adaptively select the abstraction level, can be a future direction. In addition, our method could become more interpretable and potentially more efficient if we can incorporate the geometric properties in a more explicit fashion.  
 
 Appendix:  


Decision: Strengths:
+ Paper is written clearly.
+ GenDP achieves compelling results with unseen objects. 
+ Results include ablations and evaluations that showcase the generalization aspects of GenDP.

Weaknesses:
- The approach is not entirely novel. Several prior manipulation works use 3D semantic fields as noted by all the reviewers.
- Details on how reference descriptors are obtained is missing.
- Diffusion Policy baseline should use multi-view input instead of single-view RGB-D for a fairer comparison.
- GenDP needs to be better positioned among existing works like 3D Diffuser Actor, 3D Diffusion Policies, etc.

Post rebuttal:  
The author responses clarified some of the concerns regarding novelty. Additional experiments ablated different encoders.

Reviews: review: title: title: Using Semantic Fields for Generalizable Robotics Manipulation
paper_summary: paper_summary: The authors aim to solve a behavioral cloning problem with diffusion policies. To make the model for generalizable, they adapt the perception pipeline with a 3D semantic field. They show that adding the following increases the generalization capabilities wrt. the baselines.
main_review: main_review: The paper is easy to read and the motivation is clear. Despite the impressive results of Diffusion Policies, these models lack proper generalization capabilities, in part due to the lack of proper representation of the perception. This work explores the option of using 3D semantic fields as perception representation for improving the generalization capabilities of diffusion policies.

**Strenghts**
- The selected problem is sound and relevant for the robotics community. Figuring out strategies for improved generalization is a highly relevant topic for the community.
- The selected experiments are good. Through the evaluation phase, they evaluate the influence of the semantic fields and 3D representations for generalization. The evaluation is well aligned with the contribution they are aiming to have with their presented work.

**Weaknesses**
- The authors fail to properly explain how the semantic fields are built. In the experimental section, they claim they collected 100 demonstrations for the insert pencil task. For computing the semantic fields, they make use of reference descriptors. How do you obtain them? Which images do you use from the 100 demonstrations? The authors could spend a paragraph explaining exactly how these reference descriptors are obtained.
- A highly related work is '3D Diffuser Actor: Policy Diffusion with 3D Scene Representations'. The author cites the work but they fail in explaining what makes their work different from theirs. Given both methods use 3D feature fields and diffusion policy, it might be interesting to add some lines highlighting the benefit of their method wrt. this one.
- Despite the interesting properties of the semantic fields, the authors fail to exploit properly their geometric properties. In their work, they take PointNet++ and transform the 3D semantic field into a visual feature vector that is fed into the diffusion policy. This differs from other works using Semantic Fields, such as Distilled Feature Fields that exploit the 3D geometry of the perception to generate actions. The authors could highlight this as a limitation and propose approaches for extension.
- Both semantic fields and diffusion policies are not novel in the community and the authors do not propose any geometrically meaningful approach to exploit the semantic field, but simply combine the parts through a visual feature representation.
questions: questions: - How do you select the images for the reference descriptors? Is it automatized? Could the reference descriptors represent background elements that are not relevant for solving the task?
limitations: limitations: 2
review_summary: review_summary: The paper introduces an interesting approach to improve the generalization of diffusion policies, by using 3D semantic fields as perception. The work is interesting. Nevertheless, both sematic fields and diffusion policies are not novel in the community and the authors do not propose any geometrically meaningful approach to exploit the semantic fields.Thus, it is a weak accept for me.
title: title: No main novel contribution, just a different input representation
paper_summary: paper_summary: Imitation Learning via Diffusion Policy using a multi-view semantically segmented pointcloid ("3D semantic field") as input instead of single view image inputs
main_review: main_review: The authors propose a pipeline for incorporating semantic information into pointclouds reconstructed from multiple views via descriptors/embeddings from a Vision Foundation Model.  This "3D Semantic Fields"  is then used for downstream imitation learning with a PointNet++ encoder. 

The main focus of the paper is the use of 3D semantic fields i.e. Pointcloud with dense feature descriptors with Diffusion Policy (DP). This itself doesn't warrant a novel contribution as it neither proposes any novelty in the feature descriptor input or the learning, nor is there any specific improvement done on top of DPs. It's just the use of a semantic pointcloud representation with a specific encoder suitable for such an input. While the approach shows improvement over vanilla DPs or vanilla DP with  RGBD input, or the approach without semantic pointclouds but just RGBD pointclouds, there could have been some fairer comparison with other works or with different abalations to show the success of the approach.

Major Comments:
- L31-L32 "we augment geometric information with semantics" -> Many papers that already do this 
- "resolve geometric ambiguity" It is unclear what is the main way this is resolved other than using the semantic information for resolving instead of the 3D aspects. 
- "and attend to subtle geometric details" More an outcome of the learning, than an explicit methodological/theoretical contribution. Additionally, rather than attending to the geometric details, it is the semantics that gets attended on that help resolve ambiguities, as mentioned in L188-189 " First, they help to attend to geometric details, such as the can tab, to place the can on the table upright, while ours without semantics cannot when solely relying on geometry information" This shows that it does not rely on the geometry, but more on the semantics that causes the main increase in performance.
- Line 35-36 "we introduce a novel diffusion policy framework" This is an overstatement since they just use a different encoder with off-the-shelf Diffusion policy imitation learning. There is no novelty as such in the diffusion aspect nor the policy learning, nor the representation learning.
- Line 36 "uses a scene representation in the form of 3D semantic fields." This is also done by many works, some of which are also mentioned in the related works. In L103, The authors mention that "In contrast, we extract 3D descriptor fields without distillation", however, it's not explained why that is a better approach, since some similar approaches work well with such neural feature fields for extracting semantically similar areas for doing robot manipulation.

  Florence, Peter R., Lucas Manuelli, and Russ Tedrake. "Dense Object Nets: Learning Dense Visual Object Descriptors By and For Robotic Manipulation." Conference on Robot Learning. PMLR, 2018.

  Xu, Zhenjia, et al. "Learning 3D Dynamic Scene Representations for Robot Manipulation." Conference on Robot Learning. PMLR, 2021.

  Shen, William, et al. "Distilled Feature Fields Enable Few-Shot Language-Guided Manipulation." Conference on Robot Learning. PMLR, 2023.

- L52: "to disambiguate vague geometric information" It's not that the geometric info is disambiguated, it's just that the semantic information enables the detection of relevant features irrespective of the geometry (as seen in the ablation in the appendix), which is the general contribution of such 3D semantic pointclouds

  Gu, Qiao, et al. "Conceptgraphs: Open-vocabulary 3d scene graphs for perception and planning." arXiv preprint arXiv:2309.16650 (2023).

- L127-L128 "We fuse features from multiple viewpoints by applying a weighted sum, thus obtaining the descriptor f corresponding to p" This is  similar to doing a distilliation, which the authors claim in L103 is not what they do. Here are some similar works for doing such feature fields or dense 3D descriptor pointclouds.

  Zhou, Shijie, et al. "Feature 3DGS: Supercharging 3D Gaussian Splatting to Enable Distilled Feature Fields." arXiv preprint arXiv:2312.03203 (2023).

  Shen, William, et al. "Distilled Feature Fields Enable Few-Shot Language-Guided Manipulation." Conference on Robot Learning. PMLR, 2023.

- While comparing DP+RGBD, rather than using a standard DP pipeline, using a more RGB-D oriented encoder, like Red-Net, or RD3D would have been a fairer comparison since a pointcloud specific encoder is one of the main contribution of the work. Additionally, using a Semantic-informed encoder on the RGB with diffusion policy would be another tangible baseline to run for a fairer evaluation of a semantically-aware Diffusion policy. Moreover, the current RGBD pointcloud comes from multiple views whereas a standard DP would use just a single input. Using a single view pointcloud, would be a fairer comparison to better understand how the partial observability plays a role in the learning.


It is due to these shortcomings and more importantly the lack of a strong novel contribution that makes me unable to recommend the paper for acceptance.
questions: questions: 1. What is the main novelty or contribution of the paper? How is the 3D semantic fields you show different from other works in literature that augment pointclouds with additional descriptors?
2. Please elaborate on how the approach disambiguates geometric information (which is not that it uses semantic information)
3. Please comment on the issue of baselines as mentioned in the review
4. In the explanation of attending to subtle geometric details, you highlight that the semantics helps in understanding where to grasp, however, this does not explain how the "geometric details" help the process, but rather it's the semantics that helps the learning.
limitations: limitations: 1
review_summary: review_summary: No clear novelty, just the use of a different input representation to a standard Diffusion Policy Imitation Learning.
title: title: Combination of D3fields and diffusion policy, impactful contribution, extensive analysis
paper_summary: paper_summary: Generalizable Diffusion policy with information about the complete 3D point cloud and semantics
main_review: main_review: The authors propose a generalizable 3D diffusion policy trained on the complete point cloud of the scene in conjunction with a high-dimensional descriptor field. The descriptor field processes segmentation and feature extraction via foundation model DINOv2 to construct a 3D feature cloud from RGBD images (4 cameras). The constructed feature cloud is merged with the scene point cloud for input into diffusion policy. Extensive comparison is done for generalization in terms of instance, geomtery and category. Real world experiments are also shown for the ALOHA setup.

Strengths:
This paper is built upon D^3 fields to extract the feature clouds with diffusion policy. The resulting GenDP is able to leverage semantic information to perform better than diffusion policy with raw point cloud information only.

Exploiting the representations captured by DINOv2, the trained policy is able to understand different parts of the object and correlate between seen and unseen objects of same category. for example, handles of knife and heads of different brush.

Weaknesses:
The work has limited novelty (considering diffusion policy with 3D point cloud input well explored [2]), relies on the complete point cloud of the scene and the object (more specifically) and is limited to short task lengths.

I thank the authors for presenting their method clearly. I consider this paper as an important contribution towards generalizable manipulation.

[1] https://arxiv.org/abs/2309.16118/
[2] https://arxiv.org/abs/2403.03954
questions: questions: 1. Can you provide reference images and features for tasks other than shoe? Why do you need reference features for your problem statement? What happens if you only merge scene point cloud and DINOv2 feature cloud?

2. Does the performance change if we use different reference (images I suppose from D3fields)? How do you choose them?

3. Can you clarify figure 7 further? How are you generating different similarity fields for different parts of the object?

The above clarifications can help in understanding the capabilities of the proposed approach.
limitations: limitations: 3
review_summary: review_summary: The proposed method is a strong advancement towards generalizable manipulation. However further clarifications are required.