D$^3$Fields: Dynamic 3D Descriptor Fields for Zero-Shot Generalizable Rearrangement

Authors: yixuan  wang
mingtong  zhang
zhuoran  li
tarik  kelestemur
katherine  driggs-campbell
jiajun  wu
li  fei-fei
yunzhu  li
Venue: robot-learning_org_CoRL_2024_Conference
Published: 2024-06-07 05:37:10

Abstract: Scene representation is a crucial design choice in robotic manipulation systems. An ideal representation is expected to be 3D, dynamic, and semantic to meet the demands of diverse manipulation tasks. However, previous works often lack all three properties simultaneously. In this work, we introduce D$^3$Fields---**dynamic 3D descriptor fields**. These fields are **implicit 3D representations** that take in 3D points and output semantic features and instance masks. They can also capture the dynamics of the underlying 3D environments. Specifically, we project arbitrary 3D points in the workspace onto multi-view 2D visual observations and interpolate features derived from visual foundational models. The resulting fused descriptor fields allow for flexible goal specifications using 2D images with varied contexts, styles, and instances. To evaluate the effectiveness of these descriptor fields, we apply our representation to rearrangement tasks in a zero-shot manner. Through extensive evaluation in real worlds and simulations, we demonstrate that D$^3$Fields are effective for **zero-shot generalizable** rearrangement tasks. We also compare D$^3$Fields with state-of-the-art implicit 3D representations and show significant improvements in effectiveness and efficiency. Project page: https://robopil.github.io/d3fields/

Main paper: 
 introduction: 
 The choice of scene representation is essential in robotic systems. An ideal representation is expected to be simultaneously 3D, dynamic, and semantic to meet the needs of various robotic manipulation tasks in our daily lives. However, previous research on scene representations in robotics often does not encompass all three properties. Some representations exist in 3D space [1] [2] [3] [4] , yet they overlook semantic information. Others focus on dynamic modeling [5] [6] [7] [8] , but only consider 2D data, neglecting the role of 3D space. Some other works are limited by only considering semantic information such as object instance and category [9] [10] [11] [12] [13] .In this work, we aim to satisfy all three criteria by introducing D 3 Fields, unified descriptor fields that are 3D, dynamic, and semantic. Notably, D 3 Fields are implicit 3D representations rather than explicit 3D representations like point clouds. D 3 Fields take arbitrary 3D coordinates as inputs and output both geometric and semantic information corresponding to these positions. This includes the instance mask, dense semantic features, and the signed distance to the object surface. Notably, deriving these descriptor fields requires no training and is conducted in a zero-shot manner, utilizing large visual foundation models and vision-language models (VLMs). In our approach, we employ a set of advanced models. We first use Grounding-DINO [14] , Segment Anything (SAM) [15] , XMem [16] , and DINOv2 [17] to extract information from multi-view 2D RGB images. We then project arbitrary 3D coordinates back to each camera, interpolate to compute representations from each view, and fuse these data to derive the descriptors associated with these 3D positions, as shown in Figure 1 (left). Leveraging the dense semantic features and instance mask of our representation, we achieve robust tracking 3D points of the target object instances and train the dynamics models. These learned dynamics models can be incorporated into a Model-Predictive Control (MPC) framework to plan for zero-shot generalizable rearrangement tasks.Notably, the derived representations allow for zero-shot generalizable rearrangement tasks, where the goal is specified by 2D images sourced from the Internet, smartphones, or even generated by AI models. Such goal images have been challenging to manage with previous methods, because they contain varied styles, contexts, and object instances different from the robot's workspace. Our proposed D 3 Fields can establish dense correspondences between the robot workspace and the target configurations. Given correspondences, we can define our planning cost and use the MPC framework with the learned dynamics model to derive actions for accomplishing tasks. Remarkably, this task execution process does not require any further training, offering a highly flexible and convenient interface for humans to specify tasks for the robots.We evaluate our method across a wide range of robotic rearrangement tasks in a zero-shot manner. These tasks include organizing shoes, collecting debris, and organizing office desks, as shown in Figure 1 (right). Furthermore, we provide both quantitative and qualitative comparisons with state-of-the-art implicit 3D representations to demonstrate the effectiveness and efficiency of our approach [18, 19] . Through a detailed analysis of our D 3 Fields, we offer insights into the categorylevel generalization capabilities and zero-shot rearrangement capabilities of our approach.We make three major contributions. First, we introduce a novel representation, D 3 Fields, that is 3D, semantic, and dynamic. Second, we present a novel and flexible goal specification method using 2D images that incorporate a wide range of styles, contexts, and instances. Third, our proposed robotic manipulation framework supports a broad spectrum of zero-shot rearrangement tasks.Our approach differs from Dense Object Nets (DON) in two major ways [4] :• Training Requirements: DON is trained via contrastive learning, which requires foreground and background separation and a careful selection of positive and negative pairs. This means that deploying a DON in a new environment or on new object categories requires additional training efforts. In contrast, our pipeline does not need extra training and can be applied to a diverse set of object categories by leveraging visual foundational models.• Generalization: DON is trained on small-scale datasets and may not generalize well to new object categories and scenarios. Our pipeline, however, offers stronger generalization capabilities without the need for retraining and additional data collection.2 Method Details 
related works: 
 Foundation Models for Robotics. Large Language Models (LLMs) have demonstrated promising reasoning capabilities for language. Robotics researchers have used LLMs to generate plans for manipulation [20] [21] [22] [23] . Yet, their perception modules fall short in simultaneously modeling the 3D geometry, semantics, and dynamics of objects. Meanwhile, visual foundation models, such as SAM [15] and DINOv2 [17] , have demonstrated impressive zero-shot generalization capabilities across various vision tasks. While prior visual models, like Dense Object Nets [24] , can encode similar semantic information on a small-scale dataset, these foundational models show better gen- eralization capabilities on various object categories and scenarios. However, their focus is primarily on 2D vision tasks. Grounding these models in a dynamic 3D environment remains a challenge. Recent works showcase how to ground these foundational models in the 3D world and help imitation learning to generalize [25] [26] [27] [28] . Still, these works do not emphasize dynamics learning or achieve zero-shot generalization ability.Neural Fields for Robotic Manipulation. There are various approaches leveraging neural fields as a representation for robotic manipulation [29] [30] [31] [32] [33] [34] [35] [36] [37] [38] [39] [40] [41] [42] . Among them, a series of works distilling neural feature fields from visual foundation models are closely related to us [19, [43] [44] [45] [46] . However, they often require dense camera views for a quality field, which is expensive and impractical for real-world scenarios. Also, distilled neural fields need retraining for new scenes, which is timeconsuming and inefficient. In contrast, our D 3 Fields require no training for new scenes and can work with sparse views and dynamic settings. GNFactor and FeatureNeRF train neural feature fields that can be conditioned on sparse views [18, 47] . However, such fields are often trained on a small dataset, making them hard to generalize to novel instances and scenes, whereas our D 3 Fields offer better generalization capability to new instances.We assume that there are multiple RGBD cameras with fixed viewpoints. We assume all cameras' intrinsic parameters K and extrinsic parameters T are known. The ith camera's extrinsic parameters are defined as follows:EQUATIONwhere Euclidean group SE(3) := {R, t | R ∈ SO 3 , t ∈ R 3 }. For a 3D point x in the world frame, we could obtain the pixel u i projected in ith camera and distance to ith camera r i as follows:EQUATIONwhere Proj performs perspective projection, mapping a 3D vector p = [x, y, z] T to a 2D vector q = [x/z, y/z] T .Algorithm 1 Fusion Process u i , r i ← Project(x, i), r ′ i ← R i [u i ]▷ 3D projection and depth reading 3:d i ← r ′ i -r i , d ′ i ← Truncate(d i , µ) ▷ Truncated depth difference using Eq. 3 in main paper 4:v i , w i ← Weights(d i ) ▷ Assign weights to each view using Eq. 4 in main paper 5:f i , p i ← Interpolate(W f i , W p i , u i )▷ Interpolate features using Eq. 5 in main paper 6:f , p ← Fuse(f i , p i , v i , w i ) ▷ Fuse features using Eq. 6 in main paperHere is a detailed explanation for Equation 6.• The weight w i varies based on proximity to the surface: w i = 1 when x is close, summing weights to 1, and w i approaches 0 when x is far, causing f and p to converge to 0.Equation 8 describes the process of finding the correspondence from jth sampled point's associated feature f 0 j ∈ R f to a 2D point s goal,j in the image space. The computation process consists of three steps:• α ij is the feature-space distance between jth sampled point and ith goal image pixel. We first extract goal image's feature map W goal and read out ith pixel's corresponding feature W goal [u i ]. Then we compute the L2 distance between W goal [u i ] and f 0 j and assign to α ij . • After applying the softmax to α ij over the whole image, we obtain β ij . The summation of β ij over the image space, i.e.Using Grounded-SAM [5, 6] , we could extract instance segmentation masks from each view. However, masks from different views can contain a different number of instances, and the instance IDs may not be consistent. To tackle this problem, we need to post-process the instance segmentation results. The high-level idea is to merge instances from different viewpoints based on their geometric distance.We will save all merged instances to a list. Specifically, we will first start from the first viewpoint.For each instance mask, we map them to 3D point clouds and save them into a list. Then, we move to the next viewpoint and map all instance masks to 3D point clouds. We will compare each instance with the merged instances in the list. If they have significant overlap, measured by the Intersection of Union (IoU) of the two point clouds, we will merge the instance from the new viewpoint with the merged instances in the list. This process will continue until all viewpoints have been iterated through.After merging all instances, we will filter out instances that are not stably detected. Specifically, instances that meet one of the following criteria will be filtered out:• The instance has little point cloud.• The instance is known to be a background, such as the table.• The instance overlaps with other instances, while other instances have a higher confidence.After filtering, we will assign consistent instance IDs to the instance masks in each viewpoint.After using Grounded-SAM to obtain initial masks, we use XMem to track instance masks in later frames so that we can skip time-consuming segmentation detection for subsequent frames. By applying XMem, we significantly reduce the computational cost and processing time, making the pipeline more efficient.We instantiate the dynamics model f (•, •) as graph neural networks (GNNs) that predict the evolution of particles s t ∈ R 2×ns under external actions a t . We also construct edges e t ∈ N 2×ne according to particle distance, where e t j = (u t j , v t j ) represents an edge connecting from particle u t j to particle v t j . f (•, •) consists of node and edge encodersf enc O (•, •), f enc E (•,•) to obtain node and edge representations:EQUATIONThen, we use node and edge decodersf dec O (•, •), f dec E (•,•) to predict the next time step's particle states:EQUATIONwhere N i is the index set of the edges that connect to particle i. In practice, we follow Li et al. [7] and use multi-step message passing over the graph to approximate the propagation of action impacts.To initialize keypoint tracking, we first densely sample points, which can either come from grid sampling or instances' 3D point clouds. Then, we evaluate these points using our D 3 Fields and mask out those not belonging to the desired instance. Finally, we downsample these points to the desired number using farthest point sampling.As described in Section 3.4 of the main paper, our MPC framework needs a reference camera to bridge the gap between 3D representation and 2D representation. In our work, the reference camera's extrinsic parameters are manually defined according to the tasks. Typically, it looks down at the workspace from above. Its intrinsic parameters are the same as the real camera's intrinsic parameters. The detailed MPC algorithm we used is described in Algorithm 2.For the pick-and-place tasks, our dynamics model is simplified, as the object is rigidly attached to the end-effector.We choose DINOv2 as our feature backbone because there are several great and unique properties of DINOv2, including:• Generalization: DINOv2 has demonstrated consistent feature extraction across diverse object categories and scenes, which allows us to apply our pipeline to various object categories and bridge the gap between goal image and workspace. • Refined Correspondence: DINOv2 demonstrates the capability to establish a refined correspondence, even when two images have quite different backgrounds and contexts. This is important for our since we need refined correspondence between the goal image and the workspace to define the objective function for the manipulation.However, our framework is not specific to DINOv2. If there is a better visual foundational model in the future, we could replace it easily in a plug-and-play manner.3 Additional Experiments 
problem formulation: 
 We formulate our problem as a zero-shot rearrangement problem given a 2D goal image I and RGBD images from multiple fixed viewpoints. We denote the workspace scene representation as s goal . Our goal is to find an optimal action sequence {a t } to minimize the task objective:EQUATIONwhere c( We assume that we can access multiple RGBD cameras with fixed viewpoints to construct D 3 Fields. Multi-view RGBD observations are first fed into visual foundational models. Then we obtain 2D feature volumes W. D 3 Fields are implicit functions F(•|W) defined as follows: As an overview, our pipeline first projects x into the image space of each camera. Then, we can obtain the truncated depth difference between projected depth and real depth reading using Equation 3. Afterwards, we assign weights to each viewpoint using Equation 4 and interpolate the semantic features and instance masks for each camera using Equation 5. Finally, we fuse features from all viewpoints to obtain the final descriptor using Equation 6.EQUATIONMore concretely, we map an arbitrary 3D point x to the ith viewpoint's image space. We denote the projected pixel as u i and the distance from x to the ith viewpoint as r i (Figure 3 ). By interpolating the ith viewpoint's depth image R i , we compute the corresponding depth reading from the depth image asr ′ i = R i [u i ].Then we can compute the truncated depth difference asEQUATIONwhere µ specifies the truncation threshold for the Truncated Signed Distance Function (TSDF).Given the truncated depth difference, we compute weights v i and w i for each viewpoint asEQUATIONHere is the explanation and design justification for each term. v i : It represents the visibility of x in camera i. 1 di<µ is the indicator function, which equals to 1 when d i < µ and equals to 0 otherwise. Whend i = r i -r ′ i ≥ µ,x is behind the surface, which means x is not visible in camera i and v i = 0. w i : It is the weight for the ith viewpoint. Since we only have a confident estimation when x is close to the surface, w i will decay as |d i | increases. For x that is far away, w i degrades to 0. Then we extract the semantic feature f i and instance mask p i in each viewpoint usingEQUATIONwhere DINOv2 [17] extracts the semantic feature volume W f i ∈ R H×W ×N from RGB image. W p i ∈ R H×W ×M is the instance mask volume using Grounded-SAM [14, 15] . Note that p i is a one-hot vector and already associated to ensure consistent instance indexing across different views. Finally, we fuse the semantic features and instance masks from all K viewpoints usingEQUATION)where δ is a small number to avoid numeric issues. Since the process of projection, interpolation, and fusion is differentiable, F(•|W) is differentiable when x is within the truncation threshold.This section will present how to use the dynamic implicit 3D descriptor field F(•|W) to track keypoints and train dynamics. Without loss of generality, consider the tracking of a single object instance s t ∈ R 3×ns . For clarity, we denote f and d from F(•|W) as F f (•|W) and F d (•|W). We initialize the tracked keypoints s 0 by sampling points close to the surface of the desired instance. To track keypoints s 0 , we formulate the tracking problem as an optimization problem:EQUATIONAs F(•|W) is differentiable, we could use a gradient-based optimizer. This method could be naturally extended to multiple-instance scenarios. We found that relying solely on features for tracking can be unstable. Therefore, if we know that the tracked object is rigid, we can apply additional rigid constraints and distance regularization for more stable tracking.Keypoint tracking enables dynamics model training on real data. We instantiate the dynamics model f (•, •) as graph neural networks (GNNs). We follow [48] to predict object dynamics. Please refer to [48, 49] for more details on how to train the graph-based neural dynamics model. The trained dynamics model will be used for trajectory optimization in Section 3.4.In this section, we will describe how to define the planning cost for our zero-shot rearrangement framework. As shown in Figure 2 (c), we first find the correspondence between the descriptor fields and goal image using Equation 8. Then we define the cost function c(•, •) in Equation 9to measure the distance between the current state and the goal state. Finally, we optimize the action sequence {a t } to minimize the cost function as described in Section 3.1.As described in Section 3.2, we initially sample points s 0 ∈ R 3×ns and obtain the associated features f 0 ∈ R f ×ns from the descriptor fields. We correspond s 0 to the goal image I goal to define 2D goal points s goal ∈ R 2×ns . Firstly, we compute the feature distance α ij between ith pixel u i of I goal and jth sampled point of s 0 . Then we normalize α ij using the softmax over the whole image and obtain the weight β ij . Lastly, we find the 2D point s goal,j corresponding to the jth 3D point using weighted sum. The computation process is summarized in the following:EQUATIONwhere W f goal is the feature volume extracted from I goal using DINOv2, and s is the hyperparameter to determine whether the heatmap β ij is more smooth or concentrating. Although Equation. 8 only shows a single instance case, it could be naturally extended to multiple instances by using instance mask information. Note that s goal is in the image space, and the current state representation, s t is in 3D space. To reconcile this discrepancy, we introduce a virtual reference camera. We project s t into the reference image and obtain its 2D positions s t 2D . Consequently, we define the task cost function as follows:EQUATION)Given the planning cost, we could use an MPC framework to derive the action sequence {a t } to minimize the cost function. Specifically, we use MPPI to optimize the action sequence [50] . At each time step, we sample a set of action sequences and evaluate the cost function. Then we update the action sequence using the cost function and repeat the process until convergence. This process will be repeated for each time step until the task is completed. More details regarding the method are included in the supplementary material. (b) F3RM [19] could construct a reasonable mesh in the shoe scene and establish rough correspondences, but fails in other scenes. (c) Only trained on a small dataset, FeatureNeRF [18] fails to generalize to novel scenes. The reconstructed meshes are out of camera view, and the correspondence quality is poor.For the truncation threshold µ, we set it to 0.02 across all experiments. For the prompts used for Grounded-SAM [5, 6] in Figure 7 in the main paper, from left to right, they are "shoe", "mug", "spoon", "can", and "toothpaste" respectively. The confidence threshold used for Grounded-SAM is 0.2.We also compare with several baselines, with details listed below:• Dense Object Nets (DON) [4] : We compare the effects of using different feature backbones, with DON as one of the baseline backbones. We use the pre-trained DON model since our method also uses off-the-shelf models with no re-training or additional data. Specifically, we use the model trained on the shoe class, which can be found here.• DINO [8] : Another feature backbone we baseline on is DINO, which is the precursor to DINOv2. We use the code provided by [9] to extract dense DINO features.• RGBD+DINOv2: We also compare our method to simply merging point clouds with DI-NOv2 features from multiple viewpoints [10] .• FeatureNeRF [3] : We compare our representation with other state-of-the-art 3D implicit semantic representations, including FeatureNeRF. We trained the model on the car example dataset provided by the authors. For comparison with our model, we do not distill the model from the DINO model but from DINOv2. It is worth noting that FeatureNeRF only uses one RGB image to generate the neural fields. We found that providing more views to the FeatureNeRF model during inference time leads to worse performance. Therefore, we keep its input as a single-view RGB image.• Distilled Feature Fields (F3RM) [1] : Another 3D implicit semantic representation we compare to is F3RM. We use the same training code provided by the authors, except that we distill from DINOv2 models to make it comparable with our model. In addition, we use four camera views as F3RM inputs instead of dense views as the original paper. For the dynamics training of the shoe-pushing example, we collected 20 episodes of pushing one shoe. Then, we trained a dynamics model that can take in current particles and a pushing action and predict particles in the next step.For the evaluation in the real world, we summarize the details of our tasks in Table 1 . 1 : Task Details Summary. This table summarizes our task environment, specific tasks, and objects. We evaluate our framework on eight tasks and fifteen object categories, where each object category covers several object instances with diverse appearances and shapes.We show two examples of 3D keypoint tracking in Figure 2 . In the first scenario, we track a shoe as it is pushed and subsequently flipped. The second example demonstrates tracking a shoe that is lifted and then placed down. Our system robustly tracks the shoe in 3D space. These examples underscore the effectiveness of D 3 Fields in maintaining accurate tracking in dynamic scenarios, which enables our dynamics learning capabilities.Push and Rotate Lifting We qualitatively compare the descriptor fields generated by the three methods. We extract the mesh from these fields using marching cubes, as shown in Figure 1 . We observe that our D 3 Fields could generate accurate color meshes given sparse views. FeatureNeRF could reconstruct a reasonable mesh given a single 2D image from the training distribution, as shown in Figure 1 (a). However, when it encounters a new object outside the training distribution, the reconstructed mesh will be completely off, even when we apply the image preprocessing to align the testing images with the training set in terms of image sizes, data range, and background color. Although Figure 1 (b) shows that we can reconstruct a clear mesh with dense image sequences as in the original paper, its color mesh is quite inaccurate given sparse viewpoints in our experiment setting, except for the shoe case. We use the precision-recall curve to measure the correspondence quality. Our method shows to consistently exceeds the performance of the baseline approaches, which demonstrates our method's capability to encode semantic information accurately and establish precise correspondences using the semantic information.In addition, we also measure the quantitative correspondence accuracy of our method, FeatureNeRF, and F3RM. We manually label the ground truth correspondence keypoints on the source image and the target descriptor fields. We measure the correspondence quality using the precision-recall curve, as shown in Figure 3 A larger area under the curve indicates better correspondence quality. Details regarding the precision-recall curve are provided later. We could see that F3RM and FeatureNeRF collapse to the origin point except for the shoe example for F3RM. This is because these two methods fail to reconstruct meshes given sparse observations and unseen instances. In contrast, our method shows a much better correspondence quality.To generate the precision-recall curve, we manually label one point x src on the 2D source image, and a set of corresponding 3D points x tgt . For 2D points, we obtain the associated semantic feature f src .For vertices on the reconstructed mesh, we can obtain a set of semantic features {f tgt,0 , ..., f tgt,N }.Then we compute the cosine similarity between features f src and {f tgt,0 , ..., f tgt,N }. For one similarity threshold τ , we filter out a set of points F τ with similarity scores higher than τ . Additionally, we identify the set of points G that are close to x tgt . We then define precision P τ and R τ as follows:EQUATIONBy varying τ , we can plot the precision-recall curve as shown in Figure 3 .We compare our approach using 4 views against F3RM [1] , which uses 50 views (with and without Grounding DINO + SAM supervision), on the Dense Object Nets dataset [4] .Our qualitative results are shown in Figure 4 . Similar to the main paper, We compare the reconstructed mesh, descriptor fields, and mask fields for our approach (4 views), F3RM without Grounding DINO + SAM (50 views), and F3RM with Grounding DINO + SAM (50 views). We observed that F3RM with 50 views reconstructs high-quality meshes, confirming our F3RM implementation is bug-free. Additionally, there is no significant qualitative difference between our method with 4 views and F3RM with 50 views, demonstrating the effectiveness of our representation. Lastly, we did not observe notable differences between F3RM with and without Grounding DINO + SAM, which is consistent with our previous conclusion that this additional supervision does not significantly contribute to F3RM training.Similar to our paper, we also evaluated quantitative correspondence quality using the Precision-Recall (P-R) curve, as shown in Figure 5 . A larger area under the P-R curve indicates better correspondence quality. Our results show no significant correspondence differences between our approach with 4 views and F3RM with 50 views, indicating that our representation is effective even with sparse views.In this section, we first study how different feature backbones could affect the correspondence quality. Then we show the qualitative correspondence results of our method. As mentioned in Section 3.1, we substitute our method's backbone with other pre-trained models, like DON and DINO [4, 8] . We also compare with RGBD+DINOv2 to demonstrate its effectiveness.Figure 6 shows the qualitative correspondence results of all ablation baselines. There are three key observations we can make from this figure.• Compared with RGBD+DINOv2, our method's correspondence quality is better. We achieve more accurate correspondence since our representation can amortize noise from single views by considering 3D consistency. Although RGBD+DINOv2 can achieve some spatial consistency, there are still variances in results from different viewpoints, while our method guarantees spatial consistency.• Compared with DINO, our correspondence is more fine-grained and accurate. Thanks to the advancements in foundational visual models, DINOv2 encodes more fine-grained features and enables correspondence with higher accuracy.• DON struggles to generalize to novel scenes and unseen object categories. Although the original DON shows good correspondence quality, it is trained on one type of object with a relatively small dataset. Compared with visual foundational models, it shows limited generalization capabilities in terms of scenes and object categories. We compared the reconstructed mesh, feature fields, and mask fields with F3RM using dense views. Our results showed that, despite using sparse inputs, our method did not exhibit significant qualitative performance degradation compared to F3RM under dense views, demonstrating the efficiency of our approach. We also visualize the correspondence from 2D images to our workspace as shown in Figure 7 . Specifically, we extract the DINOv2 feature of the selected pixel in the 2D image. Then we highlight the part of the 3D mesh with features close to the query feature. There are two observations regarding the qualitative correspondence results. First, the semantically similar parts are correctly matched across different instances and contexts. For example, when we select the rim of the plate in the 2D image, the corresponding part in the 3D mesh is highlighted. This matching is consistent across different object parts, such as the head and tail of the shoe, the handle and blade of the knife, and the tip and bar of the drill. Second, the correspondence is multimodal when there are multiple semantically similar object parts in the workspace. For example, when we select the spoon handle in the 2D image, multiple utensil handles in the workspace are highlighted. The correspondence qualitative results show that our D 3 Fields could establish meaningful correspondences across different instances and contexts, so that we can rely on correspondence to define the planning objective function.Similar to the main paper, we generate the precision-recall curve to quantitatively compare the correspondence quality with ablation baselines. We can make the following observations regarding the baseline correspondence results.• Compared with RGBD+DINOv2, our method shows more accurate correspondence results. This is because our D 3 Fields can average out noise from each viewpoint, while RGBD+DINOv2 accumulates noises.• DINO faces challenges in accurately distinguishing specific object components. This limitation results in less precise correspondence, as shown in Figure 3 .• Although DON can encode semantic features in seen environments and instances, it fails to generalize to novel environments and object categories. Therefore, its correspondence results are even worse than DINO, as shown in Figure 3 .In Figure 8 (a), we measure performance using the IoU between the mask of the goal image and the mask of the final state post-manipulation. Higher IoU values indicate a greater degree of alignment between the intended and achieved configurations. Our method demonstrates superior performance across five distinct object categories, consistently outshining the baseline methods. For each category, we performed 5 experiments for the evaluation results. This not only highlights its exceptional manipulation accuracy but also its robust generalization capabilities. While the DINO model exhibits some struggles, particularly in distinguishing specific object components and consequently yielding less precise results, it still performs better than DON. Although DON shows commendable results with familiar objects and configurations, its performance dips in novel scenarios, revealing a lack of generalization. These results collectively emphasize the significant advantages of our method in diverse and accurate object manipulation. the corresponding areas are highlighted in the 3D mesh. First, we observe that our representation can encode features for object parts and establish the correspondence, such as spoon tips and spoon handles. In addition, we found the correspondence can be multimodal. When the shoe head is selected, multiple shoe heads in the workspace are highlighted. At last, the correspondence is generalizable across different contexts, instances, and domains, which demonstrates our method's generalization capabilities. We measure the keypoints correspondence accuracy according to the fraction of points with accurate matches, with correct matches determined by a distance threshold. Our method is consistently better at aligning with the goal image, regardless of the chosen threshold.In Figure 8 (b), we present the correspondence results. We label 10 corresponding keypoint pairs on both the goal image and the final manipulation result to sufficiently evaluate the correspondence accuracy. The accuracy of correspondence was determined by calculating the proportion of keypoints that were accurately matched, using a predefined distance threshold as the criterion. If the distance between corresponding keypoints exceeds this threshold, they are determined as unmatched. Our method shows superior performance across various thresholds, consistently outperforming the baseline models. DINO emerges as the second-best in terms of performance, exhibiting broad applicability but with a lower precision compared to our method. Meanwhile, DON lags in performance, primarily due to its struggles with generalization in novel scenarios. These results, in conjunction with those from Figure 8 (a), reiterate our method's outstanding capabilities in both generalization and accuracy. While DINO provides reasonable applicability, it lacks the precision of our approach, and the performance of DON is hindered by its limited adaptability. additional Grounded-SAM on four scenes. We qualitatively evaluate their performance by reconstructing their mesh and visualizing descriptor fields and mask fields. Although Grounded-SAM supervision can help F3RM to segment out objects as shown in the shoe example, it does not contribute too much to reconstructing a quality mesh. In contrast, our representation can consistently reconstruct meshes and generate quality descriptor fields and mask fields across four scenes.We also trained F3RM [1] on Grounded-SAM [5, 6] and compare with our method, as shown in Figure 9 . For F3RM, we use the same setting as the main paper, except for the additional Grounded-SAM labeling. We reconstruct meshes and visualize the corresponding descriptor meshes and mask meshes, similar to what we did in the main paper. We have the following observations:• F3RM could segment out objects given additional Grounded-SAM supervision, as shown in the shoe example. This indicates that our F3RM training pipeline works as expected.• Additional Grounded-SAM supervision does not influence too much on results. We could see that reconstructed meshes are quite similar to the results from the main paper because additional Grounded-SAM supervision does not contribute too much to NeRF training. Instead, sparse camera views are the main reason for the poor performance of F3RM.• Our representation can consistently construct quality meshes and generate clear descriptor fields and mask fields, which demonstrates the effectiveness of our method.In this section, we provide more details and visualization for our debris experiments. Our initial multi-view observations and goal images are visualized in Figure 10 . In this task, we want the robot to push spreading almonds into one object pile. We could see that initially, the debris spreads over the workspace, which is quite different from the goal states. This task is challenging since the dynamics of object piles is hard to predict. In addition, such manipulation capability needs an efficient perception module so that it can gain visual feedback from the environment. Our zero-shot rearrangement framework can collect these almonds into one pile successfully using our dynamic representation and the learned dynamics model.  
experiments: 
 We design and organize our experiments to answer three key questions about our method: (1) How efficient and effective our D 3 Fields is compared to existing neural representations? (2) What kind of manipulation tasks can be enabled by our framework, and what type of generalization can it achieve?(3) Why can our D 3 Fields enable these tasks and be generalizable?In the real world, we use four RGBD cameras positioned at the corners of the workspace and employ the Kinova ® Gen3 arm for action execution. In the simulation, we use OmniGibson and the Fetch robot for mobile manipulation tasks [51] . Our evaluations span various tasks, including organizing shoes, collecting debris, tidying the table, and so on. More details are in the supplementary material.In this section, we compare our D 3 Fields with two baselines, Distilled Feature Fields (F3RM) [19] and FeatureNeRF [18] . For F3RM, we use four camera views as inputs and the DINOv2 features as the supervision and stop the distillation process at 2,000 steps, which is defaulted by the authors [19] . We first compare our D 3 Fields with F3RM and FeatureNeRF in terms of the correspondence accuracy, as shown in Figure 5 . We reconstruct the mesh from our 3D implicit representation using marching cubes. We then select DINOv2 features from the source image and visualize the We qualitatively evaluate our proposed framework on rearrangement tasks, both in the real world and in simulation, encompassing tasks such as organizing utensils, fruits, shoes, food, and mugs. The figure highlights that our representation can generalize across varied instances, styles, and contexts. For instance, in the example of organizing fruit, the goal image, unlike the workspace, is styled as a sketch drawing. Because our representation encodes semantic features, the banana in the workspace can correspond with the banana in the goal image, which allows the task to be finished. This wide range of tasks showcases the generalization capabilities of our framework.corresponding heatmap on the reconstructed feature mesh. We could see that our D 3 Fields could reconstruct the mesh with high quality and highlight semantically similar areas across different instances and contexts, while F3RM fails to reconstruct the mesh accurately, which leads to an inaccurate correspondence heatmap. Since FeatureNeRF is only pre-trained on a small dataset, it fails to construct an accurate mesh of the scene and find accurate correspondence in novel scenes, demonstrating our representation's effectiveness in reconstructing meshes and encoding semantic features.Quantitative comparisons are included in the supplementary material.We also evaluate the time efficiency of constructing the implicit representation given four RGBD views across four scenes on the machine with A6000 GPU. Our method takes 0.166±0.002 seconds, which is significantly more efficient than F3RM, which takes 88.379 ± 5.306 seconds.We conduct a qualitative evaluation of D 3 Fields in common robotic rearrangement tasks in a zeroshot manner, with partial results displayed in Figure 1 and Figure 6 . We observed several key capabilities of our framework, which are as follows: Generalization to AI-Generated Goal Images. In Figure 1 , the goal image is rendered in a Van Gogh style, which is distinctly different from those in the workspace. Since D3Fields encode semantic information, capturing shoes with varied appearances under similar descriptors, our framework can manipulate shoes based on AI-generated goal images. Compositional Goal Images. In the office desk organization task in Figure 1 , the robot first arranges the mouse and pen to match the goal image. Then, the robot repositions the mug from the top of a box to the mug pad, guided by a separate goal image of the upright mug. This example illustrates that our system can accomplish tasks using compositional goal images. Generalization across Instances and Materials. Figure 4 and Figure 1 also show our framework's ability to generalize across various instances and materials. For example, the debris collection in Figure 1 shows our framework's ability to handle granular objects. Figure 6 further shows our framework's instance-level generalization, where the goal instances distinct from ones in the workspace. Generalization across Simulation and Real World. We evaluated our framework in the simulator, as shown in the utensil organization and mug organization examples in Figure 6 . Given goal images from the real world, our framework can also manipulate objects to the goal configurations. This underscores our framework's generalization capabilities between simulation and the real world. To obtain a more thorough understanding of our D 3 Fields, we first extract the mesh using the marching cube algorithm. We evaluate vertices from the extracted mesh using our D 3 Fields and obtain the associated segmentation information and semantic features, as visualized in Figure 7 . Mask fields in Figure 7 show a distinct 3D instance segmentation in different scenarios, even clustering scenes like cans and toothpaste. The 3D instance segmentation enables the downstream planning module to distinguish and manipulate multiple instances, as shown in the mug organization tasks. Additionally, we visualize the semantic features by mapping them to RGB space using PCA. We observe that our semantic fields show consistent color patterns across different instances. In the provided shoe example, even though various shoes have distinct appearances and poses, they exhibit similar color patterns: shoe heels are represented in green, and shoe toes in red. We observed similar patterns for other object categories such as mugs and forks. The consistent semantic features across various instances help our manipulation framework to achieve category-level generalization. When encountering novel instances, our D 3 Fields can still establish the correspondence between the workspace and the goal image using semantic features. In addition, we analyze in the supplementary to show that D 3 Fields have better 3D consistency compared to simple point cloud stitching. 
conclusion: 
 In this work, we introduce D 3 Fields, which implicitly encode 3D semantic features and 3D instance masks, and model the underlying dynamics. Our emphasis is on zero-shot generalizable rearrangement tasks specified by 2D goal images of varying styles, contexts, and instances. Our framework excels in executing a diverse array of robotic rearrangement tasks in both simulated and real-world scenarios. Its performance greatly surpasses baseline methods such as FeatureNeRF and F3RM in terms of efficiency and effectiveness. Limitation. Our method lifts 2D visual foundation models to the 3D world, enabling a range of zero-shot rearrangement manipulation tasks. However, we both benefit from and are limited by their capabilities. For example, the semantic feature field is not fine-grained enough to distinguish between the right and left sides of shoes. Visual foundation models with more fine-grained semantic features are needed. In addition, rearrangement tasks could fail when they need to follow a specific manipulation order to avoid collision with other objects, e.g. a crowded scene. In the future, incorporating a task planner to handle clustered scenes could be a valuable direction. We extend the application of state-of-the-art visual foundational models to robotic manipulation tasks, offering the following improvements over prior methods:• Efficiency: Several existing works construct 3D representations from 2D models, but they are often time-consuming due to the distillation process. This limits their applicability in tasks requiring frequent visual feedback. For instance, prior works such as F3RM [1] and LERF [2] require 1.3 to 8 minutes for training. In contrast, our method takes only 0.166 seconds, making it 500 to 3,000 times faster.• Sparse Views: Some existing works, such as F3RM and LERF, require dense camera views (more than 50) to build a high-quality 3D representation, which is not suitable for typical robotic manipulation workspaces with sparse camera views (1-4). In contrast, we have developed a more efficient and effective pipeline that uses only four views.• Planning Using Reconstructed Fields: We explicitly use the reconstructed fields to find matches that define our planning objective, allowing us to employ model-based planning to solve rearrangement tasks. This contrasts with existing methods like F3RM, which require expert demonstrations for every new task.Additionally, our work distinguishes itself from FeatureNeRF by applying our representation specifically to robotic manipulation tasks, whereas FeatureNeRF does not focus on manipulation [3] . 
: 
 multi-view RGBD images and encode semantic features and instance masks using foundational models. The descriptor fields visualized in the bottom left using Principal Component Analysis (PCA) demonstrate consistent features across instances. We use our representation for rearrangement tasks given 2D goal images with diverse instances and styles in a zero-shot manner. We address pick-and-place tasks such as shoe organization and tasks requiring dynamic modeling like collecting debris. We also show that our framework can accomplish 3D manipulation and compositional task specification in the table organization task. 
 
 Appendix: β ij = 1.• Then we apply the weighted sum of pixels to obtain the corresponding pixel on the goal image s goal,j . 


Decision: **Paper summary**

This paper presents a new representation for manipulation-relevant scene features. A key advantage of this method is its performance when using sparse camera views to build a high-quality 3D scene representation. The results show how this representation enables model-based planning to solve rearrangement tasks, without requiring demonstrations like baseline approaches do.

**Review summary**


Summary of strengths:
+ This paper targets manipulation problems that are widespread in robotics and highly relevant to CoRL.
+ The proposed method leverages several pre-existing foundation models, and thus does not require any training to compute the neural field. 
+ All reviewers noted that the paper was well-written and easy to follow.


Summary of weaknesses:
- The main paper is missing quantitative results. These are included in the supplementary materials, but should be integrated into the main paper.  **[The revised paper has addressed this concern.]**
- The F3RM baseline does not perform as well as expected, leading to questions about whether it was implemented correctly. **[The revised paper has addressed this concern; the baseline performs poorly when sparse views are used, but performs as expected with more views.]**
- There is limited discussion of when/how the proposed method will fail. **[This concern is not addressed, but is a minor issue.]**

**Response to rebuttal**

The authors have extensively addressed the reviewers' concerns through additional comparisons and visualizations, and restructuring the paper to include quantitative results in the main text (rather than just in the supplementary materials). 

Two reviewers maintain their positive ratings (weak accept by FgtV and strong accept by EQkG) and one reviewer has significantly increased their rating (from strong reject to weak accept; vc8A).

**Recommendations for improvement**

* Please follow Reviewer vc8a's suggestion: "For completeness, it would be great for the authors to update the manuscript with some quantitative results comparing to F3RM as well."

Reviews: review: title: title: Review and comments on D3Fields
paper_summary: paper_summary: This paper presents a 3D representation for manipulation which has the attributes of being 3D, dynamic, and semantic. The multi-view RGBD images are taken as input to build the 3D representation. Vision foundation model features are encoded into the 3D representation. In addition, the dynamics model based on keypoints is also learned to perform trajectory optimization in manipulation. Experiments results of rearrangement tasks show the effectiveness of the proposed methods. The proposed method could also generalize to novel, AI-generated and simulated goal images.
main_review: main_review: The proposed D3Fields is a concise and effective representation for manipulation task, which mostly follows the first principle. It also compares with classic geometric representation such as pointcloud and more SOTA representation such as feature-based NeRF and show certain advantages over these two types of representations, such as more efficient than NeRF-based representation and more 3D consistent compared to point cloud stitching. The real robot experiments show promising results in terms of generalization ability of goals specified by raw RGB images. 

The paper is presented mostly clearly. The experiments are mostly well organized. However, most components of this work are built upon existing works, which might reduce the originality of the proposed method. 

Here are some concerns the author might find helpful.

In terms of trajectory optimization in the manipulation tasks, it might be interesting to compare with conventional motion planning methods since the testing scenes and objects are relatively simple. Does dynamics model learning really necessary?
questions: questions: The author could consider justifying why learning dynamics model is necessary in this manipulation task.
limitations: limitations: 3
review_summary: review_summary: This paper proposes a lightweight and effective representation which consists of semantic masks, features from vision foundation models, and dynamics models for manipulation tasks. However, most components are built upon existing work, which might hinder the originality of the contribution.
title: title: Needs more details and quantitative results
paper_summary: paper_summary: This work proposes to create a neural feature field, called D3 Fields, that leverages several foundation models. The features are computed via heuristic combinations from 4 fixed RGBD viewpoints on a tabletop manipulation environment, thus no learning is needed to compute the feature field. Given a feature field and a goal image, dense correspondences are computed and a simple cost function is optimised via MPC to obtain manipulation actions. Experiments show mainly qualitative results of how D3 Fields obtains better correspondences compared to baselines, and how it can perform rearrangement on unseen objects.
main_review: main_review: Strengths:
- The paper is well-written, clear, and easy to follow.
- The work demonstrates that one can obtain generalizable feature fields based on multiple RGBD images without having to train a network, which is an interesting concept.

Weaknesses:
- Many system and baseline details are omitted, which make the contributions hard to understand. Additionally, the ablations do not probe the system enough to understand the design choices made for the system. For example, details about how the dynamics model is trained is missing.
- The comparison to the baselines seem rocky. F3RM should be an upper bound as it is not generalizable and instead should provide a highly detailed and crisp feature field. However it is shown to fail in the manuscript and cannot even reasonably reconstruct the scene, which seems odd. More details of the F3RM training could illuminate why it is failing, or rather there may be a bug/design choice that limits its efficacy.
- Most importantly, the main paper appears to have no quantitative results, thus how this work improves the field of generalizable manipulation is completely unknown. I saw 2 subsections in the appendix about correspondense results and some manipulation, but encourage the authors to put this in the main paper so that readers can obtain insights on why the system works well.
questions: questions: Questions/Comments:
- The work seems to attack manipulation of unseen objects via generalizable feature fields. The term "zero-shot" is a bit confusing in this context as it appears there is no language being used anywhere.
- Eq 6: how come the weights are not summed to 1?
- Does the dynamics model depend on the point initialisation? How much data is used to train this? How does the dynamics model behave on seen categories vs. unseen categories?
- During manipulation, are the initial tracked keypoints s^0 selected from a single object? If so, which object is selected first and why?
- Line 151: n_s is used for the number of goal points, indicating it is the same number as the sampled points, but the softmax (Eq 8) is applied on all pixels. Is this a typo?
- Where does the virtual reference camera come from? The manuscript says top-down: in this case how are the extrinsics selected? Some goal images are not top-down, including in Fig 1 bottom right (mug) and Fig 6 column 2 (Organize Fruits).
- F3RM is trained with DINOv2. For a more comparison, it should be trained with the same foundation models used for D3 Fields (e.g. Grouding DINO + SAM).
- The F3RM paper reports 50 images from the top hemisphere of the tabletop setup for training the NeRF. Is this the case for the baseline in the paper? I suspect there may only be 4 images used to train the NeRF, which would lead to poor reconstruction results.
- For the debris experiment, what do the masks look like for the initial image and goal image? How does the learned dynamics function behave in this setting? This is quite interesting as granular material dynamics is difficult to predict.
- Section 4.4 does not show anything novel, it is just marching cubes, masks, and features. I'd recommend moving this to the appendix and replacing it with quantitative results.
- Where is X-Mem applied?
limitations: limitations: 3
review_summary: review_summary: POST-REBUTTAL UPDATE: The authors have addressed my main concern of weak baselines that provide no insight to the method. With the new comparisons against a properly trained F3RM, the paper now shows that the instantly computed D3Fields provides qualitative results that are comparable to F3RM (with the appropriate amount of images for training). For completeness, it would be great for the authors to update the manuscript with some quantitative results comparing to F3RM as well. Additionally, there are some ablations that provide more insight into design details of D3Fields. Thus, I have raised my rating to weak accept.
title: title: very relevant feature representation with evaluation on useful downstream task
paper_summary: paper_summary: The paper proposes a feature descriptor that can be queried for arbitrary 3D points in a scene. This feature descriptor is distilled from 2D foundation models such that there is a feature correspondence between a 2D image and a 3D map. This property is evaluated on a downstream task of rearranging a scene to a goal state, given by a target arrangement in a 2D image from a different scene.
main_review: main_review: strengths:
- The paper is well written and easy to follow.
- The proposed work is relevant for many downstream tasks in robotics.
- The features and rearrangement task are evaluated in a real robotic setup in a variety of scenes and objects.

weaknesses:
- The work is directly proposing foundation models as the backbone and only neural fields as an alternative. A short discussion of earlier work on dense feature correspondences, such as Dense Object Nets, would help further to categorise the work and its advantage over previous work.
- Generally, the choice of the foundation models could be better motivated. In lines 74-78, the authors claim that alternative methods that distil features, such as LeRF, "often require dense camera views for a quality field" and that their method "can work with sparse views". This statement is not supported by the paper.
- The paper only shows successful rearrangement tasks and does not discuss failure cases of the proposed approach. The limitations that are discussed mostly stem from the backbones and not from their specific application in the paper.
questions: questions: - Why is the depth R interpolated (line 112, 113) when real depth for camera view is available through the RGB-D pair? What exactly is interpolated here? How and why is this done?

- What is the benefit of fusing features in 3D compared to just projecting 2D features from single views to a single-view 3D (a.k.a. 2.5D)? Would a rearrangement task also work with single-view 3D points clouds as long as the objects of interest are in the field-of-view?

- How is the instance mask m_i (eq. 6) obtained from W^p, respectively p_i, and represented? If this is an instance index, then instance indices will not necessarily match between views and cannot simply merged over multiple views. This is broadly discussed in the supplementary material, but the paper is missing information on how to get from p_i to m_i. I suggest briefly mentioning the properties of m_i and p_i, adding an abstract function p_i -> m_i to the paper and referencing pseudo code in the supplementary material on how this mapping is done.

- Why was DINOv2 chosen to provide W^f? Is this solely for its open-set zero-shot performance which does not require collecting new training data? The supplementary material shows additional correspondence accuracy comparisons, such as to Dense Object Nets. However in the paper, the choice of DINOv2 appears mostly arbitrary. Could you better motivate the choice of DINOv2 over alternative foundation model?

- Is there any limitation in how much the starting state and the goal state can differ visually and geometrically? How much do they have to differ before no action sequence between start and target state can be found?
limitations: limitations: 2
review_summary: review_summary: The proposed implicit feature representation is very relevant to robotics and has been evaluate on a relevant task on a real robotic system. There are only minor issues regarding the choice of DINOv2 left. Post-Rebuttal: The authors addressed my concerns about DINOv2 in additional experiments.