D$^3$Fields: Dynamic 3D Descriptor Fields for Zero-Shot Generalizable Rearrangement
Authors: Yixuan Wang
Mingtong Zhang
Zhuoran Li
Tarik Kelestemur
Katherine Rose Driggs-Campbell
Jiajun Wu
Li Fei-Fei
Yunzhu Li
Venue: CoRL 2024 (2024)
Keywords: Implicit 3D Representation
Visual Foundational Model
Zero-Shot Generalization
Robotic Manipulation

Abstract: Scene representation is a crucial design choice in robotic manipulation systems. An ideal representation is expected to be 3D, dynamic, and semantic to meet the demands of diverse manipulation tasks. However, previous works often lack all three properties simultaneously. In this work, we introduce D$^3$Fields---**dynamic 3D descriptor fields**. These fields are **implicit 3D representations** that take in 3D points and output semantic features and instance masks. They can also capture the dynamics of the underlying 3D environments. Specifically, we project arbitrary 3D points in the workspace onto multi-view 2D visual observations and interpolate features derived from visual foundational models. The resulting fused descriptor fields allow for flexible goal specifications using 2D images with varied contexts, styles, and instances. To evaluate the effectiveness of these descriptor fields, we apply our representation to rearrangement tasks in a zero-shot manner. Through extensive evaluation in real worlds and simulations, we demonstrate that D$^3$Fields are effective for **zero-shot generalizable** rearrangement tasks. We also compare D$^3$Fields with state-of-the-art implicit 3D representations and show significant improvements in effectiveness and efficiency. Project page: https://robopil.github.io/d3fields/

F'eillss: Dynamic 3D Descriptor Fields for Zero-Shot Generalizable Rearrangement
Yixuan Wang"", Mingtong Zhang² Zhuoran Li Tarik Kelestemur Katherine Saiggs-Campbell' Jiajun Wu", Li Fei-Fei", Yunzhu Li'
CColumbia University, Unniversity of Illinois, Urbana- Chaanaa-aaapiinn Naational University of Singapore, 4Boston Dynamies Al Institute, Stanford University
Abstract: Scene representation crucial design choice in robotic manipulation systems. An ideal representation expected to be 3D, dynamic, and semantic meet the demands of diverse manipulation tasks. However, previous works often lack all three properties simultaneously. In this work, we introduce Fiiillss- dynamic 3D descriptor fields. These fields are implicit 3D representations that take in 3D points and output semantic features and instance masks. They can also capture the dynamics the underlying 3D environments. Specifically, we project arbitrary 3D points in the workspace onto multi-view 2D visual observations and interpolate features derived from visual foundational models. The resulting fused descriptor fields allow for flexible goal specifications using 2D images with var- ied contexts, styles, and instances. To evaluate the effectiveness of these descriptor fields, we apply our representation to rearrangement tasks in a zero-shot manner. Through extensive evaluation in real worlds and simulations, we demonstrate that D'Fields are effective for zero-shot generalizable rearrangement tasks. We also compare D'Fields with sate-of-the-as implicit 3D representations and show sig- nificant improvements in effectiveness and efficiency. Project Page
Keywords: Implicit 3D Representation, Visual Foundational Model, Zero-Shot Generalization, Robotic Manipulation
Organize
Collect Debris
Organize Office Table
Inntan Stines
Raw Visual Observation
Foundation Model
Grounded- SAM
DINOv2
Descriptor Field
(Al-Gensrated Goal Image) (Granular Material)
(Compositional Goal Image and 3D Manipulation)
Figure 1: FiFielss Representation and Application to Zero-Shot Rearrangement Tasks. Dieill take in multi-view RGBD images and encode semantic features and instance masks using foundational models. The descriptor fields visualized the bottom left using Principal Component Analysis (PCA) demonstrate consis- tent features across instances. diverse instances and styles in a zero-shot manner. We address pick-and-place tasks such as shoe organization and tasks requiring dynamic modeling like collecting debris. We also show that our framework can accomplish 3D manipulation and compositional task specification in the table organization task.
use our representation for rearrangement tasks given goal images with
Denotes equal contribution.
8th Conference Robot Learning (CoRL. 2024), Munich, Germany.

---

Introduction
The choice scene representation essential in robotic systems, An ideal representation is ex- pected to be simultaneously 3D, dynamic, and semantic to meet the needs of various robotic ma- nipulation tasks in our daily lives. However, previous research scene representations in robotics often does not encompass all three properties. Some representations exist in 3D space [1-4], yet they overlook semantic information. Others focus on dynamic modeling [5-8], but only consider 2D data, neglecting the role of 3D space. Some other works are limited by only considering semantic information such as object instance and category [9-13].
In this work, we aim satisfy all three criteria by introducing DDFields, unified descriptor fields that are 3D, dynamic, and semantic. Notably, D'Fields are implicit 3D representations rather than explicit 3D representations like point clouds. D'Fields take arbitrary 3D coordinates as inputs and output both geometric and semantic information corresponding to these positions. This includes the instance mask, dense semantic features, and the signed distance to the object surface. Notably, deriving these descriptor fields requires no training and is conducted in a zero-shot manner, utilizing large visual foundation models and vision-language models (VLMs). In our approach, we employ set of advanced models. We first use Grounding-DINS [14], Segment Anything (SAM) [15], XMem [16]. and DINOv2 [17] to extract information from multi-view 2D RGB images. We then project arbitrary 3D coordinates back to each camera, interpolate to compute representations from each view, and fuse these data to derive the descriptors associated with these 3D positions, as shown in Figure I (left). Leveraging the dense semantic features and instance mask of our representation, we achieve robust tracking 3D points of the target object instances and train the dynamics mod- Control (MPC)
els. These learned dynamics models can be incorporated into a framework plan for zero-shot generalizable rearrangement tasks.
Notably, the derived representations allow for zero-shot generalizable rearrangement tasks, where the goal is specified by 2D images sourced from the Internet, smartphones, or even generated by Al models. Such goal images have been challenging to manage with previous methods, because they contain varied styles, contexts, and object instances different from the robot's workspace. Our proposed Fieils can establish dense correspondences between the robot workspace and the target configurations. Given correspondences, we can define our planning cost and use the MPC frame- work with the learned dynamics model to derive actions for accomplishing tasks. Remarkably, this task execution process does not require any further training. offering highly flexible and convenient interface for humans to specify tasks for the robots.
We evaluate our method across wide range robotic rearrangement tasks a zero-shot man- ner. These tasks include organizing shoes, collecting debris, and organizing office desks, as shown in Figure (right). Furthermore, we provide both quantitative and qualitative comparisons with siae--oftteettte implicit 3D representations to demonstrate the effectiveness and efficiency of our approach [18, 19J. Through a detailed analysis of our DiFields, we offer insights into the category- level generalization capabilities and zero-shot rearrangement capabilities of our approach.
We make three major contributions. First, we introduce a novel representation, D'Fields, that is 3D, semantic, and dynamic. Second, we present a novel and flexible goal specification method using 2D images that incorporate wide range of styles, contexts, and instances. Third, our proposed robotic manipulation framework supports broad spectrum of zero-shot rearrangement tasks.
2 Related Works
Foundation Models for Robotics. Large Language Models (LLMs) have demonstrated promis- ing reasoning capabilities for language. Robotics researchers have used LLMs to generate plans for manipulation [20-23]. Yet, their perception modules fall short in simultancously modeling the 3D geometry, semantics, and dynamics of objects. Meanwhile, visual foundation models, such as SAM [15] and DINOv2 [17], have demonstrated impressive zero-shot generalization capabilities across various vision tasks. While prior visual models, like Dense Object Nets [24], can encode similar semantic information small-scale dataset, these foundational models show better gen-

---

Distance
d
Curz. Obs.
Goal Image
D'Fields
Multi-View Observations
Reconstructed Mesh
Poundation Model
Grounded-
SAM
DINOV2
Semantic Feature
Marching Cubes
Descriptor Field
Descriptor Map
Correspondance
F(x(W1
Descriptor Field
Instance
Probability
MSF Lass
3D Workspace
Mask Field
Planning Cost
(a) Fusion
(b) Representation
(c) Planning
Figure 2: Overview of the Proposed Framework. (a) Multi-view RGBD observations are first processed by foundation models to obtain the feature volume W The implicit function takes in arbitrary 3D points and outputs corresponding distance d, semantic features f, and instance probability p. (b) Through marching cubes, could reconstruct the mesh from the implicit signed distance function. Since our representation also encodes instances masks and semantic features for evaluated 3D points,
meshes for the mask field descriptor field as well. (c) Given a 2D goal image, we use foundation models extract descriptor map. Then we correspond 3D features to 2D features and define the planning cost based on the correspondence.
eralization capabilities on various object categories and scenarios. However, their focus primarily on 2D vision tasks. Grounding these models in a dynamic 3D environment remains a challenge. Re- cent works showcase how ground these foundational models in the 3D world and help imitation learning generalize [25-28]. Still, these works do emphasize dynamics learning achieve zero-shot generalization ability.
Neural Fields for Robotic Manipulation. There are various approaches leveraging neural fields as a representation for robotic manipulation [29-42]. Among them, series of works distilling neural feature fields from visual foundation models are closely related us [19, 43-46). However, they often require dense camera views for quality field, which is expensive and impractical for real-world scenarios. Also, distilled neural fields need retraining for new scenes, which is time- consuming and inefficient. In contrast, our Fieies require no training for new scenes and can work with sparse views and dynamic settings. GNFactor and FeatureNeRF train neural feature fields that can be conditioned on sparse views [18, 47]. However, such fields are often trained on small dataset, making them hard to generalize to novel instances and scenes, whereas our DDTields offer better generalization capability new instances.
3 Method
3.1
Problem Formulation
We formulate problem zero-shot rearrangement problem given 2D goal image I and RGBD images from multiple fixed viewpoints. We denote the workspace scene representation as 8goal Our goal is to find an optimal action sequence {a') to minimize the task objective:
min (az)
c(s". 8goul);
(1)
s.t. s¹ = g(0'),
s1+1 s'+1= S(8", (()
where c(.,,) is the cost function measuring the distance between the terminal representation sT and the goal representation Sgeal. Representation extraction function g(-) takes in the current multi-view

---

RGBD observations o' and outputs the current representation s'. f(-.-) is the dynamics function that predicts the future representation s conditioned on the current representation s and action a'. The optimization aims to find the action sequence {at} that minimizes the cost function c(s", Sgaal).
Section 3.2 describes how construct representation extraction function g(-). Section 3.3 elaborates on the dynamics function ((.)). In Section 3.4, we show how to define the cost function c(:,)).
3.2 Fields Representation
We assume that we can access multiple RGBD cameras with fixed viewpoints to construct D'Fields. Multi-view RGBD observations are first fed into visual foundational models. Then we obtain 2D feature volumes W. Fields are implicit functions F(-IW) defined as follows:
(d. (d.fpp) p) = F(x(W)
where x can be an arbitrary 3D coordinate in the world frame, and (d, f. p) corresponds to the signed distance d E R. the semantic descriptor E R and the instance probability distribution p E R&M of M instances. could different across scenarios.
As an overview, our pipeline first projects depth image
surface
into the image space of cach camera. Then, we can obtain the truncated depth difference between projected depth and real depth read- ing using Equation 3. Afterwards, we assign weights to each viewpoint using Equation 4 and interpolate the semantic features and instance masks for each camera using Equation 5. Fi- nally, we fuse features from all viewpoints to obtain the final descriptor using Equation 6.
view ra
X
F4
Figure 3: Notation Illustration. is the distance be- tween 3D point x camera and r is the interpo- lated depth from the depth image.
More concretely, we map an arbitrary 3D point x to the ith viewpoint's 's image space. We denote the projected pixel as u, and the distance from x to the ith viewpoint as r; (Figure 3). By interpolating the ith viewpoint's is depth image R,, we compute the corresponding depth reading from the depth image rf R.]u,.. Then
compute the truncated depth difference
d, = r; - r/ d' = max max(min po) -p),
where specifies the truncation threshold for the Truncated Signed Distance Function (TSDF). Given the truncated depth difference, we compute weights v; and w; for each viewpoint as
min (μ -1d1,,,.)) (dal.0)
0 = Ld wi = exp
(4)
Here is the explanation and design justification for each term. Uy: It represents the visibility of x in camera i. 1d,<a is the indicator function, which equals to
when d; < and equals to otherwise. When d, = r; - r 2 x is behind the surface, which means x is not visible in camera i and v; = 0.
Wi: It is the weight for the ith viewpoint. Since we only have a confident estimation when x is close to the surface, will decay as dd| increases. For x that is far away, w; degrades to 0. Then we extract the semantic feature f; and instance mask in each viewpoint using
Wf(u.]] p: = W.P[,l)
where DINOv2 [17] extracts the semantic feature volume W. H from RGB image. WP E & xM is the instance mask volume using Grounded-SAM [14, 15]. Note that p: is one-hot vector and already associated to ensure consistent instance indexing across different views. Finally, we fuse the semantic features and instance masks from all K viewpoints using
K d Σ dpdi
f= E w.w.f
ΣK E U.W.P.
D
8 Di=
8 SK
E
8 8
where 8 is small number to avoid numeric issues. Since the process of projection, interpolation, and fusion is differentiable, F(: is differentiable when is within the truncation threshold.

---

3.3 Keypoints Tracking and Dynamics Learning
This section will present how to use the dynamic implicit 3D descriptor field F(:(W) to track key- points and train dynamics. Without loss of generality, consider the tracking of single object in- stance of E 3x For clarity, we denote f and d from F(-W) as F+(-(W) and Fa(-IW). We initialize the tracked keypoints s" by sampling points close to the surface of the desired instance. To track keypoints 80, we formulate the tracking problem as an optimization problem:
min a+++1
|Fr(s*+(W) Fr
As F(IW) is differentiable, we could use gradient-based optimizer. This method could be natu- rally extended to multiple-instance scenarios. We found that relying solely on features for tracking can be unstable. Therefore, if we know that the tracked object is rigid, we can apply additional rigid constraints and distance regularization for more stable tracking.
Keypoint tracking enables dynamies model training on real data. We instantiate the dynamics model -()-)) as graph neural networks (GNNs). We follow [48] predict object dynamics. Please refer to [48, 49] for more details on how train the graph- based neural dynamics model. The trained dynamics model will be used for trajectory optimization in Section 3.4.
3.4 Zero-Shot Generalizable Robotic Rearrangement
In this section, we will describe how define the planning cost for zero-shot rearrangement framework. As shown in Figure 2 (c), first find the correspondence between the descriptor fields and goal image using Equation 8. Then we define the cost function c(.,) in Equation 9 to measure the distance between the current state and the goal state. Finally, we optimize the action sequence {ar} to minimize the cost function as described in Section 3.1.
As described in Section 3.2, we initially sample points s E R33x= and obtain the associated features f /xn, from the descriptor fields. We correspond 8 the goal image Igeal define compute the feature distance aij between ith pixel u, of Igeal and jith sampled point of s". Then we normalize Okj using the softmax over the whole image and obtain the weight 3.g. Lastly, we find the 2D point &geal.j corresponding to the jth 3D point using weighted sum. The computation process is summarized in the following:
2D goal points goal E _g2 Firstly,
exp (-
HxW
aij = IlW goal [u.] - 19912. Byg
H W E
exp (-sa5)
Sgoal.j =
Σ B.
(8)
where Wf is the feature volume extracted from Ignal using DINOv2, and is the hyperparameter to determine whether the heatmap 3ig is more smooth concentrating. Although Equation. 8 only shows a single instance case, it could be naturally extended to multiple instances by using instance mask information.
gual
Note that &aaal in the image space, and the current state representation, sf is in 3D space. To reconcile this discrepancy,
introduce virtual reference cam- era. We project of into the reference image and obtain its 2D positions sip- Consequently, we define the task cost function as follows:
c(s', 8gua)) ||
8gaull 112
Given the planning cost, could use MPC frame- work to derive the action sequence {ar} to minimize the cost function. Specifically, we use MPPI to opti- mize the action sequence [50]. At each time step, we sample a set action sequences and evaluate the cost function. Then we update the action sequence using the cost function and repeat the process until conver- gence. This process will be repeated for each time step until the task is completed. More details regarding the method are included in the supplementary material.
Figure 4: Object Set in Our Experiments. This figure shows diverse objects used in our experiments, expanding over 10 object types.
5

---

Sounce Image
Thrger Image
Ours
Corr. Mesis
Targer Image
F3RM
Corr. Mesh
Target Image
FestureNeRt
Corr. Meth
Figure 5: Correspondence Qualitative Comparison. We select the pixel from the source image, obtain the associated DINOv2 feature, and visualize the correspondence heatmap on the reconstructed mesh. (a) Our representation reconstructs clear mesh and corresponds from the source image to semantically similar 3D areas. (b) F3RM [19] could construct reasonable mesh in the shoe fails other scenes. (c) Only trained on small dataset, FeatureNeRF [18] fails to generalize to novel scenes. The reconstructed meshes are out of camera view, and the correspondence quality is poor.
and establish rough correspondences, but
Experiments
We design and organize our experiments to answer three key questions about our method: (1) How efficient and effective our Fieils is compared to existing neural representations? (2) What kind of manipulation tasks can be enabled by framework, and what type of generalization can it achieve? (3) Why can our Fields enable these tasks and be generalizable?
4.1 Experiment Setup
In the real world, we use four RGBD cameras positioned the corners of the workspace and employ the K R Gen3 arm for action execution. In the simulation, we use OmniGibson and the Fetch robot for mobile manipulation tasks [51]. Our evaluations span various tasks, including organizing shoes, collecting debris, tidying the table, and so on. More details are in the supplementary material.
4.2 Field Efficiency and Effectiveness
In this section, we compare our Fields with two baselines, Distilled Feature Fields (F3RM) [19] and FeatureNeRF [18]. For F3RM, we use four camera views as inputs and the DINOv2 features as the supervision and stop the distillation process at 2,000 steps, which defaulted by the au- thors [19]. We first compare our DoFields with F3RM and FeatureNeRF in terms of the correspon- dence accuracy, shown in Figure 5. We reconstruct the mesh from our 3D implicit representation using marching cubes. We then select DINOv2 features from the source image and visualize the

---

Organize
Organize
Organize
Organize (sim)
Organize (sim)
Organize Mugs (sim)
Initial Stotes
Pinal States
Goai Images
(Generalize to Novel Instances)
(AI-Generated Goal Image with Varied Styles)
(Genenalize between Real and Sim)
Figure 6: Qualitative Results. We qualitatively evaluate proposed framework rearrangement tasks, both in the real world and in simulation, encompassing tasks such organizing utensils, fruits, shoes, food, and mugs. The figure highlights that our representation can generalize across varied instances, styles, and contexts. For instance, in the example of organizing fruit, the goal image, unlike the workspace, is styled as a sketch drawing. Because our representation encodes semantic features, the banana in the workspace can correspond with the banana in the goal image, which allows the task to be finished. This wide range of tasks showcases the generalization capabilities of our framework.
corresponding heatmap the reconstructed feature mesh. We could that Diells could reconstruct the mesh with high quality and highlight semantically similar areas across different instances and contexts, while F3RM fails to reconstruct the mesh accurately, which leads to an inac- curate correspondence heatmap. Since FeatureNeRF is only pre-trained on small dataset, it fails to construct an accurate mesh of the scene and find accurate correspondence in novel scenes, demon- strating our representation's effectiveness in reconstructing meshes and encoding semantic features. Quantitative comparisons are included in the supplementary material.
We also evaluate the time efficiency of constructing the implicit representation given four RGBD views across four scenes on the machine with A6000 GPU. Our method takes 0.166=0.002 seconds, which is significantly more efficient than F3RM, which takes 88.379 + 5.306 seconds.
4.3
Zero-Shot Generalizable Rearrangement
We conduct qualitative evaluation of Fields in common robotic rearrangement tasks in zero- shot manner, with partial results displayed in Figure 1 and Figure 6. We observed several key capabilities of our framework, which are as follows:
Generalization to AI-Generated Goal Images. In Figure 1, the goal image is rendered in a Van Gogh style, which distinetly different from those in the workspace. Since D3Fields encode seman- tic information, capturing shoes with varied appearances under similar descriptors, our framework manipulate shoes based Al-generated goal images.
Compositional Goal Images. In the office desk organization task in Figure 1, the robot first arranges the mouse and pen to match the goal image. Then, the robot repositions the mug from the top of a box to the mug pad, guided by a separate goal image of the upright mug. This example illustrates that our system can accomplish tasks using compositional goal images.
Generalization across Instances and Materials. Figure 4 and Figure also show our framework's ability to generalize across various instances and materials. For example, the debris collection in Figure shows our framework's 's ability to handle granular objects. Figure further shows our frame- work's instance-level generalization, where the goal instances distinct from ones in the workspace. Generalization across Simulation and Real World. We evaluated our framework in the simulator, as shown in the utensil organization and mug organization examples in Figure 6. Given goal images from the real world, framework can also manipulate objects the goal configurations. This underscores our framework's s generalization capabilities between simulation and the real world.
7

---

Obseraation
Fields
Deseriptor Fickds
Figure 7: Representation Visualizations. To analyze our representation, we visualize the representation across different object categories. Mask fields color objects based on their instance masks, showing effective distinguishing among different instances. Descriptor fields color 3D points through PCA. The consistent color patterns across instances show that representation could encode consistent semantic features over different instances. We also demonstrate representation robustness to clustered scenes in the right two columns.
4.4 Fields Analysis
To obtain a more thorough understanding of D*Fields, we first extract the mesh using the march- ing cube algorithm. We evaluate vertices from the extracted mesh using our D'Fields and obtain the associated segmentation information and semantic features, as visualized in Figure 7. Mask fields in Figure 7 show a distinct 3D instance segmentation in different scenarios, even clustering scenes like cans and toothpaste. The 3D instance segmentation enables the downstream planning module to distinguish and manipulate multiple instances, as shown in the mug organization tasks.
Additionally, visualize the semantic features by mapping them to RGB space using PCA. We observe that our semantic fields show consistent color patterns across differrnnt instances. In the provided shoe example, even though various shoes have distinct appearances and poses, they exhibit similar color patterns: shoe heels are represented in green, and shoe toes red. We observed similar patterns for other object categories such as mugs and forks. The consistent semantic features across various instances help our manipulation framework to achieve category-level generalization. When encountering novel instances, our D F'Fies can still establish the correspondence between the workspace and the goal image using semantic features. In addition, analyze in the supplementary to show that Fieils have better 3D consistency compared simple point cloud stitching.
5 Conclusion
In this work, we introduce D'Fields, which implicitly encode 3D semantic features and 3D instance masks, and model the underlying dynamics. Our emphasis is on zero-shot generalizable rearrange- ment tasks specified by 2D goal images varying styles, contexts, and instances. Our framework excels executing a diverse array of robotic rearrangement tasks in both simulated and real-world scenarios. Its performance greatly surpasses baseline methods such FeatureNeRF and F3RM in terms of efficiency and effectiveness.
Limitation. Our method lifts 2D visual foundation models to the 3D world, enabling a range of zero-shot rearrangement manipulation tasks. However, we both benefit from and are limited by their capabilities. For example, the semantic feature field is not fine-grained enough to distinguish be- tween the right and left sides of shoes. Visual foundation models with more fine-grained semantic features are needed. In addition, rearrangement tasks could fail when they need follow spe- cific manipulation order to avoid collision with other objects, e.g. crowded scene. In the future, incorporating task planner handle clustered scenes could be valuable direction.

---

Acknowledgments
This work is partially supported by Sony Group Corporation, the Stanford Institute for Human- Centered AI (HAD), and Google. The opinions and conclusions expressed in this article are solely those of the authors and do not reflect those of the sponsoring agencies.
References
[1] L.. Manuelli, Y. Li, P. Florence, and R. Tedrake. Keypoints into the future: Self-supervised correspondence in model-based reinforcement learning. In Conference Robot Learning (CoRL), 2020.
[2] Y. Li. S. Li, V. Sitzmann, Agrawal, and A. Torralba. 3d neural scene representations for visuomotor control. arXiv preprint arXiv:2107.04004 2021.
[3] H. Shi, H. Xu, Z. Huang, Y. Li, and Wu. Robocraft: Learning to see, simulate, and shape clasto-plastic objects with graph networks. arXiv preprint urXiv:2205.02900 2022.
[4] Y. Ze, N. Hansen, Y. Chen, M. Jain, and X. Wang. Visual reinforcement learning with self- supervised 3d representations. IEEE Robotics and Automation Letters (RA-L), 2023.
[5] D. Hafner, T. Lillicrap, I. Fischer, R. Villegas, D. Ha, H. Lee, and J. Davidson. Learning latent dynamics for planning from pixels. In International conference on machine learning, pages 2555 2565. PMLR, 2019.
[6] W. Yan, A. Vangipuram, P. Abbeel, and L. Pinto. Learning predictive representations for de- formable objects using contrastive estimation. In J. Kober, F. Ramos, and C. Tomlin, editors, Proceedings of the 2020 Conference on Robot Learning, volume 155 of Proceedings of Ma- chine Learning Research, pages 564-574. PMLR, 16-18 Nov 2021.
[7] Y. Wang, Y. Li, K. Driggs-Campbell, L. Fei-Fei, and J. Wu. Synamic-Resolutiin Model Learning for Object Pile Manipulation. In Proceedings of Robotics: Science and Systems, Daegu, Republic of Korea, July 2023. doi: 1...15666...223...0
[8] M. Minderer, C. Sun, R. Villegas, F. Cole, K. P. Murphy, and H. Lee. Unsupervised learning of object structure and dynamics from videos. Advances in Neural Information Processing Systems, 32, 2019.
[9] Tremblay, T. To, B. Sundaralingam, Y. Xiang, D. Fox, and S. Birchfield. Deep object pose estimation for semantic robotic grasping of household objects. In Conference on Robot Learn- ing, pages 306-316 316. PMLR, 2018.
[10] S. Tyree, J. Tremblay, T. To, J. Cheng, T. Mosier, J. Smith, and S. Birchfield. 6-dof pose esti- mation of household objects for robotic manipulation: An accessible dataset and benchmark. In 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 13081 10081-1308. IEEE, 2022.
[11] K. M. Jatavallabhula, A. Kuwajerwala, Q. Gu, M. Omama, G. Iyer, S. Saryazdi, T. Chen, A. Maalouf, S. Li, N. V. Keetha, A. Tewari, J. Tenenbaum, C. de Melo, M. Krishna, L. Paull, F. Shkurti, and A. Torralba. ConceptFusion: Open-set multimodal 3D mapping. In I Proceedings of Robotics: Science and Systems, Daegu, Republic of Korea, July 2023, doi: 10.11600//555 2023.XIX.066.
[12] K. Mazur, E. Sucar, and A. J. Davison. Feature-realistic neural fusion for real-time, open set scene understanding. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 8201-8207. IEEE, 2023.
[13] W. Liu, Y. Du, T. Hermans, S. Chernova, and C. Paxton. Structdiffusion: Language-guide creation of physically-valit structures using unseen objects. In RSS 2023, 2023.

---

[14] S. Liu, Z. Zeng, T. Ren, F. Li, H. Zhang, J. Yang, J. Yang, H. Su, J. Zhu, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.05499 2023.
[15] A. Kirillov, E. Mintun, N. Ravi, Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y. Lo, Dollár, and R. Girshick. Segment anything. arXiv:2304.02643, 2023.
[16] H. Cheng and A. G. Schwing. XMem: Long-term video object segmentation with an atkinson-shiffrin memory model. ECCV, 2022.
[17] M. Oquab, T. Darcet, Moutakanni, H. Vo, M. Szafraniec, V. Khalidov, P. Fernandez, D. Haziza, F. Massa, A. El-Nouby, R. Howes, P.-Y. Huang, H. Xu, V. Sharma, S.-W. Li, W. Galuba, M. Rabbat, M. Assran, N. Ballas, G. Synnaeve, I. Misra, H. Jegou, J. Mairal, P. Labatut, A. Joulin, and P. Bojanowski. Dinov2: Learning robust visual features without supervision, 2023.
[18] J. Ye, N. Wang, and X. Wang. Featurenerf: Learning generalizable nerfs by distilling founda- tion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8962-8973, 2023.
[19] W. Shen, G. Yang. A. Yu, J. Wong, Kaelbling, and Isola. Distilled feature fields enable few-shot manipulation. 7th Annual Conference on Robot Learning, 2023.
[20] A. Brohan, Y. Chebotar, C. Finn, Hausman, A. Herzog, D. Ho, J. Ibarz, A. Irpan, Jang, R. Julian, al. Do can, not say: Grounding language in robotic affordances. In Conference Robot Learning, pages 287-318. PMLR, 2023.
[21] W. Huang, F. Xia, T. Xiao, H. Chan, J. Liang, P. Florence, A. Zeng. J. Tompson, I. Mordatch, Y. Chebotar, et al. Inner monologue: Embodied reasoning through planning with language models. In Conference on Robot Learning, pages 1769-1783 PMLR, 2023.
[22] J. Liang, W. Huang, F. Xia, P. Xu, K. Hausman, B. Ichter, P. Florence, and A. Zeng. Code as policies: Language model programs for embodied control. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 9493-9500. IEEE, 2023.
[23] W. Huang, Wang, Zhang, Y. Li, Wu, and L. Fei-Fei. Voxposer: Composable 3d value maps for robotic manipulation with language models. In 7th Annual Conference on Robot Learning, 2023.
[24] R. Florence, L. Manuelli, and R. Tedrake. Dense object nets: Learning dense visual ob- ject descriptors by and for robotic manipulation. In A. Billard, A. Dragan, J. Peters, and J. Morimoto, editors, Proceedings of The 2nd Conference on Robot Learning, volume 87 of Proceedings of Machine Learning Research, pages 373-385. PMLR, 29-31 Oct 2018.
[25] Y. Zhu, Z. Jiang, P. Stone, and Y. Zhu. Learning generalizable manipulation policies with object-centric 3d representations. In 7th Annual Conference on Robot Learning, 2023.
[26] T. Gervet, Z. Xian, N. Gkanatsios, and K. Fragkiadaki. Act3d: 3d feature field transformers for multi-task robotic manipulation. In Conference on Robot Learning, pages 3949-3965. PMLR, 2023.
[27] 7. Xian, Gkanatsios, Gervet, T.-W. Ke, and K. Fragkiadaki. Chaineddiffuser: Unifying trajectory diffusion and keypose prediction for robotic manipulation. In 7th Annual Conference on Robot Learning. 2023.
[28] T.-W. Ke, N. Gkanatsios, and K. Fragkiadaki. 3d diffuser actor: Policy diffusion with 3d scene representations. Arxiv, 2024.
10

---

[29] L. Zhu, A. Mousavian, Y. Xiang, H. Mazhar, J. van Eenbergen, S. Debnath, and D. Fox. Rgb- d local implicit function for depth completion of transparent objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4649-46666 110994468,,40 2021.
[30] J. Ichnowski, Y. Avigal, J. Kerr, and K. Goldberg, Dex-nerf: Using neural radiance field to grasp transparent objects. In 5th Annual Conference on Robot Learning. 2021.
[31] A. Simeonov, Y. Du, A. Tagliasacchi, J. B. Tenenbaum, Rodriguez, P. Agrawal, and V. Sitz- mann. Neural descriptor fields: Se (3)-equivariant object representations for manipulation. In 2022 International Conference Robotics and Automation (ICRA), pages 6394-6400. IEEE, 2022.
[32] D. Driess, J.-S. Ha, M. Toussaint, and R. Tedrake. Learning models as functionals of signed- distance fields for manipulation planning. In Conference on Robot Learning, pages 245-255. PMLR, 2022.
[33] Z. Jiang, Y. Zhu, M. Svetlik, K. Fang, and Y. Zhu. Synergies Between Affordance and Geome- try: 6-DoF Grasp Detection via Implicit Representations. In Proceedings of Robotics: Science and Systems, Virtual, July 2021. 10..00..56//SS.200000 XXVI.024.
[34] T. Weng, D. Held, F. Meier, and M. Mukadam. Neural grasp distance fields for robot manipu- lation. In IEEE International Conference on Robotics and Automation (ICRA), 2023.
[35] D. Driess, I. Schubert, P. Florence, Y. Li, and M. Toussaint. Reinforcement learning with neural radiance fields. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
[36] D. Shim, S. Lee, and H. J. Kim. SNeRL: Semantic-aware neural radiance fields for reinforce- ment learning. In A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett, editors, Proceedings of the 40th International Conference Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages
PMLR, 23 Jul 2023.
[37] L. Yen-Chen, P. Florence, Supervision: Learning dense object descriptors from neural radiance fields. In JEEE Con- ference on Robotics and Automation (ICRA), 2022.
Barron, T.-Y. Lin, A. Rodriguez, and Isola. NeRF-
[38] Z. Tang, B. Sundaralingam, J. Tremblay, B. Wen, Y. Yuan, S. Tyree, C. Loop, A. Schwing, and S. Birchfield. RGB-only reconstruction of tabletop scenes for collision-free manipulator control. In ICRA, 2023.
[39] A. Zhou, M. Kim, L. Wang, P. Florence, and C. Finn. Nerf the palm of your hand: Cor- rective augmentation for robotics via novel-view synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 17907-17917, 2023.
[40] N. M. (Mahi)Shafullla,, C. Paxton, L. Pinto. S. Chintala, and A. Szlam. CLIP-Fields: Weakly Supervised Semantic Fields for Robotic Memory. In Proceedings of Robotics: Science and Systems, Daegu, Republic of Korea, July 2023.
[41] Y. Wi, A. Zeng, P. Florence, and N. Fazeli. Virdo++: Real-world, visuo-tactile dynamics and perception of deformable objects. arXiv preprint arXiv:2210.03770,, 2022.
[42] Y. Wi, P. Florence, A. Zeng, and N. Fazeli. Virdo: Visio-tactile implicit representations of deformable objects. In 2022 International Conference on Robotics and Automation (ICRA), pages 3583-3590. IEEE, 2022.
[43] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, Mishkin, J. Clark, Krueger, and I. Sutskever. Learning transferable visual models from natural language supervision. In M. Meila and T. Zhang, editors, Proceedings of the 38th Inter- national Conference Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages
PMLR, 18-24 Jul 2021.

---

[44] M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin. Emerging properties self-supervised vision transformers. In Proceedings of the IEEE/CVE interna- rional conference on computer vision, pages 9650-9660, 2021.
[45] J. Kerr, C. M. Kim, K. Goldberg, A. Kanazawa, and M. Tancik. Lerf: Language embedded radiance fields. In International Conference Computer Vision (ICCV), 2023.
[46] S. Sharma, A. Rashid, C. M. Kim, J. Kerr, L. Chen, A. Kanazawa, and K. Goldberg. Lan- guage embedded radiance fields for zero-shot task-oriented grasping. In 7th Annual Conference on Robot Learning. 2023.
[47] Y. Ze, G. Yan, Y.-H. Wu, A. Macaluso, Ge, J. Ye, Hansen, L. E. Li, and Wang. Multi- task real robot learning with generalizable neural feature fields. In 7th Annual Conference on Robot Learning, 2023.
[48] Y. Li, T. Lin, K. Yi, D. Bear, D. L. Yamins, J. Wu, J. B. Tenenbaum, and A. Torralba. Visual grounding of learned physical models. In International Conference on Machine Learning. 2020.
[49] Y. Li. J. Wu, R. Tedrake, J. B. Tenenbaum, and A. Torralba. Learning particle dynamics for manipulating rigid bodies, deformable objects, and fluids. In ICLR, 2019.
[50] G. Williams, N. Wagener, B. Goldfain, P. Drews, J. M. Rehg, B. Boots, and E. A. Theodorou. Information theoretic mpe for model- reinforcement learning. In 2017 IEEE international conference on robotics and automation (ICRA), pages 1714-1721. IEEE, 2017.
[51] C. Li, Zhang, J. Wong, C. Gokmen, S, Srivastava, R. Martín-Martin, C. Wang, G. Levine, M. Lingelbach, J. Sun, M. Anvari, M. Hwang, M. Sharma, A. Aydin, D. Bansal, S. Hunter, J. H. Tang, C. Tang, F. Xia, S. Savarese, H. Gweon, K. Liu, J. Wu, and L. Fei-Fei. BEHAVIOR- 1k: A benchmark for embodied Al with 1,000 everyday activities and realistic simulation. In 6th Annual Conference on Robot Learning. 2022.
K.-Y. Kim, A. Lou, C. R. Matthews, I.
12

---

Fieillss: Dynamic 3D Descriptor Fields for Zero-Shot Generalizable Rearrangement
Yixuan Wang"", Mingtong Zhang²* Zhuoran Li Tarik Kelestemur Katherine Driggs-Campbell' Jiajun Wu Li Fei-Fei", Yunzhu Li'
CColumbia University, UUniversity of Illinois, Urbana- Chanpannn,, Naational University of Singapore, Boosonn Dynamies Al Institute, Stanford University
Contents
1
Detailed Comparisons with Prior Works
1.1
Comparisons with Prior 3D Representations using Visual Foundational Models
1.2
Comparisons with Dense Object Nets
2
Method Details
2.1
Notation: Camera Transformation and Projection
2.2
Fusion Equation Explanation
2.3
Correspondence Equation Explanation
2.4
Grounded-SAM Masks Association
2.5
Mask Tracking using XMem
2.6
Dynamics Training Details
2.7
Keypoints Tracking Initialization
2.8
Model-Predictive Control (MPC) Details
2.9
Discussion of Backbone Choice
3
Additional Experiments
3.1
Implementation Details
3.2
Keypoint Tracking Results
3.3
Mesh Comparisons with FeatureNeRF and F3RM
3.4
Quantitative Correspondence Comparisons with FeatureNeRF and F3RM
3.5
Comparisons with F3RM under Dense Views
3.6
Ablation Study: Qualitative Correspondence Comparisons
3.7
Ablation Study: Quantitative Correspondence Comparisons
10
3.8
Abaltion Study: Quantitative Manipulation Results
10
3.9
F3RM with Grounded-SAM
13
3.10
Debris Experiment Details
14
8th Conference Robot Learning (CoRL. 2024), Munich, Germany.

---

Detailed Comparisons with Prior Works
Comparisons with Prior 3D Representations using Visual Foundational Models
We extend the application of state-of-the-art visual foundational models robotic manipulation tasks, offering the following improvements over prior methods:
Efficiency: Several existing works construct 3D representations from 2D models, but they are often time-consuming due to the distillation process. This limits their applicability in tasks requiring frequent visual feedback. For instance, prior works such F3RM [1] and LERF [2] require 1.3 to 8 minutes for training. In contrast, our method takes only 0.166 seconds, making it 500 to 3,000 times faster.
Sparse Views: Some existing works, such as F3RM and LERF, require dense camera views (more than 50) build a high-quality 3D representation, which is not suitable for typical robotic manipulation workspaces with sparse camera views (1-4). In contrast, we have developed more efficient and effective pipeline that uses only four views.
Planning Using Reconstructed Fields: We explicitly use the reconstructed fields to find matches that define planning objective, allowing us employ model-basee planning to solve rearrangement tasks. This contrasts with existing methods like F3RM, which require expert demonstrations for every new task.
Additionally, our work distinguishes itself from FeatureNeRF by applying our representation specif- ically to robotic manipulation tasks, whereas FeatureNeRF does not focus manipulation [3].
1.2
Comparisons with Dense Object Nets
Our approach differs from Dense Object Nets (DON) in two major ways [4]:
Training Requirements: DON is trained via contrastive learning, which requires fore- ground and background separation and careful selection of positive and negative pairs. new object categories
This means that deploying DON in quires additional training efforts. In contrast, our pipeline does not need extra training and can be applied to a diverse set of object categories by leveraging visual foundational models.
environment
Generalization: DON is trained on small-scale datasets and may not generalize well to new object categories and scenarios. Our pipeline, however, offers stronger generalization capabilities without the need for retraining and additional data collection.
2 Method Details
2.1 Notation: Camera Transformation and Projection
We assume that there are multiple RGBD cameras with fixed viewpoints. We assume all cameras' intrinsic parameters K and extrinsic parameters T are known. The ith camera's 's extrinsic parameters are defined as follows:
R
t 1
T
0
E
SE((),
(I)
where Euclidean group (3) = [R. t
t E Rd For a 3D point x in the world frame, could obtain the pixel u, projected in ith camera and distance ith camera r, follows:
R E
303
u, Proj (K; (Rx+tt))
Ti = [0, 0, 11fh (R,x + tz),
where Proj performs perspective projection, mapping 3D vector p [r. 2)T to 2D vector q 2//².

---

Algorithm Fusion Process
1: procedure FUSION(x)
Di Input 3D point
2: U; Project(x, i), ra [u[us]
3D projection and depth reading 3: di < r. -r. 4- Truncate(di, (4) Truncated depth difference using Eq. 3 in main paper
4:
14, W;::; Weights(d,)
Assign weights each view using Eq. 4 in main paper
5:
f; P; < Interpolate(W WP. uu)
Interpolate features using Eq. 5 in main paper
6:
f.p + Fuse(f,, P&, U; Ws)
Fuse features using Eq. 6 in main paper
2.2 Fusion Equation Explanation
Here is a detailed explanation for Equation 6.
Signed Distance d: Assuming Eσ; o 0 and ignoring 8, the equation simplifies to d
In this case, the weights sum 1.
Ee
Semantic Feature f and Instance Masks p: Similarly, with E 0 and ignoring 6, E.wsp The weight w; varies based on proximity to the
Zn and p = - have f = Σ surface: wt 1 when x is close, summing weights to 1, and w; approaches 0 when x is far, causing and to converge to 0.
2.3 Correspondence Equation Explanation
Equation describes the process of finding the correspondence from jth sampled point's associated feature f.9 IR 2D point Sgoal,j in the image space. The computation process consists of three steps:
Osy is the feature-space distance between jth sampled point and ith goal image pixel. We first extract goal image's 's feature map Wgoal and read out vth pixel's corresponding feature Wgaa[ Then we compute the L2 distance between Wgoal uu] and fy and assign to aij-
After applying the softmax to ozy over the whole image, we obtain 3g. The summation of Bi over the image space, i.e. X Bij 1.
Then apply the weighted sum of pixels obtain the corresponding pixel on the goal image Sgual.j.
2.4 Grounded-SAM Masks Association
Using Grounded-SAM, [5, 6), we could extract instance segmentation masks from each view. How- ever, masks from different views can contain different number of instances, and the instance IDs may not be consistent. To tackle this problem, need post-process the instance segmentation results. The high-level idea is to merge instances from different viewpoints based their geometric distance.
We will save all merged instances to list. Specifically, we will first start from the first viewpoint. For each instance mask, we map them to 3D point clouds and save them into a list. Then, we move the next viewpoint and map all instance masks 3D point clouds. We will compare each instance with the merged instances in the list. If they have significant overlap, measured by the Intersection of Union (IoU) of the two point clouds, we will merge the instance from the new viewpoint with the merged instances in the list. This process will continue until all viewpoints have been iterated through.
After merging all instances, we will filter instances that are not stably detected. Specifically, instances that meet one of the following criteria will be filtered out:
The instance has little point cloud.
The instance is known to be background, such as the table.

---

The instance overlaps with other instances, while other instances have a higher confidence.
After filtering, we will assign consistent instance IDs to the instance masks in each viewpoint.
2.5 Mask Tracking using X.Mmm
After using Grounded-SAn SAM to obtain initial masks, we use XMem to track instance masks later frames so that we can skip time-consuming segmentation detection for subsequent frames. By apply- ing XMem, we significantly reduce the computational cost and processing time, making the pipeline more efficient.
2.6 Dynamics Training Details
We instantiate the dynamics model -) as graph neural networks (GNNs) that predict the evo- lution of particles s E R2x*. under external actions dg. We also construct edges e' E N2x according to particle distance, where ej (uf, uf) represents an edge connecting from particle uf particle o/. f(;,-) consists of node and edge encoders fo (,, SE fene (.,,) to obtain node and edge representations:
p fo (8) ac);
i i= 1, 1,..., Rsa
(3)
of = fe (8 saz)
1....,
Then, we use node and edge decoders 1 (.,)), fdee (: )) to predict the time step's particle states:
r = fdee (df),,
j j= L..... Re:
8. foc (p' Σnt). i= i
(4)
jEM.
where N₁ is the index set the edges that connect to particle i. In practice, we follow Li et al. [7] and use multi-step message passing over the graph to approximate the propagation of action impacts.
2.7 Keypoints Tracking Initialization
To initialize keypoint tracking, we first densely sample points, which can either come from grid sampling instances' 3D point clouds. Then, evaluate these points using D'Fields and mask out those not belonging the desired instance. Finally, downsample these points to the desired number using farthest point sampling.
2.8 Model-Predictivg Control (MPC) Details
As described in Section 3.4 of the main paper, our MPC framework needs a reference camera to bridge the gap between 3D representation and 2D representation. In our work, the reference cam- era's extrinsic parameters are manually defined according to the tasks. Typically, it looks down the workspace from above. Its intrinsic parameters the same the real camera's 's intrinsic parameters. The detailed MPC algorithm we used is described in Algorithm 2.
For the pick-and-pla tasks, our dynamics model is simplified, as the object is rigidly attached to the end-effector.
2.9 Discussion of Backbone Choice
We choose DINOv2 as our feature backbone because there are several great and unique properties of DINOv2, including:
Generalization: DINOv2 has demonstrated consistent feature extraction across diverse ob- ject categories and scenes, which allows us to apply our pipeline to various object categories and bridge the gap between goal image and workspace.

---

Algorithm Trajectory optimization at each MPC step
Input: Current state Sg, goal Syoul time horizon T, gradient descent iteration N the perception module h, and the dynamics module f
Output: Actions 00:T-1
Sample current action sequence 06:T-1 for i 1,.... N do
Sample M action sequences 0:7-1 current action sequence
for m = ,,..., M do
for t ....,
do
Predict the next step S + f(s;, Q(") Calculate the task loss ct + c(ST,39)
Calculate the current action sequence Tb.T-1 using the task loss cl:M
Return 0
Refined Correspondence: DINOv2 demonstrates the capability to establish refined corre- spondence, even when two images have quite different backgrounds and contexts. This is important for our tasks since we need refined correspondence between the goal image and the workspace to define the objective function for the manipulation.
However, our framework not specific to DINOv2. If there a better visual foundational model in the future, we could replace it easily in a plug-and-pl and-play manner.
Additional Experiments
3.1 Implementation Details
For the truncation threshold we set it to 0.02 across all experiments. For the prompts used for Grounde--SAM [5, 6] in Figure 7 in the main paper. from left to right, they are "shoe", "mug", "spoon", "can", and "toothpaste" respectively. The confidence threshold used for Grounded-SAM 0.2.
We also compare with several baselines, with details listed below:
Dense Object Nets (DON) [4]: We compare the effects of using different feature backbones, with DON as one of the baseline backbones. We use the pre-trained DON model since our method also uses off-the-shelf models with no re- re-training or additional data. Specifically, we use the model trained on the shoe class, which can be found here.
DINO [8]: Another feature backbone we baseline on is DINO, which is the precursor to DINOv2. We use the code provided by [9] to extract dense DINO features.
RGBD+DINOO2: We also compare our method to simply merging point clouds with DI- NOv2 features from multiple viewpoints [10].
FeatureNeRF [3]: We compare representation with other state-of-the-art 3D implicit semantic representations, including FeatureNeRF. We trained the model on the car example dataset provided by the authors. For comparison with our model, we do not distill the model from the DINO model but from DINOv2. It is worth noting that FeatureNeRF only uses one RGB image to generate the neural fields. We found that providing more views to the FeatureNeRF model during inference time leads worse performance. Therefore, we keep its input single-view RGB image.
Distilled Feature Fields (F3RM) [1]: Another 3D implicit semantic representation we com- pare is F3RM. We use the same training code provided by the authors, except that distill from DINOv2 models make it comparable with model. In addition. four camera views F3RM inputs instead of dense views as the original paper.
5

---

2D Image
Reconstructed Mesh
2D Image Sequence
Reconstructed Mesh
(a) FeatureNeRR Original Results
(b) F3RM Original Results
2D
Ours
FesstrrNNRR
F3RM
Comparisons with FeatureNeRF F3RM
Figure 1: Mesh Reconstruction Comparison. shows the reconstructed mesh of the FeatureNeRI, given a 2D image from the training distribution. This reflects that our mesh extraction process works well when input image is within the training distribution. Given a sequence of 2D images densely scanning the workspace, (b) also shows good reconstruction quality of the seene. However, when given sparse views containing novel instances, both FeatureNeRF and F3RM fail to generate accurate meshes for the scene, which demonstrates the effectiveness of our method.
For the dynamics training of the shoe- sroe-pushing example, we collected 20 episodes of pushing one shoe. Then, trained dynamics model that take in current particles and pushing action and predict particles in the next step.
For the evaluation in the real world, we summarize the details of our tasks in Table 1.
Environment
Task Name
Objects
Organize Shoes
Shoe
Collect Debris
Almonds
Organize Office Table
Mouse, Pen, Mug
Real World
Organize Utensils
Knife, Spoon, Fork. Bowl
Organize Fruits
Apple. Banana
Push Shoes
Shoe
Serve Food
Cupcake, Bread, Tomato, Lemon, Banana Mug
Organize Mugs
Simulation
Organize Shoes
Shoe
Organize Utensils
Knife, Spoon, Fork
Table I: Task Details Summary. This table summarizes our task environment, specific tasks, and objects. We evaluate our framework on eight tasks and fifteen object categories, where each object category covers several object instances with diverse appearances and shapes.

---

3.2 Keypoint Tracking Results
We show two examples of 3D keypoint tracking in Figure 2. In the first scenario, we track shoe as it is pushed and subsequently flipped. The second example demonstrates tracking shoe that is lifted and then placed down. Our system robustly tracks the shoe 3D space. These examples underscore the effectiveness of D*Fields in maintaining accurate tracking in dynamic scenarios, which enables dynamiis learning capabilities.
3.3 Mesh Comparisons with FeatureNeRR and F3RM
We qualitatively compare the descriptor fields generated by the three methods. We extract the mesh from these fields using marching cubes, as shown Figure 1. We observe that our D'Fields could generate accurate color meshes given sparse views. FeatureNeRF could re- construct reasonable mesh given single 2D image from the training distribution, shown in Figure (a). However, when it encounters a new object outside the training distribution, the reconstrueted mesh will be completely off, even when we apply the image preprocessing align the testing images with the training set in terms of image sizes, data range, and back- ground color. Although Figure (b) shows that we can reconstruct a clear mesh with dense im- age sequences as in the original paper, its color mesh is quite inaccurate given sparse view- points in our experiment setting, except for the shoe case.
Push and Rotate
Lifting
Keypoint Tracking. We apply D'eies tracking tasks showcase two tracking examples, both which involve 3D motions and partial observa- tions from single viewpoints. This shows that our rep- resentation is 3D, dynamic, and semantic.
Figure 2:
3.4 Quantitative Correspondence Comparisons with FeatureNeRF and F3RM
0.3
Pregision
0.6
0.6
ReatureNoRFF [7]
DUN [5]
0.2
0.2
0.2
DINOV2 [6]
0.8
1.0
Recall
(a) Drill
(b) Hammer
(c) Mug
(d) Shoc
Figure 3: Precision-Reddd Recall of Various Thresholds for Different Instances. The curves show how D'Fields
compares with baseline methods in terms matching quality, tested on 4 different instances: mug, bag, pan, shoc. We use the precision-reaet recall curve to measure the correspondence quality. Our method shows to con- sistently exceeds the performance of the baseline approaches, which demonstrates our method's capability to encode semantic information accurately and establish precise correspondences using the semantic information.
In addition, we also measure the quantitative correspondence accuracy of our method, FeatureNeRF, and F3RM. We manually label the ground truth correspondence keypoints on the source image and the target descriptor fields. We measure the correspondence quality using the precision-recal curve, as shown in Figure regarding the precision-rest recall curve are provided later. We could see that F3RM and FeatureNeRF collapse to the origin point except for the shoe example for F3RM. This because these two methods fail to reconstruct meshes given sparse observations and unseen instances. In contrast, method shows a much better correspondence quality.
larger area under the curve indicates better correspondence quality. Details
7

---

To generate the precision- curve, we manually label one point X on the 2D source image, and a set of corresponding 3D points Xtg-- For 2D points, we obtain the associated semantic feature fac- For vertices on the reconstructed mesh, we can obtain a set semantic features {fig,o.... fig.N }- Then we compute the cosine similarity between features fae and {fg.0: ..., fgg. J. For one similarity threshold т, we filter out set of points F, with similarity scores higher than т. Additionally, identify the set of points G that are close Xig-- We then define precision P. and R, follows:
F,NG F-l
F,
P.
R,
(5)
G
By varying 7, we can plot the precision-recall curve as shown in Figure 3.
3.5
Comparisons with F3RM under Dense Views
We compare our approach using 4 views against F3RM [1], which uses 50 views (with and without Grounding DINO + SAM supervision), on the Dense Object Nets dataset [4].
Our qualitative results shown in Figure 4. Similar to the main paper, We compare the recon- structed mesh, descriptor fields, and mask fields for our approach (4 views), F3RM without Ground- ing DINO SAM (50 views), and F3RM with Grounding DINO SAM (50 views). We observed that F3RM with 50 views reconstructs high-quality meshes, confirming our F3RM implementation is bug-free. Additionally, there is no significant qualitative difference between our method with views and F3RM with 50 views, demonstrating the effectiveness of our representation. Lastly, we did not observe notable differences between F3RM with and without Grounding DINO SAM, which consistent with our previous conclusion that this additional supervision does not signifi- cantly contribute F3RM training.
Similar to our paper, we also evaluated quantitative correspondence quality using the Precision- Recall (P-R) curve, as shown in Figure correspondence quality. Our results show no significant correspondence differences between our approach with views and F3RM with 50 views, indicating that our representation is effective even with sparse views.
larger area under the P-R curve indicates better
3.6 Ablation Study: Qualitative Correspondence Comparisons
In this section, we first study how different feature backbones could affect the correspondence quality. Then we show the qualitative correspondence results our method. As mentioned in Section 3.1, substitute our method's backbone with other pre-trained models, like DON and DINO [4, 8]. We also compare with RGBD+DINOv2 demonstrate its effectiveness.
Figure shows the qualitative correspondence results of all ablation baselines. There are three key observations we can make from this figure.
Compared with RGBD+DINOv2, method's correspondence quality is better. We achieve more accurate correspondence since representation amortize noise from single views by considering 3D consistency. Although RGBD+DINOv2 can achieve some spatial consistency, there are still variances in results from different viewpoints, while our method guarantees spatial consistency.
Compared with DINO, correspondence is fine-grained and accurate. Thanks to the advancements in foundational visual models, DINOv2 encodes more fine-grained features and enables correspondence with higher accuracy.
DON struggles to generalize to novel scenes and unseen object categories. Although the original DON shows good correspondence quality, is trained on one type of object with relatively small dataset. Compared with visual foundational models, it shows limited generalization capabilities in terms of scenes and object categories.

---

Soone
Recon Mesh
Ours (4 Views) Desttipor Fielos
Mack Fielés
F3RM (50) Views) Recon. Mesh
Deseriptor Fields
Rocon. Mesh
F3RM with Grounedd MM (50 Views)
Deterigtor Fields
Mask Pields
Figure 4: Comparisons with F3RM under Dense Views. We compared the reconstructed mesh, feature
fields, and mask fields with F3RM using dense views. Our results showed that, despite using sparse inputs, method did not exhibit significant qualitative performance degradation compared to F3RM under dense views, demonstrating the efficiency of our approach.

---

1.0
1.0
1.0
1.0
0.8
0.3
0.3
Prociston
0.6
0.6
0.6
0.4
0.4
0.2
0.2
F3RM
F3RM w/ G-SAM
0.0
0.
0.0
0.2
0.1
0.6
0.5
1.0
0.0
0.2
0.1
0.6
0.3
1.0
0.0
0.2
0.1
0.6
0.3
1.0 0.0
0.2
0.1
0.6
08
1.0
Recall
(a) Drill
(b) Hat
(c) Mug
(d) Shoe
Figure 5: Quantitative Comparisons with F3RM under Dense Views. We use the P-R curve to measure the correspondence accuracy. Our results show significant differences in correspondence between approach with views F3RM with views, indicating that our representation remains effective even with sparse
views.
We also visualize the correspondence from 2D images to our workspace as shown in Figure 7. Specifically, we extract the DINOv2 feature of the selected pixel in the 2D image. Then we high- light the part of the 3D mesh with features close the query feature. There two observations regarding the qualitative correspondence results. First, the semantically similar parts are correctly matched across different instances and contexts. For example, when we select the rim of the plate in the 2D image, the corresponding part in the 3D mesh is highlighted. This matching is consistent across different object parts, such as the head and tail of the shoe, the handle and blade of the knife, and the tip and bar of the drill. Second, the correspondence is multimodal when there are multiple semantically similar object parts the workspace. For example, when we select the spoon handle in the 2D image, multiple utensil handles in the workspace highlighted. The correspondence qualitative results show that our Fields could establish meaningful correspondences across differ- ent instances and contexts, so that we can rely correspondence to define the planning objective function.
3.7 Ablation Study: Quantitative Correspondence Comparisons
Similar to the main paper, we generate the precision-recal curve to quantitatively compare the cor- respondence quality with ablation baselines. We make the following observations regarding the baseline correspondence results.
Compared with RGBD+DINOv2, our method shows more accurate correspondence re- sults. This is because our DDFields can average out noise from each viewpoint, while RGBD+DINOv2 accumulates noises.
DINO faces challenges in accurately distinguishing specific object components. This limi- tation results in less precise correspondence, shown in Figure 3.
Although DON encode semantic features in environments and instances, it fails to generalize to novel environments and object categories. Therefore, its correspondence results are even worse than DINO, as shown in Figure 3.
3.8 Abaltion Study: Quantitative Manipulation Results
In Figure 8 (a), we measure performance using the IoU between the mask of the goal image and the mask the final state post-manipulation. Higher IoU values indicate greater degree of alignment between the intended and achieved configurations. Our method demonstrates superior performance across five distinct object categories, consistently outshining the baseline methods. For each cate- gory, we performed 5 experiments for the evaluation results. This not only highlights its exceptional manipulation accuracy but also its robust generalization capabilities. While the DINO model ex- hibits some struggles, particularly in distinguishing specific object components and consequently yielding less precise results, it still performs better than DON. Although DON shows commendable results with familiar objects and configurations, its performance dips in novel scenarios, revealing lack of generalization. These results collectively emphasize the significant advantages of our method in diverse and accurate object manipulation.
10

---

Source Image
MWW
Target Imngee
Ours
Carr. Mesh
RGBD-DINOR DINOV2 Target Image
Inaccurate
Inaccurate
Cor:. Mesh
Tanget Image
DINO
Corr. Mesh
Tirger Image
DON
Corr Mesh
Figure 6: Correspondence Quantitative Comparison. The top row shows the selected pixel from the source image, and the following have accurate correspondence, RGBD+DINOv2 corresponds some points in the background. For example, the drill tip is not accurately highlighted in the RGBD+DINOv2 example, while ours can accurately highlight the drill tip. Ours with DINO feature backbones fail to identify objects accurately, while ours with DON fail to generalize to novel seenes and novel instances.
show the corresponding areas for different methods. While method could
11

---

Souree Image
Target Image
Correspondence Mosh
Source Image
Target Imaage
Conresoondane Mesh
Figure 7: Pross-Domain Correspondence. The red triangles represent query points in the source image, and the corresponding areas are highlighted
mesh. First, we observe that our representation can encode features for object parts and establish the correspondence, such as spoon tips and spoon handles. In addition, we found the correspondence can be multimodal. When the shoe head is selected, multiple shoe heads in the workspace highlighted. At last, the correspondence is generalizable across different contexts, instances, and domains, which demonstrates our method's generalization capabilities.
1.0
0.7
D-Fiekss (Ours
Froctio of Foints with Connect Maxches
Intencctoon over over Union (IoU)
0.6
DON (Florence al
a.s-
9,6
0.3
822
D'Fields (Ouns
0.2
etal.)
DON (Pirecnze ctall)
0.0
100
150
200
Distanee Threshold (piscl)
(a) Goal Achieving Performance (higher hetter)
(b) Correspondenes Resalts
Figure 8: Quantitative Evaluation. perform real-world quantitative evaluations by measuring final goal- achieving performance and keypoints correspondence accuracy. (a) We use IoU to measure goal-achieving performance. Results indicate that our method aligns with the goal configurations much better than DON and DINO across various object categories and scenarios. (b) We mcasure the keypoints correspondence accuracy according the fraction points with accurate matches, with correct matches determined by a distance thresh- old. Our method consistently better at aligning with the goal image, regardless of the chosen threshold.
In Figure 8 (b), present the correspondence results. We label 10 corresponding keypoint pairs on both the goal image and the final manipulation result to sufficiently evaluate the correspondence curacy. The accuracy of correspondence was determined by calculating the proportion of keypoints that were accurately matched, using predefined distance threshold as the criterion. If the dis- tance between corresponding keypoints exceeds this threshold, they are determined as unmatched. Our method shows superior performance across various thresholds, consistently outperforming the baseline models. DINO emerges as the second-best in terms of performance, exhibiting broad appli- cability but with a lower precision compared our method. Meanwhile, DON lags in performance, primarily due to its struggles with generalization in novel scenarios. These results, in conjunction with those from Figure 8 (a), reiterate our method's outstanding capabilities in both generalization
12

---

and accuracy. While DINO provides reasonable applicability. it lacks the precision of our approach, and the performance of DON is hindered by its limited adaptability.
Scene
Recon Mesh
Deseripoor Fields
Ours
Maak Fields
Rocon. Mesh
Deseriptor Fields
F3RM
Mask Fields
Figure 9: Qualitative Comparison with F3RM. We compare our representation with F3RM trained with additional Grounded-SAl -AM on four seenes. We qualitatively evaluate their performance by reconstructing their mesh and visualizing descriptor fields and mask fields. Although Grounded-S SAM supervision can help F3RM to segment out objects as shown in the shoe example, it does not contribute too much reconstructing quality mesh. In contrast, representation consistently reconstruct meshes and generate quality descriptor fields mask fields across four scenes.
3.9 F3RM with Grounded-SAM
also trained F3RM [1] on Grounded-SAM [5, 6] and compare with our method, as shown in Figure For F3RM, we use the same setting as the main paper, except for the additional Grounded- SAM labeling. We reconstruct meshes and visualize the corresponding descriptor meshes and mask meshes, similar what did in the main paper. We have the following observations:
F3RM could segment out objects given additional Grounded-SAn -SM supervision, as shown in the shoe example. This indicates that our F3RM training pipeline works as expected.
Additional Grounded-SAN supervision does not influence too much results. We could see that reconstructed meshes quite similar the results from the main paper because
13

---

additional Grounded-SAN, supervision does not contribute too much to NeRF training. Instead, sparse camera views are the main reason for the poor performance F3RM.
Our representation consistently construct quality meshes and generate clear descriptor fields and mask fields, which demonstrates the effectiveness of our method.
3.10 Debris Experiment Details
In this section, we provide more details and visualization for debris experiments. Our initial multi-view observations and goal images are visualized in Figure 10. In this task, we want the robot to push spreading almonds into one object pile. We could see that initially, the debris spreads over the workspace, which is quite different from the goal states. This task is challenging since the dynamics of object piles is hard to predict. In addition, such manipulation capability needs an efficient perception module so that it can gain visual feedback from the environment. Our zero-shot rearrangement framework can collect these almonds into one pile successfully using our dynamic representation and the learned dynamics model.
RGB Observation ation
RGB Overlay With Mask
Initial Observation
Goal Image
Figure 10: Debris Experiments Details. We visualize initial multi-vien viel observations our debris experi- RGB observation with the mask. In this
ments and the corresponding goal image. The bottom row overlays the task, objective is to collect spreading debris into object pile, which is useful for tasks like cleaning.
References
[1] W. Shen, G. Yang, Yu, J. Wong, P. Kaelbling, and Isola. Distilled feature fields enable few-shot manipulation. 7th Annual Conference on Robot Learning, 2023.
[2] Kerr, M. Kim, K. Goldberg, A. Kanazawa, and M. Tancik. Lerf: Language embedded radiance fields. In International Conference on Computer Vision (ICCV), 2023.
[3] Ye, N. Wang, and Wang. Featurenerf: Learning generalizable nerfs by distilling founda- tion models. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 8962-8973, 2023.
14

---

[4] P. R. Florence, L. Manuelli, and R. Tedrake. Dense object nets: Learning dense visual ob- ject descriptors by and for robotic manipulation. In A. Billard, A. Dragan, J. Peters, and Morimoto, editors, Proceedings of The 2nd Conference on Robot Learning, volume 87 of Proceedings of Machine Learning Research, pages 373-385, PMLR, 29-31 Oct 2018.
[5] S. Liu, Z. Zeng, T. Ren, F. Li, H. Zhang, J. Yang, C. J. Yang, H. Su, J. Zhu, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.05499, 2023.
[6] A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y. Lo, P. Dollár, and R. Girshick. Segment anything. arXiv:2304.02643. 2023.
[7] Y. Li, J. J.-Y. Zhu, J. B. Tenenbaum, A. Torralba, and R. Tedrake. Propagation networks for model-based control under partial observation. In 2019 International Conference Robotics and Automation (ICRA), pages 1205-1211. IEEE, 2019.
[8] M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, Bojanowski, and A. Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF interna- tional conference on computer vision, pages 9650-9660, 2021.
(9] S. Amir, Y. Gandelsman, S. Bagon, and T. Dekel. Deep vit features as dense visual descriptors. arXiv preprint arXiv:2112.05814, 2021.
[10] M. Oquab, T. Darcet, T. Moutakanni, H. V. Vo, M. Szafraniec, V. Khalidov, P. Fernandez, D. Haziza, F. Massa, A. El-Nouby. R. Howes, P.-Y. Huang, H. Xu, V. Sharma, S.-W. Li,
W. Galuba, M. Rabbat, M. Assran, N. Ballas, G. Synnaeve, I. Misra, H. Jegou, J. Mairal, Labatut, A. Joulin, and P. Bojanowski. Dinov2: Learning robust visual features without supervision, 2023.
15


Reviews:
[{"review_id": "tN1LPHOKgN", "reviewer": "Reviewer_FgtV", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 2, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper presents a 3D representation for manipulation which has the attributes of being 3D, dynamic, and semantic. The multi-view RGBD images are taken as input to build the 3D representation. Vision foundation model features are encoded into the 3D representation. In addition, the dynamics model based on keypoints is also learned to perform trajectory optimization in manipulation. Experiments results of rearrangement tasks show the effectiveness of the proposed methods. The proposed method could also generalize to novel, AI-generated and simulated goal images.", "summary_of_recommendation": "This paper proposes a lightweight and effective representation which consists of semantic masks, features from vision foundation models, and dynamics models for manipulation tasks. However, most components are built upon existing work, which might hinder the originality of the contribution.", "technical_quality": 3, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "The author could consider justifying why learning dynamics model is necessary in this manipulation task.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ls3HAQYB8g", "reviewer": "Reviewer_vc8A", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This work proposes to create a neural feature field, called D3 Fields, that leverages several foundation models. The features are computed via heuristic combinations from 4 fixed RGBD viewpoints on a tabletop manipulation environment, thus no learning is needed to compute the feature field. Given a feature field and a goal image, dense correspondences are computed and a simple cost function is optimised via MPC to obtain manipulation actions. Experiments show mainly qualitative results of how D3 Fields obtains better correspondences compared to baselines, and how it can perform rearrangement on unseen objects.", "summary_of_recommendation": "POST-REBUTTAL UPDATE: The authors have addressed my main concern of weak baselines that provide no insight to the method. With the new comparisons against a properly trained F3RM, the paper now shows that the instantly computed D3Fields provides qualitative results that are comparable to F3RM (with the appropriate amount of images for training). For completeness, it would be great for the authors to update the manuscript with some quantitative results comparing to F3RM as well. Additionally, there are some ablations that provide more insight into design details of D3Fields. Thus, I have raised my rating to weak accept.", "technical_quality": 2, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "Questions/Comments:\n- The work seems to attack manipulation of unseen objects via generalizable feature fields. The term \"zero-shot\" is a bit confusing in this context as it appears there is no language being used anywhere.\n- Eq 6: how come the weights are not summed to 1?\n- Does the dynamics model depend on the point initialisation? How much data is used to train this? How does the dynamics model behave on seen categories vs. unseen categories?\n- During manipulation, are the initial tracked keypoints s^0 selected from a single object? If so, which object is selected first and why?\n- Line 151: n_s is used for the number of goal points, indicating it is the same number as the sampled points, but the softmax (Eq 8) is applied on all pixels. Is this a typo?\n- Where does the virtual reference camera come from? The manuscript says top-down: in this case how are the extrinsics selected? Some goal images are not top-down, including in Fig 1 bottom right (mug) and Fig 6 column 2 (Organize Fruits).\n- F3RM is trained with DINOv2. For a more comparison, it should be trained with the same foundation models used for D3 Fields (e.g. Grouding DINO + SAM).\n- The F3RM paper reports 50 images from the top hemisphere of the tabletop setup for training the NeRF. Is this the case for the baseline in the paper? I suspect there may only be 4 images used to train the NeRF, which would lead to poor reconstruction results.\n- For the debris experiment, what do the masks look like for the initial image and goal image? How does the learned dynamics function behave in this setting? This is quite interesting as granular material dynamics is difficult to predict.\n- Section 4.4 does not show anything novel, it is just marching cubes, masks, and features. I'd recommend moving this to the appendix and replacing it with quantitative results.\n- Where is X-Mem applied?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9S4lIwKrQr", "reviewer": "Reviewer_EQkG", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper proposes a feature descriptor that can be queried for arbitrary 3D points in a scene. This feature descriptor is distilled from 2D foundation models such that there is a feature correspondence between a 2D image and a 3D map. This property is evaluated on a downstream task of rearranging a scene to a goal state, given by a target arrangement in a 2D image from a different scene.", "summary_of_recommendation": "The proposed implicit feature representation is very relevant to robotics and has been evaluate on a relevant task on a real robotic system. There are only minor issues regarding the choice of DINOv2 left. Post-Rebuttal: The authors addressed my concerns about DINOv2 in additional experiments.", "technical_quality": 5, "clarity_of_presentation": 4, "potential_impact": 4, "robotics_focus": 4, "quality_of_limitations": 2, "questions_for_rebuttal": "- Why is the depth R interpolated (line 112, 113) when real depth for camera view is available through the RGB-D pair? What exactly is interpolated here? How and why is this done?\n\n- What is the benefit of fusing features in 3D compared to just projecting 2D features from single views to a single-view 3D (a.k.a. 2.5D)? Would a rearrangement task also work with single-view 3D points clouds as long as the objects of interest are in the field-of-view?\n\n- How is the instance mask m_i (eq. 6) obtained from W^p, respectively p_i, and represented? If this is an instance index, then instance indices will not necessarily match between views and cannot simply merged over multiple views. This is broadly discussed in the supplementary material, but the paper is missing information on how to get from p_i to m_i. I suggest briefly mentioning the properties of m_i and p_i, adding an abstract function p_i -> m_i to the paper and referencing pseudo code in the supplementary material on how this mapping is done.\n\n- Why was DINOv2 chosen to provide W^f? Is this solely for its open-set zero-shot performance which does not require collecting new training data? The supplementary material shows additional correspondence accuracy comparisons, such as to Dense Object Nets. However in the paper, the choice of DINOv2 appears mostly arbitrary. Could you better motivate the choice of DINOv2 over alternative foundation model?\n\n- Is there any limitation in how much the starting state and the goal state can differ visually and geometrically? How much do they have to differ before no action sequence between start and target state can be found?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: **Paper summary**

This paper presents a new representation for manipulation-relevant scene features. A key advantage of this method is its performance when using sparse camera views to build a high-quality 3D scene representation. The results show how this representation enables model-based planning to solve rearrangement tasks, without requiring demonstrations like baseline approaches do.

**Review summary**


Summary of strengths:
+ This paper targets manipulation problems that are widespread in robotics and highly relevant to CoRL.
+ The proposed method leverages several pre-existing foundation models, and thus does not require any training to compute the neural field. 
+ All reviewers noted that the paper was well-written and easy to follow.


Summary of weaknesses:
- The main paper is missing quantitative results. These are included in the supplementary materials, but should be integrated into the main paper.  **[The revised paper has addressed this concern.]**
- The F3RM baseline does not perform as well as expected, leading to questions about whether it was implemented correctly. **[The revised paper has addressed this concern; the baseline performs poorly when sparse views are used, but performs as expected with more views.]**
- There is limited discussion of when/how the proposed method will fail. **[This concern is not addressed, but is a minor issue.]**

**Response to rebuttal**

The authors have extensively addressed the reviewers' concerns through additional comparisons and visualizations, and restructuring the paper to include quantitative results in the main text (rather than just in the supplementary materials). 

Two reviewers maintain their positive ratings (weak accept by FgtV and strong accept by EQkG) and one reviewer has significantly increased their rating (from strong reject to weak accept; vc8A).

**Recommendations for improvement**

* Please follow Reviewer vc8a's suggestion: "For completeness, it would be great for the authors to update the manuscript with some quantitative results comparing to F3RM as well."
Decision: Accept