Venue: CoRL 2022 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper proposes a learning from demonstration approach based on pretrained category-level dense 3D descriptors (NDFs). The paper formulates single-step object arrangement tasks as a SE(3) transformation between two object instances. The proposed method first encodes SE(3) transformations as a set of point descriptors and then optimizes for the transformation between objects by minimizing the distance in the descriptor space. The method requires a few (10) demonstrations for each task and works for unseen object instances of known categories.
strengths_and_weaknesses: Strengths
- Clever formulation of learning from a few demonstrations.
- Multiple improvements that address the limitation of prior work (NDF).
- Nice real world demo in the supplement.

Weaknesses
- Reliance on object segmentation and complete point cloud.
- Only works for known categories. Implicitly relies on labeled 3D object databases.
- More qualitative examples would improve the presentation, such as showing the variety of objects.
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: The paper's contribution is borderline for me, but I would recommend acceptance based on the novelty and good real-robot demonstration, if not objected by other reviewers.

The paper cleverly leverages category-level descriptors that establishes correspondence between object parts to learn tasks from a few demonstrations and almost no annotation. However, a big underlying assumption is that these category level descriptors can be learned well. I have doubts about these descriptors, as they assume a full correspondence can be established between two object instances from the same category. I don't see how they will work for categories like chairs. I think a descriptor that can deal with partial-to-partial correspondences is crucial to have.

The method also assumes that object rearrangements can be simplified into SE(3) transformations, where in reality many more constraints exist, such as robot kinematics or collisions between object parts. Relying on motion planner is fine for simple scenarios, but eventually the pick and place poses need to consider these factors, especially if objects have more complex geometry and the environment is more clutterred.
issues: - Line 71 says that translation equivariance is obtained by mean-centering point clouds. How does that work? Does this assumes the point clouds are complete? A bit more details will be helpful.
- Line 128 mentioned scaling $X_A$ to have scale similar to the salient object parts. How is this scale specified? From what I understand, the objects have various different scales, right?
- How are "task-relevant part of $O_A$" (line 98) specified in the original NDF?
- Why are the EBM model (line 188-202) that learns metrics on the descriptor space not susceptible to overfitting? It seems to be trained on only 10 examples.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper proposes a way to infer the SE(3) transform to move an object (represented by a point cloud) to satisfy some required relation with another object (also represented by a point cloud) e.g. to hang a mug on a rack. Both objects should belong to some a priori known classes, for each of which a dataset of point cloud instances should exist. These datasets are used to pre-train neural networks that allow subsequent inference of a transform from a small number (around 10) demonstrations. The proposed approach is evaluated on a set of experiments, including a real robot setting, and compared with two baselines showing improvements.
strengths_and_weaknesses: The paper builds heavily on the previous work on Neural Descriptor Fields. This work enabled inferring an SE(3) transformation for moving one object to satisfy a certain relation with the other when this other object is stationary and fixed. On the one hand, this paper goes in the right direction by removing this superfluous assumption. On the other hand, the path to this is rather incremental: two Neural Descriptor Fields are used instead of one. Additional refinement is added by using an additional multilayer perceptron resulting in a slight improvement in performance. Examples, including one with a real robot, that show improvements over some baselines are present. However, no code is provided in the supplementary and no promise of publishing the code upon acceptance is made which, in my opinion, is a big drawback. 

**Minor Comments**

1. Equation (2). The property seems to be invariance rather than equivariance: usually equivariance of F:X->Y means that F(Tx)=TF(x) where T is an element of a group that acts on both X and Y, while invariance is F(Tx)=F(x). Equation (2) is similar to the latter.
2. Footnote on page 2. “T = (R, T)” probably should be “T = (R, t)”.
3. Line 93. “it’s” -> “it is”.
4. Line 106 and other places. I suggest using the usual “l^1” instead of “L1”.
5. Line 117. Maybe change “3D coordinate” to “3D point”. In my humble opinion a coordinate means each of the three numbers in a tuple representing a 3D point, not the whole tuple.
6. Equation (7). As far as I understand this is not a constrained optimization problem. Phrasing it as a constrained one seems misleading to me.
7. Lines 166-167. You write “small errors can accumulate”. What is the iterative procedure meant here? Is this the two-step procedure defined by Equations (5) and (6). If so, the word “accumulate” reads a bit weird for a two-step procedure.
8. Figure 5 (b). At the bottom left the “0.00” and “0.0” interpenetrate.
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: Although being somewhat incremental, the paper makes some progress in a relevant problem. It does not feature a (promise of) an open source implementation of the method, which is a big drawback.
issues: Please comment or act on the following.

- Line 228. Do you train the MLP on 10 examples only? If so, how do you manage to get something reasonable from it?
- Figure 6. The container lying on a side with a bottle about to drop when the manipulator lets go of it is a genuine situation from an experiment or an improperly rotated picture?
- In your bibliography, you have a number of arXiv references for papers that were published. For some entries you have page numbers and for some you do not, same for urls. Please make the bibliography consistent and cite the published versions of papers.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper presents R-NDF, a framework for pair-wise object rearrangement that is robust to pose variations in SE(3). The framework builds on prior work in Neural Descriptor Fields (NDFs) – an approach for learning spatial descriptors of objects that are robust to 6-DoF transformations. NDFs are limited to a single object, whereas R-NDF can learn geometric relationships between pairs of rigid-bodies. Experimental evaluations include simulated experiments and baseline comparisons to simple pose regression and patch matching. The results also include some ablations that investigate individual components of R-NDF. Qualitative results include real-robot demonstrations, which showcase R-NDF’s ability to precisely place mugs on wooden racks with just a handful number of demonstrations.   

strengths_and_weaknesses: **Strengths**

+ On the whole, the idea of using SE(3) equivariant frameworks for robot perception is an interesting and practical direction for vision-based manipulation. Naive image-based perception systems struggle to generalize to unseen object poses and are not robust to scene changes. Equivariance is a property that has been crucial to data efficiency and robustness in 2D vision (like with UNets), so it’s great to see equivariance being adapted to 6-DoF manipulation in 3D settings, more specifically in multi-object scenarios. While data-augmentation is one possible alternative, it’s hard to guarantee robustness to pose-variations like with equivariant frameworks.  

+ R-NDF is evaluated both in simulated and real-world setups. Particularly, the real-world results are compelling. The framework can imitate high-precision tasks with just 5-10 demonstrations, and do this with any given pose-variation. Also, this robustness to pose-variations is evaluated with simulated experiments, which might be easily reproducible if the authors open-source their code-base. 
 
+ The figures in the paper are generally helpful. Figure 1 does a good job of providing an overview of the problem. Figure 2 provides a good summary of the approach and the high-level concepts. The experimental setup in Section 6 is mostly clear, and also includes a good set of ablations investigating individual aspects of the framework like energy-model based refinement and aligning query points. 

**Weaknesses**

- One major concern is that R-NDF makes a lot of assumptions specific to the two-object setup, which could be better highlighted in the paper. Specifically, R-NDF assumes access to: (1) object segmentations, good object point-clouds, and approximate object localization (at least a centroid), (2) no distractor objects with only two-objects in the scene, (3) collision-free motion plans, and (4) pre-trained NDFs for each object. 

(1) Object segmentation and localization might be obtainable for large rigid-body objects, but it’s unclear how robust R-NDF would be in cluttered scenes with heavy occlusions and inaccurate segmentations. 

(2) The EBM formulation in Section 4 is also highly-engineered towards a setup with two-rigid objects. It’s unclear how easily extendible or generally applicable R-NDF would be to N-object rearrangement or anything that involves more than a single-step geometric transformation.

(3) Section 5 briefly mentions that a motion-planner is used to reach grasp and place poses, but how is the collision avoidance handled? Particularly for the “Mug on Rack” example, how does the motion-planner know not to collide with the wooden stem before hanging the handle on it? 

(4) How long does it take to pre-train the NDFs? The paper claims only 5-10 demonstrations are needed to learn rearrangement transformations, but there is also the cost of pre-training NDFs on object instances, right?

These limitations don’t necessarily diminish the contributions of the paper, but it might be worth highlighting these requirements for future practitioners trying to adapt the framework to real-world manipulation challenges. 

- Another issue is that the relationship between R-NDF and k-PAM (Gao et al, Manuelli et al) is not sufficiently discussed. The related work section states “.. kPAM 2.0 generalize to pairs of unseen objects, but these approaches are trained on large task-specific datasets” but the methods section for R-NDF states that “NDFs are pretrained to enable reconstruction across a large dataset of 3D shapes”. So does kPAM use more shapes than R-NDF or not? Also, is there anything preventing the experiments from using kPAM (Gao et al) as a baseline for the experiments? 

- Minor: Section 4 could be re-written for better readability. There is a lot of linear algebra interweaved with the text, which makes the sentences hard to read at times. The text could be made more accessible by providing high-level intuitions before delving into mathematical definitions. 

References: \
Gao et al: [https://groups.csail.mit.edu/robotics-center/public_papers/Gao20.pdf](https://groups.csail.mit.edu/robotics-center/public_papers/Gao20.pdf) \
Manueli et al: [https://arxiv.org/abs/1903.06684](https://arxiv.org/abs/1903.06684) 

robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: This paper proposes a novel SE(3) equivariant framework for object-rearrangement and achieves compelling real-world results with limited data. There are some issues with the presentation of methods and results, but the framework itself could be of interest to the manipulation community.

**Post Rebuttal**  
The authors spent a lot of time and effort updating the paper and adding new sections, and I really appreciate that! These updates address most of my concerns.

My only reservation is still the strong assumption about "large rigid objects". As pointed out, there has been a lot of progress in instance segmentation and 3D encoders, but it's still unclear how R-NDFs would work with objects that, to begin with, are hard to represent with segmentations or pointclouds in the real-world like particles, clothes, and liquids. But nonetheless, "larger rigid objects" still cover a broad category of manipulation problems, and overall, this paper is quite relevant to the CoRL audience. So I am keeping my score of Weak Accept.
issues: See issues mentioned in the Weaknesses section.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper builds on Neural Descriptor Fields and extends it to be used on two arbitrary objects (as opposed to one unknown and one fixed). 
This is a more difficult problem as the query pointcloud location has to be defined from demonstrations. 
The paper proposes an optimization method to consistently arrive at initial pointcloud location with a help of annotating its (initial) center.
After the pose of the query is defined on object A using NDF of A, its instance on object B is also defined using NDF of B, yielding pose descriptors Z_A and Z_B.
These pose descriptors are averaged over demonstrations.
During inference first the query pointcloud pose is defined and then the relative object pose is computed via optimization.
To improve accuracy further the paper also proposes the minimization of an Energy Based Model that fine-tunes the final pose.
The method was tested in simulations and real experiments.
strengths_and_weaknesses: 
Strengths

The paper addresses a relevant topic in robot learning and manipulation.
NDF is a promising and novel method to encode relative object poses from pointcloud recordings and it shows good potential for generalization to similar objects.
This paper further improves its applicability to two-object-class scenarios, where the two object instances might have been unseen during training.
The paper presents a reasonable and technically mostly sound approach to achieve this goal and shows evaluations to support the desgin choices.
The clarity of the paper is also good, it is easy to follow.
The figures illustrate the approach well.

Weaknesses

I found the story around the Energy Based Model minimization vague. 
While it seems to mitigate the problem of inaccurate poses (after solving (4) and (5)), for me it is not entirely clear what is the source of pose error.
The paper discusses the accumulation of errors, but why do these error appear in the first place.
Does it relate to inaccurate demonstrations, limited expressiveness of NDFs, quality of the pointcloud and its segmentation?

The overall solution consists of multiple steps starting from camera calibration, training NDFs and pose descriptors till inference in real time, point cloud segmentation, optimizations to derive relative pose and grasp pose, etc.
While a few failure points (e.g. quality of depth) have been mentioned in the limitation section, more discussion on the limitation of the whole algorithmic pipeline would be welcome.
It seems to take a substantial effort to implement the algorithm, what are the key components to pay attention to, what are prone to fail?

robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: The paper provides a well motivated improvement to NDFs, the technical quality and clarity are good, but could use more improvements (see Strengths & Weaknesses). The simulation and real world experiment support the claim of the paper. More insights into further limitations and implementation details could improve my rating.

issues: Please address the limitations of the full training and inference pipeline.

Please comment on scaling to more objects, e.g., what happens with 3 objects? For example, place bottle left to the plate and right to the box.

Can NDFs be used for tasks other than pick-and-place?
review_title: null

Meta-review: metareview: Below is a summary of the strengths and weaknesses of the paper, according to the reviewers.

Strengths:

- SE(3) equivariance is an interesting and important topic to study.
- There are several improvements over prior work of Neural Descriptor Fields (NDF).
- The method can learn high-precision tasks from just 5-10 demonstrations.
- Experiments show that the method results in improvements over baselines.
- There is a real-world demonstration of the method (although only qualitative results).
- The paper is well written, and the figures are particularly helpful.

Weaknesses:

- The contributions are incremental with respect to NDF.
- The method relies on good object segmentation and a complete point cloud to work.
- The method has several assumptions, such as no distractor objects, a pre-trained NDF for each object, and known object categories.

In the rebuttal, please address the above weaknesses, as well as the other concerns and questions raised by the reviewers.

----------

Update after the rebuttal:

All four reviewers maintained their positive opinion of the paper after the rebuttal. This is a timely paper which proposes another application of implicit neural models to robotics. Reviewers found the paper to be somewhat incremental, but still sufficient for publication at CoRL and with compelling real-world results. It will be a useful paper for anybody working in implicit neural models or dense descriptors, although it may have limited appeal more broadly across the community.
confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept (Poster)