Venue: CoRL 2022 Conference

Reviews:

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper introduces the notion of “cross-pose”, a task-specific relationship between objects that captures a desired goal configuration. The proposed “cross-pose” function maps pairs of object pointclouds to a relative pose in SE(3); the function value is defined to be identity when the objects satisfy the relative pose goal relationship. The model is trained on a dataset of demonstrations of the goal condition, e.g. a mug hung on a rack, and is designed to operate at the category level for “action” and “anchor” objects. 

The main contribution of the paper is the formulation of cross-pose and an architecture for learning it. The central novelty in the problem formulation (in particular relative to Neural Descriptor Fields) is that both the action and anchor object are encoded, enabling generalization at the category level for both (in contrast to NDF, in which only the “action” object is embedded).
strengths_and_weaknesses: The paper studies an important problem. The description and analysis of the cross-pose function are thorough. The paper achieves impressive results on the simulated mug-hanging task in comparison to a recent state-of-the-art system, and the ablation results motivate the design decisions. 

The paper has some limitations which the authors are transparent about - the strongest assumption is that there is one such “cross-pose” which satisfies the relationship. In reality, there are a manifold of poses which satisfy the goal condition for these tasks, though this is a reasonable first approximation inherited from previous work. A larger conceptual limitation is the restriction of tasks to pick and place. While a large number of household tasks may be specified by a static observation, many cannot - e.g. “carry this mug of tea” is not well specified this way. Another limitation present from the evaluation is the need to train one separate semantic model for each variant of a task - i.e. each semantic placement (right, left, in, …) requires a unique model to be trained. We’ve seen methods like CLIPort and Socratic Models use natural language in conjunction with vision to handle such variation. This isn’t the focus of this work but it warrants mention.

The related work section is relatively weak. In particular, it fails to mention the baselines compared to in the mug hanging experiment section, the most similar related work, as well as related ideas such as keypoint methods or canonicalization (e.g. NOCS). From the text alone, the novelty of the proposed method relative to these baselines is not clearly conveyed. 

The comparison to existing methods is also a bit lacking. The related methods are absent from the PartNet evaluation without mention. I believe they are omitted because the anchor object pose is also varied, where DON and NDF would require a fixed anchor. This at least warrants clarification in the text (if not outright comparison in a "fixed anchor" experiment). Moreover, only one object category experiment from NDF was replicated - it's difficult to assess the improvement in performance offered by the proposed approach with a limited evaluation.
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: I'm recommending weak rejection on the basis of the paper needing substantial improvements to the quality of presentation and potentially additional experimentation. At the moment, the novelty is not clear relative to existing baselines and the related work, experiment setup, and results lack sufficient discussion. 
issues: Please see above recommendations regarding the related work and result sections. 

I would recommend moving the success rate tables to the main paper - the error in rotation/translation is difficult to assess task performance with. 

Question regarding the partnet simulation experiment: is the anchor object pose randomized, or just the target object? The description in 6.1 is ambiguous. 

The hardware results describe at length the modifications required to achieve the real world experiments, though the mug-hanging experiments demonstrate that the method is able to learn relative grasp poses. Is there a particular reason why this was forgone for the hardware experiment?

Minor typos:

Appendix 7.1 "quarry"
Section 5.1 “a semantic goal positions”
review_title: null

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper proposed a vision system that uses pose relationships between two objects for robot manipulation. Point clouds of two interacting objects are retrieved through DGCNN and the proposed TAX-Pose method learns to predict a task-specific pose transformation from demonstrations. The prediction is then used to plan a trajectory that accomplishes the task. The system has been tested in both simulation and real-world environments and shown that the model sample efficiency that can learn from a small number of demonstrations. Dense soft correspondence + residuals allows the method to generalize to novel object instances.
strengths_and_weaknesses: The author designed a cross-pose estimation system that helped with object manipulation. Comprehensive experiments were provided to show that the proposed method is beneficial in both simulation and real world. 
The downside of the paper is that the proposed method injects a significant amount of inductive biases to solve pick-and-place-ish tasks. The scalability of the method is questionable. First the method is heavily rely on a relatively good pretrained model for point cloud extraction e.g. DGCNN as used in the paper. It turns the problem setup from image space to feature space that makes the claim of a “vision-based system” a bit weak. In addition, the feature of zero-shot sim to real transfer and generalization to novel objects of the same category likely comes from the use of DGCNN. Secondly, the scalability is significantly limited once a model is trained. It only works for the exact pair of objects and for one specific task. It is not so clear by reading the paper, but it seems a model trained for “Left” task would not be able to do “Right” task. Or a model trained for “cube-left-oven” might not be able to do “cube-left-microwave”. Thirdly, the proposed method works only for pairs of objects. It seems hard to scale the system to support multiple objects or even generalize to the number of objects. In practice, real-world tasks often involve multiple objects and the complex dynamic among objects interaction. 

As the author mentioned in the limitation section, symmetry is not handled. However, Rotational errors (ER) are reported in the evaluation metric, which might not make much sense when objects like cubes, bottles, plates etc are involved.

robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: The scalability of the method is questionable and is seems the model can only operate in a very constrained scenario. As symmetry is not haddled, the comparision between TAX-Pose and baseline methods are a bit weak. Rotational errors (ER) is no longer a fair metric. In terms of the translational errors (Et), TAX-Pose does not show much advantages. 
issues: See above
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper tackles the problem of estimating the relative pose between two objects conditioned to a specific manipulation task. This pose is then used to guide a robot manipulator to pick and place one object in a desired location relative to the other.
The authors given a formal definition of the estimation task by introducing the concept of "cross-pose".
The proposed system, called TAX-Pose, is built upon Deep Closest Point (DCP)[20], with the addition of specific components required by the manipulation task, and OMPL for motion planning.
The different components of the system are validate in ablation studies.
Experiments show that TAX-Pose achieves state-of-the-art performance and it is able to generalize to different objects of the same category.
Sim-to-real transfer is achieved without fine-tuning on some of the manipulation tasks tested in simulation. The proposed system is able to achieve much higher success rate than the baseline in the real-world.


strengths_and_weaknesses: -- Strengths

- The paper is well-written and easy to read. The authors were able to describe the problem statement and methodology in a clear and descriptive way in spite of the little space available.  The paper contains all the information and references that the reader needs to understand the proposed method and make connections to the current state-of-the-art.

- The definition of the term of "cross-pose" is intuitive and its mathematical formulation is sound. The reader needs to check the Appendix to get the full idea behind this definition.

- The proposed system, TAX-Pose, is a slightly adapted version of DCP for the task of "cross-pose" estimation. The authors propose to add a residual net block, which applies a residual vector to each point correspondence, to account for the case when the object are not in contact or overlap. I find this an elegant solution.

- The experiments are relevant. The different components of the system are validated in ablation studies. TAX-Pose achieves state-of-the-art results in simulation. Simulation to real-world transfer is achieved without fine-tuning, thanks to, according to the authors, the translation-equivariant property of TAX-Pose.

-- Weaknesses

- The originality of this work is a weakness. TAX-Pose heavily builds upon DCP for the relative pose estimation and completely rely on OMPL for planning. The main difference with respect to DCP is a revised version of the block that in DCP is called "Pointer". In this work a residual net is used to account for the case when the object are not in contact or overlap. Although this is an elegant solution, it is a minor increment with respect to the DCP architecture.

- I would suggest  to give more insights about the results. In particular:
Do the authors have an intuition why the rotation results of TAX-Pose are much better that the baselines? 
Why the sim-to-real transfer capabilities of TAX-Pose are much better than Goal-Flow? I would expect that the translation-equivariant would improve the sim-to-real transfer capabilities of TAX-Pose but the results of Tab. 2 show a big gap with respect to Goal-Flow. Could the authors explain their intuition about the reason behind this result?
Why was it not possible to evaluate the mug hanging experiments in the real-world?

robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: This paper has some potentials to have an impact in the field.
Although the methodology is an incremental contribution with respect to DCP, the authors have shown that DCP can be used to solve a relative pose problem rather than point-cloud registration problem.
The experiments are relevant and the ablation studies useful to validate the different components of the system. However, further discussions on the results would improve the quality of the work. 

**After Rebuttal**
I appreciate the authors' response to my concerns. In my opinion, this work is a good contribution for the conference. I confirm my initial recommendation of weak accept.
issues: Besides the comments regarding weaknesses in the corresponding section, I include some other minor issues below:

- The introduction of the variables $T_{\alpha}$ and $T_{\beta}$ in the section "Problem Statement" seems not necessary since such variables are not used anymore in the text. 

- It seems that \tilde{P}_{A} in Fig. 3 is call \tilde{V}_{A} in the text. Same for B.

- Reference [34] and [35] cite the ArXiv version of the papers. However, the works have been published in ICRA 22 and CoRL 18 respectively. I would suggest to update these references.
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: The paper proposes an approach to estimate the appropriate transformation to apply to an object in order to achieve the desired relative pose between two objects in a given task. Experimental evaluation shows convincing results in two different settings. 
strengths_and_weaknesses: Strengths:

1. The paper studies an interesting perspective in manipulation, treating a manipulation task as achieving a desired relative pose between objects. The proposed method is novel.
2. The paper is well-written (despite some minor notation issues) and technically sound. 
3. The experimental results are compelling.

Weaknesses:

1. The framework seems hard to extend to tasks involving more than two objects. What if a task requires five objects to achieve some desired relative poses?
2. There are some minor notation issues (see the ‘Issues’ section below).
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: The paper studies an important problem in manipulation, defining the task as achieving desired relative pose between objects, and proposes a novel framework to solve it. The paper is well-organized. Although it seems like the framework is constrained in a two-object scenario, I believe the paper is a strong contribution and can potentially have a major impact.
issues: 1. Can the authors comment on if it is possible to extend the framework to tasks that involve more than two objects?
2. I wonder if the authors have thought about using some SE(3)-equivariant network architecture like the vector neuron used in NDF [34]. Will that simplify the training process?
3. There are some minor notation issues:
    1. Should $\tilde{\mathbf{V}}_\mathcal{A}$ and $\tilde{\mathbf{V}}_\mathcal{B}$ in Equation 6 and 7 be swapped? If I understand line 167 and 168 correctly, $\tilde{\mathbf{V}}_\mathcal{A}$ is the correspondence for  $\mathbf{P}_\mathcal{B}$. In other words, $\mathbf{P}_\mathcal{A} \in \mathbb{R}^{3\times N_A}$ , then $T^{GT}_\mathcal{AB}\mathbf{P}_\mathcal{A}\in \mathbb{R}^{3\times N_A}$. However, based on the definition of $\tilde{\mathbf{V}}_\mathcal{A}$ in line 167 and 168, $\tilde{\mathbf{V}}_\mathcal{A} \in \mathbb{R}^{3\times N_B}$.
    2. What are $\tilde{\mathbf{P}}_\mathcal{A}$ and $\tilde{\mathbf{P}}_\mathcal{B}$ in Figure 3? Those symbols does not exist in the paper.
review_title: null

Meta-review: metareview: This paper defines the cross-pose, the relative pose between two objects in an object manipulation task. End-to-end systems often fail to generalize to new objects. They create a vision-based system that learns to estimate cross-pose for a given task. Using cross-pose, a motion-based planner can be used to enable the robot to complete the task.

Strengths:

* Important problem that can enable generalization across multiple tasks with much less training data.
* Clear description and analysis of cross-pose.

Weaknesses:

* Cross-pose does not generalize when there is more than one possible start/end for the task.
* Missing some related work, and not clearly differentiated from some related work.
confidence: 3: The area chair is somewhat confident
Decision: Accept (Poster)