Venue: CoRL 2023 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: 
The paper proposes an approach to behavior cloning for manipulation. The manipulation behavior is generated by composing a small set of primitives. Each primitive is defined with respect to object parts, i.e. the rim of a bowl or the handle of a mug. The primitives are learned using behavior cloning. The system is presented with a database of expert trajectories that are labeled with behavior and with the parts involved. A diffusion model is trained to reproduce the behaviors, parameterized by the shape of the objects (encoded as a point cloud) and the task. A key aspect of the work is that multiple primtives can be combined at test time. This is done simply by adding the de-noising noise from all to-be-combined primitives. The resulting trajectories are expected to achieve the objectives of each primitive.

review: 
I think the general approach here is interesting. There are a couple of key ideas: 1) creating parts-based primitives rather than object-based primitives; 2) combining diffusion policies by simply adding. I think the parts-based approach is a little unusual in the literature and makes sense and I'm happy to see it here. Similarly, the approach to combining policies is also interesting. However, I'm not at all clear on how effective the adding approach would be -- the concern would be that multiple objectives cancel each other out. It would be good to explore this question in future work.

There is a comparison to PC-DDPM and part-aware PC-DDPM. I don't know if there are other methods that should be comared against, but it seems like at least some kind of baseline is being done here.

I'm also disappointed that the parts need to be hand labeled instead of being discovered. It does seem like there is a lot of hand coded information here. Nevertheless, the overall ideas are novel and I'm not super concerned about those points.

Overall, I think this is a decent paper that has interesting ideas.

robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: I think this is a strong paper, but probably not Earth shattering. I think we should accept, but if another reviewer is more knowledgeable and feels it should be rejected, I'm fine with that.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: This paper proposes an approach for learning robot manipulation skills and generalising them to unseen objects from simulated demonstration data. The approach learns several diffusion models for a specific task, where each diffusion model is conditioned on only a single part of the function and anchor object and is responsible for capturing the distribution of valid function object poses for that task. For example, one diffusion model could be conditioned on the point cloud of the rim of a mug and bowl, while another could be on the bowl's body and the mug's handle. At test time, the output of all diffusion models is combined to predict the starting and ending pose for the function object, which, when realised, would complete the task.
review: Although the method is interesting, it appears to require a lot of work, engineering and prior knowledge to get it working for a novel skill. For example, consider the skill of pouring. 

Firstly, all demonstrations of the pouring skill must be coded in simulation. This requires engineering a heuristic for the pouring skill that will be compatible with many different function and anchor object categories and will work for multiple pouring tasks, such as: pouring from a mug into a bowl, from a pan into a bowl, from a wine glass into a container etc. Having coded such a heuristic that works reliably enough to generate multiple demonstrations involving different object categories is already equivalent to coding a policy capable of reliably executing the skill, somewhat reducing the need for the proposed method.

Secondly, all demonstrations must be annotated with part-based functional correspondences. For example, for the pouring skill this would be <align, rim(mug), rim(bowl)>, <tilt, body(mug), body(bowl)> and <facing_up, handle(mug), body(bowl)>. Decomposing a general skill into these part-based functional correspondences is additional manual work that must be carried out for each new skill we would like to teach a robot. 

Lastly, based on the part-based functional correspondences, the relevant object parts for the skill must be extracted in all demonstrated trajectories. Furthermore, during inference, they must also be segmented reliably.  

**Strengths**:
- Simultaneously predicting a motion from different parts of a function and anchor object and combining the predictions is quite appealing.

**Weaknesses**
- The proposed approach requires a lot of engineering for each new skill. Namely, the biggest issue is providing demonstrations in simulation for a given skill involving different object categories for the anchor and function object. On the one hand, the data generation process could be automated. However, after doing this, we no longer have a need to learn the skill as we already have a policy capable of executing it. On the other hand, we could collect the demonstrations manually, which would be very time-consuming.
- The proposed approach requires a lot of prior knowledge, such as part-based functional correspondences for each skill and a separate frame of reference for each considered part of the object.
- Representing general skills as a starting and ending pose is quite limited. For example, for the pouring task, the robot may spill some of the liquid as it approaches the anchor object without any constraints on the orientation of the function object during the approach phase.
robotics_focus: Relevant but unlikely to deploy to hardware in near future
quality_of_the_limitations_section: Limitations are not well addressed
summary_of_recommendation: The proposed method requires much manual engineering to set up a simulated pipeline for obtaining demonstrations. Moreover, the paper is very difficult to follow, and the experiments section is unclear, making it difficult to assess how well the method works and how it was evaluated. Moreover, it is difficult to understand how the method was deployed in the real world, given that each diffusion model makes predictions in the coordinate frame of the considered object part, which might not be annotated for novel real-world objects.
review_title: null

final_score: rating: Strong Reject: I recommend rejecting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary_of_paper: This paper proposes an approach for learning manipulation skills which involve the interaction of two objects, with one object actively controlled by the robot. The key idea is to decompose objects into their constituent parts, and define skills in terms of the correspondences between parts. The authors propose learning separate diffusion models per correspondence, which generates initial and final poses for the actively controlled object conditioned on point-clouds. Skills involving multiple correspondences are then defined by the composition of the appropriate diffusion models. The authors demonstrate their approach on two tasks and show that the method can achieve generalization to unseen object categories. The authors also include a real-world demonstration of the method. 
review: Strengths
- Authors show real-world demonstration showing impressive successful sim-to-real transfer, including generalization to unseen mugs
- The authors proposed method of using part-part correspondences as the basis of modelling manipulation actions, and the integration of this with diffusion models is very interesting and novel. I believe this paper is very relevant to the CoRL community. 

Weaknesses
The main weakness of the paper is that in places it is unclear and missing key details. In addition I believe there is room for improvement in the baselines used for comparison. 

- On line 135, and in the equation below line 144, the authors state that each individual takes as input two object point-clouds, but in the equation below line 160, these are now partial point-clouds. From the rest of the text, particularly the discussions of the baselines, I am inclined to think that the authors use partial point-clouds of the parts as input but it is unclear. 
- In the section 'Inference-time combination' the authors discuss performing the diffusion in SE3. This is further ablated in the experiments with the model with a 6D rotation parameterization. However, performing diffusion on SE3 is not a contribution of this paper, since that has been proposed in [40]. The section 'Inference-time combination' should be rewritten to make clear that this is from prior work and not part of the authors contribution. 
- I found the description of the baselines and their relationship to CPM very unclear. In particular, for both PC-DDPM and part-aware-PC-DDPM, it is very unclear to me what their input/output is. Is there a model per correspondence as with CPM, or a model per task? If neither, how are the correspondences/tasks encoded? This becomes especially important for part-aware-PC-DDPM because from the text I do not understand what the difference between part-aware-PC-DDPM and CPM is. 
- I cannot find anywhere in the text a definition of the performance metric. Is it a success rate? Over how many evaluations? The authors proposed approach shows improved performance including on unseen tasks according to this metric, but I cannot list this as a strength of the paper until I know what this performance metric means.
- The authors baselines are only other diffusion models, and all of the baselines are developed by the authors. I think they would be much more convincing if they were contrasted from another method from the literature. Perhaps something from the offline RL literature such as CQL. 
- One key limitation that is not discussed in the limitations section is the quite restrictive set of tasks that this method can be applied to, namely consisting of two-object interactions with only one object actively manipulated. 
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: I think this paper shows promise, but due to it missing some key details I do not believe it is ready for acceptance in its current form. If the manuscript was revised to improve clarity and add the key missing details (particularly the performance metric), I would be happy to change my recommendation to a weak accept. If the authors further incorporate an additional non diffusion baseline, I would change to a strong accept.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: The paper proposes composable part-based simulation – it’s tackling the problem of learning and genralizing robot manipulation skills through part-part correspondences. The method proposes the use of composable diffusion models which capture the various inter-object relationships. 

review: 
## Strengths

The method leverages the inductive bias that parts and their language descriptions capture their functional affordances, formalizing an action as a set of rules relating to object parts that can be learned. This approach could be generalized to novel unseen instances without the need for massive datasets. 

The related work is well structured and sufficient.

Part-based methods allow easy generalization to objects with slightly different part structures (such as cups to mugs).

An interesting novel application of diffusion models to robotics.

The experimental results are convincing of the method's usefulness.

## Weaknesses

The method assumes access to a set of symbolic constraints associated with each task. 
Additionally, it is unclear how constraints, which are central for this method, are defined - are they specified by a user with specific parameters optimized by the model, are they fully pre-defined, are they fully learned?

Only two tasks are considered for generalization to unseen categories, of which the second is not particularly impressive. Other tasks that could have been considered include handled object on rack, gripper on various objects.

It is unclear why the method was not compared with other baselines such as TAX pose and NDF which tackle a similar problem

The limitations are only briefly discussed.

## Other minor comments

I suggest rephrasing the research contribution of the work. A research contribution is a novel and significant addition to the field of study that advances the existing knowledge, theories or practices. 

Modelling manipulation based on part-part correspondences is not novel – many works within robot affordance learning research have done this (e.g. refer to the works of Eren Aksoy, or some of the references cited). Additionally, the third contribution is just a result of the other two.

Some citations are capitalized wrong. 

L73: should this be “within” instead of “between”

Figure 2 and its caption are unclear – How does the combined noise feed back into the sampling? 

Where is the point cloud transformer in the figure (referred to on L138)? It has no clear start or end, and the images used to represent pose are confusing. 

L145 – T0 is different from the T0 used in the equation.

robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: Overall, the method is interesting and appears useful, but there are some unanswered questions. I am recommending a weak acceptance as I believe the concerns are all addressable, but I can ultimately be convinced either way depending on the author’s responses and the comments of other reviewers.
review_title: null

Meta-review: This paper proposes a novel method based on object part decomposition and their correspondences to improve learning and generalization of robotic manipulation skills. All the reviewers recognize that the paper contains interesting ideas despite the fact that the current state of the approach requires a lot of manual engineering work. In the period of the rebuttal, the authors have provided careful answers with additional experimental results, e.g., two novel baselines,  to address the major questions of the reviewers. 3 reviewers suggest weak accept and one switches from strong reject to  strong accept after the rebuttal.
Decision: Accept (Poster)