Venue: CoRL 2023 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: This paper proposes to use diffusion models to predict rigid transformations applied to object point clouds in order to solve a manipulation task. The model gets as input 3D point clouds of the object that should be manipulated and the surrounding scene. The diffusion model iteratively outputs transformations to rearrange the object to a goal configuration. A key aspect of the approach is that multiple solutions can be predicted. The training data is generated in simulation using manually designed heuristics.
review: In general, the paper presents a novel idea of using diffusion models to solve robotic manipulation tasks. 
The paper contains a few interesting technical details, such as cropping the scene point cloud to focus on relevant parts, the loss function loss, the exact diffusion setup adjusted to predicting rigid transformations.
The reviewer appreciates the detailed appendix which clarified a lot of my initial questions.


However, I think that the paper is missing central arguments why the proposed method advances the state-of-the-art:

* The main weakness of the paper is the following: The data to train the models is generated in simulation using manually designed heuristics. While this in principle is sound and a common approach, in my opinion, one then has to show that the learned method exhibits capabilities that go beyond those manually designed heuristics. I currently do not see why the point clouds as obtained in this work cannot be transformed with the same heuristics as for generating the data.

* How does this method extend to multi-step manipulation problems?

* I suggest looking into these papers:
[1] Yifan You, Lin Shao, Toki Migimatsu, Jeannette Bohg. "OmniHang: Learning to Hang Arbitrary Objects using Contact Point Correspondences and Neural Collision Estimation". ICRA 2021.
[2] Jung-Su Ha, Danny Driess, Marc Toussaint. "Deep Visual Constraints: Neural Implicit Models for Manipulation Planning From Visual Input". RA-L 2022.
I suggest to potentially adding them as a baseline (especially OmniHang). There are also other papers exploring energy-based methods to deal with multimodality.

* Regarding the limitation section, the fact that there is no transfer to new tasks and that separate models for each task are trained could be added.
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: As written in the review above, the paper contains novel ideas and is well written.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: The high-level goal of this work is to rearrange rigid objects to  location that satisfies geometric relationship with the scene.
The author propose to use diffusion method for iterative pose denoising.
Experiments are conducted in both simulation and real-world to demonstrate the effectiveness of the propose method.
review: I'm not familiar with research on rearrangement.

## Strength: 
1. Reasonable approach and clear writing
1. Comprehensive experiments in both simulation and real-world
2. good results in both simulation and real-world.

## Weakness:

1. I'm not sure the necessity to use diffusion models in the setting, even though they are popular nowadays, What's the benefit of using diffusion models.
2. In Fig3 (b), the ablation shows that the success rate is sensitive to the crop rate. In my opinion, this somewhat indicates that the proposed method is not robust. Also, does the same cropping method is applied to the baselines as well?
3. I'm also not sure how the novelty compared to [41], which also uses the diffusion models and seems to handle a more challenging task.

## Some minor suggestions
1. in L105 $f_\theta$ should include the $pos\\_emb(t)$, as it's a input variable.

robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: I have never read paper on object rearrangement before. It's also the first time I review a paper using diffusion models, though I read some papers before.  I understand the high-level goal and approach of this work, but I'm not experienced enough to assess the practicality, effectiveness and novelty compared to previous methods.  I give a weak accept because the overall approach looks reasonable to me, the results  (especially the supplementary video) looks good, and the writing is  clear.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: The proposed system, RPDiff, addresses the problem of object rearrangement in a scene by operating directly on 3D point clouds. It overcomes challenges associated with multi-modality and generalization to diverse scene layouts. The main contributions of the paper are:

- Iterative Pose De-noising: RPDiff uses a diffusion model to perform iterative de-noising, gradually reversing the perturbation process applied to the object point cloud. This allows the system to generate diverse solutions for rearrangement tasks.

- Local Conditioning: To support generalization to novel scene layouts, RPDiff locally encodes the scene point cloud by cropping a region near the object. This allows the model to focus on task-relevant details and ignore irrelevant global structure, improving generalization.

Evaluation and Results: RPDiff is evaluated on three distinct rearrangement tasks in both simulation and the real world. It outperforms existing methods in terms of success rate and coverage.

review: Quality:
Overall, the quality of the paper is high. The authors provide a clear and well-structured explanation of their proposed system for object rearrangement using 3D point clouds. They present a good enough problem analysis and provide fair descriptions of the methodology, experiments, and results. It’s hard to evaluate the technical foundation, mostly due to too much preprocessing / manual manipulation of the data (most of the details are included in the appendix). I understand that this is unavoidable due to the inherent requirements of the problem setup, but at times it feels like some critical details were omitted to push the reader away from the relatively ‘trickier’ parts of the approach that might indeed hinder the application of the approach in broader scenarios. But it definitely addresses important challenges in the field of robotics learning, so including a better discussion of those ‘hardships’/extra steps needed would also inform the reader about the complexity of the problem.

Clarity:
The paper is generally well-written and organized. The introduction effectively motivates the problem of object rearrangement and provides a clear context for the research. The methodology section describes the proposed system in a detailed and coherent manner. The experimental setup and evaluation metrics are clearly defined. The figures and captions are informative and support the text. However, in some sections, the paper could benefit from additional clarifications, especially regarding the specific implementation details.

Originality:
The paper presents a novel approach to object rearrangement by leveraging multi-modal predictions, iterative pose de-noising, and local scene conditioning. While there have been prior works on object rearrangement and pose estimation, the combination of these techniques in the proposed RPDiff system appears to be original. The authors contribute to the field by addressing challenges associated with multi-modality and generalization in object rearrangement tasks, and they introduce a new method that surpasses existing approaches in terms of precision and coverage.

Significance:
The significance of the work is fair,, specifically in the domain of object manipulation and cleanup tasks. The ability to rearrange objects in scenes accurately and efficiently is crucial for many real-world applications. The proposed RPDiff system tackles key challenges in this area, including handling multi-modality, generalizing to diverse scene layouts, and producing precise and accurate predictions. The experimental evaluations demonstrate the superiority of RPDiff over existing methods, highlighting its practical value and potential impact on robotics research and applications. 

Strengths:

- Novel approach: The proposed Relational Pose Diffusion (RPDiff) system addresses the problem of object rearrangement in a scene using 3D point clouds, allowing it to handle unknown geometries in the real world.

- Multi-modal generation: RPDiff overcomes the challenge of fitting multi-modal demonstration data by using an iterative pose de-noising process, producing diverse and precise solutions for rearrangement tasks.

- Local conditioning: The system incorporates local conditioning by cropping a region near the object in the scene point cloud, improving generalization and precision by focusing on task-relevant details.

- Evaluation in simulation and real-world: The system is evaluated on three distinct rearrangement tasks in both simulation and the real world, demonstrating its effectiveness and applicability in different scenarios.

Weaknesses: (the authors acknowledge most of those points to some degree)

- Clarity: RPDiff employs a transformer architecture featuring self-attention. However, the authors state that they require an extra step called Local Conditioning, which involves cropping the scene around the object. The main text lacks clarity regarding the model's pipeline and requires further explanation. Consequently, it remains unclear why the additional Local Conditioning is necessary while using the self and cross attention mechanisms, as they should be capable of detecting relevant and irrelevant points in the scene.

- Data requirements: The system relies on scripted policies in simulation to obtain demonstration data, which may limit its applicability to new tasks. 

- Real-world analysis: As the authors mentioned in their limitations, they had problems with switching from simulation to real-world experiments. Consequently, all their main results were obtained solely from simulations. In the Appendix, they acknowledged that they observed worse overall prediction performance in the real world, however they did not provide specific details about the performance. 

- Sim2real gap: As highlighted by the authors as well, there is a potential sim2real gap due to training on simulated point clouds. Incorporating sim2real transfer techniques and finetuning on real-world data could improve the system's performance in real-world scenarios. However, the paper does not go into the details about how serious this gap was, how they addressed it, and it does not extensively address techniques to bridge this gap, leaving room for further investigation.

- Open-loop execution: The system executes the predicted placements in an open-loop manner, which may limit its ability to track progress and react to disturbances. As mentioned a closed-loop policy could enhance the system's performance in dynamic environments. However, how to integrate the whole approach into such a policy has not been discussed. For instance, considering the final phases of such rearrangement tasks (e.g., shelving a book between other books), the visual perception data may become less and less informative (e.g., due to occlusions).

- The paper assumes known object states and 3D models (at least in training as far as I understood), which may not be realistic in some real-world scenarios. The reliance on scripted policies for data generation limits scalability.

Minor comments: 
- the term multi-modality sounds like sensor modalities. I don’t know whether this usage is well-accepted within this (object rearrangement) context but if not, then a different naming might be more informative / less confusing.

- Generalization: The paper should be more precise about what is meant by generalization in these scenarios, while being more clear about train and test data.

- line 115: identify -> identity
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: The paper presents a well-structured demonstration of the work on using a diffusion model called RPDiff as a transformation prediction network in robotics. The authors draw from other domains and previous works to develop their approach. The writing is concise and easy to follow. However, there are some clarity issues, particularly regarding the pipeline and the necessity of local conditioning alongside self and cross attention mechanisms. The lack of real-world experimental analysis impairs the actual impact of the work. While the paper combines ideas from various domains, its main contribution seems to be the iterative use of the diffusion model, making it an incremental rather than groundbreaking work. Some important details are relegated to the appendix instead of being included in the main text.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: The authors propose an iterative diffusion method for object-scene re-arrangement. For tasks such as shelving books, the authors propose using a DNN to predict a transform a given object (e.g. book) from an initial pose to a final predicted pose, and execute this behavior in an open-loop manner. Specifically, the DNN predicts a series of transforms in an iterative manner, similar to diffusion models. The approach is able to produce multimodal predictions (e.g. multiple possible locations to shelve the book) and generalize to different environments and initial setups.
review: Strengths
* Novelty: The paper shares a similar spirit to recent works applying diffusion methods to robot manipulation ([41, 58]). This is a nice and simple formulation for leveraging diffusion models for manipulation.
* Notably, the main strength is to produce multimodal predictions.
* The authors demonstrate the ability to successfully execute their approach in the real world.
* Clarity: Overall the paper is clear.

Weaknesses
* Approach assumes that a point cloud segmentation is already provided for the scene and object (ie. the object is treated as one class, scene as another). It's not clear that this will always be easy to obtain
* Are there better alternatives to random perturbations in the forward diffusion process?
* Requires a manually defined offset based on prior knowledge of the task (line 198)
* Open loop execution

Clarity
* Page 2 describes a lot of motivation and overview of the pipeline. It would be useful to have associated images/figures throughout the explanation. For example:
    * Providing examples of the "local cropped region" around the object in line 67
    * What the "noised point cloud" looks like in line 51
    * What a "large perturbation" looks like in line 56
* The goal specification is not very clear in paragraph 1. Providing examples would help clarify things further. For example: for shelving books, is there a specific "shelving books dataset"? Is it one dataset per task, or can we use one dataset to learn multiple tasks? Is there a way to specify where to place the book, or is the goal to shelve a book in general? This is clarified only in the experiments section.
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: Overall, the paper is well written and presents a nice, simple formulation for applying diffusion models to robotics in order to achieve multimodal predictions. Moreover, the results are exciting, showing very high precision and recall in being able to achieve these multimodal predictions. The authors additionally provide real-world demonstrations. I'd be curious how the proposed method compares to recent papers such as [41, 58].
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary_of_paper: This paper presents an approach for rearranging objects in a scene to achieve a desired placing relationship, operating with geometries, poses, and scene layouts. The system produces multi-modal distributions of object transformations for rearrangement. It is claimed that it can overcome the difficulty of fitting multi-modal demonstration datasets and facilitating integration with planning algorithms that require diverse actions to search through. 
review: The proposed approach addresses a complex task of multi-modal placing. The method is sound, and the technical specifications seem to be logic and reasonable. Their results illustrate the capabilities of the framework across a diverse range of rearrangement tasks involving objects and scenes that present a large number of feasible rearrangement solutions. Overall, the paper is well written, coving the relevant literature reviews with distinct demonstration of the merits of the proposed approach. 

I recommend acceptance of this paper.

robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: This is a paper highly relevant to robotics, with learning approaches for addressing a critical robotics problem. It is well developed, and I highly recommend it for acceptance.
review_title: null

Meta-review: The reviewers agree that this paper addresses an important problem and that the proposed approach, while rather incremental, is sufficiently novel. While several questions about technical description and experimental evaluation have been addressed by the author rebuttal and discussion, some concerns such as about the limited transferability to real-world remain. 

The suggestions of the reviewers should be thoroughly incorporated in a camera-ready version of the paper.
Decision: Accept (Poster)