Shelving, Stacking, Hanging: Relational Pose Diffusion for Multi-modal Rearrangement
Authors: Anthony Simeonov
Ankit Goyal
Lucas Manuelli
Yen-Chen Lin
Alina Sarmiento
Alberto Rodriguez Garcia
Pulkit Agrawal
Dieter Fox
Venue: CoRL 2023 Poster (2023)
Keywords: Object Rearrangement
Multi-modality
Manipulation
Point Clouds
Relations
Diffusion

Abstract: We propose a system for rearranging objects in a scene to achieve a desired object-scene placing relationship, such as a book inserted in an open slot of a bookshelf. The pipeline generalizes to novel geometries, poses, and layouts of both scenes and objects, and is trained from demonstrations to operate directly on 3D point clouds. Our system overcomes challenges associated with the existence of many geometrically-similar rearrangement solutions for a given scene. By leveraging an iterative pose de-noising training procedure, we can fit multi-modal demonstration data and produce multi-modal outputs while remaining precise and accurate. We also show the advantages of conditioning on relevant local geometric features while ignoring irrelevant global structure that harms both generalization and precision. We demonstrate our approach on three distinct rearrangement tasks that require handling multi-modality and generalization over object shape and pose in both simulation and the real world. Project website, code, and videos: https://anthonysimeonov.github.io/rpdiff-multi-modal


Reviews:
[{"review_id": "gfMsmYaVf7", "reviewer": "Reviewer_AZ9E", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The high-level goal of this work is to rearrange rigid objects to  location that satisfies geometric relationship with the scene.\nThe author propose to use diffusion method for iterative pose denoising.\nExperiments are conducted in both simulation and real-world to demonstrate the effectiveness of the propose method.", "summary_of_recommendation": "I have never read paper on object rearrangement before. It's also the first time I review a paper using diffusion models, though I read some papers before.  I understand the high-level goal and approach of this work, but I'm not experienced enough to assess the practicality, effectiveness and novelty compared to previous methods.  I give a weak accept because the overall approach looks reasonable to me, the results  (especially the supplementary video) looks good, and the writing is  clear.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "I would appreciate it if the authors can address my questions on weakness part, especially on why diffusion models are required for this task given that $h_\\phi$ is required to score the prediction poses in addition to $f_\\theta$\n\nL92 mentions that grasp sampler, IK solver are required. Is it specially required for the proposed method, or is it a requirement for baselines as well?  I'm not familiar with this task. So the authors may disregard me if the question do not make sense.\n\nLastly, Equation (2) is not very clear to me: \n$\\hat T ^{(i-1)}$ and $\\hat T ^{(i)}$ are not used? Do you mean $\\hat P_O^{i-1} = \\hat T ^{(i-1)}\\hat P_O^{i}$\n", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "I'm not familiar with research on rearrangement.\n\n## Strength: \n1. Reasonable approach and clear writing\n1. Comprehensive experiments in both simulation and real-world\n2. good results in both simulation and real-world.\n\n## Weakness:\n\n1. I'm not sure the necessity to use diffusion models in the setting, even though they are popular nowadays, What's the benefit of using diffusion models.\n2. In Fig3 (b), the ablation shows that the success rate is sensitive to the crop rate. In my opinion, this somewhat indicates that the proposed method is not robust. Also, does the same cropping method is applied to the baselines as well?\n3. I'm also not sure how the novelty compared to [41], which also uses the diffusion models and seems to handle a more challenging task.\n\n## Some minor suggestions\n1. in L105 $f_\\theta$ should include the $pos\\\\_emb(t)$, as it's a input variable.\n", "extra_scores": {"impact": "2: Unsure, difficult to assess."}, "extra_text": {}}, {"review_id": "aanXfbxsHj", "reviewer": "Reviewer_vuUj", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper presents an approach for rearranging objects in a scene to achieve a desired placing relationship, operating with geometries, poses, and scene layouts. The system produces multi-modal distributions of object transformations for rearrangement. It is claimed that it can overcome the difficulty of fitting multi-modal demonstration datasets and facilitating integration with planning algorithms that require diverse actions to search through. ", "summary_of_recommendation": "This is a paper highly relevant to robotics, with learning approaches for addressing a critical robotics problem. It is well developed, and I highly recommend it for acceptance. ", "technical_quality": "Excellent: Very convincing and thorough.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "I do not have any specific questions.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The proposed approach addresses a complex task of multi-modal placing. The method is sound, and the technical specifications seem to be logic and reasonable. Their results illustrate the capabilities of the framework across a diverse range of rearrangement tasks involving objects and scenes that present a large number of feasible rearrangement solutions. Overall, the paper is well written, coving the relevant literature reviews with distinct demonstration of the merits of the proposed approach. \n\nI recommend acceptance of this paper.\n", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}, {"review_id": "QZucIKL07m", "reviewer": "Reviewer_ZYnf", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposes to use diffusion models to predict rigid transformations applied to object point clouds in order to solve a manipulation task. The model gets as input 3D point clouds of the object that should be manipulated and the surrounding scene. The diffusion model iteratively outputs transformations to rearrange the object to a goal configuration. A key aspect of the approach is that multiple solutions can be predicted. The training data is generated in simulation using manually designed heuristics.", "summary_of_recommendation": "As written in the review above, the paper contains novel ideas and is well written. ", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "See review.\n\nAdditional questions:\n- The authors state that the method generalizes over shapes. Where is this evaluated?\n- In the Neural Descriptor Fields paper I see success rates of nearly 90% for hanging a mug, why is this baseline significantly lower here?\n- I am a bit confused about the one-hot encoding in the transformer input. Is the reason to use this in order to share the point-cloud encoder between the scene and object point-clouds?\n- \"In both simulation and the real world, we manually completed offset poses for moving the object before executing the final placement.\". Can this be explained in more detail?\n- Are there ablations regarding the different loss terms?\n- What about just learning the classifier and then use sampling/gradient descent?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In general, the paper presents a novel idea of using diffusion models to solve robotic manipulation tasks. \nThe paper contains a few interesting technical details, such as cropping the scene point cloud to focus on relevant parts, the loss function loss, the exact diffusion setup adjusted to predicting rigid transformations.\nThe reviewer appreciates the detailed appendix which clarified a lot of my initial questions.\n\n\nHowever, I think that the paper is missing central arguments why the proposed method advances the state-of-the-art:\n\n* The main weakness of the paper is the following: The data to train the models is generated in simulation using manually designed heuristics. While this in principle is sound and a common approach, in my opinion, one then has to show that the learned method exhibits capabilities that go beyond those manually designed heuristics. I currently do not see why the point clouds as obtained in this work cannot be transformed with the same heuristics as for generating the data.\n\n* How does this method extend to multi-step manipulation problems?\n\n* I suggest looking into these papers:\n[1] Yifan You, Lin Shao, Toki Migimatsu, Jeannette Bohg. \"OmniHang: Learning to Hang Arbitrary Objects using Contact Point Correspondences and Neural Collision Estimation\". ICRA 2021.\n[2] Jung-Su Ha, Danny Driess, Marc Toussaint. \"Deep Visual Constraints: Neural Implicit Models for Manipulation Planning From Visual Input\". RA-L 2022.\nI suggest to potentially adding them as a baseline (especially OmniHang). There are also other papers exploring energy-based methods to deal with multimodality.\n\n* Regarding the limitation section, the fact that there is no transfer to new tasks and that separate models for each task are trained could be added.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "JgbMhmkfqa", "reviewer": "Reviewer_SEr7", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The authors propose an iterative diffusion method for object-scene re-arrangement. For tasks such as shelving books, the authors propose using a DNN to predict a transform a given object (e.g. book) from an initial pose to a final predicted pose, and execute this behavior in an open-loop manner. Specifically, the DNN predicts a series of transforms in an iterative manner, similar to diffusion models. The approach is able to produce multimodal predictions (e.g. multiple possible locations to shelve the book) and generalize to different environments and initial setups.", "summary_of_recommendation": "Overall, the paper is well written and presents a nice, simple formulation for applying diffusion models to robotics in order to achieve multimodal predictions. Moreover, the results are exciting, showing very high precision and recall in being able to achieve these multimodal predictions. The authors additionally provide real-world demonstrations. I'd be curious how the proposed method compares to recent papers such as [41, 58].", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "* How does this compare to other related works such as Diffusion Policy [58]? It would've been nice to compare against [41] as well. Even though [41] uses language input, the authors claim that the main bonus of the proposed approach is in its multimodal predictions. Is [41] able to achieve this just as well?\n* How much do you add noise? e.g. Fig. 2b shows the book is very close to the shelf, but the initial starting pose in the supplementary demo is much farther away from the shelf.\n* What is the real-world performance quantitatively? e.g. success rate\n* How are the point clouds segmented for scene and object in the real world (line 206)?\n* Fig. 3a: Why does recall terminate at ~0.68? Is it because with a large number of K samples, we don't get 1.0 recall? (this is fine, just asking for clarity)\n* What are the main failure cases of the proposed approach?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths\n* Novelty: The paper shares a similar spirit to recent works applying diffusion methods to robot manipulation ([41, 58]). This is a nice and simple formulation for leveraging diffusion models for manipulation.\n* Notably, the main strength is to produce multimodal predictions.\n* The authors demonstrate the ability to successfully execute their approach in the real world.\n* Clarity: Overall the paper is clear.\n\nWeaknesses\n* Approach assumes that a point cloud segmentation is already provided for the scene and object (ie. the object is treated as one class, scene as another). It's not clear that this will always be easy to obtain\n* Are there better alternatives to random perturbations in the forward diffusion process?\n* Requires a manually defined offset based on prior knowledge of the task (line 198)\n* Open loop execution\n\nClarity\n* Page 2 describes a lot of motivation and overview of the pipeline. It would be useful to have associated images/figures throughout the explanation. For example:\n    * Providing examples of the \"local cropped region\" around the object in line 67\n    * What the \"noised point cloud\" looks like in line 51\n    * What a \"large perturbation\" looks like in line 56\n* The goal specification is not very clear in paragraph 1. Providing examples would help clarify things further. For example: for shelving books, is there a specific \"shelving books dataset\"? Is it one dataset per task, or can we use one dataset to learn multiple tasks? Is there a way to specify where to place the book, or is the goal to shelve a book in general? This is clarified only in the experiments section.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "Gnr7HNR91cE", "reviewer": "Reviewer_HwxF", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The proposed system, RPDiff, addresses the problem of object rearrangement in a scene by operating directly on 3D point clouds. It overcomes challenges associated with multi-modality and generalization to diverse scene layouts. The main contributions of the paper are:\n\n- Iterative Pose De-noising: RPDiff uses a diffusion model to perform iterative de-noising, gradually reversing the perturbation process applied to the object point cloud. This allows the system to generate diverse solutions for rearrangement tasks.\n\n- Local Conditioning: To support generalization to novel scene layouts, RPDiff locally encodes the scene point cloud by cropping a region near the object. This allows the model to focus on task-relevant details and ignore irrelevant global structure, improving generalization.\n\nEvaluation and Results: RPDiff is evaluated on three distinct rearrangement tasks in both simulation and the real world. It outperforms existing methods in terms of success rate and coverage.\n", "summary_of_recommendation": "The paper presents a well-structured demonstration of the work on using a diffusion model called RPDiff as a transformation prediction network in robotics. The authors draw from other domains and previous works to develop their approach. The writing is concise and easy to follow. However, there are some clarity issues, particularly regarding the pipeline and the necessity of local conditioning alongside self and cross attention mechanisms. The lack of real-world experimental analysis impairs the actual impact of the work. While the paper combines ideas from various domains, its main contribution seems to be the iterative use of the diffusion model, making it an incremental rather than groundbreaking work. Some important details are relegated to the appendix instead of being included in the main text.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "- Data preparation: For finding the goal configuration, how is the transformation computed given an initial object and scene point cloud data? There is some explanation in the Appendix, but I believe this information is important enough to be included in the main text, esp. to illustrate that there is a lot of domain knowledge incorporated into the data generation process.\n\n- How viable pre-trained representations would be for overcoming the data generation challenges? How could the data requirements be reduced?\n\n- As mentioned before, why is the additional Local Conditioning necessary while using the self and cross attention mechanisms?\n\n- The paper assumes known object states and 3D models, which may not be applicable in all real-world situations. Could you discuss possible extensions or adaptations of RPDiff to handle scenarios where object states and models are uncertain or partially known?\n\n- Could you provide more insights into the computational efficiency and real-time performance of RPDiff? \n\n- The reliance on scripted policies for data generation raises concerns about scalability. Are there any plans to explore methods for collecting or generating data in a more automated or self-supervised manner? How might this impact the performance and generalization capabilities of RPDiff?\n", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Quality:\nOverall, the quality of the paper is high. The authors provide a clear and well-structured explanation of their proposed system for object rearrangement using 3D point clouds. They present a good enough problem analysis and provide fair descriptions of the methodology, experiments, and results. It’s hard to evaluate the technical foundation, mostly due to too much preprocessing / manual manipulation of the data (most of the details are included in the appendix). I understand that this is unavoidable due to the inherent requirements of the problem setup, but at times it feels like some critical details were omitted to push the reader away from the relatively ‘trickier’ parts of the approach that might indeed hinder the application of the approach in broader scenarios. But it definitely addresses important challenges in the field of robotics learning, so including a better discussion of those ‘hardships’/extra steps needed would also inform the reader about the complexity of the problem.\n\nClarity:\nThe paper is generally well-written and organized. The introduction effectively motivates the problem of object rearrangement and provides a clear context for the research. The methodology section describes the proposed system in a detailed and coherent manner. The experimental setup and evaluation metrics are clearly defined. The figures and captions are informative and support the text. However, in some sections, the paper could benefit from additional clarifications, especially regarding the specific implementation details.\n\nOriginality:\nThe paper presents a novel approach to object rearrangement by leveraging multi-modal predictions, iterative pose de-noising, and local scene conditioning. While there have been prior works on object rearrangement and pose estimation, the combination of these techniques in the proposed RPDiff system appears to be original. The authors contribute to the field by addressing challenges associated with multi-modality and generalization in object rearrangement tasks, and they introduce a new method that surpasses existing approaches in terms of precision and coverage.\n\nSignificance:\nThe significance of the work is fair,, specifically in the domain of object manipulation and cleanup tasks. The ability to rearrange objects in scenes accurately and efficiently is crucial for many real-world applications. The proposed RPDiff system tackles key challenges in this area, including handling multi-modality, generalizing to diverse scene layouts, and producing precise and accurate predictions. The experimental evaluations demonstrate the superiority of RPDiff over existing methods, highlighting its practical value and potential impact on robotics research and applications. \n\nStrengths:\n\n- Novel approach: The proposed Relational Pose Diffusion (RPDiff) system addresses the problem of object rearrangement in a scene using 3D point clouds, allowing it to handle unknown geometries in the real world.\n\n- Multi-modal generation: RPDiff overcomes the challenge of fitting multi-modal demonstration data by using an iterative pose de-noising process, producing diverse and precise solutions for rearrangement tasks.\n\n- Local conditioning: The system incorporates local conditioning by cropping a region near the object in the scene point cloud, improving generalization and precision by focusing on task-relevant details.\n\n- Evaluation in simulation and real-world: The system is evaluated on three distinct rearrangement tasks in both simulation and the real world, demonstrating its effectiveness and applicability in different scenarios.\n\nWeaknesses: (the authors acknowledge most of those points to some degree)\n\n- Clarity: RPDiff employs a transformer architecture featuring self-attention. However, the authors state that they require an extra step called Local Conditioning, which involves cropping the scene around the object. The main text lacks clarity regarding the model's pipeline and requires further explanation. Consequently, it remains unclear why the additional Local Conditioning is necessary while using the self and cross attention mechanisms, as they should be capable of detecting relevant and irrelevant points in the scene.\n\n- Data requirements: The system relies on scripted policies in simulation to obtain demonstration data, which may limit its applicability to new tasks. \n\n- Real-world analysis: As the authors mentioned in their limitations, they had problems with switching from simulation to real-world experiments. Consequently, all their main results were obtained solely from simulations. In the Appendix, they acknowledged that they observed worse overall prediction performance in the real world, however they did not provide specific details about the performance. \n\n- Sim2real gap: As highlighted by the authors as well, there is a potential sim2real gap due to training on simulated point clouds. Incorporating sim2real transfer techniques and finetuning on real-world data could improve the system's performance in real-world scenarios. However, the paper does not go into the details about how serious this gap was, how they addressed it, and it does not extensively address techniques to bridge this gap, leaving room for further investigation.\n\n- Open-loop execution: The system executes the predicted placements in an open-loop manner, which may limit its ability to track progress and react to disturbances. As mentioned a closed-loop policy could enhance the system's performance in dynamic environments. However, how to integrate the whole approach into such a policy has not been discussed. For instance, considering the final phases of such rearrangement tasks (e.g., shelving a book between other books), the visual perception data may become less and less informative (e.g., due to occlusions).\n\n- The paper assumes known object states and 3D models (at least in training as far as I understood), which may not be realistic in some real-world scenarios. The reliance on scripted policies for data generation limits scalability.\n\nMinor comments: \n- the term multi-modality sounds like sensor modalities. I don’t know whether this usage is well-accepted within this (object rearrangement) context but if not, then a different naming might be more informative / less confusing.\n\n- Generalization: The paper should be more precise about what is meant by generalization in these scenarios, while being more clear about train and test data.\n\n- line 115: identify -> identity", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers agree that this paper addresses an important problem and that the proposed approach, while rather incremental, is sufficiently novel. While several questions about technical description and experimental evaluation have been addressed by the author rebuttal and discussion, some concerns such as about the limited transferability to real-world remain. 

The suggestions of the reviewers should be thoroughly incorporated in a camera-ready version of the paper.
Decision: Accept (Poster)