FlowBot++: Learning Generalized Articulated Objects Manipulation via Articulation Projection
Authors: Harry Zhang
Ben Eisner
David Held
Venue: CoRL 2023 Poster (2023)
Keywords: Articulated objects manipulation
representation learning

Abstract: Understanding and manipulating articulated objects, such as doors and drawers, is crucial for robots operating in human environments. We wish to develop a system that can learn to articulate novel objects with no prior interaction, after training on other articulated objects.  Previous approaches for articulated object manipulation rely on either modular methods which are brittle or end-to-end methods, which lack generalizability. This paper presents FlowBot++, a deep 3D vision-based robotic system that predicts dense per-point motion and dense articulation parameters of articulated objects to assist in downstream manipulation tasks. FlowBot++ introduces a novel per-point representation of the articulated motion and articulation parameters that are combined to produce a more accurate estimate than either method on their own. Simulated experiments on the PartNet-Mobility dataset validate the performance of our system in articulating a wide range of objects, while real-world experiments on real objects' point clouds and a Sawyer robot demonstrate the generalizability and feasibility of our system in real-world scenarios. Videos are available on our anonymized website https://sites.google.com/view/flowbotpp/home


Reviews:
[{"review_id": "pDNtU6JjAwv", "reviewer": "Reviewer_frrv", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This work proposes Articulation Projection, a per-point representation of articulated motion. Building on top of this representation, this work proposes FlowBot++, a policy for manipulating articulated objects using 3D point cloud as input.\n\nThis work presents experiments showing that after trained in simulated environments, FlowBot++ can manipulate unseen articulated objects in both simulation and real-world.", "summary_of_recommendation": "This work follows a standard pipeline of first predicting articulated motion parameters and next plan trajectories to manipulate the articulated objects. This pipeline seems to have very limited novelty.\nHowever, this work is also executed quite well and show clear improvement over the previous FlowBot3D method. It also implements a system to deploy the trained model in the real world. Therefore, I think it overall provides a valuable system for generalizable articulated object manipulation.", "technical_quality": "Excellent: Very convincing and thorough.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "Most importantly, I think this work needs to provide justifications for the proposed Articulation Projection by comparing with other motion prediction methods as I have discussed in the *review* section. Currently, the baseline comparison includes a Bullet-based simulator and manipulation policies, which cannot provide much insights about the performance of the vision module itself. Ideally, this work should show the effectiveness of Articulation Projection by including a baseline comparison where the input is a point cloud (preferably generated with a realistic depth camera simulator) with segmentation, and the output is the predicted motion axis.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths\n- This work is written very clearly.\n- This work has strong motivation and execution. It identifies that the previous state-of-the-art method, FlowBot3D, lacks the ability to plan a manipulation trajectory, and proposes a a sensible representation, Articulation Projection, to overcome this limitation.\n- This work has built a system that can be deployed in the real world.\n\nWeaknesses\n- Missing comparison with previous pose or motion prediction methods.  \n  One key contribution of this work is Articulation Projection, which seems to be a pure vision module that takes a segmented point cloud and outputs the motion axis. It seems that in principle, this module can be replaced with any other articulation motion prediction method, such as [1] and [17] discussed in related work. This work is missing a comparison with these baselines.\n- It is not clear how incorrect segmentation and noisy point clouds in real-world scenario would affect the performance of this method.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "_jRsgjJaShZ", "reviewer": "Reviewer_E3aG", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "Articulated objects are ubiquitous in human environments. Interacting safely with them will be a necessity for service robots while assisting humans. The proposed method addresses this requirement and presents a novel approach to estimating articulation properties for objects without requiring prior interactions with them. The proposed method, FlowBot++, predicts dense per-point motion and articulation parameters for a provided segmented 3D point cloud of the object. Experiments demonstrate that FlowBot++ outperforms the SoTA methods in estimating motion and articulation parameters for novel objects and qualitatively establish that the estimated parameters are reasonably accurate to allow safe interactions with these objects.", "summary_of_recommendation": "The authors study an important problem and present a novel approach for solving it. The paper is well-written and effectively communicates the key concepts. The authors motivate the problem and concisely summarize their contributions. Experiments conducted both in simulation and on real-world objects demonstrate the effectiveness of the proposed approach. However, the manuscript would benefit from further refinement.\n\n**Post-rebuttal update:**\nI appreciate the authors' responses to my questions. However, I agree with other reviewers that the task completion rate alone does not provide strong evidence in favor of the efficacy of the proposed method, and adding experiments demonstrating the effectiveness of individual components would make the paper stronger. Hence, I would like to maintain my score.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "1. All robot manipulation experiments consider tasks of opening/closing the objects to their full range of motion. Consider adding experiments for partially opening and closing these objects. \n1. Please include empirical results supporting the statement: \"Empirically, the Articulation Projection prediction is sometimes less accurate than the Articulation Flow prediction\" (lines 180-181), as it influences a critical design choice for the proposed approach.\n1. Can the proposed approach handle multiple joints? If not, how can the proposed method be extended to handle multiple joints?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors tackle an important problem and present a novel approach for addressing it. The paper is well-written and communicates the main ideas comprehensively. The authors motivate the problem and state their contributions clearly. Experiments, both in simulation and on real-world objects, provide evidence in support of the work. However, the manuscript requires additional work before I feel confident recommending the work for acceptance. Detailed feedback is as follows: \n- All robot manipulation experiments consider tasks of opening/closing the objects to their full range of motion. Consider adding experiments for partially opening and closing these objects. \n- Cite previous work in Line 123.\n- Consider adding a brief justification to support the statement \"... estimating the Articulation Flow results in more generalizable prediction than estimating the articulation parameters directly\" in lines 145-146 for completeness.\n- Please include empirical results supporting the statement: \"Empirically, the Articulation Projection prediction is sometimes less accurate than the Articulation Flow prediction\" (lines 180-181), as it influences a critical design choice for the proposed approach.\n- Add details of the FlowProjNet architecture in the manuscript or in the appendix.\n- Authors of [1] presented an updated version of the work to use screw parameters for articulation model estimation in [2], which reported significantly better results. Consider comparing with [2] for a more comprehensive comparison.\n- Are the articulation parameters estimated in the previous iteration used as a prior for prediction in the successive calls during object manipulation? If not, adding them as a prior should help to improve the performance.\n- Another interesting metric to demonstrate the effectiveness of the proposed approach would be to show the robot FT sensor data during manipulation experiments. If the predictions are accurate, experienced forces and torque should be low.\n- Update Table 1 on the website.\n- Typographical:\n  - \"how any\" Line 124\n  - \"Both of both worlds\" Line 182-183\n  - Correct the spacing between lines 201-202\n\nReferences:\n[1] Jain, Ajinkya, et al. \"Screwnet: Category-independent articulation model estimation from depth images using screw theory.\" 2021 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2021.\n[2] Jain, Ajinkya, et al. \"Distributional depth-based estimation of object articulation models.\" Conference on Robot Learning. PMLR, 2022.", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}, {"review_id": "IECmt-SM9K", "reviewer": "Reviewer_uUTN", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper introduces FlowBot++, a 3D vision-based robotic system designed to understand and manipulate articulated objects like doors and drawers. Unlike previous approaches, FlowBot++ combines per-point motion and articulation parameters to achieve more accurate predictions. The system demonstrates successful articulation of various objects in simulated experiments using the PartNet-Mobility dataset and shows generalizability in real-world scenarios with real objects and a Sawyer robot. Overall, there are two main contributions: 1) per-point 3D representation of articulated objects (integrates motion under actuation and articulation parameters).\n2) demonstrations on a robot manipulation system to manipulate novel articulated objects in a zero-shot manner.", "summary_of_recommendation": "The paper was enjoyable to read, and the authors have presented a promising approach. However, there are a few points that require clarification as mentioned above, with emphasis on providing detailed analysis on generalizability and training data. Addressing the listed points will strengthen the clarity of the paper.", "technical_quality": "Excellent: Very convincing and thorough.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "Overall, the paper was a pleasure to read. However, there are a few things which I'm hoping the authors could clarify, as summarized below (see section above for details):\n* Additional interpretation for method's failure in certain categories compared to FlowBot3D (e.g., phone, laptop, button).\n* Analyze and propose ways to mitigate failure cases, with examples from the supplementary materials.\n* Compare objects in training and testing data, specifically real-robot experiment objects resembling trained objects.\n* Evaluate method's generalizability to unseen real-world objects and its robustness to input noise.\n Further, there are some typos in the paper which should be fixed (i.e. L. 182 both --> best).", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors observe that simply using articulation flow as a representation lacks the ability to predict a full, multi-step trajectory from the initial observation, hence the need for a per-point 3D representation that integrates motion flow and articulation parameters.\n\nStrengths:\n* Significant reduction in execution time compared to FlowBot3D (17.1s vs. 1.2s)\n* Real world experiments with Sawyer robot: reliable predicted trajectory, and execution is much more smooth than FlowBot3D\n\nWeaknesses:\n* Could the authors provide an interpretation for why the proposed method fails in some categories when comparing with FlowBot3D (I.e. phone, laptop, button, etc.\n* Could the authors provide an analysis of the failure cases, and ways to mitigate? Examples of failure cases were mentioned in the supplementary, however, it would be good to include an analysis as well\n* Could the authors provide a comparison of the difference between objects in training vs. testing data? It appears that the objects used in real-robot experiments are toy versions, do they resemble closely what was seen during training?\n* Generalizability: how well does the method generalize to unseen objects in the real world? How well does the method cope with input noise?", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}, {"review_id": "CyKYemi5Lkp", "reviewer": "Reviewer_8kbD", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper presents a novel approach to generalized articulated object manipulation using the concept of articulation projection. The authors propose a novel framework that leverages deep learning techniques to enable robots to manipulate a wide range of objects with articulated parts.", "summary_of_recommendation": "It is a good work but I think most of the task here is already solved therefore I recommend a weak accept ", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "One major question for me is that the task that the robot is dealing with such as opening the oven does not really require explicit learning. We can do most of the task from a control perspective with an impedance controller. I would like the authors to comment on the advantages of the proposed scheme compare with control-based approaches.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper is very clear and follows a logical flow making it a pleasure to read. The literature review is comprehensive. The reviewer especially likes the style of splitting into different category and at end of each highlighting the difference between this and aforementioned works.\n\nPros\nThe scheme is its able to generalize to objects with different articulation structures. By leveraging articulation projection, the model can transfer the learned knowledge to novel objects, even those with distinct joint configurations. This flexibility showcases the adaptability and versatility of the proposed framework.\n\nFurthermore, the paper provides comprehensive evaluations and comparisons with existing state-of-the-art methods, demonstrating the superiority of FlowBot++ in terms of manipulation accuracy, robustness, and generalization capability. The results clearly indicate that the proposed approach outperforms other methods in challenging scenarios, where object articulation and complexity play significant roles.\n\nCons please see next section.\n", "extra_scores": {"impact": "2: Unsure, difficult to assess."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers agree that the paper is well-written and enjoyable to read. I also do not find any critical issue in the paper. The reviewer raised a few points but I believe that the authors have answered them very clearly.

Some reviewers mentioned that either the impact will be small or the novelty is limited. This weakens the lowers quality of the paper. Even though all reviewers agree that the paper can be accepted, the average recommendation level is very high in this area and the paper is only borderline acceptable.

The paper is definitely robotics-oriented with good experiments on the real robot. All reviewers agree on this point.
Decision: Accept (Poster)