Diff-LfD: Contact-aware Model-based Learning from Visual Demonstration for Robotic Manipulation via Differentiable Physics-based Simulation and Rendering
Authors: Xinghao Zhu
Jinghan Ke
Zhixuan Xu
Zhixin Sun
Bizhe Bai
Jun Lv
Qingtao Liu
Yuwei Zeng
Qi Ye
Cewu Lu
Masayoshi Tomizuka
Lin Shao
Venue: CoRL 2023 Oral (2023)
Keywords: Learning from Demonstration
model-based robot learning
differentiable physics-based simulation and rendering

Abstract: Learning from Demonstration (LfD) is an efficient technique for robots to acquire new skills through expert observation, significantly mitigating the need for laborious manual reward function design. This paper introduces a novel framework for model-based LfD in the context of robotic manipulation. Our proposed pipeline is underpinned by two primary components: self-supervised pose and shape estimation and contact sequence generation. The former utilizes differentiable rendering to estimate object poses and shapes from demonstration videos, while the latter iteratively optimizes contact points and forces using differentiable simulation, consequently effectuating object transformations. Empirical evidence demonstrates the efficacy of our LfD pipeline in acquiring manipulation actions from human demonstrations. Complementary to this, ablation studies focusing on object tracking and contact sequence inference underscore the robustness and efficiency of our approach in generating long-horizon manipulation actions, even amidst environmental noise. Validation of our results extends to real-world deployment of the proposed pipeline. Supplementary materials and videos are available on our webpage.


Reviews:
[{"review_id": "yPBq6smpii", "reviewer": "Reviewer_YF65", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper presents a framwork for model-based Learning from Demonstration (LfD) in the context of contact-rich manipulation. The proposed framework contributes to the following aspects:\n1. The usage of differentiable rendering to track and reconstruct object poses in a self-supervised manner.\n2. A hierarchical planning framework that employs differentiable simulations to generate long-horizon contact actions.\n3. A sim-to-real neural policy that is trained by domain randomization.\n\nThe authors demonstrate the effectiveness of the method on several manipulation tasks, including pulling, pushing and in-hand reorientation. Results in both simulation and real world look very promising.", "summary_of_recommendation": "The paper proposes a novel framework with carefully designed perception and planning component, to solve a very challenging task, learning contact-rich manipulation from video.  The presented results are very impressive. Hence, I recommend accepting the paper.\n\n---\nAftering rebuttal, my rating remains unchanged. I agree that there isn't much novelty in the individual sub-system, but I appreciate the system-level contribution of putting all together. Reconstruction by differentiable rendering has been a hype in the vision community for a while, but it's interesting to see the reconstructed object is actually good enough for manipulation. I believe the community can be benefited from reading this paper.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "- With the proposed self-supervised reconstruction method, we're able to recover the full shape of the object. However, I'm wondering how you obtain the physical parameters, such as friction and mass.\n- Why do you need to recover the texture of the object, since it's not helpful for manipulation task. Can't you just render the point cloud and recover the shape?\n- In algo 1, line 5, how does the SelectNode work? Do you adopt any heuristic function to make the planning more efficient?\n- Questions regarding the closed-loop policy:\n    - Does it take both current pose and goal pose of the object as input? How do you obtain the pose?\n    - How do you obtain the contact configurations? Do you also apply the high-level contact planning at test time, or do you assume it's known?\n\n- Typos: \n    - Line 52: prepossess -> pre-process", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths\n- It proposes a very comprehensive system for learning from video on the challenging manipulation task, such as in-hand reorientation. It's really dense, since almost every step is tough, i.e., estimating full object shape from video, plan and optimize contact actions. It's really impressive that the authors are able to put them together and make it work.\n- The proposed hierarchical policy is very interesting. It integrates high-level contact sequence planning and low-level contact-point localization and contact-force optimization using differentiable simulation. The results look very promising.\n\nWeaknesses\n- Although in the related works section the authors cite a lot of works, the positioning and contribution of the paper isn't very clear to me. For example, it cites many papers under \"Differentiable Simulation and Rendering in Robotic Manipulation\", but it doesn't state clearly that comparing to prior works, what's the most significant difference. Instead of citing all papers at the same time, the author should select the most relevant ones and explain the key differences.  My feeling is that object tracking and reconstruction through differentiable rendering is a well-studied field. If the proposed pipeline is built off prior works, the author should cite it in the method section. If the proposed method contains new insights, the authors should also make it clear. Same thing for using differentable simulation for trajectory optimization.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "qjbC_Cl4hhH", "reviewer": "Reviewer_Qw95", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper introduces a visual learning from demonstrations (LfD) framework that first reconstructs and tracks a rigid object being manipulated in an RGB video. Given the shape and motion of the object, a set of contact points and wrenches are found to realize the reference motion in a simulator. Given the planned contact trajectory, the joint torques are recovered via the robot's kinematic Jacobian and a neural network policy is trained from perturbed data to improve the sim2real transfer. Real robot experiments on an Allegro hand are performed to demonstrate the pipeline on convex objects.", "summary_of_recommendation": "The proposed visual LfD pipeline is novel and combined various techniques from differentiable rendering and simulation to tackle the problem of learning from manipulation videos. The results are overall convincing, albeit the considered objects are simple and the considered baselines are not state of the art. The clarity of the description of the technical details (Sec. 2.1 and 2.2.) needs to be improved.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "How is the camera pose resolved? It appears to be set to $\\lbrace\\hat{\\mathcal{P}}^*_t\\rbrace$ (line 89), but it is mentioned earlier that these are the *object* poses.\n\nThe approach from Sec. 2.2 is difficult to follow. It is unclear to me why the localization of contact points involves the \"execution\" of the pose transition wrench(?) when the optimization of the contact wrench actually happens in another function (OptWrench). More details are needed on the possible combinations of contact points: do these point candidate correspond to the mesh vertices or how are they generated? Why is a wrench computed during the point localization step when the contact filtering (for which details are missing) appears to happen solely based on whether the contact wrench direction (so PD gains do not matter?) is inside the contact friction cone?\n\nWhat are the \"stationary constraints\" mentioned in line 126-127?\n\nIt is mentioned that the analytical contact wrench from ContactLoc is used to initialize the wrench optimization in OptWrench (line 158-159). In Algorithm 1 the first iteration starts with OptWrench however without previously running ContactLoc, so what initialization is used here?\n\nAre the inertia of the object considered in the OptWrench step? How are the inertial properties estimated from the observed object?\n\nThe objects reconstructed in the vision-only experiments (Figure 2 in supplemental material, or the videos from the website) are significantly more complex than the box or sphere experiments in the motion generation experiments. I wonder what the limitations are to apply this pipeline to such geometry or whether there is a preference to be limited to simple convex shapes?\n\nAs mentioned under \"weaknesses\" of this review, it is advisable to improve the performance comparison from Figure 4 to include a baseline that is actually state-of-the-art in manipulation behavior generation, such as [PGDM](https://pregrasps.github.io/), or a trajectory optimization approach that leverages the simulator gradients (e.g. iLQR, CIO) or not (e.g. MPPI).", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "## Strengths\n\nThe approach to involve a simulator in the visual LfD methodology is very interesting. The learning of the gemetric and visual shape without requiring the camera poses appears to work quite well, which is demonstrated through extensive experiments in the supplemental material (although some problems remain when the object is occluded throughout the video). The authors perform extensive experiments to demonstrate many parts of the proposed pipeline.\n\n## Weaknesses\n\nThe object reconstruction appears to have issues when parts of the object are occluded (a common problem in visual footage of manipulation behaviors), as can be seen e.g. in the coffee cup in Figure 2 of the supplemental material. The proposed diffusion model does not seem to help; the approach to combine imagined viewpoints and real images of the object for training the SDF is not explained further in the supplemental material, although it is mentioned in the paper (lines 99-100).\n\nThe baselines in the experiments are quite weak, particularly RRT search over the joint configuration space is too inefficient to be considered state of the art. A more capable baseline, such as [PGDM](https://pregrasps.github.io/) that also generates manipulation behaviors given the desired motion of an object, would be much better suited to compare the performance of the proposed framework.\n\nThe neural network policy seems to require retraining (including the generation of a training dataset) for every new object and reference video, which limits the real-time applicability to manipulation behavior generation.\n\nThe generated behaviors only imitate the motion of the object and completely discard the motion of the human hand manipulating the object. Here it would have been interesting to learn from the way the hand moves more efficient/natural-looking robot hand motions than the one-finger-at-a-time behavior that the current approach generates.\n\n## Minor issues\n\n\"Table 1\" in line 242 should refer to Figure 3", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "DxAoAm4_iF", "reviewer": "Reviewer_jZUA", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposes a pipeline for in-hand manipulation tasks that involves first using a differentiable renderer to reconstruct the object shape and pose, then using a gradient-based contact planner (combined with search over contact configurations) to find sequences of actions that move the object to a desired pose. The planner is then distilled into a neural network control policy that is demonstrated on hardware experiments.", "summary_of_recommendation": "Each of the subsystems proposed in this paper seem fairly incremental (e.g. differentiable rendering for object pose reconstruction, gradient-based planning of contact wrenches). Certain elements are interesting (combining gradient-based planning with search) but are not sufficiently explored in the paper and contain a number of heuristics that are not well justified via ablations (e.g. the random chance of transitioning contacts). If the paper were revised to clarify the novelty of each subsystem (or the novelty in combining all these subsystems, which is equally valid), as well as answering my other questions above, I would consider upgrading my recommendation.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Fair: Major revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "In addition to issues identified above, please address the following:\n\n1. How is the diffusion model in section 2.1 trained?\n2. What is the quality of the meshes that result from the differentiable rendering reconstruction process? Are the normals accurate? I would imagine that that would be important for accurate simulation.\n3. You mention that fine-tuning your neural network with REINFORCE was helpful. Did you compare with a neural network that was trained end-to-end with RL?\n4. How well will this approach generalize to more challenging manipulation tasks? For example, many manipulation tasks (e.g. pouring, insertion) are not simply about minimizing the pose error.\n5. How long does your approach take to train on new objects? How much time is spent on object reconstruction, planning, and fine-tuning the neural network?\n6. How are the contact points chosen? Are they manually specified for each object, or are they somehow learned from the reconstruction?\n\n**Update after discussion**\n\nThis paper seems like a well-implemented systems paper; if CoRL had a track for systems papers (like RSS), then this would be a clear accept. As it stands, it is still unclear to me where the contribution is. The authors attempted to clarify their contribution in their rebuttal, but seem to have just restated the contributions provided in their paper without clarifying the specific advances over prior work. For example, differentiable rendering has been used for pose estimation in prior works; what is new about the differentiable rendering sub-system in this paper? If one of the other reviewers has a clear sense of the contribution, then I will defer to them, but as it stands I don't feel confident increasing my score.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### Strengths\n\n- This work clearly required a substantial amount of engineering effort to integrate multiple systems (differentiable rendering for object reconstruction, planning contact locations, etc.). \n- The paper includes hardware demonstrations.\n\n### Weaknesses\n\n- At a number of points, the paper is unclear and defers important details to the supplementary materials. The paper should be able to clearly explain the essence of your approach without reference to the supplementary materials.\n- The reported experiments are very simple, mostly block pushing and in-hand reorientation of cubes and spheres.\n\n### Major comments\n\n1. Section 2.1 is unclear. Are you working with SDFs or with explicit meshes, or both? Consider revising this section to improve clarity.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "8QLHVXsehW", "reviewer": "Reviewer_kobW", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "In this work, the authors propose a method to reconstruct the geometry of an object from RGB demonstrations, which they use in a downstream manipulation policy which uses model-based control and planning to plan contact-rich manipulation. The authors demonstrate that their reconstruction approach is effective in simulated and real experiments, and that their planning approach can successfully manipulate objects in several real-world experiments.", "summary_of_recommendation": "Overall, I am impressed with the quality of the system that was constructed and demonstrated: going from visual demonstrations directly to contact-rich control is not an easy feat. However, I don’t think this paper is appropriate for this venue - the typical “learning” components which the community is interested in are absent, and this approach does not demonstrate abilities/behavior which the community will find relevant (i.e. generalization, flexibility, etc.). The community might be interested in the subsystems themselves at a technical level, but I feel that this work is a better fit for a pure robotics venue.\n\n---\nUpdate: based on the rebuttal and discussion with other reviewers, I now believe that this is more appropriate for the learning community than I initially believed. So I have updated from Strong Reject to Weak Accept. See comment for details.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "Can you elaborate on how you would actually use this system at test-time? In terms of specifying inputs/outputs after watching a demo.\n\nCan you elaborate more on how you align novel objects to existing demonstrations?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "# Originality\n\nThis paper proposes a reasonable system for watching a demonstration of an object being manipulated, estimating its geometry, and using that estimated geometry and a model-based planner/controller to control a robot to reorient the same object. These two sub-components are quite well-studied areas (3D reconstruction from motion, and contact-rich in-hand manipulation), so I wouldn’t consider the modules themselves novel (as I’ve seen differentiable rendering for reconstruction and contact-aware planning in prior work). However, getting a full manipulation system to work based on these two components is something I haven’t really seen before, and it’s a good system application of advances in visual reconstruction.\n\n# Quality\n\nThe paper is high quality in terms of execution. The experiments seem thorough, the description of the algorithms is complete and comprehensive (with a few minor exceptions, highlighted in questions).\n\nHowever, several of the design decisions seemed somewhat arbitrary:\nThe multi-step reconstruction procedure seems quite empirical, but no ablations were really provided for design choices. Additionally, the baseline method provided is not very strong - there are many other competitive algorithms for this sort of reconstruction - even with differentiable rendering, like the entire NeRF lineage of work - which isn’t compared against. That’s okay: but there isn’t much empirical justification for the design of their method.\nFor the MLP trained to approximate the planning process, why was REINFORCE used instead of supervised learning? Especially if the model itself is differentiable.\n\n# Clarity\n\nThe paper is reasonably well-written. However several aspects were unclear:\nHow is mass estimated / taken into account for contact planning?\nWhat is the alignment procedure for novel objects? Details were not clear in the supplement.\nWhy was diffusion necessary?\nLine 99: how are synthetic views used? This was especially unclear and the supplement did not clear things up.\n\n# Significance\n\nThe technical aspects of this paper are interesting (particularly the test-time optimization), but I suppose my biggest question is whether this method’s main contribution (reconstruction) has benefits compared to other structure-from-motion work? Each of the two major components seem related to existing work.\n\n# Relevance\n\nWhile I believe that the paper is high-quality, I don’t believe that this paper is a good fit for this venue. The two primary components of the method are a visual estimation pipeline (which uses a differentiable renderer + optimization, and incorporates no learning components), and a model-based planning and control framework which is purely physics-based. Because these components rely on their quite rigid assumptions to succeed, in my mind there is not a clear path to extend this framework to diverse classes of objects with different material properties, generalize across demonstrations, or to manipulate tasks with dynamics which are much harder to model explicitly. I suppose in principle you could replace each of these modules with a learned version which can represent their functions more flexibly, but that would be an entirely different paper.\n\nAdditionally - and I’m not totally sure about this point - but it feels a bit strange to call this method “Learning from Demonstrations”. Typically, that label is used when a policy itself is learned (could be a visual policy); but as far as I can tell, there is no learning at the policy level (except for a minor policy learning component for real-time execution) - it’s purely analytical. There’s something of an argument to be made that the visual component is learning from a demonstration, although one might quibble over whether optimizing a representation for a specific example is learning.\n\n# Limitations\nThe biggest unaddressed limitation is generalization to novel objects with different geometry at test time. The authors attempt to address this earlier in the text by saying that you can just align that object to a given demonstration, but provide no details on how to do this (and I can think of many scenarios where such an alignment wouldn’t work).", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces a pipeline for learning from demonstration that incorporates a number of elements, self-supervised pose and shape estimation, and contact sequence generation, building on differentiable rendering, with contact point and force optimisation. Reviewers initially found this a polarising paper, with two leaning to accept and two in favour of rejecting. The primary concerns were around the system's level nature of this paper and potential fit for CoRL. Reviewers felt that the overall pipeline made this an impressive systems paper, with convincing results, but that subsystems are relatively standard components, and that there is limited theoretical novelty or contribution to robot learning in these individual subsystems. There were also concerns that the policy is generated via optimisation/planning, not learned.

The authors provided a comprehensive and detailed rebuttal, introducing a number of additional experiments. The authors argued that the framework as a whole is novel, and that inference of the state and contact points so as to optimise a policy with respect to a simulation to reproduce these is still learning from demonstration. Reviewers relt that the system and additional experiments helped motivated the fit for CoRL.

I do not think that there is no place for systems papers at CoRL (historically we have given awards to the best of these). Although the idea has been broadly spoken about as a natural route forward, as mentioned by reviewers, we have not yet seen many differentiable rendering and simulation based planning pipelines working on real world manipulation tasks like these yet, so think that this is a contribution that warrants publication. I recommend that the authors update their paper to incorporate their rebuttal responses and additional experiments, which have undoubtedly strengthened the paper.
Decision: Accept (Oral)