Rethinking Optimization with Differentiable Simulation from a Global Perspective
Authors: Rika Antonova
Jingyun Yang
Krishna Murthy Jatavallabhula
Jeannette Bohg
Venue: CoRL 2022 Oral (2022)
Keywords: Differentiable simulation
Global optimization
Deformable Objects

Abstract: Differentiable simulation is a promising toolkit for fast gradient-based policy optimization and system identification. However, existing approaches to differentiable simulation have largely tackled scenarios where obtaining smooth gradients has been relatively easy, such as systems with mostly smooth dynamics. In this work, we study the challenges that differentiable simulation presents when it is not feasible to expect that a single descent reaches a global optimum, which is often a problem in contact-rich scenarios. We analyze the optimization landscapes of diverse scenarios that contain both rigid bodies and deformable objects. In dynamic environments with highly deformable objects and fluids, differentiable simulators produce rugged landscapes with nonetheless useful gradients in some parts of the space. We propose a method that combines Bayesian optimization with semi-local 'leaps' to obtain a global search method that can use gradients effectively, while also maintaining robust performance in regions with noisy gradients. We show that our approach outperforms several gradient-based and gradient-free baselines on an extensive set of experiments in simulation, and also validate the method using experiments with a real robot and deformables.


Reviews:
[{"review_id": "opAPURJpJpT", "reviewer": "Reviewer_LcC1", "summary": "The paper proposes an algorithm that combines bayesian optimization with evolutionary methods by using local \"leaps\" in the search space. For gradients, the paper uses differentiable simulators and contact rich and deformable scenarios where the optimization landscape contains many local minima.  The proposed  method does gradent free search as CMA-ES with a population mean and covariance, but uses gradient descent for updating the distribution's mean. The authors apply the method to multiple environments in 4 different differentiable simulators. The environments involve rigid bodies, fluids and deformable objects. The results show that BO-leap performs better than both CMA-ES and Bayesian Optimization and the authors illustrate the gradient field and why other algorithms do not perform well. Last, the authors apply the method to a real robot for a manipulation of a deformable cloth, but the method is used for finding \"simulation parameters\" to match the real robot's behavior.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper proposes a method that combines strengths of 2 different methods. The authors do a great job at explaning the weaknesses of both methods with examples gradient fields for existing problems. I like the diversity of the tasks, environments and simulators that the authors test their algorithm on. The examples very clearly show when the proposed algorithm would succeed while others fail. The illustrations of the results are very well presented and described.\n\nI'm slightly suprised by the results on Figure 2 (left side). I would expect CMA-ES to achieve better results with the right hyperparameters or initialization.\n\nThe application to the real hardware is actually to optimize simulation parameters to match the real robot behavior. So it is not really optimizing the behavior as it's done in simulation. We also cannot call it optimization of the simulator as it optimizes for only one real world example, so it's likely to fail at different poses. Nevertheless it shows one way to use the algorithm with a real robot, so I think it's ok. I would love to see the algorithm used to optimize a policy that is deployed to the robot. Also, I'm very curious about the potential gains on the learning curve (i.e. number of evaluations) for these tasks.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper proposes a new algorithm (to the best of my knowledge) and showcase many different type of problems where the algorithm performs better than vanilla CMA-ES or BO. The paper is overall great and the simulation results are also great, but I think that the hardware evaluation could be slightly different (i.e. optimization of a policy) or if it is tuning of a simulation, it could be based on hundreds of real robot experiments and see if the optimized simulator matches the behavior in a holdout set. Otherwise, the paper is very nice in my opinion.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "issues": "Hardware experiments can be improved (please see above).", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {}}, {"review_id": "eQNXxIELD5C", "reviewer": "Reviewer_SNTh", "summary": "Using differentiable simulators for policy optimization and system identification becomes highly non-convex for 'soft' or contact-rich settings. \nThis work proposes combining gradient-based optimization with black-box search methods to help overcome the challenges of this non-convexity. This approach is evaluted on a diverse range of simulated settings, and validated on a real-world set up. ", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I was generally happy with this submission and it is clear that it is a work of quality. Perhaps there is too much content to comforably sit within the page limit, but that was not a major issue expect that the figure and its labels are quite small to read. \n\nIt is no a controversial idea that combining Bayesian optimization and CMA-ES produces a powerful global optimizer. Although I did find some similar, uncited work (see Issues). \n\nLooking at the main text supplementary, I couldn't see results comparing the execution time of the methods considered. It would be good to see these numbers in order to assess performance vs wall-clock time, not just iterations.\n\nRegarding Bayesian optimization (line 88 -- 98), it is does not strictly use non-parametric models, just Bayesian ones. Parametric GPs such as random features approximation and neural networks features have also been used for scalability. Moreover, the 'Var' on line 92 looks like it is in math mode, not text mode.   \n\nAs a final, somewhat nitpicky comment, I personally dislike 'Rethinking' in a paper title. Most research papers involve rethinking something, so it's a rather unnecessary term. I think a title like 'Global Search in Differentiable Simulatiors with Bayesian Optimization' could be better alternative. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I was generally happy with this paper, its contribution, presentation and experiments. 10 seeds is enough for me personally. The codebase also looks good quality. I encourage the authors for a timely release so it can be adopted by the community.  ", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "Regarding the originality of this idea, I found \nSample and time efficient policy learning with cma-es and bayesian optimisation, LK Le Goff et al. ALife 2020\nwhich is somewhat similar. \nI would like the authors to comment on the differences between this approach and theres, and work in a reference into the main text. "}}, {"review_id": "MNfjJ5gSseJ", "reviewer": "Reviewer_LRvk", "summary": "In this work, the authors focus on the challenges of using differentiable simulation for learning optimal policies in tasks involving nonlinear dynamics, such as in contact-rich tasks and those involving deformable objects. The authors introduce a suite of benchmarking tasks to analyze the performance of various state-of-the-art differentiable physics simulators and gradient-based policy optimization methods. Through their study, the authors find that, often in such scenarios, the optimization landscape contains many discontinuities and spurious local optima, causing the gradient-based methods to struggle in finding globally optimal policies. To address this, the authors propose a novel Bayesian optimization scheme that uses semi-local ‘leaps’ to obtain a global search method that can use gradients effectively while being robust to noisy gradients. The authors evaluated the proposed scheme in a suite of experiments, where it outperformed the baseline methods.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The authors study challenges in using the differentiable simulation to find optimal policies for complex tasks and identify plausible reasons behind them. Understanding these bottlenecks will help the community to build better physics simulators and policy learning algorithms that can help the robots to perform complex manipulation tasks in the future. \n- The figures included in the manuscript are rather well-made and illustrate the ideas discussed comprehensively. Figure 1, illustrating the BO-Leap algorithm, describes the algorithm in a succint yet effective fashion to the readers. Figures visualizing optimization landscapes (Fig. 3, 4, 5, and 7) help the readers to better appreciate the challenges faced by the policy optimization methods in such tasks. \n- I liked that the authors also briefly described each step of Algorithm 1.  \n- Table 1 summarizing differentiable simulation frameworks, their type, supported models, and environments is quite useful to the readers to understand the capabilities of the available state-of-the-art physics simulators.\n\nWeakness:\n- A comparison of the proposed method, BO-Leap, with other standard policy optimization methods such as PPO, SAC, etc., should also be included.\n- The current manuscript requires some reorganization and updates for better readability.\n- Real-world tasks of a robot manipulating deformable objects using the policy learned with BO-Leap would demonstrate the effectiveness of the proposed policy optimization scheme better.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The authors study an important problem of understanding the challenges in using the differentiable simulation to find optimal policies for complex tasks. The manuscript presents the core ideas successfully but requires some further work. The authors must address a few issues before I can feel confident in recommending the manuscript to be accepted at the conference. If the authors address the raised concerns in their rebuttal satisfactorily, I would be happy to recommend the manuscript for publication.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {"issues": "- Rephrase the third paragraph (Lines 36-47) to make the contributions of the work clearer (same as listed in the video). The sections of the paper (sections 3 and 4) can also be rearranged to follow this list, which should enhance the overall readability of the manuscript.\nSentences in Lines 56-59, starting with “Most modern simulators…” require rephrasing for clarity.\n- Add some exemplary citations of each of the families of global optimization methods (Lines 66-70).\n- Cite the relevant work to support the claim about CMA-ES: “This method is technically not fully global — while it can overcome 86 shallow local optima, it can get stuck in deeper local optima”.\n- In section 4 para 1, please cite which previous section is being referred to in Line 148. Also, please cite the relevant work that considers contact-rich tasks in the following line (Line 149).\n- While describing each of the experiments in section 4, consider explaining the dimensionality of the optimization problem in terms of the DOF considered for the task. For example, for the 3-link cartpole task, say that the optimization problem is 400-dimensional (cart velocity and joint torques (1+3) * 100 timesteps).\n- In section 5, please include in the text that CMA-ES performs better than BO-Leap in the table and writer tasks and provide a brief discussion on why that may be the case.\n- Including a real-world task of a robot manipulating deformable objects using the policy learned with BO-Leap and comparing it with other state-of-the-art methods ( PPO, SAC, [1], [2]) would demonstrate the effectiveness of the proposed policy optimization scheme better.\n\n \nReferences:\n- [1] Hang et al.. \"Modeling, learning, perception, and control methods for deformable object manipulation.\" Science Robotics 2021.\n- [2] Zeng, et al. \"Transporter networks: Rearranging the visual world for robotic manipulation.\"  CoRL 2020\n"}}, {"review_id": "4_tCybBQhnr", "reviewer": "Reviewer_jekJ", "summary": "Differentiable simulators provide gradients to help system identification and trajectory optimization. Most of the literature on system identification and trajectory optimization uses simple differential simulators (i.e., rigid body), since deformable objects, fluids, strings, and contact-rich environments have non-differentiable areas where gradients are particularly non-smooth.\n\nThis paper compares several differentiable simulators (Nimble, Warp, DiffTaichi, Plasticine Lab) that are based on different simulation techniques (mesh-based, particle-based, and a mix of the two) on a large set of challenging environments for both trajectory optimization and system identification. \n\nFurthermore, the authors propose a novel optimization technique for this non-smooth gradient landscape. Such a technique is composed of a global search implemented with a non-parametric Bayesian technique, a semi-local search implemented with CMA-ES, and a local search via gradient descent (RandDescent).", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Contribution\n=========\n\nThe contribution is twofold: 1) providing a new analysis of differentiable simulators in contact-rich environments and in presence of fluids/deformable objects and 2) proposing a novel optimization technique that utilizes both gradient and derivative-free techniques.\n\nThe contribution, at a high level, is explained very well. The motivation for the work is clear. The placement of the paper in the relevant literature is also clear. \n\nMethod\n======\n\nAt a high level, the method is also clear. However, there are a few important concepts missing.\n\nThe authors never formalized or described the trajectory optimization problem and the system identification. This makes the empirical analysis a bit unclear to me. As far as I understood, the trajectory optimization aims to solve a problem similar to the following:\n\n$$\n\\min_{a_1, ..., a_T} || s_T - g || \\quad \\\\\n\\text{s.t.} s_0 = \\overline{s}_0, \\quad \\\\\ns\\_{t+1} = f(s_t, a_t)\n$$\n\nwhere $f$ is the simulator. One could use $\\nabla_a f(s, a, \\omega)$ to solve the problem with gradient descent (or a more complex algorithm like the one described by the authors). I think that will be very beneficial to spend a few lines introducing the problem.\n\nThe system identification part is a bit more clear to me, I guess that could be framed as\n\n$$\n\\min_\\omega \\mathbb{E}\\left[\\left(f(s_t, a_t, \\omega) - s_{t+1}\\right)^2\\right]\n$$\n\nand $\\nabla_\\omega f(s, a, \\omega)$ can help solve the problem.\n\nIn my understanding, this simple ``problem statement'' could be much more useful than the Algorithm~1, which I find honestly confusing.\n\nI agree, instead, with the current high-level description of Bayesian optimization and CMA-ES.\n\nQuestion\n----------\n\nWould the general framework of MPC help to recover bad gradients? I.e., if we take a step in the opposite direction, we could still correct the trajectory in the receding horizon... \n\nExperiments\n=========\n\nI am very impressed by the experimental section. It is true that the problem definition that I proposed above would greatly help the reading (at least I think so); but the authors did a great job in introducing the simulators, the environment, and the results. \n\nIt is possible to see that the algorithm introduced by the authors exhibits often higher performance (lower cost) in the environment tested. The authors visualize the optimization landscape. I am very impressed by the algorithm being able to solve these problems which are often (relatively) high-dimensional and rich in highly non-linear dynamics. \n\nThe real robot experiment (cloth) used for system identification is also very interesting and challenging. \n\nThe limitations section is of great value, particularly highlighting the large discontinuities of the gradients of Pinball, Swing, and Flip, preventing a correct optimization. This shows that we still need more accurate gradient estimation for highly discontinuous dynamics. Nevertheless, the potential for robotic application of differential simulators is certainly high. \n\n\nDetails\n=====\n\nI think that the term ``policy optimization'' is slightly misused. I'd call policy optimization, methods that improve the parameters of a policy. \nThe policy, in this case, would be a parametric model that outputs a sequence of actions given a starting state. \n\nI'd call instead ``trajectory optimization'' the process of finding the optimal sequence given a state. \n\nNotice that the main difference is that we can potentially run policy optimization only once, and then re-use our policy for different states, while trajectory optimization needs to be re-run for any new starting state.\n\n\nThe equation in line 89 is very obscure to me: $f(x): f(x^*) = \\min_x f(x)$. What does this notation mean??\n\n \n\n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I highly recommend this paper for this conference.\n\nThe paper deeply analyzes differentiable simulators and proposes a novel optimization framework to use for both trajectory optimization and system identification. \n\nThe analysis is very useful as differential simulators are enjoying increasing popularity in the community due to their potential. Until now, most of the literature was focused on rigid-body dynamics, but this paper analyses challenging scenarios with deformable objects (ropes, cloth), liquid (sugar), and rich contacts between the objects. \n\nI believe that this analysis can be useful for simulator designers, and to understand the current potential and limitations for system identification, trajectory optimization, and policy optimization. ", "technical_quality": "Excellent: Very convincing and thorough.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {"issues": "Missing information\n----------------------\n\nMy only issue is the lack of a proper definition of the problem:\n\n - which loss are we optimizing? how is framed the optimization problem?\n\nWriting down a nice problem statement would help the reader understand better the paper. \n\nAvoidable details\n-------------------\n\nSome details, instead, are in my opinion unnecessary in the main paper. Algorithm 1 is poorly introduced, and I believe it would require too much space to introduce it properly. Why don't delegate it to an appendix? The explanation in Section 3 is sufficiently clear to me. \n\n\nI strongly recommend the authors consider these suggestions. "}}]

Author rebuttal:

Decision: Accept (oral)