Venue: CoRL 2022 Conference

Reviews:

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: In this work, the authors focus on the challenges of using differentiable simulation for learning optimal policies in tasks involving nonlinear dynamics, such as in contact-rich tasks and those involving deformable objects. The authors introduce a suite of benchmarking tasks to analyze the performance of various state-of-the-art differentiable physics simulators and gradient-based policy optimization methods. Through their study, the authors find that, often in such scenarios, the optimization landscape contains many discontinuities and spurious local optima, causing the gradient-based methods to struggle in finding globally optimal policies. To address this, the authors propose a novel Bayesian optimization scheme that uses semi-local ‘leaps’ to obtain a global search method that can use gradients effectively while being robust to noisy gradients. The authors evaluated the proposed scheme in a suite of experiments, where it outperformed the baseline methods.
strengths_and_weaknesses: Strengths:
- The authors study challenges in using the differentiable simulation to find optimal policies for complex tasks and identify plausible reasons behind them. Understanding these bottlenecks will help the community to build better physics simulators and policy learning algorithms that can help the robots to perform complex manipulation tasks in the future. 
- The figures included in the manuscript are rather well-made and illustrate the ideas discussed comprehensively. Figure 1, illustrating the BO-Leap algorithm, describes the algorithm in a succint yet effective fashion to the readers. Figures visualizing optimization landscapes (Fig. 3, 4, 5, and 7) help the readers to better appreciate the challenges faced by the policy optimization methods in such tasks. 
- I liked that the authors also briefly described each step of Algorithm 1.  
- Table 1 summarizing differentiable simulation frameworks, their type, supported models, and environments is quite useful to the readers to understand the capabilities of the available state-of-the-art physics simulators.

Weakness:
- A comparison of the proposed method, BO-Leap, with other standard policy optimization methods such as PPO, SAC, etc., should also be included.
- The current manuscript requires some reorganization and updates for better readability.
- Real-world tasks of a robot manipulating deformable objects using the policy learned with BO-Leap would demonstrate the effectiveness of the proposed policy optimization scheme better.

robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: The authors study an important problem of understanding the challenges in using the differentiable simulation to find optimal policies for complex tasks. The manuscript presents the core ideas successfully but requires some further work. The authors must address a few issues before I can feel confident in recommending the manuscript to be accepted at the conference. If the authors address the raised concerns in their rebuttal satisfactorily, I would be happy to recommend the manuscript for publication.
issues: - Rephrase the third paragraph (Lines 36-47) to make the contributions of the work clearer (same as listed in the video). The sections of the paper (sections 3 and 4) can also be rearranged to follow this list, which should enhance the overall readability of the manuscript.
Sentences in Lines 56-59, starting with “Most modern simulators…” require rephrasing for clarity.
- Add some exemplary citations of each of the families of global optimization methods (Lines 66-70).
- Cite the relevant work to support the claim about CMA-ES: “This method is technically not fully global — while it can overcome 86 shallow local optima, it can get stuck in deeper local optima”.
- In section 4 para 1, please cite which previous section is being referred to in Line 148. Also, please cite the relevant work that considers contact-rich tasks in the following line (Line 149).
- While describing each of the experiments in section 4, consider explaining the dimensionality of the optimization problem in terms of the DOF considered for the task. For example, for the 3-link cartpole task, say that the optimization problem is 400-dimensional (cart velocity and joint torques (1+3) * 100 timesteps).
- In section 5, please include in the text that CMA-ES performs better than BO-Leap in the table and writer tasks and provide a brief discussion on why that may be the case.
- Including a real-world task of a robot manipulating deformable objects using the policy learned with BO-Leap and comparing it with other state-of-the-art methods ( PPO, SAC, [1], [2]) would demonstrate the effectiveness of the proposed policy optimization scheme better.

 
References:
- [1] Hang et al.. "Modeling, learning, perception, and control methods for deformable object manipulation." Science Robotics 2021.
- [2] Zeng, et al. "Transporter networks: Rearranging the visual world for robotic manipulation."  CoRL 2020
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper proposes an algorithm that combines bayesian optimization with evolutionary methods by using local "leaps" in the search space. For gradients, the paper uses differentiable simulators and contact rich and deformable scenarios where the optimization landscape contains many local minima.  The proposed  method does gradent free search as CMA-ES with a population mean and covariance, but uses gradient descent for updating the distribution's mean. The authors apply the method to multiple environments in 4 different differentiable simulators. The environments involve rigid bodies, fluids and deformable objects. The results show that BO-leap performs better than both CMA-ES and Bayesian Optimization and the authors illustrate the gradient field and why other algorithms do not perform well. Last, the authors apply the method to a real robot for a manipulation of a deformable cloth, but the method is used for finding "simulation parameters" to match the real robot's behavior.
strengths_and_weaknesses: The paper proposes a method that combines strengths of 2 different methods. The authors do a great job at explaning the weaknesses of both methods with examples gradient fields for existing problems. I like the diversity of the tasks, environments and simulators that the authors test their algorithm on. The examples very clearly show when the proposed algorithm would succeed while others fail. The illustrations of the results are very well presented and described.

I'm slightly suprised by the results on Figure 2 (left side). I would expect CMA-ES to achieve better results with the right hyperparameters or initialization.

The application to the real hardware is actually to optimize simulation parameters to match the real robot behavior. So it is not really optimizing the behavior as it's done in simulation. We also cannot call it optimization of the simulator as it optimizes for only one real world example, so it's likely to fail at different poses. Nevertheless it shows one way to use the algorithm with a real robot, so I think it's ok. I would love to see the algorithm used to optimize a policy that is deployed to the robot. Also, I'm very curious about the potential gains on the learning curve (i.e. number of evaluations) for these tasks.
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: The paper proposes a new algorithm (to the best of my knowledge) and showcase many different type of problems where the algorithm performs better than vanilla CMA-ES or BO. The paper is overall great and the simulation results are also great, but I think that the hardware evaluation could be slightly different (i.e. optimization of a policy) or if it is tuning of a simulation, it could be based on hundreds of real robot experiments and see if the optimized simulator matches the behavior in a holdout set. Otherwise, the paper is very nice in my opinion.
issues: Hardware experiments can be improved (please see above).
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: Using differentiable simulators for policy optimization and system identification becomes highly non-convex for 'soft' or contact-rich settings. 
This work proposes combining gradient-based optimization with black-box search methods to help overcome the challenges of this non-convexity. This approach is evaluted on a diverse range of simulated settings, and validated on a real-world set up. 
strengths_and_weaknesses: I was generally happy with this submission and it is clear that it is a work of quality. Perhaps there is too much content to comforably sit within the page limit, but that was not a major issue expect that the figure and its labels are quite small to read. 

It is no a controversial idea that combining Bayesian optimization and CMA-ES produces a powerful global optimizer. Although I did find some similar, uncited work (see Issues). 

Looking at the main text supplementary, I couldn't see results comparing the execution time of the methods considered. It would be good to see these numbers in order to assess performance vs wall-clock time, not just iterations.

Regarding Bayesian optimization (line 88 -- 98), it is does not strictly use non-parametric models, just Bayesian ones. Parametric GPs such as random features approximation and neural networks features have also been used for scalability. Moreover, the 'Var' on line 92 looks like it is in math mode, not text mode.   

As a final, somewhat nitpicky comment, I personally dislike 'Rethinking' in a paper title. Most research papers involve rethinking something, so it's a rather unnecessary term. I think a title like 'Global Search in Differentiable Simulatiors with Bayesian Optimization' could be better alternative. 
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: I was generally happy with this paper, its contribution, presentation and experiments. 10 seeds is enough for me personally. The codebase also looks good quality. I encourage the authors for a timely release so it can be adopted by the community.  
issues: Regarding the originality of this idea, I found 
Sample and time efficient policy learning with cma-es and bayesian optimisation, LK Le Goff et al. ALife 2020
which is somewhat similar. 
I would like the authors to comment on the differences between this approach and theres, and work in a reference into the main text.
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: Differentiable simulators provide gradients to help system identification and trajectory optimization. Most of the literature on system identification and trajectory optimization uses simple differential simulators (i.e., rigid body), since deformable objects, fluids, strings, and contact-rich environments have non-differentiable areas where gradients are particularly non-smooth.

This paper compares several differentiable simulators (Nimble, Warp, DiffTaichi, Plasticine Lab) that are based on different simulation techniques (mesh-based, particle-based, and a mix of the two) on a large set of challenging environments for both trajectory optimization and system identification. 

Furthermore, the authors propose a novel optimization technique for this non-smooth gradient landscape. Such a technique is composed of a global search implemented with a non-parametric Bayesian technique, a semi-local search implemented with CMA-ES, and a local search via gradient descent (RandDescent).
strengths_and_weaknesses: Contribution
=========

The contribution is twofold: 1) providing a new analysis of differentiable simulators in contact-rich environments and in presence of fluids/deformable objects and 2) proposing a novel optimization technique that utilizes both gradient and derivative-free techniques.

The contribution, at a high level, is explained very well. The motivation for the work is clear. The placement of the paper in the relevant literature is also clear. 

Method
======

At a high level, the method is also clear. However, there are a few important concepts missing.

The authors never formalized or described the trajectory optimization problem and the system identification. This makes the empirical analysis a bit unclear to me. As far as I understood, the trajectory optimization aims to solve a problem similar to the following:

$$
\min_{a_1, ..., a_T} || s_T - g || \quad \\
\text{s.t.} s_0 = \overline{s}_0, \quad \\
s\_{t+1} = f(s_t, a_t)
$$

where $f$ is the simulator. One could use $\nabla_a f(s, a, \omega)$ to solve the problem with gradient descent (or a more complex algorithm like the one described by the authors). I think that will be very beneficial to spend a few lines introducing the problem.

The system identification part is a bit more clear to me, I guess that could be framed as

$$
\min_\omega \mathbb{E}\left[\left(f(s_t, a_t, \omega) - s_{t+1}\right)^2\right]
$$

and $\nabla_\omega f(s, a, \omega)$ can help solve the problem.

In my understanding, this simple ``problem statement'' could be much more useful than the Algorithm~1, which I find honestly confusing.

I agree, instead, with the current high-level description of Bayesian optimization and CMA-ES.

Question
----------

Would the general framework of MPC help to recover bad gradients? I.e., if we take a step in the opposite direction, we could still correct the trajectory in the receding horizon... 

Experiments
=========

I am very impressed by the experimental section. It is true that the problem definition that I proposed above would greatly help the reading (at least I think so); but the authors did a great job in introducing the simulators, the environment, and the results. 

It is possible to see that the algorithm introduced by the authors exhibits often higher performance (lower cost) in the environment tested. The authors visualize the optimization landscape. I am very impressed by the algorithm being able to solve these problems which are often (relatively) high-dimensional and rich in highly non-linear dynamics. 

The real robot experiment (cloth) used for system identification is also very interesting and challenging. 

The limitations section is of great value, particularly highlighting the large discontinuities of the gradients of Pinball, Swing, and Flip, preventing a correct optimization. This shows that we still need more accurate gradient estimation for highly discontinuous dynamics. Nevertheless, the potential for robotic application of differential simulators is certainly high. 


Details
=====

I think that the term ``policy optimization'' is slightly misused. I'd call policy optimization, methods that improve the parameters of a policy. 
The policy, in this case, would be a parametric model that outputs a sequence of actions given a starting state. 

I'd call instead ``trajectory optimization'' the process of finding the optimal sequence given a state. 

Notice that the main difference is that we can potentially run policy optimization only once, and then re-use our policy for different states, while trajectory optimization needs to be re-run for any new starting state.


The equation in line 89 is very obscure to me: $f(x): f(x^*) = \min_x f(x)$. What does this notation mean??

 




robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: I highly recommend this paper for this conference.

The paper deeply analyzes differentiable simulators and proposes a novel optimization framework to use for both trajectory optimization and system identification. 

The analysis is very useful as differential simulators are enjoying increasing popularity in the community due to their potential. Until now, most of the literature was focused on rigid-body dynamics, but this paper analyses challenging scenarios with deformable objects (ropes, cloth), liquid (sugar), and rich contacts between the objects. 

I believe that this analysis can be useful for simulator designers, and to understand the current potential and limitations for system identification, trajectory optimization, and policy optimization. 
issues: Missing information
----------------------

My only issue is the lack of a proper definition of the problem:

 - which loss are we optimizing? how is framed the optimization problem?

Writing down a nice problem statement would help the reader understand better the paper. 

Avoidable details
-------------------

Some details, instead, are in my opinion unnecessary in the main paper. Algorithm 1 is poorly introduced, and I believe it would require too much space to introduce it properly. Why don't delegate it to an appendix? The explanation in Section 3 is sufficiently clear to me. 


I strongly recommend the authors consider these suggestions.
review_title: null

Meta-review: metareview: The paper proposes an optimization method that combines Bayesian optimization as the global search method and gradient based method using differentiable simulation. The method is demonstrated on an extensive set of experiments including rich contact and deformables, and is also validated on a real robot.

Most of the reviews agree that 

(1) The work has excellent presentation quality, and does a good job at motivating the work.

(2) The experimentation is impressive with diverse tasks, environments, and simulators.

(3) The novelty of the optimization framework is good.

They also brought up some missing citations, and the hardware experiment improvement suggestions.

During the reviewer discussion, there is a general acknowledgment of the good quality of the paper among most reviewers. Therefore, we recommend accepting the paper as oral.
confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept (Oral)