Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: 
This paper presents PlasticineLab, a differentiable physics environment geared towards softbody manipulation. By implementing a softened rigid-deformable contact interface, and by leveraging recent advances in softbody dynamics simulation (DiffTaichi, ChainQueen), PlasticineLab is able to provide analytical gradients which seem to outperform gradient-estimation based approaches (SAC, TD3, PPO) on few tasks.


## Strengths

**S1** The central problem tackled here is quite interesting (and challenging)! There is a growing interest in the ML community wrt differentiable simulation techniques and in particular, their applicability to learning dynamics.

**S2** The paper is extremely well-written and easy to follow. While this builds heavily on DiffTaichi and ChainQueen, it is commendable that this paper came across as self-contained.

**S3** I believe the characterization of related work is fair. An interesting point made here was that TDW and SAPIEN do not provide assets for soft-body simulation. I am unsure if that's entirely true, but the lack of assets is indeed an issue for softbody simulation. It will be a welcome contribution if this paper were to make these 3D assets publicly available.

**S4** Conceptually, this paper claims that inductive biases (arising from simulation of deformable objects) should be exploited wherever possible. In particular Fig. 4 and Table 1 seem to indicate that gradient-based optimization (using differentiable simulation) consistently outperforms RL techniques in 8/10 tasks. To one's anticipation, gradient-based optimization seems to achieve significantly faster convergence (seems like two orders of magnitude), which is an impressive feat.

**S5** It is interesting to see that it is possible to differentiate through contacts across rigid and deformable objects. To my knowledge, this has not been demonstrated before (has been tangentially discussed in [C]) and is a significant contribution.


## Weaknesses

**W1** The paper could benefit from an explicit exposition of critical design choices that affect differentiability. While PlasticineLab uses a particle-based model for representing and simulating soft-bodies, alternatives in the form of (usually tetrahedral) mesh-based representations exist. It appears that particle systems are chosen to enable trivial differentiability (e.g. the material-point method in the absense of contact forces is analytically differentiable)

**W2** An important detail which I couldn't find in the paper and/or supplementary material is how many of the parameters are simultaneously observable. For example, if the masses of the particles and the manipulator contact parameters are both unspecified, wouldn't this lead to problems in observability (i.e., both quantities cannot be simultaneously solved for, resulting in ill-behaved gradients)?

**W3** Does the approach assume a one-one correspondence between the predicted and target shape? While this might seem a reasonable assumption, I believe gradient computation is cumbersome (and perhaps ambiguous?) were this to be relaxed?

**W4** Another crucial detail that the paper does not seem to get through. As with most other "differentiable physics" approaches, unmodelled effects in the dynamical system might limit the applicability of the system. Since the physics engine only implements forces and softbody dynamics that are "predetermined", I would imagine it is hard to emulate real-world effects such as wear-and-tear, sophisticated contacts, and material properties. In favor of the paper though, I feel this detail might also be out of scope to an extent. (Recent approaches such as Neural dynamical systems [A] and Learning physical constraints by neural projections [B] come to mind, to handle some of these concerns).


## Summary

While the differentiable simulation aspect of the paper is not substantially novel (building atop DiffTaichi, ChainQueen, and soft contact models - Stomakhin et al. 2013), the overall system is impressive and addresses a gap in the differentiable physics community. Simulating differentiable softbody dynamics, as well as interaction with (a limited class of) rigid bodies could open up interesting avenues in reinforcement learning and softbody manipulation.

[A] Neural dynamical systems: balancing structure and flexibility in physical prediction. arXiv 2020

[B] Learning physical constraints with neural projections. arXiv 2020

[C] Scalable differentiable physics for learning and control. ICML 2020

review_title: Interesting contribution

final_score: rating: 7: Good paper, accept
review_content: ####  Summary

In this work, the authors present PlasticineLab, a new framework for soft-body manipulation tasks for Reinforcement Learning and planning algorithms. The environment consists of a novel soft (i.e., deformable, plastic) material termed Plasticine which is complex to model and manipulate because of the inherently complex high-dimensional governing equations and the large number of degrees of freedom associated with soft materials. The PlasticineLab framework proposes 10 novel tasks involving manipulation of the soft plasticine material. The authors show thorough empirical analysis that traditional state of the art model-free reinforcement learning algorithms fail to effectively learn the task even after a substantial amount of training. Thus effectively showcasing the complexity of the proposed tasks and the inability of state of the art RL models to model the proposed tasks.

#### Positives:

1. Novel tasks: Each task poses a different challenge: E.g., some tasks involve flattening the plasticine, other involve pinching the plasticine while yet other tasks involve grasping one or multiple plasticine objects (at one or multiple points) and deforming it or moving it in some required manner. 

2. The variety of tasks test various facets of RL like long-term planning especially in the case of multi stage tasks.

3. Another major effort prelavent in the paper is that the authors have chosen to use a differentiable physics engine using the DiffTaichi system thereby making the gradients available for planning and control algorithms. 

4. The paper highlights through empirical results, the superiority of gradient-based approaches (over model-free RL approaches) that leverage the underlying differentiable physics engine toward learning the required tasks.

5. An important facet of a benchmark is to propose tasks that are sufficiently complex for the current state of the art procedures. The authors employ 3 state of the art model-free RL algorithms and show that these RL models perform poorly in a majority of the 10 tasks. Torus, RollingPin, Move tasks are the only three tasks where the model-free procedures are able to perform somewhat comparably with the gradient-based planning approaches which themselves perform well in all but the Writer, Pinch and TripleMove tasks. The last task involves multi-object manipulation and requries long-term planning and hence  

#### Concerns:

1. In figure 4. the Adam optimizer and the GD which both seem to consistently accumulate the greatest rewards also have high variance, some commentary about how this can be explained would help the reader better contextualize the results.


2. As one of the main claims of the paper is the challenge of soft-body manipulation and the proposal of a framework for the same, it is imperative to demonstrate the variation of the degree of difficulty with increase (or decrease) in rigidity of the materials being manipulated. A comparative analysis such as this, demonstrating for example the variation in IOU error of the best performing RL model with increasing in yield stress for plasticine would serve to showcase the actual challenge posed by soft-body material mainpulation in the context of the current proposed framework. Ofcourse since decreasing softness and increasing rigidity is most likely not as simple as increasing a single number such as yield stress, this is a minor concern and more a suggestion toward a holistic analysis of the proposed framework.


#### Minor Details & Suggestions:

1. Extrapolation is an important facet of learning algorithms in general. Since one of the suggestions of the current work is to present the PlasticineLab framework as a way to not only characterize RL and gradient-based algorithms but also combine these two families of methods, it is also important to evaluate the performance of these models on unseen but related tasks e.g., manipulating a table with fewer or grater number of legs, trying to place more than 3 objects at specified locations.

2. The citation relating to the paper by Avila et al. titled End-to-end differentiable physics for learning and control published in the Advances in Neural Information Processing Systems conference in 2018 seems to be repeated. 

3. Another potential direction of the current framework could be using PlasticineLab to learn policies which might be transferred to the real-world (similar to the task mentioned in [1]). If feasible, adding some brief commentary about this in the context of [1] might open up further avenues of exploration for plasticinelab.

#### References:

1. Matas J, James S, Davison AJ. Sim-to-real reinforcement learning for deformable object manipulation. arXiv preprint arXiv:1806.07851. 2018 Jun 20.
review_title: Authors propose framework consisting of a set of soft-body manipulation tasks. Framework has 10 varied tasks which test different facets of reinforcement-learning algorithms. A differentiable physics engine is used as the core of the framework allowing learning, planning procedures to leverage task gradients while learning to perform each task. Empirical results on multiple state of the art (SOTA) model-free RL models are convincing and show that proposed tasks are too complex for SOTA models.

final_score: rating: 7: Good paper, accept
review_content: The paper introduces a new open-source simulation benchmark for soft robotics. The simulation environment builds on top of DiffTaichi, an existing differentiable simulator which enables end-to-end differentiability. The paper proposes 10 different tasks, each with 5 variations and evaluates both RL-based policy learning methods and gradient-based optimization methods on those tasks. The results suggests neither current RL-based methods nor gradient-based method can solve most of the tasks efficiently, especially for those require long-term planning.

Overall the paper is well-written and the contribution is well-argued. I have a few comments / questions as follows:
- The simulator only considers the state of the end-effector of the manipulator. It would be great to consider higher DOF soft manipulators (e.g. [1][2]) which would further benefit the soft robotics community.
- Given the randomness and the nature of the RL algorithms [3], the evaluation in section 5.2 should be done with at least multiple random seeds with multiple trials per seed to make the benchmarking results  statistically significant.
- Similarly for Section3, In Table 1, it would be great to see the standard deviation in addition to the average value. It's also not clear how many trials were conducted in order to get the numbers shown in both Table 1 and the plots in Fig. 4.

[1] Della Santina, Cosimo, et al. "Dynamic control of soft robots interacting with the environment." 2018 IEEE International Conference on Soft Robotics (RoboSoft). IEEE, 2018.

[2] George Thuruthel, Thomas, et al. "Control strategies for soft robotic manipulators: A survey." Soft robotics 5.2 (2018): 149-163.

[3] Khimya Khetarpal, Zafarali Ahmed, Andre Cianflone, Riashat Islam, Joelle Pineau. Reproducibility in Machine Learning Workshop, ICML 2018
review_title: A new simulation benchmark for soft robotics

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: PlasticineLab

The paper presents a new soft-body manipulation benchmark for RL and differentiable planning.
The presented simulation suite is very interesting and the contribution is solid.

Strength:
- new simulation benchmark with features that are not yet well explored
- differentiable physics to open up possibilities for planning methods
- tasks are difficult enough to be challenging for a while
- baseline results are provided

Weaknesses:
- only the computation times would be good to add

Presentation:
The paper is clearly written and easy to follow.

Ways to improve the paper:
- wall-clock times would really be very useful. Both for the forward pass as well as a backward pass through the entire horizon with Adam. Maybe also some notes on how it can be parallelized since you have a CUDA implementation.


Details:
- p5 last paragraph: "For any grid points with a signed distance d..:" The formulation is not clear enough. 
 Do you mean with positive signed distance. Prob not, because you can also have penetration. But why would a point then not have a distance to the rigid body?  
- same paragraph: "By definition, s decays exponentially with d until d becomes negative (when penetration occurs)" Well it decays with increasing distance (and then it cannot become negative if it increases...)
- 5.1: IoU definition: Are S always positive? I don't exactly understand what the mass tensor S is. I know it from rigid body dynamics, but this does not seem to be the same here. Can you clarify this better such that it becomes clear why the formula describes and IoU.
- Fig 4. Consider removing the grey background that seaborn uses automatically. The plots will look much cleaner and better visible. 



review_title: Very interesting and potentially impacting work on soft-body manipulation

Meta-review: This paper proposes a new differentiable physics benchmark for soft-body manipulation. The proposed benchmark is based on the  DiffTaichi system. Several existing reinforcement learning algorithms are evaluated on this benchmark. The paper identify a set of key challenges that are posed by this specific benchmark to RL algorithms. Short horizon tasks are shown to be feasible by optimizing the physics parameters via gradient descent. The reviewers agree that this paper is very well-written, the  problem tackled in it is quite interesting and challenging, and the use of differentiable physics in RL for manipulating soft objects quite intriguing.
Decision: Accept (Spotlight)