Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The manuscript proposed a benchmark platform for deformable object manipulation tasks: DaXBench. DaxBench contains several environments with a mixture of fluid, fabric, and elastoplastic materials. Due to the support of JAX, DaXBench seems to support multi-GPU simulation and training easily. Also, it promotes differentiable simulation. The authors compared a set of controller optimization methods on the proposed benchmark platform and shared their insights.
strength_and_weaknesses: Strength:
- As far as I know, DaXBench is the first deformable object manipulation benchmark platform based on JAX.
- This manuscript compared a large set of methods on their proposed method, which provides the readers with a good sense of the performance of their simulator and the compared methods.
- The provided code represents the visible reproducibility of the work.

Weaknesses:
- I wonder how the *mixture* is handled in these demos: are they two-way coupled, one-way coupled, or any other coupling was used?
- I personally did not follow the *Lazy Dynamic Update* part: MPM will require a grid for simulation. Does it mean the overall grid is 32*6*32, or is the grid actually 128*128*128, but only a small region of it is activated? My personal feeling is this setting only applies to a small fraction of the proposed environments, is it?
- My main reservation is the comparison at the framework level. JAX has a very similar position to taichi and Nvidia warp, which are more famous for physical simulation. And I would expect they have similar performance to JAX-implemented environments, is it? Is DaXBench distinguishable from them because of user-friendliness? Or better interface to deep learning modules? Or speed?
clarity,_quality,_novelty_and_reproducibility: The manuscript is nicely written. I did not follow the *Lazy Dynamic Update* part and hope for more explanation of it.
summary_of_the_review: This manuscript proposed DaxBench, the first deformable object manipulation benchmark platform based on JAX. I appreciate the work from the authors to build the simulation framework and share it with the community, but I expect some explanations and comparisons to taichi/warp.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper presents a benchmark that can simulate and evaluate a variety of manipulation tasks on deformable objects, including fluid, cloth, rope, and elastoplastic objects. The simulator is implemented using Jax, therefore it can also provide analytic gradients, making it compatible with recent popular control algorithms designed for differentiable simulation. 

A series of comparisons between differentiable-simulation-physics methods and gradient-free ones are further conducted. The experiments cover a wide range of categories of algorithms, e.g. reinforcement learning, imitation learning, and planning. Based on these comprehensive evaluations, the authors conclude the limitation and advantages of the differentiable-physics-based methods.

In the end, the authors successfully conduct a real-world Push-Rope task using the control policy from the simulation, which can validate the fidelity of the simulator.

strength_and_weaknesses: Strength

In my view, the experiments are the highlights of this paper. The authors implement the recently proposed differentiable simulation papers in their benchmark DaX, and compare them with gradient-free methods. The experiments include various robotic learning algorithms. This systematic evaluation indicates that the differentiable-physics-based method can indeed have better performance in some cases, but they also suffer from limitations as many gradient-based methods do, i.e. sensitive to the optimization landscape and initialization. And it also reiterates the importance of exploration in those differentiable-physics-based methods. Moreover, the authors perform real-world experiments, making the benchmark more convincing.

Weaknesses

Since the research of differentiable physics is progressing fast, the authors might better illustrate and clarify its difference from related literature. For example, PlasticineLab is another platform for differentiable soft body manipulation. Since both DaX (fluid, elastoplastic, and mixture) and PlasticineLab are using MPM as the governing dynamics, it is not apparent to me why PlasticineLab cannot support liquid and mixture (in Table 1) while DaX can. Moreover, it is unclear to me how the solid ingredients are represented in Pour-Soup. Are they elastoplastic objects or purely rigid bodies? Is there a coupling issue between the fluid particles and the ingredients?
Furthermore, the Saving Memory strategy in Section 3.1 is the same as the checkpoint scheme in [1] (Section 4.2), where the back-propagation trades time for memory usage by recomputing the intermediate variables in each step.

[1] Qiao, Yi-Ling, Junbang Liang, Vladlen Koltun, and Ming C. Lin. "Efficient differentiable simulation of articulated bodies." In International Conference on Machine Learning, pp. 8661-8671. PMLR, 2021.

clarity,_quality,_novelty_and_reproducibility: The Quality of this paper is quite good to me as it covers different algorithms, tasks, and physics systems. The analysis after the experiments is insightful and valuable for future research.

The novelty of this paper is okay as it designs an efficient differentiable framework to simulate soft bodies. It could be better if it can provide more quantitative evaluations of the performance (e.g. comparison with PlasticineLab)

The experiments and methods look reasonable. And the authors promise to release the entire benchmark and the code, I think this paper should be reproducible.

The clarity is fine. Here are some sentences that I might need more clarification on,

‘The policy complexity grows exponentially with the planning horizon, known as “curse of dimensionality”;’ - This is not how I usually understand the “curse of dimensionality”. Are there more concrete citations or examples of the exponential complexity w.r.t. the horizon?

‘Macro-actions reduce the dimension of the action space and the effective horizon, and hence increase the scalability requirement on the DOM methods’ - Why does it increase the requirements instead of making the scaling easier?

summary_of_the_review: In my view, this paper provides a unified platform to evaluate the soft-body manipulation tasks and differentiable physics algorithms. It conducts comprehensive experiments and provides useful insights for future work. The sim-to-real experiment makes it more convincing. One thing it can improve is its differentiation from previous methods. In total, I think it is a good paper.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper introduce DaXBench, which is a very strong differentiable simulator for various DOM tasks. The authors examined various algorithms for the given tasks.
strength_and_weaknesses: Strength:
- A very respectable implementation efforts was put into this work. I think this simulator will be of interest to many in the community.
- A large range of learning algorithms are studied. Offering some interesting insights.


Weakness:
- Some of the more recent DOM works are not referenced to, all of these works offer insights / simulator for DOM:
* ThreeDWorld: A Platform for Interactive Multi-Modal Physical Simulation, Gan et al, Neurips 2021
* ACID: Action-Conditional Implicit Visual Dynamics for Deformable Object Manipulation Bokui Shen, Zhenyu Jiang, Christopher Choy, Leonidas J. Guibas, Silvio Savarese, Anima Anandkumar, Yuke Zhu, RSS, 2022
* VIRDO: Visio-tactile Implicit Representations of Deformable Objects., Youngsun Wi, Pete Florence, Andy Zeng, Nima Fazeli. ICRA 2022.
- I'm not confident about the sim2real part. Saying the simulator has sim2real ability is a pretty big claim. However, only a demo video is provided without any systematic evaluation.
clarity,_quality,_novelty_and_reproducibility: Great on all fronts.
summary_of_the_review: Very nice paper. I think it's a strong paper and definitely worth acceptance. I wish the authors can address the weakness i mentioned above.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposes a benchmark for deformable object manipulation, which is based on differentiable physics and uses the latest JAX framework. The support by JAX and differentiable property will benefit many algorithms in this direction. All reviewers are positive about the contribution. Some issues regarding the details are raised by the reviewers and then addressed by the authors. It is a clear case of an accepted research work at ICLR. 
justification_for_why_not_higher_score: Already highest level of score. 
justification_for_why_not_lower_score: The meta-reviewer shares the same excitement as all other reviewers that this work is an important contribution to the community.
Decision: Accept: notable-top-5%