Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper presents a new benchmark environment for manipulation tasks. The environment supports both traditional rigid-body manipulation tasks and soft-body tasks, while unified interface allows a wide range of algorithms. Results show that the asynchronous rendering and render server approach provides an improved FPS for CNN-based policy learning.
strength_and_weaknesses: Strength:
1.	The proposed environment supports a wide range of observation and controller types and supports mainstream algorithms in sense-plan-act, RL and IL frameworks.
2.	The tasks include those from previous works and some newly proposed ones, covering a wide aspect of challenges in manipulation. The support for soft-body tasks is of great importance.
3.	The asynchronous rendering approach and render server implementation allow improved rendering performance and reduced memory usage.

Weaknesses:
1.	All the proposed manipulation tasks are based on robotic arms and lack support for other actuators (e.g. Dexterous hands).
2.	Since different simulators have different parameters that may drastically affect the overall performance, please include the details of parameters for each simulator (e.g. Did you use the GPU pipeline in IsaacGym or just used the CPU pipeline, the substeps for physics simulation, the number of the total vertex in the simulation). 
3.	The performance comparison is carried out on RGBD input settings. The performance of pure state input and point-cloud input is not reported.
4.	The effectiveness and accuracy of demonstration conversion are neither discussed in detail nor measured quantitatively.
5.	Point-cloud observations in some tasks contain ground truth segmentation, which cannot be easily obtained in the real world. This barrier for sim-to-real transfer could be avoided with a better observation design.

clarity,_quality,_novelty_and_reproducibility: The paper is of good quality. The illustrations and experimental results are clear to me.
summary_of_the_review: This work proposes several improvements to the current simulator for manipulation. The improvements in rigid-body soft-body interface and simulation performance, along with the variety of supported tasks, is of great value. This work's weakness includes omitted details from different sections and an issue in observation. Overall, I am leaning towards acceptance.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a new benchmark for robotic manipulation that supports various types of manipulation tasks with improved efficiency. The main contributions are:
- Rich task collections, assets, and demonstrations.
- Multi-controller support and action space conversion.
- Efficient 2-way coupling rigid-MPM simulation.
- Efficient parallelization for visual RL training.
strength_and_weaknesses: Strengths:
- Variety of the manipulation tasks: stationary/mobile-base, single/dual-arm, rigid/soft-body, different action spaces and controllers. These, in combination, provide richer families of tasks than previous benchmarks.
- Good software engineering techniques for efficient learning with multi-physics and visual input.

Weaknesses:
- Lack of different grippers and arms support - only Panda arm and two-finger/suction grippers are implemented. To facilitate dexterous manipulation and more sim-to-real transfer, it is important to support more industrial arms (IIWA, UR5e, etc.) with multi-finger grippers.
- Limitations are not presented in the paper. It would be important to understand what are the pros and cons of this benchmark compared to existing ones in a clear manner (e.g. having a table showing the comparison with popular benchmarks on multiple dimensions/features). For example, Maniskill2 does not seem to support photorealism, domain randomization, cloth simulation, scaling to thousands of parallel environments (like Isaac Gym), etc., which exist in some of the previous benchmarks.

Questions:
- While claiming Maniskill2 is the first to support real-time simulation and rendering of MPM material, I don't see concrete statistics supporting this. What is the speed comparison against naive MPM simulation or PlasticineLab?
- Is two-way rigid-soft coupling necessary for manipulation tasks in Maniskill2? Though I believe it is more realistic (compared to one-way coupling), for manipulation purposes, the rigid grippers are usually much heavier than the soft materials being manipulated. In this case, maybe one-way coupling is realistic enough and more efficient. It would be necessary to showcase where the two-way coupling is a must for some manipulation tasks.
clarity,_quality,_novelty_and_reproducibility: This paper is sufficiently novel and clearly written, with abundant technical details presented.
summary_of_the_review: I recommend acceptance based on the benchmark novelty and the engineering efforts for simulation efficiency. In my opinion, this paper has no major flaws, and I believe there is enough motivation for the community to use this benchmark to facilitate future research.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors propose ManiSkill2, a benchmark for robotic manipulation. It builds upon ManiSkill 1.0 and has better support for different types of gripper-based manipulations, uses significantly more interactive objects, runs much faster, and includes soft-body simulations. The authors show experiments on a few tasks, 4+ million frames of gripper-based interactions, which can be used for imitation learning, and include pickup object sim2real results.
strength_and_weaknesses: ### Strengths
* The extent of the manipulation capabilities and support is great.
* The number of objects used for a manipulation benchmark is quite impressive.
* The performance benchmarks and simulation speeds are impressive. I appreciate all the engineering effort that went into this paper!
* The sim2real results are neat.


### Weaknesses

* **W1. Lack of discussion on objects.** The authors claim they have 2000+ object models, a significant increase over the 162 objects in ManiSkill 1.0. However, there isn't much of a discussion on where these objects come from. Where do they come from? Are they going to be released (and under what license)? What is the distribution of the object categories? To what extent are they interactive/articulated?

* **W2.** The backgrounds and setup of the scenes are quite weak. In comparison to ManipulaTHOR, Habitat 2.0, and iGibson 2.0 for manipulating different objects, which are in household environments, the gray backdrop behind the objects makes the environments feel like a toy environment. It would be much better if the manipulation of objects happened in the context of other objects (e.g., when turning on a faucet, the agent is always in a bathroom/kitchen), when opening the cabinet, the agent is always in a kitchen or bedroom, etc. It doesn't seem like the tasks are very realistic to what one might want a robot to do in the real-world. The one task used for sim2real in this paper is merely picking up a fairly visually distinct cube.

* **W3.** I wish the paper was formatted a bit differently, and instead focused mostly on experiments in the main part of the paper, instead of discussing technical details in sections 3 and 4. The main part of the paper doesn't seem like much of the content in these sections is necessary for most users of ManiSkill2 to read. It seems like much of this could have gone into the appendix.
clarity,_quality,_novelty_and_reproducibility: **R1.** On reproducibility: I appreciate the release of all the code and simulator functionality. A lot of work has gone into this, and it will benefit the community.

**R2.** Much of the work seemed to focus on clear next steps from ManiSkill 1.0. The authors make many incremental improvements, which in aggregate will be helpful to the community.
summary_of_the_review: The engineering effort that went into this is quite impressive to support such a wide variety of agent-based manipulations. However, the work is pretty incremental, focusing on a few clear next steps from the previous work in ManiSkill 1.0. I would have also liked the authors to start moving towards more realistic task setups that a robotic agent might actually encounter in the real-world. Currently, the agents are just operating with isolated objects on a grayscale background, and this isn't practically useful for most tasks we want robots to perform.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The authors identify a core part of embodied AI to be manipulation of diverse objects and generalization of manipulation abilities across object instances.  They introduce a new suite of manipulation tasks with object-level variations to support research on these challenges.  The aim of this task suite is to support vision-based policies and fast simulation.  This is done using Warp, a JIT framework from Nvidia that allows for specification of tasks using python, but faster execution.  A speed comparison is performed, demonstrating good performance relative to relevant alternatives.  Demonstrations are also included, and tests demonstrating the plausibility of sim2real are also included in the paper.

The reviewers all found this paper and benchmark to be a valuable contribution to the field.  Though there were some issues raised about performance comparisons details, features (e.g., gripper options and visual backgrounds), omitted details (e.g. objects included), etc, these were reasonably well addressed by author responses.
justification_for_why_not_higher_score: Scope of contribution.
justification_for_why_not_lower_score: Core quality and relevance for this audience.
Decision: Accept: poster