Venue: NeurIPS 2021 Track Datasets_and_Benchmarks Round2

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The paper introduces a generalizable manipulation skill benchmark with large intra-class topological and geometric variations. Compared with previous work, the paper focuses more on the object-level generalizability of manipulation skills. To support the new task, high-quality object data and large-scale demonstrations datasets are collected. The new challenging task may attract a wide range of research fields including 3D computer vision, reinforcement learning as well as robotics.
strengths: - The proposed task is challenging with the high value of the real-world application. The benchmark supports 3D perception tasks, learning of policies from interactions as well as learning-from-demonstrations, which will attract a broad set of researchers. 
- The provided dataset is of high quality and the demonstrations dataset is of large scale. 
- The selected baselines are reasonable and also show the challenges of the object-level generalization task, which can guide future research.
weaknesses: - The number of intra-class instances and categories is limited, which limits the generalizability to the real world.
- All the settings are in a simulation, the selected objects are hand-made CAD models. It's helpful to add some reconstructed objects which are closer to the real world and also, provide some real frames and tracks for the testing in the real world scenario to evaluate the sim-to-real gap.
correctness: Yes.
clarity: Yes, the paper is well written and easy to understand.
relation to prior work: Yes.
additional feedback: None
review_title: A good benchmark for generalizable manipulation research

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: ManiSkill is a large-scale open-source benchmark for physical manipulation skill learning over a diverse set of articulated objects from 3D visual inputs.  It consists of four tasks filtered by difficulty; the need for dual-arm collaboration and the number of DoF involved in solving a task.  Baselines are agents which use PointNets and an optional Transformer layer and are trained by a mixture of Behavioural Cloning and (Offline) RL.  Results measured in success rate show that this is a challenging benchmark.
strengths: Significance:
* As mentioned in Section 1, ManiSkill addresses shortcomings of existing datasets by testing for object-level generalization; variation within those objects, and using egocentric 3D vision (state/pointcloud/rgbd), as well as many different tracks the benchmark can be evaluated on.

Relevance:
* Numerous methods to train policies (behavioral cloning/RL with sparse/dense rewards) make this accessible for researchers from many backgrounds.

Accessibility and Accountability:
* The paper is very easy to read.  There is a discussion of the as well as a disclaimer that this is not a generic learning-from-demonstrations benchmark (line 114) which helps focus the work.
* The paper also explains how ManiSkill is limited and provides future directions to remedy this as well as an assurance that the benchmark will be continually updated and monitored.  This is very good to read!
weaknesses: There are no outstanding weaknesses -- any issues I have with the submission are minor.

Demonstrations generated by reinforcement learning may be easier to learn from and may have lower diversity than those provided by humans.  Some care should be taken when explaining why demonstrations from RL are suitable.  Moreover, discussions over the number of demonstrations and whether this is appropriate for an eventual human use-case should also be provided.

The authors could have perhaps had more discussions on why they chose the architectures they benchmarked to assure future practitioners that these are the best architectures that their research should build on.
correctness: If the demonstrations are obtained from a trained RL agent, more explanation is needed for why this trained RL agent does not suffice to solve ManiSkill.
clarity: The paper is very well-written.  There was a good use of bold font and underlining throughout which drew to the structure of the piece and enabled the reader to quickly find key points.  In particular, the abstract is excellently written.  Researchers who have previously worked with this benchmark who need a reminder could very quickly glean high-level information to refresh their memory.  

There are some minor grammatical errors (eg. line 1, poses -> pose).

Minor point, terms such as revolute joint, prismatic joint introduced in lines 177 and 182 are introduced, and perhaps could be supported by explaining how they look like with aid of a figure.  Perhaps this could be a modification of Figure 3.

I didn't quite follow the divide-and-conquer approach mentioned in lines 108-111 or how a reward function was automatically generated when it was introduced for the first time in lines 108-111 and presumed it was expanded on in Section 2.4?  If so, a link to Section 2.4 should have been provided for better clarity.
relation to prior work: Much related work was referenced throughout, with up to ten papers cited in one go to support a point.  Section 1 comprehensively introduces related work (eg lines 54/55) and explains how they are lacking.  This has the added bonus of being laid out in a well-structured format.
additional feedback: The paper was an enjoyable read overall and the issues I've raised are only minor in comparison.
review_title: Enjoyable Read, Very Minor Clarifications Needed

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The authors propose ManiSkill to benchmark manipulation skills over diverse objects in a full-physics simulator.  Panoramic cameras are simulated that captures ego-centric point clouds or RGB-D images. The benchmark is designed to serve a broad set of researchers interested in manipulation research, and supports the learning of policies from interactions. Besides, a large number of high-quality demonstrations are provided to support learning-from-demonstrations methods.
strengths: * The dataset contains various geometry and topology in high quality, which is important for evaluating the generalizability of the physical manipulation algorithms.

* Challenges and demonstrations are provided in a large amount of objects and trajectories.

* The 3D deep learning-based policy baselines are presented.
weaknesses: The amount of objects categories are limited, and the four tasks are relatively simple. However, these data are sufficient as a novel benchmark for physical manipulation skill, and has the potential to be extended.
correctness: The construction of the dataset and evaluating methods are technically sound.
clarity: The paper is well written and easy to follow.
relation to prior work: The discussion about prior works are clear.
additional feedback: None.
review_title: The benchmark provides a wealth of data and demonstrations that benefit the research community

Meta-review: This submission proposes a manipulation benchmark, with the focus of dealing with different objects in 3D simulation. All of the reviewers have rated the submission very highly, and agreed that it will be useful for a broad range of disciplines (e.g. robotics, RL). The paper is well written, the dataset is well documented, and benchmark results are provided. For this reason I recommend acceptance of this submission.
Decision: Accept