Venue: NeurIPS 2021 Track Datasets_and_Benchmarks Round2

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This paper describes a new simulation platform for studying dynamic manipulation of deformable objects. The platform is built on top of PyBullet, and consists of a suite of tasks involving different object categories and manipulation goals. The suite contains more task and object types than prior environments for deformable objects, and is built to be customizable. Initial experiments demonstrate how the platform can be used for representation learning and reinforcement learning research.
strengths: + Addresses a need articulated by the robotics community (customizable environments for studying deformable objects)
+ More diversity and customizability than prior such environments
+ Clear presentation and extensive documentation in the code repo
weaknesses: - The pipeline for customizing action and observation spaces, and reward functions, is not described in the main paper
- The VAE and RL experiments are not especially illuminating

-- Customizability --

I think the focus on customizability is great, and deserves a bit more detail. For example: does the platform support observation spaces that are not pixels? How would a user go about specifying a new action space or reward function? Can the fidelity of the simulation be changed based on user demands?

-- VAE and RL experiments --

The VAE experiments don’t fully describe the evaluation metric. From the figure I can infer that it is reconstruction error. Yet the text alludes to the fact that VAEs might learn a structured latent space and the latent representation could be useful for tracking holes, etc. I think it would be more interesting to investigate the VAE variants with respect to the latent structure they learn, rather than with respect to their ability to reconstruct test images. RL on top of VAE embeddings could also be interesting.

In general, both the VAE and RL experiments serve as a good demonstration of how the platform might be used, but do not yield many new insights into unsupervised learning or RL with deformable objects. I think that's okay, since that's not the focus of the paper, but perhaps could be improved.


correctness: All claims appear correct to me: as far as I can tell, the dataset is constructed in a sound way.
clarity: The writing is quite clear overall. 

The figures are also helpful, but could be improved. Many of the figures are raw screenshots without annotation. I think it could help to add more annotation, labeling different parts of the visualizations and pointing out qualitative things to look at. Figure 2 is a good example of what I'm thinking of; that kind of markup could be added to other figures as well.

Figures 11 and 12 show raw TensorBoard screenshots, which I don't think are very helpful. I take it that the intent is to show how the system is integrated with TensorBoard, but this is so standard that I think just stating it in text is enough. To me it would be better to use the space to show interesting results of the experiments, without the distraction of TensorBoard details.

Details on some of the tasks are missing. For example, the mathematical form of the actions space and reward function is only given for some tasks. I also could not easily find these details in the wiki. This information should be added somewhere in text, so that users don't have to hunt through the code.

Minor comments: 
- “DEDO” is referred to, in Table 1, before being defined, in Section 3. Also, Table 1 is not referred to in the main text.
- Sewing-v0: why not select from all 22000 meshes? Why only 1200?
- Line 273: “researches” —> “researchers”
relation to prior work: The relationship to prior work is well discussed. Table 1 is very helpful.
additional feedback: None
review_title: A solid contribution to promote research on deformable objects

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: In this paper, the authors present a set of environments for reinforcement learning for control theory. These environments focus on the manipulation and movement of deformable objects in 3D space. The environments are simulated with PyBullet and equipped with an OpenAI gym interface. The authors also provide some reference implementations of unsupervised representation learning methods and trained several RL algorithms using well-known open-source packages.

The paper's main contribution is a set of new environments for 3D manipulation and control of deformable objects. The authors provide a wide range of object topologies and procedural materials with a suite of tasks like hanging bags and buttoning cloth. Additionally, the environments come with customizability and extension to adjust to a user's given goals.
strengths: --The authors provide a suite of 3D manipulation benchmarks for tasks with deformable objects. This type of benchmark is quite hard to make since it requires an understanding of graphics and physics engines. Therefore, this will allow researchers to focus on generating control policies.  
--The code is accessible and is reasonably extensible, with particular strength in bringing in custom assets.
weaknesses: --It is unclear if these simulators are sufficient for sim2real transfer of policies or unsupervised learning approaches. Therefore, the usefulness of the simulators is limited to understanding how to build policies for complex control problems and the link between control policies and observation representations.
--The paper would benefit from a clear statement, via a table, of the runtime speeds of the suite of simulators.
--The lack of a code testing suite will make maintaining the benchmarks challenging and limit the extensibility of the benchmarks.
correctness: The claims appear to be correct.
clarity: The paper is reasonably well written and straightforward.
relation to prior work: The references to prior work appear adequate.
additional feedback: The paper lacks a titled section that discusses the limitations and scope of the benchmarks.
review_title: A suite of benchmarks for the 3D manipulation and control of deformable objects

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The paper presents benchmark tasks that focus on interaction involving soft bodies or deformable-body. The paper presents a task divided into 4 types hanging garment by hook, put cloth holes through a button, throwing hoops and lasso around the pole, and putting the garment on a mannequin. The environment is based on Pybullet with is open source and supports customization and addition of external 3d models. The paper also presents benchmarking results on these tasks
strengths: The paper is very well written, each task is explained in detail. The code and documentation are also very comprehensive.
The environment presented has some strong points:
1. Variety of 3d models of soft bodies and textures
2. Ease of use
3. Authors show benchmark results on these tasks using widely used RL algorithms like PPO and SAC
weaknesses: I am a bit concerned with how similar all tasks are in terms of executing them. For example, hang garment task and buttoning cloth seems to require the same kind of skill. Similarly, hang a mask, hang a backpack, and put a dress garment seems to also require the same set of skills. Most of these tasks could be grouped under the same category with slight variation in them for generalization. I feel this is not a comprehensive environment for training on deformable objects in general but rather a training on a specific type of deformable object. I am interested to know if authors have thought about including few other scenarios that involved soft bodies in some other context. 
correctness: To the best of my knowledge, the implementation of the environment sounds correct although I am not an expert in 3d env development.
clarity: The paper is clearly written.
relation to prior work: The paper cites a number of environments that support deformable objects.
additional feedback: None
review_title: Review for Dynamic Environments with Deformable Objects

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: Dynamic Environments with Deformable Objects (DEDO) is a set of tasks that involve deformable objects with special attention paid to everyday deformable objects such as masks, bags, clothes, and rope. The environments are implemented in PyBullet and use a publicly available repository of garment object models but are easily extensible to custom object models. A subset of the tasks is evaluated in unsupervised learning and reinforcement learning settings. 
strengths: * Manipulation of deformable/cloth objects is an interesting and underexplored topic.
* The code for this benchmark is well-organized and functional
* The authors use a free and open-source physics simulator
weaknesses: * It seems that there are already existing RL benchmarks for deformable objects. This benchmark has some more varied objects such as garments, but it's not clear what interesting challenges these novel objects pose to RL algorithms that aren't already in existing deformable benchmarks such as SoftGym. 
* Authors claim that their platform makes it easier to add custom objects (you have to recompile SoftGym), but this seems like a minor technical difference.
* The authors make the statement, "It is likely that learning structured latent states from simulations that involved only rigid objects was much more tractable than dealing with highly deformable objects. This indicates the need for further research with tasks that include deformables". While highlights an interesting property of unsupervised learning approaches, this conclusion could just have easily been made from existing benchmarks for deformable objects and is not specific to this set of deformable objects.
* More could be done to make these tasks resemble real-world robotics tasks. An environment with floating fingers at prespecified grasp anchors seems too oversimplified. This concern appears to be justified by how easily PPO can solve these tasks with almost no variability. 
* Even though each instance of a task has different garments of the same class (e.g. bags with different textures), it's unclear how much this affects the policy. Looking at the learning curves, it appears the specific garment doesn't affect the policy at all. There should be more variance in different instances of a task that demonstrates the necessity of using the observations.
* In the conclusion section, the authors state that they "plan to add mobile manipulators to the scenes and experiment with household tasks." This addition would certainly go a long way to make the tasks more interesting, challenging, and transferrable to real-world domains.
correctness: * Why was no hyperparameter search done for the various RL algorithms? I think this is common practice for benchmarks to compare algorithms fairly by doing a hyperparameter search for each algorithm instead of simply using the defaults.
* What do the error bars on the plots mean?
* Even though authors supposedly tested on multiple RL algorithms, only results for a single RL algorithm (PPO) are shown.
* Results for the lasso and buttoning tasks are not shown even though those seem to be the most difficult tasks.
* Only the results for two of the tasks are shown in the unsupervised learning setting.
clarity: * The hyperparameters used for the RL algorithms are not specified in the text.
* The architectural details for any of the evaluations (unsupervised or RL) are also not described in the text.
* If more room is needed for experimental details and further evaluation, it seems like many of the figures are redundant/unnecessary. 
relation to prior work: All related work work is discussed.
additional feedback: While this is an interesting direction, the tasks in this work don't present any unique difficulties not present in other deformable benchmarks, and the learned policies are not transferable to real-world robotics domains. In addition, the evaluation is not sufficient as the paper lacks many task evaluations, baselines, and experimental details.
review_title: DEDO Review

Meta-review: All reviewers agree on acceptance. I recommend the authors to take into account the reviewers' comments to improve the paper for the camera-ready version.
Decision: Accept