Venue: NeurIPS 2022 Track Datasets and Benchmarks

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: * They chose IsaacGym. In my opinion it is currently the best library for the robotics or physics research. It makes possible to run one experiment in 30 minutes instead of the days if use Mujoco instead.
* First released benchmark for bimanual dexterous hands manipulation with such a good performance.
* Added comparison between environments and human skills at different age. Which demonstrated that current skills are pretty basic for 
 human and achievable by baby which cannot be said about computer.
* There were introduced 20 different tasks and tested on the most popular RL algorithms including Meta RL and Offline RL. Surprisingly SAC didn't work well. Which may require additional investigation.
strengths: * easy to use environments.
* good efficiency
* pretty successful attempt to create a benchmark in which you can compare ai with a human skills which might have positive impact in future general ai research.
* Provided  relationship between their dexterous tasks and the 70 motor skills of humans at different ages.
weaknesses: * Didn't include observation space design for different tasks. It is impossible to understand what is the difference in observation between MARL and Single Agent versions without looking into the source code.
* Same issue with reward. Reward function was not  described in paper. Rewards in different even similar tasks have very different scale. Didn't provide maximum reward for each task. For example in 'Two Catch Underarm' algorithms achieved only one catch. Looks like to solve it you need to get 27+ reward.
* No analysis of the different algorithm  performance.
* for some envs 100m of the steps was not enough.
correctness: Benchmark was done using a couple of on-policy MARL algorithms and one off-policy SAC which is the SOTA for a lot of continuous action space tasks. But in this case SAC didn't work well which is consistent with IsaacGym ShadowHand environment but requires additional investigation in future.
clarity: The paper is well written and easy to follow. The visuals and tables significantly help my understanding of the paper.
But not enough explanation about how reward is calculated. Also environments were designed in the way that on some of them we see reward ~1000 and ~20 on others. I think this part could be unified.
relation to prior work: One of the most related prior work which  is the IsaacGymEnvs shadow hand environment which worked as a baseline for current envs.
Also they mentioned OpenAI's Rubik's Cube. But this work's goals are much different from prior works and it doesn't make sense to compare them a lot.
additional feedback: Overall pretty good benchmark. But you can add more information.
I'd like to see in the paper:
1) observation, reward and action configuration for different tasks. Or at least demonstrate how you build it for one example task.
2) Maybe normalize reward so it will have same scale for different tasks.
3) It would be nice to add support for the default IsaacGymEnvs RL library rl-games.
4) Explain benchmark results. Why PPO is the best and SAC cannot even solve it.
review_title: Good environments and tasks to benchmark existing RL algorithms but lacks some technical details. 

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This submission proposes Bimanual Dexterous Hands Benchmark (Bi-DexHands), a simulator that involves two dexterous hands with tens of bi-manual manipulation tasks and thousands of target objects (mainly come from YCB and SAPIEN).

The authors claim Bi-DexHands has five key features: Isaac Gym Efficiency, Comprehensive RL Benchmark, Heterogeneous-agent Cooperation, Task Generalization, and Cognition.

To the best of my knowledge, Bi-DexHands is the first simulation environment which focus on bimanual dexterous hand manipulation tasks. This benchmark will probably attract researchers from robotics and reinforcement learning.
strengths: - Though dexterous hand manipulation is an important topic and draws much attention, there are no good benchmarks for dexterous hand manipulation in the bimanual setting. This benchmark provides a good test bed for bimanual dexterous hand manipulation.

- Benchmarked many different categories of  RL methods, including single-agent RL, Multi-agent RL (MARL), Offline RL, Multi-task RL, and Meta-RL.

- The simulation speed is decent, though it is mainly because of Issac Gym.

- The documentation is good overall, though there are still some minor issues. 
weaknesses: - Simulation 
  
  - According to the figures shown in the Github repo, I feel there might be some simulation issues presented in this benchmark, e.g.,
    
    - In [this figure](https://github.com/PKU-MARL/DexterousHands/blob/main/assets/image_folder/scissors.gif), the scissors start to move before being touched.
    
    - In [this figure](https://github.com/PKU-MARL/DexterousHands/blob/main/assets/image_folder/pen.gif?raw=true), the movement of pen cap seems incorrect.
    
    - As far as I know, the [simulation of opening the cap](https://github.com/PKU-MARL/DexterousHands/blob/main/assets/image_folder/bottle_capv2.gif) of bottle is non-trivial. Can you provide more details about how you implement this? And where are the object models come from?
  
  - Simulating everything correctly is tricky, but this is somehow very important in build simulated robotic tasks, especially when using RL agent (which always exploits the bug in the simulator)

- Performance metrics
  
  - Currently, the performance on the tasks is measured by average rewards, but this is not very intuitive. For example, people cannot easily understand whether an agent with ~100 reward is good or bad, without looking into the details of this task. 
  
  - Therefore, I hope there can be more intuitive ways to measure the performance. One possible solution can be using some binary-based metrics like success rates.

- Observations of the tasks
  
  - The observations are state-based (a vector describing the states of everything in the environment). This is good to kick off the research, but not a realistic setting. In real world, agent always needs to estimate the states of other objects by visual observations. It would be better to consider incorporating visual observations in these tasks.

- Benchmarked methods
  
  - Only RL-based methods are benchmarked. However, other methods like motion planning, model predictive control (MPC) are also widely used in robotics research. If this benchmark is not purely designed for RL, then other types of methods should also be benchmarked.
  
correctness: Yes
clarity: The overall structure of the paper is clear and easy to follow.



Something I want the authors to clarify:

- What controllers are used to control the dex hands? And how do you determine what controllers to use? I guess something like PID controllers are used. 

relation to prior work: Yes. 

However, I did not find an explanation about how the objects from YCB and SAPIEN are used. For example, which task uses YCB objects and which task uses SAPIEN objects? How are the objects selected?
additional feedback: None
review_title: A benchmark for bimanual dexterous hand manipulation

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This paper contributes a novel bi-manual dexterous manipulation benchmark for RL. The highlight is the elaborated design of 20 tasks that covers different difficulty levels equivalent to growth month of a baby, as well as the highly paralleled running platform for RL training based on the new simulator Issac Gym. The authors also provide RL baseline implementations and evaluate them based on categories. It's great the authors make the work open-source already and has got 20 forks and 177 stars on github, showing a wide impact.
strengths: As claimed in the paper, I agree that the proposed work has the following strengths:
1. Support most kinds of RL algorithms in one platform (this will definitely broaden the impact and benefit multiple RL directions while also provide a testbed to benchmark different methods)
2. Highly paralleled, able to sample actions and train efficiently on GPU (they describe in the appendix the improvement of running in parallel on one GPU instead of multiple ones required by Issac Gym originally)
3. Provide a set of objects to test task generalizability over target objects (I believe this generalization is essential to extend RL algorithms to a broader use case that it could possibly adapt to unseen objects after being trained on a publlic object set)


weaknesses: 1. Sensor simulation: I don't understand why this platform only provides point cloud given by depth camera. Other modalities such as RGB, normal, shading, etc. from multiple, configurable views should also be provided, to explore direct end-to-end training from raw sensor input, rather than known object target information like their 6D poses. This point should be an easy add-on though.
2. Ease of task customization: I read through the paper and appendix but didn't find detailed descriptions on designing customized tasks. The authors only mention that objects from provided YCB and SAPIEN datasets could be used to replace the current basis, but how to generate a ground truth target pose or trajectory for every new object, or even a completely new task, is not given. This point is essential for users who want to develop based on this platform and create their own specific tasks.
3. Different reward design for RL: In the appendix, each of 20 tasks is detailed with a given reward function with some hardcoded numbers. I wonder how are these numbers obtained by the authors and whether there are pre-experiments testing different types of rewards, because the reward function design is important to RL and the design details should be explored.
4. The evaluation of some RL algorithms are not finished yet.
correctness: The submission is constructed in a sound way. I got most questionable points answered in the supplementary material.
clarity: The paper is well written overall. I recommend a proof-reading on grammars and put more essential details about the dataset itself to the main paper from supplementary materials (a summary of A.1 and A.2)
relation to prior work: Yes, this work is novel and different from previous works. I recommend the authors considering more diverse tasks and algorithms mentioned in these previous works:

https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=8989777
https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=9196527
https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=9035030
https://arxiv.org/pdf/1904.11382.pdf
https://arxiv.org/pdf/2010.08587.pdf
additional feedback: This work is really a good contribution to the community. Besides the points I mentioned in weakness section, I am curious on several points on further extending this work:
1. How good is the physical contact rich actions simulated in the proposed platform, or Issac Gym?
2. How hard is it to import other dexterous manipulators and design new tasks to the workspace?
3. How to further evaluate the trained RL agents in real world?
review_title: Review of paper 'Towards Human-Level Bimanual Dexterous Manipulation with Reinforcement Learning'

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The paper proposes the Bimanual Dexterous Hands Benchmark (Bi-DecHands) which involves two dexterous hands and twenty manipulation tasks. The authors demonstrate efficient RL training, thanks to Issac Gym, and benchmark SOTA single-agent/multi-agent RL, offline RL, multi-task RL, and meta RL on these tasks.

strengths: The main contribution is threefold: a) relatively comprehensive (bi-manual) dexterous hand manipulation benchmarks, b) very fast simulation speed, c) comprehensive baselines methods across single/multi-agent RL, offline RL, and multi-task/meta RL. It’s pretty relevant for the community that is interested in (bi-manual) dexterous hand manipulation. The benchmark is open-souced. 

weaknesses: The paper does have a few major weaknesses. It motivates the benchmark by arguing that the tasks are designed to match different levels of human motor skills according to cognitive science literature. However, this claim seems to be built upon some shaky ground. The translation from “human skill” to “robotic task” seems a bit arbitrary. “Child places 10 pellets in the bottle” got translated into “put a ball into a bucket”. “Infant differentiate individual finger movement” got translated into “reorient a Rubrik's cube”. It’s very much unclear how these tasks are constructed and whether they really capture the motor skills of infants and children. Secondly, Two Catch Underarm was labeled as “adult” while it’s reasonable to assume many dexterous children/teenagers could do it. Also, in the “conclusion” section, the authors claimed that “the current reinforcement learning can reach the level of 48-months infants”, which is a very dramatic over-generalization. A 48-month infant can do way more tasks than pouring water and catching objects. Finally, a less important point, quite a few tasks are not inherently bi-manual, so it seems very forced to have both hands there (e.g. re-orientation, stack block, even door opening, etc)
correctness: There are a few questions regarding the correctness of the dataset. Firstly, it’s unclear how the authors propose to leverage the “thousands of objects” from SAPIEN dataset. In the paper, only ten to twenty objects were observed in the task. It’s still unclear how someone can use different object models for these tasks (e.g. for pouring water, how to use different models for the pot and the cup?). On a related note, the task setup seems to be specific for each object model. For example, in the “Lift Pot” task, the reward function relies on the “distance from the left hand to the left handle”. However, different models of the pot category can have a different number of handles and the handles can be located at different positions. Are these locations programmatically accessible from SAPIEN? It’s unlikely the case. Also, many tasks require a notion of the “goal pose”, which can also depend on the exact object models used. For example, in the “Two Catch Underarm” task, the goal poses probably should depend on the size of the objects being thrown. In general, it’s very unclear how these task setups can be object-agnostic, i.e. no need to hard-code these “handle positions” and “goal poses” manually. 

Furthermore, the authors implemented 5 single-agent RL algo., 5 multi-agent RL algo., 3 multi-task RL algo. and 2 meta-learning RL algo., which is really great and comprehensive. However, in experimental evaluation, only 2 single-agent RL algo., 2 multi-agent RL algo., 1 multi-task RL algo., and 1 meta-learning RL algo. are evaluated. It would be great if the authors can benchmark the performance of all the baselines that they implemented and include the additional results in the rebuttal. 

The current experimental results that the authors presented also raise a few questions. First of all, it’s not convincing why the SAC does not work on almost any tasks. The authors tried to offer their hypothesis in Section C of the appendix, but it’s still not 100% convincing. For example, from Figure 6 of the appendix, it seems that the SAC training becomes less stable when the number of parallel environments increases, which is very counter-intuitive. The authors should consider significantly increasing the replay buffer size of SAC (currently it’s only 5000, as specified in Table 43 in the appendix). Since the observation space only contains states (not images, more compact), so this should be doable. Also, maybe increasing batch size would help. Finally, the meta-RL baseline ProMP completely fails to solve the tasks, even in the simple scenario of ML1, in which the tasks clearly share some underlying structure. It’s recommended that the authors double-check the implementation of ProMP for any potential issues.  

clarity: The paper is relatively well-written, but the overall text contains a large number of typos and grammatical errors. The task names are also not consistent throughout the paper (compare table 1, table 2, figure 3, and figure 5 in the appendix and Appendix A.2 Detailed components of tasks).

relation to prior work: In the related work section, the authors mentioned previous learning-based and non-learning-based methods that tackle dextrous hand manipulation and also robot manipulation in general. However, they didn’t compare their benchmark with other robotic manipulation benchmarks (e.g. robosuite, RLBench, metaworld, etc). These benchmarks are slightly different because they don’t use dexterous hands, but it’s still relevant enough to be included and compared. 


additional feedback: The supplementary video is helpful. Thank you.
review_title: Solid benchmark for bimanual dexterous manipulation; room for improvement for related work, task setup, and experimental evaluation.

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This paper introduces Bi-DexHands, a simulation suite for bimanual dexterous hand manipulation that is built in Isaac Gym, enabling fast GPU-accelerated physics simulation. The authors benchmark a comprehensive set of different approaches covering reinforcement learning, multi-agent RL, offline RL, multi-task RL, and meta-RL. Their baseline results show that while PPO-based algorithms can perform well at single-task settings, multi-task learning still poses a considerable challenge for current algorithms on Bi-DexHands.

strengths: 1. This paper introduces a comprehensive benchmark with 13 bimanual dexterous manipulation tasks of varying complexity, based on different levels of human motor skill development. The authors benchmark different baselines covering reinforcement learning, multi-agent RL, offline RL, multi-task RL, and meta-RL. The code and tasks are well-documented, such as details on observations, actions, and rewards for all tasks in the Appendix.

2. The suite of tasks is built off Isaac Gym, enabling efficient scaling by using GPU-based physics simulation. This lowers the barrier for those with smaller compute-budgets to run experiments. 

weaknesses: 1. The paper is missing an upfront discussion of limitations, as well as a broader impact statement. These do not appear in the main body, or the Appendix. While some limitations are mentioned in Section 6, it is in the context of future directions. 
2. The paper would benefit from improving the clarity of the writing, see Additional Feedback section.

correctness: To my knowledge, yes.

clarity: The paper can be difficult to understand at times, and would benefit from a thorough editing readthrough. See Additional Feedback.  

relation to prior work: Yes, to my knowledge, no prior work has presented a simulated suite of tasks for bimanual dexterous manipulation.

The related work section would benefit from additional discussion of prior work on single-hand dexterous manipulation [1, 2, 3, 4] and cooperative multi-agent environments, such as [5].

1. https://arxiv.org/abs/2111.03062
2. https://arxiv.org/abs/2108.05877
3. https://arxiv.org/abs/2203.06173
4. https://arxiv.org/abs/2204.12490
5. https://arxiv.org/abs/2105.12196

additional feedback: L66: Can you clarify what “enormous target objects” means?

L267: Does “MuJoCo” mean the “D4RL-MuJoCo” tasks?

L267-269: Can you what is meant by “out-of-distribution action is more severe”? What is the distribution shift?

For Table 4, is it difficult to gauge the success of methods by averaging reward across tasks. Is there some measure of progress or success for each of the tasks? Even a normalized measure would be better, similar to Figure 4. 

L206-207: I appreciate the analogies of task difficulty to human measures. However, I think analyzing the performance of different RL algorithms on the proposed benchmark is more nuanced than “precisely point out agents’ intelligence level”. 

---

Minor comments (this is not a comprehensive list, and a thorough editing pass is recommended):

L9: “Issac Gym”, typo

L24: “the human hands”, grammar 

L45: “og”, typo

L109: “toward master”, grammar

L253-255: grammar

The paper occasionally uses “Shadow hand”, which should probably be capitalized to “Shadow Hand”?

L210: Should be capitalized as “Meta-World”
review_title: Official Review of Paper277 by Reviewer wDmx

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The Bi-DexHands Benchmark is a new Isaac Gym simulator that supports experiments with bi-dexterous robotic manipulation. It covers a wide range of settings (Single-agent/Multi-agent RL, Offline RL, Multi-task RL, and Meta RL), and includes a variety of tasks requiring two hands, along with several implementations of corresponding algorithms. Baseline results exercise the simulator, and demonstrate significant research challenges left in bi-dexterous manipulation.
strengths: The benchmark is highly relevant to current work in applying the various RL approaches to robotics challenges. 

The simulator is very fast, particularly since observations are state-based only. Image-based observations will decrease performance.

The reported experiments are sufficient to show the benchmark’s usefulness for continued research. The most simple tasks can be solved with standard RL techniques, given state observations, along with enough computation and reward shaping.

The paper presents a good analysis of relative PPO vs. SAC performance.

weaknesses: The benchmark currently supports only state-based observations, which greatly limits its utility to the community. But this is a reasonable first step to take, and it should be possible to add image-based observations in the future. 

Many of the RL algorithms provided have not yet been tested, so bugs remain. 

Support for applying the algorithms to physical shadow hands would be a significant extension of the framework.

Generally, the benchmark is still a work in progress. This is not too surprising, given its ambitious breadth. Still, Bi-DexHands is a solid initial step, and will hopefully become more complete soon.

correctness: The datasets and evaluations look reasonable.
clarity: The documentation and the paper are fairly clear and easy to follow. Community involvement can be expected to drive improvements.
relation to prior work: The relevant prior work is covered well in the paper.
additional feedback: It would be helpful to include pointers and comments related to the physical shadow hands, and how closely they may or may not relate to the shadow hands simulated here.
review_title: A limited but welcome contribution to robotic benchmarks of current interest.

Meta-review: metareview: The paper provides a variety of environment simulators and settings involving bimanual dexterous manipulation. These settings encompass many popular paradigms in RL today, including offline RL and multi-agent RL. I am glad to see that the reviewer discussion was fruitful and led to an improved submission. I would further encourage the authors to include more results with visual observations (as suggested by Reviewer EVVH). Some sort of visual input is becoming increasingly common in real-world RL-trained robots, and so benchmarks for this setting are important.
confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept