MuJoCo Manipulus: A Robot Learning Benchmark for Generalizable Tool Manipulation

Authors: Jonathan Zamora, Daniel Seita, Yue Wang
Primary area: datasets and benchmarks
Keywords: robotics, deep reinforcement learning, benchmark

Abstract:
We propose MuJoCo Manipulus, a novel open-source benchmark powered by the MuJoCo physics simulation engine, designed to accelerate advances in robot learning for tool manipulation. Our benchmark includes a diverse set of tasks for tool manipulation --- a domain where the field currently lacks a unified benchmark. Different research groups rely on custom-designed tasks or closed-source setups, limiting cross-comparability and hindering significant progress in this field. To that end, our benchmark provides 16 challenging tool manipulation tasks, including variants of Pouring, Scooping, Scraping, Stacking, Gathering, Hammering, Mini-Golf, and Ping-Pong. The benchmark supports both state-based and vision-based observation spaces, is fully integrated with the Gymnasium API, and seamlessly connects with widely used Deep Reinforcement Learning libraries, ensuring easy adoption by the community. We conduct extensive reinforcement learning experiments on our benchmark, and our results demonstrate that there is substantial progress to be made for training tool manipulation policies. Our codebase and additional videos of the learned policies can be found on our anonymous project website: https://mujoco-manipulus.github.io

Decision: reject
Reviewer scores: 3, 5, 3, 3, 3

Reviews and discussion:
<thread>
creation date:
1729977836360

thread_type:
official_review

<thread_object>
creator: reviewer_vjeu

<content>
summary:
The paper introduces MuJoCo Manipulus, a new benchmark for robotic tool manipulation. Built on the MuJoCo simulation platform, the benchmark includes 14 tasks and offers a user-friendly gym interface. The paper also provides baseline evaluations using SAC and PPO algorithms on the proposed tasks.

strengths:
1. The paper is well-structured and presents its contributions clearly.
2. Tool manipulation is an important and timely topic in robotics, and the introduction of a new benchmark in this area is a valuable contribution.

weaknesses:
1. A significant limitation of the benchmark is that the tasks are based on floating control points, rather than simulating direct interaction between the robot and the tool. This abstraction reduces the realism of the tasks and poses challenges for sim-to-real transfer, which is critical in robotic manipulation.
2. Despite the claim of offering a "diverse set of challenging tool manipulation tasks," the benchmark consists of only 14 tasks, with a significant portion focused on pouring. This narrow focus could limit the utility of the benchmark for evaluating a broader range of tool manipulation skills.

questions:
I have no major technical questions, as the paper is relatively straightforward. However, I am unsure whether ICLR is the most appropriate venue for this work. While the benchmark provides meaningful contributions to the field of robotics, it lacks substantial algorithmic innovation or methodological novelty that would typically align with the scope of ICLR. A robotics-focused conference may offer a more fitting platform for this type of work.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730167049097

thread_type:
official_review

<thread_object>
creator: reviewer_dwwo

<content>
summary:
a Mujoco benchmark for policy learning including pouring, stacking, scooping, and various miscellaneous tasks, each with multiple tool options

strengths:
This work introduces a MuJoCo simulator featuring a range of tasks, including pouring, stacking, scooping, and various miscellaneous tasks, each with multiple tool options.

The benchmark supports both state-based and image-based representations, compatible with standard reinforcement learning algorithms such as PPO and SAC.

weaknesses:
The novelty of this benchmark appears limited. Many tasks resemble simplified, toy examples, with objects floating in the air rather than being manipulated by robotic arms. This raises concerns about the practical value of the benchmark for advancing robot learning.

The content is incomplete, with multiple paragraphs repeating similar ideas. While the state space for different tasks is described, there is no mention of the action space, reward function, or terminal states.

There is no comprehensive comparison with previous benchmarks, leaving the unique contributions of this benchmark unclear.

The number of objects or tools available for each task is limited, which restricts the potential for in-category object manipulation learning.

questions:
1. Could different robot manipulators be added to the simulator? Most robotic arms are limited by their mechanical design and current motion planning algorithms, making it challenging to follow the trajectory of floating objects.

2. Could this benchmark support in-category object variations for each task? Adding this feature would greatly benefit the robot learning community.

3. A comparison table highlighting the differences and contributions of this benchmark compared to others would be very helpful.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730430654318

thread_type:
official_review

<thread_object>
creator: reviewer_aaox

<content>
summary:
This paper introduced a MuJoCo-based simulation framework named "MuJoCo Manipulus," which features tool-use tasks such as pouring, stacking, scooping, gathering, etc. In these benchmarking tasks, an "idealized" agent manipulates a free-floating tool to physically interact with objects and accomplish manipulation goals. Specifically, the initial release of this benchmark includes 12 tools spanning two categories: Kitchen Tools and Home Tools, which were either hand-designed or drawn from existing asset libraries (e.g., TACO and MuJoCo). The authors highlighted several engineering features offered by their framework, including support for end-to-end RL with the MuJoCo physics engine, integration with the Gymnasium interface, and state/vision-based observations. As an initial attempt toward solving these tasks, they evaluated two popular deep RL algorithms, PPO (on-policy) and SAC (off-policy), with both state-based and vision-based observations. These baseline methods achieved mixed results in these tasks, suggesting room for future development.

strengths:
- **Problem motivation.** Tool use is an important family of robot manipulation problems to study, as it represents a high level of physical intelligence manifested in intelligent species and requires multi-step, contact-rich interactions with multiple objects. Building systematic benchmarks for robotic tool manipulation would facilitate reproducible research in this problem domain.
- **User-friendly design.** The proposed benchmark was built on the MuJoCo simulation engine. Its designs prioritized flexibility and ease of use, with the standardized Gymnasium APIs and support for state-based/vision-based observations for end-to-end reinforcement learning. These designs could reduce the barrier of entry for researchers to leverage this simulator for research.
- **Baseline evaluations.** Experiments with standard RL algorithms, including PPO and SAC, presented a baseline view of existing RL methods' strengths and limitations. These preliminary results provide insight into the major challenges and motivate potential avenues for future work.

weaknesses:
While the benchmark is well-motivated and carefully designed, its current version bears major weaknesses:

- Although the authors motivated this benchmark for robot learning and manipulation research, this paper did not use realistic robot models or controllers. Instead, it used an idealized "point mass" robot with simple Cartesian action spaces. This design severely abstracts away the challenges of contact-rich robot manipulation problems in the real world. Compared to existing manipulation benchmarks, such as ManiSkill2 (Gu et al., 2023) and Robosuite (Zhu et al., 2020), which aimed to model robot dynamics faithfully, the simplified robot model could hinder the ability to deploy the learned models on physical robots and thus limit its impact in advancing robot learning research.
- The authors motivated the benchmark as a "wider-scale tool manipulation benchmark with substantially more tasks." However, the current offering of the tasks is fairly limiting compared to several existing simulation frameworks for robot navigation and manipulation (Nasiriany et al., 2024; Szot et al., 2021; Li et al., 2022), which provides 100-1000 tasks in large-scale home environments. In contrast, the proposed benchmark only provides 14 simplified tasks in a restricted tabletop workspace.
- The authors highlighted the "Unified API for Reinforcement Learning with MuJoCo" as the main feature of the benchmark and spent a great amount of text in Sec 3 explaining these design choices. However, these features seem to be directly inherited from MuJoCo and its derived open-source tools/software, such as Gymsium. MuJoCo provides native support for state-based/vision-based observations, which has been commonly used in existing frameworks such as Robosuite (Zhu et al., 2020). Therefore, it remains unclear what technical challenges the authors have tackled to construct this benchmark and what are unique designs of this benchmark that have not been done in prior work.
- The experiments of this paper focus on evaluating off-the-shelf RL methods instead of innovating new ones. More qualitative analyses and ablative studies should be done to understand the limitations of these methods and provide insight into how these algorithms could be improved to better tackle these tasks. The mixed success rates also indicate the challenges of defining informative (dense) reward functions without extensive reward engineering. More reward design techniques could be explored to improve the quality of these tasks for RL purposes.

questions:
In addition to addressing my comments in the Weaknesses section above, I have the following questions:
- I would like to hear the authors' explanation of the technical contributions of this work in comparison to existing MuJoCo-based simulations and tools. The APIs, simulation speed, and environments do not seem particularly novel. Can you please clarify how this effort differentiates (improves upon) robot learning simulations, especially the ones built with MuJoCo?
- Why would different tasks come with different action spaces, from three to five DoFs? For example, the Pouring tasks have 4D action spaces, while the Stacking tasks have 3D ones. Would it be beneficial to unify the actions across these tasks to enable multi-task RL research with the benchmark?
- Could you comment on the potential of transferring the model trained on MuJoCo Manipulus to a real robot? How could you bridge the embodiment gap between the idealized robot used in this work and the kinematic designs of real robots?
- The RL results reported that "the PPO+RGB and SAC+RGB methods were often better than PPO+State and SAC+State." This result is interesting and a bit counterintuitive, as visual RL is usually more expensive to train due to the high-dimensional visual observations as input to the policy. Could you provide more insight why we see this result?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730558023124

thread_type:
official_review

<thread_object>
creator: reviewer_eogs

<content>
summary:
This paper presents Mujoco Manipulus, a tool use benchmark for RL research.  This benchmark is built on top of Mujoco and gynmasium, and supports a range of tasks and tools such as scooping, stacking and gathering.  The authors evaluate PPO and SAC on these tasks using both vision and state information, demonstrating that these tasks are nontrivial to solve.

strengths:
+ Developing new benchmarks for robotic skills and manipulation is an important contribution to the community.
+ The paper is well presented and easy to understand.
+ The authors present a clear picture of related work.

weaknesses:
- The size of the benchmark is relatively small.  There are only a few tasks and objects available.
- The complexity of these tasks is relatively low.  The agent has direct control over the tools (non-dexterous) and requires only a few degrees of freedom to move them.
- The realism is fairly limited, while at the same time the simulator uses Mujoco on the CPU.  Most recent robotics benchmarks are moving toward either greater realism or very-high-throughput simulation on the GPU, and so this benchmark, which lacks both, feels somewhat behind the Pareto frontier.
- It's not clear how challenging this benchmark is.  The authors run PPO and SAC on these tasks and show that they struggle to solve them out of the box.  In many robotics applications though, a large degree of reward shaping or learning from demonstrations is frequently necessary to solve real world problems, and it's not clear why those approaches wouldn't work here as well.
- A more thorough comparison against relevant tasks from Maniskill2 and Robosuite is necessary to establish the utility/necessity of this new benchmark.  Those two settings contain some tool use, are the tasks here harder than those?  More comprehensive?  Given the simplicity of the models presented here, it seems like the others should be more realistic.  What new features does this benchmark include that are not covered elsewhere?  If the only contribution here is to package together certain types of tasks that exist across many other benchmarks, that could still be useful to the community, but more work needs to be done to establish the added value of the dataset relative to what's already out there.
- Along the same lines, in a more general sense it is not clear what utility the broader community gets from this new benchmark.  Given the limited realism, this doesn't seem like it advances our practical ability to train real-world robots (sim-to-real does not seem to be much of a consideration here).  Given the small size, it doesn't add much to the already substantial physically grounded simulation benchmarks for RL.  The authors motivate this benchmark by arguing that there is not a dedicated tool-use benchmark for robot manipulation, but there are several that contain tool use as a subset, and it's not clear what this new one adds to them.

questions:
The authors mention Maniskill2 and Robosuite, but say that only a portion of those tasks cover tool use/manipulation.  How would the current benchmark compare in size/complexity/realism with a subset of the Maniskill2 and Robosuite tasks that only cover tool use/manipulation?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730818233313

thread_type:
official_review

<thread_object>
creator: reviewer_p8et

<content>
summary:
The paper presents a benchmark designed for robotic tool manipulation using the MuJoCo physics engine. It offers a diverse set of 14 tool manipulation tasks across categories such as pouring, stacking, scooping, gathering, hammering, and scraping. Each task features specific tools (e.g., cup, ladle, hammer) and allows for both state-based and vision-based learning. The benchmark integrates with Gymnasium API and popular reinforcement learning (RL) libraries, making it easily used by the research community.

strengths:
1. The paper is well-written with clear images.
2. It presents convenient APIs and environments, built on the MuJoCo engine, incorporating various tools and objects.
3. Several commonly used RL algorithms were tested.

weaknesses:
1. The benchmark lacks completeness; as a robot learning environment, it does not include any robots, and the tasks and action spaces are very simple.
2. Novelty and contribution are severely concerned - it feels more like an engineering project. Essentially, it imports objects into MuJoCo, designs tasks around them, and tests some off-the-shelf algorithms.
3. The action space appears to involve directly transforming the pose of the objects (tools) rather than applying interaction with them, which raises severe concerns about whether true physical simulation is involved.
4. Current SOTA simulation benchmarks could support the tasks presented in this paper by simply importing the relevant 3D objects. 

In summary, this is a reasonable project but lacks completeness (at the very least, a robot should be included as a robot learning environment) and novelty, so I recommend rejecting it.

questions:
No additional questions.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733056667862

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response to Reviewers (1/3)

comment:
Dear Reviewers,

Thank you for your time reviewing our work, and for providing detailed feedback. We appreciate each of your reviews, and have worked hard to address several of your concerns. Below, we provide a summary of key updates to our work:

1. **Updated Results w/ New Dense Reward Functions**
    - We agree with reviewers that reward engineering is highly important for our Reinforcement Learning benchmark, and have spent time improving our Reward Functions for all tasks. In the updated draft of our work, please see Figure 9, where you can find Success Rate curves for the 16 Tool Manipulation tasks in our benchmark. These results showcase the effectiveness of our new rewards, and indicate that there is still a wide margin of improvement to be made in our more challenging tasks.
        1. Our benchmark’s harder tasks include `Scoop Particle`, `Scoop Cube`, `Stack Plates` , `Stack Bowls`, and `Ping Pong`. We provide a table below, showing the Final Average Success Rates across 5 seeds for our 5 most-challenging tasks:
            
            
            | Task ID | # of Training Steps | CrossQ | SAC | PPO | Num Steps per Episode | Total Episodes of Experience for Training |
            | --- | --- | --- | --- | --- | --- | --- |
            | Scoop Particle | 300,000 | **98.4** | 43.6 | 0 | 200 | 1500 |
            | Scoop Cube | 300,000 | **98.4** | 18.4 | 0 | 200 | 1500 |
            | Stack Plates | 300,000 | **76.8** | 0 | 0 | 200 | 1500 |
            | Stack Bowls | 300,000 | **34.4** | 0 | 0 | 200 | 1500 |
            | Ping Pong | 50,000 | 0 | 0 | **27.6** | 50 | 1000 |
            | **Hammer Nail** | 30,000 | 99.6 | 98 | 84 | 200 | **150** |
        - The above table highlights several areas of improvement for our harder tasks.
            1. **Sample-Efficiency of RL Algorithms**: We include the results of our **easiest** task, `Hammer Nail`, where all 3 of our baselines learned near-perfect control policies with just 150 episodes of experience during training. For our harder tasks, our strongest baseline **CrossQ** required **10x the amount of episodic experience** during training to learn good tool manipulation policies, while other baselines struggled significantly to match this performance.
            2. **Off-Policy RL for Short-Horizon Dynamic Tasks**: In `Ping Pong`, a dynamic task where a paddle must quickly hit a ball, we observed empirically that CrossQ and SAC struggled to conservatively hit the Ping Pong Ball. In Figure 7 of our paper, the Failure case for `Ping Pong` showcases at t=3 the ball being hit too aggressively. This was a frequent problem with CrossQ and SAC, while PPO did not have these issues. We believe for short-horizon dynamic tasks, there is more to be investigated about why Off-Policy RL methods struggle compared to On-Policy RL methods.
            3. **Learning Reward Functions**: In our `Scoop Particle` and `Scoop Cube` tasks, our human-engineered reward function still required 300,000 steps of training (1500 episodes of experience) to learn good tool manipulation policies. This is large in-comparison to tasks like `Hammer Nail`, which can be learned in just 30,000 steps of training (150 episodes of experience). We believe that learning a reward function from videos, or from structured data such as graphs, could provide a better reward signal for tasks like `Scoop Particle` and `Scoop Cube`, and enable progress in area (1) (i.e. better reward functions can improve the sample-efficiency of RL algorithms).
    - For complete details of each task category’s reward function design, success conditions, and environment resets, please see Section A.4 in our new Appendix.
        
        
2. **Updated Website with New Visualizations and Figures**
    - We have updated our project website, [https://mujoco-manipulus.github.io](https://mujoco-manipulus.github.io/), to include visualizations of our work. This includes an Overview of our Simulation Framework (Figure 2 in our paper), videos of successful policy rollouts for each of our 16 tasks, and a results section with Success Rate curves for each task (Figure 9 in our paper).
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposed a benchmark designed for robotic tool manipulation using the MuJoCo physics engine. It was reviewed by four field experts and received unanimously negative evaluations. The main concerns raised include a lack of significant technical contributions and weak results. The AC finds no reasons to recommend acceptance. The authors are encouraged to address the reviewers' feedback and refine the work for submission to other venues.