Robotics benchmarking: where we are, and what comes next
Shared test objects remain useful today. Alongside them, benchmarks such as BEHAVIOR 1K describe a thousand household activities, including tasks involving liquids, cloth and changes such as heating an object. Completing these activities can require a robot to interpret an instruction, choose the right objects and coordinate many actions. Each of those abilities needs testing, as does the robot's ability to use them together.
Robotics benchmarks now test all of these abilities. This review looks at the main approaches, with a focus on manipulation: robots grasping, moving and using objects. We will look at what the tests measure, what their results have revealed, and why repeating an experiment on hardware remains so difficult. That brings us to a question we have been thinking about at ProjectSim: what comes next in robotics benchmarking?
What robotics benchmarks measure
A benchmark defines a shared test and a way to score it. Some evaluate a particular component of a robot system. BOP, for example, measures object pose estimation: finding an object's position and orientation from images. GraspNet evaluates proposed grasps, including where to place the gripper and how to orient it. Shared images and annotations let the teams developing these methods compare them without having to compare an entire robot system at the same time.
Choosing a grasp is one part of completing a pick and place task. The robot also has to lift the object, move it and release it in the right place. Benchmarks such as MetaWorld, RLBench and ManiSkill test task execution by running a policy, the system that selects the robot's actions, inside a simulated environment. In closed loop evaluation, the policy receives new observations as it acts, so it can respond to what actually happens. If an object slips, for example, later actions must account for its new position. These environments can be reset to compare policies over repeated attempts.
Collections of tasks also let us study how a robot learns. LIBERO tests knowledge transfer and lifelong learning: whether learning one task helps with another, and whether a policy retains earlier skills as it learns new ones. Its task suites vary objects, spatial relationships and goals. CALVIN tests how a policy uses skills together by asking it to follow a sequence of language instructions. The environment is not reset between instructions, so each task must begin from the state left by the previous one. Success on isolated tasks therefore does not guarantee success on the whole sequence.
RoboCasa365 also tests how well policies combine skills. It supplies 365 tasks across 2,500 kitchen environments. In the paper's initial comparison, the robot policy GR00T N1.5 received additional training on 30,000 demonstrations across 300 tasks. It was then evaluated on 50 tasks. Its average success rate was 43% on tasks built around individual skills, 9.6% on tasks combining skills in ways included in that additional training set, and 4.4% on tasks with combinations absent from it. In this experiment, completing combined activities was much harder than performing individual skills. The separate scores make that difference visible; one overall average would conceal it.
The objects and instructions can introduce further difficulties. BEHAVIOR 1K includes liquids and cloth, which flow or deform during interaction. These tests require simulation of fluid flow and deformation, in addition to the motion of rigid objects. VLABench includes sorting books by publication year, which requires working out the correct order as well as moving the books. Here, an incorrect arrangement could come from a reasoning error even if every grasp succeeds.
An apparently successful robot may also rely less on its instruction than we expect. In LIBERO Plus, some evaluated policies continued to perform familiar tasks when the instruction was left empty. When the instruction changed the target object, they often continued to act on the original one. Success with the usual instruction had hidden a weakness in following a different request in the same scene.
“So it followed the instruction?” GIF via GIPHY.
LIBERO Plus and Colosseum V2 test robustness by varying conditions such as the scene's appearance, object positions or instructions. These tests ask whether performance survives a specified change. A policy can complete many tasks in familiar conditions and still fail when the camera moves. Reporting the effect of each change identifies which camera views, layouts or instructions cause performance to deteriorate.
Physical benchmarks test execution with real sensors, contact and hardware. FurnitureBench provides repeatable furniture assembly tasks. The Functional Manipulation Benchmark includes grasping, reorientation and precise insertion. A part that is slightly misaligned can catch on an edge and fail to fit. Physical testing measures whether the robot can complete that interaction on hardware; success in simulation alone cannot establish this.
Some projects also share access to the physical testing. RoboChallenge evaluates submitted policies remotely on a common set of real robot tasks. RoboArena combines tests from multiple sites. Evaluators choose tasks and compare pairs of policies without knowing their identities, and those comparisons produce a ranking. RoboChallenge lets policy developers use a common physical test setup; RoboArena gathers comparisons from different physical settings.
Beyond manipulation, Habitat includes navigation and object rearrangement, HumanoidBench tests tasks requiring coordinated control of a humanoid's body, and RoboCup@Home evaluates domestic service tasks that include human interaction. These extend evaluation to reaching the work area, balancing while acting and responding to people.
The field has therefore built tests for individual components, complete tasks, unfamiliar conditions and combinations of abilities. Their scores answer different questions. Accurate pose estimation can help a robot grasp an object, but it does not establish that the robot can follow an instruction to put that object away. A score can also reward completing a task in a way that bypasses the ability we intended to test.
When the score only checks whether the blocks ended up in the box. GIF via Tenor.
The problem with physical robot tests
A comparison is easier to trust and build on when another team can repeat it. For large language models, evaluation is relatively straightforward for many tests: download the same inputs, run another model and apply the same scoring code. SWE bench, for example, pairs software issues with executable tests that check proposed code changes.
That shared test makes it easier to find out which changes improve a system. The SWE agent study used SWE bench to investigate how an LLM should interact with a codebase. The authors kept the underlying model fixed and compared different tools for viewing, searching and editing files. They could measure which interface changes helped the agent resolve more issues, and other teams developing coding agents could test those ideas on the same problems.
A simulated robot environment can also be shared as software. Reproducing a physical robot test requires more. Sharing a policy and its code does not reproduce the camera calibration, gripper pads, table surface or object placements. Those differences can change both what the policy sees and what happens when it acts.
Suppose a bowl starts a little farther from the gripper than it did in a previous trial. The robot might nudge it instead of grasping it. The next camera image then shows the bowl in a different position, and the policy must react to a different situation. Replaying the original commands will not recreate the original trial. Even with careful setup, we need repeated attempts to estimate success rates and determine whether a measured improvement exceeds normal variation between trials.
Those attempts require physical work. Hardware needs maintaining, objects resetting, cameras calibrating and recordings checking. Testing in a second room means setting up and documenting another experiment. The number of trials matters, too: with a small trial set, a modest improvement is hard to distinguish from chance, and a rare failure may never appear.
The collection effort behind DROID gives a sense of the scale involved. Its 76,000 demonstrations, covering 350 hours of interaction, took 50 people across 13 institutions a year to collect. DROID is a training dataset, rather than an evaluation benchmark, but it illustrates the work required to collect diverse physical experience. This is why robotics needs substantially more investment in this work, including maintaining the setups that make continued collection and testing possible.
The recordings also need enough documentation for someone else to use them. The LeRobot dataset team's account describes missing task annotations, ambiguous camera labels and inconsistent action formats. Without that information, the person loading the dataset may not know what task was attempted, which camera produced an image or how to interpret an action value. Common formats and explicit metadata make the collected data easier to combine and reuse.
For an evaluation dataset, that documentation also needs to identify which objects, rooms and tasks were excluded from training. An unseen object, an unseen room and an unseen combination of familiar actions test different kinds of generalisation. That distinction determines what a result tells us about performance beyond the training examples.
What we would like to borrow from the LLM ecosystem is the ability to repeat and challenge a result without rebuilding the experiment from scratch. Repeated, comparable tests would also help us investigate which changes to a robot's training lead to better performance, just as the SWE agent study could investigate changes to a coding agent.
What robotics benchmarks should tell us next
Given the cost of collecting physical experience, we think benchmarks should help decide what a policy needs to practise next. A useful test can identify a weakness, suggest experience that might address it and measure whether that experience helps. This would give us evidence for one of the most expensive decisions in robot learning: what data to collect next.
Consider a policy that usually picks up a part and carries it to the right place, but struggles to insert it. The failed attempts show a recurring pattern: once the part catches on the rim, the robot keeps pushing from the same angle. That suggests a specific addition to training: demonstrations of withdrawing the part, realigning it and trying again. Those demonstrations might be more useful than the same amount of additional practice on the whole task. A benchmark that shows where attempts fail gives us a reason to test that choice.
Work on recovery is beginning to make this connection. REBOOT pairs precision assembly failures with recovery demonstrations. LIBERO RECOVER studies this in simulation by adding recovery examples to a policy's training data. The extra examples helped policies complete more tasks when they were placed directly into a situation left by an earlier failure, such as a bowl displaced by a failed grasp. They made little difference when policies attempted tasks from the beginning, encountering their own mistakes along the way.
That result shows why finding a weakness and collecting examples to address it are only part of the work. We also need to test what the policy learned from those examples. For the insertion problem, we could compare additional recovery demonstrations with an equal amount of general task practice, then measure both recovery performance and overall task success. Repeating such comparisons would help us choose training experience on the basis of measured improvement. This is the role we want benchmarking to play in the next round of robot learning.
Our first experiment: narrowing the real-to-sim gap
We would like to create much of that targeted practice in simulation, where a policy could attempt many variations of a difficult interaction without someone physically resetting the task each time. For this to help, the simulation needs to preserve the difficulty we want the policy to overcome. If a part slides into place in simulation when it catches on the real rim, simulated practice has removed the problem the robot needs to learn to handle.
This is why our first experiment at ProjectSim focuses on the real-to-sim gap: differences between policy performance in a physical setup and its simulated reconstruction. We are investigating whether more faithful assets and scenes, including object geometry, appearance and physical properties, bring simulated performance closer to the results observed on the robot. We are also developing benchmarking measurements to assess how much those changes help.
This draws on work such as SIMPLER, which compares policy performance in real and simulated setups, and SimFoundry, which reconstructs interactive scenes from real video. In our study, an external evaluator ran and scored physical trials on a setup with two robot arms. We reconstruct object geometry and appearance from scene captures and assign physical properties such as mass and friction. We also use a Gaussian splat, a 3D representation built from images, to reproduce the background seen by the policy.
We then run the same policies on the corresponding simulated tasks and compare their task scores and the stages they reach with the physical results. These measurements let us ask whether better assets and scenes reproduce real robot performance more faithfully.
Starting views for one policy on a bowl stacking task: physical setup on the left, ProjectSim reconstruction on the right.
If that approach holds up across more tasks and policies, we want to use the resulting benchmarks to find policy weaknesses and design simulated experience to address them. We could run that practice at scale, without collecting each additional training example on a physical robot, then test whether it improves performance on the real task. Our present experiment investigates the fidelity of the reconstructions; using them to guide and improve policy learning is the next step we want to explore.
Conclusion
From shared objects to tests of complete activities, robotics benchmarks have steadily expanded what we can learn about robot capability. They now help us see where a task breaks down, which changes a policy can handle and which abilities it still lacks. The next phase should connect that understanding to the choice of experience used to train the policy.
Our contribution begins with testing whether better simulated assets and scenes bring policy performance closer to reality. If they do, those scenes could help us investigate weaknesses found during evaluation and provide the practice needed to address them. Benchmarking would then connect successive rounds of testing and learning: each evaluation helps determine what to practise, and the next measures what improved. With realistic simulation, that cycle could run at a scale that physical data collection alone cannot support.
