BiGym: A Demo-Driven Mobile Bi-Manual Manipulation Benchmark

Authors: nikita  chernyadev
nicholas  backshall
xiao  ma
yunfan  lu
younggyo  seo
stephen  james
Venue: robot-learning_org_CoRL_2024_Conference
Published: 2024-06-06 20:16:25

Abstract: We introduce BiGym, a new benchmark and learning environment for mobile bi-manual demo-driven robotic manipulation. BiGym features 40 diverse tasks set in home environments, ranging from simple target reaching to complex kitchen cleaning. To capture the real-world performance accurately, we provide human-collected demonstrations for each task, reflecting the diverse modalities found in real-world robot trajectories. BiGym supports a variety of observations, including proprioceptive data and visual inputs such as RGB, and depth from 3 camera views. To validate the usability of BiGym, we thoroughly benchmark the state-of-the-art imitation learning algorithms and demo-driven reinforcement learning algorithms within the environment and discuss the future opportunities.

Main paper: 
 introduction: 
 Machine learning benchmarks are of significant importance for measuring and understanding the progress of research algorithms. Examples of notable benchmarks include ImageNet [1] for image understanding, KITTI [2] for autonomous driving, and SQuAD for language-based question answering [3] . In robotics, prior benchmarks have greatly reduced the cost of iterating and developing algorithms. Examples include OpenAI Gym [4] , DeepMind Control Suite [5] , and MetaWorld [6] . However, all of these benchmarks focus on pure reinforcement learning (RL) with dense shaped rewards, limiting their application in long-horizon manipulation tasks where accurately defining reward functions is challenging.While crafting reward is difficult, obtaining expert trajectories, such as those from human demonstrations, is relatively straightforward. This advantage has boosted the popularity of demonstrationdriven methods within the robot learning community, manifesting as both imitation learning (IL) [7, 8, 9, 10, 11, 12, 13] and demo-driven RL [14, 15, 16, 17] . To support the research of building demo-driven agents, RLBench [18] was created with a wide range of single-arm fixed manipulation tasks with expert demonstrations generated by motion planners. Using motion planners allows RLBench to generate a large amount of demonstration data purely in simulation, however, the output trajectories are often either unnatural due to the inherent randomness in sampling-based planners, or have an unrealistically narrow trajectory distribution when compared to noisy real-world human demonstrations. Moreover, the community-progress is beginning to plateau on a large number of RLBench tasks, in particular with recent 3D next-best pose agents [10, 11, 16, 17, 19, 20, 21] .These limitation highlights the need for a new benchmark which provides: (1) more natural demonstrations like those seen in real-world robot data and (2) a set of new challenging tasks where state-of-the-art algorithms are likely to perform poorly. To this end, we present BiGym, a demodriven mobile bi-manual manipulation benchmark with a humanoid embodiment. BiGym covers 40 visual mobile manipulation tasks, ranging from simple tasks like moving plates between drainers to interacting with articulated objects such as dishwashers (see Figure 1 ). Unlike prior humanoid benchmarks [22, 23] that focus only on RL with dense shaped reward functions, which may lead to undesired behaviors [24] , we provide for each task only sparse rewards but with 50 demonstrations, allowing evaluation of both IL and RL algorithms. Additionally, compared to previous benchmarks that rely on expert demonstrations generated by planners [18] , the human-collected demonstrations in BiGym are much more realistic and multi-modal (see Figure 3 ), better reflecting the trajectories of real-robot movements. Finally, BiGym considers locomotion and mobile bi-manual manipulation challenges separately; specifically, BiGym allows users to switch between the whole-body mode, which jointly considers locomotion and manipulation, and a bi-manual mode, which focuses on upper-body mobile manipulation while controlling the lower body with fixed controllers (see Figure 2 ). This separation of action modes enables researchers to better investigate and benchmark the capability of various algorithms with different focuses, i.e., locomotion control and mobile bi-manual manipulation solely. Code for BiGym is available on our project website. 
related works: 
 With the rapid progress in robot learning algorithms, the role of benchmarks has become crucial as a tool to understand the effect of various algorithmic design choices and compare algorithms in the same setup. There have been a series of benchmarks for complex manipulation tasks. Most of the existing benchmarks consider a single-arm manipulation scenario. The IKEA furniture assembly environment [25] , BEHAVIOUR [26] , and Habitat [27] provide a wide range of long-horizon household object (mobile) manipulation tasks. They emphasise long-horizon planning capabilities but employ abstract low-level actions that overlook physical interactions. Some benchmarks focus on more realistic settings with physics interactions [18, 28, 29, 30, 31, 32] and mainly support training RL agents. Notably, James et al. [18] provide APIs to generate expert demonstrations with motion planners. As a result, it is widely used for benchmarking IL [10, 11, 21] and demo-driven RL algorithms [16, 17, 19, 20] . Concurrent to our work, RoboCasa [33] constructs realistic environments with human demonstrations, but only for single-arm tasks. Unlike these benchmarks that only consider a single-arm manipulation setup, BiGym provides a variety of mobile bi-manual manipulation tasks.Bi-manual manipulation benchmarks consider controlling two arms or floating dexterous hands to interact with the environment [34, 35, 36] . More recently, benchmarks for humanoid robots have been introduced. For instance, LocoMujoco [22] focuses on locomotion control of different types of humanoids with two arms, but does not include manipulation tasks. As a concurrent work, HumanoidBench [23] focuses on benchmarking RL algorithms with task-specific shaped rewards on 15 locomotion and 12 manipulation tasks. In contrast, BiGym supports benchmarking both IL and RL algorithms by providing 40 tasks with human-collected demonstrations. These demonstrations exhibit realistic noisy trajectories that cover a wider data distribution compared to planner-generated demonstrations [18] , thus enabling the evaluation that better reflects the real-world performance of algorithms. We provide a detailed comparison across benchmarks in Table 1 . 
bigym: 
 We present BiGym, a demo-driven mobile bi-manual manipulation benchmark. BiGym consists of 40 mobile bi-manual manipulation tasks, ranging from simple target reaching to complex dishwasher cleaning tasks. To evaluate IL and demo-driven RL algorithms in a realistic scenario with noisy, multimodal demonstrations, BiGym provides human-collected demonstrations for all tasks. We describe which challenges BiGym presents (see Section 3.1), the simulation platform (see Section 3.2), and details on human demonstration datasets (see Section 3.3) and tasks (see Section 3.4).We design BiGym to pose the following challenges:Partial Observability. BiGym tasks are formulated as a partially observable Markov decision process (POMDP) [37] observations {o t , s t } T t=1 and actions {a t } T t=1 , which is non-trivial given the curse of history and the curse of dimensionality of POMDPs [38, 39] .Complex Task Space. Mobile bi-manual manipulation introduces a much more complex task space compared to fixed single-arm settings. This is because, with the presence of dual arms and the mobility of the agent, there may exist multiple ways of solving a single task. For example, to grasp a cup on the side of a table and put it into the closed drawer, the robot can consider several different ways: (1) pick up the cup with one hand, pull the drawer with the other hand, and put the cup into the drawer; (2) pull the drawer with one hand, pick up the cup with the same hand, and put the cup into the drawer. The mobility of the robot also allows navigating to the target with different routes. As a result, even if a mobile robot has a comparable number of degrees of freedom to a fixed robot, the task space is highly multi-modal and significantly more complex. To test the capabilities of robot learning algorithms in such scenarios, BiGym offers a wide range of tasks featuring complex task spaces and human-collected demonstrations with diverse modalities.Long Task-Horizon and Sparse Reward. In common household scenarios, the agent will need to perform long-horizon tasks, which are composed of a series of sub-tasks, which require both task-level planning and low-level motion planning. For example, to load plates into the dishwasher, the agent should correctly locate the plates, open the dishwasher, pull out the trays, accurately put the plates on the trays, and finally, close the door. In addition, due to the complex task space, it is extremely difficult to properly define reward functions. If one can define such rewards, the agent may easily fall into local minimas by learning to exploit such sub-optimal shaped rewards. Thus, BiGym instead provides a set of sparsely-rewarded tasks along with noisy human-collected demonstrations, to evaluate the performance of IL and demo-driven RL algorithms in a more realistic setup.Realistic Multi-Modal Demonstrations. In contrast to prior benchmarks that generate expert demonstrations with motion planners [18, 31, 40] , BiGym provides human-collected demonstrations that are highly noisy and multi-modal. Specifically, we design BiGym tasks to be solvable in multiple ways to induce a multi-modal demonstration distribution. For instance, in reach target multi modal task, reaching the target can be achieved with either left or right hand, as shown in Figure 2(b) . This design enables us to evaluate the capabilities of robot learning algorithms using more realistic demonstrations, rather than synthetic demonstrations consisting of unnatural trajectories (see Figure 3 ).We build BiGym simulation environments based on MuJoCo [42] (see Figure 2 (a) for the illustration of the whole system). We brief the core design choices below and more details are in Appendix A.Humanoid Body Configurations. We implement the platform with the Unitree H1 robot given its publicly available model [43] . As the original H1 comes with no grippers, we attach an additional Robotiq 2F-85 gripper with an actuated wrist joint to each arm. We note that it is easy to swap the parallel gripper with other dexterous manipulators, but we leave it for future study as we observe that parallel grippers are sufficient for current tasks. Observation Spaces. As shown in Figure 2 (a), we mount three cameras on the robot: the forehead, the left wrist, and the right wrist. Each camera can generate both RGB and depth observations, which supports a diverse types of algorithms which use either type of observation. As a result, the observation space is defined as O = {I head , I left , I right , D head , D left , D right , s proprio }, where I is the RGB image, D is the depth image, and s proprio is the proprioception state of the robot. Additional observations, e.g., the gripper poses and robot poses, can also be easily obtained if required.Action Modes. It remains unclear to the robotics community what action modes are the best for complex embodiment in mobile bi-manual manipulation tasks. Thus in BiGym, we provide flexible configurations for users to customise the action modes they want to use, and leave the choice to the users. Specifically, we provide two off-the-shelf action modes: the whole-body action mode and the bi-manual action mode, with either delta or absolute actions. For the whole-body action mode, we allow full control of the humanoid joints. This allows studying whole-body manipulation with locomotion. With the bi-manual action mode, we simplify the control by treating the lower-body of the humanoid as an omni-directional floating base controlled by classic controllers. In this case, we can focus on upper-body bi-manual mobile manipulation skills.Scenes. The scenes in BiGym are created from MuJoCo MJCF models using a custom objectoriented API based on dm control [5] . All MJCF models provided in BiGym were created from publicly available 3D models. Many other 3D models were processed to be used in BiGym: meshes were decimated to reduce the total number of polygons, moving parts of articulated objects were separated, required joints and actuators were added, and convex collision meshes were created.Currently, BiGym provides 46 high-quality assets that could be reused to facilitate the creation of new environments. In addition to rigid object models, BiGym offers a set of articulated models, such as a dishwasher and customisable kitchen modules.API. The interface for the benchmark follows the standard Gymnasium APIs [44] for training IL and RL agents. A typical workflow of RL agents training is demonstrated in Figure 4 .One of the key design choices in BiGym is to provide a fixed number of human-collected demonstrations for each task. This allows BiGym benchmark to better reflect the challenges of real-world robot learning, which involves dealing with noisy, multi-modal demonstrations in contrast to synthetic demonstrations generated by motion planners [18] . We describe BiGym's human demonstration dataset collection and management system as below and more details are in the appendix.Demo Collection Pipeline. We use VR 2 to collect demonstrations by virtually tele-operating the H1 robot in simulation (using the 6 DoF poses of the headset and controllers). The headset pose controls the position and orientation of the H1 body, and the 6 DoF poses of the controllers are used to operate the arms. We solve the inverse kinematics for each arm and then reorient the grippers to match the orientation of the respective controller.Down-Sampling Demonstrations. The control frequency of demonstrations can significantly affect the horizon length of tasks and the ability to capture fine grained control, both of which can greatly affect task success [45] . To provide users with the flexibility in selecting action frequencies, we capture demonstrations at the frequency of physics calculation (500 Hz) and provide the functionality to down-sample the demonstrations to a desired frequency (20-500 Hz).Demonstration Management. To minimise the use of storage for saving demonstrations, we save lightweight demonstrations that only contain control signals (actions). We then provide a tool that enables users to pull such lightweight demonstrations and replay them to obtain full demonstrations with user-specified observations such as RGB or depth images. These demonstrations are cached in users' local storage so that users do not have to re-download or replay the same demonstrations again.Tools. BiGym provides two tools for demonstration management. The demo player tool provides various demonstration-related functionalities such as downloading, deleting, verifying, replaying at different frequencies, converting, and re-recording demonstrations. The demo recorder enables users to easily collect data by streamlining the process of recording demonstrations with VR.BiGym provides 40 high-quality tasks with human-collected demonstrations to support the research on household mobile bi-manual manipulation. We describe the tasks and their configurations below.Reach Target Tasks. Different from the standard reach target tasks in single-arm settings, we consider three variants of reach target tasks:(1) reach target single: The robot must use a specified wrist to reach a coloured target.(2) reach target multi modal: The robot must reach the target with either left or right wrist. This induces a multi-modal demonstration distribution of reaching the target with different hands. We expect the policy to understand this multi-modality during training.(3) reach target dual: In this task, the robot must reach two targets, one with each arm. The success criteria require both wrists to be aligned with corresponding targets. Once this criteria is met, the targets are highlighted to provide visual feedback.Tasks. We then consider table-top manipulation which requires the robot to interact with rigid-body objects, e.g., plates and cups. The challenge here is that the robot should be able to identify remote objects and perform manipulation tasks that require moving the base together with the arms. We introduce the following tasks:(4) stack blocks: Move blocks across the table, and stack them in the target area.(5) move plate: Move the plate between two draining racks. ( 6) move two plates: Move two plates simultaneously from one draining rack to the other. ( 7) flip cup: Flip the cup, initially positioned upside down on the table, to an upright position. (8) flip cutlery: Take the cutlery from the static holder, flip it, and place it back into the holder.In addition to simple single-object manipulation tasks, we introduce complex manipulation tasks that require interaction with articulated objects, where it is crucial to understand the object's kinematics to perform constrained motion planning. The scenarios include: Dishwasher and Kitchen Counter tasks.Dishwasher Tasks. We consider a set of tasks which require interactions with the articulated dishwasher, ranging from sliding trays to long-horizon unloading tasks: Kitchen Counter Tasks. In addition, BiGym considers a more complex kitchen counter scenario with multiple challenging articulated objects, e.g., the cupboard, the drawer, etc. Similar to the dishwasher tasks, we provide a range of short and long-horizon tasks as below:EQUATION(40) store groceries upper: Place a random set of groceries in cabinets and shelves on the wall.Reward Functions. We provide sparse rewards for all the tasks based on success detector: a reward of 1 is given for reaching the successful criteria and 0 otherwise. Details are available in Appendix B. 
experiments: 
 The contribution of the paper is BiGym. However, in this section, we aim to validate that current algorithms can attain some degree of performance on all BiGym tasks, even if minimal. To this end, we conduct experiments with both state-of-the-art IL and demo-driven RL algorithms. Specifically, we focus on the following seven general robot learning algorithms:IL Algorithms. We aim to investigate how different policy representations contribute to the final performance of the algorithms on BiGym, which provides highly noisy and multi-modal demonstrations. In pursuit of this goal, we consider the following algorithms: standard Behaviour Cloning (BC), Action Chunking Transformers (ACT) [9] which trains a transformer model [46] to predict a sequence of actions, and Diffusion Policies [8] which trains a diffusion model to approximate the expert action distribution. In particular, we do not benchmark against the popular 3D next-best pose agents [10, 11, 16, 17, 19, 20, 21] since they reply on heuristic-based key-frame extraction methods which only apply to single fixed arms [17] ; thus, they are not currently applicable to the mobile bi-manual manipulation morphology.RL Algorithms. We mainly consider demo-driven RL algorithms which support training with expert demonstrations. Specifically, we focus on off-policy algorithms and offline RL algorithms that have demonstrated good capabilities in online settings. We consider the following algorithms:DrQV2 [47],Advantage Weighted Actor-Critic (AWAC) [48] , Implicit Q-Learning (IQL) [49] , and Coarse-to-fine Deep Q-Network (CQN) [50] . We note that BiGym tasks can be extremely challenging for RL algorithms due to their sparse reward, partial observations, and complex dynamics. To provide a reference for future studies, we provide the results of all the methods as-is with the common set of hyperparameters, instead of tuning their performance for individual BiGym tasks.We provide experimental results and discussions in Appendix C. 
discussions: 
 Limitations. Currently, BiGym has the following main limitations. Firstly, although BiGym has posed a series of challenging tasks of whole-body mobile bi-manual manipulation, most tasks are still at the skill level, requiring less complex long-horizon task and motion planning, except for the " long" tasks, e.g, unload cups long. To fully benchmark the capability of algorithms in an ultimate household environment, such tasks are necessary. In addition, BiGym only supports the Unitree H1 robot at the moment. More different embodiments will be included for future tasks.Opportunities and Future Works. BiGym presents various future research opportunities, including but not limited to: (1) exploring better network architectures for approximating multi-modal noisy human demonstrations; (2) studying better belief estimation mechanisms for the POMDP in the mobile manipulation context; (3) investigating better collaboration modes between arms on mobile platforms; (4) whole-body motion planning which improve the efficiency and performance of mobile agents while navigating in cluttered environments.We introduce BiGym, a new and challenging benchmark for demo-driven mobile bi-manual manipulation. BiGym covers 40 challenging tasks of mobile bi-manual manipulation, ranging from simple target reaching to complex dishwasher manipulation tasks. Built upon the humanoid embodiment of Unitree H1 robot, BiGym allows users to flexibly customise the action modes: the whole-body mode and the bi-manual mode. Furthermore, we provide multi-modal and noisy human-collected demonstrations for all BiGym tasks, exhibiting realistic trajectories compared to synthetic ones generated by motion planners. In our experiments, we validate the usability of BiGym by benchmarking state-of-the-art IL and RL algorithms.In this section, we provide additional details about BiGym.Observation Spaces. For image observations, we allow users to specify the resolution of the images, where the default resolution is 84×84. Higher resolution may allow learning better policies, but we find the default value works across tasks. In the whole-body mode, the proprioception state s fb proprio ∈ R 76 = {s rb qpos , s rb qvel , s grip }, where s rb qpos ∈ R 37 is the joint angle positions of the robot, s rb qvel ∈ R 37 is the corresponding velocities, and s grip ∈ R 2 is the gripper opening amount of both grippers. On the contrary, the bi-manual mode greatly simplifies the locomotion by replacing the lower-body control with a predefined controller, i.e., a floating base. This reduces the dimension of s rb qpos and s rb qvel to s bm qpos ∈ R 29 and s bm qvel ∈ R 29 . Furthermore, an additional state s base = (x, y, z, θ) ∈ R 4 is included in s proprio to indicate the position and orientation of the floating base. As a result, s bm proprio ∈ R 64 = {s bm qpos , s bm qvel , s base , s grip } Action Spaces. In the whole-body mode where the agent has the full control over the body, an action space A wb ∈ R 23 is defined as A wb = {A arms , A legs , A torso , A grip }, where A arms ∈ R 10 controls both arms, A legs ∈ R 10 controls the legs, A torso ∈ R 1 controls the main torso joints, and A grip ∈ R 2 controls the opening amount of grippers. In bi-manual mode, the user controls the floating base instead of the leg joints. Therefore the action space becomes A bm ∈ R 16 = {A arms , A base , A grip }, with A base ∈ R 4 controlling the delta actions (δx, δy, δz, δθ) of the base.Simulation Performance. We present the simulation speed in Figure 5 . The benchmark was done on a headless server of NVIDIA L4 GPU and Intel Xeon Gold 6438Y+ CPU, in a single process. Benefiting from the highly optimised MoJoCo engine, BiGym runs at around 400FPS to 1400FPS depending on the number of cameras. The performance could be further improved by using parallel environments or MuJoCo XLA, which speeds up the execution with XLA just-in-time compilation. In this section, we detail the definitions of all task success detectors.Reach Target Tasks.(1) reach target single: The distance from the robot left wrist to the target is smaller than a tolerance value. The default tolerance value is 0.1.(2) reach target multi modal: The distance from either the robot left wrist or the right wrist is smaller than a tolerance value. The default tolerance value is 0.1.(3) reach target dual: The distance from the left wrist and the right wrist to their corresponding goals are smaller than a tolerance value. The default tolerance value is 0.1.(4) stack blocks: The three blocks are stacked on each other, i.e. in collision with each other, in a target region on the table.(5) move plate: The following conditions must be met: (a) the orientation of the plate is upright, (b) the plate is not colliding with the table, (c) the plate is colliding with the rack and (d) the robot has released the plate from its gripper. ( 6) move two plates: Transfer two plates to the target rack and meet all conditions similar to the move plate task. (13) dishwasher load plates: All plates are in collision with the bottom tray of the dishwasher and the robot has released the plates from it's grippers. ( 14) dishwasher load cups: All cups are in collision with the middle tray of the dishwasher and the robot has released the cup from its gripper. ( 15) dishwasher load cutlery: All cutlery are in collision with the dishwasher cutlery basket and the robot has released the cutlery from its gripper. ( 16) dishwasher unload plates: All plates are moved from the bottom tray of the dishwasher to the drainer on the table, and placed onto the rack positioned on the counter-top. (17) dishwasher unload cups: All cups are moved from the middle tray of the dishwasher to the cabinet and in collision with the cabinet counter. Additionally, all cups are released from the robot gripper.(18) dishwasher unload cutlery: All cutlery are moved from the dishwasher basket to the tray and are in collision with the tray. (19) dishwasher unload plate long: All conditions of dishwasher close and dishwasher unload plates must be met. Additionally, all plates are placed inside the wall cabinet. Finally, all joint angles of the wall cabinet doors are close to 0 with a tolerance. The default value is 0.1. (20) dishwasher unload cup long: Similar to dishwasher unload plate long but with cups. (21) dishwasher unload cutlery long: Similar to dishwasher unload cutlery long but with cutlery and instead of the cabinet, the cutlery must be placed in a closed drawer.Kitchen Counter Tasks. 24) drawers open all: The joint angles of all drawers are close to 1 with a tolerance value. The default value is 0.1. ( 25) drawers close all: The joint angles of all drawers are close to 0 with a tolerance value. The default value is 0.1. ( 26) wall cupboard open: The joint angle of two doors of the wall cupboard is close to 1 with a tolerance value. The default value is 0.1. (27) wall cupboard close: The joint angle of two doors of the wall cupboard is close to 0 with a tolerance value. The default value is 0.1. ( 28) cupboards open all: The joint angles of the two doors and all drawers of the kitchen set are close to 1 with a tolerance value. The default value is 0.1. ( 29) cupboards close all: The joint angles of the two doors and all drawers of the kitchen set are close to 0 with a tolerance value. The default value is 0.1. (30) take cups: All cups are in collision with the counter on the table and the robot has released the cups from its gripper. (31) put cups: All cups are in collision with the cupboard shelf and the robot has released the cups from its gripper. (32) pick box: The box is in collision with the counter and the robot has released the box from its grippers. (33) store box: The box is in collision with the shelf and the robot has released the box from its grippers. (34) saucepan to hob: The saucepan is in collision with the hob and the robot has released the saucepan from its grippers. (35) store kitchenware: Both the saucepan and the pan are in collision with the shelf, and the robot has released the objects from its grippers. (36) sandwich toast: All the following conditions must be met: (a) The sandwich is in collision with the pan. (b) The orientation of the sandwich is either up or down. (c) The pan is in collision with the hob. (37) sandwich flip: Similar to sandwich toast. In addition the sandwich orientation must be flipped. (38) sandwich remove: All the following conditions must be met: (a) The sandwich is in collision with the board. (b) The orientation of the sandwich is either up or down. (39) store groceries lower: All items are in collision with the shelf of the cabinet below the counter. Additionally, all items are released from the robot gripper. (40) store groceries upper: All items are in collision with the shelf of the cabinet on the wall. Additionally, all items are released from the robot gripper.We implemented all algorithms using PyTorch [51] .ACT. Following the official implementation 3 , we train a ResNet-18 encoder [52] to extract visual features and a transformer model to predict a sequence of actions. Inputs to the transformer model are multi-view image features and proprioceptive features from a conditional variational autoencoder (CVAE) [53] . During execution, we use receding horizon control for all tasks by training the policy to output an action sequence of length 16 and executing only the first step in the sequence. Following the official implementation, we enable temporal ensembling to improve the smoothness of the policy. Diffusion Policy. Our implementation of Diffusion Policy closely follows the official release 4 . To be consistent with ACT, we use ResNet-18 as vision encoders for all camera observations. As discussed in Chi et al. [8] , the Diffusion Policy is susceptible to the choice of backbones and their parameters: the UNet-1D backbone might outperform the causal transformer backbone in certain tasks and vice versa. Thus, we benchmark both the UNet-1D backbone and the causal Transformer backbone, and report the highest achieved performance between them in our main results. In addition, for all Diffusion Policy variants, we use action sequence length of 16 and execution length of 1, which we find to achieve strong performance in general. Following ACT, we also enable temporal ensembling for Diffusion Policies, which we find to be crucial for stabilising the inference.Other Baselines. For BC and demo-driven RL baselines, we adopt the same network architectures which consist of an CNN-based image encoder and a fully-connected output head. The image encoder encodes each camera image with 3 layers of CNNs, each has kernel size 3 and 32 channels. In between the layers, we use SiLU activation function [54] and layer normalisation [55] . We flatten the CNN features and concatenate with the proprioception states to form the final observation feature vector. The head has 2 fully connected layers of dimension 512, and bottlenecks the output to dimension 64. After the bottleneck layer, we normalise the output with layer normalisation followed by tanh activation.Training Details. We use a frame stack of 4, Adam optimiser [56] with a learning rate of 0.0001, and batch size of 256 for all IL and demo-driven RL algorithms. In addition, specifically for all RL algorithms, we follow AW-Opt [57] and keep the demonstration ratio for each batch to be 50% by using a separate demonstration replay buffer. This helps the exploration of the agent during sparse reward settings. We run 150K training steps for IL algorithms and 100K steps for demo-driven RL methods. We observe that all algorithms converge after 100K steps and longer training does not give additional performance boost. All results are averaged over the last 3 checkpoints.In Table 2 , we provide the performance of IL and demo-driven RL methods on 40 BiGym tasks. Overall, we observe that BiGym tasks are challenging and pose a variety of unique and interesting challenges for future researches. We outline our observations as below:The mobile manipulation of articulated or rigid-body objects is challenging for the state-ofthe-art algorithms. BiGym has presented a series of tasks that involve interactions with articulated or rigid-body objects, which typically require high-precision manipulation, e.g., move two plates, cupboards open all, and stack blocks. When coupled with the mobile base, these tasks become more challenging because (i) accurately measuring the grasping poses while moving is hard, and (ii) correctly estimating the posterior distribution of the states given partial history information of a POMDP is difficult. For instance, while we observe that ACT and Diffusion Policy achieve the overall best performance across all tasks, they still struggle in the seemingly simple tasks, e.g., stack blocks, which requires the agent to pick 3 cubes, and stack them to a target region located on the other side of the table. We believe a more robust system with stronger memory mechanisms to track the "beliefs", i.e., estimating the posterior distributions of the states, is necessary to solve such challenging BiGym tasks.The long-horizon tasks in BiGym requires both task and motion planning of the agent. BiGym introduces a series of long-horizon tasks, e.g., dishwasher unload cups long and put cups. All algorithms fail on these tasks. Intuitively, these tasks are composed of multiple sub-tasks, and the difficulty level of achieving these long-horizon tasks grows exponentially at the same time. As model-free agents, our baselines are not capable of performing task-level reasoning. Hierarchical methods [12] could work as a better policy representation for these tasks. We leave it for future study.The complex policy space of BiGym requires carefully designed agent architectures. We observe that almost on all tasks, ACT and Diffusion Policy achieves superior performance to BC and demodriven RL baselines. We hypothesize this is because both ACT and Diffusion Policy utilise powerful policy classes based on generative representation learning, i.e., CVAE and Diffusion models, and they also use expressive network architecture such as transformers or UNets. In contrast, BC and all demo-driven RL approaches use simple CNN + MLP architectures. It is likely that these weaker architectures struggle to deal with the complex multi-modal noisy demonstrations introduced in BiGym. We believe this can motivate future research on finding appropriate policy representations for mobile bi-manual manipulation.Demo-driven RL approaches struggle with the complex task space and sparse reward in BiGym. We observe that demo-driven RL algorithms fail on most of the BiGym tasks. For instance, CQN [50] , which exhibits strong performance on fixed single-arm demo-driven RL setups, fails to solve most of the BiGym tasks. It is notable that all RL algorithms only achieve non-zero success rates on simple tasks with little interaction with the objects, e.g., reach target single and top drawer close, and completely fail to solve all the other tasks. We hypothesise this is because (i) the presence of mobile base makes it more difficult for agents to explore meaningful state. e.g. an erroneous base turning action can easily cause robot to lose view of the objects. and (ii) RL agents struggle to learn value-functions on long-horizon BiGym tasks with sparse reward.Valve Index VR headset, two controllers and two base stationshttps://github.com/tonyzhaozh/alohahttps://github.com/real-stanford/diffusion_policy 
: 
 : BiGym focuses on mobile manipulation with home assistance humanoids. We provide 40 tasks ranging from simple mobile target reaching to complex dishwasher manipulations. Specifically, each task comes with demonstrations recorded by human demonstrators and can be used to benchmark both imitation learning and reinforcement learning algorithms. 
 
 Appendix:  


Decision: Strengths:
- Interesting and timely topic of bimanual manipulation, with a rich set of tasks consisting of both bimanual and whole-body manipulation.
- Reviewers agree that the baselines and experiments are insightful, showing gaps in SoTA on the proposed benchmark. 

Weaknesses:
- Reviewers raise concerns about the whole-body motion demos and controller and its impact on the benchmark’s realism, especially sim2real. 
- The experiments and analysis should be in the main paper, instead of in the appendix. Details about the controller used for collection demonstrations, and a limitation section should also be added to the paper.

Post-rebuttal:

The key points addressed in the rebuttal include:
- Limitation section: Authors have added a limitations section to the paper.
- Experimental details: Authors have described details of the controller and teleoperation.
However, some concerns raised by reviewers around sim2real still remain.

Reviews: review: title: title: Extensive mobile bimanual benchmark with 40 skills of ranging difficulty, but baseline evaluation should be included in the main paper and expanded.
paper_summary: paper_summary: The paper presents a mobile bimanual benchmark in simulation with 40 human-demonstrated skills (50 demos each) of ranging difficulty. The skills include reach target, table-top manipulation, dishwasher, and kitchen counter tasks. The skills are benchmarked using 3 imitation learning baselines and 4 RL baselines. The results demonstrate that ACT and diffusion policy imitation learning algorithms perform best but leave a lot of room for improvement, and even completely fail on some tasks that are long-horizon and/or involve mobile navigation and manipulation. The conclusions are we need algorithms with longer temporal memory and for RL potentially hierarchical approaches to deal with the sparse reward on long-horizon tasks.
main_review: main_review: Strengths:
* The paper topic is pertinent to the research community as mobile manipulation/humanoid hardware has become popular. A benchmark in this space it timely and needed by the research community.
* Generally the paper is well written and easy to read.
* The tasks and design choices are detailed, explained, and illustrated.
* The experiments clearly show a gap on the benchmark for the considered baselines and the discussion of the results is reasonable and insightful.

Weaknesses:
* The experimental results (or some subset of them) should really be in the main paper. There is a lot of repetition currently in the main paper regarding the 40 tasks and the contents of the benchmark. I would cut down on some of the repetition in favor of including the experimental results and discussion.
* The discussion suggests that longer temporal memory could help ACT/diffusion policy do better on the long-horizon tasks, while hierarchical RL methods could help with the sparse reward. Although I do not expect these changes to completely solve the benchmark (nor should they), it would be pertinent to show that these are reasonable directions. Adding some memory to diffusion policy for example and trying out an existing hierarchical RL algorithm seems like a reasonable course of action.
* It would be great to demonstrate that the simulation findings transfer to hardware.
* The limitation section is missing.

* The following is a non-exhaustive list of typos and grammatic errors.
1. Line 127: 'which supports a diverse types of algorithms which use either type of observation' - incorrect grammar and poor wording.
2. Line 255: 'since they reply on'.
3. Line 569: 'The long horizon tasks in BiGym requires'.
4. Line 574: 'We leave [this] for future work.'
5. The references have some consistency issues and need to be proofread for typos. For example, conference acronyms are sometimes included (e.g., [8]) and sometimes not (e.g., [1]). Some conference names were not capitalized (e.g., [1]). When possible, the conference venue should be cited instead of ArXiv. The references should in general be proofread (e.g., capitalization errors in the title of [4, 11], the year is listed twice in [42], etc.).
questions: questions: In addition to the weaknesses listed above, I have the following questions for the authors.
1. Is 50 demonstrations per skill enough, particularly for long horizon tasks? How was this number chosen?
2. Is saving action-only demonstrations that can be replayed later to get the observations sufficient? During replaying, are the rollouts of the demonstrations the same or stochastically different?
limitations: limitations: 1
review_summary: review_summary: Overall, I think the mobiel bimanual benchmark is a good addition to the research community. My main points of feedback are: include the experiments in the main paper, try out the couple of ideas suggested (temporal memory for imitation learning and hierarchical RL), and demonstrate that the skills from the simulation benchmark transfer in some way to hardware.
title: title: New MuJoCo-based mobile bi-manual manipulation benchmark
paper_summary: paper_summary: The authors proposed BiGym -  a benchmark based on MuJoCo and learning environment for mobile bi-manual manipulation with 40 tasks, with 50 human-collected demos for each task. It provides visual inputs as RBGD with overhead and wrist camera views. The paper provides extensive benchmarking results of current SOTA RL and IL methods.
main_review: main_review: **Pros**

- Timely support for both learning paradigms - provide collected demo for each task to support benchmarking both RL and imitation learning algorithms.

- Flexible as a benchmark - Supports both locomotion and mobile bi-manual manipulation, and allows switching between whole-body and bi-manual to focus on whole-body locomotion with manipulation, or focus on manipulation specifically.

- Includes relatively extensive benchmarking on current state of the art algorithms in IL and RL on the provided tasks.

- Open-sourced benchmarking will benefit the community and push for better benchmarking for mobile bimanual tasks!

- Provide a longer task horizon and the option of treating the lower-body as floating base controlled by the classical controller, to focus on upper-body bi-manual skills.

- The supplement provides extensive benchmarking experiment results between IL methods (BC, ACT, Diffusion Policy) and RL (DrQV2, AWAC, IQL, CQN) on the provided tasks, and finds that RL generally suffer in performance compared to IL methods.  	


**Cons**

- Right now, the simulator only supports Unitree H1 humanoid with robotiq parallel gripper, it would be great for future iterations to include dextrous hands and dextrous manipulation tasks that require dexterous hands. 

- The tasks included in this benchmark is fairly simple and easy for humanoid/bimanual setup. Future iterations could benefit from including tasks that require more whole-body manipulation, including using the robot arm in addition to robot end effector to complete the tasks. 

**Others**

- It would be very helpful for the authors to provide some evaluation experiments on sim2real transfer gap. For example, some of the table-top task setup could be replicated in real world, and some preliminary real-world eval results on the policy trained from simulation would be useful to provide insights on how physically transferable the policy trained from simulation physics to real world.
questions: questions: - It would be very helpful for the authors to provide some evaluation experiments on sim2real transfer gap. For example, some of the table-top task setup could be replicated in real world, and some preliminary real-world eval results on the policy trained from simulation would be useful to provide insights on how physically transferable the policy trained from simulation physics to real world.
limitations: limitations: 2
review_summary: review_summary: Great system paper with a new benchmark to push for mobile bi-manual manipulation, but more real-world result is needed to distinguish it from concurrent related works.
title: title: Review
paper_summary: paper_summary: The paper presents a simulated benchmark and demonstration dataset for bimanual manipulation tasks based on the UniTree H1 humanoid robot. The benchmark features a variety of bimanual tasks situated in a kitchen environment. It supports both full-body control and bimanual mode with a floating base. The paper also includes results in comparing popular behavior cloning methods and offline RL methods.
main_review: main_review: Strengths:
+ Timely research. A benchmark suite on humanoid bimanual manipulation with demonstration data is of great interest to the robot learning community.
+ Code will be released upon publication.
+ The paper presents baseline results for most popular BC and offline RL methods, which will be useful reference points for future research.


Weaknesses:
- While-body control mode is claimed as a main contribution of the work. However, I can’t seem to find any details about the full-body controller while the robot is being teleoperated. How is it implemented? From the demonstration video, it seems that the leg and the torso are independently controlled: torso translation and rotation do not seem to correlate with the leg movement. For example, the legs maintain the same gaiting while the robot is crouching, which seems physically impossible. Also the upper body rotation seems to be limited to yaw and has no roll or pitch in the videos. 
- If the whole-body control mode is indeed simplified, I’d like to better understand how this affects the realism of the benchmark. For example, most full-body tasks such as lifting heavy items require complex coordination of upper and lower body motion. If the benchmark cannot expose these kinds of challenges, would algorithms tested in simulation reflect their real-world performances?

=== Post rebuttal:

The authors have addressed some of my concerns, e.g., all the videos demonstrated are actually bimanual mode, not whole-body mode. This left me wondering what the whole-body control mode would look like. Regardless, I believe Bigym would be of wide interest to the CoRL community and I therefore have updated my rating to weak accept.
questions: questions: See "weaknesses"
limitations: limitations: 2
review_summary: review_summary: I'm willing to change my rating if the authors can adequately address my question regarding full-body control.
title: title: The paper is structured and easy-to-follow, however, there are a few concerns need to be addressed.
paper_summary: paper_summary: The paper presents a benchmark data set for (mobile) bimanual manipulation tasks. The data is collected from human demo with a comprehensive bimanual tasks.
main_review: main_review: This paper introduces BiGym, a new benchmark and learning environment for mobile bi-manual robotic manipulation. 

The paper is easy-to-follow, and I appreciate a lot on the literature review as it distinct itself from existing works.

The key contributions and strengths of BiGym include:
- A diverse set of 40 tasks in home environments, ranging from simple target reaching to complex kitchen cleaning scenarios. This provides a good breadth of challenges.
- Human-collected demonstrations for each task, which capture more realistic and multi-modal trajectories compared to synthetic planner-generated demos used in some previous benchmarks. 
- Support for multiple observation types including proprioceptive data and visual inputs from 3 camera views. This flexibility is valuable.
- Benchmarking of state-of-the-art imitation learning and demo-driven reinforcement learning algorithms to validate the usability of the environment.
- A flexible framework allowing users to switch between whole-body control and simplified bi-manual control modes.

Some comments:
- The robot's bipedal movement seem unrealistic from the simulation? I therefore don't think the data is useful for whole body mobile manipulation
- I would like to see a bit of reasoning for the selected tasks.
- I also didn't see a limitation section?
questions: questions: The bipedal movement seems very unrealistic, how are you going to resolve this issue?
The reason of selecting the tasks is unclear to me.
Elaboration on limitation?
limitations: limitations: 1
review_summary: review_summary: I think this is a good paper for CoRL if my concerns are addressed.