Mobile ALOHA: Learning Bimanual Mobile Manipulation using Low-Cost Whole-Body Teleoperation

Authors: zipeng  fu
tony z zhao
chelsea  finn
Venue: robot-learning_org_CoRL_2024_Conference
Published: 2024-06-07 13:57:22

Abstract: Imitation learning from human demonstrations has shown impressive performance in robotics. However, most results focus on table-top manipulation, lacking the mobility and dexterity necessary for generally useful tasks. In this work, we develop a system for imitating mobile manipulation tasks that are bimanual and require whole-body control. We first present Mobile ALOHA, a low-cost and whole-body teleoperation system for data collection. It augments the ALOHA system with a mobile base, and a whole-body teleoperation interface. Using data collected with Mobile ALOHA, we then perform supervised behavior cloning and find that co-training with existing static ALOHA datasets boosts performance on mobile manipulation tasks. With 50 demonstrations for each task, co-training can increase success rates by up to 90\%, allowing Mobile ALOHA to autonomously complete complex mobile manipulation tasks such as sauteing and serving a piece of shrimp, opening a two-door wall cabinet to store heavy cooking pots, calling and entering an elevator, and lightly rinsing a used pan using a kitchen faucet.  We will open-source all the hardware and software implementations upon publication.

Main paper: 
 introduction: 
 Imitation learning from human-provided demonstrations is a promising tool for developing generalist robots, as it allows people to teach arbitrary skills to robots. Indeed, direct behavior cloning can enable robots to learn a variety of primitive robot skills ranging from lane-following in mobile robots [2] , to simple pick-and-place manipulation skills [3, 4] to more delicate manipulation skills like spreading pizza sauce or slotting in a battery [5, 1] . However, many tasks in realistic, everyday environments require whole-body coordination of both mobility and dexterous manipulation, rather than just individual mobility or manipulation behaviors. For example, consider the relatively basic task of putting away a heavy pot into a cabinet in Figure 1 . The robot needs to first navigate to the cabinet, necessitating the mobility of the robot base. To open the cabinet, the robot needs to back up while simultaneously maintaining a firm grasp of the two door handles, motivating whole-body control. Subsequently, both arms need to grasp the pot handles and together move the pot into the cabinet, emphasizing the importance of bimanual coordination. Along a similar vein, cooking, cleaning, housekeeping, and even simply navigating an office using an elevator all require mobile manipulation and are often made easier with the added flexibility of two arms. In this paper, we study the feasibility of extending imitation learning to tasks that require whole-body control of bimanual mobile robots. Two main factors hinder the wide adoption of imitation learning for bimanual mobile manipulation. (1) We lack accessible, plug-and-play hardware for whole-body teleoperation. Bimanual mobile manipulators can be costly if purchased off-the-shelf. Robots like the PR2 and the TIAGo can cost more than $200k USD, making them unaffordable for typical research labs. Additional hardware and calibration are also necessary to enable teleoperation on these platforms. For example, the PR1 uses two haptic devices for bimanual teleoperation and foot pedals to control the base [6] . Prior work [7] uses a motion capture system to retarget human motion to a TIAGo robot, which only controls a single arm and needs careful calibration. Gaming controllers and keyboards are also used for teleoperating the Hello Robot Stretch [8] and the Fetch robot [9], but do not support bimanual or whole-body teleoperation. (2) Prior robot learning works have not demonstrated high-performance bimanual mobile manipulation for complex tasks. While many recent works demonstrate that highly expressive policy classes such as diffusion models and transformers can perform well on fine-grained, multi-modal manipulation tasks, it is largely unclear whether the same recipe will hold for mobile manipulation: with additional degrees of freedom added, the interaction between the arms and base actions can be complex, and a small deviation in base pose can lead to large drifts in the arm's end-effector pose. Overall, prior works have not delivered a practical and convincing solution for bimanual mobile manipulation, both from a hardware and a learning standpoint. We seek to tackle the challenges of applying imitation learning to bimanual mobile manipulation in this paper. On the hardware front, we present Mobile ALOHA, a low-cost and whole-body teleoperation system for collecting bimanual mobile manipulation data. Mobile ALOHA extends the capabilities of the original ALOHA , the low-cost and dexterous bimanual puppeteering setup [1] , by mounting it on a wheeled base. The user is then physically tethered to the system and backdrives the wheels to enable base movement. This allows for independent movement of the base while the user has both hands controlling ALOHA . We record the base velocity data and the arm puppeteering data at the same time, forming a whole-body teleoperation system. On the imitation learning front, we observe that simply concatenating the base and arm actions then training via direct imitation learning can yield strong performance. Specifically, we concatenate the 14-DoF joint positions of ALOHA with the linear and angular velocity of the mobile base, forming a 16-dimensional action vector. This formulation allows Mobile ALOHA to benefit directly from previous deep imitation learning algorithms, requiring almost no change in implementation. To further improve the imitation learning performance, we are inspired by the recent success of pretraining and co-training on diverse robot datasets, while noticing that there are few to none accessible bimanual mobile manipulation datasets. We thus turn to leveraging data from static bimanual datasets, which are more abundant and easier to collect, specifically the static ALOHA datasets from [1, 10] through the RT-X release [4] . It contains 825 episodes with tasks disjoint from the Mobile ALOHA tasks, and has different mounting positions of the two arms. Despite the differences in tasks and morphology, we observe positive transfer in nearly all mobile manipulation tasks, attaining equivalent or better performance and data efficiency than policies trained using only Mobile ALOHA data. This observation is also consistent across different class of state-of-the-art imitation learning methods, including ACT [1] and Diffusion Policy [5] . The main contribution of this paper is a system for learning complex mobile bimanual manipulation tasks. Core to this system is both (1) Mobile ALOHA, a low-cost whole-body teleoperation system, and (2) the finding that a simple co-training recipe enables data-efficient learning of complex mobile manipulation tasks. Our teleoperation system is capable of multiple hours of consecutive usage, such as cooking a 3-course meal, cleaning a public bathroom, and doing laundry. Our imitation learning result also holds across a wide range of complex tasks such as opening a two-door wall cabinet to store heavy cooking pots, calling an elevator, pushing in chairs, and cleaning up spilled wine. With co-training, we are able to achieve over 80% success on these tasks with only 50 human demonstrations per task, with an average of 34% absolute improvement compared to no co-training.  
related work: 
 Mobile Manipulation. Many current mobile manipulation systems utilize model-based control, which involves integrating human expertise and insights into the system's design and architecture [12, 13, 14, 15, 6] . A notable example of model-based control in mobile manipulation is the DARPA Robotics Challenge [16] . Nonetheless, these systems can be challenging to develop and maintain, often requiring substantial team efforts, and even minor errors in perception modeling can result in significant control failures [17, 18] . Recently, learning-based approaches have been applied to mobile manipulation, alleviating much of the heavy engineering. In order to tackle the exploration problem in high-dimensional state and action spaces of mobile manipulation tasks, prior works use predefined skill primitives [19, 20, 21] , reinforcement learning with decomposed action spaces [22, 23, 24, 25, 26] , or whole-body control objectives [27, 28, 29] . Unlike these prior works that use action primitives, state estimators, depth images or object bounding boxes, imitation learning allows mobile manipulators to learn end-to-end by directly mapping raw RGB observations to whole-body actions, showing promising results through large-scale training using real-world data [3, 30, 31] in indoor environments [32, 31] . Prior works use expert demonstrations collected by using a VR interface [33] , kinesthetic teaching [34] , trained RL policies [35] , a smartphone interface [36] , motion capture systems [7] , or from humans [37] . Prior works also develop humanoid teleoperation by using human motion capture suits [38, 39, 40, 41] , exoskeleton [42, 43, 44, 45] , VR headsets for visual feedbacks [46, 47, 48, 49] , and haptic feedback devices [50, 51] . Purushottam et al. develop an exoskeleton suit attached to a force plate for whole-body teleoperation of a wheeled humanoid, However, there is no low-cost solution to collecting whole-body expert demonstrations for bimanual mobile manipulation. We present Mobile ALOHA for this problem. It is suitable for hour-long teleoperation, and does not require a FPV goggle for streaming back videos from the robot's egocentric camera or haptic devices. Imitation Learning for Robotics. Imitation learning enables robots to learn from expert demonstrations [2] . Behavioral cloning (BC) is a simple version, mapping observations to actions. Enhancements to BC include incorporating history with various architectures [53, 54, 55, 3] , new training objectives [56, 57, 5, 1, 58] , regularization [59] , motor primitives [60, 61, 62, 63, 64, 65] , and data preprocessing [10] . Prior works also focus on multi-task or few-shot imitation learning, [66, 67, 68, 69, 70, 71, 72, 73] , language-conditioned imitation learning [74, 75, 55, 3] , imitation from play data [76, 77, 78, 79] , using human videos [80, 81, 82, 83, 84, 85, 86, 87] , and using task-specific structures [88, 89, 75] . Scaling up these algorithms has led to systems adept at generalizing to new objects, instructions, or scenes [90, 55, 3, 91, 92] . Recently, co-training on diverse real-world datasets collected from different but similar types of robots have shown promising results on single-arm manipulation [4, 93, 94, 95, 96] , and on navigation [97] . In this work, we use a co-training pipeline for bimanual mobile manipulation by leveraging the existing static bimanual manipulation datasets, and show that our co-training pipeline improves the performance and data efficiency of mobile manipulation policies across all tasks and several imitation learning methods. To our knowledge, we are the first to find that co-training with static manipulation datasets improves the performance and data efficiency of mobile manipulation policies. 
mobile aloha hardware: 
 We develop Mobile ALOHA, a low-cost mobile manipulator that can perform a broad range of household tasks. Mobile ALOHA inherits the benefits of the original ALOHA system [1] , i.e. the low-cost, dexterous, and repairable bimanual teleoperation setup, while extending its capabilities beyond table-top manipulation. Specifically, we incorporate four key design considerations: 1) Mobile: The system can move at a speed comparable to human walking, around 1.42m/s. 2) Stable: It is stable when manipulating heavy household objects, such as pots and cabinets. 3) Whole-body teleoperation: All degrees of freedom can be teleoperated simultaneously, including both arms and the mobile base. 4) Untethered: Onboard power and compute. We choose AgileX Tracer AGV ("Tracer") as the mobile base following considerations 1 and 2.Tracer is a low-profile, differential drive mobile base designed for warehouse logistics. It can move up to 1.6m/s similar to average human walking speed. With a maximum payload of 100kg and 17mm height, we can add a balancing weight low to the ground to achieve the desired tip-over stability. We found Tracer to possess sufficient traversability in accessible buildings: it can traverse obstacles as tall as 10mm and slopes as steep as 8 degrees with load, with a minimum ground clearance of 30mm. In practice, we found it capable of more challenging terrains such as traversing the gap between the floor and the elevator. Tracer costs $7,000 in the United States, more than 5x cheaper than AGVs from e.g. Clearpath with similar speed and payload. We then seek to design a whole-body teleoperation system on top of the Tracer mobile base and ALOHA arms, i.e. a teleoperation system that allows simultaneous control of both the base and the two arms (consideration 3). This design choice is particularly important in household settings as it expands the available workspace of the robot. Consider the task of opening a two-door cabinet. Even for humans, we naturally step back while opening the doors to avoid collision and awkward joint configurations. Our teleoperation system shall not constrain such coordinated human motion, nor introduce unnecessary artifacts in the collected dataset. However, designing a whole-body teleoperation system can be challenging, as both hands are already occupied by the ALOHA leader arms. We found the design of tethering the operator's waist to the mobile base to be the most simple and direct solution, as shown in Figure 2 (left). The human can backdrive the wheels which have very low friction when torqued off. We measure the rolling resistance to be around 13N on vinyl floor, acceptable to most humans. Connecting the operator to the mobile manipulator directly also enables coarse haptic feedback when the robot collides with objects. To improve the ergonomics, the height of the tethering point and the positions of the leader arms can all be independently adjusted up to 30cm. During autonomous execution, the tethering structure can also be detached by loosening 4 screws, together with the two leader arms. This reduces the footprint and weight of the mobile manipulator as shown in Figure 2 (middle). To improve the ergonomics and expand workspace, we also mount the four ALOHA arms all facing forward, different from the original ALOHA which has arms facing inward. To make our mobile manipulator untethered (consideration 4), we place a 1.26kWh battery that weights 14kg at the base. It also serves as a balancing weight to avoid tipping over. All compute during data collection and inference is conducted on a consumer-grade laptop with Nvidia 3070 Ti GPU (8GB VRAM) and Intel i7-12800H. It accepts streaming from three Logitech C922x RGB webcams, at 480x640 resolution and 50Hz. Two cameras are mounted to the wrist of the follower robots, and the third facing forward. The laptop also accepts proprioception streaming from all 4 arms through USB serial ports, and from the Tracer mobile base through CAN bus. We record the linear and angular velocities of the mobile base to be used as actions of the learned policy. We also record the joint positions of all 4 robot arms to be used as policy observations and actions. We refer readers to the original ALOHA paper [1] for more details about the arms. With design considerations above, we build Mobile ALOHA with a $32k budget, comparable to a single industrial cobot such as the Franka Emika Panda. As illustrated in Figure 2 (middle), the mobile manipulator can reach between 65cm and 200cm vertically relative to the ground, can extend 100cm beyond its base, can lift objects that weight 1.5kg, and can exert pulling force of 100N at a height of 1.5m.We include more technical specifications of Mobile ALOHA in Figure 2 (right). Beyond the off-theshelf robots, we open-source all of the software and hardware parts with a detailed tutorial covering 3D printing, assembly, and software installation. 
co-training with static aloha data: 
 The typical approach for using imitation learning to solve real-world robotics tasks relies on using the datasets that are collected on a specific robot hardware platform for a targeted task. This straightforward approach, however, suffers from lengthy data collection processes where human operators collect demonstration data from scratch for every task on the a specific robot hardware platform. The policies trained on these specialized datasets are often not robust to the perceptual perturbations (e.g. distractors and lighting changes) due to the limited visual diversity in these datasets [98] . Recently, co-training on diverse real-world datasets collected from different but similar types of robots have shown promising results on single-arm manipulation [4, 95, 93, 96] , and on navigation [97] . In this work, we use a co-training pipeline that leverages the existing static ALOHA datasets to improve the performance of imitation learning for mobile manipulation, specifically for the bimanual arm actions. The static ALOHA datasets [1, 10] have 825 demonstrations in total for tasks including Ziploc sealing, picking up a fork, candy wrapping, tearing a paper towel, opening a plastic portion cup with a lid, playing with a ping pong, tape dispensing, using a coffee machine, pencil hand-overs, fastening a velcro cable, slotting a battery, and handling over a screw driver. Notice that the static ALOHA data is all collected on a black table-top with the two arms fixed to face towards each other. This setup is different from Mobile ALOHA where the background changes with the moving base and the two arms are placed in parallel facing the front. We do not use any special data processing techniques on either the RGB observations or the bimanual actions of the static ALOHA data for our co-training. Denote the aggregated static ALOHA data as as D static , and the Mobile ALOHA dataset for a task m as D m mobile . The bimanual actions are formulated as target joint positions a arms ∈ R 14 which includes two continuous gripper actions, and the base actions are formulated as target base linear and angular velocities a base ∈ R 2 . The training objective for a mobile manipulation policy π m for a task m isE (o i ,a i arms ,a i base )∼D m mobile L(a i arms , a i base , π m (o i )) + E (o i ,a i arms )∼Dstatic L(a i arms , [0, 0], π m (o i )), where o i is the observation consisting of two wrist camera RGB observations, one egocentric top camera RGB observation mounted between the arms, and joint positions of the arms, and L is the imitation loss function. We sample with equal probability from the static ALOHA data D static and the Mobile ALOHA data D m mobile . We set the batch size to be 16. Since static ALOHA datapoints have no mobile base actions, we zero-pad the action labels so actions from both datasets have the same dimension. We also ignore the front camera in the static ALOHA data so that both datasets have 3 cameras. We normalize every action based on the statistics of the Mobile ALOHA dataset D m mobile alone. In our experiments, we combine this co-training recipe with multiple base imitation learning approaches, including ACT [1] , Diffusion Policy [5] , and VINN [57] . 
tasks: 
 We select 7 tasks that cover a wide range of capabilities, objects, and interactions that may appear in realistic applications. We illustrate them in the Appendix Figure 3 . For Wipe Wine, the robot needs to clean up spilled wine on the table. This task requires both mobility and bimanual dexterity. Specifically, the robot needs to first navigate to the faucet and pick up the towel, then navigate back to the table. With one arm lifting the wine glass, the other arm needs to wipe the table as well as the bottom of the glass with the towel. This task is not possible with static ALOHA, and would take more time for a single-armed mobile robot to accomplish. For Cook Shrimp, the robot sautes one piece of raw shrimp on both sides before serving it in a bowl. Mobility and bimanual dexterity are also necessary for this task: the robot needs to move from the stove to the kitchen island as well as flipping the shrimp with spatula while the other arm tilting the pan. This task requires more precision than wiping wine due to the complex dynamics of flipping a half-cooked shrimp. Since the shrimp may slightly stick to the pan, it is difficult for the robot to reach under the shrimp with the spatula and precisely flip it over. For Rinse Pan, the robot picks up a dirty pan and rinse it under the faucet before placing it on the Wipe Wine ( drying rack. In addition to the challenges in the previous two tasks, turning on the faucet poses a hard perception challenge. The knob is made from shiny stainless steel and is small in size: roughly 4cm in length and 0.7cm in diameter. Due to the stochasticity introduced by the base motion, the arm needs to actively compensate for the errors by "visually-servoing" to the shiny knob. A centimeter-level error could result in task failure. For Use Cabinet, the robot picks up a heavy pot and places it inside a two-door cabinet. While seemingly a task that require no base movement, the robot actually needs to move back and forth four times to accomplish this task. For example when opening the cabinet door, both arms need to grasp the handle while the base is moving backward. This is necessary to avoid collision with the door and have both arms within their workspace. Maneuvers like this also stress the importance of whole-body teleoperation and control: if the arms and base control are separate, the robot will not be able to open both doors quickly and fluidly. Notably, the heaviest pot in our experiments weighs 1.4kg, exceeding the single arm's payload limit of 750g while within the combined payload of two arms. For Call Elevator, the robot needs to enter the elevator by pressing the button. We emphasize long navigation, large randomization, and precise whole-body control in this task. The robot starts around 15m from the elevator and is randomized across the 10m wide lobby. To press the elevator button, the robot needs to go around a column and stop precisely next to the button. Pressing the button, measured 2cm×2cm in size, requires precision as pressing the peripheral or pressing too lightly will not activate the elevator. The robot also needs to turn sharply and precisely to enter the elevator door: there is only 30cm in clearance between the robot's widest part and the door. For Push Chairs, the robot needs to push in 5 chairs in front of a long desk. This task emphasizes the strength of the mobile manipulator: it needs to overcome the friction between the 5kg chair and the ground with coordinated arms and base movement. To make this task more challenging, we only collect data for the first 3 chairs, and stress test the robot to extrapolate to the 4th and 5th chair. For High Five, the robot needs to go around the kitchen island, and whenever a human approach it from the front, stop moving and high five with the human. After the high five, the robot should continue moving only when the human moves out of its path. We collect data wearing different clothes and evaluate the trained policy on unseen persons and unseen attires. While this task does not require a lot of precision, it highlights Mobile ALOHA's potential for studying human-robot interactions.We want to highlight that for all tasks mentioned above, open-loop replaying a demonstration with objects restored to the same configurations will achieve zero whole-task success. Successfully completing the task requires the learned policy to react close-loop and correct for those errors. We believe the source of errors during the open-loop replaying is the mobile base's velocity control. As an example, we observe > 10cm error on average when replaying the base actions for a 180 degree turn with 1m radius. 
experiments: 
 We aim to answer two central questions in our experiments. (1) Can Mobile ALOHA acquire complex mobile manipulation skills with co-training and a small amount of mobile manipulation data? (2) Can Mobile ALOHA work with different types of imitation learning methods, including ACT [1] , Diffusion Policy [5] , and retrieval-based VINN [57] We conduct extensive experiments in the real-world to examine these questions. Please check the Appendix B and C for ablations and user studies, and our project website for qualitative results and videos.. As a preliminary, all methods we will examine employ "action chunking" [1] , where a policy predicts a sequence of future actions instead of one action at each timestep. It is already part of the method for ACT and Diffusion policy, and simple to be added for VINN. We found action chunking to be crucial for manipulation, improving the coherence of generated trajectory and reducing the latency from per-step policy inference. Action chunking also provides a unique advantage for Mobile ALOHA: handling the delay of different parts of the hardware more flexibly. We observe a delay between target and actual velocities of our mobile base, while the delay for position-controlled arms is much smaller. To account for a delay of d steps of the mobile base, our robot executes the first k -d arm actions and last k -d base actions of an action chunk of length k.We start with ACT [1] , the method introduced with ALOHA, and train it on all 7 tasks with and without co-training. We then evaluate each policy in the real-world, with randomization of robot and objects configurations as described in Figure 3 . To calculate the success rate for a sub-task, we divide #Success by #Attempts. For example in the case of Lift Glass and Wipe sub-task, the #Attempts equals the number of success from the previous sub-task Grasp Towel, as the robot could fail and stop at any sub-task. This also means the final success rate equals the product of all sub-task success rates. We report all success rates in Table 1 . Each success rate is computed from 20 trials of evaluation, except Cook Shrimp which has 5.With the help of co-training, the robot obtains 95% success for Wipe Wine, 95% success for Call Elevator, 85% success for Use Cabinet, 85% success for High Five, 80% success for Rinse Pan, and 80% success for Push Chairs. Each of these tasks only requires 50 in-domain demonstrations, or 20 in the case of High Five. The only task that falls below 80% success is Cook Shrimp (40%), which is a 75-second long-horizon task for which we only collected 20 demonstrations. We found the policy to struggle with flipping the shrimp with the spatula and pouring the shrimp inside the white bowl, which has low contrast with the white table. We hypothesize that the lower success is likely due to the limited demonstration data. Co-training improves the whole-task success rate in 5 out of the 7 tasks, with a boost of 45%, 20%, 80%, 95% and 80% respectively. For the remaining two tasks, the success rate is comparable between co-training and no co-training. We find co-training to be more helpful for sub-tasks where precise manipulation is the bottleneck, for example Press Button, Flip Shrimp, and Turn On Faucet. In all of these cases, compounding errors appear to be the main source of failure, either from the stochasticity of robot base velocity control or from rich contacts such as grasping of the spatula and making contact with the pan during Flip Shrimp. We hypothesize that the "motion prior" of grasping and approaching objects in the static ALOHA dataset still benefits Mobile ALOHA, especially given the invariances introduced by the wrist camera [99] .We also find the co-trained policy to generalize better in the case of Push Chairs and Wipe Wine. For Push Chairs, both co-training and no co-training achieve perfect success for the first 3 chairs, which are seen in the demonstrations. However, co-training performs much better when extrapolating to the 4th and 5th chair, by 15% and 89% respectively. For Wipe Wine, we observe that the co-trained policy performs better at the boundary of the wine glass randomization region. We thus hypothesize that co-training can also help prevent overfitting, given the low-data regime of 20-50 demonstrations and the expressive transformer-based policy used.We train two recent imitation learning methods, Diffusion Policy [5] and VINN [57] , with Mobile ALOHA in addition to ACT. Diffusion policy trains a neural network to gradually refine the action prediction. We use the DDIM scheduler [100] to improve inference speed, and apply data augmentation to image observations to prevent overfitting. The co-training data pipeline is the same as ACT. VINN trains a visual representation model, BYOL [101] and uses it to retrieve actions from the demonstration dataset with nearest neighbors. We augment VINN retrieval with proprioception features and tune the relative weight to balance visual and proprioception feature importance. We also retrieve an action chunk instead of a single action and find significant performance improvement similar to [1] . For co-training, we simply co-train the BYOL encoder with the combined mobile and static data.In Table 2 , we report co-training and no cotraining success rates on 2 real-world tasks: Wipe Wine and Push Chairs. Overall, Diffusion Policy performs similarly to ACT on Push Chairs, both obtaining 100% with co-training. For Wipe Wine, we observe worse performance with diffusion at 65% success. The Diffusion Policy is less precise when approaching the kitchen island and grasping the wine glass. We hypothesize that 50 demonstrations is not enough for diffusion given its expressiveness: previous works that utilize Diffusion Policy tend to train on upwards of 250 demonstrations. For VINN + Chunking, the policy performs worse than ACT or Diffusion across the board, while still reaching reasonable success rates with 60% on Push Chairs and 15% on Wipe Wine. The main failure modes are imprecise grasping on Lift Glass and Wipe as well as jerky motion when switching between chunks. We find that increasing the weight on proprioception when retrieving can improve the smoothness while at a cost of paying less attention to visual inputs. We find co-training to improve Diffusion Policy's performance, by 30% and 20% for on Wipe Wine and Push Chairs respectively. This is expected as co-training helps address overfitting. Unlike ACT and Diffusion Policy, we observe mixed results for VINN, where co-training hurts Wipe Wine by 5% while improves Push Chairs by 20%. Only the representations of VINN are co-trained, while the action prediction mechanism of VINN does not have a way to leverage the out-of-domain static ALOHA data, perhaps explaining these mixed results. 
conclusion, limitations and future directions: 
 In summary, our paper tackles both the hardware and the software aspects of bimanual mobile manipulation. Augmenting the ALOHA system with a mobile base and whole-body teleoperation allows us to collect high-quality demonstrations on complex mobile manipulation tasks. Then through imitation learning co-trained with static ALOHA data, Mobile ALOHA can learn to perform these tasks with only 20 to 50 demonstrations. We are also able to keep the system accessible, with under $32k budget including onboard power and compute, and open-sourcing on both software and hardware. Despite Mobile ALOHA's simplicity and performance, there are still limitations that we hope to address in future works. On the hardware front, we will seek to reduce the occupied area of Mobile ALOHA. The current footprint of 90cm x 135cm could be too narrow for certain paths. In addition, the fixed height of the two follower arms makes lower cabinets, ovens, and dish washers challenging to reach. We are planning to add more degrees of freedom to the arms' elevation to address this issue. On the software front, we limit our policy learning results to single task imitation learning. The robot can not yet improve itself autonomously or explore to acquire new knowledge. In addition, the Mobile ALOHA demonstrations are collected by two expert operators. We leave it to future work for tackling imitation learning from highly suboptimal, heterogeneous datasets.expertise. None of them have used Mobile ALOHA before. We start by allowing each participant to freely interact with objects in the scene for 3 minutes. We held out all objects that will be used for the unseen tasks during this process. Next, we give each participants two tasks: Wipe Wine and Use Cabinet. An expert operator will first demonstrate the task, followed by 5 consecutive trials from the participants. We record the completion time for each trial, and plot them in Figure 5 . We notice a steep decline in completion time: on average, the time it took to perform the task went from 46s to 28s for Wipe Wine (down 39%), and from 75s to 36s for Use Cabinet (down 52%). The average participant can also approach speed of expert demonstrations after 5 trials, demonstrating the ease of use and learning of Mobile ALOHA teleoperation.  
 
 Appendix: 8th Conference on Robot Learning (CoRL 2024), Munich, Germany. 


Decision: The reviewers found many strengths in this submission, but also clearly articulated some questions for the rebuttal phase. 
Thank you for your responses.

Reviews: review: title: title: Low-Cost Whole-Body Mobile Manipulation Robot Leveraging Co-Training on Tabletop Data
paper_summary: paper_summary: The paper introduces a new low-cost whole-body robot designed for mobile manipulation. It demonstrates that co-training with a larger dataset of tabletop manipulation significantly enhances the performance of mobile manipulation tasks.
main_review: main_review: **Strengths**

1. **Low Cost and Accessibility**: The robot's low-cost design makes it financially accessible for a wide range of research labs. This enables research in mobile manipulation by allowing more institutions to participate. The authors have documented the materials and design process, ensuring that other researchers can easily replicate and build upon their work.

2. **Teleoperation System for Data Collection**: The authors introduce a teleoperation system enables efficient gathering of training data. The show the effectiveness of the collected data in their experiments.

3. **High Success Rate with Minimal Demonstrations**: Achieving up to a 90% success rate with only 50 demonstrations is a significant accomplishment. This efficiency indicates the robustness of their learning algorithms and the effectiveness of the training data.

4. **Using ALOHA Data for Co-Training**: By incorporating data from ALOHA (825 episodes), the authors significantly improve the robot's performance. Co-training on this dataset allows the robot to learn a broader range of skills and adapt to various manipulation tasks more effectively. This is a very interesting finding for the community, showing that transferring skills across different tasks using demonstration data is feasible.

5. **Significant Performance Improvement**: The co-training method achieves a 34% absolute improvement over prior methods. This substantial gain shows the effectiveness of their approach and its potential impact on the field.

6. **Effective in Fine-Grained Manipulation Tasks**: The robot works well with fine-grained manipulation tasks, showing controlled movements capability. This capability is crucial for applications that require detailed and careful handling of objects.

7. **Continuous Evaluation and Robustness**: The authors do continuous evaluation through tasks such as pushing chairs. This continuous testing ensures that the robot can perform reliably on the task and shows that the results are reliable and not cherry-picked.
questions: questions: 1. **Feasibility in Real Houses**: The robot's large size may limit its practicality in navigating real houses, where space constraints are common. Addressing this issue is crucial for real-world applications in home environments. Do authors have any thoughts or further plans on how to deploy this setup in indoor environments beyond labs?

2. **Costly Data Collection for Long-Horizon Tasks**: Collecting 50 trajectories for long-horizon mobile manipulation tasks can be time-consuming and labor-intensive. Clarification is needed on the time required to gather data for the longest trajectories and how this impacts overall performance and efficiency.

3. **Generalization to New Environments**: The authors should provide more information on how well the robot generalizes to new, unseen environments. Understanding the limits of its adaptability is essential for evaluating its real-world applicability.

4. **Robustness to Environmental Changes**: The robot's performance under varying conditions, such as changes in background, clutter lighting, and dynamic obstacles (such as humans walking around) needs further exploration. These factors can significantly impact the robot's ability to perform tasks reliably.

5. **Battery Life and Operational Duration**: Information on how long the robot can operate untethered on a single charge would help determine its suitability for extended tasks.

6. **Scaling Data Collection**: Given the high cost associated with data collection, what do authors think about potential methods for scaling up this process? 

7. **Navigation Duration and Environment Observability**: The paper should specify the duration of navigation tasks and whether the environment is partially or fully observable. This information is crucial for understanding the robot's navigation capabilities and its ability to handle complex environments.

8. **Collision Frequency and Handling**: Information on the frequency of collisions and how the robot handles them would provide insights into its safety and reliability.
limitations: limitations: 2
review_summary: review_summary: I recommend accepting the paper for publication with minor revisions. The paper introduces a low-cost whole-body mobile manipulation robot and demonstrates significant performance improvements through co-training on a larger dataset. Its strengths include hardware accessibility, high success rates with minimal demonstrations, and substantial performance gains. However, the paper would benefit from addressing the robot's feasibility in real-world home environments, its generalization to new settings, and its robustness to environmental changes.    **After Rebuttal**: I read the rebuttal and other reviews. I thank the authors for the clarifications and I still vote for the acceptance of the paper.
title: title: The paper presents Mobile ALOHA, a cost-effective, whole-body teleoperation system that improves bimanual mobile manipulation tasks through co-training with static datasets, enhancing performance and efficiency in complex tasks with minimal demonstrations.
paper_summary: paper_summary: The paper introduces Mobile ALOHA, a low-cost, whole-body teleoperation system designed for bimanual mobile manipulation tasks. By augmenting the existing ALOHA system with a mobile base, the system allows for the collection of comprehensive bimanual manipulation data through teleoperation. The researchers utilize this data to train robotic policies using supervised behavior cloning. Their key finding is that co-training with static ALOHA datasets significantly improves the performance and success rates of mobile manipulation tasks. The paper demonstrates that with only 50 demonstrations per task, Mobile ALOHA can autonomously complete complex tasks like cooking, cleaning, and using elevators. This approach not only reduces the cost and complexity of hardware but also leverages existing datasets to enhance learning efficiency and performance in mobile robotic manipulation. The study underscores the potential of co-training with diverse datasets to develop more capable and efficient robotic systems.
main_review: main_review: **Strengths**
- **Cost-Effective Solution**: The system is low-cost, making advanced robotic manipulation accessible to more research labs.
- **Improved Task Performance**: The co-training method significantly boosts the success rates of complex tasks with minimal demonstrations.
- **Open-Source Accessibility**: The paper commits to open-sourcing all hardware and software implementations, promoting transparency and further research.

**Weaknesses**
- **Limited Novelty of Technical Methods**: This work is largely an incremental improvement on the existing ALOHA system, with a primary focus on hardware system evaluation. It merges with the previous dataset and employs similar methods, which limits the overall novelty. The primary innovation is the hardware setup, rather than new technical methods or algorithms, potentially reducing the paper's impact.

- **Potential Generalization Issue**: For some evaluations, the robot seems to consistently start from the same condition or using the very similar pots or opening the same cabinets, raising concerns about the system's ability to generalize to different initial conditions or environments. For example, while the authors likely tested the robot with various chairs, the results might be really good too, but this paper doesn't present these results. For the contribution to mobile and bimanual manipulation to be robust, the paper needs to show these robustness and versatility.

- **Handling Partial Dataset with Zero-Padding**: The method of merging two datasets by simply zero-padding the base action and third view camera raises questions about its scalability and rigor in handling diverse tasks, ie the actions can be values of zeros by nature. This approach may not be sufficient for robustly integrating partial data from different sources, potentially leading to suboptimal performance in tasks requiring more complex data handling and integration strategies.
questions: questions: [Issue 1] **Generalization and Robustness in Different Conditions**: Does quantitative evaluation (success metrics) include the trials with randomness shown in the example videos? Can the authors provide more detailed results or experiments showing the robot's performance starting from varied initial conditions or handling different types of chairs, pots with different weights or disturbances from the environments (move the objects or add weights into the pot during execution) to better demonstrate its generalization capabilities?

[Issue 2] **Zero-Padding**: How do the authors justify the robustness and scalability of their zero-padding method for merging datasets? Are there any additional experiments or analyses that support the effectiveness of this approach in handling diverse tasks and partial data?
limitations: limitations: 2
review_summary: review_summary: The paper introduces Mobile ALOHA, a low-cost, whole-body teleoperation system for bimanual mobile manipulation, demonstrating significant improvements in task performance through co-training with static datasets. While the system's affordability and open-source nature are strengths, there are concerns about its generalization capabilities and robustness in varied scenarios. Overall, while the hardware innovation is notable, the limited novelty in technical methods suggests that further demonstrations of robustness and scalability are needed to fully validate the approach.
title: title: The paper is well-written and easy to follow, especially with extensive hardware evaluations. An additional contribution of this paper to the robotics community is its intent to open-sourcing the system. Discussions on imitation learning involving heterogeneous robotic systems would be beneficial.
paper_summary: paper_summary: This paper presents Mobile ALOHA, a low-cost mobile manipulation platform for data collection and imitation learning. In this system, the original ALOHA system is augmented with a mobile base and a whole-body interface, enabling imitation of bimanual mobile manipulation tasks. Through co-training with static ALOHA datasets, it is demonstrated that Mobile ALOHA can learn various complex mobile manipulation tasks with small number of demonstrations in hardware experiments.
main_review: main_review: **Strengths:**

This paper is well-written and easy to follow with ample references. The paper includes detailed descriptions of the proposed hardware platform and experimental results. Supplemental videos demonstrate the effectiveness and robustness of the proposed learning framework in hardware experiments using the proposed platform. In addition, the paper aims to open source the system design. Open-sourcing the system could be potentially beneficial for the robotics community for advancement of future research.


**Weaknesses:**

In the current design, joint positions are directly used as a part of the mobile manipulation policy. This is possible because the leader and the follower arms share identical configurations. Thus, the collected datasets are applicable only in the specific platform. The paper uses the existing static ALOHA datasets for bimanual arm actions, but again this is possible since both static and mobile ALOHA systems have the common arm configurations. It would be informative if the authors discuss the case of imitation learning involving heterogeneous robotic systems.

**Minor comments:**

- page 3, Figure 2: Figure appears to be small, which could be enlarged for better readability.

- page 8, line 323: Reference (Zhao et al.) should be provided in the format with a number [1]. 

- The following paper (*1) conceptually employs a similar imitation learning approach to learning a mobile manipulation task with a teleoperated mobile manipulator platform, which could be mentioned in the paper.

(*1) Hiroshi Ito, Kenjiro Yamamoto, Hiroki Mori, and Tetsuya Ogata, "Efficient multitask learning with an embodied predictive model for door opening and entry with whole-body control", Science Robotics, Vol. 7, Issue 65, 2022.

- To make the paper be self-contained and more informative, brief explanations of the learning methods such as ACT, Diffusion Policy and VINN should be provided.
questions: questions: - How about the case of imitation learning involving heterogeneous robotic systems?
- Would it be possible to provide experimental videos at normal speed for teleoperation to better demonstrate the usability of the system?
- Can the learned policy generalize in the novel situations? For example, in the case of Wipe Wine task, can it work with the different table and glass?
limitations: limitations: 3
review_summary: review_summary: Overall, the paper is well-written and easy to follow, especially with extensive hardware evaluations. The intent to open source the system would be beneficial to the robotics community. However, at this moment, generalization of the learning framework using heterogeneous robotic systems is not very clear.