UMI-on-Legs: Making Manipulation Policies Mobile with Manipulation-Centric Whole-body Controllers
Authors: Huy Ha
Yihuai Gao
Zipeng Fu
Jie Tan
Shuran Song
Venue: CoRL 2024 (2024)
Keywords: Manipulation
Visuo-motor Policy
Whole-body Controller
TL;DR: A framework for scalable quadruped manipulation, combining manipulation policies trained from real-world demonstrations with whole-body controllers trained from simulation.

Abstract: We introduce UMI-on-Legs, a new framework that combines real-world and simulation data for quadruped manipulation systems.  We scale task-centric data collection in the real world using a handheld gripper (UMI), providing a cheap way to demonstrate task-relevant manipulation skills without a robot. Simultaneously, we scale robot-centric data in simulation by training a whole-body controller.  The interface between these two policies are end-effector trajectories in the task-frame, which are inferred by the manipulation policy and passed to the whole-body controller for tracking. We evaluate UMI-on-Legs on prehensile, non-prehensile, and dynamic manipulation tasks, and report over 70% success rate for all tasks. Lastly, we also demonstrate the zero-shot cross-embodiment deployment of a pre-trained manipulation policy checkpoint from a prior work, originally intended for a fixed-base robot arm, on our quadruped system. We believe this framework provides a scalable path towards learning expressive manipulation skills on dynamic robot embodiments.

UMI on Legs: Making Manipulation Policies Mobile with Manipulation-Centric Whole-body Controllers
Huy Ha*.12, Yihuai Gao*.1, Zipeng Fu', Jie Tan?, and Shuran Song!2
"Stanford University CClumbia University. Google DeepMind
Abstract: We introduce UMI-on-Legs, new framework that combines real-world and simulation data for quadruped manipulation systems. We scale task-centric data collection in the real world using hand-held gripper (UMI), providing cheap way to demonstrate task-relevant manipulation skills without robot. Simultaneously, scale robot-centric data in simulation by training whole-body controller for task bosk-trakking without task sim- ulation setups. The interface between these two policies is end-effector trajectories in the world frame, inferred by the manipulation policy and passed to the whole-body controller for tracking. We evaluate UMl-on-Legs on prehensile, non-prehensile, and dynamic manipulation tasks, and report over 70% success rate on tasks. Further, conduct in-the-wild, OOD evaluation for both policies in 5 unique locations (indoor/outdoorr, deformallehhard ground) and report 60% success rate. Lastly, we demonstrate the zero-shot cross cross-embodee emsoddimodm deployment of pre-trained manipulation policy checkpoint from prior work, originally intended for a fixed-base robot arm, on our quadruped system. Please checkout our website for robot videos, code, and data: https://umi-on weaeiiooo--lggggggiee github.io/
Keywords: Manipulation, Visuo-motor policy, Whole-body Control
Task-Centric Human Demonstrations in the Real World
UMI
Legs
Robot-Centric Reinforcement Learning in Simulation
Fig UM--on-Legs. We achieve fully autonomous, expressive, real-world manipulation skills on quadrupeds by combining real-world demonstrations using hand-held grippers (left) and simulation-trained whole-body controllers (right). Our framework allows the porting of existing "table-top" manipulation policies to mobile manipulation while enhancing mobility and power from the quadruped's legs.
1 Introduction
Collecting robot data in simulation versus in the real world are two distinct pathways for scaling up robot learning approaches, each with its challenges. Real-world robot teleoperation allows for direct human demonstration of new tasks. However, is often bottlenecked by the robot's physical hanlware - in addition to the cost and safety concerns, the involvement of robot hardware usually makes the collected data robot-specific (e.g., uses body commands) or even irrelevant whenever the robot's physical form changes. In contrast, simulation provides safe exploration and infinite resets for any new robot design. However, often bottlenecked by the task diversity accurately simulating diverse objects and their dynamics and defining all tasks and their associated rewards remains a significant challenge.
* Indicates equal contribution
8th Conference Robot Leamiing (CoRL 2024), Munich, Germany.

---

In this work, we introduce UMI-on-Legs, quadruped manipulation system that demonstrates an alternative way to combine real-world and simulation data:
We propose using a hand-held gripper (i.e., UMI [T]) to demonstrate task-centric data without requiring robot hardware. We model this manipulation data using a diffusion policy [2] that agnostic to the robot embodiment beyond the end-effector. By allowing human operators collect the data directly, we can easily capture a diverse set of challenging tasks, bypassing the need for real-world robot enrroo.n-error.
Scaling task-centric data in the real world without a robot:
Scaling robot-centric data in simulation without task simulations: We train whole-body controller (WBC) in massively parallelized simulation [3-7], whose sim2real transfer to a wide range of terrain conditions [7-9] is a well-studied problem. By training control policies to track end-effector trajectories instead interacting with objects, we can bypass the difficulty of setting up realistic manipulation assets, engineering task rewards, and modeling object dynamics.
iPhone Odometry
A key design choice in the framework is the interface be- tween the two policies: it must be simple enough for non- experts use, yet expressive enough for complex manipulation tasks. Most prior legged manipulation systems use body ve- locity commands with single-step. body-frame end-effector targets [8, 10-12), which are embodiment-speciff and in- sufficient for dynamic manipulation. We propose using end- effector trajectories in the world-frame, where the manipulation policy infers these from gripper-centric image observations. The whole-body controller then converts these trajectories into embodiment-speecffee controls, enabling cross-embodiment gen- cralization of the manipulation policy and providing preview for anticipating motions for the controller.
Fisheye Camera
Onboard Power Compute
Fig 2: In-the-wild Manipulation, on Legs, Fea-
turing a selfaccottined iPhone odometry and on- board power' compute, our mobile manipulation system is complete for in-the-wild manipulation.
To support system deployment, we present an accessible, real-time odometry solution, based on the Apple's ARKit. Instead of assuming tightly integrated and precisely calibrated/yynchoonized system [13-15], odometry approach takes the self-contained, compact form factor of iPhone.
We tested UMI-on-Legs with complex tasks requiring prehensile actions (pick & place), non-prehensile whole-body actions (weight pushing), and dynamic actions (tossing) in challenging scenarios with on- moaddrd sensing and in-the-wild generalization. Thanks to our interface design, our asynchronous bi-level policy formulation robustly executes these tasks, achieving over 70% all tasks. In summary, the main contributions of this work are:
UMI-on-Legs, a framework for combining real-world and simulation data, which enables learning expressive manipulation skills on mobile robots using no real-world robot data. Namely, real-world data collected without any robots [1] can be distilled into manipulation skills for different mobile robot platforms using robot-specific controllers trained entirely in simulation.
Within this framework, propose manipulation-centrie whole-body controller, using end-effector trajectory interface. This simple interface enables zero-shot emboddendd deployment of existing manipulation policies while being expressive enough to represent complex manipulation skills.
A real-world deployment system featuring real-time, robust, and accessible odometry approach for in wild task-space tracking, addressing a common bottleneck in achieving fast and precise manipulation in mobile manipulation systems.
2 Related Work
Mobile Pick-and Systems. Prior works have shown that pick-and place quadrupeds trained simulation
can be deployed in real-world environments using object detectors for visual domain transfer [12, 16). Wheeled systems demonstrate more complex manipulation tasks using carefully engineered primitives [17-21]. enabling
perception and planning [17, 20, 22] with foundation models [23-26), online learning [18, 19), or trajectory
optimization [21]. Recently, Mobile-ALOH.. [27] proposed wheeled-bimanual platform for demonstration collection. However, these systems rely quasi-static manipulation or heavy bases with a low center of mass.

---

Learning-base -aatid Controllers for Robot-specific Teleoperation. Reinforeement learning [28] (RL) with massively parallel simulators [3, 7] is the dominant approach for learning dynamic locomotion [6, 29- 31] and manipulation skills [32-34] in legged robots. While domain adaptation methods [7-9] allow deployment in different environments, mobill-manipulatiion on quadrupeds still typically requires human AlcoperatioonAApilTaaaa [35] during deployment for teleoperation [8, 10, []. Fully-autonomous deployment is limited due to the high cost of collecting real world robot data, which depends robot-specific commands and requires the robot's physical presence. Recently, He et al. combines behavior cloning with an RL controller, but their approach relies on robot-specific demonstrations generated heuristically in simulation, limiting scalability, expressivity, and usability. Furthermore, their system is constrained by a tethered setup with external cameras for April Tag tracking.
Cross-embodiment Manipulation. In pursuit of robust and generalizable manipulation, behavior cloning works has innovated on policy architectures [2, 37, 38], sensor placements [37], action spaces [38], and data considerations like quality, size, and cost [37, 39-43J. Key design choices have emerged, including pre-trained visual encoders [1, 25, 39], diffusion based action decoding [2, 39, 44], and end-effector sequence prediction [1, 38, 44]. Recent work [1, 45] shows zero-shot cross- cross-embodiment visuo-motor policy transfer using only ego-centric visual inputs. However, by predicting target pose sequences, these approaches assume precise low-level controllers for tracking.
3 Method: Universal Manipulation Interface on Legged Robots
UMI-on-Lege Legs (Fig 3) comprises of two main components: (1) a high-level diffusion--asel manipulation policy [2] which takes as input wrist-mounted camera views and outputs sequences of end-effector pose targets into the future in the camera frame, and (2) a low-level whole-body controller which tracks end-effector pose targets by outputting joint position targets for both the legs and arm. We train the manipulation policy using data collected in the real world using UMI [1], a hand-held gripper data collection device and train the WBC entirely in simulation using massively parallelized simulators [3].
Our choice of using world-frame end-effector trajectories as the interface has the following benefits:
Intuitive demonstration: Using end-effector trajectories instead of robot-specific low-level actions, we allow intuitive task demonstration by non-expert users using hand-held devices like UMI [1]-
High-level intention from preview horizon: With the preview horizon of future targets, the whole body controller can anticipate upcoming movements. For instance, the robots should brace accordingly if high-velocity toss coming up. Meanwhile, if the target is moving within the arm's 's reach range, the body should lean instead of taking step, which risks shaking the end effector.
Precise and stable manipulation in the world-fran framm: Unlike most of legged manipulation systems that uses body-framd tracking, our controller tracks action in the task-space (Fig 4) that is persistent regardless of base movement, enabling precise and stable manipulation.
Asynchronous multi-frequeney execution: The interface defines natural inference hierarchy that allows a fow-frequency manipulation policy (1-5Hz) to coordinate with a high-frequency low-level controller (50Hz) for handling drastically different sensor and inference latencies.
Compatible with any trajectory- manipulation policy: Our interface supports the plug-and-pla of any trajectory-based manipulation policy [1, 38, 39, 43-45). With the rise of policies trained on diverse datasets [39-41, 45, 46), our manipulation-centrie WBC can accelerate the porting of existing "table-top" manipulation skills to "mobile" manipulation.
3.1 Manipulation Policy with Behavior Cloning
Following the default configuration from Chi et al., we U-Net [47] architecture diffusion policy [2] (Fig 3a) with the DDIM scheduler [48] and a pre-trained CLIP vision encoder [25] to map from a context of visual observations to end-effector pose and gripper command trajectories. We use a longer action horizon of 64 to provide more information into the future for the low-level controller: For our cup rearrangement task, we directly use UMI [I]'s cup rearraggeennt checkpoint. For the pushing and tossing task, we collect data and train diffusion policies from scratch. Tasked with processing high-dimensional visual data, this network runs much slower than the controller. However, its output pose trajectory can be densely interpolated to get inputs for the controller, which outputs position targets for 12 leg joints and 6 arm joints. Meanwhile, its output gripper command trajectory can be interpolated can fed directly into the gripper joint.
3

---

a) Manipulation Policy
b) The Interface
c) Whole-body Controller
base state azm state leg state Last action
Arm
Diffusion Policy
MLP
Leg
SoHz
2Hz
Task-Space End-Effector
RGB
Action
< Behavior-Clone in Real World
Trajectory
Reinforcement Learn in Sim
Fig 3: Method Overview. Our system takes input RGB images from GoPro and infers camera- -ramme end-effector trajectory using a diffusion policy (a), trained using real-world UMI demonstrations, We transform this trajectory into the task bask-s and use it as the interface to the WBC. This controller (c) outputs joint position targets at 50Hz, which PD controllers subsequently tracks.
3.2 Whole-body Controller with Reinforcement Learning
To track end-effector trajectories predicted from the manipulation policy, we propose to train a whole body controller with reinforcement learning in simulation infer arm and leg joint targets. Notably, setting up simulation to track these manipulation end-effector trajectories does not require setting up the manipulation task and environment. This design greatly alleviates a key bottleneck in using simulation data.
Worrddfrraee Trajectory Tracking of Manipulation Trajectories. Prior works [8, 10-12] train WBCs by sampling end-effector poses in the body-frame, simplifying policy optimization but failing to teach whole-body coordination to compensate for base movements (Fig 4b). This limitation more pronounced when the arm's momentum affects the base, especially with lightweight systems dynamic movements. In contrast, train controller to track trajectories in the world-fram (Fig 4a), teaching the robot to maintain its gripper pose by compensating for body movements. We use task trajectories from UMI [1], which are more representative of real manipulation tasks. In Š 4, we show our controller generalizes across tasks without sacrificing tracking performance. Deployment details are in $ 3.3.
Observation space. The observation space includes robots' 18 joint positions and velocities, the base orientation and angular velocity, the previous action, and the end-effector trajectory inferred by the manipulation policy. We represent end effector pose using 3D vector for the position and the 6D rotation representation [49]. We densely sample target poses from -60ms to 60ms at 20ms interval relative to the current time, which informs the controller of current velocity and acceleration. In addition, we include the target 1000ms into the future, which helps the controller prepare to take foot steps if necessary.
Rewards. The task objective rewards the policy for minimizing the position error épos and orientation error Com to the target pose follows: exp(-( fre + fim where a represents scaling terms based ope
on precision requirements. This formulation entangles position and orientation, resulting in more ideal behavior compared treating them separately, which often leads to high precision in only one aspect. We also found that implementing exploration early while ensuring high precision with further training. In addition, we incorporate common regularizations [7-9, 12], as detailed in the supplementary material.
curriculum for both position and orientation is essential for enabling
Policy network architecture. We train a multi-layer perceptron (MLP) controller which maps from observations target joint positions for both the legs (12 DOF) and the arm DOF) (Fig 3c). A forward pass for this controller takes s=0.66ms on the robot's on-board CPU, and is called at 50Hz during deployment. The joint position targets are tracked using separate PD controllers for the legs and the arm.
3.3 System Integration
Robot system setup. The robot system (Fig. 2) consists of a 12-DOF Unitree Go2 quadruped robot and 6-DOF ARX5 robot arm, both powered by the battery of Go2. We customize the ARX5 arm with Finray grippers and a GoPro to match the UMI gripper [1]. Our whole-body controller runs on the Go2's Jetson while our diffusion policy inference runs on a separate desktop's 's RTX 4090 over an internet connection. We mount an iPhone for pose estimation and connect it to the Jetson through an Ethernet cable.
Rim2Real Transfer. To improve robustness, we apply random push forces and randomize joint friction, damping, contact frictions, body/arm masses, and centers of mass. Domain randomization of the end-effector's mass and center of mass simulates the effects of grasped objects without explicitly modeling them. Modeling

---

Manipulation Policy (2Hz)
Base Perturbation Movements
(Tracked by iPhone @ 60 H2)
Whole-body Controller (50Hz)
e) World-FF Fodd-Frrame (Ours)
Controller compensates movements
Manipulation Policy places cup onto sauce!
(Prior Work)
Controller
Body
follows movements
Manipuletion Policy compensatee
movements
Fig World- v.s. body-frame tracking. Our whole-body controller (WBC) learns track the target trajectory in world-frame (a), effectively compensating base perturbations and, therefore, frees up the manipulation policy to focus making task progress. In contrast, most of existing WBCs use body-frame tracking (b)(8, 10-12], trained to follow base perturbations. In effect, they defer world-frame tracking responsibilities to the low-rate manipulation policy and fail to react quickly to body perturbations.
20ms control latency during training was essential. Additionally, account for odometry noise, we randomly transport the robot every 20 seconds mid-episode. More details are provided in the appendix.
Accessible real-time odometry. The lack of real-time onboard task-space tracking is a common limitation in prior quadruped manipulation works. By assuming external tracking with motion capture [32] and/or April'aags [8, 11, 36], their systems can't deployed fully-autonomouslyy the wild. our system, we address this drawback with an iPhone, mounted on the robot's base. Our choice of the rear mounting site avoids adding extra weight the robot arm, prevents arm-phone collisions, and minimizes motion blur and visual occlusion. In contrast many existing robust, real-time odometry solutions [13-15], our odometry solution has self-contained, compact form-factor and uses only ubiquitous consumer electronic devices.
Asynchronous Transform Book-keeping, During deployment, the manipulation policy, WBC, and iPhone pose estimator run asynchronously. Let be the current timestep and T the action horizon of the manipulation policy. The iPhone ARSession's frame serves the world-frame, initialized when the iOS app opens. For each camera- frame action trajectory 0++T from the manipulation policy, transform it into the world-frame using the camera-pose recorded at time t. This pose is derived from the iPhone's 's timestamped world-frame pose and joint state forward kinematics. For all timesteps t' E [t,t+T], the controller densely interpolates the end-effector trajectory then transform it into the rripper's frame for controller observations.
4 Experiments
We designed series of experiments simulation and the real world to validate our key design decisions, which aims to answer the following key questions:
Capability, Is our system UMI-on-Legs able to leamn complex and challenging manipulation skills such as whole-body dynamics actions (e.g., tossing)?
Robustness. Can whole- body controller handle unexpected external perturbations and object dynamics, encountered during manipulation?
Scalability, Can we use cross- embodimen manipulation data collected entirely absent of the quadruped?
Metries. In simulation, we report average position error (cm), orientation error (radians), survival (%), and electrical power usage (kW) over 500 episodes. Here, survival is for the episode only the policy lasts for 17 seconds without a terminal collision (any robot link contact that not the robot's feet or gripper). In real-world experiments, we report average success rate over episodes unless specified otherwise.
Training Data. We train a WBC and a manipulation policy for each task on only data from those tasks: 100 trajectories for tossing, 200 for pushing, and 1000 for cup rearrangement. An exception is the cup task manipulation policy, where we directly use the public checkpoint from UMI [1], and only train a WBC for this task. To investigate the effects of adding more task trajectories, we finetuned "Ours (Tossing)" on 300 trajectories, 100 from tossing, pushing, and cup rearrangement each, leading to "Ours (Unified)".
Comparisons. We ablate design choices for the WBC in Table 1. Starting with approach, Ours (Tossing), we remove preview (i.c., trajectory observations), world-frame tracking formulation, and
5

---

1) Leap and Toss
Tuck and Curi
Land
3 Ours
1) Leap and Panick
2) Drop and Land
100%
85%
70%
b) No Preview
75%
0%
Mocap
Mocap
iPhone
Replay
Pollcy
Palicy
c) Comparison
Fig. 5: Dynamic Tossing Requires Whole-body Coordination. Our controller (top row) effectively executes reliable lossing strategy, despite limited arm strength and body inertia, in three stages. First, as the arm accelerates forward, the back legs lift, creating leap and loss motion (la, green). To avoid falling forward, the robot tucks its arms and legs inwards (2a, orange), generating backward torque that shifts the front legs' contact points ahead of its center of mass for landing (3a). contrast, the controller without preview information (bottom row) attempts to leap after the fast target acceleration but lacks foresight, resulting in dropped ball. Check out our robot videos our website!
manipulation trajectory training trajectories. Removing all three simultaneously leads to baseline similar to DeepWBC [8]. Finally, to investigate the efficacy of our iPhone-based odometry system, we compare against an OptiTrack motion capture system in all our real-world experiments, which serves as the oracle odometry. For real-world tossing, we also show the no-pre no-preview baseline's 's behavior in real (Fig. 5).
4.1 Capability: Whole-Body Dynamic Tossing
Our first experiment aims at validating the system's 's capability, i.c., whether our policy interface can capture complex manipulation skills. For this, we chose a particularly challenging task dynamic tossing. The robot, with the center of its body 150em away from and facing the target bin, must toss pre-grasped balls towards a target bin 90cm away, where a trial is successful if lands within 40cm of bin's center.
Task difficulties. Most of existing tossing systems assumee a carefully designed primitive [17, 50]. Re- cently, Chi et al. [1] demonstrated that learning from hand-held gripper demonstrations also results in ca- pable tossing skills, but has only been shown to work with analytical controllers on afixed-based arm. For legged systems, reliable tossing depends on the dy- namic body coordination. Thus, in order to achieve a successful toss, the controller must discover the right stance, utilize power from all of its degrees of freedom, and leamn the coordination required to achieve precise tracking throughout the dynamic tossing motion.
Approach Units
Pos Err Om Err Survival
Power kWL
cm.
degl
% 1
Ours (Unified)
2.18
3.10
99.8
3.72
Ours (Tossing)
2.12
3.35
98.4
3.82
Preview
3.02
4.23
93.0
3.95
World-space
15.49
15.55
0.0
4.74
UMI Traj
2.48
15.67
97.4
3.69
(-) Leg Ctrl
4.01
5.72
91.0
3.05
IK Fixed Leg
11.69
15.30
0.0
3.02
DeepWBC [8]
22.2
66.22
0.0
5.92
Table 1: Sim Tossing. Averaged over 500 episodes.
Dynamic tossing requires dynamic whole-body coordination. Through leaming to track end-effector trajectories, whole-body controller discovered whole-body tossing strategy (Fig 5). The strategy involved dynamic whole-body coordination, involving power from all joints during the toss, momentarily balancing on one foot/two feet, and effective usage of body mass inertia to regain balance. Overall, our system achieves 75% and 70% when using motion capture and iphone odometry, respectively.
Effects of the interface design. From ablations and baseline comparisons (Table 1), we observed that training on world-frame manipulation trajectories was crucial removing either world-frame tracking manipulation trajectories ((-) Task-space and (-) UMI Traj) significantly hurts position and/or orientation precision. Meanwhile, removing trajectory information from the WBC's observation prevents the controller from anticipating future motions, resulting in dangerous reaching behaviors ( (-5.4% survival) and more jittery actions (+130W compared to our approach, as demonstrated in Fig 5.
Improved tracking from diverse UMI trajectories. When finetuned on additional trajectories from pushing and tossing, we observe that our aassing-peetraine WBC not only retained its tracking abilitics, but

---

Push
Catch
b) Obj Variation
100%
95%
90%
static friction
dynamic slide
75%
10
50%
25%
Vel
-10
0%
Time [s]
0.0
0.5
1.0
1.5
2.0
2.5
3.0
Mocap iPhone
a) Pushing Sequence
c) Comparison
Fig 6: Robustness to Unexpected Dynamics. When faced with unexpected resistance from the kettlebell's friction, controller continuously tries to push forwards. This leads wiggling motion along with increased pushing force until static friction breaks into dynamic slide, which point the robot catches itself by dropping front feet forwards. We observed this push-and-catch strategy while the robot pushes both 10lbs 20Ibs (b), despite the latter succumbing to static friction more often. We also report comparable performance with the motion capture oracle (c).
also learned safer behavior. Quantitatively, its collision rate (1 survival) improves over the tossing only WBC by 25% (from 1.6% to 1.2%, Table 1) while reducing its power usage by 100W.
Failure analysis. Despite the highly-dynamic nature of this motion, the controller manages safe landings during all motion capture evaluation trials. To further investigate whether tossing misses were due to imprecise WBCs or noisy end-effector trajectories, we tested our controller with calibrated tossing trajectory replay with motion capture, which effectively acts as an "oracle" manipulation policy, and report +10% improvement (Fig. 5). We hypothesize that scaling up tossing data collection further could close this gap.
4.2 Robustness: End-effector Reaching Leads to Robust Whole-body Pushing
Our second task stress tests the controller's ability to handle unexpected and unseen external dynamics in zero-shot fashion. To evaluate this capability studied kettlebell pushing (Fig 6), where goal of this task is to push 10lbs kettlebell, such that it slides upright into goal region. Between each episode, the kettlebell's distance from its goal is varied within the range [80cm,120cm] 120cm]. and the robot starts off with its center 60cm behind the kettlebell with its grippers grasping the kettlebell's handle.
Task Difficulties. Although the end effector motion involved may be simple, the whole body controller need to adapt significantly changed dynamic models. In order be successful, the WBC must be robust to feet slippage, overcome static friction, and adapt to when static friction suddenly breaks into a slide. In addition, small imprecision in the pushing contact point could lead to the kettlebell toppling over.
Robustness to large disturbances. To our surprise, even with such a large disturbance, our controller was able to maintain balance and finish the task, without ever training on simulated forces or weights. Our controller completes the task with 95% and 90% success rates for motion capture and iPhone deployment respectively. We observed that as soon as the WBC observes larger position tracking erro caused by the weight, it changes its strategy to lean forward in order to exert more force a strategy which would cause the robot to fall forwards in absence of the kettlebell. By sending only end effector trajectories without any base velocity, the controller can also move its legs forward to reach faraway target poses. In addition, we also tested the policy with a 20Ibs kettlebell, and observed successful episodes out of 5.
Failure analysis. The most common failure case in this task due to the kettlebell toppling sideways. In addition, observed that our hardware system often overheats during this task's evaluation, and requires frequent cool-down breaks. Besides obvious hardware improvement, we hypothesize training with simulated forces [10] could lead to more elegant, energy efficient pushing behavior.
4.3 Scalability: Plug-and-plat Cruss-Embodiment Manipulation Policies
In our third experiment, we validate the feasibility of deploying a pre-trained manipulation policy from prior work [1] with our WBC, without fine-tuning, demonstrating zero-shot generalization to unseen embodinent*.. We evaluate this on the In-the-wild Cup Rearrangement Task, where the robot must place an espresso cup on saucer with the handle facing left. The cup and saucer are randomly positioned within
'Zero-sho shot refers the manipulation policy, which is not trained on the target embodiment
7

---

Flxes-bass
hardttrraah
0C0 terrahn
100%
85%
80%
!!!
75%
72%
50%
25%
0%
UR5 Engneored Ctri
Mocap iPhone In-the-wild
Learned Whole body Chi
Fig 7: Making Existing Manipulation Policies Mobile. Although intended for exee-based ams with larger reach range and precise controllers (left, from Chi et al.. the In-the-wild cup rearangement policy from Chi et al. was able to achieve 80-85% success rates zero-shot on our quadruped system with our learned whole-body controller when tested on hard ground (center figure). In out-o cut-of-distribution (OOD) terrain settings (outdoor environments with deformablc/unsaall terrain), our system with iPhone tracking achieves 60% success rate.
a 20cm radius, with random cup orientation. The robot starts 30cm from the table edge, facing the table. Success is defined as the cup being upright on the saucer, with the handle within 15 of left. This task always performed in an unseen environment for both the manipulation policy and WBC.
Task difficulties. The key challenge for this experiment is to effectively leverage cross-embodiment manipulation policy. In the original UMI paper [1]. the policy was trained on data collected by humans, and intended for industrial robot arms (e.g., URS and Franka) with a fixed-base and analytical controller. In this experiment, we are directly deploying this pre-trained policy on an 18 DoF' legged manipulator. Moreover, the intrinsic difficulty of the pushing and rearranging task also tests the system ability perform precise DoF movements, including prehensile and non-prehensile manipulation. This kind precise and gentle actions have been traditional difficult for low-cost legged manipulation systems.
Results and Findings. Without fine-tuning the manipulation policy, our system achieved success rates of 85% and 80% using motion capture and iPhone odometry on hard flat ground, respectively, despite the cross-embodiment gap. The manipulation policy, originally designed for UR5e system with a larger kinematic reach than our ARX5 arm, often predicts action sequences that span a larger workspace. To accommodate this, system dynamically tilts and leans its base to increase reach and counterbalance the arm. We evaluated our system with four different cups in five locations, including various indoor and outdoor terrains like soft carpet and unstable surfaces, which were out-of distriibbutoo (OOD) for the WBC, trained only on hard flat ground. Despite this, we achieved a 60% success rate (27/45 trials), with 72% (13/18) of failures attributed to manipulation errors (e.g.. dropping or pushing the cup out of view) and 28% (5/18) due to controller shakes on OOD terrain.
4.4 Limitations
Although UMI-on-Legs has demonstrated various manipulation skills, several limitations remain that future work could address. First, our system currently supports only gripper-based manipulation. A promising direction is to extend our interface to facilitate whole-body manipulation [32, 51, 52]. Second, while our embodiment- aggossic interface is scalable, incorporating embodiment-sppe specifti constraints into the high-level manipulation policy crucial. Lastly, for a complete mobile manipulation platform, our system could benefit from features like collision avoidance [53], force feedback, and force-based control [10].
Conclusion
We introduce UMI-on-Legs, a framework that combines real-world human demonstrations and simulation- based reinforcement learning. With world-frame end-effector trajectories as the interface, our formulation allows intuitive demonstrations from hand-held devices yet is expressive enough represent complex manipulation skills such as dynamic tossing and pushing, and precise pick and place.
Although, as community, we've yet to have a unanimous foundation manipulation policy, we believe our framework an important step towards enabling the transfer such policies to different robot platforms. Looking forward, we're optimistic that tackling limitations in our interface ($ 4.4) could take us even closer towards a future where separate people can develop ever more general visuo-motor policies and robust WBCs. knowing that they can be plugged-and played together in the end.
8

---

Acknowledgements
We would like to Pete Florence and Jonathan Tompson for many fruitful discussions in the project. We thank Cheng Chi, Zhenjia Xu, and Chuer Pan for guidance with UMI, and Haochen Shi, Antonio Loquercio, Haoran He, and Jiyuan Shi for tips with real-world deployment of RI. controllers. A special thanks to Zhenjia Xu, for giving us the idea and starter code to try out iPhone ARKit camera pose estimation. We thank Yifan Hou, Qi Wu, Shuo Yang. Tianyu Li, Xuxin Cheng. and Kexin Shi for fruitful discussions about controller formulation, implementation and deployment. We thank Jingyun Yang and Zi-ang Cao for helpful instructions set up the motion capture system. For paper writing and figures, we thank Zeyi Liu, Austin Patel, Mengda Xu, Dominik Bauer, Xiaomeng Xu, Mandi Zhao, and Samir Gadre for helpful feedback. Finally, we thank Carl Vondrick for providing the compute. This work was supported in part by the Google Research Award, NSF Award #2132519, #2143601, #2037101. We would like to thank ARX for the ARX robot hardware. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied of the sponsors.
References
[1] C. Chi, Z. Xu, C. Pan, E. Cousineau, B. Burchfiel, S. Feng, R. Tedrake, and S. Song. Universal manipulation interface: wn-the-wild robot teaching without in-the-wild robots. Proceedings of Science and Systems (RSS), 2024.
Roboties:
[2] C. Chi, S. Feng. Y. Du, Z. Xu, E. Cousineau, B. Burchfiel, and S. Song. Diffusson policy: Visuomotor policy learning via action diffusion. In Proceeulings of Robotics: Science and Systems (RSS), 2023.
[3] V. Makoviychuk, L. Wawrzyniak, Y. Guo, M. Lu, K. Storey, M. Macklin, D. Hoeller, N. Rudin, A. Allshire, A. Handa, et al. Isaac gym: High performance gpu-based physics simulation for robot learning. arXiv preprint arXiv:2108. 10470, 2021.
[4] E. Todorov, T. Erez, and Y. Tassa. Mujaco: A physics engine for model-based control. In 2012 JEEE/S) international conference on intelligent robots and systems, pages 5026-5033. IEEE, 2012.
[5] C. D. Freeman, F. Frey, A. Raichuk, S. Girgin, I. Mordatch, and O. Bachem. Brax-a differentiable physics engine for large scale rigid body simulation. arXiv preprint Stti::2006.12886 2021.
[6] K. Caluwaerts, A. Iscen, C. Kew, W. Yu, T. Zhang, D. Freeman, K.-H. Lee, L. Lee, S. Saliceti, V. Zhuang, et al. Barkour: Benchmarking animal-leve agility with quadruped robots. arXiv preprint arXiv:2305.1 14654, 2023.
[7] N. Rudin, D. Hoeller, P. Reist, and M. Hutter. Learning to walk in minutes using massively parallel deep reinforcement learning. In Conference on Robot Learning, pages 91-100. PMLR, 2022.
[8] Z. Fu, X. Cheng, and D. Pathak. Deep whole-body control: Learning unified policy for manipulation and locomotion. In Conference Robot Learning, pages 148.149. PMLR, 2023.
[9] A. Kumar, Z. Fu, D. Pathak, and J. Malik. Rma: Rapid motor adaptation for legged robots. arXiv preprint arXiv:2107.0403: 2021.
[10] T. Portela, G. B. Margolis, Y. Ji, and P. Agrawal. Learning force control for legged manipulation. arXiv preprint arXiv:2405.01402. 2024.
[11] G. Pan, Q Ben, 7. Yuan, G. Jiang, Y.Ji, J. Pang, H. Liu, and H. Xu. Roboduet: A framework affording mobile-manipulation and cross-embodiment. 2024.
[12] M. Liu, Z. Chen, X. Cheng, Ji, R. Yang, and X. Wang. Visual whole-body control for legged loco-manipulation. arXiv preprint arXiv:2403. 16967, 2024.
[13] W. Xu, Y. Cai, D. He, J. Lin, and F. Zhang. Fast-lio2: Fast direct lidar-inertial odometry. IEEE Transactions on Robotics, (8(4):2053-2072 2022.

---

[14] D. Wisth, M. Camurri, and M. Fallon. Vilens: Visual, inertial, lidar, and leg odometry for all all-teerran legged robots. IEEE Transactions on Robotics, 39(1):309-322 2022.
[15] Hartley, M. G. Jadidi, L. Gan, J.-K. Huang. W. Grizzle, and M. Eustice. Hybrid contact preintegration for visual-inertial-contie state estimation using factor graphs. In 2018 JEEE/RS/ International Conference on Intelligent Robots and Systems (IROS), pages
IEEE, 2018.
[16] N. Yokoyama, A. Clegg. J. Truong, E. Undersander, T.-Y. Yang, S. Amaud, S. Ha, D. Batra, and A. Rai. Asc: Adaptive skill coordination for robotic mobile manipulation. IEEE Robotics and Automation Letters, 9(1):799-786 2023.
[17] J. Wu, R. Antonova, A. Kan, M. Lepert, A. Zeng, S. Song, J. Bohg. S. Rusinkiewicz, and T. Funkhouser. Tidybot: Personalized robot assistance with large language models. Autonomous Robots, 2023.
[18] H. Xiong, R. Mendonca, K. Shaw, and D. Pathak. Adaptive mobile manipulation for articulated objects in the open world. arXiv preprint arXiv:2401. 01244011.44033 14403, 2024.
[19] C. Sun, J. Orbik, C. M. Devin, B. H. Yang, A. Gupta, G. Berseth, and S. Levine. Fully autonomous real-world reinforeement learning with applications to mobile manipulation. In Conference on Robot Learning, pages 308-319. PMLR, 2022.
[20] M. Ahn, Brohan, N. Brown, Y. Chebotar, O. Cortes, David, C. Finn, C. Fu, K. Gopalakrishnan, K. Hausman, et al. Do i can, not as say: Grounding language in robotic affordances. arXiv preprint arXiv:2204.01691, 2022.
[21] T. Lew, S. Singh, M. Prats, J. Bingham, J. Weisz, B. Holson, X. Zhang, V. Sindhwani, Y. Lu, F. Xia, et al. Robotic table wiping via reinforeement learning and whole-body trajectory optimization. 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 7184-7190. IEEE, 2023.
[22] W. Huang, P. Abbeel, D. Pathak, and I. Mordatch. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. In International Conference on Machine Learning, pages 9118-9147. PMLR, 2022.
[23] X. Gu, T.-Y. Lin, W. Kuo, and Cui. Open-vocabulary object detection via vision and language knowledge distillation. arXiv preprint arXiv:2104. 13921, 2021.
[24] A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitchead, A. C. Berg, W.Y Lo, et al. Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4015 2023.
[25] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. Leamning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763. PMLR, 2021.
[26] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, Dhariwal, A. Neelakantan, Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. Advances neural information processing systems, 33:1877-1901, 2020.
[27] Z. Fu, T. 7. Zhao, and C. Finn. Mobile aloha: Learning bimanual mobile manipulation with low-cost whole-body teleoperation. In arXiv, 2024.
[28] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization 2017.
algorithms. arXiv preprint
[29] X. Cheng, K. Shi, A. Agarwal, and D. Pathak. Extreme parkour with legged robots. arXiv preprint arXiv:2309.144 14341, 2023.
[30] Z. Zhuang, Z. Fu, J. Wang, C. Atkeson, S. Schwerffeger, C. Finn, and H. Zhao. Robot parkour learning. arXiv preprint arXiv:2309.05665, 2023.
10

---

[31] G. B. Margolis, G. Yang, K. Paigwar, T. Chen, and P. Agrawal. Rapid locomotion via reinforcement learning The International Journal of Robotics Research, 43(4):572-587, 2024.
[32] P. Arm, M. Mittal, H. Kolvenbach, and M. Hutter. Pedipulate: Enabling manipulation skills using a quadruped robot's leg. arXiv preprint arXiv:4402. 2024.
[33] C. Lin, X. Liu, Y. Yang, Y. Niu, W. Yu, T. Zhang, J. Tan, B. Boots, and D. Zhao. Locoman: Advancing versatile quadrupedal dexterity with lightweight loco-manipulators. arXiv preprint arXiv:2403.18197, 2024.
[34] Y. Ji, G. B. Margolis, and P. Agrawal. Dribblebot: Dynamic legged manipulation in the wild. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 5155-5162. IEEE, 2023.
[35] E. Olson. Apriltag: A robust and flexible visual fiducial system. In 2011 IEEE international conference on robotics and automation, pages 3400-3407, IEEE, 2011.
[36] Z. He, K. Lei, Y. Ze, K. Sreenath, Z. Li, and H. Xu. Learning visual quadrupedal loco-manipulation from demonstrations. arXiv preprint 24XX::2403.2022 2024.
[37] A. Mandlekar, D. Xu, J. Wong, Nasiriany, C. Wang. R. Kulkami, L. Fei-Fei, S. Savarese, Y. Zhu, and Martfn-Mrrtn. What matters in learning from offline human demonstrations for robot manipulation. arXiv preprint arXiv:2108.03299 2021.
[38] T. Z. Zhao, V. Kumar, S. Levine, and C. Finn. Learning fine fineinra bimanual manipulation with low-cost hardware, arXiv preprint 1rXx::3304.1300, 2023.
[39] A. Khazatsky, Pertsch, S. Nair, A. Balakrishna, S. Dasari, S. Karamcheti, S. Nasiriany, M. K. Srirama,
L. Y. Chen, K. Ellis, et al. Droid: A large-scale in-the-wild robot manipulation dataset. arXiv preprint WrXi:22403.1244, 2024.
[40] F. Ebert, Y. Yang. K. Schmeckpeper, B. Bucher, G. Georgakis, K. Daniilidis, C. Finn, and S. Levine Bridge data: Boosting generalization of robotic skills with cross-domain datasets. arXiv preprint arXiv:2109.13396 2021.
[41] H. R. Walke, K. Black, T. Z. Zhao, Q. Vuong, C. Zheng, P. Hansen-Estruch, A. W. He, V. Myers, M. J. Kim, M. Du, et al. Bridgedata v2: A dataset for robot learning at scale. In Conference on Robot Learning, pages 1723-1736. PMLR, 2023,
[42] P. Wu, Y. Shentu, Z. Yi, X. Lin, and P. Abbeel. Gello: general, low-cost, and intuitive teleoperation framework for robot manipulators. arXiv preprint arXiv:2309.13087 2023.
[43] H. Ha, P. Florence, and S. Song. Scaling up and distilling down: Language-guided robot skill acquisition. In Conference on Robot Learning, pages 3766-3777. PMLR, 2023.
[44] O. M. Team, D. Ghosh, H. Walke, K. Pertsch, K. Black, O. Mees, S. Dasari, J. Hejna, T. Kreiman, C. Xu, et al. Octo: An open-source generalist robot policy. arXiv preprint arXiv:2405. /2213, 2024.
[45] J. Yang, C. Glossop, A. Bhorkar, D. Shah, Q Vuong, C. Finn, D. Sadigh, and S. Levine. Pushing the limits of cross-embodiment learning for manipulation and navigation. arXiv preprint arXiv:2402. 19432, 2024.
[46] A. Padalkar, A. Pooley, A. Jain, A. Bewley, A. Herzog, A. Irpan, A. Khazatsky, A. Rai, A. Singh, A. Brohan, et al. Open x-embodiment: Robotic leamning datasets and rt-x models. arXiv preprint arXiv:2310.08864, 2023.
[47] O. Ronneherger, P. Fischer, and T. Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical image computing and computer-assisted Intervention-MICCAI 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part III 18, pages Springer, 2015.
11

---

[48] J. Song, C. Meng, and S. Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:001000250. 2020.
[49] Y. Zhou, C. Barnes, J. Lu, J. Yang, and H. Li. On the continuity of rotation representations in neural networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5745-5753, 2019.
[50] A. Zeng, S. Song, J. Lee, A. Rodriguez, and T. Funkhouser. Tossingbot: Learning to throw arbitrary objects with residual physics. IEEE Transactions on Robotics, 3604):307711119, 2020.
[51] S. Jeon, M. Jung, S. Choi, B. Kim, and Hwangbo. Learning whole-body manipulation for quadrupedal robot. IEEE Robotics and Automation Letters, 91):699-706, 2023.
[52] X. Cheng, A. Kumnr, and D. Pathak. Legs manipulator: Pushing quadrupedal agility beyond locomotion. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 5106-5112. IEEE, 2023.
[53] T. Miki, J. Lee, L. Wellhausen, and M. Hutter. Learning to walk in confined spaces using 3d representation. arXiv preprint 2rX1v:2403.00187 2024.
7 Things that did not work
In this section, we discuss things we tried, but did not improve the performance or caused other issues. We hope that through openly discussing our unsuccessful attempts, the community could gain a more complete understanding the project, learn from our mistakes, and improve upon our attempts. To complement this section, we've 'e also included robot failure videos
website.
7.1 Privileged policy distillation and observation history.
We tried to include privileged information (kp, kd, friction, damping, ground truth poses, base velocities, etc.) to train privileged policy and distill it through supervised learning and online regularization. However, it didn't I provide performance boost and introduce instability when we include observation history longer than 1 step. This could be because that the Python ROS2 timer is imprecise, thus the observation history can easily be out distribution given the wrong history timestamps. This was less of an issue on the Unitree Al platform, which ships with real-time kernel.
7.2 Precise grasping for tossing.
To train diffusion policies for grasping and tossing, we collected 500 episodes human demonstration. However in the grasping phase, the small shakes from the controller results in distributional drift in visuo-motor manipulation policy. We hypothesize that robust, fully-autonomous grasping and tossing could be achieved with more data.
7.3 System Reliability for Fully-untethered Deployment.
Although the system is capable of various tasks in the real world without any external cables, the reliability is in general not high enough. We summarize some hardware and system level challenges we encountered as follows. Given a small buse-to ense-to-arm weight ratio, the joints of the dog are likely to overheat in to 30 minutes according to the robot posture and then goes to error mode. Therefore, we needed frequently cool down the motors and fine-tune the policy more energy-efficient posture. On the power front, the battery supplies different voltages depending how full it is. When fully charged, the voltage is too high for the arm, and the setup requires voltage adaptor. When closer to empty, the policy's behavior is more dampened.
12

---

7.4 Velocity integration
We used the iOS ARKit to run VIO on an iPhone 15 Pro. However; the estimated camera pose sometimes drift heavily under dynamic actions including tossing, We also measure the delay of the pose estimator. Although the Ethernet communication time (between the iPhone and the onboard Jetson) is negligible, we found the the latency from the movement the pose update is still roughly 140ms. This latency introduced significant Sim2Real gap during deployment, which manifests as low frequency oscillations which slowly diverges.
Using low-latency foot contacts, joint position/velocitiss, and IMU readings, we can estimate the base velocity for each timestamp. Using base velocity estimates from 140ms into the past, we can integrate every received iPhone pose forward in time by 140ms. However, the shaking behavior due to the latency not addressed, which can only be mitigated by higher action rate regularization fine-tuning. As the ARKit pose estimation runs at 60Hz, we expect better implementation could achieve shorter latency.
8 Deployment
In this section we elaborate important details in real-world deployment and Sim2Real transfer.
8.1 iPhone placement
We mounted the iPhone on the back of the robot with following considerations::) Since the iPhone is facing back, the robot arm will not be in the view during manipulation, so ARKit's visual-inertial odometry can better track the surrounding environment. 2) iphone is mounted at a fixed angle 60 to the back plane of the Go2, thus the camera pointing upwards even when Go2 slightly bending down. This increases the number of visual features in the iPhone camera, thus provides more robust tracking compared to our original design, which was 90' mount. 3) This position is kinematically unreachable for the robot arm, the arm will unlikely damage the iPhone.
8.2
Robot URDF
For our highly dynamic tasks, we observed that the domain randomization and adaptive policies [8, 9] were insufficient to bridge the gap between heavily mis-specified model (provided by the manufacturer) and the real system, gap that is exacerbated when performing dynamic arm movements. We found that disassembling the ARX5 arm, reweighing each component, and recomputing mass, center of mass and inertia matrices in the URDF were crucial for both simulation stability as well as successfully real-world deployment.
8.3 Latency
While all observations and actions are perfectly synchronized in simulation, the communication and code runtime of the real-wor weal-world robot system introduce a significant amount of latency in different aspects.
The most important latency comes from the robot joint state observation and the joint command execution. This includes the motor encoder readout, ROS2 communication, whole-body controller inference, action sent back motors and being executed on motors. Since we were unable to exactly measure all the latency sources, swept the end-to-end latency from Oms 30ms with 5ms interval in simulation and find that 20ms works the best in our real-world system. We observed high frequency shaking if the latency is mismatched in simulator and real-world robots.
Robot pose estimators (motion capture and iphone) also introduce latency. We observed an 8ms latency from the motion capture system and 140ms latency from the iphone ARKit. We simulated 10ms pose latency in simulation close the sim2real gap. We also implemented inertial-legged velocity estimation to integrate the latest pose for iPhone, but the performance didn't 't improve significantly. See 7.4 for detailed explanation.
We also observe latency in the Python ROS2 program. Due to the global interpreter lock, all the Python ROS2 callback functions have to run altemaaely, so one callback function will block the others if it takes too
OOr code, data, and checkpoints will be released.
13

---

long. We optimize the run time of all callback functions and detach the callback functions that take too long to other ROS2 nodes. This allows the joint ohservation update to run closer to 200Hz and policy inference closer to 50Hz. A C++ implementation with more precise timer and lower latency can achieve better performance given the same checkpoint and evaluation setup.
8.4 Safety
We observed that directly deploying the whole-body controller checkpoints in the simulation section led to some safety issues and addressed them by fine-tune tine-tuning the checkpoints with more conservative reward schemes:
Shaking: To reduce the shaking and oscillation behavior of the robot, we increased the action rate regularization. The controller will get less reward if the predicted action is farther away from the previous action. We also disabled the on board lidar that introduces shaking behavior.
The calf joint of the robot applies the output torque. Therefore, the required ouiput torque of the motor is higher than the other joints, leading to frequent overheating and emergency shutdown. We observed that increased torque regularization during training allows the controller to run up to 30 minutes continuously!
Overheat Shutdowns:
linkage configuration rather than directly
Unsafe Configurations: In the simulator, the controller is likely to twist the legs to achieve higher precision with less movements, but this makes it unsafe to deploy in the real world. We added reward terms to regularize the body to a more balanced pose, so that the center of mass can roughly stay in the center.
Training
9.1 Manipulation Policy
In this section, we report hyperparameters used for training our manipulation diffusion policy. Visual observation and proprioception horizon means how many history image and robot states (with 0. interval) are used as input to the policy. Output Steps means the action length of the diffusion policy output. Execution Steps and Exccution Frequency indicate how many steps of the diffusion policy output actually executed in the real- world robot. To enable more stable end-effector trajectory updates, we add linear interpolation between the executed trajectory and the newly-updated trajectory for duration of "Trajectory Update Smoothing Time".
9.2 Whole-Body Dynamic Tossing
We increased the inference and execution steps for smoother toss, since the trajectory is less likely to update during the dynamic toss. We also increased the execution frequency to get farther toss. Please refer to Table 2 for detailed parameters.
Hyperparameters
Values
Training Set Trajectory Number
500
Diffusion Policy Visual Observation Horizon
2
Diffusion Policy Proprioception Horizon
4
Diffusion Policy Output Steps
64
Diffusion Policy Execution Steps
40
Diffusion Policy Execution Frequency
12Hz
Trajectory Update Smoothing Time
0.1s
Table 2: Hyper parameter set of the dynamic tossing task.
sPlease check out evaluation videos our website, which shows our controller running continuously for 20-30
minutes.
14

---

9.3 End-effector Reaching Leads to Robust Whole-body Pushing
Since the trajectory motion is much simpler, we collected fewer human demonstrations for the diffusion policy. We increased the "Trajectory Update Smoothing Time" to prevent heavy shakes due to the trajectory update. Please refer to Table 2 for detailed parameters.
Hyperparameters
Values
Training Set Trajectory Number
25
Diffusion Policy Visual Observation Horizon
2
Diffusion Policy Proprioception Horizon
2
Diffusion Policy Output Steps
32
Diffusion Policy Execution Steps
10
Diffusion Policy Execution Frequency
10Hz
Trajectory Update Smoothing Time
0.3s
Table 3: Hyper parameter set of the whole-body pushing task.
9.4 Plug-and-play Cruss-Embodiment Manipulation Policies
As the UMI cup rearrange task requires much higher precision but is quasi-static, we decreased the execution frequency for better stability. As mentioned in the main text, we directly use the publicly released checkpoint from Chi et al.. We did not collect any extra data for this task.
Hyperparameters
Values
Training Set Trajectory Number
1400
Diffusion Policy Visual Observation Horizon Diffusion Policy Proprioception Horizon
2
Diffusion Policy Output Steps
16
Diffusion Policy Execution Steps
8
Diffusion Policy Execution Frequency
5Hz
Trajectory Update Smoothing Time
0.1s
Table 4: Hyper parameter the UMI cup rearrangement task.
9.5 In-the-wild Cup Rearrangement Evaluation
We evaluated the whole-body controller on the cup rearrangement task in the wild. We tested it in some challenging scenarios including unstable terrain (grass, dirt, collapsible table), direct sunlight which made the robot easily overheated, and unseen tables and cups to show the generalizability of the diffusion policy, Please checkout the supplementary video for more details.
9.6 Reward Terms
During RL training of the whole body controller, we include the following reward terms:
Joint Limit, Joint Acceleration, Joint Torque, Root Height, Collision, Action Rate: We use the same definition as prior works [7, 8].
Body-EE Alignment: We regularize joint 0 and joint 3 of the arm (the joints that mostly affect the yaw angle of the gripper) to stay close to their initial pose. This allows the arm to be aligned with the body.
Even Mass Distribution: We regularize the standard deviation of the force of the four feet be lower. The motors are less likely to overheat if the body mass is distributed more evenly on the four legs.
15

---

Feet Under Hips: We regularize the feet's planar position to be close to that of their respective hips, for all four legs. This regularization improves the stability of the standing pose.
Pose Reaching: We minimize the position error Epu. and orientation error Fum to the target pose using an unified reward function: exp(- fre Fum )). We apply o curriculum to the reward functions:
the error get smaller, decrease for more peaky reward function, which encourages the controller to further reduce reaching error pps is set to [2, 0.1, ....0.... 0.1, 0.0,0 0.01, 0.0055 respectively; dom is set to
when the position error is smaller than [100, 0..,8,.5,0.4,0.2.2. 18.0,4.0..00,1100. when the orientation error smaller than 1.0.0,,.0,0.800000.. respectively.
Name
Weight
Joint Limit
-10
Joint Acceleration
-2.5e-7
Joint Torque
-le-4
Root Height
-1
Collision
-1
Action Rate
-0.01
Body-EE Alignment
Even Mass Distribution
Feet Under Hips
-1
Pose Reaching
Table 5: Reward terms.
10
Evaluation
10.1
Real World Tossing
To achieve a pre-grasped state during tossing evaluations, we handed the policy the tennis ball. We count episodes where the robot falls during the toss failure. Finally, measure the distance from the location where the ball first lands the center of the bin, and report success if lands within 40cm of the bin's center.
10.2 Simulation Ablations
each episode, random grasp and toss end-effector trajectory sampled from our dataset ($ 9.2), the robot uniformly randomfy initialized in the range of (-5.0cm,5.0ees for and y position, and [-0.1 Iadd Irad] for orientation, and its initial joint positions sampled uniformly within 5% of that joints' full range, centered around the default joint configuration. Position and orientation errors are averaged over all timesteps, while survival is 1 only if the policy lasts for 17 seconds without terminal collision. Here, terminal collision is defined as any robot part contact that is not the robot's 's feet or gripper.
Similar to training, evaluation includes pose latency of 10ms and the full range of domain randomization reported in Table 6.
Hyperparameters
Values
Init XY Position
[-0.1m,0.1m
Init Z Orientation
10.05rad,,0055rad
Joint Damping
[0.01.0.5]
Joint Friction
[0.0,0.05]
Geometry Friction
[
Mass Randomization
[-0.25,0.25]
Center of Mass Randomization
1-0.1m,0.Im
Table 6: Domain Randomization Hyperparameters.
All approaches were trained for 4000 iterations, and the performance at checkpoint 4000 is reported. For Ours (Unified), we took Ours (Tossing) and finetune it for 1000 steps 300 trajectories, 100 from each
16

---

of the three tasks studied. Power usage report is electrical power usage based on real hardware's voltages, manufacturers' reported torque constants, and the simulation's 's motor torques.
11 Misc.
Cost of the system. We report the cost of our entire robot system based on the market price in Table which roughly 1/4 of other quadruped manipulation systems [10, 12]
Item
Cost(S)
Unitree Go2 Edu Plus
12,500.00
ARX5 Robot Arm
10,000.00
GoPro Hero9
210.99
GoPro Media Mod
79.99
GoPro Max Lens Mod
68.69
iPhone 15 Pro
999.00
Elgato Capture Card
147.34
Total
24,006.01
Table 7: System Cost.
17


Reviews:
[{"review_id": "orVF70wdLP", "reviewer": "Reviewer_ZZoE", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 4, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposes UMI-on-legs which integrates the Universal Manipulation Interface (UMI) from prior work with a quadruped robot base. This is enabled by substituting the task-space end-effector controllers used in UMI for a RL whole-body controller that controls the legs and arm jointly to track task-space manipulator commands.", "summary_of_recommendation": "The paper makes a clear contribution to the field of legged mobile manipulation. It would be enhanced by clarifying points of the training procedure and ablating claims about the role of the legged base in the realized behaviors.", "technical_quality": 3, "clarity_of_presentation": 5, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "“To provide the controller with relevant reference trajectories, we use trajectories collected with UMI [1].” .. “removing either task-frame tracking or manipulation trajectories ((-) Task-space and (-) UMI Traj) significantly hurts position and/or orientation precision.” I would like more details about the trajectories used to train the RL controller. How many trajectories are there and what kinds of tasks do they represent? E.g. are the trajectories in 4.2 and 4.3 also seen during training or are they out-of-distribution for the RL policy? If I wanted to add a new task with UMI, would it also be important to retrain the WBC on some task-related trajectories, or does the policy learn a very general tracking capability?\n\nIt would be really cool to see the system accomplish tasks that specifically require legs. For example, can the robot walk over some uneven terrain (inaccessible to a wheeled robot) before performing the manipulation, while carrying the UMI payload? This could even be done by switching to Unitree’s default controller, if it’s hard to include non-flat terrain in the RL training.\n\nFurther thoughts on this topic: “For legged systems, reliable tossing depends on the body placement and the tossing motion. Thus, in order to achieve a successful toss, the controller must discover the right stance, utilize power from all of its degrees of freedom, and learn the dynamic whole-body coordination required to achieve precise tracking throughout the dynamic tossing motion.” Can this be quantified? For example, you could consider training an ablation policy (a) where the robot’s legs have fixed targets and the action space only controls the arm joints. Or you could consider an ablation (b) where the legs have fixed targets and the gripper is controlled with IK. I think it’s reasonable to expect (b) will not be performant because the base will shake — but this would be nice to show. I think (a) could actually perform better because the policy can still learn to compensate the arm’s commands for the motion of the base. If the full system delivers the best performance, the gap would quantify how much the legs are really contributing power or stabilization towards the task, vs. primarily acting as a complication to the dynamics that needs to be overcome. I think both are suggested in different parts of the paper and we don't really know which is being evidenced here.\n\n\"With an expressive embodiment-agnostic interface in place, we can separately develop ever more general visuo-motor policies and robust WBCs, knowing that they can be plugged-and-played together in the end.\" As mentioned in the limitations section, there is the outstanding problem of making the high-level policy aware of the embodiment-specific constraints. With this problem unsolved, can we really claim yet that developing more general versions of these components will lead to a plug-and-play solution?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rVCf0CA2Sv", "reviewer": "Reviewer_H1aK", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 4, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper presented a framework for training mobile manipulation policies using UMI decide that uses task-space trajectory and was tested on prehensile, non-prehensile, and dynamic actions", "summary_of_recommendation": "I recommend accepting the paper for its thoughtful hardware setup, choice of challenging tasks, thoroughness in experiments, and clarity in communication.", "technical_quality": 4, "clarity_of_presentation": 5, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "1. What exactly is task frame? The world frame? If so, why don't you call it the world frame? I was expecting to see it described in more detail in page 4, section 3.2 line 142. \n2. How do you adapt the trajectory-based policy from table-top to floor-top? For example, if you have a policy trained for tossing on the table using UMI data that a human collected while standing. How do you use that policy on the robot dog with its limited height? From the figures, it looks like the human has to kneel down to collect tossing and pushing data.\n3. How much data was collected to train for each task that achieved > 70% success rate? What's the distribution of initial states (e.g. height of tables in the picking task, range of robot initial base configuration and ball poses in tossing task) in demonstrations, and how different are the initial states in testing? I see in the supplementary that grasping and tossing took 500 demonstrations. How much would performance drop if only a subset of those demonstrations is used for training?\n4. Is the robot's initial base positions obtained using April Tags in the environment?\n5. The low-level RL policy is only used to train to each any end-effector poses, as opposed to follow certain demonstration trajectories for tasks? Would it be helpful to include task trajectories (like tossing) in training the low-level policy?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "R4Y7K1PcgW", "reviewer": "Reviewer_uNw1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper describes a learning-based framework for manipulation with a robot arm (gripper) mounted on a quadruped base. A diffusion policy is trained via demonstrated motions to map end-effector camera input to target position trajectories for the end-effector in the task frame.  A full-body low-level controller is then trained to map the target trajectories to joint poses for the robot. Experiments are provided on three tasks both in simulation and the real-world.", "summary_of_recommendation": "Overall the paper is considering an interesting problem and has developed a natural and potentially effective approach. We are shown successful sim-to-real performance on 3 tasks. Given the claims of generality, the paper would have been more convincing if a wider variety of tasks was considered. The description of the system and intuition behind the results can be improved significantly.", "technical_quality": 3, "clarity_of_presentation": 2, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "The gripper state did not appear to be part of the observation space or controllable by the learned policy. How is the gripper handled in this work? \n\nWhat exactly is \"survival %\"? Is it the same as success? For example, in tossing one might miss the goal but still not fall over. \n\nWhy is the low-level controller able to learn object manipulation when the simulations do not appear to involve objects or simulated disturbances that would be similar to such object interactions? \n\nAm I misunderstanding what you mean by \"zero-shot cross-embodiment\"? \n\nWhat trajectories are used to train the low-level controller?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: **Pre-rebuttal**

This paper receives a mixed review from three reviewers.

The reviews acknowledge the following strengths:
- Effectiveness of the whole-body controller trained with RL (**ZZoE**, **H1aK**).
- Showcasing a way of leveraging hand-held gripper data for quadruped manipulator (**ZZoE**, **H1aK**).
- Versatility of hardware setup (**H1aK**).
- Clarity of presentation (**ZZoE**, **H1aK**).
- Potential infrastructure contribution if open sourced (**ZZoE**).

The following weaknesses are mentioned:
- Lack of details on training (**ZZoE**, **uNw1**).
- Insufficient demonstration on tasks that require a legged base (**ZZoE**).
- Concern and confusion on the "zero-shot cross-embodiment" claim (**ZZoE**, **uNw1**).
- Limited technical novelty (**H1aK**).
- Lacking on the evaluated task number and variation (**H1aK**, **uNw1**).

---
**Post-rebuttal**

The paper initially received two weak accept and one weak reject. The reviewer gave the weak reject acknowledged the merit of the rebuttal and raised the assessment to weak accept. This leads to a positive consensus among three reviewers.

Overall the paper has delivered a novel and interesting framework for combining quadruped whole-body control with task frame human demonstrations. AC agrees with the assessments of the reviewers and thereby recommends accept.
Decision: Accept