Deep Whole-Body Control: Learning a Unified Policy for Manipulation and Locomotion
Authors: Zipeng Fu
Xuxin Cheng
Deepak Pathak
Venue: CoRL 2022 Oral (2022)
Keywords: Mobile Manipulation
Whole-Body Control
Legged Locomotion

Abstract: An attached arm can significantly increase the applicability of legged robots to several mobile manipulation tasks that are not possible for the wheeled or tracked counterparts. The standard modular control pipeline for such legged manipulators is to decouple the controller into that of manipulation and locomotion. However, this is ineffective. It requires immense engineering to support coordination between the arm and legs, and error can propagate across modules causing non-smooth unnatural motions. It is also biological implausible given evidence for strong motor synergies across limbs. In this work, we propose to learn a unified policy for whole-body control of a legged manipulator using reinforcement learning. We propose Regularized Online Adaptation to bridge the Sim2Real gap for high-DoF control, and Advantage Mixing exploiting the causal dependency in the action space to overcome local minima during training the whole-body system. We also present a simple design for a low-cost legged manipulator, and find that our unified policy can demonstrate dynamic and agile behaviors across several task setups. Videos are at https://maniploco.github.io

Deep Whole-Body Control: Learning a Unified Policy for Manipulation and Locomotion
Zipeng Fu'
Xuxin Cheng" Carnegie Mellon University
Deepak Pathak
Figure We present a framework whole-hody control a legged robot with robot arm attached. Left half shows how whole-body control achieves larger workspace by leg bending and stretching. Right half shows different real-world tasks, including wiping whiteboard, picking up cup. pressing door-open buttons, placing, throwing cup into garbage bin and picking clustered environments. Videos on the project website.
Abstract: An attached arm can significantly increase the applicability of legged robots to several mobile manipulation tasks that are not possible for the wheeled tracked counterparts. The standard modular control pipeline for such legged manipulators is to decouple the controller into that manipulation and locomotion. However, this is ineffective. It requires immense engineering support coordina- tion between the arm and legs, and error can propagate across modules causing non-smooth unnatural motions. It is also biological implausible given evidence for strong motor synergies across limbs. policy for whole-body control of legged manipulator using reinforcement learn- ing. We propose Regularized Online Adaptation to bridge the Sim2Real gap for high-Dof control, and Advantage Mixing exploiting the causal dependency in the action space to overcome local minima during training the whole-body system. We also present a simple design for a low-cost legged manipulator, and find that our unified policy can demonstrate dynamic and agile behaviors across several task setups. Videos are https: /atp://mmammccoee gi github..t.
this work, we propose to learn a unified
Keywords: Mobile Manipulation, Whole-Body Control, Legged Locomotion
Introduction
Locomotion has seen impressive performance in the last decade with results challenging outdoor and indoor terrains, otherwise unreachable by their wheeled or tracked counterparts. However, there are strong limitations to what a legged-only robot can achieve since even the most basic everyday tasks, besides visual inspection, require some form of manipulation. This has led to widespread interest and progress towards building legged manipulators, i.e., robots with both legs and arms, primarily
"equal contribution; ZF is now at Stanford University.
6th Conference on Robot Learning (CoRI. 2022), Auckland, New Zealand.

---

Training in Simulation
Env Extrinsics (x)
Body Mass,
End-ET End-Elfector Mass
Privileged info Encoder
Manipulation Advantage Function muatip
CoM, Friction,
(d]
Ap- l/mll
Arm Strength, (,) Leg Strength
Unified Policy (xt)
Regulárize
Supervise
Locomotion Adventage Function A
Mole youy foll
Adaptation Module
commands
(B) Deployment in Real World
Teleoperation
Demonstrations
Unified Policy (zt)
App 9p-1
50Hz
Adaptation Module (p) 50Hz
Figure 2: Whole-hody control framework. During training, unified policy is learned by conditioned environment extrinsics. During deployment, adaptation module is reused without any real-world fine-tuning. The robot can commanded in various modes including teleoperation, vision and demonstration replay.
achieved so far through physical modeling of dynamics [1, 2, 4, 5, 6]. However, modeling legged robot with an attached arm is dynamic, high-DoF, and non-smooth control problem, requiring substantial domain expertise and engineering effort on the part of the designer. The control frameworks are often modular where kinematic constraints are dealt with separately for different control spaces [7], thus limited to operating in constrained settings with limited generalization. Learning- based methods, such as reinforcement learning (RL), could help lower the engineering burden while aiding generalization to diverse scenarios.
However, recent learning-basee approaches for legged mobile manipulators [8] have also followed their model-based counterparts [9, 10] by using modular models in semi-coupled fashion control the legs and arm. This is ineffective due to several practical reasons including lack of coordination between the arm and legs, error propagation across modules, and slow, non-smooth and unnatural motions. Furthermore, it is far from the whole-bodd motor control in humans where studies suggest strong coordination among limbs. In fact, the control of hands and legs so tied together that they form low-dimension synergies, as outlined over 70 years ago in seminal series of writings by Russian physiologist Nikolai Bernstein (11, 12, 13]. Perhaps the simplest example how is hard for humans move arm and the corresponding leg in different motions while standing. The whole-body control should not only allow coordination but also extend the capabilities of the individual parts. For instance, our robot bends stretches its legs with the movement of the arm to extend the reach of the end-effector as shown in Figure 1.
Unlike legged locomotion, it is not straightforward to scale the standard sim2real RL to whole-body control due to several challenges: (a) High-Dol -DF control: Our robot shown Figure has total 19 degrees of freedom. This problem is exacerbated in legged manipulators because the control is dynamic, continuous and high-frequency, which leads to an exponentially large search space even in few seconds of trajectory. (b) Conflicting objectives and local minima: Consider when the arm tilts to the right, the robot needs to change the walking gait to account for the weight balance. This curbs the locomotion abilities and makes training prone to learn only
mode (manipulation or locomotion) well. (c) Dependency: Consider picking an object on the ground, the end-effector of the arm needs support from the torso by bending legs. This means the absolute performance of manipulation is bounded until legs can adapt.
In this work, we present both hardware setup for customized low-cost fully untethered legged manipulators and method for learning one unified policy to control and coordinate both legs and arm, which is compatible with diverse operating modes as shown in Figure 1. We use unified policy for whole-body control, i.e. to control the joints of the quadruped legs as well as the manipulator to simultancously take the arm end-effector to desired poses and command the quadruped to move

---

Command Following (ftollowing)
Energy (Fencryy) )
Alive (rative)
pmanip
0.5 c -pp.op Ip
0.004
Σ jjaam joints
loco
0.5
cund
+ 0.15 Myyo
0.00005 Lieleg joints I
0.2 + 0.5 2
Table 1: Both manipulation and locomotion rewards follow: Ptailowing + Veeergyy + Talive which encourages
command following while penalizes positive mechanical energy consumption to enable smooth motion [17]. Denote forward base linear velocity Uz, yaw angular base velocity wiyaw, tonque 7, joint angle velocity q-
in desired velocities. The key insights of the method that we can exploit the causal structure in action space with respect to manipulation and locomotion to stabilize and speed up learning, and adding regularization to domain adaptation bridges the gap between simulation with full states and real world with only partial observations.
We perform evaluation on our proposed legged manipulator. Despite immense progress, there exists no easy-to-use legged manipulator for academic labs. Most publicized robot is Spot Arm from Boston Dynamics [14], but the robot comes with pre-designed controllers that cannot be changed. Another example is the ANYmal robot with a custom arm [8] from ANYBotics. Notably, both these hardware setups are expensive (more than 100K USD). We implement a simple design of low-cost legged Gol robot [15] with low-cost arm on top (hardware costs 6K USD). Our legged manipulator can run fully untethered with modest on-board compute. We show the effectiveness of our learned whole-bodd body controller for teleoperation, vision- vision-guided control as well as open-loop control setup across tasks such picking objects, throwing garbage, pressing buttons walls etc. Our robot exhibits dynamic and agile leg-arm coordinated motions as shown in videos at https //mani ///////mmmmmoooooo/o github. io.
2 Method: A Unified Policy for Coordinated Manipulation and Locomotion
We formulate the unified policy л as one neural network where the inputs are current base state shase E IRS (row, pitch, and base angular velocities), arm state stm E IR (joint position and leg R (joint position and velocity of each leg joint, and
velocity of each arm joint), leg state 8 foot contact indicators), last action a E R 18 end-effector position and orientation command [psmd _ E SE(3). base velocity command [c_ndd wyyw J, and environment extrinsics 2, E R (details in Section 2.2). The policy outputs target arm joint position at E R and target leg joint leg E 12, which are subsequently converted to torques using PD controllers. We use position a, joint-space position control for both legs and the arm. As opposed to operational space control of the arm, joint-space control enables learning to avoid self-collision and smaller Sim-to-Ran gap, which is also found to be useful in other setups involving multiple robot parts, like bimanual manipulation [16].
We use RL to train our policy π by maximizing the discounted expected return E, ( where r2 is the reward at time step t, 7 the discount factor, and i' the max- imum episode length. The reward is the sum of manipulation reward manip and locomotion reward placo as shown in Table 1. Notice that
Command Vars u Cmd I
Training Ranges 0.9]
Test Ranges [0.8, 1.0]
[-1.0, 1.0]
[-1.-] -.7] [.7. 1]
[0.2, 0.7]
[0.6, 0.8]
P
[ 2tz 22/ 2n/51
2n 2 2n/51
y
3x / 3n/51
33 3n/51
Tin
[1,
[0.5. 1]
Table 2: Ranges for uniform sampling of command variables
we use the second power of energy consumption at each leg joint to encourage both lower aver- age and lower variance across all leg joints. We follow the simple reward design that encourages minimizing energy consumption from [17].
We parameterize the end-effector position com- mand n in spherical coordinate (l,p, v). where / is the radius of the sphere and p and are the pitch and yaw angle. The origin of the spherical coordinate system is set the base of the arm, but independent of torso's 's height, row and pitch (details in Supplementary). We set the end-effector pose command end by interpolat- ing between the current end-effector position p and randomly sampled end-effector position
Env Params
Training Ranges
Test Ranges
Base Extra Payload
[-0.5, 3.0]
[5.0, 6.0]
End-Effector Payload
[0. 0.1
[0.2, 0.3]
Center of Base Mass
[-0.15, 0.15]
[0.20, 0.20]
Arm Motor Strength
[0.7. 1.3]
[0.6, 1.4]
Leg Motor Strength Friction
[0.9. 1.1]
[0.7, 1.3]
[0.25, 1.75]
[0.05, 2.5]
Table 3: Ranges for uniform sampling of environment parameters
3

---

pand every Tuj seconds:
Pi
t iraj
p +
(1
t Tims
and
1
[0, Timg
pmnd is resampled if any pi leads to self-collision or collision with the ground d is uniformly sampled from SD(3) space. Table 2 lists the ranges for sampling of all command variables.
2.1 Advantage Mixing for Policy Learning
Training a robust policy for a high-DoF robot is hard. In both manipulation and locomotion learning literature, researchers have used curriculum learning to ease the learning process by gradually increasing the difficulty of tasks so that the policy can learn to solve simple tasks first and then tackle difficult tasks [18, 19, 20]. However, most of these works require many manual tunings of a diverse set of the curriculum parameters and careful design of the mechanism for automatic curriculum.
Instead of introducing large number of eurricula the learning and environment setups, we rely on only one curriculum with only one parameter to expedite the policy learning. Since we know that manipulation tasks mostly related to the arm actions and locomotion tasks largely depends on leg actions, we can formulate this inductive bias in policy optimization by mixing advantage functions for manipulation and locomotion to speed up policy learning. Formally, for policy with diagonal Gaussian noise and a sampled transition batch D, the training objective with respect to policy's parameters 0z
1 J(0x) = D
Σ (SLIa)ED
log ((a um |s)) (Amanip + (.A + log (a,"" st) (3Ammmip + A Aloon
is the curriculum parameter that linearly increases from 0 to I over timesteps Tmix: 3 = min(t/T (t/T/t 1). mmanip and Aloco advantage functions based y manip and placo respectively. Intuitively, the Advantage Mixing reduces the credit assignment complexity by first attributing differ- ence in manipulation returns arm actions and difference in locomotion returns to leg actions, and then gradually anneal the weighted advantage sum to encourage learning arm and leg actions that help locomotion and manipulation respectively. We optimize this RL objective by PPO [21].
2.2 Regularized Online Adaptation for Sim-to-Real Transfer
Much prior work on Sim-to-Res transfer utilize the two-phase teacher-student scheme to first train a teacher network by RL using privileged information that is only available in simulation, and then the student network using onboard observation history imitates the teacher policy either in explicit action space or latent space [22, 23, 24, 25J. Due to the information gap between the full state available to the teacher network and partial observability of onboard sensories, the teacher network may provide supervision that is impossible for the student network predict, resulting in realizability gap. This problem is also noted Embodied Agent community [26]. In addition, the second phase can only start after the convergence of the first phase, yielding extra burdens for both training and deployment.
To tackle the realizability gap and to remove the two-phase pipeline, we propose Regularized Online Adaptation (shown in Figure 2). Concretely, the encoder μ takes the privileged information as input and predict an enviornment extrinsics latent 2" for the unified policy to adapt its behavior in different environments. The adaptation module estimates the environment extrinsics latent 2 by only condition on recent observation history from robot's 's onboard sensories. We jointly train μ with the unified policy end-to-end by RL and regularize z" to avoid large deviation from d estimated by the adaptation module. The adaption module o is trained by imitating z" online. We formulate the loss function of the whole learning pipeline with respect to policy's 's parameters 0z, privileged information encoder's parameters 0,., and adaptation module's 's parameters 0g as
L(0,, 0s) (Ox, 02) + A||2" sgl2>]]la + ||sg[z"] 121
where J(O,,, 0,)) is the RL objective discussed in Section 2.1, sg[:] is the stop gradient oper- ator, and X the Laguagrian multiplier acting as regularization strength. The loss function can be minimized by using dual gradient descent: 0,, 0, + arg ming, 8, (())))))(

---

Survival f
Base Accel. 4
Vel Error 4
EE Eror +
Tot. Energy 4
Unified (Ours)
97.1 + 0.61
1.00 = 0.03
0.31 # 0.03
0.63 = 0.02
50 + 0.90
Separate
92.0 + 0.90
1.40 = 0.04
0.43 1 0.07
0.92 = 0.10
51 4: 0.30
Uncoordinated
94.9 = 0.61
1.03 0.01
0.33 = 0.01
0.73 = 0.02
0.28
Table 4:
Comparison of unified policy with separate policies for legs-arm, and one uncoordinated policy. The unified policy achieves the best performance given energy consumption. The test ranges in Table 2.
0g + arg ming, ((s.a)><(.....)) [L], and +- A + Dx with step size a. This optimization process is known to converge under mild conditions [27. 28]. In practice, we alternate the optimization process of the unified policy and encoder μ1 and the one of adaptation module by fixed number of gradient steps. À increases from O to by a fixed linear scheme. Notice that RMA [22] is a special case of Regularized Online Adaptation, in which the Laguagrian multiplier À is set to constant zero and the adaptation module Φ starts training only after convergence of the policy and the encoder
Deployment During deployment, the unified policy and adaptation module executes jointly onboard. To specify commands, we develop three interfaces: teleopertion by joysticks, closed-loop control by using RGB tracking, and open-loop reply of human demonstrations, Details are in Section 3.3.
3 Experimental Results
6DoF Arm
3.1 Robot System Setup
The robot platform is comprised of Unitree Gol
Onboard Compute Power
quadraped [15] with 12 actuatable DoFs, and robot
RGB Camera
arm which is the 6-DoF Interbotix WidowX 250s [29] with a parallel gripper. We mount the arm on top of the quadruped. The RealSense D435 provides RGB visual information and mounted close to the gripper of X. Both power of Gol and WidowX are provided by Gol's 's onboard battery. Neural network inference is also done onboard of Gol. Our robot system uses only onboard computation and power so it is fully untethered.
Go1 Quadruped
Figure 3: Robot system setup
3.2 Simulation Experiments
The purpose our simulation experiments to address the following questions:
Does the unified policy improves over separate policies for the arm and legs? If so, how?
How Advantage Mixing helps learning the unified policy?
What's 's the performance of Regularized Online Adaptation compared with other Sim2Real methods?
Baselines and Metrics:
We compare method with the following baselines:
Separate policies for legs and the arm: one policy controls legs based on the quadraped observation, and another policy controls the arm based on arm observation.
2. One uncoordinated policy: Same as unified policy which observes aggregate state base, legs and the arm, but only yumip is used to train arm actions, and only y. for leg actions.
3. Rapid Motor Adaptation (RMA) [22]] Two-phase teacher-student baseline.
4. Expert policy: the unified policy using the privileged information encoder 2"
5. Domain Randomization: the unified policy trained without environment extrinsics 2.
We report following metrics: (1) sur- vival percentage, (2) Base Accel: an- gular acceleration of base, (3) Vel Er-
Arm workspace (m²) 1 Survival under perturb 1
Unified (Ours)
0.82 0.02
0.87 0.04
Separate
0.58 0.10
0.64 0.06
ror: LI error between base velocity commands and actual base velocity, (4) EE Error: LI error between end- effector (EE) command and actual EE pose, (5) Tot. Energy: total energy consumed by legs and the arm. All metrics are normalized by episode length. All experiments tested over 3 randomly initialized networks and 1000 episodes each. Details simulation and training are in Supplementary.
Uncoordinated
0.65 1 0.02
0.77 1 0.06
Table 5: unified policy, legs help increase the arm workspace and the arm helps the quadruped to stabilize.
5

---

Realizability Cup
||2
2 l
Survival
Base Accel. 4
Vel Eroo 4
EB Error L
Tor. Energy
Domain Randomization
95.8 - 0.2
0.44 + 0.00
0.46 0.00
0.40 0.00
21.9 0,53
RMA [22] Regularized Online Adapt (Ours)
0.31 + 0.01
95.2 0.2
0.54 + 0.02
0.41 = 0.00
0.26 0.0
27.3 = 0.96
2e-4 =0.00
97.4 0.1
0.51 0.02
0.39 0.01
0.21 0.00
25.9 0.56
Expert w/ Reg.
97.8
0.52 + 0.02
0.40 0.01
0.21
0.00
25.8 2 0.19
Expert w/o Reg.
98.3 0.2
0.51 +: 0.02
0.39 = 0.00
0.21 0.00
25.6 0.30
Table 6: Regularized Online Adaptation outperforms other basclines with the smallest imitation error which helps it to have the same performance the expert policy which uses privileged information to predict envi- ronment extrinsics. Expert policy trained with regularization term ||2 sg[="]] has negligible performance degradation compared with expert trained without regularization. Test ranges in Table 3. Domain Random- ization learns to just stand in most cases, hence, trivially collapsing to low Tot. Energy and Base Accel.
Improvements of the Unified Policy over Baselines: In Table 4, unified policy outperforms separate and uncoordinated policies because both the arm and leg actions are trained with the sum of reward for manipulation and locomotion are given with observations for the arm, legs and the quadraped base, while baselines struggle to maintain a small base acceleration, which results in larger error in command velocity following and inaccurate EE pose following.
Unified Policy Increases Whole-body Coordination: Table shows that our unified policy pro- motes whole-body coordination where (1) leg actions will help the arm to achieve larger workspace via bending for lower EE commands and standing up high for higher EE commands, and (2) arm will help the robot balance under larger perturbation (1.0 m/s initial velocity of base) resulting in higher survival rate of the unified policy. We estimate the the arm workspace via calculating the volume of the convex hull of 1000 sampled EE poses, subtracted by the volume of cube that encloses the quadruped.
Advantage Mixing Helps Learning the Unified Policy: Without Advantage Mixing, the unified pol- icy has difficulty in credit assignment, resulting in the policy first learns EE command following but ignores the locomotion task. As shown in Figure 4, Advan- tage Mixing helps the policy to focus on each task first and then merge them together, which induces a curriculum-like mechanism to speed up training. Details in Supplementary.
20-
Episode Vel Error [m/5 (s)
15-
10-
5-
Advantage Mixing (Ours)
No Advantage Mixing
Robust OOD Performance of Regularized Online Adaptation: We find that our Regularized Online Adaptation is more robust than RMA and Domain Randomization (DR), tested in environments with out-of-distribution (OOD) environment parameters in Table 3. In RMA, it is not guaranteed the estimated environment extrinsics by the adaptation module can imitates the one learned by the expert. With Regularized Online Adaptation, the expert learns to predict environment extrinsics with regularization from the adaptation module, thus tiny imitation error, resulting in 20% reduction in EE Error. Table 6 shows that adding regularization to expert has negligible negative impact on performance, while every metric gets improved compared to RMA due to smaller latent imitation error. Note that DR has better base acceleration and total energy just stands in place under difficult environments.
0-
1000
2000
3000
4000
5000
Iteration
Figure 4: Advantage Mixing helps the unified policy to learn to follow the command velocity much faster (aggregated Vel Error over episodes decreases sharply) than without mixing.
3.3 Real-Worle Experiments
We use the built-in Gol MPC controller and the IK solver for operational space control of Widow as the bascline in the real world, which we refer to as MPC+IK. More details are in the Supplementary.
Teleoperation: We specify EE position command pmd by parameterizing pi41 ppmd + Ap, where Ap (Ai, Ap, Ay) is specified by two joysticks. With human in the loop, we can command the end-effector to reach points within or outside of training distribution. In Figure 5, we analyze the whole-body control in the real world, and show that the quadruped's d's base rotation (y-quad pquad yuud) strongly correlates with the EE position command pand This indicates that our unified policy enables whole-body coordination where the leg joints, as well the arm joints, help reaching.

---

0.4
0.4
= 0.45
L5
R0aa (rad)
0.55
0.2
(rad]
0.2
0.65
-0.
0.0
1.5
0.0
-0.2
0.4
P. (rad)
(rad)
(a) Body pitch paund correlates with command EE pitch with various lengths 1.
(b) Body pitch y quad correlates with command EE pitch with various yaws y.
Figure 5: Real-world whole-body control analysis. (a) We fix command EE yaw y = and change command EE pitch p and length When p has a large magnitude, the quadruped will pitch upward or downward help the arm reach for its goal. With larger' (goal far away). the quadruped will pitch more to help. (b) When the magnitude of command EE yaw is closer to 1.578 (arm turns to side of the torso). the quadruped will roll to help the arm. When y the quadruped pitches downward instead of roll sideways help the arm.
Easy Tasks
bc Hard Tasks
X
Suocess
Success
54
(a) Our method: success in both easy and hard tasks with coordinated bchaviors.
(b) MPC+IK: failure in both scenarios. Left: fail due to missed cup. Right: fail due to solf-collision.
Figure 6; Comparison of method and the baseline controller (MPC+IK) in vision- vision-guided pick-up tasks. We sample diffeernnt points around the robot the target pick up position. Easy tasks: points are normal distance from the robot. Hard tasks: when the point is very close to the front feet and are hard to reach without whole-body control. More hard tasks are in the Supplementary. Videos are at https: h//////mmppppp
github. io
Vision-Guided Tracking: In addition joystick control by humans, we also show successful picking tasks using visual feed- back from an RGB camera. We mount a Realsense D435i camera near the gripper of the arm and use AprilTag [30] to get the relative position between the gripper and the object to be picked up. AprilTag is a visual fiducial system popular in robotics research using simple 2D black and white blocks encode pose information. We first get the translation of the AprilTag in the camera frame pag [2 y ""
Success Rate
TTC
Failure Rate
Self-Collision Rate
Easy tasks (tested on 3 points)
Ours
0.8
5s
0
MPC+IK
0.3
17s
0.4
0.3
Hard tasks (tested 5 points)
Ours
0.8
5.68
0
MPC+IK 0.1
22.0s
0.2
0.5
Table 7: Comparison of our method v.s. MPC+IK on pick-up tasks, pad the goal position sampled from the points the ground. TTC is the average time completion. Each task performance averaged
real-world trials.
Then we design and use simple yet effective position feedback controller to set the current EE position command y.nu Kf a where K = [-1.5, 1.5, 0.1]" is a gain vector for position control. In Figure 6 and Table 7, we compare our method and the baseline (MPC+IK) in several pick-up tasks by measuring the success rate, average time to to completion (TTC), IK failure rate, and self-collision rate for every setting. We initialize the robot to the same default configuration and before exccution.
Analysis of Success and Failure Modes: Our method succeeds most of times on easy tasks without visible performance drop in hard task. The failed trials of our method are largely due the mismatch between the actual cup position and the AprilTag position, which can be mitigated by using two AprilTags and averaging their poses (details in the Supplementary). Since the visual estimation not the focus of this work, we infer that our method has higher precision and higher efficiency tasks than MPC+IK. MPC+IK succeeds some the easy tasks and fails due to singularity or self-collision In hard tasks, the major failure cause is self-collision given the cup is too close to the
pick-up
7

---

Snapshots
Body Angle (deg)
20
Roll
Yaw
FR
0.5
Time
Figure 7: The arm follows demonstration trajectory to pick up cup while walking. The start position is p = (0.5, -0.5, - 1.2), at the right upper side of the robot and the end position is ymm = (0.55, -0.9, 0.4). on the left lower front side close to the ground. Tuy 2.5s. The robot initially stands on the ground and then is commanded by constant forward velocity ennd 0.35. Meanwhile the position command changes. When EE position command is high. the quadruped starts to walk without significant tilting behavior with a natural walking gait. As the EE position command moves below the torso, the quadruped starts to pitch downwards, roll the left and yaw slightly to the right to help the arm reach the goal.
body. Notice that the TTC of MPC+IK is also longer than our method because solving online IK and operational space control more computationally demanding than joint position control (ours).
Open-loop Control from Demonstration: In this part, we analyze how agile walking is coupled with dynamic arm movement, The robot is given pre-defined end-effector trajectory to follow in an open-loop manner while being commanded walk at the same time. Results in Figure 7 show agility and dynamic coordination of our legged manipulator uneven grass terrain powered by our whoy control method.
Related Work
Legged Locomotion Traditional model-based control methods for legged robots have shown success but often require controllers to be meticulously designed and many manual tunings [1, 3, 4, 5, 3 32, 33, 34, 35, 36, 6, 37]. The extra weight and movement of robot arm on top of the legged robot will make such design process more challenging. Recent advances in reinforcement learning enable legged robots to traverse challenging terrains and adapt to changing dynamics [38, 24, 22, 39, 40, 41, 42, 43, 44, 45, 46, 47, 17, 48, 49, 50). However these works only focus on the mobility part and few interactions with objects the environment by manipulation are studied.
Mobile Manipulation Adding mobility manipulation is studied in [51, 52, 53, 8, 55, 10, 9, 56, 57, 58]. Advances have also been made the field of biped humanoid [59, 60, 61, 62). More recently, Ma et al. [8] proposed using MPC controller track the desired end-effector position of the arm mounted on a quadruped with RL policy to maintain balance. However, the controllers for legs (RL) and arm (model-based) are separate modules and no dynamic movements demonstrated. In [55], language models are used to guide a mobile robot to finish different tasks using the arm. However, the manipulation and mobility are utilized in decoupled step-by-step manner.
5 Discussion and Limitations
We proposed hardware setup as well as an algorithm to learn whole-body control of a legged robot with robotic arm. Our policy shows coordination between legs and arm while being able to control them in a dynamic manner. Although we have shown preliminary results on object interaction (e.g. picking, pressing, erasing), incorporating general-purpose object interaction (e.g. occlusion and soft object) into the our unified policy is challenging open research direction. There are several ways in which the current methodology could be extended, such as, learning vision-based policies from egocentric camera mounted on torso [63] and on the arm, climbing the obstacle using front legs to pick something up on the table where the arm alone cannot reach, and etc. We believe this paper provides a first step towards several of such future directions.

---

Acknowledgments
We would like to thank Chris Atkeson for high-quality feedback, and Kenny Shaw, Russell Mendonca, Ellis Brown, Heng Yu, Unitree Roboties (Irving Chen, Yunguo Cui and Walter Wen) and staff at CMU Tech Spark for help in real-world experiments, hardware design and assembly. This work is supported in part by DARPA Machine Common Sense grant and ONR N00014-22-1-296
References
[1] H. Miura and Shimoyama. Dynamic walk of biped. IJRR, 1984.
[2] M. H. Raibert. Hopping in legged myddemg-modeling and simulation for the two-dimensional one-legged case. IEEE Transactions on Systems, Man, and Cybernetics, 1984.
[3] H. Geyer, A. Seyfarth, and R. Blickhan. Positive force feedback in bouncing gaits? Proceedings of the Royal Society of London. Series B: Biological Sciences, 2003.
[4] K. Yin, K. Loken, and M. Van de Panne. Simbicon: Simple biped locomotion control. ACM Transactions on Graphics 2007.
[5] K. Sreenath, H.-W. Park, I. Poulakakis, and J. W. Grizzle. A compliant hybrid zero dynamics controller for stable, efficient and fast bipedal walking on mabel. IJRR, 2011.
[6] M. Hutter, C. Gehring, D. Jud, A. Lauber, C. D. Bellicoso, V. Tsounis, J. Hwangbo, K. Bodie, P. Fankhauser, M. Bloesch, al. Anymal-a highly mobile and dynamic quadrupedal robot. In IROS, 2016.
[7] L. Sentis and O. Khatib. A whole-body control framework for humanoids operating in human environments. In ICRA, 2006.
[8] Y. Ma. F. Farshidian, T. Miki, Lee, and M. Hutter. Combining (earning-basee locomotion policy with model-based manipulation for legged mobile manipulators. RA-I., 2022.
[9] C. D. Bellicoso, K. Krämer, M. Stäuble, D. Sako, F. Jenelten, M. Bjelonic, and M. Hutter. Alma-articulated locomotion and manipulation for a torque-controllable robot. In ICRA, 2019.
[10] S. Zimmermann, R. Poranne, and S. Coros. Go fetch!-dynamic grasps using boston dynamics spot with external robotic arm. In ICRA, 2021.
[11] N. Bernstein. The co-ordination and regulation of movements. The co-ordination and regulation of movements, 1966.
[12] M. L. Latash. The bliss (not the problem) of motor abundance (not redundancy). Experimental brain research, 217(1):1-5, 2012.
[13] M. Bruton and N. O'Dwyer. Synergies in coordination: comprehensive overview of neural, computational, and behavioral approaches. Journal of Neurophysiology, 12066):261-22774, 2018.
[14] B. Dynamics. Spot Arm. http::/wwwwwuuiitre www.uuiiieeeccoom com/proouucss/g///
[15] X. Wang. Unitree gol. https: h/wp::://www..
[16] S. Kataoka, S. K. S. Ghasemipour; D. Freeman, and I. Mordatch. Bi-manual manipulation and attachment via sim-to-real reinforcement learning. arXiv preprint arXiv:2203.08227. 2022.
[17] Z. Fu, A. Kumar, J. Malik, and D. Pathak. Minimizing energy consumption leads to the emergence of gaits in legged robots. In CoRL, 2021.
[18] G. B. Margolis, G. Yang. K. Paigwar, T. Chen, and P. Agrawal. Rapid locomotion via reinforce- ment learning. RSS, 2022.
[19] N. Rudin, D. Hoeller, P. Reist, and M. Hutter. Learning to walk in minutes using massively parallel deep reinforcement learning. In CoRL, 2022.

---

[20] I. Akkaya, M. Andrychowicz, M. Chociej. M. Litwin, B. McGrew, A. Petron, A. Paino, M. Plappert. G. Powell, R. Ribas, et al. Solving rubik's cube with a robot hand. arXiv preprint arXiv: 11Xiv:1910.07113, 2019.
[21] Schulman, Wolski, P. Dhariwal, A. Radford, and O. Klimov, Proximal policy optimization algorithms. arXiv:1707.06:2 2017.
[22] A. Kumar, Z. Fu, D. Pathak, and J. Malik. RMA: Rapid Motor Adaptation for Legged Robots. In RSS, 2021.
[23] G. B. Margolis, T. Chen, K. Paigwar, X. Fu, D. Kim, S. bae Kim, and P. Agrawal. Learning to jump from pixels. In CoRL, 2021.
[24] J. Lee, J. Hwangbo, L. Wellhausen, V. Koltun, and M. Hutter. Learning quadrupedal locomotion over challenging terrain. Science Robotics, 2020.
[25] T. Miki, J. Lee, J. Hwangbo, L. Wellhausen, V. Koltun, and M. Hutter. Learning robust perceptive locomotion for quadrupedal robots in the wild. Science Robotics, Jan. 2022.
[26] L. Weihs, U. Jain, I.JJ Liu, J. Salvador, S. Lazebnik, A. Kembhavi, and A. Schwing. Bridging the imitation gap by adaptive insubordination. NeurIPS, 2021.
[27] S. Levine, C. Finn, T. Darrell, and P. Abbeel. End-to-end training of deep visuomotor policies. JMLR, 2016.
[28] H. Wang and A. Banerjee. Bregman alternating direction method of multipliers. NeurIPS, 2014.
[29] WidowX 250 robot 6DOF X-Series robotic arm. https://www tto.srrro...r...rroot.ot........ http/vidoxx-255-rrodr arrm--rm---- 6dof. aspx.
[30] E. Olson. AprilTag: A robust and flexible visual fiducial system. In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), pages 3400-3407. IEEE, May 2011.
[31] A. M. Johnson, T. Libby, E. Chang-Siu, M. Tomizuka, R. J. Full, and D. E. Koditschek. Tail assisted dynamic self righting. In Adaptive Mobile Robotics. World Scientific, 2012.
[32] M. Khoramshahi, H. Bidgoly, S. Shafiee, A. Asaei, A. ljspeert, and M. N. Ahmadabadi. Piecewise linear spine for speed-energy efficiency trade-off in quadruped robots. Robotics and Autonomous Systems, 2013.
[33] A. D. Ames, K. Galloway, K. Sreenath, and J. W. Grizzle. Rapidly exponentially stabilizing control Lyapunov functions and hybrid zero dynamics. IEEE Transactions on Automatic Control, 2014.
[34] D. J. Hyun, Lee, S. Park, and S. Kim. Implementation of trot-to-gallop transition and subsequent gallop on the mit cheetah i. IJRR, 2016.
[35] M. Barragan, N. Flowers, and M. Johnson. MiniRHex small, open-source, fully pro- grammable walking hexapod. In RSS Workshop, 2018.
[36] G. Bledt, M. J. Powell, B. Katz, J. Di Carlo, P. M. Wensing, and S. Kim. Mit cheetah 3: Design and control of a robust, dynamic quadruped robot. In IROS, 2018.
[37] C. S. Imai, M. Zhang. Y. Zhang, M. Kierebinski. R. Yang, Y. Qin, and X. Wang. Vision-guides quadrupedal locomotion in the wild with multi-modal delay randomization. 2021.
[38] J. Hwangbo,, J. Lee, A. Dosovitskiy, D. Bellicoso, V. Tsounis, V. Koltun, and M. Hutter. Learning agile and dynamic motor skills for legged robots. Science Robotics, 2019.
[39] 7. Li, X. Cheng, Peng, P. Abbeel, Levine, Berseth, and K. Sreenath. Reinforcement learning for robust parameterized locomotion control of bipedal robots. In ICRA, 2021.
10

---

[40] X. B. Peng, E. Coumans, T. Zhang. T.-W. E. Lee, J. Tan, and S. Levine. Learning agile robotic locomotion skills by imitating animals. In RSS, 2020.
[41] W. Yu, J. Tan, C. K. Liu, and G. Turk. Preparing for the unknown: Learning universal policy with online system identification. In RSS, 2017.
[42] W. Yu, C. K. Liu, and G. Turk. Policy transfer with strategy optimization. In ICLR, 2018.
[43] W. Zhou, L. Pinto, and A. Gupta. Environment probing interaction policies. In ICLR, 2019.
[44] W. Yu. V. C. V. Kumar, G. Turk, and C. K. Liu. Sim-to-real transfer for biped locomotion. In IROS, 2019.
[45] W. Yu, Tan, Y. Bai, E. Coumans, and S. Ha. Learning fast adaptation with meta strategy optimization. RA-L, 2020.
[46] X. Song, Y. Yang, Choromanski, K. Caluwaerts, W. Gao, C. Finn, and Tan. Rapidly adaptable legged robots via evolutionary meta-learning, In IROS, 2020.
[47] I. Clavera, A. Nagabandi, S. Liu, R. Fearing, Abbeel, S. Levine, and C. Finn. Learning to adapt in dynamic, real-world environments through meta-reinforcement learning. In ICLR, 2019.
[48] L. Smith, J. C. Kew, X. B. Peng. S. Ha, J. Tan, and S. Levine. Legged robots that keep on learning: Fine-tuning locomotion policies in the real world. In ICRA, 2022.
[49] R. Yang, M. Zhang, N. Hansen, H. Xu, and X. Wang. Learning vision sidion- quadrupedal locomotion end-to-end with cross-modal transformers. In ICLR, 2022.
[50] Z. Fu, A. Kumar, A. Agarwal, H. Qi, J. Malik, and D. Pathak. Coupling vision and proprioception for navigation of legged robots. CVPR, 2022.
[51] C. Sun, J. Orbik, C. M. Devin, B. H. Yang, A. Gupta, G. Berseth, and S. Levine Fully autonomous real-world reinforcement learning with applications to mobile manipulation. In CoRL, 2022.
[52] J. Wong, A. Tung, A. Kurenkov, A. Mandlekar, L. Fei-Fei, S. Savarese, and R. Martín-Martin Error-aware imitation learning from teleoperation data for mobile manipulation. In CoRL, 2022.
[53] D. Honerkamp, T. Welschehold, and A. Valada. Learning kinematic feasibility for mobile manipulation through deep reinforcement learning. RA-L, 2021.
[54] X. Ding and F. Yang. Study on hexapod robot manipulation using legs. Robotica, 34(2):
468 468-481 181, Feb. 2016.
[55] M. Ahn, A. Brohan, N. Brown, Y. Chebotar, O. Cortes, B. David, C. Finn, K. Gopalakrishnan. K. Hausman, A. Herzog, D. Ho, J. Hsu, J. Ibarz, B. Ichter, A. Irpan, E. Jang, R. J. Ruano, K. Jeffrey, S. Jesmonth, N. J. Joshi, R. Julian, D. Kalashnikov, Y. Kuang, K.-H. Lee, S. Levine, Lu, L. Luu, C. Parada, P. Pastor, J. Quiambao, K. Rao, J. Rettinghouse, D. Reyes, P. Sermanet, N. Sievers, C. Tan, A. Toshev, V. Vanhoucke, F. Xia, T. Xiao, Xu, S. Xu, and M. Yan. Do as can, not as say: Grounding language in robotic affordances. 2022.
[56] Xia, C. Li, R. Martín-Martin, O. Litany, A. Toshev, and S. Savarese. Relmogen: Lever- aging motion generation in reinforcement learning for mobile manipulation. arXiv preprint Sttt::2008.077999 2020.
[57] K. Dong, K. Pereida, F. Shkurti, and A. P. Schoellig. Catch the ball: Accurate high-speed motions for mobile manipulators via inverse dynamics learning. In IROS, 2020.
[58] K. Blomqvist, M. Breyer, A. Cramariuc, J. Förster, M. Grinvald, F. Tschopp, J. J. Chung, L. Ott, Nieto, and R. Siegwart. Go fetch: Mobile manipulation in unstructured environments. ICRA Workshop, 2020.
11

---

[59] S. Kuindersma, R. Deits, M. Fallon, A. Valenzuela, H. Dai, F. Permenter, T. Koolen, P. Marion, and R. Tedrake. Optimization based locomotion planning, estimation, and control design for the atlas humanoid robot. Autonomous robots, 2016.
[60] S. Feng, E. Whitman, X. Xinjilefu, and C. G. Atkeson. Optimization based full body control for the atlas robot. In International Conference on Humanoid Robots, 2014.
[61] J. Chestnutt, M. Lau, G. Cheung, J. Kuffner, J. Hodgins, and T. Kanade. Footstep planning for the honda asimo humanoid. In ICRA, 2005.
[62] N. A. Radford, Strawser, K. Hambuchen, J. S. Mchling, W. K. Verdeyen, A. S. Donnan, J. Holley, J. Sanchez, V. Nguyen, L. Bridgwater, et al. Valkyrie: Nasa's 's first bipedal humanoid robot. Journal of Field Robotics, 2015.
[63] A. Agarwal, A. Kumar, J. Malik, and D. Pathak. Legged locomotion in challenging terrains using egocentric vision. In Conference on Robot Learning (CoRL., 2022.
12


Reviews:
[{"review_id": "sg_FezEskpy", "reviewer": "Reviewer_3ot8", "summary": "This paper presents a novel learning framework for quadruped to learn coordinated dynamic manipulation and locomotion tasks. Instead of training two separate policies for manipulation and locomotion tasks, the proposed approach learns a single unified policy for arm and leg joints motion. The authors use advantage mixing to reduce the coupling between manipulation and locomotion tasks, thereby improving the learning efficiency. Moreover, the authors propose a novel Regularized Online Adaptation algorithm (similar to RMA) to improve the sim-to-real transfer performance. The proposed framework was in simulation and real-world experimentations through several tests. The results show improved learning efficiency, operation capability, and success rates over baseline approaches.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strength\n\n- It uses advantage mixing to improve the learning efficiency of coupled tasks without compromising on traditional decoupled loco-manipulation approaches.\n- This paper shows that coordinating arm manipulation and leg locomotion could improve the robot’s operation space and overall stability.\n- The Regularized Online Adaptation algorithm appears to provide good sim-to-real performance.\n- The authors clearly explained their simulation and experimental setups and results.\n- The trained neural-network policy can realize faster motion when compared to online optimal control (MPC-IK).\n\n### Weakness:\n\n- The dimension of the inputs of the unified policy network is large, which would potentially lead to a very large neural network. While this is a common approach in many end-to-end learning for robotics, it may lead to sampling inefficiency and extended training time. How long does it take to train a policy in simulation?\n- The policy outputs joint position commands and then uses PD controllers to track the target positions. However, the joint tracking performance in simulation and robot hardware could be significantly large due to model mismatch, potentially introducing a large sim-to-real gap. How did you account for the PD tracking errors in experiments?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Overall, the paper is well written and the results are interesting. While the robot does not move as smooth as SOTA robots in experiment, it is understandable considering the relatively cheap robotic arm being used. The authors presented a comprehensive framework to learn coordinated loco-manipulation policy with an ability to bridge the sim-to-real transfer gap. While some additional details (such as training time, etc.) can be provided, the work presented would provide an excellent example of the advantages of coordinated loco-manipulation policies in legged robots. I highly recommend to accept the paper to CoRL.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "- Some additional details regarding learning efficiency can be provided.\n- How did the difference in PD tracking in simulation and hardware being addressed?\n- There are some minor typos. Please carefully proofread the paper."}}, {"review_id": "sNV7CttgMg1", "reviewer": "Reviewer_Hcpy", "summary": "The paper presents a novel method for whole body control of a legged manipulator using sim-to-real reinforcement learning. The method is based on a teacher-student framework similar to [1], where an encoder uses privileged information in simulation to predict latent environment extrinsics and an adaptation module learns to mimic it from a history of sensor measurements. The output of the adaptation module is fed into a whole body control policy, and the entire system is trained end-to-end. In contrast to [1], the encoder output is regularized to overcome the realizability gap, and the system is trained in a single phase. The authors also propose advantage mixing to provide a natural curriculum for policy learning. Extensive simulation and hardware experiments demonstrate that the learned controller can successfully coordinate the quadrupedal base and manipulator to perform a variety of complex tasks.\n\n[1] A. Kumar, et. al., RMA: Rapid Motor Adaptation for Legged Robots, RSS 2021.\n", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\nComprehensive Experimental Evaluation\n\nThe real-world robot experiments are an impressive demonstration of learning based control working on complex tasks that require coordination between large degrees of freedom as well as generating dynamic motions. The simulated experiments and baseline comparisons clearly demonstrate the benefits of every of the proposed training methodology.\n\nTraining \n\nThe paper provides novel contributions to the training strategy. Regularizing the encoder output to overcome the realizability gap allows training the encoder and adaptation module in a single phase, which helps reduce the overall training time. Furthermore, the idea of advantage mixing to exploit the causal structure of the locomotion and manipulation is a simple yet effective technique for avoiding local minima.\n\n**Weaknesses**\n\n1. It would be helpful if the authors discuss more details of the low-cost hardware setup and how it can be re-created by other researchers. \n\n2. The limitations section does not provide any insights into scenarios where the current method will fail and how it can be improved. In particular, the authors should discuss the limits of sim-to-real when it comes to both locomotion and manipulation (especially cases that involve object interaction that can be hard to simulate), and how the current methodology can be potentially adapted to them. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "My recommendation for strong accept is based on the impressive real-world results, a thorough evaluation of different components and interesting improvements in the training pipeline. In the future it would be interesting to see how such approaches can be used to autonomously perform more complicated and fine-grained manipulation tasks.  \n", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "The paper draft can be significantly improved from a proof-reading pass to improve several grammatical errors. A few examples,\n\nLine 107 - “rely on” instead of “relies on”\nLine 127 - “start” instead of “starts”\n\nThe authors should fix these in the final version and address the comments in the weaknesses section.\n"}}, {"review_id": "WS5G8IKokqu", "reviewer": "Reviewer_KgX5", "summary": "The authors present a lowcost quadrupped robot with a manipulator. They then introduce a controller that, given commands for the end-effector and base, learns a single policy to control both arm and locomotion to track this command.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- the advantage mixing approach seems simple and sensible for this domain\n- the regularization during the teaching process seems to work well\n- the experiments and videos demonstrate good behavior and the usefullness of the approach\n\nWeaknesses\n- the paper does not make fully clear from the beginning that the task is to follow given commands. E.g. l76 p^cmd, v^cmd are just appearing as given. It would be beneficial for clarity to describe that the task is to follow given commands earlier on.\n- the paper proposes to jointly train teacher and student instead of in two phases, but it does not evaluate the impact of this. It would be good to compare to the \"standard\" approach of training in two phases.\n- generally the experiments could benefit from a few more additions, comparisons, see list of issues below\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Overall a good paper. Though additional comparison and a bit more clarity in certain parts would be very beneficial.\n\nPost rebuttal: the authors have addressed my main concerns, with the changes discussed in the process I think the paper is ready to be accepted and I am happy to raise my score", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {"issues": "- l65: what does low-cost mean in comparison to the 100k for ANYmal?\n- l141: \"This optimization process is known to converge.\" I believe convergence is only guarantueed under certain assumptions.\n- l166: Why not give the \"separate\" baseline access to the same observations, i.e. both base and arm state?\n- Minor: there are a number of spelling/grammar mistakes that could be cleared up \n- l200: This section should make clear what is OOD. At the moment the reference to table 3 is hidden in the caption.\n- l239: TTC is never explained in the text\n- Table 7: would be beneficial to not just have 1 difficult point\n- l 251: grouping 17 references into a single parantheses seems a bit much\n- Sec. 5: obstacle avoidance? As I understand it, the current policy is blind?"}}, {"review_id": "STZCCzklJ2_", "reviewer": "Reviewer_bkQw", "summary": "This paper presents a method for learning a coordinated policy to control a quadruped with mounted arm. There are two key algorithmic components. The first is advantage mixing, wherein the locomotion and manipulation policies are first trained independently, then gradually integrated together with shared reward terms. The second key algorithmic component is Regularized Online Adaptation, a modification of RMA where the environment encoder with privileged information is updated throughout training to more closely match the adaptation module, which has no privileged information. The resulting whole-body controller rolls and pitches the body to help the arm reach, as well as successfully trotting and reaching. This is demonstrated on hardware in three control scenarios.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n-  Very nice results. It's awesome to see the torso tilt to stretch the arm farther, especially in the video.\n-  Techniques are relatively simple, but they seem to make a big difference in the learned policy.\nThorough ablation studies.\n-  I like the idea of Regularized Online Adaptation. I wasn't convinced at first, but after thinking about it, here is my reasoning: Suppose there is some privileged information the encoder is encoding that the adaptation module simply cannot infer from the information it has. The policy may come to rely on that information, but the adaptation module cannot provide it. By training the encoder to match the adaptation module, you ensure the encoder isn't passing on any information the adaptation module cannot infer.\n\nWeaknesses:\n-  Section 2.2 (Regularized Online Adaptation for Sim-to-Real Transfer) could be clearer. In particular it would benefit from a brief summary of RMA and a more thorough explanation of the differences between Regularized Online Adaptation and RMA.\n-  Line 234 says \"The failed trials of our method are largely due to the mismatch between the actual cup position and the AR tag position.\" I understand that vision is not the focus of this work, but as this comparison with MPC+IK is an important result, this visual estimation issue should be resolved. As one simple option, why not put an April tag on each side of the cup and average the coordinates?\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "There are a couple places where the paper should be more clear, but overall it is convincing. The problem addressed is an interesting one: yes, robots with both arms and legs should ideally coordinate their movements, as humans do. The two non-standard algorithmic components (advantage mixing and regularized online adaptation) make sense and are well-investigated with ablations and comparison methods. And the results look good: the robot twists its torso to help the arm reach.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {"issues": "-  Section 2.2 should be clarified. In particular it would benefit from a brief summary of RMA and a more thorough explanation of the differences between Regularized Online Adaptation and RMA.\n-  From line 93, I understood that the spherical coordinate system for the end effector was relative to the robot's torso, which would mean the legs could not help the arm reach. From the supplementary material I realize that the coordinate system is actually independent of the torso's height, roll, and pitch. This should be stated in the main paper.\n\nTypos:\n-  Line 2, \"picking in [cluttered] environments.\"\n-  Line 63, \"pre-designed controllers that [can't] be changed.\"\n-  Line 73, \"([roll], pitch, and base angular velocities)\"\n-  Line 139, \"[Lagrangian] multiplier\"\n-  Line 148, \"open-loop [replay] of human demonstrations\"\n"}}]

Author rebuttal:

Decision: Accept (oral)