Guided Reinforcement Learning for Robust Multi-Contact Loco-Manipulation
Authors: Jean Pierre Sleiman
Mayank Mittal
Marco Hutter
Venue: CoRL 2024 (2024)
Keywords: Whole-body Loco-Manipulation
Reinforcement Learning
Legged Mobile Manipulators
TL;DR: This work proposes a task-agnostic RL formulation that leverages a single demonstration to train robust policies for complex multi-contact behaviors, such as traversing spring loaded doors.

Abstract: Reinforcement learning (RL) has shown remarkable proficiency in developing robust control policies for contact-rich applications. However, it typically requires meticulous Markov Decision Process (MDP) designing tailored to each task and robotic platform. This work addresses this challenge by creating a systematic approach to behavior synthesis and control for multi-contact loco-manipulation.
We define a task-independent MDP formulation to learn robust RL policies using a single demonstration (per task) generated from a fast model-based trajectory optimization method. Our framework is validated on diverse real-world tasks, such as navigating spring-loaded doors and manipulating heavy dishwashers. The learned behaviors can handle dynamic uncertainties and external disturbances, showcasing recovery maneuvers, such as re-grasping objects during execution. Finally, we successfully transfer the policies to a real robot, demonstrating the approach's practical viability.

Guided Reinforcement Learning for Robust
Multi-Contact Loco-Manipulation
Jean- Sleiman" ETH Zurich jsleimandethz.c ch
Mayank Mittal" ETH Zurich and NVIDIA mi mittalma@e tta.ma@ethh..d
Marco Hutter ETH Zurich mahutter@ethz. ch
Abstract: Reinforcement learning (RL) often necessitates a meticulous Markov Decision Process (MDP) design tailored to each task. This work aims to address this challenge by proposing a systematic approach to behavior synthesis and con- trol for multi-contact mec-mmanipulation tasks, such as navigating spring-loadet doors and manipulating heavy dishwashers. We define task-independent MDP train RL policies using only single demonstration per task generated from model-based trajectory optimizer. Our approach incorporates an adaptive phase dynamics formulation to robustly track the demonstrations while accommodating dynamic uncertainties and external disturbances. We compare our method against prior motion imitation RL works and show that the learned policies achieve higher success rates across all considered tasks. These policies learn recovery maneuvers that are not present in the demonstration, such as re-grasping objects during ex- ecution or dealing with slippages. Finally, successfully transfer the policies to real robot, demonstrating the practical viability of our approach. For videos, please check: https:///oggeerrboiicc githubi..hgu........ io/ htt///uiderrrrrr ra-oodrrraloa/////
Keywords: Loco-Manipulation, Reinforcement I Learning, Motion Imitation
Single demonstration
Large-scale simulation-bass based RL
Sim aon-to-real transfer
Start
Guided exploration using demonstration
(1.))
Task-agnostic rewards specification
Learned reacitivity and robustness
Figure 1: A framework for learning Dcco-maniuulation tasks, such traversing spring-loaded loaded door and ma- nipulating dishwashers. single demonstration guides the RL training process to learn multi-contact behaviors (such as using the feet or the arm for interaction) without task-specific handcrafted rewards.
Introduction
Model-based optimal control techniques, such as trajectory optimization (TO) and model-predictio control (MPC), valued for their inherent versatility. They generate range of easily inter- pretable behaviors, facilitating intuitive adjustments the problem formulation [1, 3]. However, these methods are sensitive to modeling mismatches and violation of assumptions. Conversely, rein- forcement learning (RL) has demonstrated remarkable success in developing robust control policies for various contact-rich tasks, such as legged locomotion [4, 5]. micommanipulation [6, 7]. and dex- terous manipulation [8, 9]. Despite this, RL suffers from high sample inefficiency, the emergence of
'Tese authors contributed equally. Their names listed in alphabetical order.
8th Conference Robot Learning (CoRL. 2024), Munich, Germany.

---

unnatural behaviors, and the need for labor-intensive reward design. In this work, we aim to leverage the complementary strengths of TO and RL to mitigate their weaknesses.
We build upon the versatile TO-based framework from Sleiman et al. [10], which demonstrates the discovery of long-horizon multi-modal behaviors for poly-articulated systems. This framework can safely and reliably solve complex loco- cerommaapulation tasks, accounting for the multiple interaction regions in the object (handle, surfaces) and the robot (individual limbs). These behaviors are chal- lenging to achieve with pure RL, even with an extensive task- sesk-specific design process. However, the successful execution of TO-based behaviors depends heavily on reasonably accurate environment models and minimal external disturbances. This reliance is due to the underlying MPC-based con- troller's strict adherence to tracking open-loop references without the ability replan or respond to significant deviations, such slippages. This limitation of the MPC controller motivates our work: using RL obtain policies for the robust execution of contact-rich interaction plans.
In this paper, we propose a Markov Decision Process (MDP) to efficiently train robust loco- manipulation policies, particularly for articulated object interaction, with task-agnostic rewards and hyperparameter tuning. Our approach uses dynamically feasible demonstrations generated from a TO-based framework [10] to guide the RL agent in learning complex behaviors. By maintaining consistent MDP parameters and utilizing only one demonstration per task, we present an approach to train control policies that track the generated demonstrations while handling modeling uncertainties, external disturbances, and unforeseen events such as handle slippage. We benchmark our approach against prior motion imitation works on four loco- loco- manipulation tasks: door pushing and pulling and dishwasher opening and closing. Finally, we deploy the trained policies on a quadrupedal mobile manipulator, showcasing their robustness to unknown object models and reactivity to slippages.
2 Related Work
Imitation learning has been successfully applied to various manipulation tasks through its algorith-
mic variants such as behavioral cloning (BC) [11], nemoo-uggmettd policy gradient [12, 13] (which bootstraps the RL policy through BC), and more recently, generative models [14, 15]. However, these approaches often struggle with distributional shifts and require substantial training data, pos- ing challenges for high-dimensional robotic systems.
An alternate class of methods relies on state-only references to guide an RL agent toward desired behaviors. This approach commonly involves conditioning a low-level policy on references ob- tained either offline from motion library or online by a separate high-level module. For instance, Bergamin et al. [16] incorporates motion matcher that selects and blends the best-fitting motion clip from an extensive MoCap database based on handerafted features. The works from Kang al. [17] and Jenelten et al. [18] update the policy with on-demand "optimal" references that are com- puted in an MPC-like fashion. While MPC can provide precise guidance, its high computational demands can significantly slow down the RL training process.
Another common approach is inverse RL, where a reward function is defined through the demon- strations. Adversarial motion priors (AMP) [19, 20, 21] formulate style reward that is maximized when the learned policy yields state transitions similar those in the dataset. Subsequent vari- ants [22, 23, 24] propose a hierarchical architecture to address the mode-collapse issue in adversar- ial learning setups. However, these methods still require task-specific goal-directing rewards and a large amount of data to learn the style reward. In contrast to AMP-style approaches, motion- imitation methods guide RL policies to robustly track target trajectories directly [25, 26, 27, 28). They use reward terms that encourage adherence to the demonstrations while allowing exploration around the reference motions.
Our work falls into the category of motion-imitation RL without any task-specific objectives. It closely relates to the work from Fuchioka et al. [26] and Bogdanovic et al. [27], as we aim to directly track and robustly stabilize offline trajectories generated using a fast trajectory optimizer. Differently from these works, we introduce domain-specific considerations relevant to multi-contact loco-manipulation tasks. Moreover, we propose a task-independent MDP formulation based on an

---

Locomanipulation Planner
Demonstration Guided Controller
Environment
Inputss
MCP
Reforence Solector
Sparse Desorpttonn
Bobonar Object
X
rr
Policy
Roloor
Oiject affonds
M
Adeptice Phase Dynemics
f(b. m. 6uk
6
Zp. 3p
The loco-manipulation planner [10] generates references in the form of multi-modal plans con- sisting of continuous trajectories X and manipulation schedules M These are used by the demonstration- guided controller to select x" and m² adaptively based on the task phase & and track them robustly. The controller receives full-state feedback and sends joint position commands to the robot.
Figure 2:
adaptive phase dynamics model, which is crucial for successful task execution in the presence of large modeling uncertainties and external disturbances.
3 Approach
Our proposed approach consists of two steps, as illustrated in Fig. 2. First, we employ the plan- ner from Sleiman et al. [10] to generate whole-body multi-contact behaviors for various loco- manipulation tasks. Subsequently, we train a neural network using RL to reliably track these behav- iors while leveraging only one pre-computed trajectory per task as an "expert demonstration". We train this policy entirely in simulation with domain randomization to achieve a successful transfer to the real robot. At deployment, the applicabiliity in the wild, our primary focus is devising an MDP for learning robust behaviors across different loco-manipulation scenarios without requiring task-specific handcrafting.
assume access to the object's state. While this assumption limits
3.1 Problem Formulation
We generate a single demonstration for each task using the multi-contact planner (MCP) from [10]. This planner takes the task specification and set of user-defined object affordances (e.g. handle or surfaces) and the robot's end-effectors (e.g. the tool on the arm or the feet) for interaction as inputs. It then searches for possible robot-object interactions to provide a physically consistent demonstration based on a nominal robot and object model.
The demonstrations from the planner consist of the continuous robot and object state references X* -[x; )=s and the manipulation schedule M (m;) Jrme with Tamk being the demonstra- tion's duration. In the case of articulated object interaction, the state reference æt includes the robot's base pose (rrîB. biB) in fixed inertial frame T, the robot's joint positions q;. and the object's joint positions e The contact mode m;; specifies the interaction type for each end-effector (none, prehensile, or non- -reeensees) and the object contacts involved at that timestep. Since the demonstrations for each task have varying lengths, we encode the notion of time into the task-phase [0, 1], where o 0 and ø imply the start and end of the demonstration, respectively [25].
The objective of the learning agent is track these references while staying robust against variations in the kinematic and dynamic properties of the object, external disturbances, and unforeseen events such as slippages. In the naïvest way, one could increase the task phase linearly with time and track the corresponding references [25, 26). However, as we show later in Sec. 5. this approach limits the success rate during object manipulation. For instance, if the door handle slips away and the door closes, strictly following the references will cause the robot get stuck behind the door. Instead, want give the robot the time recover and, thus, grow the task phase adaptively.
3.2 Adaptive Task Phase Dynamics
Linearly evolving the task phase with time (also referred to in this paper as the "nominal" for- mulation) can be written as: Φ:+1 o, + - the environment time-step. As motivated earlier, seek adaptive mechanism that adjusts the phase & depending on the current robot and object states. To this end, propose the phase dy-
Tie dt, where o denotes the nominal phase and dt is
3

---

Trask dt, where the task phase rate uf = 0f + oi dv is determined by a state-dependent reference u = (a x², m²) and learnable residual term 6, (scaled by 01 >> 0). We opt for reward-dependent that reflects the task-level tracking accuracy:
namics: P1+1= =
of
i (a. x." m II min (k- ex-exxp(-x (2, x", m 1).
(1)
where 1 and < are fixed constants, and L is the number of task-level rewards Ti,t, with rs.e S 0 at all time-steps. For articulated object interaction, L with terms corresponding the tracking rewards for the robot's base pose and the object's joint positions, specified in Sec. 3.3.
Intuition behind 6 The term 8 effectively pauses the phase evolution for large deviations from the current reference. As the tracking improves, it gradually approaches the nominal phase rate (++ 1). Since adhering strictly to the reference may not always be the best strategy to complete the task, we introduce a clipping in Eq. to encode perfect tracking only within a certain margin defined by l and X. This operation is particularly useful when handling discrepancies between the nominal model used by the planner and the real model employed during training and deployment. The resulting task phase dynamics with 8 induces curriculum-like like effect during training. At the start of the training, when the tracking is poor, it helps the agent learn recover to single reference state. Eventually, the agent improves, it learns to follow more of the demonstration.
Intuition behind d.: In some instances, unforeseen slippage or large disturbances could render the object uncontrollable due to complete loss of contact, resulting in significant deviations from the reference pose. In these situations, the term 6 + O, and the robot cannot recover. To enable motion recovery in such scenarios, we introduce a residual phase 6e that allows potentially speeding up, slowing down, and even decreasing the phase whenever necessary. This residual phase outputted from the policy, allowing adapt the task phase dynamics via learnable parameters.
3.3 Rewards
To design task-agnostic reward functions such that loco meco-manipulation behaviors in the generated demonstrations can be learned through the same MDP formulation, we split the rewards into three parts: aeffrrenc--taccking rewards, task progress rewards, and penalties for smooth motions.
frackkng reward: These are simply defined through the tracking errors between the current state æ and the reference state x= from the demonstration:
y track
|| rri 11 + W2 1| E 1 W3 /|g; 97 1
(2)
+ W4 1biject 11q. 112 + ws pprehensite ||TTEE irrnll
where wi 0, H is non-fixed handle frame that is initialized at and moves with the object, and 1.. 1.....1.. (m²) denotes indicator functions that depend the reference manipulation mode m+.
The indicator functions help avoid incurring irrelevant penalties that do not coincide with the behav- ior's mode schedule. The function Lojject is true only when at least one end-effector is either already in contact or is establishing contact with the object. Essentially, it activates the object tracking er- ror only when the object's state is controllable and essential for the task completion. Meanwhile, prekensite true when prehensile interaction is active. This choice helps improve the accuracy of prehensile contact and is particularly useful when randomize the object models during training, as the location of the object frame H varies.
Task progress reward: This reward term encourages the learning agent to progress in the task. is defined as: , = 81 [6 exp(-K2 | - where K1, K2 > O, and e and ø are the reference task-phase rate and the nominal phase from Sec. 3.2. When the tracking is poor, 80 + and the agent gets less reward for the task progress, i.e., yP - 0. Consequently, the agent gets encouraged maximize the reference tracking reward and minimize the tracking deviation.
These terms penalize the robot's joint accelerations and torques, base velocity in unde- sired directions, and abrupt action changes. Since these standard penalties used in RL, we skip detailing them here for space reasons and refer the reader to Appendix C.2 for further details.
Penalties:

---

3,4 Observation and Action Spaces
Observations: The observations comprise the tracking errors in the robot and object states, the po- sitions and velocities of all end-effector frames participating in prehensile interactions, the previous action, and the task-phase parameters. For legged mobile manipulator with only one prehensile end-effector E on the arm, the observations can be denoted Or (oe O2 Op a:-1 ob), with:
ITIB ITiB
B
IB _ $
BWIB dj Op
ITIE EVIF
0e
Qj qj
06
0
(3)
qo %
d
The choice of the observation terms Oe and o, resemble that of classical PD control law, where o, captures the position tracking error and o, is the velocity tracking with zero velocity targets. Effectively, the learned policy is Additionally, while including Op is not essential, we notice that having it improves the training it climinates the need to learn the mapping from the robot's state to operational space quantities.
complex tracking controller around the demonstrations.
The actions a, = (a ou,e) are interpreted as the residuals over the robot's reference joint positions qj., and the reference phase rate 0 from Sec. 3.2. The robot's actions are sent its actuators as joint position commands: gf = Th + 02 agg, with 02 > 0. While the planner [10] also provides reference joint velocities, we observed that including them in feed-forward control resulted in poorer performance. This finding aligns with that from Fuchioka et [26].
Actions:
3.5 Training Setup
Initial Distribution: At environment resets, we apply the reference state initialization (RSI) strat- egy from imitation-based RL setups [25, 26]. In RSI, we randomly sample the initial phase Pinit [0, 1] and spawn the robot and object at their corresponding reference states. However, differently from RSI, we randomize the robot configuration uniformly around the reference state when snit O. This alleviates the need always start the robot the initial reference state deployment and lets the agent learn the necessary recovery actions to stay near the demonstration.
The learning task then to complete the remainder of the task from these varying initial states. This approach exposes the policy to regions of the state space that are relevant to reference behavior and are hard reach independently. Additionally, removes the need for the episode length to be least long the demonstration Trask-
Domain Randomization (DR): Unlike [10], where the MPC-based tracking controller's 's success relies on reasonable knowledge of the environment, we aim to make the learned policy robust to modeling uncertainties and disturbances through DR. This involves varying the properties of the robot (such as friction and base mass), the kinematic and dynamic models of the object (such as the door dimensions and spring loading parameters), and adding regular external disturbances to the system. For further details, please refer to Appendix C.4
Curriculum: We introduce a curriculum that incrementally increases the external disturbances, observation noise, reward penalties, and initialization offsets. The curriculum level brand E [0, 1] is updated according to the following rule:
Ak rand
Fand 0.25 if Pe < 0.75 with Pa. = Oinit
rand + 0.25 if Ps > 0.95
(4)
Dinit
The term Pe represents the progress of the task phase relative to the nominal one. As the agent becomes proficient in tracking the demonstrations, it progresses to higher difficulty level with larger DR and penalties. Conversely, if its performance is poor, moves lower level.
Experimental Setup
We validate our framework legged mobile manipulator consisting of the quadruped ANYmal-D [29] with 6-DoF robotic arm [30]. We consider four tasks: Door Push/Pull: traversing spring- loaded push or pull door, and Dishwasher Open/Close: opening closing a heavy dishwasher.
5

---

Posh
Distwassber Open
ARM
Dishwaeher Close
Door Pull
10
10
Time
Nonprehenstle Contacts
Prehanstle Arm Contact
Prehonatic Contact
Nomprahonate Surface Contact
Figure Manipulation schedules in the generated mocommaniuulation demonstrations. the quadrupedal mobile manipulator, ALMA, the end-effectors are: ARM: tool attached a 6-DoF robotic arm, LF: left front foot, RF: right front foot, LI:: left hind foot, and RH: right hind foot.
Demonstration generation: Fig. illustrates the manipulation schedules M of the generated demonstrations using [10]. Simpler tasks like door pushing are solvable with single contact mode, while more complex tasks, such as door pulling, require sequencing multiple robot-object interaction modes. Navigating through a spring-loaded pull door stands out as the most complex task, both in terms of behavior discovery and execution time. For additional details, please check Appendix B.
Large-scale RL training: We simulate the training environment NVIDIA Isaac Gym [31]. The simulator runs at 200 Hz. and we decimate the policy to run 50 Hz. All policies are trained using the Proximal Policy Optimization (PPO) [32] algorithm. Training converges within 6 hours per task single NVIDIA RTX 4090Ti. Notably, we apply the same MDP formulation across all tasks with identical environment and agent hyperparameters. These are specified in Appendix C.
Baselines: Behavior cloning (BC) requires labeled actions and large training dataset. Given that we operate with a few demonstrations that also lack action labels, we do not compare our method against BC. Instead, we evaluate our method against other motion-imitation RL formulations:
Nominal phase: This formulation is based on existing works in motion motion-imitation RL [25, 26]. It uses the task phase dynamics with و = and no learned residual phase rate.
Nearest Neighbor Search (NNS): An alternative phase update mechanism inspired by the time-independent training stage of [27]. It aims to minimize the distance between the current system state and the closest reference in the demonstration dataset.
5 Results and Analysis
5.1 Simulation Experiments
We evaluate each learned policy in 4096 environments with randomized robot and object properties and random pushes. We report the success rates across the four tasks in Table 1. We observe that our proposed method performs significantly better than other motion-imitation RL setups, without including the residual phase action 6,. This result highlights the significance of integrating the reward-driven adaptive phase dynamics. The nominal phase update can reliably accomplish the relatively straightforward task of navigating through push doors. However, it strug- gles with more complex tasks, achieving notably lower success rates for door pulling. Meanwhile, the NNS strategy fails across all the tasks. While experimenting with various training setups to in- corporate NNS, we consistently found the phase stuck at certain points along the reference path. We hypothesize this occurs because the policy exploits the NNS update mechanism to achieve a state where the phase remains unchanged, thereby maximizing the reference tracking reward.
Comparisons:
Ablations: To evaluate the need for demonstrations as whole-bode body trajectories, omit the robot's joint-level references, relying on partial task-level references that pertain solely to the floating base

---

Table I: Comparison and ablation study of different MDP formulations to train RL policies for the four loco- manipulation tasks. The average success rate is reported over 4096 episodes, with success defined the successful execution of more than 95% of the demonstration.
Category
Door Push
Dishwasher
Dishwasher Close
Door Pull
Open
NNS-based
00.00%
00.00%
00.00%
00.00%
Nominal Φ
76.83%
53,44%
70.68%
15.38%
Adaptive w/ Residual 6, (Ours)
98.36%
98.60%
96.46%
96.33%
Ours w/o Joint References
97.17%
00.00%
00.00%
00.00%
Ours w/o Curriculum
97. 97.69%
95.25%
97.38%
95.46%
Ours w/o Residual 6g
96.80%
97.22%
97.68%
95.21%
Reference
Adaptive
Adaptive w/ Residual
With nominal
With
1.75
Door Augle (vad)
1.25-
Success Ratio
0.75
0.50
0.75
0.50
0.00
0.00
1.00
Door (in
1.00
0.80
Task Phuse
0.60
Shozrs Ratio
0.75
0.50
0.20
0.25
0.00
0.00
15 (5)
20
25
0
15
20
25
3.0 60 9.0 120150180
Time (S)
Handle Offiet (in
Figure 4: Trajectories for the door-pulling task involving loss of contact (indicated by shaded regions). The top and bottom rows show the door and task phase trajectories.
Figure Effect randomization for the door pulling task. Grey: The DR region used dur- ing training. Red: The nominal parameter.
and the object trajectories. As shown in Table 1, only the door pushing remains solvable under this scheme. The reliance on the robot's foot contacts for manipulation necessitates whole-body guidance to solve the other tasks. Moreover, we see that the curriculum from Sec. 3.5 helps improve tasks. During training, we observe that the curriculum also enables faster convergence and smoother policies. Adding the residual phase rate only improves the performance slightly in these comparisons. However, as shown in Fig. 4 for the door-pulling task, relying solely on the reference 6 for the phase dynamics brings the task progression to a halt during an inadvertent loss of contact. In contrast, the learned residual action decreases the phase, allowing the robot to re-establish contact with the handle and successfully complete the task.
the agent's performance across
Robustness Evaluation: While the generated loco-manipulation plan is done for nominal param- eters of the object, we train the RL policies with various randomizations, as mentioned in Sec. 3.5. To assess the robustness of the learned policies, we investigate their performance on different door parameters for the door-pulling task. We remove all DRs during these evaluations to ensure trolled setting. For comparison, we train an RL policy with no DR as an alternative to the MPC- based controller in [10]. Based on Fig. 5, the policy trained only with nominal door properties fails quickly when the variations become too large. In contrast, the policy trained with random doors supports a wider range of door parameters. When the handle offset is significant, the policy trained with random doors displays "search" behavior to find and grasp the handle. Meanwhile, the policy with nominal door only works reliably well around the handle length. This analysis demonstrates that although the generated demonstrations are not for the simulated door models, the RL policy discovers behaviors around them that help it generalize to large set of doors.
7

---

b
C
P
Time
Figure 6: Hardware deployment for traversing spring-loaded (a) pull and (b) push doors and manipulating a heavy dishwasher to (c) open and (d) close it. For videos, please check the website.
Multi-task Learning from Demonstrations We experiment with multi-task training setup where a single policy jointly learns all four tasks. It takes a one-hot vector to indicate the desired demon- stration to execute. We train the policy with the same hyperparameters and number of environments from before and obtain a reduced average success rate (84% instead of 97%). By simply doubling the number of environments during training (i.e., collecting twice the number of samples), ob- serve that achieves the performance of individual task-specific policies.
5.2 Real-World Deployment
We demonstrate the proposed approach on all four tasks on hardware. A motion capture system provides the handle location at the start and the joint angle for the door and the dishwasher. For the robot, rely its onboard state estimation. For each task, perform six continuous runs, where we manually reset the object, place the robot randomly in front of the object, and execute the policy. As shown in Fig. 6, the deployed policies exhibit smooth motions and preserve the multi-contact behavior from the demonstration. They complete all the tasks six times in a row. Additionally, the policies yield different motions every time, which particularly noticeable during traversing a pull door. They are able handle random base placement, recover from slippages, and re-grasp the handle when misses it. Please refer the supplementary video observe these behaviors.
5.3 Limitations
On hardware deployment, we notice that if the robot starts too far away or too close to the object, aggressively tries to adjust its configuration. This, at times, led the robot falling down. However, in practice, navigation planner can appropriately place the robot in front of the object, only local adjustments are needed, which the learned policies can handle. Additionally, while the policies can handle large intra-object category variations, they may still fail on certain object instances. For instance, if the door is too small, the policy fails to traverse through it as it collides with the door. In such cases, we need to use more demonstrations for the training. Lastly, a natural extension this work is to devise perceptive loco-manipulation policies that rely on onboard sensing [5, 7].
6 Conclusion
Our work integrates model-based TO and model-free RL. to develop robust tracking policies for multi-contact loco-manipulation tasks. Central to our approach a task-agnostic MDP formulation that utilizes loco- manppuullllnnnnn demonstrations generated by precise trajectory planner. Unlike pre- vious imitation-based RL methods that depend time-based nominal phase updates, we showed that state deltedde adaptive phase dynamics facilitate successful task execution despite model- ing inaccuracies and significant external disturbances. We validated our framework on quadrupedal mobile manipulator performing four complex long-horizon tasks, such as navigating spring-loaded doors and manipulating heavy dishwashers. Additionally, we demonstrated that the learned policies transfer to hardware successfully and can effectively recover from slippages and missed handles, overcoming the limitations of the MPC-based tracking controller from [10].

---

Acknowledgments
This research was supported by the European Union's Horizon Europe Framework Programme un- der grant agreements No 101070596, 101121321, and 852044.
References
[1] M. Bjelonic, R. Grandia, M. Geilinger, O. Harley, V. S. Medeiros, V. Pajovic, E. Jelavic,
S. Coros, and M. Hutter. Offline motion libraries and online mpe for advanced mobility skills. The International Journal of Robotics Research, 41, 2022.
[2] I. Mordatch, E. Todorov, and Z. Popovic. Discovery of complex behaviors through contact- invariant optimization. ACM Transaction of Graphics, 31(4):43:1-33:8 2012.
[3] Y. Tassa, T. Erez, and E. Todorov. Synthesis and stabilization of complex behaviors through online trajectory optimization. In IEEE/RSJ International Conference Intelligent Robots and Systems, pages 9906-4913. SEEE,2012
[4] X. Cheng, K. Shi, A. Agarwal, and D. Pathak. Extreme parkour with legged robots. In IEEE International Conference Robotics and Automation, pages 1143-114550 IEEE, 2023.
[5] Miki, J. Lee, Hwangbo. L. Wellhausen, V. Koltun, and M. Hutter. Learning robust per- ceptive locomotion for quadrupedal robots in the wild. Science Robotics, 7(62), 2022.
[6] Y. Ma, F. Farshidian, and M. Hutter. Learning arm-assisted fall damage reduction and re- covery for legged mobile manipulators. In IEEE International Conference on Robotics and Automation, pages 12149-12181 12155, IEEE, 2023.
[7] M. Liu, Z. Chen, X. Cheng, Ji, R. Qiu, Yang, and X. Wang. Visual whole-body control for legged loco-manipulation. In Conference on Robot Learning. PMLR, 2024.
[8] O. M. Andrychowicz, B. Baker, M. Chociej, Jozefowicz, B. McGrew, J. Pachocki, A. Petron, M. Plappert, G. Powell, Ray, al. Learning dexterous in-hand manipulation. The International Journal of Robotics Research, 39(1):3-20, 2020.
[9] A. Handa, A. Allshire, V. Makoviychuk, A. Petrenko, R. Singh, J. Liu, D. Makoviichuk, K. Van Wyk. A. Zhurkevich, B. Sundaralingam, et al. Dextreme: Transfer of agile in-hand manipulation from simulation reality. In IEEE International Conference Robotics and Automation, pages 5977-5984. IEEE, 2023.
[10] Sleiman, F. Farshidian, and M. Hutter. Versatile multicontact planning and control for legged loco muco-manipulation. Science Robotics, 8(81), 2023.
[11] A. Mandlekar, D. Xu, J. Wong, S. Nasiriany, C. Wang, R. Kulkarni, L. Fei-Fei, S. Savarese, Y. Zhu, and R. Martin-Martin. What matters in learning from offline human demonstrations for robot manipulation. In Conference on Robot Learning, pages 1678-1690 PMLR, 2021.
[12] A. Rajeswaran, V. Kumar, A. Gupta, G. Vezzani, J. Schulman, E. Todorov, and S. Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstra- tions. In Roboties: Science and Systems, 2018.
[13] Y. Qin, H. Su, and X. Wang- From one hand multiple hands: Imitation learning for dexterous manipulation from single-cames teleoperation. IEEE Robotics and Automation Letters, 7(4): 10873-10881, 2022.
[14] C. Chi, S. Feng, Y. Du, Z. Xu, E. Cousineau, B. Burchfiel, and S. Song. Diffusion policy: Visuomotor policy learning via action diffusion. In Robotics: Science and Systems, 2023.
[15] S. Lee, Y. Wang, H. Etukuru, H. Kim, N. M. M. Shafiullah, and L. Pinto. Behavior generation with latent actions. In International Conference Machine Learning, volume 235. pages 26991-27008. PMLR. 2024.

---

[16] K. Bergamin, S. Clavet, D. Holden, and J. R. Forbes. Drecon: driven responsive control of physics-based characters. ACM Transaction of Graphics, 38(6))200:1 1-::::::::::111111-1 2019.
[17] D. Kang, J. Cheng, M. Zamora, F. Zargarbashi, and S. Coros. RI model-based control: Using on-demand optimal control to learn versatile legged locomotion. IEEE Robotics and Automation Letters, 8(10):61196662 2023.
[18] F. Jenelten, J. He, F. Farshidian, and M. Hutter. DTC: deep tracking control. Science Robotics, 9(86), 2024. URL https: //tp:///iiiooo / 112//cirobotics adh5401.
[19] X. B. Peng, Z. Ma, P. Abbeel, S. Levine, and A. Kanazawa. AMP: adversarial motion priors for stylized physics-based character control. ACM Transaction of Graphics, 40(4))144:11144:20) 2021.
[20] A. Escontrela, X. B. Peng, W. Yu, T. Zhang, A. Iscen, K. Goldberg, and P. Abbeel. Adversarial motion priors make good substitutes for complex reward functions. In IEEE/RSJ International Conference Intelligent Robots and Systems, pages 25-32. IEEE, 2022.
[21] J. Bae, J. Won, D. Lim, C. Min, and Y. M. Kim. PMP: learning physically interact with environments using part-wise motion priors. In ACM SIGGRAPH, pages 64:1-64:10 ACM, 2023.
[22] X. B. Peng, Y. Guo, L. Halper, S. Levine, and S. Fidler. ASE: large-scale reusable adversarial skill embeddings for physically simulated characters. ACM Trunsaction of Graphics, 41(4):
94:1-94:17, 2022. URL https: /00g/////0/001/100/// 10.0...1555 /145/3528223.330 3533000. 3530110.
[23] C. Li, S. Blaes, P. Kolev, M. Vlastelica, J. Frey, and G. Martius. Versatile skill con- trol via self-supervised adversarial imitation of unlabeled mixed motions. In IEEE Inter- national Conference on Robotics and Automation, pages 2944-2950. IEEE, 2023. URL
https://dol: http://///oooo///.../// 1109/ 1.009/CCRA88991 2023. 10160421.
[24] C. Tessler, Y. Kasten, Y. Guo. S. Mannor, G. Chechik, and X. B. Peng. CALM: conditional adversarial latent models for directable virtual characters. In ACM SIGGRAPH, pages 37:1- 37:9. ACM, 2023. URL. /tttps:///o: htt.://..r///..11///1/111111
[25] X. B. Peng, P. Abbeel, S. Levine, and M. van de Panne. Deepmimic: example-guided deep reinforcement learning of physics-based character skills. ACM Transaction of Graphics, 37 (4):143, 2018.
[26] Y. Fuchioka, Z. Xie, and M. van de Panne. Opt-mimic: Imitation of optimized trajectories for dynamic quadruped behaviors. In IEEE International Conference on Robotics and Automation, pages 5092-5098. IEEE, 2023.
[27] M. Bogdanovic, M. Khadiv, and L. Righetti. Model-free reinforcement learning for robust locomotion using demonstrations from trajectory optimization. Frontiers Robotics AI, 9, 2022.
[28] S. Dasari, A. Gupta, and V. Kumar. Learning dexterous manipulation from exemplar object trajectories and pre-grasps. In IEEE International Conference on Robotics and Automation, pages 3889-3896. IEEE, 2023.
[29] ANYbotics.
ANYmal D Autonomous Legged Robot, 2024.
URL:
wttpiiic...cc.coooooomooooeeeeee com/ homm/anyall-aa -eggeal--egeed-robot
[30] DuaTic. DynaArm revolutionary lightweight yet highly dynamic robotic arm, 2024. URL: https://duatic..cmm
[31] V. Makoviychuk, Wawrzyniak, Guo, M. Lu, Storey, M. Macklin, Hoeller, N. Rudin, A. Allshire, A. Handa, and G. State. Isaac gym: High performance GPU based physics sim- ulation for robot learning. In Neural Information Processing Systems Track on Datasets and Benchmarks 1, 2021.
10

---

[32] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv, abs/1707.06347, 2017. URL. htt.ttt/p///iii org/abs/ http:///aaaaiiioorr/////////// 06347..
[33] M. Bloesch, M. Hutter, M. A. Hoepflinger, S. Leutenegger, S. Gehring, C. D. Remy, and R. Siegwart. State estimation for legged robots. consistent fusion of leg kinematics and imu. In Robotics: Science and Systems (RSS), 2012.
[34] Sleiman, Farshidian, M. V. Minniti, and M. Hutter. A unified MPC framework for whole- body dynamic locomotion and manipulation. IEEE Robotics and Aulomation Letters, 6(3):
4688 1598,4695,.0 2021.
[35] N. Rudin, D. Hoeller, P. Reist, and M. Hutter. Learning to walk in minutes using massively parallel deep reinforcement learning. In Conference Robot Learning, volume 164, pages 91-100. PMLR, 2021.

---

A Notations/ Symbols
Table Definition Symbols
Symbol
Description
g
Value of the quantity g from the demonstration
g
Value of the quantity from the nominal phase formulation
ABAB E R
Position of frame B from frame A in frame A Orientation of frame in frame A
PAB E SO(3)
APAB E R3
Linear velocity of frame B from frame A in frame A
AWAB E R3
Angular velocity of frame from frame A in frame A
8: SO(3) x S3(3) +33
Difference between two orientations as a rotation error vector [33]
B
Frame attached to the base of the robot
E
Frame attached to the arm's is end-effector of the robot Frame attached to the handle's center on the object
H
Inertial frame (fixed to the initial frame H on execution)
heom
Centroidal momentum of the robot
9b
(ITIB. IBB)
Base pose of the robot Joint positions of the robot
Qo
Joint positions of the object Joint velocities of the robot Joint velocities of the object Joint acceleration of the robot Joint torques applied to the robot
gj
Qo
qj
m
Manipulation contact mode between robot and the object Kinematic state of the robot
Fr
( TIB, PIB: 93)
EG go: Qo)
Kinematic state the object Kinematic state of the robot and the object
(Ir, Lo.
M {ma] 1==1
Link
Manipulation schedule (Sequence of manipulation contact modes)
X = [r 2in4
Pink
Kinematic state trajectory (Sequence of kinematic states)
T Tins
Duration (in s) of the trajectory for the task
Lobject == Lobject (m)
True iff at least one end-effector is either already in contact with or is establishing contact with the object
prechensite 1 prebensile (m)
True iff prehensile interaction is active
[0,
Phase signal that helps index the reference trajectory (X", MM)
Task phase rate (first order-dynamics model for Task phase rate based
60
reward-dependent functions Nominal phase signal computed using vd Residual phase rate (output from the policy)
Tie
6,
dt
Step-size (in s) of the environment Observation from the environment at time-step / Action applied to the environment at time-step Reward from the environment at time-step t
O:
0:
r: B Demonstrations from the Loco-Manipulation Planner
The mecommanipllttion planner from Sleiman et al. [10] efficiently generates physically con- sistent demonstrations for our proposed framework. The planner relies on a high-fidelity model that integrates the robot's full centroidal dynamics and first-order kinematics with the object's full dynamics [34]. This helps ensure that the discovered behaviors are dynami- cally feasible. The planner outputs the following the continuous states and system inputs: = (x, a.,) = (heom 9b qj qo qo) and u (we q)), where the robot state ær includes the centroidal momentum heom, base pose qb. and joint positions qi whereas the object state æ, con- sists of its generalized coordinates qo and velocities qo- The control input u is composed of the robot's joint velocities qj and the contact wrenches we acting at the robot's 's end-effectors.
12

---

Komprehennile Contocts
Prefeesile
Prefienlle Contact Nonprehersile Surface
Figure 7: Illustration the accommanipulation tasks: traversal of large articulated object, and ii) dishwasher manipulation. The user-defined robot end- and-effcctor contacts and object affordances are highlighted.
Moreover, from a set of user-defined object affordances and the robot's ' end-effectors for interac- tion, a discrete variable my represents the manipulation contact mode. contact mode is state- action pair, where the contact state encodes possible robot-object interaction combinations, and contact-swiccinng action indicates whether contact established, broken, maintained. By ploiting loco-man peccommanpulation-speeeee for multi-modal sequence via sampling-ban based bi-level search over manipulation modes m, and continuous state-input trajectories (atx (t), uu (4)). aiming to connect the start and goal states. It TO while fixing the discovered contact
pruning rules, the planning algorithm in [10] efficiently solves
then refines the resulting plan through a single sequence. We refer the reader to [10] for further details on the multi-contact planner.
While the references generated from the planner contain the control inputs (t), these signals usually tracked on hardware through whole-body quadratic programming (QP) controller. This controller computes the necessary joint torques to achieve the desired motions. However, the QP controller's robustness is limited because of its several assumptions, such as no slippages and pre- cise command tracking. We use only the reference states X and contact modes M to address these limitations and guide the RL training process. This approach allows the NN policy learn the underlying actuator dynamics during training and adapt better to the inherent uncertainties and variations encountered during real-world operations.
Table summarizes the single demonstrations generated for each task. In less than minute, all demonstrations are discovered on an Intel Core 17-10750H CPU@2.6GE hexacore processor. Nav- igating through spring-loaded pull door stands out as most complex task. This can be attributed to the long time horizon, the requirement for stable prehensile interaction, and the multiple con- tact transitions involved. Discovering these modes using standard RL would necessitate carefully designing handerafted rewards, which we want to alleviate through formulation.
Table 3: Computation time and trajectory duration for demonstrations generated using the planner [10].
Task
Computation Time (s) Trajectory Duration (s)
Trajectory Length
Door Push
6.8
16.8
1 1195
Dishwasher Open
25.0
11.2
814
Dishwasher Close
23.1
12.6
902
Door Pull
44.2
23.8
1725
C MDP Formulation and PPO Training
This section summarizes the terms that formulate the proposed MDP and the learning algorithm.
13

---

C.1 Observation Terms
Table 4 specifies the observation terms and the noise added to them during training. The critic ob- tains the same observations the actor but without any noise applied. Importantly, for the adaptive- phase dynamics formulation, the previous action d;-1 comprises both the robot commands d and the residual phase ó..
Table 4: Observation Terms Summary. We do not perform any scaling or clipping on individual observation terms. All noise models are additive in nature.
Term Name
Definition
Noise
Robot Base Position Difference
ITI
ITiB
u(-0.05, 0.05) u(-0.1,0.1)
Robot Base Orientation Difference
4 0
4 iB
Robot Base Linear Velocity
BVIB
u(-0.1.0.1)
Robot Base Angular Velocity
BWIB
u(-0.2,0.2)
Robot Joint Position Difference
qj 9}
u(-0.01,0.01)
Robot Joint Velocity
9:
u(-1.5,
Robot Arm End-effector Position
IT
4(-0005,0000
Object Joint Position Difference
T
0(00. 0.05)
Nominal Task Phase
Adaptive Task Phase
Ø
U ((-00005, 0.005)
Adaptive Task Phase Speed
8
u(-0.005, 0.005)
Previous Action
C.2 Reward Terms
To design task-agnostic reward function, we split the reward function into generic reference- tracking terms that stabilize the open-loop trajectories and standard penalty terms that ensure smooth lyand rregularize For the adaptive phase formulation, the reward also includes the task progress: total = -total + r The individual terms are:
motions: plotal prack
track
= wi ||ITIB 1rri 2| W2 1 E B + uz 1|q; 97 12
+ W4 1 abject l1q. 1 + W/s prrchensite 1|TTEE sr1n11
p Ki [00 epppeexp(-
x]]]]
regularize = 31 ||| 32 || 2 B3
|0y 2 B1 11Buiill Bs BBaww
+ B6 la, - 1..
where symbols have their meanings from Table The corresponding weights are Table
C.3 Initial-State Distribution
We apply additive offsets to the robot's reference configuration at dinit = so that the policy is robust to varying initial locations of the robot in front of the door. Ideally, we would like to apply this at any randomly sampled Φ; doing so non-trivial due to the difficulty filtering invalid collision configurations.
C.4 Domain Randomization
Domain randomization helps mitigate overfitting to specific models and addresses inherent unmod- elled effects by introducing variability during training. In our setup, takes the following form:
Object's kinematics: These include object dimensions (e.g. door width and height), posi- tioning of object affordances (e.g. handle location the panel), and handle types (cylinder box). For every object category, we load 128 different kinematic variations.
Object's dynamies: These include friction and restitution, spring-damping coefficients for the hinge joint, and constant force/torque offset the hinge joint.
14

---

Table 5: Reward Terms Summary. The environment scales the reward weights with the time-step de [35]. For brevity, we drop the time-step & from individual quantities unless necessary. We use the same reward weights for all the loco- moc-manipulation tasks.
Term Name
Definition
Weight
Robot Base Position Tracking
ITIB
1T
-0.2
Robot Base Orientation Tracking
E
+
-0.2
1B
Robot Joint Position Tracking
||
qj
1
-0.2
Object Joint Position Tracking
object
90 L
-0.2
Robot Arm End-effector Position Tracking
prehemsile
|1TE
1I1H
-10.0
Action Rate
a,
0.05
Robot Base Linear Velocity (along 2)
|
-0.5
Robot Base Angular Velocity (along zy)
[8 18
-0.05
Robot Joint Velocity
9
110
10
Robot Joint Acceleration
-1.0 10
Robot Applied Joint Torque
|T
-2.
10
Task Progress (only with adaptive phase)
d exp( expp(11..0 10.0 ||
25.0
Similar to object dynamics, vary the friction and restitution within (0.4, 1.0). Additionally, the mass of the robot's base is randomized within = 10% of its nominal values.
Robot's dynamics:
External disturbances: At randomly sampled episode intervals, external pushes ap- plied to both the robot and the object. For the robot, this implies adding random velocity (pushes) to the robot's base. For the door, this is done by applying randomly sampled external force on the door panel.
C,5 Termination Term
We trigger episode termination when the robotic system loses balance or episode length times out. Typically, we infer fall from significant force acting the robot's base, indicating ground tact. However, distinguishing the source of this force becomes challenging during oocommanipulation tasks, as contact between the robot's base and an object is both expected and sometimes permissible. Thus, we rely on the base to not drop below 0.3 m to correctly detect falls.
C.6 Adaptive-Phase Hyperparameters
This section lists the hyperparameters for the remainder of the MDP formulation. These include parameters for scaling the input actions and those for the adaptive-phase formulation in Eq. 1.
Table 6: MDP Hyperparameters.
Hyperparameter
Value
Episode length
158
Simulation time-step
0.005
Control decimation
4
Residual phase action scale:
0.01
Robot action scale: 02
0.5
Adaptive phase (1):
-0.2
Adaptive phase (I): w2
-0.2
Adaptive phase (1): W4
-0.2
Adaptive phase (1): À
-50.0
Adaptive phase (1): A:
10.0
C.7 Learning Algorithm
For each task, train the policy using the on-policy RL algorithm, Proximal Policy Optimization (PPO) [32]. The actor and critic networks are designed as Multi-Layer Perceptron (MLP) with
15

---

[256 x 128 x 64] hidden-layer structure and an ELU activation function. A complete list of hyperparameters and their values is specified in Table
Table PPO Hyperparameters
Hyperparameter
Value
Empirical Normalization
True
Learning Rate (start of training)
le-3
Learning Rate Schedule
"adaptive" (based on KI KI--diergeece [35])
Discount Factor
0.99
GAE Discount Factor
0.95
Desired KL-divergence
0.01
Clip Range
0.2
Entropy Coefficient
0.0
Value Function Loss Coefficient
1.0
Batch Size
245,760 (4096 60)
Mini-Batch Size
61,440 (4096 15)
Number of Epochs
5
Number of iterations
10,000
D Supplementary Discussions
D.I Object Locking Mechanism
Our policy is trained on doors where handles are treated fixed object-attached links. However, typical door can only be opened after being unlocked using its handle. By adapting our environment to incorporate door-locking mechanism, pushing the handle downwards, but with low success rates. A possible way to resolve this issue is using an asymmetric actor-critic structure, where the critic obtains the handle angle.
current setup results in behaviors involving the robot
D.2 Unknown Object Type
In real-world scenarios, expect the robot to autonomously traverse diverse doors without speci- fying the type of door (i.e., push or pull door). One way to achieve this objective would be to first train a multi-task policy that can execute both door-traversal behaviors and then separately train door-type estimator that outputs the appropriate task command to the policy.
D.3 Unknown Object State
In the current hardware experiments, we rely explicit sensors such as door encoders to obtain the joint position of the panel. However, in more realistic scenarios, this information needs to be extracted from the robot's onboard sensors. One mechanism to achieve this goal by leveraging teacher-student training and treating the current policy as the teacher policy [5].
D.4 Applicability to Other Scenarios
In this work, we demonstrated the approach to multi-contact tasks that primarily involved articulated object manipulation. However, there an open question on the generalization of the approach to other tasks and robots. For tasks where single demonstration sufficient solve the task, we be- lieve our method should work in these scenarios. However, for tasks where complete re-planning necessary (for instance, the rearrangement of objects), training online replanner becomes essary provide new references for the policy to track. We leave this part of our future work.
16


Reviews:
[{"review_id": "v2RTo3nWE8", "reviewer": "Reviewer_9jLh", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "Robust loco-manipulation achieved via sim2real RL, guided by a single demonstration. The key development is a state-dependent adaptive phase variable that captures task progress, which enables guidance from a demonstration while allowing adaptability in the presence of model mismatch and external disturbances. The authors demonstrate its applicability in the context of simulated and real door opening tasks.", "summary_of_recommendation": "I recommend this paper for acceptance because the adaptive phase formulation is interesting and effective for solving a suite of door opening tasks robustly, and because the questions of robustness and reduction of human effort in policy generation are topical in the robotics/machine learning research community right now. However, I suggest that the authors restructure the methods section to increase clarity and weaken their claims on the generality of the method (unless they demonstrate its applicability beyond door opening).", "technical_quality": 3, "clarity_of_presentation": 3, "potential_impact": 2, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "1. Please provide an intuitive explanation of o1, o2, and o3 in the Observations section, as well as ensuring that all of the terms are defined (for example, the starred terms are not defined, q_0 and q_j are not defined, and the E frame is not defined).\n2. In the actions section, please justify why sigma_1 is necessary.\n3. I find the setup of Nominal Phase-Based Formulation in 3.1.1 and Adaptive Phase-Based Formulation in 3.1.2 to be unnecessarily confusing. Instead, I recommend presenting the full adaptive phase based-formulation, and simply highlighting (where appropriate) what the formulation would reduce to in the nominal phased-based case (like you do in lines 140-142 when describing the relationship between your adaptive phase rate and the nominal phase rate).\n4. \"The resulting dynamics induce a guiding effect during training similar to that achieved in reward-driven curriculum learning\" (line 151-152), please provide some empirical or mathematical justification for this claim.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DpR7Joz8N0", "reviewer": "Reviewer_kZSQ", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 2, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposes a method to enable a quadruped loco-manipulator to navigate doors and open/close dishwashers. The approach first uses a planner to generate a trajectory of high-level waypoints to accomplish the task. These waypoints are sent to a controller, which keeps track of progress towards the goal along the trajectory, and sends intermediate subgoals to the low-level policy. The policy takes in a target state and manipulation mode as input, and is trained using PPO. The approach is demonstrated in both simulation and on a real robot.", "summary_of_recommendation": "This work is promising but needs improvements in presentation, comparison to another method, and real robot implementation", "technical_quality": 2, "clarity_of_presentation": 3, "potential_impact": 2, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "Presentation\n- The message of the paper can be refined, perhaps not claiming a general method if it cannot be backed up\n- Some of the terminology in the paper is not widely used in RL and yet are not introduced or defined anywhere. Here are a few examples: references, schedule, phase.\n\nEvaluation:\n- There are ablations performed on the method itself but there does not seem to be a comparison to a widely-used existing approach as a baseline\n\nReal robot:\n- The robot should complete the task or at least the authors should explain why it is only able perform half of the task\n- If the long pauses are due to code implementation, that should be addressed\n\nMinor issues:\n- Line 20: \"has shown demonstrated\" -> \"has demonstrated\"\n- Line 53: \"behaviors.This\" -> \"behaviors. This\"", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TVtehuDuD5", "reviewer": "Reviewer_1Sf9", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper proposes a generic MDP framework that includes a task phase variable to train loco-manipulation RL policies, guided by trajectory optimization.", "summary_of_recommendation": "I recommend the acceptance of this paper. The proposed MDP framework for guided RL and adaptive phase dynamics offer a promising approach to handling complex, contact-rich tasks. However, including comparisons with baseline methods and providing clearer intuitions for the unique design components would strengthen the paper further.", "technical_quality": 3, "clarity_of_presentation": 3, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "1. Could you explain the physical meaning and effect of the residual action $\\delta_v$ more clearly?\n2. Could you give an overview of the proposed formulations and what are their advantages over existing ones?\n3. Although the NNS results in 0% success rate, it could be informative to visualize their trajectories as well. Are they stuck at the onset of unforeseen slippage and large disturbances?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: Update:
The authors have addressed reviewers' concerns during the rebuttal. There is an interesting idea to this paper that would benefit the CoRL community, and I found the new videos quite convincing.

====================================


Overall, the paper proposes an interesting method (adaptive phase variable) that can capture task progress for door-opening tasks, and its experimental results in simulation and real world are convincing.

However, the paper needs to tone down the argument about the generality of the proposed method -- as pointed out by several reviewers, it seems that there are several tasks where this approach is difficult to apply (see ones suggested by 9jLh). The paper needs to clearly articulate the class of problems that the method *can* solve, instead of claiming that the proposed method is task-agnostic. Lastly, I too agree with the reviewers that the method section is confusing to understand.
Decision: Accept