Guided Reinforcement Learning for Robust Multi-Contact Loco-Manipulation

Authors: jean-pierre  sleiman
mayank  mittal
marco  hutter
Venue: robot-learning_org_CoRL_2024_Conference
Published: 2024-06-07 13:29:56

Abstract: Reinforcement learning (RL) has shown remarkable proficiency in developing robust control policies for contact-rich applications. However, it typically requires meticulous Markov Decision Process (MDP) designing tailored to each task and robotic platform. This work addresses this challenge by creating a systematic approach to behavior synthesis and control for multi-contact loco-manipulation.
We define a task-independent MDP formulation to learn robust RL policies using a single demonstration (per task) generated from a fast model-based trajectory optimization method. Our framework is validated on diverse real-world tasks, such as navigating spring-loaded doors and manipulating heavy dishwashers. The learned behaviors can handle dynamic uncertainties and external disturbances, showcasing recovery maneuvers, such as re-grasping objects during execution. Finally, we successfully transfer the policies to a real robot, demonstrating the approach's practical viability.

Main paper: 
 introduction: 
 Model-based optimal control techniques, such as trajectory optimization (TO) and model-predictive control (MPC), are valued for their inherent versatility. They can generate a range of easily interpretable behaviors, facilitating intuitive adjustments to the problem formulation [1, 2, 3] . However, these methods are sensitive to modeling mismatches and violation of assumptions. Conversely, reinforcement learning (RL) has demonstrated remarkable success in developing robust control policies for various contact-rich tasks, such as legged locomotion [4, 5] , loco-manipulation [6, 7] , and dexterous manipulation [8, 9] . Despite this, RL suffers from high sample inefficiency, the emergence of unnatural behaviors, and the need for labor-intensive reward design. In this work, we aim to leverage the complementary strengths of TO and RL to mitigate their weaknesses.We build upon the versatile TO-based framework from Sleiman et al. [10] , which demonstrates the discovery of long-horizon multi-modal behaviors for poly-articulated systems. This framework can safely and reliably solve complex loco-manipulation tasks, accounting for the multiple interaction regions in the object (handle, surfaces) and the robot (individual limbs). These behaviors are challenging to achieve with pure RL, even with an extensive task-specific design process. However, the successful execution of TO-based behaviors depends heavily on reasonably accurate environment models and minimal external disturbances. This reliance is due to the underlying MPC-based controller's strict adherence to tracking open-loop references without the ability to replan or respond to significant deviations, such as slippages. This limitation of the MPC controller motivates our work: using RL to obtain policies for the robust execution of contact-rich interaction plans.In this paper, we propose a Markov Decision Process (MDP) to efficiently train robust locomanipulation policies, particularly for articulated object interaction, with task-agnostic rewards and hyperparameter tuning. Our approach uses dynamically feasible demonstrations generated from a TO-based framework [10] to guide the RL agent in learning complex behaviors. By maintaining consistent MDP parameters and utilizing only one demonstration per task, we present an approach to train control policies that track the generated demonstrations while handling modeling uncertainties, external disturbances, and unforeseen events such as handle slippage. We benchmark our approach against prior motion imitation works on four loco-manipulation tasks: door pushing and pulling and dishwasher opening and closing. Finally, we deploy the trained policies on a quadrupedal mobile manipulator, showcasing their robustness to unknown object models and reactivity to slippages. 
related work: 
 Imitation learning has been successfully applied to various manipulation tasks through its algorithmic variants such as behavioral cloning (BC) [11] , demo-augmented policy gradient [12, 13] (which bootstraps the RL policy through BC), and more recently, generative models [14, 15] . However, these approaches often struggle with distributional shifts and require substantial training data, posing challenges for high-dimensional robotic systems.An alternate class of methods relies on state-only references to guide an RL agent toward desired behaviors. This approach commonly involves conditioning a low-level policy on references obtained either offline from a motion library or online by a separate high-level module. For instance, Bergamin et al. [16] incorporates a motion matcher that selects and blends the best-fitting motion clip from an extensive MoCap database based on handcrafted features. The works from Kang et al. [17] and Jenelten et al. [18] update the policy with on-demand "optimal" references that are computed in an MPC-like fashion. While MPC can provide precise guidance, its high computational demands can significantly slow down the RL training process.Another common approach is inverse RL, where a reward function is defined through the demonstrations. Adversarial motion priors (AMP) [19, 20, 21] formulate a style reward that is maximized when the learned policy yields state transitions similar to those in the dataset. Subsequent variants [22, 23, 24] propose a hierarchical architecture to address the mode-collapse issue in adversarial learning setups. However, these methods still require task-specific goal-directing rewards and a large amount of data to learn the style reward. In contrast to AMP-style approaches, motionimitation methods guide RL policies to robustly track target trajectories directly [25, 26, 27, 28] . They use reward terms that encourage adherence to the demonstrations while allowing exploration around the reference motions.Our work falls into the category of motion-imitation RL without any task-specific objectives. It closely relates to the work from Fuchioka et al. [26] and Bogdanovic et al. [27] , as we aim to directly track and robustly stabilize offline trajectories generated using a fast trajectory optimizer. Differently from these works, we introduce domain-specific considerations relevant to multi-contact loco-manipulation tasks. Moreover, we propose a task-independent MDP formulation based on an  
approach: 
 Our proposed approach consists of two steps, as illustrated in Fig. 2 . First, we employ the planner from Sleiman et al. [10] to generate whole-body multi-contact behaviors for various locomanipulation tasks. Subsequently, we train a neural network using RL to reliably track these behaviors while leveraging only one pre-computed trajectory per task as an "expert demonstration". We train this policy entirely in simulation with domain randomization to achieve a successful transfer to the real robot. At deployment, we assume access to the object's state. While this assumption limits the applicability in the wild, our primary focus is devising an MDP for learning robust behaviors across different loco-manipulation scenarios without requiring task-specific handcrafting.We generate a single demonstration for each task using the multi-contact planner (MCP) from [10] . This planner takes the task specification and a set of user-defined object affordances (e.g. handle or surfaces) and the robot's end-effectors (e.g. the tool on the arm or the feet) for interaction as inputs. It then searches for possible robot-object interactions to provide a physically consistent demonstration based on a nominal robot and object model.The demonstrations from the planner consist of the continuous robot and object state references X * = {x * t } Ttask t=1 and the manipulation schedule M * = {m * t } Ttask t=1 with T task being the demonstration's duration. In the case of articulated object interaction, the state reference x * t includes the robot's base pose ( I r * IB , Φ * IB ) in a fixed inertial frame I, the robot's joint positions q * j , and the object's joint positions q * o . The contact mode m * t specifies the interaction type for each end-effector (none, prehensile, or non-prehensile) and the object contacts involved at that timestep. Since the demonstrations for each task have varying lengths, we encode the notion of time into the task-phase φ ∈ [0, 1], where φ = 0 and φ = 1 imply the start and end of the demonstration, respectively [25] .The objective of the learning agent is to track these references while staying robust against variations in the kinematic and dynamic properties of the object, external disturbances, and unforeseen events such as slippages. In the naïvest way, one could increase the task phase linearly with time and track the corresponding references [25, 26] . However, as we show later in Sec. 5.1, this approach limits the success rate during object manipulation. For instance, if the door handle slips away and the door closes, strictly following the references will cause the robot to get stuck behind the door. Instead, we want to give the robot the time to recover and, thus, grow the task phase adaptively.Linearly evolving the task phase with time (also referred to in this paper as the "nominal" formulation) can be written as: φt+1 = φt + 1 Ttask • dt, where φ denotes the nominal phase and dt is the environment time-step. As motivated earlier, we seek an adaptive mechanism that adjusts the phase φ depending on the current robot and object states. To this end, we propose the phase dy-namics:φ t+1 = φ t + v φ t T task • dt, where the task phase rate v φ t = vφ t + σ 1 • δ v,tis determined by a state-dependent reference vφ := vφ (x, x * , m * ) and a learnable residual term δ v (scaled by σ 1 > 0). We opt for a reward-dependent vφ that reflects the task-level tracking accuracy:EQUATIONwhere k > 1 and λ < 0 are fixed constants, and L is the number of task-level rewards r i,t , with r i,t ≤ 0 at all time-steps. For articulated object interaction, L = 3 with terms corresponding to the tracking rewards for the robot's base pose and the object's joint positions, as specified in Sec. 3.3.The term vφ effectively pauses the phase evolution for large deviations from the current reference. As the tracking improves, it gradually approaches the nominal phase rate (→ 1). Since adhering strictly to the reference may not always be the best strategy to complete the task, we introduce a clipping in Eq. 1 to encode perfect tracking only within a certain margin defined by k and λ. This operation is particularly useful when handling discrepancies between the nominal model used by the planner and the real model employed during training and deployment.The resulting task phase dynamics with vφ induces a curriculum-like effect during training. At the start of the training, when the tracking is poor, it helps the agent learn to recover to a single reference state. Eventually, as the agent improves, it learns to follow more of the demonstration.Intuition behind δ v : In some instances, unforeseen slippage or large disturbances could render the object uncontrollable due to a complete loss of contact, resulting in significant deviations from the reference pose. In these situations, the term vφ → 0, and the robot cannot recover. To enable motion recovery in such scenarios, we introduce a residual phase δ v that allows potentially speeding up, slowing down, and even decreasing the phase whenever necessary. This residual phase is outputted from the policy, allowing it to adapt to the task phase dynamics via learnable parameters.To design task-agnostic reward functions such that loco-manipulation behaviors in the generated demonstrations can be learned through the same MDP formulation, we split the rewards into three parts: reference-tracking rewards, task progress rewards, and penalties for smooth motions.Reference-tracking reward: These are simply defined through the tracking errors between the current state x and the reference state x * from the demonstration:EQUATION+ w 4 • ✶ * object • ||q o -q * o || 2 + w 5 • ✶ * prehensile • || I r IE -I r IH || 2, where w i < 0, H is a non-fixed handle frame that is initialized at I and moves with the object, and✶ * ••• := ✶ ••• (m * )denotes indicator functions that depend on the reference manipulation mode m * .The indicator functions help avoid incurring irrelevant penalties that do not coincide with the behavior's mode schedule. The function ✶ * object is true only when at least one end-effector is either already in contact or is establishing contact with the object. Essentially, it activates the object tracking error only when the object's state is controllable and essential for the task completion. Meanwhile, ✶ * prehensile is true when a prehensile interaction is active. This choice helps improve the accuracy of prehensile contact and is particularly useful when we randomize the object models during training, as the location of the object frame H varies.Task progress reward: This reward term encourages the learning agent to progress in the task. It is defined as:r φ = κ 1 • vφ • exp(-κ 2 • ||φ -φ|| 2 ), where κ 1 , κ 2 > 0, and vφ and φ are the reference task-phase rate and the nominal phase from Sec. 3.2. When the tracking is poor, vφ → 0 and the agent gets less reward for the task progress, i.e., r φ → 0. Consequently, the agent gets encouraged to maximize the reference tracking reward and minimize the tracking deviation.Penalties: These terms penalize the robot's joint accelerations and torques, base velocity in undesired directions, and abrupt action changes. Since these are standard penalties used in RL, we skip detailing them here for space reasons and refer the reader to Appendix C.2 for further details.Observations: The observations comprise the tracking errors in the robot and object states, the positions and velocities of all end-effector frames participating in prehensile interactions, the previous action, and the task-phase parameters. For a legged mobile manipulator with only one prehensile end-effector E on the arm, the observations can be denoted as o t = (o e o v o p a t-1 o φ ), with:EQUATIONThe choice of the observation terms o e and o v resemble that of a classical PD control law, where o e captures the position tracking error and o v is the velocity tracking error with zero velocity targets. Effectively, the learned policy is a more complex tracking controller around the demonstrations. Additionally, while including o p is not essential, we notice that having it improves the training as it eliminates the need to learn the mapping from the robot's state to operational space quantities.Actions: The actions a t = (a qj ,t , δ v,t ) are interpreted as the residuals over the robot's reference joint positions q * j,t and the reference phase rate vφ t from Sec. 3.2. The robot's actions are sent to its actuators as joint position commands: q cmd j,t = q * j,t + σ 2 • a qj,t , with σ 2 > 0. While the planner [10] also provides reference joint velocities, we observed that including them in feed-forward control resulted in poorer performance. This finding aligns with that from Fuchioka et al. [26] .Initial Distribution: At environment resets, we apply the reference state initialization (RSI) strategy from imitation-based RL setups [25, 26] . In RSI, we randomly sample the initial phase φ init ∈ [0, 1] and spawn the robot and object at their corresponding reference states. However, differently from RSI, we randomize the robot configuration uniformly around the reference state when φ init = 0. This alleviates the need to always start the robot at the initial reference state at deployment and lets the agent learn the necessary recovery actions to stay near the demonstration.The learning task is then to complete the remainder of the task from these varying initial states. This approach exposes the policy to regions of the state space that are relevant to the reference behavior and are hard to reach independently. Additionally, it removes the need for the episode length to be at least as long as the demonstration T task . Domain Randomization (DR): Unlike [10] , where the MPC-based tracking controller's success relies on a reasonable knowledge of the environment, we aim to make the learned policy robust to modeling uncertainties and disturbances through DR. This involves varying the properties of the robot (such as friction and base mass), the kinematic and dynamic models of the object (such as the door dimensions and spring loading parameters), and adding regular external disturbances to the system. For further details, please refer to Appendix C.4Curriculum: We introduce a curriculum that incrementally increases the external disturbances, observation noise, reward penalties, and initialization offsets. The curriculum level l rand ∈ [0, 1] is updated according to the following rule:EQUATIONThe term p φ represents the progress of the task phase relative to the nominal one. As the agent becomes proficient in tracking the demonstrations, it progresses to a higher difficulty level with larger DR and penalties. Conversely, if its performance is poor, it moves to a lower level. 
experimental setup: 
 We validate our framework on a legged mobile manipulator consisting of the quadruped ANYmal-D [29] with a 6-DoF robotic arm [30] . We consider four tasks: Door Push/Pull: traversing a springloaded push or pull door, and Dishwasher Open/Close: opening or closing a heavy dishwasher. Demonstration generation: Fig. 3 illustrates the manipulation schedules M * of the generated demonstrations using [10] . Simpler tasks like door pushing are solvable with a single contact mode, while more complex tasks, such as door pulling, require sequencing multiple robot-object interaction modes. Navigating through a spring-loaded pull door stands out as the most complex task, both in terms of behavior discovery and execution time. For additional details, please check Appendix B.We simulate the training environment in NVIDIA Isaac Gym [31] . The simulator runs at 200 Hz, and we decimate the policy to run at 50 Hz. All policies are trained using the Proximal Policy Optimization (PPO) [32] algorithm. Training converges within 6 hours per task on a single NVIDIA RTX 4090Ti. Notably, we apply the same MDP formulation across all tasks with identical environment and agent hyperparameters. These are specified in Appendix C.Baselines: Behavior cloning (BC) requires labeled actions and a large training dataset. Given that we operate with a few demonstrations that also lack action labels, we do not compare our method against BC. Instead, we evaluate our method against other motion-imitation RL formulations:• Nominal phase: This formulation is based on existing works in motion-imitation RL [25, 26] . It uses the task phase dynamics with v φ = 1 and no learned residual phase rate.• Nearest Neighbor Search (NNS): An alternative phase update mechanism inspired by the time-independent training stage of [27] . It aims to minimize the distance between the current system state and the closest reference in the demonstration dataset.5 Results and Analysis 
simulation experiments: 
 Comparisons: We evaluate each learned policy in 4096 environments with randomized robot and object properties and random pushes. We report the success rates across the four tasks in Table 1 .We observe that our proposed method performs significantly better than other motion-imitation RL setups, even without including the residual phase action δ v . This result highlights the significance of integrating the reward-driven adaptive phase dynamics. The nominal phase update can reliably accomplish the relatively straightforward task of navigating through push doors. However, it struggles with more complex tasks, achieving notably lower success rates for door pulling. Meanwhile, the NNS strategy fails across all the tasks. While experimenting with various training setups to incorporate NNS, we consistently found the phase stuck at certain points along the reference path. We hypothesize this occurs because the policy exploits the NNS update mechanism to achieve a state where the phase remains unchanged, thereby maximizing the reference tracking reward.Ablations: To evaluate the need for demonstrations as whole-body trajectories, we omit the robot's joint-level references, relying on partial task-level references that pertain solely to the floating base and the object trajectories. As shown in Table 1 , only the door pushing remains solvable under this scheme. The reliance on the robot's foot contacts for manipulation necessitates whole-body guidance to solve the other tasks. Moreover, we see that the curriculum from Sec. 3.5 helps improve the agent's performance across most tasks. During training, we observe that the curriculum also enables faster convergence and smoother policies. Adding the residual phase rate only improves the performance slightly in these comparisons. However, as shown in Fig. 4 for the door-pulling task, relying solely on the reference vφ for the phase dynamics brings the task progression to a halt during an inadvertent loss of contact. In contrast, the learned residual action decreases the phase, allowing the robot to re-establish contact with the handle and successfully complete the task.Robustness Evaluation: While the generated loco-manipulation plan is done for nominal parameters of the object, we train the RL policies with various randomizations, as mentioned in Sec. 3.5.To assess the robustness of the learned policies, we investigate their performance on different door parameters for the door-pulling task. We remove all DRs during these evaluations to ensure a controlled setting. For comparison, we train an RL policy with no DR as an alternative to the MPCbased controller in [10] . Based on Fig. 5 , the policy trained only with nominal door properties fails quickly when the variations become too large. In contrast, the policy trained with random doors supports a wider range of door parameters. When the handle offset is significant, the policy trained with random doors displays a "search" behavior to find and grasp the handle. Meanwhile, the policy with a nominal door only works reliably well around the handle length. This analysis demonstrates that although the generated demonstrations are not for the simulated door models, the RL policy discovers behaviors around them that help it generalize to a large set of doors. Multi-task Learning from Demonstrations We experiment with a multi-task training setup where a single policy jointly learns all four tasks. It takes a one-hot vector to indicate the desired demonstration to execute. We train the policy with the same hyperparameters and number of environments from before and obtain a reduced average success rate (84% instead of 97%). By simply doubling the number of environments during training (i.e., collecting twice the number of samples), we observe that it achieves the performance of individual task-specific policies.We demonstrate the proposed approach on all four tasks on hardware. A motion capture system provides the handle location at the start and the joint angle for the door and the dishwasher. For the robot, we rely on its onboard state estimation. For each task, we perform six continuous runs, where we manually reset the object, place the robot randomly in front of the object, and execute the policy. As shown in Fig. 6 , the deployed policies exhibit smooth motions and preserve the multi-contact behavior from the demonstration. They complete all the tasks six times in a row. Additionally, the policies yield different motions every time, which is particularly noticeable during traversing a pull door. They are able to handle random base placement, recover from slippages, and re-grasp the handle when it misses it. Please refer to the supplementary video to observe these behaviors.On hardware deployment, we notice that if the robot starts too far away or too close to the object, it aggressively tries to adjust its configuration. This, at times, led to the robot falling down. However, in practice, a navigation planner can appropriately place the robot in front of the object, so only local adjustments are needed, which the learned policies can handle. Additionally, while the policies can handle large intra-object category variations, they may still fail on certain object instances. For instance, if the door is too small, the policy fails to traverse through it as it collides with the door. In such cases, we need to use more demonstrations for the training. Lastly, a natural extension of this work is to devise perceptive loco-manipulation policies that rely on onboard sensing [5, 7] . 
conclusion: 
 Our work integrates model-based TO and model-free RL to develop robust tracking policies for multi-contact loco-manipulation tasks. Central to our approach is a task-agnostic MDP formulation that utilizes loco-manipulation demonstrations generated by a precise trajectory planner. Unlike previous imitation-based RL methods that depend on time-based nominal phase updates, we showed that our state-dependent adaptive phase dynamics facilitate successful task execution despite modeling inaccuracies and significant external disturbances. We validated our framework on a quadrupedal mobile manipulator performing four complex long-horizon tasks, such as navigating spring-loaded doors and manipulating heavy dishwashers. Additionally, we demonstrated that the learned policies transfer to hardware successfully and can effectively recover from slippages and missed handles, overcoming the limitations of the MPC-based tracking controller from [10] . Moreover, from a set of user-defined object affordances and the robot's end-effectors for interaction, a discrete variable m k represents the manipulation contact mode. A contact mode is a stateaction pair, where the contact state encodes possible robot-object interaction combinations, and a contact-switching action indicates whether a contact is established, broken, or maintained. By exploiting loco-manipulation-specific pruning rules, the planning algorithm in [10] efficiently solves for a multi-modal sequence via a sampling-based bi-level search over manipulation modes m k and continuous state-input trajectories ⟨x k (t), u k (t)⟩, aiming to connect the start and goal states. It then refines the resulting plan through a single long-horizon TO while fixing the discovered contact sequence. We refer the reader to [10] for further details on the multi-contact planner.While the references generated from the planner contain the control inputs u k (t), these signals are usually tracked on hardware through a whole-body quadratic programming (QP) controller. This controller computes the necessary joint torques to achieve the desired motions. However, the QP controller's robustness is limited because of its several assumptions, such as no slippages and precise command tracking. We use only the reference states X * and contact modes M * to address these limitations and guide the RL training process. This approach allows the NN policy to learn the underlying actuator dynamics during training and adapt better to the inherent uncertainties and variations encountered during real-world operations.Table 3 summarizes the single demonstrations generated for each task. In less than a minute, all demonstrations are discovered on an Intel Core i7-10750H CPU@2.6GHz hexacore processor. Navigating through a spring-loaded pull door stands out as the most complex task. This can be attributed to the long time horizon, the requirement for a stable prehensile interaction, and the multiple contact transitions involved. Discovering these modes using standard RL would necessitate carefully designing handcrafted rewards, which we want to alleviate through our formulation.  
learned reacitivity and robustness: 
 Task-agnostic rewards specification Guided exploration using demonstration Sim-to-real transfer Large-scale simulation-based RL Single demonstration Figure 1 : A framework for learning loco-manipulation tasks, such as traversing a spring-loaded door and manipulating dishwashers. A single demonstration guides the RL training process to learn multi-contact behaviors (such as using the feet or the arm for interaction) without task-specific handcrafted rewards. 
 
 Appendix: This section summarizes the terms that formulate the proposed MDP and the learning algorithm. • External disturbances: At randomly sampled episode intervals, external pushes are applied to both the robot and the object. For the robot, this implies adding random velocity (pushes) to the robot's base. For the door, this is done by applying a randomly sampled external force on the door panel.We trigger episode termination when the robotic system loses balance or episode length times out. Typically, we infer a fall from a significant force acting on the robot's base, indicating ground contact. However, distinguishing the source of this force becomes challenging during loco-manipulation tasks, as contact between the robot's base and an object is both expected and sometimes permissible. Thus, we rely on the base to not drop below 0.3 m to correctly detect falls.This section lists the hyperparameters for the remainder of the MDP formulation. These include parameters for scaling the input actions and those for the adaptive-phase formulation in Eq. 1. For each task, we train the policy using the on-policy RL algorithm, Proximal Policy Optimization (PPO) [32] . The actor and critic networks are designed as a Multi-Layer Perceptron (MLP) with a [256 × 128 × 64] hidden-layer structure and an ELU activation function. A complete list of hyperparameters and their values is specified in Table 7 . Our policy is trained on doors where handles are treated as fixed object-attached links. However, a typical door can only be opened after being unlocked using its handle. By adapting our environment to incorporate a door-locking mechanism, our current setup results in behaviors involving the robot pushing the handle downwards, but with low success rates. A possible way to resolve this issue is using an asymmetric actor-critic structure, where the critic obtains the handle angle.In real-world scenarios, we expect the robot to autonomously traverse diverse doors without specifying the type of door (i.e., push or pull door). One way to achieve this objective would be to first train a multi-task policy that can execute both door-traversal behaviors and then separately train a door-type estimator that outputs the appropriate task command to the policy.In the current hardware experiments, we rely on explicit sensors such as door encoders to obtain the joint position of the panel. However, in more realistic scenarios, this information needs to be extracted from the robot's onboard sensors. One mechanism to achieve this goal is by leveraging teacher-student training and treating the current policy as the teacher policy [5] .In this work, we demonstrated the approach to multi-contact tasks that primarily involved articulated object manipulation. However, there is an open question on the generalization of the approach to other tasks and robots. For tasks where a single demonstration is sufficient to solve the task, we believe our method should work in these scenarios. However, for tasks where a complete re-planning is necessary (for instance, the rearrangement of objects), training an online replanner becomes necessary to provide new references for the policy to track. We leave this as part of our future work. 


Decision: Update:
The authors have addressed reviewers' concerns during the rebuttal. There is an interesting idea to this paper that would benefit the CoRL community, and I found the new videos quite convincing.

====================================


Overall, the paper proposes an interesting method (adaptive phase variable) that can capture task progress for door-opening tasks, and its experimental results in simulation and real world are convincing.

However, the paper needs to tone down the argument about the generality of the proposed method -- as pointed out by several reviewers, it seems that there are several tasks where this approach is difficult to apply (see ones suggested by 9jLh). The paper needs to clearly articulate the class of problems that the method *can* solve, instead of claiming that the proposed method is task-agnostic. Lastly, I too agree with the reviewers that the method section is confusing to understand.

Reviews: review: title: title: New formulation for updating phase enables more robust loco-manipulation for door opening
paper_summary: paper_summary: Robust loco-manipulation achieved via sim2real RL, guided by a single demonstration. The key development is a state-dependent adaptive phase variable that captures task progress, which enables guidance from a demonstration while allowing adaptability in the presence of model mismatch and external disturbances. The authors demonstrate its applicability in the context of simulated and real door opening tasks.
main_review: main_review: This work claims to introduce a task-independent MDP formulation, based on an adaptive phase dynamics model, that allows the RL agent to learn robust policies guided by a single demonstration. 

It has several strengths:
1. The adaptive phase dynamics formulation is an interesting idea with demonstrated effectiveness in robustly solving door opening tasks.
2. This contribution reduces the burden of manual tuning in sim2real RL, in that it reuses the same MDP formulation and algorithm 
hyperparameters for solving a suite of tasks, which is an important goal in machine learning research for robotics.
3. The task of opening a spring-loaded door is sufficiently hard, and the real-world behavior demonstrated in the supplemental video looks convincingly robust, to motivate the author's proposed approach.

It also has several weaknesses:
1. The organization of the methods section is confusing, in that the equations that describe the formulation the authors actually use are broken up into two sections. I suggest consolidating sections 3.1.1 and 3.1.2 so that the equations that describe the method are in one place, and noting where appropriate what the formulation reduces to in the nominal phase case.
2. The authors claim that the MDP formulation is task-independent, but only demonstrate its application to a suite of door opening tasks. One thing I notice about this class of tasks is that progress in door opening tasks can be well-captured by a phase variable that can progress forward or backward (ex. if you fail to keep a spring loaded door open, it will shut, and look similar to the door at phase = 0). How would this formulation perform if task progress is less obviously captured by a phase variable, and failure can easily bring the robot/object states out of distribution of the initial demonstration (ex. if the task is stacking blocks, and at one point the stack topples and the blocks are now in a completely different arrangement)?
3. The work would be more impactful if demonstrated on real observations rather than ground-truth state.
questions: questions: 1. Please provide an intuitive explanation of o1, o2, and o3 in the Observations section, as well as ensuring that all of the terms are defined (for example, the starred terms are not defined, q_0 and q_j are not defined, and the E frame is not defined).
2. In the actions section, please justify why sigma_1 is necessary.
3. I find the setup of Nominal Phase-Based Formulation in 3.1.1 and Adaptive Phase-Based Formulation in 3.1.2 to be unnecessarily confusing. Instead, I recommend presenting the full adaptive phase based-formulation, and simply highlighting (where appropriate) what the formulation would reduce to in the nominal phased-based case (like you do in lines 140-142 when describing the relationship between your adaptive phase rate and the nominal phase rate).
4. "The resulting dynamics induce a guiding effect during training similar to that achieved in reward-driven curriculum learning" (line 151-152), please provide some empirical or mathematical justification for this claim.
limitations: limitations: 3
review_summary: review_summary: I recommend this paper for acceptance because the adaptive phase formulation is interesting and effective for solving a suite of door opening tasks robustly, and because the questions of robustness and reduction of human effort in policy generation are topical in the robotics/machine learning research community right now. However, I suggest that the authors restructure the methods section to increase clarity and weaken their claims on the generality of the method (unless they demonstrate its applicability beyond door opening).
title: title: Promising work but needs further improvement
paper_summary: paper_summary: This paper proposes a method to enable a quadruped loco-manipulator to navigate doors and open/close dishwashers. The approach first uses a planner to generate a trajectory of high-level waypoints to accomplish the task. These waypoints are sent to a controller, which keeps track of progress towards the goal along the trajectory, and sends intermediate subgoals to the low-level policy. The policy takes in a target state and manipulation mode as input, and is trained using PPO. The approach is demonstrated in both simulation and on a real robot.
main_review: main_review: This paper proposes a method to address four challenging tasks and shows nice results in simulation. The approach is also partially demonstrated on a real robot system. Ablations validate the efficacy of various components, and domain randomization during training improves the robustness of the behavior.

However, I have concerns about the generality of the approach and the real robot implementation. The authors describe a goal of devising a generic formulation that works across scenarios without task-specific handcrafting. However, the lack of breadth in evaluation tasks and the complexity of the approach do not support the claim of a method that is general. This paper reads to me as a specific instantiation of a hierarchical-RL-based approach for these specific tasks.

For the real robot evaluation, the video is rather confusing. The robot does not complete the task, it just opens the door and then video stops there, the robot does not walk through the door. Also, the real robot is completely stationary during most of the video. What is happening during those long pauses? Did the code crash?

Another weakness of the real world implementation is the assumption of access to ground truth states. The authors implement this by installing an encoder on the door hinge and putting fidicial markers on the door surface. This limits the practicality of the approach if the ultimate goal is deployment in the real world.
questions: questions: Presentation
- The message of the paper can be refined, perhaps not claiming a general method if it cannot be backed up
- Some of the terminology in the paper is not widely used in RL and yet are not introduced or defined anywhere. Here are a few examples: references, schedule, phase.

Evaluation:
- There are ablations performed on the method itself but there does not seem to be a comparison to a widely-used existing approach as a baseline

Real robot:
- The robot should complete the task or at least the authors should explain why it is only able perform half of the task
- If the long pauses are due to code implementation, that should be addressed

Minor issues:
- Line 20: "has shown demonstrated" -> "has demonstrated"
- Line 53: "behaviors.This" -> "behaviors. This"
limitations: limitations: 3
review_summary: review_summary: This work is promising but needs improvements in presentation, comparison to another method, and real robot implementation
title: title: Guided Reinforcement Learning with adaptive phase dynamics offer a promising approach to handling contact-rich loco-manipulation tasks
paper_summary: paper_summary: The paper proposes a generic MDP framework that includes a task phase variable to train loco-manipulation RL policies, guided by trajectory optimization.
main_review: main_review: Strengths:
1. The paper is well motivated and easy to follow.
2. The ablations are carefully designed to justify each design component, including the adaptive task phase variable, joint references and curriculum learning.
3. Both simulation and hardware experiments show promising performance against disturbances on multi-contact loco-manipulation tasks., using only a single demonstration per task.

Weaknesses:
1. Lack of Comparison with Baselines: the paper does not compared with other baselines such as vanilla IL or RL from scratch to validate the effectiveness of imitation-based RL.
2. Uncommon MDP Formulation Designs: some of the formulation designs are not common in MDP (e.g., the task phase variable, residual action). The paper would benefit from a high-level intuition and clear explanations of these designs to highlight how they differ from existing formulations and why they are necessary.
3. Limited Real-World Experiments: while the simulation includes the dishwasher task, the hardware experiments are only limited to pushing or pull the door (without actually traversing through the door).
questions: questions: 1. Could you explain the physical meaning and effect of the residual action $\delta_v$ more clearly?
2. Could you give an overview of the proposed formulations and what are their advantages over existing ones?
3. Although the NNS results in 0% success rate, it could be informative to visualize their trajectories as well. Are they stuck at the onset of unforeseen slippage and large disturbances?
limitations: limitations: 3
review_summary: review_summary: I recommend the acceptance of this paper. The proposed MDP framework for guided RL and adaptive phase dynamics offer a promising approach to handling complex, contact-rich tasks. However, including comparisons with baseline methods and providing clearer intuitions for the unique design components would strengthen the paper further.