WoCoCo: Learning Whole-Body Humanoid Control with Sequential Contacts

Authors: chong  zhang
eth  zurich
wenli  xiao
tairan  he
guanya  shi
Venue: robot-learning_org_CoRL_2024_Conference
Published: 2024-06-04 19:14:01

Abstract: Humanoid activities involving sequential contacts are crucial for complex robotic interactions and operations in the real world and are traditionally solved by model-based motion planning, which is time-consuming and often relies on simplified dynamics models. 
Although model-free reinforcement learning (RL) has become a powerful tool for versatile and robust whole-body humanoid control, 
it still requires tedious task-specific tuning and state machine design and suffers from long-horizon exploration issues in tasks involving contact sequences. In this work, we propose WoCoCo (Whole-Body Control with Sequential Contacts), a unified framework to learn whole-body humanoid control with sequential contacts by naturally decomposing the tasks into separate contact stages. Such decomposition facilitates simple and general policy learning pipelines through task-agnostic reward and sim-to-real designs, requiring only one or two task-related terms to be specified for each task. We demonstrated that end-to-end RL-based controllers trained with WoCoCo enable four challenging whole-body humanoid tasks involving diverse contact sequences in the real world without any motion priors: 1) versatile parkour jumping, 2) box loco-manipulation, 3) dynamic clap-and-tap dancing, and 4) cliffside climbing. We further show that WoCoCo is a general framework beyond humanoid by applying it in 22-DoF dinosaur robot loco-manipulation tasks. Website: lecar-lab.github.io/wococo/.

Main paper: 
 introduction: 
 Humanoids are designed to operate in and interact with environments like humans do, which often requires the fulfillment of sequential contacts during task execution [1] . Provided specific contact plans, the typical solution is to employ model-based motion planning or trajectory optimization to generate whole-body references for tracking [2, 3, 4] . Although motion planning can be powerful for motion synthesis, it is often time-consuming and relies on simplified reduced-order dynamics models, which may affect the motion quality and the real-world performance [5, 6, 7, 8, 9, 10] .Model-free reinforcement learning (RL) has demonstrated remarkable robustness against model mismatch and uncertainties, and enabled real-time agile motions on legged robots [11, 12, 13, 14, 15] . However, these works focus on standard locomotion tasks (e.g., walking) without the necessity to fulfill specific contact sequences. Although some recent works have achieved RL-based locomotion with constrained footholds [16, 17, 18, 19, 20] , they are heavily tuned for specific scenarios.8th Conference on Robot Learning (CoRL 2024), Munich, Germany. Similarly, in existing works that showcase task-aware contact sequences in real-world humanoids, such as soccer playing [21] and loco-manipulation [22, 23, 24] , each RL policy is specifically tuned for a particular task or transition in the state machine. Distinct formulations and rewards are required for different policies, limiting their practical application in long-horizon tasks. Sferrazza et al. [25] trained policies for multiple dynamic manipulation tasks with a shared hierarchical RL architecture, yet they did not address sim-to-real concerns or propose unified contact-related rewards. Xiao et al. [26] enabled humanoids to track desired contacts with objects, but the controller was limited to animation purposes and required human motion references. To summarize, existing works on RL have shown potential in robustness and versatility, while a systematic and general method for controlling real-world humanoids under desired contact sequences using RL is missing.In comparison, a single model-based solver such as Crocoddyl [5] can address multiple tasks with different contact sequences, requiring only slight adjustments to costs (few intuitive task-related terms). The question is: How can we achieve such simplicity and adaptability with an RL framework? Besides, regarding effective policy learning, we also identify three challenges: (1) Contacts are sparse, especially when coupled with other whole-body motion goals such as balancing and posture maintenance; (2) Robots may avoid exploring the whole long horizon due to the compounding risks; and (3) Sim-to-real transfer is non-trivial, and is often achieved through domain randomization [27] and task-irrelevant regularization rewards, which can hinder exploration.In this work, we address the question by proposing WoCoCo, a general RL framework for wholebody humanoid control with sequential contacts. In WoCoCo, we reformulate the problem as the sequential fulfillment of multiple contact stages (detailed in Section 2), which also breaks down the exploration burden into separate stages. This then transforms each challenge to a question: Q1: How to reach desired contact states within each stage? Q2: How to streamline exploration across multiple contact stages? and Q3: How to develope a compatible sim-to-real pipeline?We jointly tackle Q1 and Q2 by a concise yet effective WoCoCo reward design (detailed in Section 3.1), which is a combination of dense contact rewards, stage count rewards, and curiosity rewards. The dense contact rewards outperform standard 0-1 rewards [24] by counting every correct and incorrect contact, thereby guiding the policy more effectively. On top of that, the stage count rewards are proposed based on the number of fulfilled contact stages. This drives the robot to explore further stages to maximize cumulative rewards, thus mitigating the shortsightedness caused by the RL policy strategically staying in the current stage to avoid potential failures. To better facilitate exploration, we propose a task-agnostic curiosity reward term. Via detailed ablation analyses, we show that the WoCoCo reward design is both effective and minimal in Section 5.In addressing Q3, we also propose a general sim-to-real pipeline with domain randomization and regularization rewards (Section 3.2). Inspired by [28] , we design a curriculum with three training stages: initially training without domain randomization, then training with domain randomization, and finally increasing the weights of regularization rewards. This curriculum reduces the exploration burden introduced by sim-to-real modules in training. To summarize, our contributions are:1. We propose WoCoCo, a general framework for RL-based whole-body humanoid control under sequential contact plans, with natural task decomposition based on contact stages. 2. We showcase how WoCoCo's task-agnostic designs empower end-to-end RL to tackle four challenging humanoid tasks and a 22-DoF dinosaur robot task, showing versatility and universality. 3. We validated the learned RL policies for the four aforementioned humanoid tasks in the real world, as shown in Fig. 1 and the videos. To our knowledge, these are the first instances of each task being solved by a single end-to-end RL policy.2 Overview: Learning with Sequential Contacts and Task DecompositionConsidering a wide range of robotic tasks requiring active contacts for environment interactions, such as parkour jumping and loco-manipulation, we decompose these tasks into multiple contact stages i ∈ {0, 1, . . . , I} based on the desired contact sequences, as illustrated in Fig. 1 . The robot is expected to sequentially fulfill these stages, where the fulfillment of each stage is defined as the simultaneous fulfillment of a contact goal g con i (defining the contact states that certain end effectors should reach) and a task goal g task i (defining additional task-specific requirements). In this paper, we study tasks where contact stages are predefined (e.g., heuristically designed), and our method can seamlessly be integrated with high-level contact planners (e.g., [29] ). For example, in the parkour jumping task (Section 4.1 and Fig. 1(B )), each stepping stone corresponds to a contact stage where achieving correct foot contacts defines the contact goals, and maintaining upper body posture forms the task goals. Upon fulfilling a stage, checked by sensors or human observation, the robot advances to the next stage after a predefined arbitrary time period.To develop RL-based controllers for these tasks, we formulate the policy learning problem as an extended Markov Decision Process (MDP) M = ⟨S, A, T , R, γ, G con , G task ⟩ of state s t ∈ S, action a t ∈ A, transition probability T , reward r t ∈ R, discount factor γ, contact goal g con i ∈ G con , and task goal g task i ∈ G task . The objective is to maximize the expected return E [ t γ t r t ] by finding an optimal policy a t = π * (s t |g con i:I , g task i:I ). We define our rewards as r = r WoCoCo + r reg task-agnosticEQUATIONwhere r WoCoCo is the task-agnostic rewards (detailed in Section 3.1), r reg is the task-agnostic regularization rewards for sim-to-real transfer (detailed in Section 3.2 and Appendix F), and r task is the task-related rewards with few intuitive terms (detailed in Section 4 for different tasks).We employ Proximal Policy Optimization (PPO) [30] with symmetry augmentation [31] (detailed in Appendix D) to optimize the policy in Isaac Gym [32] simulation based on the parallel RL framework in [33] . All policies trained in this work are end-to-end MLP policies, while our framework does not restrict the policy architecture. The policy observations can include proprioception, exteroception (optional), and goal-related observations, which are detailed in Appendix H. The policy outputs a t are joint target positions tracked by low-level PD controllers to actuate the motors.The remainder of the paper is organized as follows: In Section 3, we detail our task-agnostic r WoCoCo reward terms and sim-to-real designs. In Section 4, we show how our framework, WoCoCo, can be applied to a variety of challenging dynamic tasks with flexible definitions and representations of contact and task goals. We conduct further analyses and ablation studies in Section 5, and discuss the limitations and future works in Section 6. 
wococo rewards and sim-to-real transfer: 
 This section presents our novel reward designs to overcome the challenges discussed in Section 1 and the sim-to-real pipeline.We propose WoCoCo rewards which comprise three task-agnostic terms:EQUATIONwhere r con is the contact rewards, r stage is the stage count rewards, and r curi is the curiosity rewards.As mentioned in Section 1, r con densifies the contact state reaching rewards, r stage incentivizes exploration across multiple contact stages, and r curi further facilitates exploration in the state space. These reward terms are detailed in the following subsections.Denser Contact Rewards: Every Contact Matters. r con encourages correct contacts with task goal fulfillment, and offers denser rewards than 0-1 rewards1 by additionally rewarding each correct contact while penalizing each wrong one. We define it asEQUATIONwhere n con is the maximal number of end effectors involved in the contact sequence, n corr and n wrong are respectively the number of end effectors with correct and wrong contacts at the current timestep, n stage is the number of fulfilled stages, F con and F task are respectively the boolean values for whether the contact goal or the task goal of the current stage is fulfilled. These symbols are further exemplified in Fig. 7 in Appendix A. The coefficients are designed to avoid local maxima, and the penalty for n wrong is masked during the first contact stage (n stage = 0) to encourage exploration and avoid invalid episode reset (otherwise, the robot may immediately receive penalties upon reset).Stage Count Rewards: Do More, Get More. Exploring new contact stages can come with failures and penalties, while staying at the current one may bring positive rewards. Hence, it is necessary to drive the agent towards new stages via rewards. To this end, we define the stage count reward asEQUATIONand the condition F task is to avoid intentional unfulfillment of the task goal for cumulated stage count rewards.Curiosity Rewards: Drive the Exploration. Curiosity rewards have been used to encourage exploraton in the RL context [34, 35] . For high-dimensional observations, random network distillation (RND) [36] has been proposed as a flexible and effective way for curiosity-driven exploration. In robot control, Schwarke et al. [24] have successfully applied RND to real-world whole-body locomanipulation problems, while the curiosity observations are task-specific based on expert insights.In this work, we aim to define curiosity rewards with task-agnostic observations that can be redundant and high-dimensional. We find that RND can over-explore states that do not generate meaningful behaviors, similar to what is reported in the RND work's Section 3.7 [36] . Instead, we propose to use count-based curiosity rewards via random neural network (NN) based hash, inspired by Tang et al. [37] and Charikar [38] .To be specific, we define a task-agnostic set of curiosity observations o curi (detailed in Appendix I). With a randomly initialized and frozen NN f curi : R dim(ocuri) → R dim(hash) , each o curi is hashed to a bucket with the indexEQUATIONwhere BIN2DEC interprets an array of boolean values as a binary number and converts it to the decimal format2 , and the curiosity reward is based on how many times the robot has visited states hashed in the same bucket:EQUATIONWe find our curiosity rewards powerful and stable in facilitating exploration, even with task-agnostic curiosity observations across different challenging whole-body tasks. Based on the hash mechanism, overfitting of random networks can be constrained by the numerical decay as the #. visits increases.Following existing works [39] , we use domain randomization [27] and regularization rewards to enable sim-to-real transfer. The details are presented in Appendix. Notably, our domain randomization settings and regularization rewards are shared across all humanoid tasks.We also apply a curriculum to reduce the exploration burden posed by domain randomization and regularization rewards, inspired by Li et al. [28] . Specifically, 1) we first train policies without domain randomization until they converge, 2) then resume training with domain randomization until convergence, and 3) afterwards increase the weights of regularization terms by 20% per 2000 iterations until they double, inducing more conservative behaviors. The curiosity rewards are activated only in 1). Following Li et al. [28, 40] , we stack 3 control steps of previous joint states and actions, and append them to the policy observations to enhance the robustness by temporal memory. 
case studies: 
 In this section, we show how our framework, WoCoCo, can be applied to various challenging tasks with different contact sequences. As mentioned, we use the same r WoCoCo and r reg terms for different tasks, and the only task-specific adjustments are one or two very intuitive task rewards (introduced in each subsection). For brevity, we present the task definition, reward intuitions, and results here, detailing the observations and reward designs in Appendix.Parkour jumping by humanoids is a highly challenging dynamic task demonstrating advanced agility with precise landing, as showcased by Boston Dynamics (BD) [41] . However, BD's parkour motions come from a behavior library through offline trajectory optimization [42] , which can limit versatility when deployed in the wild or when additional upper body motion is required for specific tasks. Li et al. [43] have achieved continuous bipedal jumping based on online model-based optimization, while only double-foot forward jumps without upper body tasks are supported. Li et al. [44] use RL to learn double-foot jumping in the 3D space, yet their method does not support continuous jumps, relies on a motion reference, and does not consider humanoids with upper body motions.In contrast, we show that WoCoCo can enable end-to-end RL-based versatile parkour jumping with 1) single/double-foot contact switch, 2) controlled landing in the 3D space, and 3) upper body posture tracking, without any motion reference.Task Definition. As shown in Fig. 2 , we train the humanoid to jump over stones with various contact sequences, where each stone makes a contact stage. The contact goal is to have the correct foot (left/right/double) contact the stone, and the task goal is to maintain specified upper body postures ("hug"/"relax"). This setup challenges the robot to accurately execute foot contacts while adjusting its upper body posture during highly dynamic and coupled movements.Reward. There is only one task-related reward term, encouraging tracking of the elbow position and orientation in the base frame to fulfill the task goal.Results. The results are shown in Fig. 2 , demonstrating the humanoid's capability to perform versatile continuous jumping while tracking upper body postures, and robustness against perturbations such as unseen gravels. In the real world, we only tested double-contact sequences with one or two continuous jumps due to facility constraints. Yet, the robot exhibited highly dynamic and adaptive behaviors for different stone heights and distances.Box loco-manipulation is an important application of humanoids and has been well studied with model-based controllers [45] . However, model mismatch and perturbations such as uneven terrains pose significant challenges to these controllers, for which RL can be a promising solution [13, 22] .That said, existing RL-based works either depend on finite state machines and train separate policies for each state transition with distinct formulations, rewards, and posture priors such as stance width [22, 23] , or are limited to short-distance movements [24] . In this paper, we show that with provided current and goal positions3 of the box, an end-to-end RL policy can control the humanoid to first approach the box and then transport it to the destination without any posture prior. The learned whole-body coordination can also enhance the motion efficiency, as observed in the existing works on quadrupeds [46, 47, 48, 49] .Task Definition. We define two contact stages. In the first stage, the contact goal is to place hands on both sides of the box, while the task goal is always fulfilled. In the second stage, the contact goal is to maintain hand contact with the box sides, and the task goal is to transport the box close to the destination. Placement at the destination is also feasible by modifying the contact goals to a virtual one. By defining the contact sequence solely on the hands, we leverage RL to achieve robust locomotion while simplifying the whole task.Reward. There are two task-related reward terms, which incentivize minimizing the distances between the hands and the box, and between the box and its destination.Efficient Turning in 0. 3 , the humanoid can efficiently turn, transition seamlessly between walking and picking, and simultaneously approach the destination while picking up the box. It can also recover after stepping on a belt tied to itself, showcasing robustness.Humanoids may also entertain with dynamic dancing skills. BD has achieved impressive dancing with model-based control and offline trajectory optimization [50] . Existing RL-based methods can track human references [39, 51] , yet unable to ensure accurate tapping on the ground. Here we show WoCoCo can enable RL-based dynamic dancing with accurate tapping and optional clapping.Task Definition. In this task, contact stages are assigned to feet and hands. As shown in Fig. 4 , there are three moves to compose diverse contact sequences where "Left" and "Middle" can transit to each other and so do "Right" and "Middle". In each contact stage, the task goal is to position the hands within the black bounding boxes (predefined in the base frame). The contact goal requires foot contact with the ground in their corresponding bounding boxes (predefined in the world frame), plus hand self-collision if the move is "Left" or "Right".Middle Right Results. We successfully learned the policy with real-world deployment, as shown in Fig. 4 .Cliffside climbing is a representative task requiring precise movement of all limbs to support the humanoid. Though model-based controllers [9, 52, 53, 54] have showcased success in such problems, we prove RL is also a promising solution for fast and resilient multi-contact locomotion.Task Definition. In this task, the contact sequences are tracked to make the humanoid move along the cliffside, as shown in Fig. 5 . In each contact stage, the task goal is always fulfilled. The contact goal requires both hands to touch the goal regions on the wall, while both feet need to stand on their goal regions on the ground. Each end effector's goal region is bounded by a 2-d square. Reward. There are two task-related reward terms. The first encourages the humanoid to face the wall, and the second incentivizes precise movement of end effectors by minimizing the distances between the end effectors and the centers of their goal contact regions.Results. The learned cliffside climbing behavior is shown in Fig. 5 . The policy is robust against pushes and unseen gravels in simulation. In the real world, the cliff is replaced by a board held by a human, and the humanoid can adapt to varying contact forces on the hand during the interaction.To show WoCoCo can generalize to other embodiments, we trained a 22-DoF dinosaur robot (adapted from [55] ) to perform a ball loco-manipulation task. This involves pushing a ball to a specified destination using one of its six end effectors (head, tail, and four feet). The task definition mirrors that of box loco-manipulation, except that the desired contact point is the projection of the destination through the ball's center to its surface, facilitating the ball's movement toward the destination. Contacts made by the end effector with the ball near this point fulfill the contact goal. The results are shown in Fig. 6 .Figure 6 : We train the dinosaur robot to push the ball towards destinations with different end effectors. By altering the destinations, we make the robot generate ball trajectories forming "WoCoCo". 
analyses and ablations: 
 Given that versatile parkour jumping is arguably the most challenging task, and training every task for the same setting is costly, we do analyses and ablation studies based on the jumping task. The learned behaviors of baselines are visualized in Fig. 8 in Appendix B.Ablating Dense Contact Rewards. With 0-1 contact rewards r 0-1 con = c 0-1 F con F task , the humanoid cannot explore to jump over the stones, and tracks upper body postures without moving. This proves the necessity of our dense contact rewards.Ablating Stage Count Rewards. Without the stage count rewards, the humanoid intentionally does not fulfill the contact goal to avoid progressing to further contact stages, while still obtaining other rewards. This verifies the effectiveness of our proposed stage count rewards.Ablating Curiosity Rewards. Without the curiosity rewards, the humanoid cannot jump over the stones, and tracks upper body postures without moving, which means under-exploration. With RNDbased curiosity rewards, the humanoid learns to lean backward in a risky way, which aligns with the observations by Burda et al. [36] : the agent may over-explore a dangerous behavior pattern while staying alive, as such states are rare in the agent's experience compared to safer ones. In comparison, our curiosity rewards achieves effective exploration without overfitting specific behaviors.Empirical Benefits of WoCoCo. With WoCoCo, the humanoid demonstrates high agility and motion efficiency. These motions are not constrained by simplified models and motion priors. Besides, by training with diverse task configurations, the learned RL policies can fulfill versatile contact goals. The policies also showcase robustness against perturbations such as unseen gravels.Training Stability. Despite the stochasticity of curiosity-driven exploration as shown in [24] , our method has been stable against random network initialization and exploration. This is shown by Fig. 10 in Appendix C where we plot the learning curves for five different random seeds. 
limitation and future works: 
 One limitation of our work is the lacking knowledge of when the controller will fail. In contrast, model-based methods can explicitly tell whether they can find a feasible solution. Therefore, we may explore failure predictors [56] and other safety assessment methods in the future [57] . Besides, if the contact sequence length is unknown a priori, we may need heuristic reward clamping to avoid the robot exploiting the stage count reward.We currently rely on motion capture as a prototype, and we will try to incorperate onboard sensing in the future. We will also explore sampling-based [53] or LLM-based [26] high-level planners, while currently we predefine contact sequences based on heuristics. Another limitation is the requirement for explicit contact feedback (by contact sensors or human observers) to switch contact stages, a process that might be implicitly managed by the policy in the future. Supplementary Only one task-related reward is introduced for parkour jumping:EQUATIONwhere |err rot | and |err pos | are respectively the rotation and position errors of the elbow w.r.t. the desired orientations and positions in the base frame. We have w task = 30.The task rewards include two terms:r task = w box exp(-d box2dest ) • 1(n stage > 0) • 1(|θ dest | < π 2 )+w hand exp(-EQUATIONwhere d box2dest is the distance between the box and the destination, d left2box and d left2box are respectively the distances from the left hand and the right hand to the corresponding box side center, θ dest and θ box are respectively the direction angles of the destination and the box in the base frame. These two terms encourage approaching and moving the box. We have w hand = 100, w box = 200.For dinosaur ball loco-manipulation, we remove the direction angle conditions.The task rewards include two terms:EQUATIONwhere b y lh and b y rh are respectively the y values (displacement from the sagittal plane) of the left hand and the right hand in the base frame, I lh and I rh are respectively the signs of the y values for the centers of the left and right hand bounding boxes in the base frame, d lf2box and d rf2box are respectively the distances from the left foot and right foot to the centers of their bounding boxes. These two terms encourage arm spread with precise footholds. We have w hand = 5, w foot = 10.The task reward includes two terms: rwhere θ wall is the angle of the wall relative to the base yaw, and d ee2goal is the distances from the four end effectors to the centers of their goal regions. These two terms encourage the humanoid to face the wall while maintaining precise end effector placement. We have w base = 50, w ee = 5.The policy observations in this work consist of two parts for all policies: proprioception and goal representations. Camera and LiDAR observations are not used here but our framework does not exclude them. The goal representations are provided by the motion capture system as a prototype.We use the following proprioception observations (shared across all tasks): joint positions, joint velocities, previous actions, base linear and angular velocities, and projected gravity. Joint positions, joint velocities, and previous actions are stacked by 3 control steps.For versatile parkour jumping, we feed the future two stages' goal representations to the policy network, so the robot can adapt the foothold for future goals. For other tasks, we feed only the current stage's goal representations. This design allows our policies to generalize to varying sequence lengths during deployment.The contact goal is represented by the corner points of each foot's next two stones in the base frame, which are set to zeros when the foot is intended to be in the air. The task goal is represented by the desired elbow orientation and position in the base frame.The contact goal representation is the center points of the box sides in robot's base frame, and the task goal representation is the destination position in the base frame.We use a combined contact-and-task observation: the one-hot vector for the case (Left or Middle or Right) plus the robot's x, y, and yaw values in the world frame. This accommodates scenarios where the humanoid is required to dance within a fixed area.The task goal is always fulfilled so we only need the contact goal representation: corner positions in the base frame for all the goal regions of the current and next contact stages.Actors and Critics are all MLPs with [512, 256, 128] hidden units.Reward 1 if the contact stage is fulfilled, else 0. Can be defined as FconFtask using the symbols in Eq. 3.For example, if fcuri outputs [1.5, -0.2, 0.4], the binary number is 101 and the bucket id is 5.Referred to as "destination" to avoid confusion with contact/task goals. 
 
 Appendix:  


Decision: The paper proposes WoCoCo, a unified framework for whole-body humanoid control tasks involving sequential contacts. The key idea is to use contact positions as high-level command representations to design task-agnostic reward functions for low-level control policies of humanoid robots. The paper received positive initial reviews from the three reviewers. The reviewers pointed out the work's strengths as 1) using contacts for reward formulation is general and applicable for diverse and challenging control tasks, and 2) the trained policies were successfully transferred from simulation to the real world. Meanwhile, they also pointed out its weaknesses as 1) manual regularization was required for walking but needs to be turned off for jumping, which limits the framework's flexibility, 2) manual efforts were required for tuning the weighting terms of reward components for individual tasks, and 3) writing clarify can be further improved, and some technical details should be further discussed.

**Post-rebuttal update:**

The AC appreciated the detailed responses in the authors' rebuttal. Toward the end of the discussion period, all three reviewers remained positive about this submission. Although some lingering (and minor) issues have been mentioned by Reviewer MQ1w and HtJA, which were not fully resolved by the rebuttal, the AC believes that this work has made sufficient technical contributions and demonstrated impressive behaviors on the humanoid robot. Therefore, the AC concurs with the reviewers and recommends accepting this paper for CoRL.

Reviews: review: title: title: Review
paper_summary: paper_summary: This paper aims to propose a unified framework for whole-body humanoid control tasks with sequential contacts. It describes a general reward recipe that can be applied to any task involving sequential contacts, and shows that it yields successful policies for four challenging humanoid tasks.
main_review: main_review: Strengths:
- The paper makes a good novel proposition of what is currently missing from RL whole-body control: a “systematic and general method” for specifying the task, and proposes a good answer with the contact goal reward scheme that is shown to represent and facilitate optimization for diverse challenging tasks.
- The stage count rewards and curiosity rewards are task agnostic.
- The sim-to-real recipe seems good since it transfers diverse and dynamic behaviors to the real robot.

Weaknesses:
- The framework requires some manual regularizations for walking that need to be turned off when jumping. I'm curious if they can be replaced with WoCoCo rewards or there is a limitation in representing e.g. trotting interleaved with other contact based objectives.
- The weighting of the contact, stage count, curiosity reward, and task rewards still needs to be tuned for each task. So, while an impressive amount of the task spec is task agnostic, there is still room to further reduce the effort.
questions: questions: The framework does still require manual regularizations for walking that need to be turned off when jumping. Why not manually define a trotting or standing contact schedule for the feet during locomotion tasks (e.g.  https://arxiv.org/abs/2011.01387) and fully standardize the WoCoCo framework? Is there a limitation that things get tricky here, when the number of sequential contacts is large or unknown a priori, as with the foot contacts when walking to pick up a box? I guess you need to skip the stage reward and impose a manual timing for those trotting feet contacts..

Formulating the problem as sequential contacts might be limiting if the robot has grippers or hands. What you really want to know is not whether the hand is in contact but whether it is grasping, which is not an easy query to the simulator. Do you foresee any limitations of WoCoCo for embodiments with non-point end effectors?
limitations: limitations: 3
review_summary: review_summary: The paper proposes a novel formulation for humanoid whole-body control with sequential contacts, with impressive sim-to-real results. I have a few minor questions related to possible extensions.
title: title: Valuable insight in using contact sequence as a means of conveying motion command to humanoid policy.
paper_summary: paper_summary: This paper proposes a general framework of using contact positions and robot base pose as a high-level command representation to control the low-level humanoid robot policy. The author demonstrates the effectiveness of the proposed method in various case studies, including *Versatile Parkour Jumping*, *Dynamic Clap-and-Tap*, and *Bidirectional Cliffside Climbing*.
main_review: main_review: This paper describes a valuable formulation of using contact sequences to convey motion commands to the humanoid policy. This framework could potentially impact the general humanoid algorithm implementation. The author provides a clear and well-organized description of the motivation, proposal, and experiment results. However, it is hard to picture each case study's implementation and the numerical formulation. The technical details could be improved.

**strength**
* The proposal of using contact as a high-level command representation is quite a universal and important direction to the humanoid low-level control.

* This work demonstrates the importance of using contact as a high-level command with sufficient case studies.

* The insight of adding correct and wrong contacts into reward formulation is valuable.

**weakness**
* The title of Section 3 *How to train your Droid* does not sound academic. It would be better to rephrase it.

* The author fails to explain how the contact points are assigned in the real world, especially in the case of *Versatile Parkour Jumping*.
questions: questions: 1. Is all the contact goal $g_i^\text{con}$ and task goal $g_i^\text{task}$ fed into the policy network in each timestep? If not, how does the author select the goal to feed into the policy network?

2. How does the author specify the goal positions in the robot base frame in the real world, since the author claims that they do not use exteroception? For example the platform position in the *Bidirectional Cliffside Climbing* case.

3. How does the system determine whether a contact stage is fulfilled given a state at timestep $t$?

4. How does the author train the policy to fulfill the contact stages one by one? In the example of the *Dynamic Clap-and-Tap* case, the policy could easily learn to fulfill the similar contact stages first, then the different contact stages. How does the author prevent this from happening?
limitations: limitations: 3
review_summary: review_summary: This paper provide valuable insight of how this contact sequences as command can be deployed in the real-world. But the technical details should be improved. I recommend a weak acceptance on this paper.
title: title: The significance of the paper is considered strong.
paper_summary: paper_summary: This paper proposes WoCoCo, a whole-body control framework for humanoid robots using reinforcement learning. In WoCoCo, a task was decomposed into several contact stages, and policies were learned such that the contact and task goals at each stage were satisfied. To enable the transfer of policies to a real robot, WoCoCo learned the policies through a training curriculum using domain randomization and regularization rewards.
main_review: main_review: Strengths:
* This paper is significant because it shows that the proposed framework can generate a variety of motions in real humanoid robots. 
* The accompanying video and the video available on the project web page are very impressive. 

Weaknesses:
* The abstract and introduction need to be improved. For example, the authors criticize conventional RL for requiring careful design of state machines and distinct rewards, but WoCoCo also requires decomposition of tasks into contact stages and design of task-specific reward functions. It is a subjective to say that the decomposition into contact stages and the design of the reward function in WoCoCo are simpler, so the improvement from the conventional RL is a bit unclear. Furthermore, it is not clear why (1)-(3) are the primary challenges to the question "How can we achieve such simplicity and adaptability with an RL framework? It is also not clear why challenges (1)-(3) turns into the three key questions Q1-Q3.
* Lack of experimental validation makes it difficult to admit that the development of the training curriculum is a contribution of this paper. Thus, the methodological contribution of this paper appears to be weak, as it only proposes the design of an effective reward function. I would like to see the effectiveness of the training curriculum if it could be easily demonstrated, e.g., through an ablation study.
* It would be easier to understand the effectiveness of the proposed reward function if the learning curves for the ablation study in Section 5 and Appendix B were also shown.
* I think it would be better to delete the DANCING WITH SKULLS quote on page 4, as it is not clear to readers who have not read the RND work [36] and would make this paper less self-contained.
* I recommend that the authors describe the details of real robot control. For example, the following details would be useful
(i) Have system identifications of robots and actuators been performed to reduce the sim-to-real gap?
(ii) At what hertz are the policies updated?
(iii) At what hertz is the low-level PD controlled?
(iv) Is a low-pass filter applied to the policy?
* Typo on page 8: Besides, By training with ...
questions: questions: Issues that need to be addressed by the authors are listed in Weaknesses in the Review section.
limitations: limitations: 3
review_summary: review_summary: The contribution of this paper is sufficient: strength in significance of the paper. The experiments are performed in not only simulation but also real environments.