RobotKeyframing: Learning Locomotion with High-Level Objectives via Mixture of Dense and Sparse Rewards

Authors: fatemeh  zargarbashi
jin  cheng
dongho  kang
robert  sumner
stelian  coros
eth  zürich
  switzerland
Venue: robot-learning_org_CoRL_2024_Conference
Published: 2024-06-04 18:12:11

Abstract: This paper presents a novel learning-based control framework that uses keyframing to incorporate high-level objectives in natural locomotion for legged robots. These high-level objectives are specified as a variable number of partial or complete pose targets that are spaced arbitrarily in time. Our proposed framework utilizes a multi-critic reinforcement learning algorithm to effectively handle the mixture of dense and sparse rewards. Additionally, it employs a transformer-based encoder to accommodate a variable number of input targets, each associated with specific time-to-arrivals. Throughout simulation and hardware experiments, we demonstrate that our framework can effectively satisfy the target keyframe sequence at the required times. In the experiments, the multi-critic method significantly reduces the effort of hyperparameter tuning compared to the standard single-critic alternative. Moreover, the proposed transformer-based architecture enables robots to anticipate future goals, which results in quantitative improvements in their ability to reach their targets.

Main paper: 
 introduction: 
 Legged robots hold a great promise for becoming household companions [1] or automated performers in the entertainment industry [2, 3] . In these applications, it is crucial for robot controllers to perform natural and directable behavior from simple high-level user command inputs beyond the typical commands used in the robotic domain such as joystick velocity commands [4, 5] or target base position [6, 7] .In the character animation domain, a widely used technique for specifying character behavior from simple and sparse inputs is keyframing [8, 9] . It involves defining the target position or kinematic pose of the character at particular points in time, allowing animators to create smooth movements by interpolating between these keyframes. Despite its proven effectiveness within the kinematic animation pipeline, incorporating keyframing for achieving time-specific targets remains unexplored in the realm of physics-based robot control.With inspiration from the character animation technique, we aim to equip legged robots with more refined control by incorporating sparse and temporal high-level objectives as keyframes. The primary goal of this work is to develop a locomotion controller that enables the robot to fulfill specified partial or full-pose targets while infilling natural behavior during the intermediate periods. This goal aligns with recent advancements in using reinforcement learning (RL) for legged robots due to their promising robustness and flexibility [10, 11] . However, learning a policy that accurately meets keyframes without imposing undesired constraints at intermediate periods presents challenges, particularly due to the need to handle sparsity in the keyframe objectives. Acquiring effective policies requires a meticulous reward design procedure that carefully balances these sparse rewards with other dense rewards which are crucial for regularizing and encouraging natural motion.In this work, we present a novel framework that unifies timed high-level objectives with natural locomotion of legged robots through temporal keyframes. Along with the imitation objective similar to Peng et al. [12] for natural motion generation, our pipeline allows specifying full or partial highlevel targets, including base position, orientation, and joint postures. We propose using a multi-critic RL framework to address the challenge of managing groups of sparse and dense rewards by learning distinct value functions. Our method also employs a novel transformer-based architecture to encode a variable number of goals with arbitrary time intervals. Unlike typical sequence-to-sequence transformers [13] , we propose a lightweight sequence-to-token module that can be used autoregressively within a feedback control loop. We demonstrate the effectiveness of our framework through experiments both in simulation and on real-world hardware. Our policies successfully guide the robot to meet multiple keyframes at the required times, for both position and posture targets. Furthermore, the multi-critic approach showcases better convergence with less hyperparameter tuning compared to the conventional single-critic method. Our experiments also reveals that using a transformer-based encoder to anticipate future goals significantly enhances goal-reaching accuracy.The contribution of this paper is threefold: (i) We introduce RobotKeyframing, a novel learningbased framework for integrating high-level objectives in natural locomotion of legged robots; (ii) We propose using multi-critic RL to effectively handle the mixture of dense and sparse rewards, along with a sequence-to-token encoder to accommodate a variable number of keyframes; (iii) We validate the effectiveness of our method through extensive experiments in simulation and on hardware. 
reinforcement learning for legged robots: 
 Over the last decade, reinforcement learning has been increasingly applied to develop locomotion policies for legged robots [4, 14, 15] . The primary focus has been to achieve robust control policies that can accurately track velocity commands from joysticks [11, 16, 17] . More recently, researchers have attempted to enhance the versatility of legged robot controllers by incorporating high-level objectives, particularly through position-or orientation-based targets [18, 19, 20] . This high-level control is typically accomplished through hierarchical frameworks, where a high-level policy is learned to drive a low-level controller [7, 21, 22] . Conversely, end-to-end approaches aim to develop a unified policy for both high-and low-level control, allowing high-level objectives to directly influence low-level decisions [6, 18, 23] . However, the aforementioned methods typically urge the robot to reach a target as fast as possible, lacking refined control over the temporal profile of achieving the target. Inspired by keyframing in animation, this work aims to further expand control over robot motion by incorporating multiple keyframes as input to the control policy, thereby enabling robots to generate diverse behaviors in reaching targets. We further enhance this versatility by allowing partial or full targets, including base position, orientation, and joint postures.Synthesizing naturalistic behavior from existing motion datasets while fulfilling spatial or temporal conditions has been extensively studied in the character animation domain [24, 25, 26, 27] . Existing research for generating natural motions between keyframes [28, 29, 30] has mainly focused on the kinematic properties of characters and thus cannot be directly applied to physics-based characters or robots, whose dynamic interactions with the environment require consideration of both kinematics and dynamics. Various efforts have also been made to combine kinematic motion generation with physically controlled robots to achieve natural behavior on hardware [31, 32, 33, 34] . Another thread of research focuses on controlling characters in physically simulated environments, incorporating motion datasets as demonstrations [35, 36, 37, 38] . Some of these methods have also been successfully transferred to robot control for quadrupeds or humanoids [39, 40, 41, 42] . Among these works, Adversarial Motion Priors (AMP) [12] provides a flexible way to encourage the policy to have natural, expert-like behavior by connecting generative adversarial networks (GAN) [43] with RL given an offline motion dataset. We also incorporate an AMP-based imitation objective to encourage naturalistic motion for the policy and further extend it to infilling keyframes for robots. 
problem setup: 
 To integrate high-level control objectives into the robotic control framework, we employ sparse keyframes that require a robot to achieve specific goals at predetermined times. Each keyframe contains a full or partial combination of a variety of targets such as global base position p ∈ R 3 , global base orientation ( φ, ζ, ψ) ∈ R 3 where ϕ, ζ, ψ denote roll, pitch, and yaw angles respectively, and full posture specified by joint angles θj ∈ R Nj where N j is the number of joints. Each keyframe is also assigned with a specific time t ∈ R in the future at which the robot is expected to meet the goals. In summary, the high-level objectives are specified through these keyframesK = k 1 , k 2 , ..., k n k ,where k i = ĝ, t i and ĝ ⊂ p, φ, ζ, ψ, θj . Here, n k ≤ N k where n k and N k denote the actual and the maximal number of keyframes, respectively. We aim to support an arbitrary number of keyframes, allowing for the flexible specification of high-level objectives only as needed.The main goal is to train a locomotion policy for legged robots that not only meets these keyframes but also maintains a natural style during the intervals between them. To avoid undesired restrictions on the intermediate periods, policy's task performance is evaluated exclusively at the designated times, making the keyframe objectives temporally sparse. However, relying solely on keyframes to train the control policy may result in undesirable motions. Thus, it is crucial to have additional rewards for regularizing and promoting a natural motion style. In this regard, we incorporate AMP [12] as a general style guide for the robot, encouraging the policy to behave naturally and similarly to an offline motion dataset from dogs [44] . The style and regularization rewards are evaluated at every step of the episode, making them temporally dense. The mixture of sparse and dense rewards presents a unique challenge that is difficult to manage effectively with standard RL frameworks. Details on the observation, action, reward definitions, and training setup can be found in Appendix A.Modern RL algorithms [45, 46, 47] typically employ the actor-critic paradigm, where the actor decides the action to take, and the critic evaluates the action by estimating the value function. To effectively manage a complex mixture of temporally dense and sparse rewards, we employ a multicritic (MuC) RL framework by Martinez-Piazuelo et al. [48] as shown in Fig. 2 . It involves training a set of critic networks {V ϕi } n i=0 to learn distinct value functions associated with different reward groups {r i } n i=0 . Similar concepts have been used to balance a set of dense rewards [49, 50] ; however, we extend the multi-critic method to the context of dense and sparse reward combination, where we demonstrate its distinct advantages. We design each reward group to contain either exclusively dense or sparse rewards. This division is essential for effectively managing the distinct temporal characteristics of each reward type and facilitates value estimation.We integrate the multi-critic concept to Proximal Policy Optimization (PPO) [46] , as shown in Alg. 1. Particularly, each value network V ϕi (•) is trained independently for a specific reward group r i with temporal difference loss,EQUATION)where Êt is the empirical average and γ is the discount factor. The value functions calculated by each critic are used to individually estimate the advantage { Âi } n i=0 for each reward group. Subsequently, these advantages are synthesized into a policy improvement step by calculating the multi-critic advantage as a weighted sum of the normalized advantages from each reward groupEQUATIONwhere µ Âi and σ Âi are the batch mean and standard deviation of the advantage from group i. Similar to PPO, the surrogate loss for policy gradient is clippedEQUATIONwhere α t (θ) and ϵ respectively denote the probability ratio and the clipping hyperparameter. This formulation integrates feedback from both dense and sparse rewards into the policy update, facilitating a balanced and effective learning process. Rollout policy π θ to fill the buffer.for each mini-batch do 5:Estimate Âi for each r i .Compute ÂMuC with Eq. 2.Update policy with Eq. 3.Update each critic with Eq. 1.end for 10: end for Assigning distinct critics for dense and sparse rewards helps achieve each set of objectives more effectively while reducing the reliance on extensive hyperparameter tuning. To illustrate this, consider a simple scenario with an episode length of T involving two types of rewards: a temporally dense reward r d that is active at every step and a temporally sparse reward r s that is only active at the final step of an episode r s,t = rs , t = T 0, otherwise.In the conventional single-critic RL, the total reward of each time step t is typically computed as a linear combination of different reward terms r t = w s r s,t + w d r d,t . The value in this scenario isEQUATIONWe define the reward sparsity ratio as the number of dense reward steps per sparse reward horizon, which is here equal to T . The second term in Eq. 5 consists of a summation over T -t individual reward terms, whereas the first term includes only a single component. This highlights the impact of different reward sparsities on the learning process, suggesting that the weight of reward groups must be adjusted for different sparsity ratios to achieve a proper balance. This challenge is amplified when the sparsity ratio changes between episodes, for example, when keyframe timings are randomly sampled within a range. These variations can complicate the hyperparameter tuning process and hinder the efficacy of the learning algorithm.In the multi-critic approach, on the other hand, the advantage for each reward group is normalized independently, ensuring that a fixed weight ratio for the advantages is adequate to maintain the desired balance, regardless of variations in the sparsity ratio. This method decouples reward frequency and magnitude from the learning process, enabling more effective policy optimization and reducing the effort for manual hyperparameter tuning.The transformer framework [51] has achieved great success in modeling sequential data not only in the natural language processing [52, 53] but also in other areas including robotics [54] . The attention mechanism, serving as the core of transformer networks, models the correlation between each element of the input sequence and reweights them accordingly. To handle a variable number of keyframes in our problem, we utilize a transformer-based encoder to process the sequence of goals for both the policy and critics. However, unlike the typical application of transformers in sequenceto-sequence tasks, we adapt the architecture to function in a sequence-to-token manner, as shown in Fig. 3 . This adaptation makes it suitable for autoregressive feedback control in robotic systems.Figure 3 : Policy with transformer-based keyframe encoder.In our system, each input token x i t is a vector corresponding to a particular keyframe. At every time step t, each keyframe k i is transformed spatially and temporally into a robot-centric view, resulting in a goal error ∆g i t and a calculated time to goal ti -t. These are then concatenated with the robot state s t to form a single token. Additionally, we incorporate a self-goal keyframe, x 0 t , as the first token in the sequence. This token represents a state with zero error and zero time to goal, which ensures that the control system remains operational despite the absence of active goals or after achieving all goals. The transformer encoder receives the sequence of tokens as a matrix X t = [x 0 t ; . . . ; x n k t ], where x 0 t = [s t , 0, 0], and x i t = [s t , ∆g i t , ti -t] for i = 1, . . . , n k . In scenarios where the number of active keyframes is less than the maximum capacity of the system, we apply masking to ignore the surplus tokens and focus only on the relevant keyframes. Furthermore, we also apply masking to keyframes once their designated time is reached and surpassed by a few steps. This practice prevents past goals from inappropriately influencing the long-term behavior of the policy. The output from the transformer encoder is then forwarded to a max-pooling layer, which condenses the encoded goal features for delivery to the subsequent multilayer perceptrons (MLP). By leveraging transformer's ability to handle sequences of varying lengths, our architecture can effectively integrate multiple and variable numbers of goals into the control process. 
results: 
 The control policies are trained for quadruped robots with 12 degrees of freedom (DoF) using Isaac Gym [55] . At the start of each episode, the robot is either set to a default state or initialized according to a posture and height sampled from the dataset, a technique known as Reference State Initialization (RSI) [56] . We incorporate a learning curriculum, beginning with keyframes entirely sourced from reference data and progressively increasing the proportion of randomly generated keyframes, with time intervals, position targets, and yaw angles each sampled from a predetermined range. In this section, we present the qualitative and quantitative experiment results in simulation and on hardware.We demonstrate that our trained policy effectively reaches keyframes at the designated times through several simulation experiments. Given keyframes consisting of position goals, our policy reaches its targets with notable precision, as illustrated in Fig. 4a by the horizontal trajectories for two example scenarios with different number of keyframes. Furthermore, our framework offers control over target reaching time and can generate diverse behaviors for the same targets by specifying different time profiles. This is depicted in Fig. 4b through snapshots of robot motion when provided with keyframes consisting of the same position goal, but different target times. Full posture targets are also supported along with position and orientation goals. Fig. 5 shows snapshots of the robot motion given different keyframe scenarios, highlighting that our policy accurately meets its full posture targets while maintaining a natural style. Further quantitative evaluation of keyframe matching is provided in Appendix A.6.In this section, we conduct a comparative analysis between multi-critic and single-critic approaches in the keyframing setup. Learning curves for both methods are presented in Fig. 6 , with each method trained across three different ranges of sparsity ratios by sampling keyframes with varying time The learning curves reveal that the multi-critic algorithm achieves a similarly fast convergence without retuning the advantage weights for different scenarios. In contrast, the single-critic method displays significant delays in reward increase due to the sparser nature with longer keyframe horizons, underscoring the efficiency of the multi-critic in reducing the need for extensive manual hyperparameter tuning. This feature makes multi-critic particularly valuable in environments with varying reward sparsities.An advantage of using a transformer-based encoder is that it enables the policy to incorporate multiple and a varying number of goals as input. If the goals are temporally close to each other, awareness of future goals influences the robot's motion to achieve all of them more accurately. The phenomenon of future goal anticipation is demonstrated in Fig. 7 where we compare a policy aware of all goals and a policy only aware of the immediate next goal, both trained with only position goals in the keyframe. The policy trained with multiple keyframes adopts a larger yaw angle at the first goal, leaning more towards the second one to be able to reach it with higher accuracy. Table 1 provides a quantitative comparison of the two policies across three different scenarios: straight, turn and slow turn, the latter featuring a longer time horizon for the second goal. The results indicate that future goal anticipation helps the policy to adjust its motion while approaching earlier goals to gain better accuracy for the subsequent targets. This is particularly important when keyframes are temporally close, resulting in higher accuracy gains in fast and dynamic movements, compared to slower ones.We validate our method through extensive hardware experiments using the Unitree Go2 [57] , a 12-DoF commercial quadruped robot. Fig. 8 illustrates the outcomes of a policy that manages up to 5 positional goals arranged in different courses, and a policy trained for full pose targets that successfully drives the robot to achieve various posture keyframes. These experiments underscore the adaptability and effectiveness of our keyframing approach in enhancing high-level control in robotic systems. Readers are encouraged to watch the videos provided in the supplementary material for a more comprehensive presentation of these results. 
discussion: 
 Conclusion: This paper presents RobotKeyframing, a learning-based control framework designed to incorporate high-level objectives into the natural locomotion of legged robots through a sequence of keyframes, leveraging transformer models and multi-critic reinforcement learning. Through simulation and hardware experiments, we demonstrated the effectiveness of our approach: The sparse reward imposed by keyframe objectives is effectively handled by a multi-critic PPO algorithm. In addition, the transformer-based architecture is adaptive to various number of target keyframes and improves accuracy in reaching targets through future goal anticipation.Limitations and future work: First, if the timing values are infeasible for the specified goals, the robot may fail to meet the targets. However, it is worth noting that such cases do not result in uncontrolled behaviors, such as falling down. Second, our approach inherits the mode collapse issue from the AMP framework [12] , which can be mitigated in future research through the integration of style embeddings. Third, the performance of our policy is currently limited by the motions in the dataset, restricting its ability to generalize to out of distribution motions or targets. Looking ahead, our method can be expanded to incorporate diverse types of goals in the keyframes, such as end-effector targets or more intuitive high-level inputs such as skill or text. Additionally, RobotKeyframing can be extended to more complex characters and potentially used for physics-based motion in-betweening in the character animation domain.Table A2 : Goal reward terms Goal positionΦ i (K(p -pi , 0.2, 0)) Goal roll Φ i (K(ϕ -φi , 0.1, 0)) Goal pitch Φ i (K(ζ -ζi , 0.1, 0)) Goal yaw Φ i (K(ψ -ψi , 0.3, 0)) Goal posture Φ i (K(θj -θi j , 0.2 √ 12, 0))Goal rewards are defined with a temporally sparse kernel Φ i (x)EQUATIONand only activated when the corresponding timestep for that goal ti is reached in the episode. The detailed reward terms are defined in table A2 .For advantage weights (Eq. 2), we recommend starting with a 1:1 ratio and making minor adjustments based on specific preferences. In our experiments, we used ω style = 0.5, ω goal = 0.5, ω reg = 0.1, maintaining equal weights for goal and style, with a lower weight for the regularization critic.We use a database of motion capture from dogs introduced by Zhang et al. [44] . The motions are retargeted to the robot skeleton using inverse kinematics for the end-effectors' positions with some local offsets to compensate for the different proportions of the robot and dog. A subset of around 20 minutes of data was used, removing the undesired motions such as smelling the ground, walking on slopes, etc. We augment this dataset with other motion clips animated by artists to include more diversity in the dataset. The frame rate is adjusted to that of the simulation, i.e. 50 frames per second. In addition, the linear and angular velocity of the base are precomputed for each frame using a finite difference method and stored in the dataset.We utilize Isaac Gym [55] for simulating the physical environment. At the start of each episode, the robot is initialized either to a default state (20% probability) or using Reference State Initialization (RSI) (80% probability). For RSI, a random frame from the reference motion trajectory provides the height, orientation, joint angles, base linear velocity, base angular velocity, and joint velocities for initializing the robot. RSI plays a crucial role in capturing and learning the specific style of motion, as highlighted in previous studies such as Peng et al. [56] .We employ two keyframe sampling strategies: dataset-based and random sampling. In the datasetbased approach, a motion trajectory and starting frame index are initially sampled. Then the time interval between keyframes are uniformly drawn from a specified range. The corresponding frames from the dataset are used to obtain the global position, orientation, and posture of the robot at the keyframe times. The global horizontal positions are subsequently converted to relative local positions with respect to the starting frame and then transformed into positions in the simulator's world coordinates by accounting for the robot's initial state. The yaw angle is similarly adjusted, while height, roll, pitch, and joint postures are directly used as target values.In the random sampling strategy, time intervals are sampled from a predefined range, similar to the first strategy. The radius and direction of each keyframe, relative to the previous one are uniformly sampled to compute the keyframe's global horizontal position. Additionally, the change in yaw angle is sampled and added to the previous keyframe's yaw. The roll and pitch angles, joint postures and height are taken directly from a reference frame randomly selected from the dataset. This ensures that the posture is feasible and compatible with the height. Details of the sampling ranges are provided in Table A3 .8th Conference on Robot Learning (CoRL 2024), Munich, Germany. 
 
 Appendix:  


Decision: The paper presents a keyframe-based learning framework for legged locomotion. Reviewer aL5k is excited about the significance of the proposed work, along with its originality. On the other hand, Reviewer i8Sh questioned that the proposed multi-critic framework is not directly related to the proposed concept. Reviewer fZ8e raised concerns about the reproducibility and hardware evaluation. We would like to encourage the authors to respond to all the reviewer's comments.

The authors provided thorough feedback on the raised concerns, and the reviewers seemed satisfied with the responses. Particularly, the authors communicated well with Reviewers i8Sh and fZ8e. To this end, we recommend accepting this manuscript.

Reviews: review: title: title: Clear paper, strong contribution
paper_summary: paper_summary: To implement keyframe-based control of quadrupeds, the paper proposes to train multiple critics to separately handle sparse and dense rewards. A transformer-based keyframe encoder is used. The method is shown to work better compared to having a single critic or a policy only aware of the next goal (instead of anticipating future goals). Great explanatory video is provided.
main_review: main_review: Quality

High-quality, clean, well-written paper, with clear and well-presented idea. The authors have employed rigorous experimental methods and provided substantial evidence to support their claims. The use of both simulation and hardware experiments enhances the robustness of the findings. The inclusion of detailed figures and tables, such as the learning curves for single- and multi-critic approaches, as well as the quantitative comparison of goal anticipation policies, aids in comprehending the results effectively.

Clarity

The paper is written with a high degree of clarity. Each section logically flows into the next, providing a coherent narrative from the introduction of the problem to the presentation of results and discussions. The explanations of complex concepts, such as the transformer-based encoder and the multi-critic reinforcement learning (RL) approach, are articulated well.

Originality

The originality of this work is notable. The application of transformer-based models to handle sequences of keyframes in robotic control is innovative and demonstrates a creative extension of techniques traditionally used in natural language processing. Furthermore, the multi-critic RL approach to handle varying reward sparsities presents a novel contribution to the field of reinforcement learning and robotic control.

Significance

The significance of this work is high. It addresses key challenges in the control of quadruped robots, particularly in dynamic environments with varying keyframe targets. The demonstrated improvements in policy learning efficiency and goal accuracy can have substantial implications for the development of more autonomous and adaptable robotic systems. The successful deployment of these methods on hardware, such as the Unitree Go2 robot, underscores the practical relevance of the research.
questions: questions: Reward Weights and Sparsity:
- How sensitive is the performance of the proposed approach to the choice of weights  w_s  and  w_d  in the reward function? What guidelines can you provide for tuning these parameters?

Scalability and Generalization:
- How well do you expect the proposed methods to scale to different types of robots or environments? Have you considered testing the framework on robots with significantly different morphologies or in more complex environments?

Data Efficiency:
- The paper mentions the efficiency of policy learning. Can you provide more details on the data efficiency of your approach compared to other state-of-the-art methods?

Public Availability:
- Is there any plan to make the code and datasets used in this research publicly available? This would greatly benefit the research community and facilitate further advancements in the field.
limitations: limitations: 3
review_summary: review_summary: Solid clean paper with a clear straightforward contribution and strong demonstration on real hardware
title: title: Good execution for a new task/framework
paper_summary: paper_summary: This paper presents a method for learning a “keyframe” policy that can satisfy a sequence of keyframe poses. This is done through a transformer based architecture to handle a variable number of goal keyframes as input, along with a multi-critic RL process to more evenly weight dense vs. the sparse keyframe matching rewards. There is also flexibility in what the keyframes specify, allowing for any full or partial combination of base pose and joint angles. Simulation results are shown tracking both position waypoint goals along with full posture key frames for walking, jumping, and paw raising motions. Ablation studies vs. a single critic method are presented, showing the robustness of using multiple critics with varying reward sparsities. Finally hardware results are shown of the system running on a Unitree Go2 quadruped doing locomotion and posture matching tasks.
main_review: main_review: Overall I think the work is fairly strong. The novelty is relatively limited and is more of a system putting together different previous researched techniques (with some modifications), but I think is moving towards a good direction for pose matching. I usually see works more along the lines of either matching whole trajectories or “style” based methods, but a keyframe based approach as shown here so that behaviors can be specified in an animation like style is more convenient than whole trajectories but more precise than style methods. This work is a good first step towards this, and if more improvements can be made in the future I think this can be a useful framework for control.

## Strengths:

The proposed method can achieve (at least visually) very good matching of keyframes, even for those more difficult ones when the robot is not in contact with the ground like jumping. This can allow for very easy specification of detailed behaviors. Being able to command behaviors through an animation like keyframing can be very useful. The system is a very sound combination of techniques and has some good ideas. 

I particularly like the separation of sparse vs. dense rewards, which is a good application of multi objective/critic techniques and I think is very helpful. The simple example given at the end of section 3.2 is a good illustration of the need for something like this when there are rewards that can only be achieved a few times per episode.

The text and method are for the most part clear and understandable. Explanation details are mostly thorough and go into the sufficient greater detail of the training in the appendix.


The technical details of the methods and analysis of the algorithm seem to be all correct to my understanding.


## Limitations:

The overall novelty of the proposed method is a little limited. While I don’t think they have been put together for this specific keyframe matching task, and have slight variations, the basic idea of each proposed technique has been done before. 
The multi-ciritic paradigm is just multi-objective RL and the method used here is pretty much the same as those used in the works that you reference (48, 49, 50), just that you have a “special” choice of what your reward groups are (dense vs. sparse).
For example, Radosavovic et. al [1] also use a sequence-to-token transformer architecture for their policy (current and previous observations are the tokens in their case rather than goals) in an autoregressive manner.
I’m not really sure if both of these points are really novel and thus if point ii (line 55 in the text) is really a contribution. I think the contribution here is really the system as a whole used for this keyframe matching task, i.e. contribution i.

I would like to see some evaluation of the actual keyframe matching. The only data I see for the goal error is Table 1, and this seems to be for just a body position keyframe. I would like to see how accurately the learned policy can match given full pose keyframes like those shown in Figure 5. Another table similar to Table 1, but instead reports average pose error (maybe need to do some scaling so joint angles and body pose/orientation are on the same scale?) for the keyframe scenarios in Figure 5 would be good to add. I think this is pretty important since keyframe matching is the main purpose and goal of your system.

I would maybe consider changing the title (if possible) to not include “locomotion”, since I think that locomotion is not really a part of your goal and the purpose of the proposed method is for matching keyframes instead. Of course you can keyframe a locomotion gait, but I think this is probably not a very convenient way to specify locomotion behaviors and the results are less performant, robust, and natural looking than all of the previous works (which you reference too, [5, 7, 10-12, 14, 15, etc.]). The main purpose and novel part that you are proposing is more general in that you can match arbitrary poses.

The limitation that is mentioned in Section 5 that the keyframe poses is limited to motions in the dataset is a pretty large limitation. As it is right now, I don’t think this system has an actual use case over just AMP. However I do understand that this is a first work for this method, and that future works can have improvements and perhaps have more generalization that would put this above trajectory imitation works like AMP. 

Video comments:
Can you use some sort of texture for the floor in the simulation results? The pure white floor makes it hard to judge where the floor actually is and when the feet are actually in contact with it or not. 


[1] Ilija Radosavovic et al. ,Real-world humanoid locomotion with reinforcement learning. Sci. Robot. 9,eadi9579(2024). DOI:10.1126/scirobotics.adi9579
questions: questions: I don’t fully get the use of a transformer-based architecture since it can’t really handle arbitrary number of keyframes as used here. There is still a fixed maximum number of keyframe goals it can handle and masking is still required. How important is the self attention for this system? If you need to have masking anyway, why can you not just individually encode these keyframes and pass them directly into the MLP? Why not have some positional encoding of each keyframe so that the policy can handle any number of keyframe goal inputs?

Why does the state need to be part of goal tokens X_t? It seems like the state is the same across each goal token, and the state is already inputted to the MLP layers. What is the intuition for why the state should influence which goal to pay attention to? How does the policy perform if the state is not part of the goal tokens?

Since there is the self-goal keyframe which has 0 error and 0 time to goal, how does the policy differentiate between this self-goal keyframe and a masked keyframe (which I assume is also just all zeros)? I guess in the self-goal case the state will not masked, is this the purpose of including the state in the goal keyframes? 

You mention that past goal keyframes are masked as well, are they just masked in their current place or are the next goal keyframes “moved up”? For example, if at the start the goal tokens are inputted like $(x^0_t, x^1_t, …, x^{n_k}_t)$ then some time later (let’s say at $t = w$) after 2 goals have been reached are goal tokens like $(0, 0, x^2_w, …, x^{n_k}_w)$ or would it be like $(x^2_w, …, x^{n_k}_w, 0, 0)$? The text sounds like it does the first option, but I think the second option makes more sense? Along the same line, is the self-goal keyframe ever masked? Or will it always be a part of the input?

How important is being aware of all next goals rather than just the next two? The evaluation provided (Table 1) only tests when there are 2 keyframes, not when there are more than 2. I can understand that being aware of more than just the next goal is useful so that the policy can prepare (as Table 1 shows), but how far in the future is it necessary to look ahead? If for example there are 3 keyframe goals, is it necessary to know all 3 or can they just be given two at a time (1st and the 2nd, and then the 2nd and 3rd after the 1st goal is past). I think this is an important evaluation to do, since it answers if handling a variable number of keyframes (a large part of your method) is even necessary or just a small fixed lookahead works for all cases. 

What are the weightings ($w_i$) to used to weight the sparse vs. dense advantages? How is this hyperparameter tuned?
limitations: limitations: 3
review_summary: review_summary: The novelty is a bit limited but it is moving towards a good direction. I think that this keyframe approach can have some advantages over the style based methods that are more commonly used today. Full generalization has yet to be shown in this work but it is a first step towards some thing that could do general pose matching, which would be pretty powerful.
title: title: Review
paper_summary: paper_summary: The paper presents a reinforcement learning pipeline for learning for learning quadruped locomotion policies in simulation and demonstrates transfer to a Unitree Go2. Whereas prior methods are often restricted to velocity commands, the proposed method can set a variable number of keyframes consisting of timestamps, target poses and joint configurations. The keyframe reward, which is sparse as it is only given at the respective timestamps, is complemented by an AMP style reward obtained from the popular dog motion capture dataset by Zhang et al. 2018. The main contribution is the use of a multi-critic approach, that rather than trading-off the different reward terms (style reward and keyframe reward) independently estimates the advantage functions for the these rewards, normalizes them, and weights them for trading-off. While this multi-critic approach is not novel, the idea is relatively novel, hasn't been applied to locomotion and seems interesting, because in contrast to reward-weights, the weights on the advantage functions work well for different degrees of sparsity (timegaps between keyframes). The paper presents nice empirical results, as the quadruped performs natural locomotion while reaching the desired keyframes at the request timestamps.
main_review: main_review: While the technical contribution is quite limited, the proposed multi-critic approach could potentially have impact by allowing to learn policies that follow more expressive commands, which is important for hierarchical control.	

The paper is well-written and easy to follow. However, the pipeline is not described in sufficient detail. In its current form, the paper is not reproducible. Providing source code would be very helpful for filling the blanks. Please refer to my questions below for examples. In particular, I am wondering how the keyframes are tokenized and sampled and how exactly the initial states for the RSI are sampled.

The main strength of the paper are the good empirical results. However, the paper does not provide quantitative results on the hardware, but only shows some videos of successful attempts few (I assume to some extend tuned) keyframe-scenarios. The paper would be much stronger, when using held-out demonstrations (ideally from a different dataset) to sample keyframes and quantitatively evaluating their reward in simulation and on the robot.
questions: questions: How exactly are the keyframe tokens passed to the transformer? A token typically refers to a single integer, however, I assume that the state, relative goal, and time-delta are not discretized into a single integer. 

The appendix provides some information on sampling the keyframes. However, I think it would be useful to describe the process in more detail.

It would also be useful to describe the sampling of the initial state (with RSI) in more detail, e.g. by providing code or pseudo-code.

Appendix states that the "previous actions" are part of the state. Does it refer to the actions of the last time step, or a longer history of actions?

I assume the loss in Eq. 1 uses a target network. It would be could to indicate this in the equation.

What are the "time steps" in the x-axis of Fig. 6? Are these RL iterations or is there missing some factor to bring them to the right order of magnitude?

According to the appendix, the RL policy runs at 50Hz. What are the control frequencies of the PD controller? 

The joint soft limits (Table A1) seem to bias the joint position towards the middle position, $1/2 (\boldsymbol{\theta}\_{j,min} - \boldsymbol{\theta}\_{j,max})$. Does this joint configuration correspond to a reasonable nominal posture for the robot?

How often is the discriminator updated during RL?
limitations: limitations: 3
review_summary: review_summary: Good work with sufficient relevance to the field but insufficient description of the methodology.