RobotKeyframing: Learning Locomotion with High-Level Objectives via Mixture of Dense and Sparse Rewards
Authors: Fatemeh Zargarbashi
Jin Cheng
Dongho Kang
Robert Sumner
Stelian Coros
Venue: CoRL 2024 (2024)
Keywords: Legged robots
Multi-Critic Reinforcement Learning
Motion Imitation
TL;DR: This paper presents a novel learning-based control framework that incorporates high-level objectives for the natural locomotion of legged robots through keyframing.

Abstract: This paper presents a novel learning-based control framework that uses keyframing to incorporate high-level objectives in natural locomotion for legged robots. These high-level objectives are specified as a variable number of partial or complete pose targets that are spaced arbitrarily in time. Our proposed framework utilizes a multi-critic reinforcement learning algorithm to effectively handle the mixture of dense and sparse rewards. Additionally, it employs a transformer-based encoder to accommodate a variable number of input targets, each associated with specific time-to-arrivals. Throughout simulation and hardware experiments, we demonstrate that our framework can effectively satisfy the target keyframe sequence at the required times. In the experiments, the multi-critic method significantly reduces the effort of hyperparameter tuning compared to the standard single-critic alternative. Moreover, the proposed transformer-based architecture enables robots to anticipate future goals, which results in quantitative improvements in their ability to reach their targets.

RobotKeyframing: Learning Locomotion with High-Level Objectives via Mixture of Dense and Sparse Rewards
Fatemeh Zargarbashi Jin Cheng'
Dongho Kang'
Robert Sumner² "DisneyRescarchStuiis Switzerland
Stelian Coros'
'ETH Zürich, Switzerland aaemmch.zargabbashh inf.ethz.ch
Abstract: This paper presents novel learning-based control framework that uses keyframing to incorporate high-level objectives in natural locomotion for legged robots. These high-level objectives are specified as a variable number of partial complete pose targets that are spaced arbitrarily in time. Our proposed frame- work utilizes a multi-critic reinforcement learning algorithm to effectively handle the mixture of dense and sparse rewards. Additionally, it employs a transformer- based encoder to accommodate variable number of input targets, each associated with specific time-to-arrivals. Throughout simulation and hardware experiments. we demonstrate that our framework can effectively satisfy the target keyframe sequence at the required times. In the experiments, the multi-critie method sig- nificantly reduces the effort of hyperparameter tuning compared to the standard single-critic alternative. Moreover, the proposed transformer- architecture enables robots to anticipate future goals, which results quantitative improve- ments in their ability to reach their targets.
Project website: https: //sitessttee/etteee//seeeeeeeess /tt.sss.ooooosee.....//oooooooee www.segoooggo.ooooooo..o.o..o.o. http:/iiee/roott-keyttee
Keywords: Legged Robots, Multi-Critic Reinforcement Learning, Transformer, Motion Imitation
Go2
202
GoZ
Figure 1: RobotKeyframing: Locomotion policy trained with framework meets the keyframes with position and full posture targets (yellow) at specified times on hardware experiments.
1 Introduction
Legged robots hold great promise for becoming household companions [1] or automated perform- ers in the entertainment industry [2, 3]. In these applications, is crucial for robot controllers to perform natural and directable hehavior from simple high-level user command inputs beyond the typical commands used in the robotic domain such as joystick velocity commands [4, 5] or target base position [6, 7].
In the character animation domain, widely used technique for specifying character behavior from simple and sparse inputs is keyframing [8, 9]. It involves defining the target position or kinematic
8th Conference Robot Learning (CoRL. 2024), Munich, Germany.

---

pose of the character at particular points in time, allowing animators to create smooth movements by interpolating between these keyframes. Despite its proven effectiveness within the kinematic animation pipeline, incorporating keyframing for achieving time-specific targets remains unexplored in the realm of physics-based robot control.
With inspiration from the character animation technique, we aim equip legged robots with more refined control by incorporating sparse and temporal high-level objectives as keyframes. The pri- mary goal of this work is to develop a locomotion controller that enables the robot to fulfill specified partial or full-pose targets while infilling natural behavior during the intermediate periods. This goal aligns with recent advancements in using reinforcement learning (RL) for legged robots due to their promising robustness and flexibility [10, 11]. However, learning a policy that accurately meets keyframes without imposing undesired constraints intermediate periods presents challenges, par- ticularly due to the need to handle sparsity in the keyframe objectives. Acquiring effective policies requires meticulous reward design procedure that carefully balances these sparse rewards with other dense rewards which are crucial for regularizing and encouraging natural motion.
In this work, we present a novel framework that unifies timed high-level objectives with natural lo- comotion of legged robots through temporal keyframes. Along with the imitation objective similar to Peng al. [12] for natural motion generation, our pipeline allows specifying full partial high- level targets, including base position, orientation, and joint postures. We propose using multi-critic RL framework to address the challenge of managing groups of sparse and dense rewards by learning distinct value functions. Our method also employs novel transformer based architecture to encode a variable number of goals with arbitrary time intervals. Unlike typical
torennce---seeeecce trans- formers [13], we propose lightweight sequence-to-token module that can be used autoregressively within a feedback control loop. We demonstrate the effectiveness of our framework through exper- iments both in simulation and real-world hardware. Our policies successfully guide the robot to meet multiple keyframes at the required times, for both position and posture targets. Furthermore, the multi-critic approach showcases better convergence with less hyperparameter tuning compared to the conventional single-critic method. Our experiments also reveals that using a transformer-basee encoder to anticipate future goals significantly enhances goal-reaching accuracy.
The contribution of this paper is threefold: (i) We introduce RobotKeyframing, a novel learning- based framework for integrating high-level objectives in natural locomotion of legged robots; (ii) We propose using multi-critic RL to effectively handle the mixture of dense and sparse rewards, along with sequence- to-token encoder to accommodate variable number of keyframes; (iii) We validate the effectiveness of our method through extensive experiments in simulation and on hardware.
2 Related Work
2.1 Reinforcement Learning for Legged Robots
Over the last decade, reinforcement learning has been increasingly applied to develop locomotion policies for legged robots [4, 14, 15]. The primary focus has been to achieve robust control policies that can accurately track velocity commands from joysticks [11, 16, 17]. More recently, researchers have attempted to enhance the versatility of legged robot controllers by incorporating high-level ob- jectives, particularly through position- or orientation-baser targets [18, 19, 20]. This high-level con- trol is typically accomplished through hierarchical frameworks, where a high-level policy is learned to drive a low-level controller [7, 21, 22). Conversely, end-to-end approaches aim to develop uni- fied policy for both high- and low-level control, allowing high-level objectives to directly influence low-level decisions [6, 18, 23]. However, the aforementioned methods typically urge the robot to reach a target as fast as possible. lacking refined control over the temporal profile of achieving the target. Inspired by keyframing in animation, this work aims to further expand control over robot motion by incorporating multiple keyframes as input to the control policy, thereby enabling robots to generate diverse behaviors in reaching targets. We further enhance this versatility by allowing partial or full targets, including base position, orientation, and joint postures.

---

2.2 Natural Motion for Characters and Robots
Synthesizing naturalistic behavior from existing motion datasets while fulfilling spatial or temporal conditions has been extensively studied in the character animation domain [24, 25, 26, 27]. Existing research for generating natural motions between keyframes [28, 29, 30] has mainly focused on the kinematic properties of characters and thus cannot be directly applied to physics- based characters or robots, whose dynamic interactions with the environment require consideration of both kinemat- ics and dynamics. Various efforts have also been made to combine kinematic motion generation with physically controlled robots to achieve natural behavior on hardware [31, 32, 33, 34]. Another thread of research focuses controlling characters in physically simulated environments, incorpo- rating motion datasets as demonstrations [35, 36, 37, 38). Some of these methods have also been successfully transferred to robot control for quadrupeds or humanoids [39, 40, 41, 42]. Among these works, Adversarial Motion Priors (AMP) [12] provides a ficxible way to encourage the pol- icy to have natural, expert-like behavior by connecting generative adversarial networks (GAN) [43] with RL given offline motion dataset. We also incorporate an AMP-based imitation objective to encourage naturalistic motion for the policy and further extend it to infilling keyframes for robots.
3 Method
3.1 Problem Setup
To integrate high-level control objectives into the robotic control framework, we employ sparse keyframes that require a robot to achieve specific goals at predetermined times. Each keyframe con- tains a full or partial combination of variety of targets such as global base position P E R global base orientation (ő, () E R$ where ④. C, denote roll, pitch, and yaw angles respectively, and full posture specified by joint angles 0, E IR,; where Nj is the number of joints. Each keyframe is also assigned with a specific time f E IR in the future at which the robot is expected to meet the goals. In summary, the high-level objectives are specified through these keyframes K (k*, .k2.....k''' k**)
where k' = (g. b)² and 9 c { p. S. &. 0,)) Here, ny I NN where n2 and N₂ denote the ac- tual and the maximal number of keyframes, respectively. We aim support arbitrary number of keyframes, allowing for the flexible specification of high-level objectives only as needed.
The main goal is to train a locomotion policy for legged robots that not only meets these keyframes but also maintains a natural style during the intervals between them. To avoid undesired restrictions on the intermediate periods, policy's 's task performance is evaluated exclusively at the designated times, making the keyframe objectives temporally sparse. However, relying solely on keyframes to train the control policy may result in undesirable motions. Thus, it is crucial to have additional re- wards for regularizing and promoting natural motion style. In this regard, we incorporate AMP [12] as a general style guide for the robot, encouraging the policy to behave naturally and similarly to an offline motion dataset from dogs [44]. The style and regularization rewards are evaluated at ev- cry step of the episode, making them temporally dense. The mixture of sparse and dense rewards presents unique challenge that is difficult to manage effectively with standard RL frameworks. De- tails on the observation, action, reward definitions, and training setup can be found in Appendix A.
3.2 Multi-Critic RL for Dense-Sparse Reward Mixture
Modern RL algorithms [45, 46, 47] typically employ the actor-critic paradigm, where the actor decides the action to take, and the critic evaluates the action by estimating the value function. To effectively manage a complex mixture of temporally dense and sparse rewards, we employ a multi- critic (MuC) RL framework by Martinez-Piazuue a set of critic networks (Vo, to learn distinct value functions associated with different reward groups {ra) Similar concepts have been used to balance set of dense rewards [49, 50]; however, extend the multi-critic method to the context of dense and sparse reward combination, where demonstrate its distinct advantages. We design each reward group to contain either exclusively
et al. [48] as shown in Fig. 2. It involves training
3

---

dense or sparse rewards. This division is essential for effectively managing the distinct temporal characteristics of each reward type and facilitates value estimation.
We integrate the multi-critic concept Proximal Policy Optimization (PPO) [46], as shown in Alg. 1. Particularly, each value network (+) is trained independently for specific reward group Ti with temporal difference loss,
[(de) Te, [IIrA + Vo. Vo, (s.
where E, is the empirical average and is the discount factor. The value functions calculated by each critic are used to individually estimate the advantage {A.) for each reward group. Subsequently, these advantages are synthesized into policy improvement step by calculating the multi-critic ad- vantage as weighted sum of the normalized advantages from each reward group
a
A,
MA AA
AMuC Σ
where μí and A are the batch mean and standard deviation of the advantage from group i. Similar PPO, the surrogate loss for policy gradient is clipped
LCLIP-MUC (0) = E, min (ar (or()) clip(a: e, e)Aara,))]
(3)
where ar (0) and e respectively denote the probability ratio and the clipping hyperparameter. This formulation integrates feedback from both dense and sparse rewards into the policy update, facili- tating balanced effective learning process.
Algorithm Multi-Critic PPO
Action
1: Initialize policy parameters and parame- ters of each critic, Φ;- for n 1 to do
Policy (Actor)
Environment
Observation
3: Rollout policy ma to fill the buffer.
for each mini-batch do Estimate A, for each ri-
5:
Policy Optimization
Compute Âmuc with Eq. 2. Update policy with Eq. 3. Update each critic with Eq. 1.
Critics 0....,
end for 10: end for
Figure 2: Multi-Critic RL.
Assigning distinct critics for dense and sparse rewards helps achieve each set of objectives more fectively while reducing the reliance on extensive hyperparameter tuning. To illustrate this, consider simple scenario with episode length of T involving two types of rewards: temporally dense reward that is active at every step and temporally sparse reward r, that is only active at the final step of an episode
Pss t=
Tef = 10
(4)
otherwise.
In the conventional single-critic RL, the total reward of each time step is typically computed as linear combination of different reward terms s,t ward,t The value in this scenario is
T
(sa)) = E way( (T 0F + wa
*
rd,k
(5)
Amnt
We define the reward sparsity ratio as the number of dense reward steps per sparse reward horizon, which is here equal to T. The second term in Eq. consists of summation over T' f individual reward terms, whereas the first term includes only single component. This highlights the impact of different reward sparsities on the learning process, suggesting that the weight of reward groups must

---

be adjusted for different sparsity ratios to achieve proper balance. This challenge is amplified when the sparsity ratio changes between episodes, for example, when keyframe timings are randomly sampled within a range. These variations can complicate the hyperparameter tuning process and hinder the efficacy of the learning algorithm.
In the multi-critic approach, the other hand, the advantage for each reward group is normalized independently, ensuring that fixed weight ratio for the advantages is adequate to maintain the de- sired balance, regardless of variations in the sparsity ratio. This method decouples reward frequency and magnitude from the learning process, enabling more effective policy optimization and reducing the effort for manual hyperparameter tuning.
3.3 Transformer-based Keyframe Encoding
The transformer framework [51] has achieved great success in modeling sequential data not only in the natural language processing [52, 53] but also in other areas including robotics [54]. The attention mechanism, serving as the core of transformer networks, models the correlation between each element the input sequence and reweights them accordingly. To handle a variable number of keyframes in our problem, we utilize a transformer based encoder to process the sequence goals for both the policy and critics. However, unlike the typical application of transformers in sequence- to-sequence tasks, we adapt the architecture function in sequence-to-token manner, shown in Fig. 3. This adaptation makes it suitable for autoregressive feedback control in robotic systems.
se I-goal
goel
goel
2
goal n
state
state
error to goal
time goa
MLP
action
layers
masked mult--hadd sell-attention
pacling
Figure 3: Policy with transformer-basee keyframe encoder.
In our system, each input token ot is a vector corresponding to particular keyframe. At every time step t, each keyframe k' is transformed spatially and temporally into a robot-centric view, resulting in goal error Ag; and a calculated time to goal l t. These are then concatenated with the robot state 8; to form a single token. Additionally, we incorporate a self- keyframe, xf, as the first token in the sequence. This token represents a state with zero error and zero time to goal, which ensures that the control system remains operational despite the absence of active goals or after achieving all goals. The transformer encoder receives the sequence of tokens matrix X, = [zf: x where x! [s,, O, 0), and ni (sc. Ag i (] for i l,.... Tkk-
In scenarios where the number of active keyframes is less than the maximum capacity of the system, we apply masking to ignore the surplus tokens and focus only on the relevant keyframes. Further- more, we also apply masking to keyframes once their designated time reached and surpassed by few steps. This practice prevents past goals from inappropriately influencing the long-term behavior of the policy. The output from the transformer encoder is then forwarded to max- max-pooling layer, which condenses the encoded goal features for delivery to the subsequent multilayer perceptrons (MLP). By leveraging transformer's 's ability to handle sequences of varying lengths, our architecture can effectively integrate multiple and variable numbers goals into the control process.
Results
The control policies are trained for quadruped robots with degrees of freedom (DoF) using Isaac Gym [55]. At the start of each episode, the robot is either set to default state initialized according to posture and height sampled from the dataset, a technique known as Reference State Initialization

---

Zargel dime *2
2.0
<K
1.5
(m)
m00 m111
Torget
0.5
X (m)
(m)
1
(22x
(a)
(b)
Figure 4: a) Horizontal trajectories of the robot base given two sets of position goals (dots). b) Spec- ifying different temporal profiles generates diverse behaviors for the same position goal.
(RSI) [56]. We incorporate learning curriculum, beginning with keyframes entirely sourced from reference data and progressively increasing the proportion of randomly generated keyframes, with time intervals, position targets, and yaw angles each sampled from a predetermined range. In this section, we present the qualitative and quantitative experiment results in simulation and on hardware.
4.1 Keyframe Matching
We demonstrate that trained policy effectively reaches keyframes at the designated times through several simulation experiments. Given keyframes consisting of position goals, policy reaches its targets with notable precision, as illustrated in Fig. 4a by the horizontal trajectories for two example scenarios with different number of keyframes. Furthermore, our framework offers control over target reaching time and can generate diverse behaviors for the same targets by specifying different time profiles. This depicted in Fig. 4b through snapshots of robot motion when provided with keyframes consisting of the same position goal, but different target times. Full posture targets are also supported along with position and orientation goals. Fig. shows snapshots of the robot motion given different keyframe scenarios, highlighting that policy accurately meets its full posture targets while maintaining natural style. Further quantitative evaluation of keyframe matching is provided in Appendix A.6.
4.2 Multi-Critic RL
In this section, we conduct comparative analysis between multi-critic and single-critic approaches in the keyframing setup. Learning curves for both methods are presented in Fig. 6, with each method trained across three different ranges of sparsity ratios by sampling keyframes with varying time
RS
Figure 5: Snapshots the robot motion given keyframes with full postures: moving forward (top), jumping (middle) and raising the paw up (bottom). Target keyframes are displayed in yellow.

---

Single-Critic
Multi-Critic
1.0
Goal reward
0.8
[50-75
[75-100]
Geal reward
0.8
0.6
0.4
0.2
0.2
[25-50]
[00-55
0.0
175-100)
2000
4000
6000
8000
10000
2000
4000
6000
8000
10000
Training iteration
Training iteration
Figure 6: Convergence comparison of single-critic (left) and multi-critic (right) for different ranges of keyframe time horizons ((55, 50), [50, 75]. [75, 100J) with fixed weights.
1l
1=1.5 s
(a)
(m)
/=0x s
F= S
1=1.5 8
y (m)
(b)
x (m)
Figure 7: The policy aware of all goals (a) adjusts its yaw angle earlier to better reach the second goal compared to the policy only aware the next goal (b). Keyframes are placed at and 1.5 seconds in time. Left: snapshots, right: trajectories.
horizons. Initially, reward and advantage weights are tuned separately for single- and multi-critic according the time horizon range [25, 50]. New policies are then trained using the same weights for another two scenarios of time horizons, 50, 75] and [75, 100). The learning curves reveal that the multi-critic algorithm achieves similarly fast convergence without retuning the advantage weights for different scenarios. increase due to the sparser nature with longer keyframe horizons, underscoring the efficiency of the multi-critic in reducing the need for extensive manual hyperparameter tuning. This feature makes multi-critic particularly valuable in environments with varying reward sparsities.
contrast, the single-critic method displays significant delays in reward
4.3 Future Goal Anticipation
An advantage of using transformer-based encoder is that it enables the policy to incorporate multi- ple and a varying number of goals as input. If the goals are temporally close to each other, awareness of future goals influences the robot's motion to achieve all of them more accurately. The phe- nomenon of future goal anticipation is demonstrated in Fig. 7 where we compare a policy aware of all goals and a policy only aware of the immediate next goal, both trained with only position goals in the keyframe. The policy trained with multiple keyframes adopts a larger yaw angle the first goal, leaning more towards the second one be able reach with higher accuracy. Table provides quantitative comparison of the two policies across three different scenarios: straight, turn and slow turn, the latter featuring a longer time horizon for the second goal. The results indicate that future goal anticipation helps the policy to adjust its motion while approaching earlier goals to gain better accuracy for the subsequent targets. This is particularly important when keyframes are temporally close, resulting in higher accuracy gains in fast and dynamic movements, compared to slower ones.
Hardware Deployment
We validate our method through extensive hardware experiments using the Unitree Go2 [57], 12-DoF commercial quadruped robot. Fig. illustrates the outcomes of policy that manages up to 5 positional goals arranged in different courses, and policy trained for full pose targets that

---

First Goal
Straight
Turn
Turn (Slow)
Straight
Aware of goals
0.0872
0.0336
0.0781
0.0236
0.0806 + 0.0265
Aware of next goal
0.0898
0.0317
0.0841 1 0.0335
0.0787 1 0.0208
Second Goal
Straight
Turn
Turn (Slow)
Tum
Aware goals
0.0472 0.0187
0.3340 + 0. 0.1162
0.0566 + 0.0804
Aware of next goal
0.1332 0.0605
0.7271
+
0.1711
Table 1: Average position error (m) for three keyframe scenarios (depicted right) across 20
experiments. The policy aware of all goals achieves better accuracy reaching them.
Figure 8: Hardware deployment of RobotKeyframing for position targets (top), and full-pose targets (bottom). Posture keyframes are displayed in yellow.
successfully drives the robot to achieve various posture keyframes. These experiments underscore the adaptability and effectiveness of our keyframing approach in enhancing high-level control in robotic systems. Readers are encouraged to watch the videos provided in the supplementary material for a more comprehensive presentation of these results.
Discussion
Conclusion: This paper presents RobotKeyframing, a learning- control framework designed incorporate high-level objectives into the natural locomotion legged robots through a sequence of keyframes, leveraging transformer models and multi-critic reinforcement learning. Through sim- ulation and hardware experiments, demonstrated the effectiveness of approach: The sparse reward imposed by keyframe objectives is effectively handled by multi-critic PPO algorithm. In addition, the transformer buuseddmer-bused architecture is adaptive to various number of target keyframes and improves accuracy reaching targets through future goal anticipation.
Limitations and future work: First, if the timing values are infeasible for the specified goals, the robot may fail meet the targets. However, is worth noting that such cases do result in uncon- trolled behaviors, such falling down. Second, our approach inherits the mode collapse issue from the AMP framework [12], which can be mitigated in future research through the integration of style embeddings. Third, the performance of our policy is currently limited by the motions in the dataset, restricting its ability to generalize to out of distribution motions or targets. Looking ahead, our method can expanded to incorporate diverse types of goals in the keyframes, such as end-effector targets or more intuitive high-level inputs such skill or text. Additionally. RobotKeyframing can be extended to complex characters and potentially used for physics-based motion in in-betweening in the character animation domain.

---

Acknowledgments
The authors would like thank Miguel Zamora for valuable discussions. This work was supported by the Furopean Research Council (ERC) under the European Union's Horizon 2020 research and innovation program (grant agreement No. 866480).
References
[1] Sony Electronics. Aibo companion robot. https://electtronccssu www.///mo////eeceeeeeeeeeeoooooa
[2] M. Hopkins, G. Wiedebach, K. Cesare, J. Bishop. E. Knoop, and M. Bächer, In- teractive design of stylized walking gaits for robotic characters. ACM Transactions
on Graphics (ToG), 2024. URL http:///a...aaaeeeeeeearcc...ola
[3] Walt Disney Imagineering, A new approach to disney's robotic character pipeline. https: 2023.
[4] Lee, Hwangho, I.. Wellhausen, Koltun, and M. Hutter. Learning quadrupedal comotion over challenging terrain. Science Robotics, 5(47)xeabc5986, 2020. doi:10.1126/ scirobotics.abc5986.
[5] T. Miki, J. Lee, J. Hwangbo, L. Wellhausen, V. Koltun, and M. Hutter. Learning robust per- ceptive locomotion for quadrupedal robots in the wild. Science Robotics, 7(62))eab,2822, 2022.
[6] N. Rudin, D. Hoeller, M. Bjelonic, and M. Hutter. Advanced skills by learning locomotion and local navigation end-to-end 2022 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 2497-2503. IEEE, 2022.
[7] D. Hoeller, N. Rudin, D. Sako, and M. Hutter. Anymal parkour: Learning agile navigation for quadrupedal robots. Science Robotics, 9(8))eadi7566, 2024.
[8] D. Sturman. Interactive key frame animation of 3-d articulated models. In Graphics Interface, volume 86, 1984.
[9] Thomas and O. Johnston. Disney animation: The illusion of life. Disney Editions, 1981.
[10] Hwangho, J. Lee, A. Dosovitskiy, D. Bellicoso, V. Tsounis, V. Koltun, and M. Hutter. Learn- ing agile and dynamic motor skills for legged robots. Science Robotics, 4(26))caau5872, 2019.
[11] A. Kumar, Z. Fu, D. Pathak, and J. Malik. Rma: Rapid motor adaptation for legged robots. Robotics: Science and Systems, 2021.
[12] X. B. Peng, Z. Ma, Abbeel, S. Levine, and A. Kanazawa. Amp: adversarial motion priors for stylized physics-based character control. ACM Transactions on Graphics (ToG), 40(4), jul 2021. ISSN 0730-0301. doi:10. 045110111////4452664555555
[13] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, N. Gomez, E. Kaiser, and I. Polo- sukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[14] J. Tan, T. Zhang, E. Coumans, A. Iscen, Y. Bai, D. Hafner, S. Bohez, and V. Vanhoucke. Sim- to-real: Learning agile locomotion for quadruped robots. arXiv preprint arXiv: 1XXx::8804113333 2018.
[15] J. Hwangbo, J. Lee, A. Dosovitskiy, D. Bellicoso, V. Tsounis, V. Koltun, and M. Hutter. Learn ing agile and dynamic motor skills for legged robots. Science Robotics, 4(26)xcau58872, 2019.

---

[16] A. Agarwal, A. Kumar, J. Malik, and D. Pathak. Legged locomotion in challenging terrains using egocentric vision. In K. Liu, D. Kulic, and J. Ichnowski, editors, Proceedings of The 6th Conference on Robot Learning, volume 205 of Proceedings of Machine Learning Research, pages 403-415. PMLR, 14-18 Dec 2023.
[17] G. B. Margolis and P. Agrawal. Walk these ways: Tuning robot control for generalization with multiplicity of behavior. In Conference on Robot Learning, pages 22-31. PMLR, 2023.
[18] J. Cheng, M. Vlastelica, P. Kolev, C. Li, and G. Martius. Learning diverse skills for local navigation under cunti-constraint optimality. arXiv preprint arXiv:2310.02440, 2023.
[19] T. Miki, J. Lee, L. Wellhausen, and M. Hutter. Learning to walk in confined spaces using 3d representation. arXiv preprint arXiv:2403.000 00187, 2024.
[20] P. Arm, M. Mittal, H. Kolvenbach, and M. Hutter. Pedipulate: Enabling manipulation skills using a quadruped robot's s leg. arXiv preprini arXiv: 10837, 2024.
[21] J. Truong, D. Yarats, T. Li, F. Meier, S. Chermova, D. Batra, and A. Rai. Learning naviga- tion skills for legged robots with learned robot embeddings. In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 484-491. IEFE, 2021.
[22] J. Lee, M. Bjelonic, A. Reske, L. Wellhausen, T. Miki, and M. Hutter. Learning robust au- tonomous navigation and locomotion for wheeled-legged robots. Science Robotics, 9(9)): eadi9641, 2024.
[23] X. Cheng, K. Shi, A. Agarwal, and D. Pathak. Extreme parkour with legged robots. arXiv preprint arXiv:2309. 14341, 2023.
[24] D. Holden, T. Komura, and J. Saito. Phase-functioned neural networks for character control. ACM Transactions on Graphics (TOG), 36(4):1-13, 2017.
[25] H. Zhang, S. Starke, T. Komura, and J. Saito. Mode-adaptive neural networks for quadruped motion control. ACM Transactions on Graphics (TOG), 37(4):1-11, 2018.
[26] H. Y. Ling, F. Zinno, G. Cheng, and M. Van De Panne. Character controllers using motion vaes. ACM Transactions Graphics (TOG), 3944):0011,002 2020.
[27] G. Tevet, S. Raab, B. Gordon, Y. Shafir, D. Cohen-or, and A. H. Bermano. Human motion diffusion model. In The Eleventh International Conference Learning Representations, 2023. URL. https: : http///ppeeeeeeeeeeeeeeeeeeeeeee
[28] F. G. Harvey, M. Yurick, D. Nowrouzeezahrai, and C. Pal. Robust motion in in-betweening. ACM Transactions Graphics (TOG), 1,
[29] Qin, Y. Zheng, and K. Zhou, Motion in-betweening via two-stage transformers. ACM Transactions Graphics (TOG)., 11 2022.
[30] Starke, S. Starke, T. Komura, and F. Steinicke. Motion in in-beteennig with phase manifolds. Proc. ACM Comput. Graph. Interact. Tech., 6(3), aug 2023. 145/366615///
[31] D. Kang, F. De Vincenti, N. C. Adami, and Coros. Animal motions on legged robots using nonlinear model predictive control. In 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 11955-11962. IEEE, 2022.
[32] D. Kang, Zimmermann, and Coros. Animal gaits on quadrupedal robots using motion matching and mosel-based control. In 202/ IEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pages 8500-8501 IEEE, 2021.
[33] X. Huang, Y. Chi, R. Wang, Z. Li, X. B. Peng, Shao, B. Nikolic, and K. Sreenath. Diffuse- loco: Real-time legged locomotion control with diffusion from offline datasets. arXiv preprint arXiv:2404.199 192644 2024.
10

---

[34] I. Radosavovic, B. Zhang, B. Shi, J. Rajasegaran, S. Kamat, T. Darrell, K. Sreenath, and J. Ma- lik. Humanoid locomotion as next token prediction. arXiv preprint arXiv:2402. /9469, 2024.
[35] X. B. Peng, P. Abbeel, S. Levine, and M. Van de Panne. Deepmimie: Example-guided deep inforcement learning of physics-based character skills. ACM Transactions On Graphics (TOG), 37(4):1-14, 2018.
[36] K. Bergamin, S. Clavet, D. Holden, and J. R. Forbes. Drecon: data-driven responsive control of physics-based characters. ACM Trunsactions Graphics (TOG), 38(6):1-11, 2019.
[37] X. B. Peng, Z. Ma, Abbeel, Levine, and Kanazawa. Amp: Adversarial motion priors for stylized physies-based character control. ACM Transactions on Graphics (TOG), 40(4): 1-20, 2021.
[38] H. Yao, Z. Song, B. Chen, and L. Liu, Controlvae: Model-based learning of generative con- trollers for physics-based characters. ACM Transactions on Graphics (TOG), 41(6):1-16, 2022.
[39] A. Escontrela, X. B. Peng, W. Yu, T. Zhang, A. Iscen, K. Goldberg, and P. Abbeel. Adver- sarial motion priors make good substitutes for complex reward functions. In 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 25-32. IEEE, 2022.
[40] T. Li, Y. Zhang, C. Zhang, Q. Zhu, J. Sheng, W. Chi, C. Zhou, and L. Han. Learning terrain- adaptive locomotion with agile behaviors by imitating animals. In 2023 IEEE/RSJ Interna- tional Conference on Intelligent Robots and Systems (IROS), pages 339-345. IEEE, 2023.
[41] H. Shi, Li, Q Zhu, J. Sheng, L. Han, and M. Q.-H. Meng. An efficient model- approach on learning agile motor skills without reinforcement. arXiv preprint arXiv:2403.01962 2024.
[42] Q. Zhang, P. Cui, D. Yan, J. Sun, Y. Duan, A. Zhang, and R. Xu. Whole-body humanoid robot locomotion with human reference. arXiv preprint arXiv:2402. 18294, 2024.
[43] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial networks. Communications of the ACM, 63(11):139- 144, 2020.
[44] H. Zhang, S. Starke, T. Komura, and J. Saito. Mode adaptivee neural networks for quadruped motion control. ACM Transactions on Graphics (loG), 37(4), jul 2018. ISSN 0730-0301.
[45] Schulman, S. Levine, Abbeel, M. Jordan, and Moritz, Trust region policy optimization. In International conference on machine learning, pages 1889-1897. PMLR, 2015.
[46] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 1707.06347, 2017.
[47] T. Haarmoja, A. Zhou, P. Abbeel, and S. Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International conference on machine learning, pages 1861-1870. PMLR, 2018
[48] J. Martinez-Piazuelo, D. E. Ochoa, N. Quijano, and L. F. Giraldo. A multi-critic reinforcement learning method: An application to multi-tank water systems, JEEE 8: 8:732277-173238, 2020. doi: 10. 109/ACCESS 20t..2020.0022000
[49] S. Mysore, G. Cheng, Y. Zhao, K. Saenko, and M. Wu. Multi-critic actor learning: Teaching rl policies to act with style. In International Conference on Learning Representations, 2021.
[50] Xu, X. Shang, V. Zordan, and Karamouzas. Composite motion learning with task control. ACM Transactions Graphics (TOG), 42(4), jul 2023. ISSN 0730-0301. doi: 10.1145/3592447.

---

[51] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, N. Gomez, L. Kaiser, and I. Polo- sukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[52] H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprini 2rXiv:2302. 13777, 2023.
[53] A. Chowdhery, S. Narang, Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton, S. Gehrmann, et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1-113 2023.
[54] A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, J. Dabis, C. Finn, Gopalakrishnan, K. Haus- man, A. Herzog, Hsu, et al. Rt-1: Robotics transformer for real-world control at scale. arXiv preprint arXiv:2212.066817, 2022.
[55] V. Makoviychuk, L. Wawrzyniak, Guo, M. Lu, K. Storey, M. Macklin, Hoeller, N. Rudin, A. Allshire, A. Handa, et Isaac gym: High performance gpu-based physics simulation for robot learning. arXiv preprint arXiv:2108. 10470, 2021.
[56] X. B. Peng, P. Abbeel, S. Levine, and M. Van de Panne. Deepmimic: Example-guided deep re- inforcement learning of physics-based character skills, ACM Trunsactions on Graphics (TOG), 37(4):1-14, 2018.
[57] Unitree Robotics. attps://vne www.:/uiitreeecoooooeeeo
[58] D. Kang, J. Cheng, M. Zamora, Zargarbashi, and S. Coros. RI+ model-based control: Using on-demand optimal control to learn versatile legged locomotion. IEEE Robotics and Automa- tion Letters, 2023.
[59] Han, Q. Zhu, Sheng, C. Zhang, Li, Zhang, Zhang, Y. Liu, C. Zhou, Zhao, et al. Lifelike agility and play in quadrupedal robots using reinforcement learning and generative pre-trained models. Nature Machine Intelligence, pages 1-12, 2024.
12

---

A Appendix
A.1 Observation and Action Space
The observation of the policy is composed of two main components: state observation and goal observation. State observation at time include the linear velocity (v) and angular velocity (w) of the base in local coordinates, current joint angles (0j), current joint velocities (0j), projected gravity in the base frame (gprof), base height (h) and previous actions from the last timestep (Aprev),
8, {v, w. Oj. 0j, Sproj h aprev
A variable number of keyframes K (k' k²,..... km*) are specified as targets for the robot. At each time step each keyframe kt transformed spatially and temporally into rohot-centric view. Then, the goal observation is prepared by calculating the remaining time to goal f - and the error to target goals (Ag').
Agi C (Api. Aø', AÇ", Aeb A0)].
Here, Api denotes the error between robot base position and keyframe position in the base coordi- nate frame, Δ0j is the error in joint angles, and Aø', Aç* and A denote the errors roll, pitch and yaw angles, respectively, which are wrapped to (-7, m].
The policy receives the sequence of tokens X, (a ........ az"" as input to the encoder, where æς = (sr, 0, 0), and æi = (st, Agi, - t) for i = 1,... keyframe encoding, the extra tokens can be masked to enable arbitrary number of goals. In addition, keyframes with a time over one second past the current time are also masked avoid any long-term influence on reaching the future goals.
Thanks to the transformer-based
The action (ar) space of the policy is set to target joint angles, which are tracked using a PD con- troller to compute the motor torques.
A.2 Reward Terms
We include three groups of rewards in this framework: regularization, style, and goal. For each reward group, the final reward is computed as multiplication of individual reward terms,
II Tt-
(8)
group
&Egroup
Regularization rewards are designed to provide a smooth output of the policy and consist of several terms defined in Table A1. Here, IC is an exponential kernel function defined in Eq. where and 5 are the sensitivity and tolerance of the kernel function, respectively.
maax (0, ||||| 5)
K((x, o, exp
To generate natural motion between the keyframes, we use AMP proposed by Peng et al. [12], which involves training discriminator D to identify motions that are similar to those of the offline expert dataset. The style reward defined based on the discriminator output of the latest state transition of the robot (s,-1, st),
Tsyle max (1 -1- (.) -))
(10)
Table A1: Regularization reward terms
Action rate
K. (á, 8.0, 0)
Base horizontal acceleration
C( 8.0, 0)
Joint acceleration
K((,, 150.0, 10.0)
Joint soft limits
IC (max (0) 0y.ar 0,
0, 0),0),
13

---

Table A2: Goal reward terms
Goal position
('(((p 0',0.2,00020000 0))
Goal roll
('((( - 0.,,0.0))
Goal pitch
6'(K(( - ç", 0.1,0))
Goal yaw
4'(((V - 00,3
Goal posture
4' 4'(X(0; 03.0.2v12.
Goal rewards are defined with temporally sparse kernel +'(x)
Ir t t= ja
('(x)
(11)
0, otherwise
and only activated when the corresponding timestep for that goal 1' is reached in the episode. The detailed reward terms are defined in table A2.
For advantage weights (Eq, 2), we recommend starting with a 1:1 ratio and making minor adjust- ments based specific preferences. In our experiments, used Watyle 0.5, Wgool 0.5, Wreg 0.1, maintaining equal weights for goal and style, with lower weight for the regularization critic.
A.3 Dataset Preparation
We use a database of motion capture from dogs introduced by Zhang et al. [44]. The motions are retargeted to the robot skeleton using inverse kinematics for the end-effectors positions with some local offiets compensate for the different proportions of the robot and dog. A subset of around 20 minutes of data used, removing the undesired motions such smelling the ground, walking on slopes, etc. We augment this dataset with other motion clips animated by artists include more diversity in the dataset. The frame rate is adjusted to that of the simulation, i.e. 50 frames per second. In addition, the linear and angular velocity of the base are precomputed for each frame using a finite difference method and stored in the datasct.
A.4 Training Procedure
We utilize Isaac Gym [55] for simulating the physical environment. At the start of each episode, the robot is initialized either to default state (20% probability) using Reference State Initialization (RSI) (80% probability). For RSI, random frame from the reference motion trajectory provides the height, orientation, joint angles, base linear velocity, base angular velocity, and joint velocities for initializing the robot. RSI plays crucial role in capturing and learning the specific style of motion, as highlighted in previous studies such as Peng et al. [56].
We employ two keyframe sampling strategies: dated and random sampling. In the dataset- based approach, motion trajectory and starting frame index are initially sampled. Then the time interval between keyframes are uniformly drawn from a specified range. The corresponding frames from the dataset are used to obtain the global position, orientation, and posture of the robot at the keyframe times. The global horizontal positions are subsequently converted to relative local positions with respect to the starting frame and then transformed into positions in the simulator's world coordinates by accounting for the robot's initial state. The yaw angle is similarly adjusted, while height, roll, pitch, and joint postures are directly used target values.
In the random sampling strategy, time intervals are sampled from predefined range, similar to the first strategy. The radius and direction of each keyframe, relative to the previous one are uniformly sampled to compute the keyframe's 's global horizontal position. Additionally, the change in yaw angle is sampled and added the previous keyframe's yaw. The roll and pitch angles, joint postures and height are taken directly from reference frame randomly selected from the dataset. This ensures that the posture is feasible and compatible with the height. Details of the sampling ranges provided in Table A3.
14

---

Table A3: Keyframe sampling ranges
Quantity
Time interval
Waypoint radius (m)
Waypoint direction (rad)
Delta yaw angle (rad)
(steps)
Range
25, 50]
0.5, 1.0
n/3
/T33, 7/3)
Our methodology incorporates learning curriculum, beginning with keyframes entirely sourced from reference data and progressively increasing the proportion of randomly generated keyframes. The meticulous sampling of target keyframes is critical for ensuring their feasibility and preventing them from impeding effective policy learning.
We train the policy to handle a maximum number of keyframes, randomly selecting the actual num- ber of keyframes for each episode. To avoid negative impacts on training, unused goals are masked when input into the transformer encoder. For stability, the episode does not terminate immediately after the last goal is reached; instead, it terminates approximately setup for full keyframe comprising time, position, roll, pitch, yaw, and posture targets with up to maximum keyframes requires approximately 17 hours on system equipped with Nvidia GeForce RTX 4090.
second later. The training
A.5 Hardware Implementation Details
Domain randomization is added during training to achieve robust policy that can be executed on hardware. Similar to Kang et al. [58], we randomize friction coefficients, motor stiffness and damping gains and actuator latency. Furthermore, we add external pushes during training. Details of domain randomization ranges are provided in Table A4. Although joint limits are softly taken into account in the simulation, we found it crucial terminate episodes when reaching joint limits to ensure stable deployment on hardware. We use motion capture system to receive the global position and orientation of the robot. These are used to compute the relative errors to the target goals and are then passed to the policy. Other observations are computed based on the outputs from the state estimator. Our locomotion control policy runs at 50 Hz and updates the joint angle targets. At a lower-level, the robot's built-in motor controller operates at 1000 Hz, updating the torque targets for each joint. To enhance computational efficiency during RL training, we simulate the robot and the PD controller 200 Hz, which has proven effective in practice.
Table A4: Domain randomization ranges
Quantity
Range
Friction Coefficient
[0.25, 1.75]
Push velocity (m/s)
[0.0, 1.0]
Push angular velocity (m/s)
[000, 1.0)
Stiffness multiplier
[0.9, 1.1]
Damping multiplier
[0.9, 1.1]
Actuator lag steps
[0,6]
A.6 Keyframe Matching
We provide quantitative analysis of keyframe matching performance for the three scenarios from Fig. 5 with full-pose keyframes in Table A5. The mean and standard deviation of each entry is from 20 independent evaluations. The posture error reported is the root mean squared error (RMSE) of all joint angles. The results indicate that our trained policy tracks the keyframes with good precision and consistency.
To assess the generalization capability of our method, we evaluated the trained policy using three different keyframe sampling strategies:
15

---

Table A5: Keyframe matching error for three different scenarios
Scenario
Walk (goal 1)
Walk (goal 2)
Jump
Paw up
Distance error (m)
0.048 + 0.011
0.020 + 0.003
0.118 + 0.012
0.044 + 0.015
Roll error (deg)
3.094 + 0.327
1.621 + 0.074
1.473 0.579
1.209 + 0.733
Pitch error (deg)
0.332 0.178
0.934 = 0.235
1.558 = 0.940
0.516 0.246
Yaw error (deg)
1.209 + 0.384
2.120 + 0.367
7.110 + 1.060
2.773 1.301
Posture error (deg)
2.469 + 0.309
2.595 + 0.126
3.226 + 0.103
4.698 + 0.808
Dataset:
Sampling the full pose (position, orientation, and posture) only from trajectories in the reference dataset.
Random: Sampling the posture from the reference dataset while randomly sampling the
planar position and yaw angle targets.
Evaluation dataset: Sampling the full pose from the evaluation dataset, which was not seen by the policy during training. For the evaluation dataset, use few locomotion clips from the open-sourced dataset from Han et al. [59].
The mean and standard deviation for each strategy are derived from 20 independent evaluations and shown Table A6. The results indicate that our policy generalizes outside the trained dataset in terms of position and yaw angle when exposed the randomly sampled target during training. Surprisingly, the policy achieves decent keyframe tracking in position and orientation well as postures with the evaluation dataset which is promising generalization results, full generalization to postures that differ significantly from those in the training dataset is yet to be achieved.
during training. Although Table A6 shows
Table A6: Keyframe matching performance across different keyframe sampling strategics
Keyframes source
Dataset
Random
Evaluation dataset
Distance error (m)
0.060 0.050
0.142 + 0.087
0.064 + 0.060
Roll error (deg)
1.873 + 1.888
3.181 = 3.466
3.231 + 9.053
Pitch error (deg)
3.084 = 6.747
4.260 = 9.202
1.438 + 2.842
Yaw error (deg)
2.394 = 1.775
8.754 = 6.220
2.200 + 2.235
Posture error (deg)
5.119 3.143
4.640 = 1.606
9.207 + 7.122
A.7
Future Goal Anticipation
Details of target keyframes used for Table 1 are given in Table A7.
Table A7: Details of keyframe scenarios
First Goal
Second Gaol
Scenario
Time (steps)
Position (m)
Time (steps)
Position (m)
Straight
50
(0, 0.32, 1.0)
75
(0, 0.32, 2.0)
Turn
50
(0, 0.32, 1.0)
75
(1.0, 0.32, 1.5)
Turn (Slow)
(0, 0.32, 1.0)
100
(1.0, 0.32, 1.5)
16

---

A.8 Training Hyperparameters
Table A8 provides details of hyperparameters used for training.
Table A8: Summery of training hyperparameters
Number of environments
4096
Number of mini-bateion
Number of learning epochs
5
Learning rate
0.0001
Entropy coefficient
0.02
Target KL divergence
0.02
Gamma
0.99
Lambda
0.95
Discriminator learning rate
0.0003
Discriminator update per epoch
20
Discriminator batch size
768
Transformer encoder layers
2
Transformer heads
Transformer feed-forward dimensions
512
MLP dimensions
512, 256
Initial standard deviation
Activation function
ELU
17


Reviews:
[{"review_id": "4wOAZGv4bP", "reviewer": "Reviewer_aL5k", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 4, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "To implement keyframe-based control of quadrupeds, the paper proposes to train multiple critics to separately handle sparse and dense rewards. A transformer-based keyframe encoder is used. The method is shown to work better compared to having a single critic or a policy only aware of the next goal (instead of anticipating future goals). Great explanatory video is provided.", "summary_of_recommendation": "Solid clean paper with a clear straightforward contribution and strong demonstration on real hardware", "technical_quality": 4, "clarity_of_presentation": 5, "potential_impact": 4, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "Reward Weights and Sparsity:\n- How sensitive is the performance of the proposed approach to the choice of weights  w_s  and  w_d  in the reward function? What guidelines can you provide for tuning these parameters?\n\nScalability and Generalization:\n- How well do you expect the proposed methods to scale to different types of robots or environments? Have you considered testing the framework on robots with significantly different morphologies or in more complex environments?\n\nData Efficiency:\n- The paper mentions the efficiency of policy learning. Can you provide more details on the data efficiency of your approach compared to other state-of-the-art methods?\n\nPublic Availability:\n- Is there any plan to make the code and datasets used in this research publicly available? This would greatly benefit the research community and facilitate further advancements in the field.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bj3aewBeGy", "reviewer": "Reviewer_i8Sh", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper presents a method for learning a “keyframe” policy that can satisfy a sequence of keyframe poses. This is done through a transformer based architecture to handle a variable number of goal keyframes as input, along with a multi-critic RL process to more evenly weight dense vs. the sparse keyframe matching rewards. There is also flexibility in what the keyframes specify, allowing for any full or partial combination of base pose and joint angles. Simulation results are shown tracking both position waypoint goals along with full posture key frames for walking, jumping, and paw raising motions. Ablation studies vs. a single critic method are presented, showing the robustness of using multiple critics with varying reward sparsities. Finally hardware results are shown of the system running on a Unitree Go2 quadruped doing locomotion and posture matching tasks.", "summary_of_recommendation": "The novelty is a bit limited but it is moving towards a good direction. I think that this keyframe approach can have some advantages over the style based methods that are more commonly used today. Full generalization has yet to be shown in this work but it is a first step towards some thing that could do general pose matching, which would be pretty powerful.", "technical_quality": 4, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "I don’t fully get the use of a transformer-based architecture since it can’t really handle arbitrary number of keyframes as used here. There is still a fixed maximum number of keyframe goals it can handle and masking is still required. How important is the self attention for this system? If you need to have masking anyway, why can you not just individually encode these keyframes and pass them directly into the MLP? Why not have some positional encoding of each keyframe so that the policy can handle any number of keyframe goal inputs?\n\nWhy does the state need to be part of goal tokens X_t? It seems like the state is the same across each goal token, and the state is already inputted to the MLP layers. What is the intuition for why the state should influence which goal to pay attention to? How does the policy perform if the state is not part of the goal tokens?\n\nSince there is the self-goal keyframe which has 0 error and 0 time to goal, how does the policy differentiate between this self-goal keyframe and a masked keyframe (which I assume is also just all zeros)? I guess in the self-goal case the state will not masked, is this the purpose of including the state in the goal keyframes? \n\nYou mention that past goal keyframes are masked as well, are they just masked in their current place or are the next goal keyframes “moved up”? For example, if at the start the goal tokens are inputted like $(x^0_t, x^1_t, …, x^{n_k}_t)$ then some time later (let’s say at $t = w$) after 2 goals have been reached are goal tokens like $(0, 0, x^2_w, …, x^{n_k}_w)$ or would it be like $(x^2_w, …, x^{n_k}_w, 0, 0)$? The text sounds like it does the first option, but I think the second option makes more sense? Along the same line, is the self-goal keyframe ever masked? Or will it always be a part of the input?\n\nHow important is being aware of all next goals rather than just the next two? The evaluation provided (Table 1) only tests when there are 2 keyframes, not when there are more than 2. I can understand that being aware of more than just the next goal is useful so that the policy can prepare (as Table 1 shows), but how far in the future is it necessary to look ahead? If for example there are 3 keyframe goals, is it necessary to know all 3 or can they just be given two at a time (1st and the 2nd, and then the 2nd and 3rd after the 1st goal is past). I think this is an important evaluation to do, since it answers if handling a variable number of keyframes (a large part of your method) is even necessary or just a small fixed lookahead works for all cases. \n\nWhat are the weightings ($w_i$) to used to weight the sparse vs. dense advantages? How is this hyperparameter tuned?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zBcwh1t5kX", "reviewer": "Reviewer_fZ8e", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper presents a reinforcement learning pipeline for learning for learning quadruped locomotion policies in simulation and demonstrates transfer to a Unitree Go2. Whereas prior methods are often restricted to velocity commands, the proposed method can set a variable number of keyframes consisting of timestamps, target poses and joint configurations. The keyframe reward, which is sparse as it is only given at the respective timestamps, is complemented by an AMP style reward obtained from the popular dog motion capture dataset by Zhang et al. 2018. The main contribution is the use of a multi-critic approach, that rather than trading-off the different reward terms (style reward and keyframe reward) independently estimates the advantage functions for the these rewards, normalizes them, and weights them for trading-off. While this multi-critic approach is not novel, the idea is relatively novel, hasn't been applied to locomotion and seems interesting, because in contrast to reward-weights, the weights on the advantage functions work well for different degrees of sparsity (timegaps between keyframes). The paper presents nice empirical results, as the quadruped performs natural locomotion while reaching the desired keyframes at the request timestamps.", "summary_of_recommendation": "Good work with sufficient relevance to the field but insufficient description of the methodology.", "technical_quality": 3, "clarity_of_presentation": 3, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "How exactly are the keyframe tokens passed to the transformer? A token typically refers to a single integer, however, I assume that the state, relative goal, and time-delta are not discretized into a single integer. \n\nThe appendix provides some information on sampling the keyframes. However, I think it would be useful to describe the process in more detail.\n\nIt would also be useful to describe the sampling of the initial state (with RSI) in more detail, e.g. by providing code or pseudo-code.\n\nAppendix states that the \"previous actions\" are part of the state. Does it refer to the actions of the last time step, or a longer history of actions?\n\nI assume the loss in Eq. 1 uses a target network. It would be could to indicate this in the equation.\n\nWhat are the \"time steps\" in the x-axis of Fig. 6? Are these RL iterations or is there missing some factor to bring them to the right order of magnitude?\n\nAccording to the appendix, the RL policy runs at 50Hz. What are the control frequencies of the PD controller? \n\nThe joint soft limits (Table A1) seem to bias the joint position towards the middle position, $1/2 (\\boldsymbol{\\theta}\\_{j,min} - \\boldsymbol{\\theta}\\_{j,max})$. Does this joint configuration correspond to a reasonable nominal posture for the robot?\n\nHow often is the discriminator updated during RL?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper presents a keyframe-based learning framework for legged locomotion. Reviewer aL5k is excited about the significance of the proposed work, along with its originality. On the other hand, Reviewer i8Sh questioned that the proposed multi-critic framework is not directly related to the proposed concept. Reviewer fZ8e raised concerns about the reproducibility and hardware evaluation. We would like to encourage the authors to respond to all the reviewer's comments.

The authors provided thorough feedback on the raised concerns, and the reviewers seemed satisfied with the responses. Particularly, the authors communicated well with Reviewers i8Sh and fZ8e. To this end, we recommend accepting this manuscript.
Decision: Accept