SLR: Learning Quadruped Locomotion without Privileged Information
Authors: Shiyi Chen
Zeyu Wan
Shiyang Yan
Chun Zhang
Weiyi Zhang
Qiang Li
Debing Zhang
Fasih Ud Din Farrukh
Venue: CoRL 2024 (2024)
Keywords: Locomotion
Reinforcement Learning
Privileged Learning

Abstract: Traditional reinforcement learning control for quadruped robots often relies on privileged information, demanding meticulous selection and precise estimation, thereby imposing constraints on the development process. This work proposes a Self-learning Latent Representation (SLR) method, which achieves high-performance control policy learning without the need for privileged information. To enhance the credibility of our proposed method's evaluation, SLR is compared with open-source code repositories of state-of-the-art algorithms, retaining the original authors' configuration parameters. Across four repositories, SLR consistently outperforms the reference results. Ultimately, the trained policy and encoder empower the quadruped robot to navigate steps, climb stairs, ascend rocks, and traverse various challenging terrains.

SLR: Learning Quadruped Locomotion without Privileged Information
Shiyi Chen', Zeyu Wan', Shiyang Yan', Chun Zhang"', Weiyi Zhang', Qiang Li Debing Zhang', Fasih Ud Din Farrukh
Tsinghua University, SShenzhen Technology University
Self-learning Latent Representation No Privileged Information Training
Friction,
Linear
Velocity, Height Map,
Contact,
Vision,
Figure 1: We propose a framework for training a robust quadruped lo- comotion policy without relying on privileged information. The robot effectively navigates challenging terrains, showcasing adaptive loco- motion skills acquired through self-learning.
The recent mainstream reinforcement learning control for quadruped robots often relies on privileged information, demanding meticulous selection and precise estimation, thereby imposing constraints on the development pro- cess. This work proposes Self-learning Latent Representation (SLR) method, which achieves high-performance control policy learning without the need for privileged information. To enhance the credibility of the proposed method's 's eval- uation, SLR was directly compared with state-of-the-art algorithms using their open-source code repositories and original configuration parameters. Remark- ably, SLR surpasses the performance of previous methods using only limited pro- prioceptive data, demonstrating significant potential for future applications. Ulti- mately, the trained policy and encoder empower the quadruped robot traverse various challenging terrains. Videos of our results can be found on our website: https://1ichens tettth...oSs.t.
Abstract:
Keywords: Locomotion, Reinforcement Learning, Privileged Learning
Introduction
Humans and animals inherently possess locomotion abilities, enabling them to traverse various com- plex terrains. In contrast, gait control for robots is highly challenging. Model-based methods
"Corresponding Author
8th Conference Robot Learning (CoRL. 2024), Munich, Germany.

---

have achieved some success by leveraging robots' mechanical structures and dynamic principles [1, 2, 3, 4, 5]. However, finding a balance hetween model accuracy and computational efficiency remains difficult, especially for real-time applications.
Additionally, designing these models requires deep understanding of a robot dynamics, posing significant challenge for researchers. As result, Reinforcement Learning (RL) methods are be- coming increasingly popular. By simulating real-world environments and training policies with customized reward functions, these methods enable robots to perform complex locomotion tasks in real-time [6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18).
The recent mainstream RL applications in quadruped robots rely on privileged learning meth- ods [19]. In real-world scenarios, robot's interaction with its enviromment is modeled as Partially Observable Markov Decision Process (POMDP). Solely relying proprioceptive sor measurements, robot cannot fully perceive external environmental information, limiting its decision-making capabilities. Consequently, many studies leverage the "observability" advantages of simulation platforms. During training, various physical parameters (such as friction coefficients [10, 20, 21], restitution coefficients [9, 21], and scan-dots of the terrain [14, 22]) are artificially added as privileged information to help the robot understand both itself and the external environment.
Unlike human cognition, which navigates terrains without explicit knowledge of physical parame- ters, neural network-basse robots may not benefit from adding such parameters. This is because the parameters are not used to solve dynamic equations but are simply input into the neural network. As a result, these privileged information pieces may not enhance the interpretability or effectiveness of neural network-based agents.
Therefore, instead of relying on manually chosen physical parameters to construct privileged in- formation, this work explores whether it is possible for robots to learn latent representation of environmental states by themselves? It is proposed that intelligent robots should be capable of self- learning latent representations, which are inherently more suitable than those predefined by humans.
To achieve this, we propose the Self-learning Latent Representation (SLR) algorithm, which gen- erates latent representations guided by RI. Markov process without privileged information. This self-learning approach enables the robot grasp latent environmental features and exhibit general- ized locomotion capabilities across challenging terrain, as illustrated in Figure 1.
Our results show that the SLR algorithm, which operates without privileged information, outper- forms traditional privileged learning methods. It consistently aligns with actual terrain conditions across various terrains. Implemented in existing open-source repositories and evaluated in the same environments previous studies, the SLR algorithm demonstrates state-of-the offthe-art (SOTA) perfor- mance in both simulations and real-world applications. This suggests that the self-learning approach has considerable potential for future expansion.
2 Related Work
Privileged learning in RL-based methods can be divided into explicit estimation and implicit esti- mation methods based on the target of supervised learning. In this paper, directly fitting specific physical parameters from privileged information is referred as explicit estimation, while fitting the latent representation of privileged information classified as implicit estimation.
Explicit Estimation. [6] concurrently trains policy network and a state estimator, which include real-world parameters that are difficult to obtain accurately, such as linear velocity, foot height, and contact probability. [13] sets friction coefficients and stiffness coefficients as privileged information, inferring these from observation history to assist the robot in domain randomizations. But in the real world, the quadruped robot's foot contact time very short during fast running, making difficult fully perceive ground friction. Therefore, [12] proposes learning infornation-gathering behaviors by adding active estimation reward, which increases the accuracy of estimates for privileged information disparities.

---

Self-learning Latent Representation
Triplet Optimization
Random Sample
Replay
24-11
1
2t+1 <-
TransModel
=g/-1
Encodor
O2
Policy Gradient
Proximal Policy Optimization
Figure 2: Illustration of SLR training framework. All dashed lines represent the network updating process. The translucent fuchsia lines indicate the encoder updates through backpropagation from the Critic network, the Transition model, and random sampling. The remaining solid lines represent the network's forward inference process.
Implicit Estimation. To train quadruped robots on complex terrain, estimating a large amount of privileged information explicitly is challenging. common approach to encode this high- dimensional information into a latent representation. [10] utilizes teacher encoder compress data such friction and terrain height into latent representation, and then student adaptation module learn to infer this latent representation from observation history. [11] optimized the teacher-student training process from [10] in one stage by regularizing the teacher's actual privileged latent and supervising the student's estimated privileged latent. The algorithm of [9] also includes a teacher- student policy. During training, the teacher's s encoder is used, and the student's adaptation module output aligns with the teacher's encoder. For deployment, the student's adaptation module and the teacher's trained policy are utilized together. [8] employs the Asymmetric Actor-Crite (AAC) [7] method, feeding privileged information the critic network while using encoder-decoder estimators to assist the actor in imagining this privileged information.
3 Method
3.1 Problem Formulation
Our goal is to construct one-stage end-to-end system based on RL, using proprioceptive sensor data as input for measuring and controlling joint movements.
Observation Space. The observations o, consist of dimensions, including the robot's base an- gular velocity w, commands from the joystick c==lvend _ , ,. J. measurement of the gravity vector gt. joint positions 0 and velocities Ú, and the actions az-1 taken by the robot at the previous time step.
Action Space. The action space is 12-dimensional vector a, corresponding the four legs of the quadruped robot, with each leg having three motor drive units. The neural network's output is converted into actual torque 7 through PD controller.
Reward Functions. The reward functions we used during the training are shown Table A1, which come from [10, 16, 23].
Domain Randomizations. For the training of our method, domain randomizations used. The details are shown in Table A2, which come from [8, 21].

---

3.2 Framework Overview
The proposed training framework utilizes the Markov Decision Process (MDP) to guide self-learning of the latent representation through state transitions, distinctions, and cumulative rewards, without relying on manually defined privileged information. All networks in this framework are multi-layer perceptrons (MLPs). Table A3 in the Appendix details the MLP structures, and Table A4 lists the training hyperparameters. The training framework is illustrated in Figure 2.
Encoder: In this architecture, the encoder's input consists of the observation history of', which is composed of proprioceptive information or from the previous 10 time steps. The encoder outputs a 20-dimensional latent representation 2 of the observation history:
24 =600
Actor-Critic: The policy (Actor) and Critic network are trained jointly via Proximal Policy Op- timization (PPO) algorithm [24]. The policy takes as input the current proprioceptive observation or and the latent representation 24, and it outputs the joint position aç. The Critic network, which shares the same input as the policy, outputs the state value et- worth noting that, we turn off the gradient of 2 the policy and turn on in the Critic network, using the backpropagation of the Critic network to update the encoder the direction of the maximum cumulative reward:
(e (ot, sg. sg(=+)
2/4 = V (ot,
(3)
where sg[[] is the stop gradient operator.
Transition Model: The Transition model simulates the real state transitions of the environment p ($t+1 st, a.). Its input is the latent representation 24 and the action ag and it outputs the next time step's latent estimation ε
(4) Loss Function: We align the estimated Æ from the state transition model with the actual latent state 2t+1 at time + 1, while ensuring distinctiveness from other latent states 2ttn at different times & + n. Drawing inspiration from [25, 26), we formulate a latent representation loss function utilizing a triplet loss, denoted as Luip-
4+1 = p(21, 0z)
Cuip (2++1, ž-11 2t4n) max (1|2+1 - 2 || ztn + m, O). s.t. n n 1
(5)
where is margin that is enforced between žt+1 and žttt pairs, set to 1.0 in the experiment.
This updating strategy empowers the encoder to comprehend the dynamics of environmental state transitions and extract environmental attributes by assimilating state-action pairs derived from the MDP rollout. Finally, the triplet loss Luip multiplied by a triplet coefficient and added the PPO loss Lpp [24] for network updating. Details of simulation and training are in Appendix 1.
4 Experiments
4.1 Ablation Study for Latent Representations
To evaluate the proposed algorithm against traditional privileged learning algorithms, an ablation study was conducted. In this setup, commonly used privileged information e, R10 was chosen, including parameters such friction, restitution, foot height, and foot contact. The methods using different types of latent representations
follows:
1) Implicit: The policy input includes the implicit privileged latent 4, which is encoded from privileged information e, by adopting the teacher policy approach in [9, 10].
2) Explicit: The policy input includes the explicit privileged information ((o,").
3) SLR w/ explicit: The SLR policy input is the concatenation of the self-learning latent 2, and explicit privileged latent é,

---

12.0
8.0
9.0
Mean Ruwward
Terrain Level
3.0
Explict implitit
2.0
explict SLR w/ implict:
excpict SUR w/ implicit
Implisi
SUR wio latent
SUR weo letent
1500
3000
4500
b000
1500
3000
4500
6000
Earations
Rerations
Figure 3: Ablation study training curves, curves are averaged over 3 seeds. The shaded area repre- sents the standard deviation across seeds, and the curves are smoothed using Gaussian filtering.
4) SLR w/ implicit: The SLR policy input is the concatenation of the self- self-learning latent 2 and implicit privileged latent 4,
5) SLR w/o latent: The SLR policy input does not include any latent representations and does not utilize an encoder.
To ensure the fairness of quantitative comparisons, the configurations used in the experiments all based the default settings in the code repository [27]. Training conducted for 6000 iterations under multiple terrains by default, and the results are presented in Figure 3. And we can draw the following conclusions:
The SLR algorithm outperforms traditional explicit and implicit privileged learning methods, suggesting that unified privileged information struggles to effectively represent diverse terrains.
2. Incorporating additional privileged latent information reduces the SI.R algorithm's performance, possibly due to increased policy input disrupting the critic's guidance on updating the self-learning latent.
4.2 Latent Representation Analysis
Identifying and distinguishing terrain types is crucial for robotic systems [28]. To evaluate the self-learned latent representation, conducted simulation-based analysis. The test environment includes four terrains: upward slope, descending stairs, flat ground, and ascending stairs. The robot navigates these terrains sequentially (Figure 4), and the latent representation output is recorded at each step. We then apply t-distributed stochastic neighbor embedding (t-SNE) to project these complex latent representations into a two-dimensional space for analysis. The latent representations from Implicit and SLR are examined separately, as shown in Figure
Different Terrain Representation. As shown in Figure 5. The self-learned latent representations reveal distinct ting-shaped regions (labeled A. B, C, D) for each terrain. Notably, regions A and D exhibit similarity, suggesting the robot perceives up slope and ascending stairs as similar processes. In contrast, representations trained with the Implicit method present scattered points, indicative of overlapping privileged information across terrains.
Terrain Transition Representation. Notably, in the SLR latent representations, when the robot transitions from one terrain to another, all four ring-shaped latent representations show "tails", For instance, when transitioning from an up slope to descending stairs, the latent rep- resentation in region A has tails extending to the right, with the tail's end having color sim- ilar to the lower left corner of region B. This indicates that the robot is near the boundary between terrains at that moment, and our latent representations are effectively indicating such terrain transi- tions.
Figure 4: t-SNE test terrains.
5

---

400
0.
0.2
Figure 5: t-SNE visualization of Implicit (left) and SLR (right). Color intensity represents cumu- lative steps across four terrains. The privileged latent distribution is discrete and weakly correlated with terrain. In contrast, the SLR latent trajectories align precisely with the terrains traversed by the robot, with each ring-like representation accompanied by "tail", indicating terrain transitions.
5 Results
5.1 Compared Methods
Previous comparative studies of robot algorithms often lack fairness due to the absence of a univer- sal benchmark robotics. Comparing algorithms within one's own environment is not objective, as reward functions and hyperparameters are typically optimized for the researcher's algorithm. Additionally, replicating previous algorithms in new code repository can lead incomplete repro- ductions, potentially affecting their performance.
In this study, these issues are addressed by directly implementing the SLR algorithm in the open- source code repositories previous SOTA works [9, 11, 13, 27, 29]. Only the algorithm framework modified while keeping other variables consistent with the original settings. This approach en- sures fair comparison between method and the publicly available SOTA algorithms The algorithms which are evaluated given below:
MoB[13]: Explicit estimation, trained flat ground with the Unitree Gol robot.
RLvRL[9]: Implicit estimation, trained on flat ground with the Cheetah robot.
ROA[11]: Implicit estimation, trained on custom fractal noise environment using a Unitree Gol robot equipped with a manipulator.
HIM[29]: Explicit and implicit estimation, trained in multi-terrain environments with the Unitree Aliengo robot.
Baseline(27]: No encoder, no privileged information, trained in multi-terrain environments with the Unitree robot.
5.2 Simulation
All five code repositories implemented reinforcement learning based the PPO algorithm, with training conducted the IsaacGym platform [27, 30] using three different random seeds. The training curves are illustrated in Figure 6. Additionally, to evaluate the locomotion capabilities, experiments on velocity tracking performance were conducted on flat terrain. The evaluation metrics included Linear Velocity Tracking Error (LVTE) and Angular Velocity Tracking Error (AVTE), both measured using Mean Square Error (MSE). The evaluation results are summarized in Table 1.
Prrevious evaluations a single repository achieved noticeably better performance (see Section 4.1). To
provide more conclusive evidence, we extended experiments.

---

MoB
ROA
Baseline
6.0
30.0
0.9
12.0
12.5
4.5
7.5
0.6
30.0
3.0
5.0
6.0
Moan
5.0
1.5
2.5
3.0
2.5
0.0
2000 4000 500D
8000
300 1000
1500
2000
1000 2000 3000
4000
500 1000
1500
2000
1000
2000
3000
Rerasions
Iterations
Itorations
Rteations
Iterations
SURtours)
Figure 6: Training curves from various repositories show that the mean reward of SLR surpasses those of Baseline, RLvRL, and ROA, is slightly higher than MoB, and is comparable to HIM. The shaded area represents the standard deviation across seeds, and the curves are smoothed using Gaus- sian filtering. Note: SLR achieved a higher terrain level than HIM, indicating superior performance.
Code Repository
Metrics
Ranges
Method
MoB
RLvRL
ROA
HIM
Baseline
Previous
0.058
0 003
0.150+0.001
0.014
0.021 00000
[111]] m/s
m/s
Ours
0.029:00.04
0.024+0.000
0.136 00.000
0.013#0.00z
0.011 #0.004
LVTE
Previous
0.18140.014
0. 0.108=20.000
N/A
0.019
0.074=0:007
[-2,2] m/s
Ours
0.124200022
0.106:0.000
N/A
0.017:0.002
0.066:10.004
Previous
0.035 0.7:0000002 0.00
0. 182 007
1.100
0.013 009
0.137=0.013
[-1,1]
Ours
0.049:200001
0.057
1.09640.0 10 025
0.014
0.07720.019
AVTE
Previous
0.131#0.015 #0.015
0.347=#0.00
N/A
0.038=0.019
0.315=0.022
[-2.2] rad/s
Ours
0.086=00000
0.142#0.002
N/A
0.038 023
0.223
Table Velocity tracking errors various environments. The SLR algorithm demonstrates superior velocity tracking capabilities compared to the previous algorithm in most cases. Top performances are highlighted bold. Note: N/A indicates that the ROA code repository does not support rapid locomotion.
Based on the evaluation results of the five code repositories, the proposed SLR algorithm generally achieves higher mean rewards and superior velocity tracking capabilities compared to the original implementations. From these results, we can draw the following conclusions:
1. The SLR algorithm achieves mean rewards comparable to those of the recently released top- performing HIM algorithm, but with a higher terrain level (see Figure 8), indicating superior per- formance. This demonstrates that learning from limited proprioceptive data can rival advanced privileged learning methods.
The SLR algorithm's 's effectiveness in managing complex tasks with MoB (multiple commands) and ROA imanipulator-equipeer repositories suggests that self-learning latent representations offer more general benefits than manually selected privileged information.
3. The SLR algorithm demonstrates enhanced tracking capabilities across various velocity ranges. One reason for this is that SLR can implicitly infer velocity from proprioceptive data. Additionally, the Critic network optimizes for maximum cumulative rewards, with velocity tracking as the most significant term in the reward function. Consequently, the backpropagation in the Critic network naturally guides the latent representation to optimize tracking performance.
5.3 Deploy in Real-World
The trained policies are deployed on the Unitree Go2 robot in real-world as depicted in Figure 7. Performance evaluation of the policy was conducted across various indoor and outdoor terrains, and comparative analyses were performed against code repositories [13, 27, 29] and Unitree built-in MPC control method. Each environment was tested 10 times. As presented in Table 2, the results demonstrate the superior efficacy of our policy in real-world scenarios.
7

---

Discussion and Limitations
In this study, we present a quadruped locomotion algorithm that operates without privileged infor- mation. Relying solely on limited proprioceptive data, the algorithm achieves SOTA performance. Looking ahead, we anticipate that future self-leerning approaches will be able integrate privileged information and external perceptions, leading to even more impressive outcomes.
While blind policy demonstrates robust motion, achieving superior trajectory planning neces- sitates the use of vision sensors. Moving forward, we aim to enhance quadruped locomotion by integrating visual information for tackling even more complex challenges.
(a) Indoor Scenarios
(b) Outdoor Scenarios
Figure 7: Indoor and outdoor experiment settings. The left column (a) showcases indoor scenarios, with the top row depicting the long stairs and the bottom row featuring the step obstacle. The right column (b) illustrates outdoor scenarios, depicting the robot moving through bush, cobbled road, and climbing curb in the top row, while the bottom row showcases the robot encountering slope, stairs, and rock.
Scenarios
Terrain
Metries
Ours
HIM
MoB
Baseline
MPC
Stairs
Number
136.4 124.
107.4 107.4*19.
19.1
0.0
0.0
93.54-32.9
Indoor
Step
Height (cm)
35.4 222
30.7
6.1
5.3
15.6
Bush
100
100
50
100
Cobbled Road
100
100
100
30
90
Curb
Success rate (%)
100
100
20
0
40
Outdoor
Earthen Slope
100
100
40
0
60
Stairs
100
100
0
0
20
Rock
70
30
Comparison of deployment methods across various terrains. Each method was tested 10 times per terrain. The "Number " indicates the total stairs (16 x 29 cm) climbed by the robot from start to fall, and "Height" shows the maximum step height the robot resents a 95% confidence interval. A successful trial means the robot completes the terrain without intervention or failure within the allotted time. Top performances are in bold.
Table 2:
handle. The + symbol rep-

---

Acknowledgments
We would like express our gratitude Han Yu for providing invaluable assistance in our de- ployment work. Additionally, this work is supported by the National Natural Science Foundation of China (No.U20A20220).
References
[1] G. Bledt, M. Powell, B. Katz, Di Carlo, P. M. Wensing, and S. Kim. Mit cheetah 3: Design and control of robust, dynamic quadruped robot. In 2018 TEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 0245-22522 IEEE, 2018.
[2] C. D. Bellicoso, F. Jenelten, C. Gehring, and M. Hutter. Dynamic locomotion through online nonlinear motion optimization for quadrupedal robots. IEEE Robotics and Automation Letters, 3(3):2261-2268, 2018.
[3] Y. Ding, A. Pandala, C. Li, Y.-H. Shin, and H.-W. Park. Representation-free free model predictive control for dynamic motions quadrupeds. IEEE Transactions on Robotics, 37(4): 1et11114-1111 2021.
[4] A. Bouman, M. F. Ginting, N. Alatur, M. Palieri, D. D. Fan, Touma, T. Pailevanian, S.-K. Kim, K. Otsu, J. Burdick, et al. Autonomous spot: Long-range autonomous exploration of extreme environments with legged locomotion. In 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 2518-2525. IEEE, 2020.
[5] C. Gehring, P. Fankhauser, L. Isler, R. Diethelm, S. Bachmann, M. Potz, L. Gerstenberg, and M. Hutter. Anymal in the field: Solving industrial inspection of an offshore hvde platform with a quadrupedal robot. In Field and Service Robotics: Results of the 12th International Conference, pages 247-260. Springer, 2021.
[6] G. Ji, Mun, H. Kim, and J. Hwangbo. Concurrent training of control policy and a state estimator for dynamic and robust legged locomotion. IEEE Robotics and Automation Letters, 7(2):463-44677,)) 2022.
[7] L. Pinto, M. Andrychowicz, P. Welinder, W. Zaremba, and P. Abbeel. Asymmetric actor critic for image-hased robot learning. arXiv preprint arXiv: 1710:06710 2017.
[8] I. M. A. Nahrendra, B. Yu, and H. Myung. Dreamwaq: Learning robust quadrupedal lo- comotion with implicit terrain imagination via deep reinforcement learning. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 5078-5084. IEEE, 2023.
[9] G. B. Margolis, Yang, K. Paigwar, T. Chen, and P. Agrawal. Rapid locomotion via rein- forcement learning. The International Journal of Robotics Research, 43(4):572-587, 2024.
[10] A. Kumar, Z. Fu, D. Pathak, and Malik. Rma: Rapid motor adaptation for legged robots.
2021.
[11] Z. Fu, X. Cheng, and D. Pathak. Deep whole-body control: Learning unified policy for manipulation and locomotion. In Conference on Robot Learning, pages 138-149, PMLR, 2023.
[12] G. B. Margolis, X. Fu, Y. Ji, and P. Agrawal. Learning to physical properties with active sensing motor policies. Conference on Robot Learning, 2023.
[13] G. B. Margolis and P. Agrawal. Walk these ways: Tuning robot control for generalization with multiplicity of behavior. In Conference Robot Learning, pages 22-31. PMLR, 2023.
[14] X. Cheng, K. Shi, A. Agarwal, and D. Pathak. Extreme parkour with legged robots. arXiv preprint arXiv:2309.14344 14341, 2023.

---

[15] Z. Zhuang, Z. Fu, J. Wang, C. Atkeson, S. Schwertfeger, C. Finn, and H. Zhao. Robot parkour learning. arXiv preprint 2xXxv:3309006666. 2023.
[16] A. Agarwal, A. Kumar, J. Malik, and D. Pathak. Legged locomotion in challenging terrains using egocentric vision. In Conference robot learning, pages 403-415. PMLR, 2023.
[17] He, C. Zhang, W. Xiao, G. He, Liu, and G. Shi. Agile but safe: Learning collision-fiee high-speed legged locomotion. arXiv preprint arXiv:2401.15883 2024.
[18] R. Yang, M. Zhang, N. Hansen, H. Xu, and X. Wang. Learning vision-guided quadrupedal lo- comotion end-to-end with cross mrossl transformers. arXiv preprint arXiv:2107.03996, 2021.
[19] D. Chen, B. Zhou, V. Koltun, and P. Krähenbühh. Learning by cheating. In Conference Robot Learning, pages 66-75. PMLR, 2020.
[20] Lee, Hwangbo, L. Wellhausen, Koltun, and M. Hutter. Learning quadrupedal locomo- tion over challenging terrain. Science robolics, 5(47)xeabc5986, 2020.
[21] J. Wu, G. Xin, C. Qi, and Y. Xue. Learning robust and agile legged locomotion using adver- sarial motion priors. IEEE Robotics and Automation Letters, 2023.
[22] T. Miki, J. Lee, J. Hwangbo, L. Wellhausen, V. Koltun, and M. Hutter. Learning robust per- ceptive locomotion for quadrupedal robots in the wild. Science Robotics, 7(62))abb,2822, 2022.
[23] Z. Fu, A. Kumar, J. Malik, and D. Pathak. Minimizing energy consumption leads to the emer- gence of gaits in legged robots. arXiv preprint arXiv:2111.01674, 2021.
[24] Schulman, Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 1707.06347777 2017.
[25] F. Schroff, D. Kalenichenko, and J. Philbin. Facenet: A unified embedding for face recogni- tion and clustering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 115-823, 2015.
[26] A. Zhang, R. McAllister, R. Calandra, Y. Gal, S Levine. Learning invariant representations for reinforcement learning without reconstruction. arXiv preprint arXiv:2006. 10742, 2020.
[27] N. Rudin, D. Hoeller, P. Reist, and M. Hutter. Learning walk in minutes using massively parallel deep reinforcement learning. In Conference on Robot Learning, pages 91-100. PMLR, 2022.
[28] H. Karnan, E. Yang. Farkash, G. Warnell, J. Biswas, and Stone. Sterling: Self-supervised terrain representation learning from unconstrained robot experience. In 7th Annual Conference on Robot Learning, 2023.
[29] J. Long. Z. Wang, Q. Li, L. Cao, J. Gao, and J. Pang. Hybrid internal model: Learning agile legged locomotion with simulated robot response. In The Twelfth International Conference on Learning Representations, 2024.
[30] V. Makoviychuk, L. Wawrzyniak, Y. Guo, M. Lu, K. Storey, M. Macklin, D. Hoeller, N. Rudin, A. Allshire, A. Handa, et al. Isaac gym: High performance gpu-based physics simulation for robot learning. arXiv preprint arXiv:2108. 10470, 2021.
10

---

A Appendix
A.1 Self-learning Latent Representation Details
Algorithm Self-learning Latent Representation
1: Randomly initialize adaptation module Ø, transition model μ., and policy π
2: Initialize empty replay buffer D
3: for itr 1.2.... do
4: On < env.reset()
5: for = 0, 0..,... T do
6:
2t t x((o²)
7:
Or t- (0;;, =z)) 0t+1, t cnv.step(ar) Store ar, Tr: in end for
8:
9:
10:
2++1 t ((0 2+11 t p1l2, at), 24tn RandomSample(D)
12:
Compute Lmp (Eq.5) and Cppo [24]
Update n, jr, by optimizing Lppo + aLuup
14: Empty D
end for
We presented the details of the Self-learning Latent Representation in Section 3.2 of the main paper. We set H/ to 10 and the triplet coefficient a to 1.0.
A.2 Reward Terms Detail
In Table Al, is the linear velocity, o is the tracking shaping scale equal 0.25 here, hurge is the desired base height corresponding to ground, tarper and p' are the desired feet position and real feet position in z-axis of robot's frame and vay is the feet velocity in xy-plane of robot's frame.
Reward
Equation
Weight
Powers
| 2
-2e-5
Linear velocity tracking
exp
cmd
Uay
1.0
Angular velocity tracking
exp
0.5
Linear velocity penalty z-axis
-2.0
Angular velocity penalty
||Wag 177
-0.05
Joint acceleration penalty
|100
-2.ee7
Base Height penalty
(h larget
h)
-10.0
Joint torques
|||
Action rate
|1a Q--10 113
-0.01
Action smoothness
[g 20, + a2221
-0.11
Foot clearance
Σi D- (p. Largel
p²))
-0.01
Orientation
11.1113
-022
Table A1: Reward Terms

---

A.3 Domain Randomizations
Parameters
Min,
Unit
Body Mass
nominal value
Kg
Link Mass
nominal value
Kg
CoM
(..001/){(0.110.0x)(000000. [-1.3]
m
Payload Mass
Kg
Ground Friction
[0.2,2.75]
Ground Restitution
[0.0,1.0]
Motor Strength
1.28 [ motor torque
Nm
Joint Kr
[0.8, x2
Joint K2
x value
Initial Joint Positions
rad
System Delay
[0.3At]] [,0300]x{=33,330x[{33303
External Force
N
Table A2: Domain Randomizations and their Respective Range
A.4 Network Architecture
Module
Inputs
Hidden Layers
Outputs
Encoder
ои off ,t off ,t
[256, 128] [512, 256, 128] [512, 256, 128]
At
Actor
Og
Critic
TransModel
2x a0t
[256, 128]
R++1
Table A3: Network Architecture
A.5
Hyper Parameters for Training
Hyperparameter
Value
Clip range Entropy coefficient
0.2
0.01
Discount factor
0.99
GAE discount factor
0,95
Desired KL-divergence
0.01
Learning rate
le-3
Adam epsilon
1e-8
Replay Buffer Size
4096x24 24
Triplet loss coefficient
1.0
Table A4: Hyper Parameters for Training
12

---

A.6 Terrain Level Training Curves
In the Baseline [27] and HIM [29] code repositories, the default configuration includes a terrain curriculum. Consequently, some agents may progress to harder terrains, potentially resulting in lower returns. To verify the proposed algorithm's actual performance, terrain level training curves have been added, shown in Figure 8.
Considering Figure 6 and Figure together, it is evident that the actual performance of the SLR algorithm surpasses both of these methods.
6.0
Terrain Level
Terrain Level
4.0
4.0
2.0
2.0
SLR(ours) Baseline
SLR SLR(ours)
HIM
1000
2000
500
1000
1500
Iterations
Iterations
Figure 8: Terrain level training curves for Baseline and HIM code repositories.
13


Reviews:
[{"review_id": "HBaa99LSgL", "reviewer": "Reviewer_ztJ4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposes SLR, a representation learning technique for RL locomotion based on triplet optimization.", "summary_of_recommendation": "I like the idea of contrastive learning to form a better latent representation during RL training. The authors should clarify the significance of comparing training reward curves in these environments and revise the results for clarity. Comparison to a recent/concurrent related work would strengthen the paper's impact.", "technical_quality": 3, "clarity_of_presentation": 3, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "- Figure 6 reports the reward curves for various repositories. Is this the reward from the training rollouts or in a fixed evaluation environment? Some of these environments have a performance-dependent curriculum, e.g. on velocity or terrain, so that a higher training reward does not necessarily reflect higher performance, since the robot might be training on easier terrains or tasks. I believe Table 1 provides a fair comparison evaluating Author vs SLR on a fixed evaluation environment which would be correct, but this should be explicitly discussed. The authors should add some discussion on this to indicate what can be interpreted from each result.\n- Similarly, Figure 3 reports reward curves that I expect are based on training returns; but my understanding of [27] is that there is a terrain curriculum, where some policies may advance to significantly harder terrains, and might also be responsible for a lower return. The authors should add (1) plot of the terrain level progression during training, and (2) evaluation of the policies on a fixed evaluation environment (e.g. all level 5 terrains)\n- Table 2 shows that SLR can traverse terrains well, but it should be noted that the MoB and RLvRL methods are trained on flat ground. It would be more straightforward to emphasize the performance of ablations of SLR (SLR w/ explicit privileged latent, SLR w/ implicit privileged latent, SLR w/o latent) in one table and the performance vs. other controllers (MoB, RLvRL, MPC) in a separate table, and I would find the ablation table much more interesting. The naming of \"Baseline\" is not descriptive and is also placed far from SLR which makes it harder to interpret what I want to know from the current table.\n- Comparison with DreamWaq would strengthen the paper, since it also claims to learn a better latent representation for locomotion; but I know there is no official code release so this might take more work to provide.\n- https://arxiv.org/abs/2312.11460 also proposes a contrastive-based approach to legged locomotion. On one hand, the code was only released after the CoRL submission deadline so this might be considered a concurrent work. But I think comparing HIM vs SLR would strengthen the impact of the paper because I would know which contrastive technique yields better performance.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TAlF9MWz96", "reviewer": "Reviewer_fW7r", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 4, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposes a Self-learning Latent Representation (SLR) method for robust quadrupedal locomotion over complex terrains. Unlike state-of-the-art (SOTA) learning-based methods, which rely on meticulously chosen privileged information in simulations to learn a latent representation of the physical properties of the environment, SLR self-learns these latent representations guided by state-transitions, state distinctions, and cumulative rewards under a Markov Decision Process (MDP). The authors demonstrate successful deployment of the learned policy both in simulation and on real robots across various complex terrains. Additionally, they compare the effectiveness of the SLR method against four SOTA open-source references, highlighting its robustness and adaptability in real-world applications.", "summary_of_recommendation": "I would like to recommend this paper for acceptance. This paper presents a significant and original contribution to the field of learning-based locomotion for quadrupedal robots. The novel approach of self-learning latent representations without relying on privileged information simplifies the training process and demonstrates robust performance across various complex terrains. While there are areas for improvement as mentioned in the weakness and rebuttal section, the strengths of this work make it a valuable addition to the research community.", "technical_quality": 4, "clarity_of_presentation": 4, "potential_impact": 4, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "**Issues**\n\n1. In Section 3.2, please provide more details of the neural networks used in your framework, either in the manuscript or supplementary material. Specifically, what is the dimension of 𝑧𝑡 ? What is the size of the MLP for the encoder and the state transition model?\nIn line 112, please clarify how the state transition model and encoder share the same network structures despite having different inputs.\n2. What is the size of the Replay Buffer?\n3. In the loss function, what is the value of the margin 'm' used?\n4. In Section 5 (Results), please consider citing the acronyms of reference methods for clarity.\n5. What was the reason that RMA [10] was not considered a potential reference for comparison? While RLvRL [9] uses RMA to train its policy, [9] is more biased towards rapid locomotion tasks. It would be interesting to see a performance comparison against RMA [10] as it has also shown similar robust behaviors in indoor and outdoor environments.\n\n[9] G. B. Margolis, G. Yang, K. Paigwar, T. Chen, and P. Agrawal. Rapid locomotion via reinforcement learning. The International Journal of Robotics Research, 43(4):572–587, 2024.\n\n[10] A. Kumar, Z. Fu, D. Pathak, and J. Malik. Rma: Rapid motor adaptation for legged robots. 2021.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0cakKwTx3B", "reviewer": "Reviewer_Bcyx", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "Incorporating privileged information to learn a latent vector encoding various environmental parameters or using it to estimate hard-to-compute parameters have proven to be very effective techniques in the domain of quadrupedal locomotion. The given paper operates in the blind locomotion setting and tries to estimate these environmental parameters without explicitly using privileged information. The proposed method instead relies on using just proprioceptive information to encode the observation history into a latent embedding and learning a state transition estimator for the latent. This latent embedding allows the learned policy to operate in different terrains and transition smoothly between them, and be robust to external disturbances. The authors term this the Self-learning Latent Representation (SLR) method.", "summary_of_recommendation": "While the authors show improvement in performance compared to the baselines, the main differentiating factor lies in the interpretability of the latent. It might be interesting to extend it to more dynamic and agile behaviors, especially trying to transition between such motions. The paper could be improved further by providing additional details about the implementation of the proposed method and a slightly more thorough substantiation of the claims.", "technical_quality": 3, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 2, "questions_for_rebuttal": "* It would be great if the authors could provide more details about the training process for triplet optimization. For example, it is unclear how the other latents are sampled to enforce distinctiveness, or what the frequency of updates for the encoder is. It might be nice to explicitly show the overall sequence in the form of pseudocode or an algorithm. This will help with reproducibility.\n* Given that some of the papers cited by the authors report more more dynamic behaviors (such as Extreme Parkour and Rapid locomotion via reinforcement learning), it might be nice to have higher target velocity ranges, especially for comparison against the MoB and RLvRL baselines which seem to have better linear velocity tracking but worse angular velocity tracking than the proposed method. \n* Another minor suggestion is to cite the respective paper with the baseline in section 5.1 as this will further help improve clarity.\n* While the real-world deployment performance is impressive, it would further help to explicitly specify the success criteria and relevant metrics (such as distance covered before a fall or time before a fall etc) for further transparency.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lM1fTdYFvd", "reviewer": "Reviewer_b8HJ", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper aims to propose a quadrupedal locomotion learning framework that does not need privileged information during training stage.", "summary_of_recommendation": "The paper is more convincing as new results and explanation provided during rebuttal.", "technical_quality": 3, "clarity_of_presentation": 3, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "I have no question for rebuttal.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: I agree with the majority of the reviewers and recommend the acceptance of the paper. The reviewers are mainly positive about the paper (3 out of 4 weak accept). The main strength of the paper are
* the simplicity of the algorithm which does not rely on privileged information or student-teacher methods
* the fair evaluation of the algorithms within the original environments
* the real-world experiments of indoor and outdoor environments

I want to especially highlight the fairness of the evaluation as the author used the original environments of the previously published works and evaluated their methods under their conditions. The main weakness of the paper is the limited (potentially marginal) improvement over prior art and the debatable novelty as pointed out by reviewer b8HJ, who was initially not very constructive but improved during the rebuttal. While one could argue in either direction for novelty, I personally would consider the work novel as the exact combination has not been shown in the literature! 

I recommend accepting the paper because it proposes a simple algorithm that works on the physical system and the paper provides a fair comparison to prior art. I am very certain in my evaluation as the paper was reviewed by many reviewers that extensively published on the topic.
Decision: Accept