SoloParkour: Constrained Reinforcement Learning for Visual Locomotion from Privileged Experience
Authors: Elliot Chane-Sane
Joseph Amigo
Thomas Flayols
Ludovic Righetti
Nicolas Mansard
Venue: CoRL 2024 (2024)
Keywords: Reinforcement Learning
Agile Locomotion
Visuomotor Control
TL;DR: A novel method to train visual locomotion policies end-to-end with constrained reinforcement learning

Abstract: Parkour poses a significant challenge for legged robots, requiring navigation through complex environments with agility and precision based on limited sensory inputs.
In this work,  we introduce a novel method for training end-to-end visual policies, from depth pixels to robot control commands, to achieve agile and safe quadruped locomotion.
We formulate robot parkour as a constrained reinforcement learning (RL) problem designed to maximize the emergence of agile skills within the robot's physical limits while ensuring safety. 
We first train a policy without vision using privileged information about the robot's surroundings. 
We then generate experience from this privileged policy to warm-start a sample efficient off-policy RL algorithm from depth images.
This allows the robot to adapt behaviors from this privileged experience to visual locomotion while circumventing the high computational costs of RL directly from pixels.
We demonstrate the effectiveness of our method on a real Solo-12 robot, showcasing its capability to perform a variety of parkour skills such as walking, climbing, leaping, and crawling.

SoloParkour: Constrained Reinforcement Learning for Visual Locomotion from Privileged Experience
Elliot Chane-Sanc" Joseph Amigo*12 Thomas Flayols', Ludovic Righetti Nicolas Mansard 13
'LAAS Université de Toulouse, CNRS, Toulouse, France Machhnes in Motion Laboratory, New York University, USA
Artificial and Natural Intelligence Toulouse Institute, Toulouse, France https: tttps://gepetto. gi github... io/SoloParkour/
Figure 1:
The open open-hardware quadruped robot Solo- 12 performs agile skills that are reminiscent of parkour, such as walking, climbing high steps, leaping over gaps, and crawling under obstacles.
Parkour poses significant challenge for legged robots, requiring navi- gation through complex environments with agility and precision based on limited sensory inputs. In this work, we introduce novel method for training end-to- end visual policies, from depth pixels to robot control commands, to achieve agile and safe quadruped locomotion. We formulate robot parkour as a constrained reinforcement learning (RI.) problem designed to maximize the emergence ag- ile skills within the robot's physical limits while ensuring safety. We first train a policy without vision using privileged information about the robot's 's surroundings. We then generate experience from this privileged policy to warm-star sample ef- ficient oof-po RL algorithm from depth images. This allows the robot to adapt behaviors from this privileged experience to visual locomotion while circumvent- ing the high computational costs RL directly from pixels. We demonstrate the effectiveness of our method on real Solo-12 robot, showcasing its capability perform a variety parkour skills such as walking. climbing. leaping, and crawl- ing.
Abstract:
Keywords: Reinforcement Learning, Agile Locomotion, Visuomotor Control
'Equal contribution. Correspondence elliot. chane-sane@laass
8th Conference Robot Learning (CoRL. 2024), Munich, Germany.

---

Introduction
In recent years, training locomotion policies in simulation using reinforcement learning (RL) then transferring them to real robots has proven highly effective in mastering agile skills [1, 2. 3, 4, 5, 6, 7, 8, 9, 10). Notably, [11, 12, 13, 14] have successfully demonstrated a range of dynamic skills akin the athletic maneuvers seen in parkour, including walking, climbing, jumping, and crawling, which necessitates tight coordination between vision and control. In this work, we are interested in developing parkour skills from onboard depth inputs for Solo- 12, a lightweight, open- hardware quadruped robot [15], using this sim-to-real approach. Solo is a lighter prototype than other quadruped robots experimented in parkour projects (e.g. Atlas, Go-1, or Anymal). Safe visual locomotion with Solo is then at its highest stake, to prevent exceeding the limited torque range of its motors and breaking the 3D-printed plastic shells and exposed electronic components in unexpected impacts or falls.
Yet, training visual locomotion policies presents significant challenges due to the limited field of view and dynamic, lag-prone visual perception during dynamic movements. Previous methods for end-to-end control from pixels [16, 12. 13] typically involve a two-stage process, where a policy is first trained with privileged information about the robot's 's surroundings, then the learned behaviors are distilled into a visual policy using imitation learning [17]. However, such an approach relies on the unrealistic assumption that privileged information can be fully reconstructed from a history of depth images. Indeed, such privileges may reveal information obscured behind obstacles, outside the field of view of the egocentrie camera, simply missing from the depth image stream due to lags. This gap between the privileged information and the actual visual inputs may result in behaviors that a vision-based policy cannot replicate accurately. In that case, an optimal vision policy would differ from the one using privileged information. It would, for example, try to gather more information before crossing an obstacle to handle the occlusion. Ideally, training RL directly from pixels would ensure the policy learns behaviors effectively based on its actual visual sensors, but this method is impractical due the high computational costs associated with generating depth images in simulations [16].
To address these challenges, we propose SoloParkour, a novel approach for training parkour lo- comotion policies using depth inputs. Our method frames parkour as a constrained reinforeement learning problem [18, 19, 20]. This framework allows the RL agent to explore the full capabilii ties of the robot while explicitly preventing unsafe actions, ensuring that the agent can aggressively optimize performance without compromising the safety of the robot. We propose a novel method to train the visual locomotion policy end-to-end from pixels using sample-efficient off- off-policy RL algorithm. Building upon recent advancements in accelerating RL using demonstrations from pre- vious controllers [21, 22], our algorithm not only learns from its own experiences but also leverages experience generated by the privileged policy. This approach enables the RL agent to rapidly ac- quire agile skills and adapt behaviors from the privileged experience to suit the specific limitations of its visual sensors, bypassing the computational burden associated with RL from pixels while still ensuring the development of agile and safe legged locomotion skills.
We demonstrate the effectiveness of our approach in simulation. We then directly deploy the learned policy on real Solo-12 12 robot and demonstrate the effective acquisition of parkour skills such as crawling, lcaping, and climbing obstacles from depth image (see Figure 1). Our approach pushes the robot to its limits: it manages to clear obstacles 1.5 times higher than its height despite the robot being significantly less powerful than the ones typically used in parkour experiments.
In summary, our contributions are as follows:
we cast parkour learning from depth images as a constrained RL problem,
we introduce computationally efficient RL algorithm to train end-to-end visual locomo- tion policies with significant improvement over methods based on distillation
and we validate the effectiveness of our approach in simulation and real Solo- 12 robot to perform parkour skills, outperforming the best movements ever generated with this robot

---

and reaching performances comparable to recent parkour achievements despite more lim- ited actuation range.
2 Related Work
Agile Locomotion RL has demonstrated tremendous success obtaining robust and adaptive controllers for legged robot [23, 3, 24, 5, 25J. This includes agile skills such as high-speed run-
ning [6, 11, 26, 8J, recovering from falling [27, 28, 29, 9]. jumping [2, 4, 7. 22, 30, 31, 32, 33], climbing obstacles [1. 34, 35, 36, 37, 16, 38, 39, 20, 40, 14, 12, 13] bipedal walking with quadruped robots [41, 30, 22, 13, 42] and walking inside confined spaces [43, 44, 40]. Learning multi-skill locomotion policies, as required in parkour, can be done by training separate policies for each skill, then coordinating them with high-level planner [28, 45, 46, 14] or distilling them into a single policy [11, 12, 42]. Instead, we follow [13] and learn multi-task policies directly through RL.
Safe Locomotion Safety mechanisms have been implemented to ensure safer outcome while per- forming agile skills [47, 8]. Following [18, 19, 20), we employ constraints alongside rewards in RL to deter undesired behaviors. This only allows for aggressive optimization of agility while ensuring safety but also simplifies the process of reward tuning for RL. In particular, we exploit the reformulation of Constraints as Terminations [20] (CaT), which was demonstrated as an overhead on top of on-policy RL algorithms [48], and which we extend to the off- off-policy formulation, more suitable to our case as explained below.
Sample-Efficient Learning [49, 22] accelerate RL with demonstrations obtained from trajectory optimization. Other works aimed exploring RL methods sample efficient enough train loco- motion policies directly in the real-world [29, 50]. Our approach repurposes many of these designs to bypass the computational cost of RL from pixels while obtaining pure end-to-end RL method, unlocking several advantages exhibited below.
Vision-Based Locomotion Prior methods often separate perception from control using inter- mediate representations such as elevation maps [51, 52, 53, 54, 55, 5, 35, 56, 14, 20], trace- ability maps [57, 58] or visual odometry [59, 60, 61, 62], for downstream planning and con- trol [63, 64, 65, 66, 67, 68]. Recently, locomotion from pixels [69, 4, 39, 16, 12, 13] has emerged as powerful paradigm that more tightly coordinates vision and control, often relying on teacher-student approaches (typically by distilling an observvtioon-rrivilgeed policy), yet raising some limited action- perception behaviors.
3 Method
3.1 Agile and Safe Parkour Learning Problem Formulation
Our goal is train parkour policy in simulation using RL and transfer it a real Solo- quadruped robot. To this end, consider an infinite, discounted, constrained Markov Decision Process A,,,A, T,er) with state space S, action space A, reward function : S A R, discount factor y, dynamics T S x A + S and constraints {c: S x A - R, i E 7). Constrained RL aims to find a policy : S - A that maximizes the discounted sum of future rewards:
max EA
Σ
r ar) r=
(1)
while satisfying the constraints Ger under the state-action policy visitation distribution:
PP ((.,))=7777 (c((s,)) a) 0] S <; E
where any value of cr(s, a) above 0 corresponds to the magnitude of the violation of the i-th straint by taking action in state
3

---

(a) Crawl
(b) Climb
(c) Leap
(d) Hurdle
Terrains used to train SoloParkour in simulation: the crawl parkour contains floating objects the robot must crawl under, the step and hurdle parkour contain obstacles for the robot to climb up and down, and the leap parkour contains gaps over which the robot must leap.
Figure 2:
history of proprioceptive measurements of the positions and velocities of all 12 joints of the robot and of the orientation and angular velocity of the base of the robot, of previous action samples, command vector an in the x-y space given by the user, and of depth images 'o...." The action d, corresponds to desired joint position offsets with respect to a default joint configuration, that are then converted to torques through proportional- derivative (PD) controller operating at a higher frequency than the neural policy.
States and Actions The state & corresponds
Rewards We use similar reward function to [13] that measures progress towards a direction below a commanded velocity threshold (yet without the additional burden of defining subgoals):
cmd
miii(i;
cmd
|
where v is the velocity the base relatively in the robot frame. We found that this design was simple while being sufficient for the emergence of agile locomotion skills.
Constraints Greedily maximizing Eq. leads to obvious impractical behaviors, in particular exceeding the real robot capabilities and impossible sim-to-real transfer. To achieve safe and transferable behavior, enforce set of constraints o that the robot should adhere to. We use straightforward constraint formulations to limit the torques, accelerations, velocities and positions of the joints, avoid unwanted contacts between the robot and the terrain, encourage the robot to head toward the commanded direction and push a specific walking style. For instance, constraints for the torque on the k-th joint and for the orientation of the base along the roll axis can be respectively written as:
Cuoquee, |T&l lim and Carima
|ori
or
where 7 lim and ori are limits that the robot should avoid exceeding. The detailed formulations of constraints are provided Appendix A. We found that the constrained RL formulation, as opposed to reward penalties usually done in RL for locomotion, was crucial for the effective and safe transfer of agile locomotion skills on our real Solo-12 robot while being easier to tune.
Terrains The policy trained to traverse diverse challenging terrains in the IsaacGym simula- tor [70] to learn variety of agile locomotion skills, as illustrated in Figure 2. The focus of this work being on parkour, we consider learning locomotion skills such as walking, climbing, leaping, and crawling with four terrain types illustrated in Figure 2. single policy is jointly trained curriculum of variations of these terrains with increasing difficulty [5].
3.2 Visual Policy Learning
Ideally, we would like to train the policy from depth pixels to actions using deep RL in order to leamn behaviors that best use the limited sensory inputs from the depth cameras. However, directly training the vision policy end-to-end from scratch with RL is impractical, as depth images are costly to render simulation. An alternative approach is to first train a policy that has access to cheap- to-compute information only accessible in simulation about the terrain surrounding the robot, then distill the behaviors learned from this privileged information into visual policy observing a history of depth images using imitation learning [16, 13, 12]. While the experimental setup can be chosen

---

Stage Reinforcement Learning with Privileged Information
Stage Visuomotor Reinforcement Learning from Privileged Experience
PPO +CaT
DDPG +aaT
Ceiling height
Online Replay
Buffer
Privileged Pollcy pils
D
Visual Policy
Depth Images
Privileged Experience
Hoightmap scan,
propriacaption + commands...
propriacaption commands...
Figure 3: SoloParkour leverages two-stage RL approach to train visual locomotion policies in simulation. Stage I: we train privileged policy that observes heightmap scan of its surroundings and the height of the nearby floating objects using PPO with Constraints as Terminations (CaT) [20]. Stage 2: we train policy from depth pixels using variant of DDPG with CaT that learns from dataset of privileged experience collected using the Stage I policy.
to reduce the distillation gap (e.g. by choosing specific sensors providing extensive observability), there is in general no guarantee that the learned behaviors will transfer well to the visual policy due to the observability gap between the privileged information and the visual sensors. For instance, elevation maps may contain information about surfaces that are not visible from the camera inputs. We then propose generic solution mitigating these two problems.
Our approach, SoloParkour, is illustrated in Figure 3. Following prior works [16, 12, 13), first learn privileged policy that has access to cheap-to-compute information about the terrain sur- rounding the robot. Unlike [16, 12, 13], we propose use this privileged policy to warm-start the reinforcement learning of an end-to-end visual policy. This way, we let the RL agent decide on the best behaviors given its limited sensory inputs while bypassing the computational complexity of training from depth images.
Stage Privileged Policy Learning We first train a privileged policy πFn that has access to cheap-to-compute information about the terrain surrounding the robot. In place of histories of depth images and proprioception, the robot observes the privileged state <!tiv that includes the current proprioceptive measures, velocity commands, and the previous action well of its surrounding and the ceiling height. In crawl terrains, the ceiling height corresponds to the distance from the ground to levitating block when such a block is near the robot. In other terrains, or when no levitating blocks are nearby, the ceiling height is set to an arbitrarily high default value. We found that this provides sufficient information about the terrain geometry to train the privileged policy for the proposed terrain tracks.
height-scan map
The privileged policy is parameterized by Multi-Layer Perceptron (MLP) trained using Proximal Policy Optimization [48] (PPO), an on-policy RL method widely used in learning based locomotion. We use CaT [20] to enforce the constraints.
Stage 2: End-to-End Vision-Based RL from Privileged Experience To overcome the com- putational complexity of RL from pixels, we propose to adapt Reinforcement Learning with Prior Data [21] (RLPD) to transfer the experience from the privileged policy to visual locomo- tion and learn from depth images in a sample efficient manner. We build on design principles from [29, 21, 22] and implement off-policy RL with high update-to-data ratio. Compared to on-policy approaches such PPO, this minimizes the number of depth image rendering steps in simulation in favor of increased updates from the privileged and online experience.
More precisely, we employ a variant of Deep Deterministic Policy Gradient [71] (DDPG) that uses two replay buffers: its online replay buffer Donline and a privileged experience buffer Dpriv Deriv constructed by employing the fully trained ,ivv from Stage to generate demonstrations that in- clude the depth modality ddpph During policy learning, batches of experiences are sampled in equal proportion from Donline and Deriv to train the policy and its Q-function. We use REDQ [72] and
5

---

Layer Normalization [73] to stabilize Q-learning at high update-to-datee ratio [74, 21, 22]. Impor- tantly, while the visual actor π is trained end-to-end from a history of depth images, we found that training the critic from privileged state si rather than on the full state &t in an asymmetric actor- critic fashion [75] was faster and more stable. We incorporate CaT [20] an off-policy manner to learn visual locomotion that keeps satisfying the constraints.
The resulting RL algorithm is sample efficient enough to train our visual policies, parameterized by a ConvNet [76] to process depth images individually. Gated Recurrent Unit [77] to handle histories of observations, and an MI. head, with RI directly from pixels in simulation.
Experiments
4.1 Experimental setup
The policies are trained in the IsaacGym [70] simulator using massively parallel environments. The full policy learning pipeline can be trained on a single NVIDIA RTX 4090 GPU in less than 20 hours. After training in simulation, the controller is directly deployed on a real Solo-12 quadruped robot. The policy runs at 50Hzz on a Raspberry Pi 5. Target joint positions are sent to the onboard PD controller running at H We use an Intel RealSense D-405 stereo camera to capture depth images and process them to resolution 48 48, While the depth images are rendered every 5 environment steps in simulation (i.e. 10Hz in the time reference of the simulation), we provide the images at the speed of the depth pipeline on the real robot at 30Hz.
Simulation and robot
When standing, the height of Solo 26cm and its body length is 45cm, which similar to the Unitree Go-1 used [12, 13]. However, Go-1 has a trust-tt-weiihtt ratio 2 to 3 times superior to Solo. Thus, we don't 't expect to overcome obstacles as challenging as [12, 13].
Baselines and ablations We validate our approach in simulation and compare SoloParkour to the following baselines and ablations.
DAgger [17]: the method used in [13, 12, 16] distill the privileged policy into the visual policy using imitation learning through action relabelling with the privileged policy.
Behavior Cloning (BC): distilling the privileged policy by training the visual policy directly with supervised learning on the privileged experience Ppriv-
Privileged Reconstruction: training the vision module reconstruct the privileged infor- mation from the history of depth and proprioceptive inputs, then reemploy the Stage policy based on these reconstructions, aimed to resemble [16, 56).
From Scratch: ablation of our approach where we train the visual policy with RL from scratch, without privileged experience D Privv
No Priv. Critic: an ablation of our approach where the critic of Stage observes histories of depth images instead of the simplified privileged information.
Visual RL w/o CaT: an ablation of our approach where we train the visual RL policy with- out constraints (but from experience from the constrained Stage 1 policy).
We first train a privileged policy from Stage Then, except for BC which is much faster to train as it doesn't require querying the simulator, we train all these baselines and ablations with the same computational budget. Finally, we compare their performance by executing the learned policies in simulation and measuring the distance traveled in the terrains of Figure 2.
4.2 Simulation Experiments
Comparison to supervised distillation In Figure 4a, we compare SoloParkour against DAgger, BC and Privileged Reconstruction. For comparison, we also report the performances of the privi- leged policy used to train these methods. SoloParkour performs roughly as well as the privileged policy on the hurdle, step, and crawl tracks and marginally worse on the lepp track. It outperforms the supervised distillation baselines on all terrains and at almost all levels of difficulty. The differ- ence is particularly significant against DAgger and BC on the leap terrains, where the robot must

---

Climb
Crawl
Hurdie
Leas
100
Completinn (%C
Obstaule treight (um)
Ceiling frerghh (cr))
Obssaatle Tre ghil (cm)
Gop length (cm)
Privileged policy
Degger
Privileged Reconstruction
SoloParkour
(a) Comparison between SoloParkour and imitation- distillation methods.
Completion (%)
100
50
10
20
30
40
34
32
30
28
26
10
20
30
20
40
60
Obbstacle beight (cm)
Ceiling height (cm)
Obstacle The glill (sm)
lengith (cin)
No Privi. Critic
-
From Scratch
Visual RL we CaT - SoloParkour
(b) Ablations of the proposed methods.
Figure 4: Average terrain completion (over 4 training seeds) by obstacle dimension for each policy.
perform complex motion to overcome the gaps, requiring perfect timing succeed. On these ter- rains, DAgger and BC often fail to propel themselves enough and miss the edge of the landing side of the platform, or ignore the gap and fall into it. Meanwhile, Privileged Reconstruction struggles on the crawl and climb terrains, where occlusions hinder accurate reconstruction of the surrounding terrain geometry. We attribute these differences to our end-to-end training from pixels with the RL objective, which results tighter coordination between vision and control, understood as a local action- perioppper refinement.
Ablative analysis In Figure 4b, examine the importance of two design choices for SoloParkour given the same computation budget. Training the visual policy from scratch without privileged experience (from Scratch) is much less sample-efficient and achieves poor performances overall, validating the importance of transferring privileged experience from the Stage policy. Having the critics process the full-depth observation instead of only the privileged information (No Priv. Critic) causes additional computation overheads during Q-learning, which are yet unnecessary to train good visual policies our setting. As a result, given the same computation budget, No Priv. Critic processes fewer samples and achieves lower performances than the full SoloParkour method.
Constraints satisfaction In Figure 5, examine the vio- lation of the torque constraint for the front left shoulder joint and the base orientation constraint, as outlined in (4). We fo- cused exclusively on successful trajectories where the robot managed to navigate the entire level order to exclude data from extreme states, such as when the robot is falling outside of the track or gets endlessly stuck on a high obstacle. Hence, report results for only for DAgger, as it fails to overcome larger gaps. Thanks to off-policy CaT, SoloParkour maintains constraint compliance effectively after visuomotor RL, with torque violations under 4% and base orientation violations under 10% on most levels, despite the highly dynamic skills involved to overcome the obstacles. These results are comparable to those of the privileged policy and supervised distillation baselines. By contrast, Visual RL w/o CaT achieves higher terrain completion rates but at the cost of significantly increased constraint violations, ren- dering the policies unsuitable for real-world robot transfer. Additional results on the satisfaction of broader set of constraints are discussed in Appendix C.
Step
Leap
Crawl
Average
85%
70%
100%
85%
Table 1: Success rates achieved by the policies for different obstacles on the real robot, averaged over training seeds and 10 trials per ob- stacle per seed.
gap lengths the leap track
7

---

Climb
Crawl
Hurdie
Leap
Constraint violation (%)
20-
10-
32
Obslatle freight (em))
Celling
treight (Lm)
Obrkkacle Treig it (ur)
Gap length (cm)
Privileged potcc
Dagger
Privlegeed Recorstruction
SoloParkour
(a) Constraint for the basc orientation around the roll axis
Constraint vialation (%)
Climb
Loap
20
10-
0-
10
20
30
40
34
32
30
28
26
10
20
30
20
40
60
Obstacle heigght (crn)
Ceiling heightt (cm)
Obskacle Trigg (cm)
length (cm)
Privlleged policy
Dagger
-
BC
Visual RL WD CaT
Privileged Reconstruction
SoloParkour
(b) Torque constraint for front left knee joint
Figure 5: Constraint violations (in %) when the policies successfully traverse the terrain level by
obstacle dimension, averaged across 4 training seeds.
4.3 Real-Robot Experiments
We deploy SoloParkour directly the real Solo-12 12 and evaluate it to traverse challenging obstacles that involve climbing, crawling, and leaping (see Figure 1). We found that the policy handled the depth vision signals sent by the camera, responding synchronously to obstacles at the right time. Table I reports the results achieved by SoloParkour. Our approach successfully climbs a 40cm step (1.5 times the robot's 's height), leaps over a 35cm gap (78% its length), and crawls under obstacles 20cm above the ground with high repeatability.
Limitations
The experiments on the real hardware have revealed that our policy, while respecting the motor limitations, is likely hitting the battery current limits. This issue was not anticipated, hence not modeled neither in the simulation nor in the imposed constraints. It likely was amplified during the course of the experimental study by damaging the batteries when exceeding their capabilities. An exciting direction is to properly model this physical effect in the constrained-Min which has the potential to improve the performances of all other robots in future parkour research.
Moreover, current parkour learning approaches, including ours, require that simulation terrains be manually constructed for each specific skill. Consequently, robots only learn new skills by designing types of terrain. Future work could explore procedural or generative simulators to create more diverse and realistic environments for training locomotion policies and transition from baseh to RGB vision for legged robots.
6 Conclusion
We introduced SoloParkour, a novel method that combines constraints and sample-efficient RL from privileged experience to train agile and safe locomotion for legged robots end-to-end from depth pixels. We found that constraints simplify the work of designing the MDP while leading to more consistent results. The end-to-end resolution leads better training convergence compared to previ- teacher-student approaches. We also experimentally brought the lightweight robot Solo- 12 closer to its limit than ever before, achieving various parkour stages despite severe actuator limitations.

---

Acknowledgements
This work was funded in part by ANITI (ANR-19-P31A-0000) COCOPIL (Région Occitanie, France), PEP# O2R (AS2 ANR-2--EXOD-006) Dynamograde (ANR-21-LCV3-40022, ROBO- TEX 2.0 (ROBOTEX ANR- SUR-I0-EQPX-44440 and MIRRE--ANR-J--ESR--OO Science Foundation (grants 2026479, 2222815 and 2315396). It was granted access the HPC resources of IDRIS under the allocations AD011012947 and AD01 1015316 made by GENCI.
and the National
References
[1] J. Lee, J. Hwangbo, L.. Wellhausen, V. Koltun, and M. Hutter. Learning quadrupedal locomo- tion over challenging terrain. Science robotics, 5(47)xcabc5986, 2020.
[2] G. Bellegarda, C. Nguyen, and Q. Nguyen. Robust quadruped jumping via deep reinforcement learning. arXiv preprint arXiv:2011.07085 2020.
[3] Fu, Kumar, J. Malik, and Pathak. Minimizing energy consumption leads to the emer- gence of gaits in legged robots. In Conference Robot Learning (CoRL), 2021.
[4] G. B. Margolis, T. Chen, K. Paigwar, X. Fu, D. Kim, S. Kim, and P. Agrawal. Learning to jump from pixels. arXiv preprint 1iX.::2110.15344 2021.
[5] N. Rudin, D. Hoeller, P. Reist, and M. Hutter. Learning to walk in minutes using massively parallel deep reinforcement learning. In Conference on Robot Learning, pages 91-100. PMLR, 2022.
[6] G. Bellegarda, Y. Chen, Z. Liu, and Q. Nguyen. Robust high-speed running for quadruped robots via deep reinforcement learning. IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2022.
[7] Z. Li, X. B. Peng, Abbeel, S. Levine, G. Berseth, and K. Sreenath. Robust and ver- satile bipedal jumping control through multi-task reinforcement learning. arXiv preprint arXiv:2302.09450, 2023.
[8] T. He, C. Zhang, W. Xiao, G. He, C. Liu, and G. Shi. Agile but safe: Learning collision-free high-speed legged locomotion. arXiv preprint arXiv:2401.17583, 2024.
[9] S. Inoue, K. Kawaharazuka, K. Okada, and M. Inaba. Body design and gait generation of chair- type asymmetrical tripedal low-rigidity robot. 2024 IEEE 7th International Conference on Soft Robotics (RoboSoft), pages 593-600. IEEE, 2024.
[10] Y. Ma, W. Liang, H. Wang, S. Wang, Zhu, L. Fan, O. Bastani, and D. Jayaraman. Dreureka: Language model guided sim-to-real transfer. In Robotics: Science and Systems (RSS), 2024.
(11) K. Caluwaerts, A. Iscen, J. C. Kew, W. Yu, T. Zhang, D. Freeman, K.-H. Lee, I. Lee, S. Sal- iceti, V. Zhuang, et al. Barkour: Benchmarking animal-level agility with quadruped robots. arXiv preprint arXiv:2305. 14654, 2023.
[12] Z. Zhuang, Z. Fu, J. Wang, C. Atkeson, S. Schwertfeger, C. Finn, and H. Zhao. Robot parkour learning. In Conference on Robot Learning (CoRL.), 2023.
[13] X. Cheng, Shi, A. Agarwal, and D. Pathak. Extreme parkour with legged robots. arXiv preprint 113.::::339.14441 2023.
[14] D. Hoeller, N. Rudin, D. Sako, and M. Hutter. Anymal parkour: Learning agile navigation for quadrupedal robots. Science Robotics, 2024.

---

[15] F. Grimminger, A. Meduri, M. Khadiv, J. Viereck, M. Wüthrich, M. Naveau, V. Berenz, S. Heim, F. Widmaier, T. Flayols, et al. An open torque-controlled modular robot architec- ture for legged locomotion research. IEEE Robotics and Automation Letters, 5(2):3650-3657, 2020.
[16] A. Agarwal, A. Kumar, J. Malik, and D. Pathak. Legged locomotion in challenging terrains using egocentric vision. In Conference on robot learning, pages 403-415 PMLR, 2023.
[17] S. Ross, G. Gordon, and Bagnell. reduction of imitation learning and structured predic- tion no-regret online learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pages 627-635. JMLR Workshop and Conference Proceed- ings, 2011.
[18] Lee, L. Schroth, V. Klemm, M. Bjelonic, A. Reske, and M. Hutter. Evaluation constrained reinforcement learning algorithms for legged locomotion. arXiv preprint arX1v:3399114330, 2023.
[19] Y. Kim, H. Oh, Lee, J. Choi, G. Ji, M. Jung, D. Youm, and J. Hwangbo. Not only rewards but also constraints: Applications on legged robot locomotion. IEEE Transactions on Robotics, 2024.
[20] E. Chane-Sane, P.-A. Leziart, T. Flayols, O. Stasse, P. Souères, and N. Mansard. Cat Constraints as terminations for legged locomotion reinforcement learning. In JEFE/SI International Conference on Intelligent Robots and Systems (IROS), 2024.
[21] P. Ball, L. Smith, I. Kostrikov, and S. Levine. Efficient online reinforcement learning with offline data. In International Conference Machine Learning, pages 1577-1594. PMLR, 2023.
[22] L. Smith, J. C. Kew, T. Li, L. Luu, X. B. Peng. Ha, Tan, and S. Levine. Learning and adapting agile locomotion skills by transferring experience. Proceedings of Roboties: Science and Systems, 2023.
[23] X. B. Peng, E. Coumans, T. Zhang, T.-W. Lee, J. Tan, and S. Levine. Learning agile robotic locomotion skills by imitating animals. arXiv preprint arXiv:2004.00784, 2020.
[24] A. Kumar, Z. Fu, D. Pathak, and J. Malik. Rma: Rapid motor adaptation for legged robots. arXiv preprint arXiv:2107.0400 2021.
[25] M. Aractingi, P.-A. Léziart, T. Flayols, J. Perez, T. Silander, and P. Souères. Controlling the solo12 quadruped robot with deep reinforcement learning. Scientific Reports, 13(1):11945, 2023.
[26] G. B. Margolis, G. Yang, K. Paigwar, T. Chen, and P. Agrawal. Rapid locomotion via rein- forcement learning. The International Journal of Robotics Research, 43(4):572-587, 2024.
[27] Hwangbo, J. Lee, A. Dosovitskiy, D. Bellicoso, V. Tsounis, V. Koltun, and M. Hutter. Learn- ing agile and dynamic motor skills for legged robots. Science Robotics, 4(26):eau5572,, 2019.
[28] C. Yang, K. Yuan, Q. Zhu, W. Yu, and Z. Li. Multi-expert learning of adaptive legged locomo- tion. Science Robotics, 5(49))eabb2177 2020.
[29] L. Smith. J. C. Kew, X. B. Peng. S. Ha, J. Tan, and S. Levine. Legged robots that keep learning: Fine-tuning locomotion policies in the real world. In International Conference on Robotics and Automation (ICRA), 2022.
[30] C. Li, M. Vlastelica, S. Blaes, J. Frey, Grimminger, and G. Martius. Learning agile skills via adversarial imitation of rough partial demonstrations. In Conference Robot Learning, pages 342-352. PMLR, 2023,
10

---

[31] R. Yang, Z. Chen, J. Ma. C. Zheng, Y. Chen, Q. Nguyen, and X. Wang. Generalized ani- mal imitator: Agile locomotion with versatile motion prior. arXiv preprint arXiv:2310.01408, 2023.
[32] Z. Li, X. B. Peng, P. Abbeel, S. Levine, G. Berseth, and K. Sreenath. Reinforcement learning for versatile, dynamic, and robust bipedal locomotion control. arXiv preprint arXiv:2401.16889, 2024.
[33] C. Zhang, J. Sheng, T. Li, H. Zhang, C. Zhou, Q. Zhu, R. Zhao, Y. Zhang, and I.. Han. Learning highly dynamic behaviors for quadrupedal robots. arXiv preprint arXiv:2402.13473 2024.
[34] Siekmann, K. Green, Warila, A. Fern, and J. Hurst. Blind bipedal stair traversal via sim- to-real reinforcement learning. arXiv preprint arXiv:2105.0832. 2021.
[35] T. Miki, J. Lee, J. Hwangbo, L. Wellhausen, V. Koltun, and M. Hutter. Learning robust per- ceptive locomotion for quadrupedal robots in the wild. Science Robotics, 7(62))eab,2822, 2022.
[36] D. Hoeller, N. Rudin, C. Choy, A. Anandkumar, and M. Hutter. Neural scene representation for locomotion on structured terrain. IEEE Robotics and Automation Letters, 7(4):8667 -674,, 2022.
[37] N. Rudin, D. Hoeller, M. Bjelonic, and M. Hutter. Advanced skills by learning locomotion and local navigation end-to-end. In 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 2497-2503. IEEE, 2022.
[38] R. Yang, G. Yang, and X. Wang. Neural volumetric memory for visual locomotion control. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1430-1440, 2023.
[39] A. Loquercio, A. Kumar, and J. Malik. Learning visual locomotion with cross-modal super- vision. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 7295-7302. IEEE, 2023.
[40] Y. Cheng, H. Liu, G. Pan, L. Ye, H. Liu, and B. Liang. Quadruped robot traversing 3d complex environments with limited perception. arXiv preprint MxXi:22404.18225 2024.
[41] Y. Fuchioka, Xie, and M. Van Panne. Opt-mimic: Imitation of optimized trajectories for dynamic quadruped behaviors. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 5092-5098. IEEE, 2023.
[42] X. Huang, Y. Chi, R. Wang, Z. Li, X. B. Peng, S. Shao, B. Nikolic, and K. Sreenath. Diffuse- loco: Rime-time legged locomotion control with diffusion from offline datasets. arXiv preprint arXiv:2404. 19264, 2024.
[43] Miki, J. Lee, I.. Wellhausen, and M. Hutter. Learning to walk in confined spaces using 3d representation. International Conference on Robotics and Automation (ICRA), 2024.
[44] Z. Xu, A. H. Raj, X. Xiao, and P. Stone. Dexterous legged locomotion in confined 3d spaces with reinforcement learning. arXiv preprint arXiv:2403.03845 2024.
[45] S. Kim, M. Sorokin, J. Lee, and S. Ha. Humanconquad: human motion control of quadrupedal robots using deep reinforcement learning. In SIGGRAPH Asia 2022 Emerging Technologics, 2022.
[46] X. Huang, Z. Li, Xiang, Y. Ni, Y. Chi, Y. Li, L. Yang, X. Peng, and Sreenath. Cre- ating dynamic quadrupedal robotic goalkeeper with reinforcement learning. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2023,

---

[47] T.-Y. Yang, T. Zhang, L. Luu, S. Ha, J. Tan, and W. Yu. Safe reinforcement learning for legged locomotion. In 2022 IEFE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 1554-2461. IEEE, 2022.
[48] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 1rcXiv:1707... 1707.06377, 2017.
[49] M. Bogdanovie, M. Khadiv, and L.. Righetti. Model-free reinforcement learning for robust locomotion using demonstrations from trajectory optimization. Frontiers in Robotics and AI, 9:854212, 2022.
[50] Wu, A. Escontrela, D. Hafner, P. Abbeel, and K. Goldberg. Daydreamer: World models for physical robot learning- In Conference on Rohot Learning, pages 2226-2240. PMLR, 2023.
[51] A. Kleiner and C. Dornhege. Real-time localization and elevation mapping within urban search and rescue scenarios. Journal of Field Robotics, 24(8-9):223-745 2007.
[52] X. B. Peng, G. Berseth, K. Yin, and M. Van De Panne. Deeploco: Dynamic locomotion skills using hierarchical deep reinforcement learning. Acm transactions on graphics (tog), 36(4): 1-13, 2017.
[53] P. Fankhauser, M. Bloesch, and M. Hutter. Probabilistic terrain mapping for mobile robots with uncertain localization. IEEE Robotics and Automation Letters, 3(4):3019-3026, 2018.
[54] S. Gangapurwala, M. Geisert, R. Orsolino, M. Fallon, and I. Havoutis. Real-time trajectory adaptation for quadrupedal locomotion using deep reinforcement learning. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 5973-5979. IEEE, 2021.
[55] V. Tsounis, M. Alge, Lee, F. Farshidian, and M. Hutter. Deepgait: Planning and control of quadrupedal gaits using deep reinforcement learning. IEEE Robotics and Automation Letters, 5(2):3699-3700 2020.
[56] H. Duan, B. Pandit, M. S. Gadde, B. J. van Marum, J. Dao, C. Kim, and A. Fern. Learning vision-based bipedal locomotion for challenging terrain. IEEE International Conference Robotics and Automation (ICRA), 2024.
[57] R. O. Chavez-Gareia. Guzzi, L. M. Gambardella, and A. Giusti. Learning ground traversabil- ity from simulations. IEEE Robotics and Automation letters, 1695 2018
[58] B. Yang, L. Wellhausen, T. Miki, M. Liu, and M. Hutter. Real-time optimal navigation planning using learned motion costs. In IEEE International Conference on Robotics and Automation (ICRA), pages 9283-9289. IEEE, 2021.
[59] M. Bloesch, S. Omari, M. Hutter, and R. Siegwart. Robust visual inertial odometry using a direct ekf-based approach. In 2015 IEEE/RSJ international conference on intelligent robots and systems (IROS), pages 298-304. IEEE, 2015.
[60] D. Wisth, M. Camurri, and M. Fallon. Robust legged robot state estimation using factor graph optimization. IEEE Robotics and Automation Letters, 4(4):4507-4514, 2019.
[61] D. Wisth, M. Camurri, and M. Fallon. Vilens: Visual, inertial, lidar, and leg odometry for all-terrain legged robots. IEEE Transactions on Robotics, 39(1):309-326, 2022.
[62] R. Buchanan, M. Camurri, F. Dellaert, and M. Fallon. Learning inertial odometry for dynamic legged robot state estimation. In Conference on Robot Learning, pages 1575-1584. PMLR, 2022.
[63] M. Wermelinger, P. Fankhauser, Diethelm, Krüsi, R. Siegwart, and M. Hutter. Navigation planning for legged robots in challenging terrain. In IEEE/RSJ International Conference Intelligent Robots and Systems (IROS), 2016.
12

---

[64] C. Mastalli, M. Focchi, I. Havoutis, A. Radulescu, S. Calinon, J. Buchli, D. G. Caldwell, and C. Semini. Trajectory and foothold optimization using low-dimensional models for rough terrain locomotion. In IEEE International Conference on Robotics and Automation (ICRA), 2017.
[65] O. A. V. Magana, V. Barasuol, M. Camurri, L. Franceschi, M. Focchi, M. Pontil, D. G. Cald- well, and C. Semini. Fast and continuous foothold adaptation for dynamic locomotion through cnns. IEEE Robotics and Automation Letters, 4(2):2140221) 2019.
[66] F. Jenelten, T. Miki, A. E. Vijayan, M. Bjelonic, and M. Hutter. Perceptive locomotion in rough terrain-online foothold optimization. IEEE Robotics and Automation Letters, 5(4):5370-5376, 2020.
[67] D. Kim, D. Carballo, J. Di Carlo, B. Katz, G. Bledt, B. Lim, and S. Kim. Vision aided dynamic exploration of unstructured terrain with small-scale quadruped robot. In IEEE International Conference on Robotics and Automation (ICRA), 2020.
[68] A. Agrawal, S. Chen, A. Rai, and K. Sreenath. Vision-aides dynamic quadrupedal locomo- tion on discrete terrain using motion libraries. In International Conference on Robotics and Automation (ICRA), 2022.
[69] W. Yu, D. Jain, A. Escontrela, A. Iscen, P. Xu, E. Coumans, Ha, J. Tan, and T. Zhang. Visual- locomotion: Learning to walk on complex terrains with vision. In 5th Annual Conference on Robot Learning, 2021.
[70] V. Makoviychuk, I.. Wawrzyniak, Y. Guo, M. Lu, K. Storey, M. Macklin, D. Hoeller, N. Rudin, A. Allshire, A. Handa, et al. Isaac gym: High performance gpu-based physics simulation for robot learning. arXiv preprint arXiv:2108. 10470. 2021.
[71] T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra. Continuous control with deep reinforcement learning. arXiv preprint (tt::::009022771 2015.
[72] X. Chen, C. Wang, Z. Zhou, and K. W. Ross. Randomized ensembled double Ielanninn: Learning fast without model. In International Conference Learning Representations, 2021.
[73] Ba, R. Kiros, and G. Hinton. Layer normalization. arXiv preprint arXiv: 2016.
[74] L. Smith, I. Kostrikov, and S. Levine. A walk in the park: Learning to walk in 20 minutes with model-free reinforcement learning. arXiv preprint (2Xxx22008.0786) 2022.
[75] L. Pinto, M. Andrychowicz, P. Welinder, W. Zaremba, and P. Abbecl. Asymmetrie actor critic for image-basee based robot learning. Proceedings of Robotics: Science and Systems, 2018.
[76] Y. LeCun, L. Bottou, Y. Bengio, and Haffner. Gradient-based learning applied document recognition. Proceedings of the IEEE, (6(11))2278-2324 1998.
[77] J. C.hung, C. Gulcehre, K. Cho, and Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv:1142.3555 2014.
[78] S. Huang, R. J. Dossa, C. Ye, J. Braga, D. Chakraborty, K. Mehta, and J. G. Araújo. Cleanrl: High-quality single-file file implementations of deep reinforcement learning algorithms. Journal of Machine Learning Research, 13(274):1-18 2022. URLhttp: jtlllorr///////l////////////ll htrg/apprssv23/ 21-1342. html.
13

---

A Implementation Details
A.1 Rewards and Constraints
We use the reward function from [13] that measures progress toward specific direction. To always have positive rewards, we add survival bonus of 0.5 at each time step, and, following [5], clip total rewards below 0.0.
To enforce the constraints, we follow CaT [20] and reformulate the constrained RL problem into the following RL problem:
Σ II 7(1 o(s,, av)) (()) r(se, (
max E TIUST
(5)
with termination probabilities (86, az). Following CaT, we define the termination probabilities as:
max SE7
P;
mas clip(
max 0, 1),
(6)
where cf max(O, ca(s, a)) is the violation of constraint i, ci is exponential moving average of the maximum constraint violation over the last batch of experience collected in the environment, and pm a hyperparameter that controls the maximum termination probability for the constraint i.
Table lists all the constraints used. Following [20], we separate constraints between hard con- straints, where pm 1.0, and soft constraints, where p, increases throughout the course of training, allowing the RL agent discover agile locomotion during the early stage of training while enforcing more the constraints later to ensure safe behaviors. To encourage the emergence of specific locomotion style, some constraints are activated only in specific settings. For instance, the Stand still constraints Caill are only active when no velocity command is provided, whereas the Base orientation and Number of foot contacts are only active on flat terrains and on early terrain levels.
We found that rescaling the constraint violations by the square root function C₁ +- v ot helps CaT be less sensitive to extreme values of constraint violations.
Type
Expression
Hard
Cond.
Knee or base collision Foot contact force
Cknee/base contact
knce/base contact
Cfoot contact,
|||
foot,
film
X
Foot stumble
Catmmlee;
xy
foot;
12
foot,
X
Heading
Chcading
langle base
angle emd 7
angle Jim
X
Torque
Gurque,
|x
X
Joint velocity
Gjoint velocity, &
14x
lim
X
Joint acceleration
Gjoint acceleration
= 1gh| q
lim
X
Action rate
Caction rates,
A
A.tt" 1.x
gdss lim
X
Joint limits min
Goingm
joint min joint,
Joint limits max
Gjoinge joint,
joint
Foot air time
Cair time;
des aii time;
time
Base orientation (roll-axis)
Corian
orii
ori
Base orientation
Cori base orixy ll
base lim
Number of foot contacts
Ca foot contacts
Y/joot contact
foot contact
Stand still
Cstill
q" l|
Estil
Table 2: List of constraints, where Hard indicates whether each row corresponds to a hard constraint and Cond. indicates whether a constraint is active only under certain conditions.
14

---

A.2 Policy Learning
We built our RL algorithm with the CleanRL [78] implementations of PPO and DDPG. During priv- ileged policy learning, we linearly increase the damping parameter (Kd) of the PD controller from 0.05 to 0.2 but keep it fixed at 0.2 during visual policy learning. Indeed, we empirically observed that RL discovers agile skills more easily with lower Kd but policies with higher Kd transfer better to the real Solo-12.
Privileged Policy Learning An MLP parametrizes the privileged policy with hidden dimensions [512, 256, 128] and elu activations. We use PPO [48] with 4096 actors in parallel in simulation. The training procedure is very similar to [5, 13, 20].
Visual Policy Learning The actor processes depth images using a vision neural network consist- ing of three blocks of convolution with leaky ReI.U activations, followed by max pooling and linear layer to produce the depth embeddings. Random translation, random noise, and random cutout are applied to the depth images during training. The actor then processes the history of pro- prioceptive information, actions, and depth embeddings with one-layer Gated Recurrent Unit [77] (GRU) of hidden size 256. This GRU is followed by a MLP with hidden dimensions [512, 256, 128) and elu activations. The output of the final layer is processed by tanh activation function and rescaled to produce the 12-dimensional action vector. We used the action bounds observed in the privileged experience buffer to rescale the actions given by the actor.
The critic network is parameterized by MLP with hidden dimensions [512, 256, 128]. layer nor- malization and elu activations. While the actor observes the full state 8r, which includes history of depth images, the critics process the privileged state priv which includes privileged heightmap scan and ceilings instead of high-dimensional images.
We generate trajectories from the privileged policy that amounts to 2 million state-action samples and store them in the privileged experience buffer Dariv whereas online experience is collected by 256 actors in parallel into the online replay buffer Doaline We store the constraint violations cf of both online and privileged experience in their respective replay buffers to recompute the termination probabilities on the fly during off-policy learning. Both Deriv and Danline store privileged infor- mation at every step and depth image every 5 environment steps. During training, we only give the vision network one image every five timesteps and then replicate the depth latent five times to match the sequence length of the other observations. The online replay buffer stores the GRU hidden latent produced by the online actors whereas the privileged replay buffer stores zeros for these latents. This is done to initialize the first hidden of the DDPG actor correctly during off- off-polii offppolicy training.
We train the visual policy using a variant of RLPD [21]. We build upon DDPG [71] with an update- to-data ratio of 8 during policy evaluation. We use REDQ [72] with 10 critics and an ensemble of random critic targets.
A.3 Baselines
The BC baseline uses the same neural network architecture as SoloParkour, as described in Section A.2. We train the BC policies to regress the action based on the history of observations on the same dataset demonstrations Deni generated by the privileged policy pii as SoloParkour.
The DAgger baseline uses the the Stage 1 policy şivv as teacher for action relabelling. We found that starting the DAgger policy from the BC-pretrained weights greatly improves online learning efficiency.
neural network architecture as SoloParkour and BC. We employ
For the Privileged Reconstruction baseline, we use the same observation space and neural archi- tecture as SoloParkour, BC and DAgger for the vision module except that the output of the GRU is projected the privileged information space through linear layer. We employ the privileged policy priv as frozen MLP head and only train the vision module reconstruct the privileged information from the history of past observations and actions. Similar to [16] and unlike [56], we found it highly
15

---

beneficial to train the vision network with experience generated by the resulting online policy, rather than training solely on the fixed dataset Πaav generated by the privileged policy.
The No Priv. Critie ablation processes visual input in the critic network instead of privileged infor- mation. Its vision module follows the same architecture the visual policy.
For the Visual RL w/o CaT ablation, we use the same dataset of privileged experience Dr as the one used to train all other methods (except for the From Scratch ablation that doesn't learn from any demonstration), but we remove the constraints for Stage 2 RL. Note that the privileged experience Den was generated with zad which was trained with Constrained RI. and therefore satisfies the constraints.
B Real Robot Setup
We use the Solo-12 quadruped robot for experiments. We built custom 3D-printed plastic piece mount the Intel RealSense D405 stereo camera observing in front of the robot. We the Python wrapper of librealsense capture depth images resolution 424 240. We resize and crop the images to 48 x 48 and apply the librealsense postprocessing hole-filling filter. Depth images are preprocessed in a separate thread on a separate CPU as they come, at around 30Hz. The visual policy runs at 50Hz using ONNX and produces target joint angles to torque by PD controller with stiffness Kp = 4.0 and damping Kd = 0.2 running at 10KHz. Hence, depth embeddings are updated at a higher frequency at inference than during training. All the computation is done through Python scripts by the onboard Raspherry Pi 5. Velocity commands sent the embedded controller via wireless gamepad.
C Additional Results
In Figure 6 and 7, we present further results constraint satisfaction for SoloParkour and the base- lines introduced in Section 4.1. SoloParkour demonstrates high constraint satisfaction, highlighting the effectiveness of our approach in achieving safe yet agile locomotion skills over challenging ter- rains using vision.
16

---

Climb
Crawi
Hurdie
Constraint violation (%)
2-
28
26
10
20
Oboitnnle Highht [sm)
Oc ling Reightt (sm)
Obrstatue treight (um)
Gup lengih (Lri)
Privileged policy
Dagger
BC
Visua wr Cal
Privileged Recorstruction
SotoParkout
(a) Torque constraint for the front left shoulder joint
Climb
Crawl
Hurdle
Loop
Constraint violation (%)
10
20
34
28
26
10
20
80
Obstacle height
ling height (cm)
Obstacle height
langth
Privilgged policy
Degger
BC
Visual
CaT
Privileged Reconstruction
SoloParkour
(b) Joint velocity constraint for the front left knee
Olibb
Crawi
Hurdie
Losp
Consiraint violation
3
0
Obstacle he ght (cm)
Coiling height (cm)
Obstacle height (cm)
Gap longth (cm)
Privilaged policy
Degger
CaT
Privllged Reconstruction
SoleParkour
(c) Joint limit constraint for the front left shoulder
Climb
Leap
Constraint vialation (96)
20
10
10
20
40
34
32
26
10
20
30
20
Obstacle height
Ceiling
Obstacle
(cm)
length (cm)
Privileged policy
Degger
BC
Visual RL W/ CaT
Privilgged Reconstruction
SaloParkour
(d) Joint limit min constraint for the front left knee
Figure 6: Constraint violations (in %) when the policies successfully traverse the terrain level by obstacle dimension, averaged across training seeds.
17

---

Climb
Crawl
Hurdie
Constraint violation (%)
4-
10
28
26
10
20
Obslanile Highht [cm)
Oc ling Feigght (sm)]
Obostatle treight (um)
Gup lengihh (Lri)
Privileged policy
Dagger
BC
Visua wr Cal
Privileged Recorstruction
SotoParkout
Air time constraint for the hind right foot
Climb
Crawl
Hurdle
Losp
Constraint violation (%)
10
40
34
32
30
60
Obstacle height
Cailing
heigght
Obstaacl heig
langth (cm)
Privllgeed policy
Degger
BC
Visual
CaT
Privileged Reconstruction
SoloParkour
(b) Heading constraint between the direction of the robot and the commanded direction
Constraint violation (96)
Climb
Crawl
Hurdle
Losp
80-
Obstacle hoight (cm)
Cailing helght (cm)
Obstacle helg nt (cm)
Gap longth (cm)
Privileged policy
Degger
CaT
Privllggd Reconstruction
SoteParkour
(c) Collision constraint for the base of the robot
Hurdle
Leap
Constaaitt vialation (96)
2-
10
20
30
40
34
30
28
26
10
20
30
20
40
80
Obstacle height
[cm]
Obstacte
length (cml
Privileged policy
Degger
BC
Visual RLL W/22 CaT
Privilgeed Reconstruction
SoloParkour
(d) Collision constraint for the hind left knee
Figure 7: Constraint violations (in %) when the policies successfully traverse the terrain level by obstacle dimension, averaged across training seeds.
18


Reviews:
[{"review_id": "MC6WoYflwI", "reviewer": "Reviewer_vns5", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 4, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This work presents an approach to training agile-legged robot using visual observation. The paper proposes to use a privileged policy, that has access to information about the terrain around the robot at the first stage to generate trajectories. Then in the second stage, the agent is trained with the help of the generated trajectories while using only depth images for policy training and additional privileged information for a critic. The experimental section considers different approaches for such tasks showing the superiority of the method in simulation. The paper also presents experiments on physical robot.", "summary_of_recommendation": "The paper is convincing and well-written. Results are extensively demonstrated on hardware.", "technical_quality": 5, "clarity_of_presentation": 5, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "* Typo on line 271 - “MPD”", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "relQj6znkW", "reviewer": "Reviewer_uDaM", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 2, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "Parkour presents a significant challenge for legged robots, requiring agile and precise navigation through complex environments with limited sensory inputs. This paper introduces a novel method for training end-to-end visual policies, transforming depth pixels into robot control commands to achieve agile and safe quadruped locomotion. The approach frames robot parkour as a constrained reinforcement learning (RL) problem, aiming to maximize agile skill emergence within the robot's physical limits while ensuring safety. Initially, a policy is trained without vision using privileged information about the robot's surroundings. This privileged policy generates experience to warm-start a sample-efficient off-policy RL algorithm from depth images, enabling the robot to adapt behaviors to visual locomotion without the high computational costs of RL directly from pixels. The method's effectiveness is demonstrated on a real Solo-12 robot, showcasing its ability to perform various parkour skills such as walking, climbing, leaping, and crawling.", "summary_of_recommendation": "While the manuscript demonstrates a comprehensive system for training visual locomotion on a real quadrupedal robot and introduces innovative methods to enhance state-of-the-art techniques, several weaknesses need to be addressed to strengthen the paper. These include the lack of comparison with other baseline methods that use depth/point cloud features, unclear impacts of the CaT on visual locomotion with insufficient data provided, and missing critical details about the privileged critic's inputs and the robot's behavior in the hardware video. Addressing these concerns would significantly enhance the manuscript's contribution to the field and my support for its publication.", "technical_quality": 3, "clarity_of_presentation": 3, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 2, "questions_for_rebuttal": "**1. Comparison of Other Baselines:** The paper proposes an incremental improvement for policy distillation on visual locomotion tasks, with comparisons to action distillation methods like Dagger and BC shown in Figure 4a. However, it would be beneficial to include the folllowing references and compare the following methods, which distill depth/point cloud into visual features either as an implicit feature vector (partially shown in [1]) or as an explicit local heightmap neural-network-based method [2, 3], and then feed them into the privileged policy for online execution. This approach avoids potential suboptimal results from action imitation. While hardware comparisons are challenging, simulations could enhance the discussion of a comprehensive set of methods for visual locomotion.\n\n[1] https://vision-locomotion.github.io/\n\n[2] https://arxiv.org/abs/2306.14874 \n\n[3] https://arxiv.org/abs/2309.14594 \n\n**2. Effects of the CaT:** Although the original CaT is previously published, its overall effects on visual locomotion remain unclear. Constraints like unwanted collisions are critical for achieving agile locomotion over terrains. The paper only presents results on torque limits in Table 1 and Figure 5. A more thorough investigation of CaT's impact on specific visual locomotion tasks is necessary to determine its usefulness in this domain. The data collected for Table 2 and Figure 5 appear insufficient; a larger dataset would provide a better understanding of CaT's effect on learned behavior. Additionally, it is crucial to assess whether CaT's absence during the distillation phase affects the final policy. What if the CaT is removed during distillation?\n\n**3. Missing Important Details:** \n- Does the privileged critic only use an additional heightmap compared to a regular critic? If not, what are the inputs? What is the rationale for choosing the privileged observation?\n- In the hardware video, why does the robot squat after walking through the ceiling-type terrain? Is it because the GRU overfits to the ceiling, causing the robot to remain crouched afterward?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eJdJmfQ3Cv", "reviewer": "Reviewer_1Wye", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This work proposes employing constraint as termination in training the extreme challenging parkour tasks for quadurped robot. Using DDPG with RLPD, the author trains a visual parkour policy that reduces the chances of violating the torque constraint in the parkour tasks.", "summary_of_recommendation": "This paper provides valuable insight on the constraint violation of the deployed policy. However, the experiments are not convincing enough to show the effectiveness of reducing the torque constraint violation. I recommend a weak reject on this paper.", "technical_quality": 3, "clarity_of_presentation": 5, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 2, "questions_for_rebuttal": "1. Since the author trains the deployable policy $\\pi$ based on a privileged policy $\\pi^\\text{priv}$ from PPO, does the deployable policy reuse any weights of the privileged policy $\\pi^\\text{priv}$?\n\n2. What is the linear velocity of the robot during the parkour course?\n\n3. Is it possible to directly set the hardware constraint in the PD controller? Does it help with the torque constraint in the real-world robot?\n\n4. What are the major causes of failures in the leaping task?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper demonstrates a learning-based Parkour task on the Solo quadrupedal robot using constrained RL. Overall, the reviewers agreed that the results are convincing and impressive. The paper is also clearly written. However, they also encouraged the authors to include more analysis, particularly comparisons against baselines. Please refer to all the reviewers for more details.

During the rebuttal phase, we got polarized recommendations from the reviewers. One reviewer seemed to be happy about the result and increased the score to SA. On the other hand, another reviewer still remains negative despite a few resolved concerns. As an area chair, I am a bit toward the positive side in the hope that this paper with dense hardware experiments can still be a good reference to future researchers, even when there are somewhat similar papers out there. Therefore, I would like to recommend accepting this manuscript.
Decision: Accept