Hybrid Internal Model: Learning Agile Legged Locomotion with Simulated Robot Response
Authors: Junfeng Long
ZiRui Wang
Quanyi Li
Liu Cao
Jiawei Gao
Jiangmiao Pang
Venue: ICLR 2024 poster (2024)
Keywords: Reinforcement Learning
Quadrupedal Locomotion
Internal Model
Primary area: applications to robotics, autonomy, planning

Abstract: Robust locomotion control depends on accurate state estimations. However, the sensors of most legged robots can only provide partial and noisy observations, making the estimation particularly challenging, especially for external states like terrain frictions and elevation maps. Inspired by the classical Internal Model Control principle, we consider these external states as disturbances and introduce Hybrid Internal Model (HIM) to estimate them according to the response of the robot. The response, which we refer to as the hybrid internal embedding, contains the robot’s explicit velocity and implicit stability representation, corresponding to two primary goals for locomotion tasks: explicitly tracking velocity and implicitly maintaining stability. We use contrastive learning to optimize the embedding to be close to the robot’s successor state, in which the response is naturally embedded. HIM has several appealing benefits: It only needs the robot’s proprioceptions, i.e., those from joint encoders and IMU as observations. It innovatively maintains consistent observations between simulation reference and reality that avoids information loss in mimicking learning. It exploits batch-level information that is more robust to noises and keeps better sample efficiency. It only requires 1 hour of training on an RTX 4090 to enable a quadruped robot to traverse any terrain under any disturbances. A wealth of real-world experiments demonstrates its agility, even in high-difficulty tasks and cases never occurred during the training process, revealing remarkable open-world generalizability.

Published as a conference paper at ICLR 2024
HYBRID INTERNAL MODEL: LEARNING AGILE LEGGED LOCOMOTION WITH SIMULATED ROBOT RESPONSE
Junfeng ; Zirui Wang Quanyi Jiawei Gao! Liu Cao Jiangmiao Pang |X
'OpenRobotLab, Shanghai AI Laboratory, "Zhejiang University, Tsinghua University
Teacher Module
Factor Module
Lidar
Camera
Mimic
Mimic
Student Module
Adaptation Module
ONLY require 200 M Frames (1 hour RTX 4090) Efficient Learning
Mator
Minimal Sensors
No Mimic Learning
Hybrid Internal Model
Sprinting
Stairs
Rough Slopes
High Platform
Substantial Agility
Figure 1: Our locomotion policy drive robots to walk across any terrain under any disturbances.
Key insight lies alternatively estimating environmental dynamics with the response of the robot.
ABSTRACT
Robust locomotion control depends on accurate state estimations. However, the sensors of most legged robots can only provide partial and noisy observations, making the estimation particularly challenging, especially for external states like terrain frictions and elevation maps. Inspired by the classical Internal Model Control principle, consider these external states as disturbances and introduce Hybrid Internal Model (HIM) to estimate them according to the response the robot. The response, which refer to the hybrid internal embedding, contains the robot's 's explicit velocity and implicit stability representation, corresponding to primary goals for locomotion tasks: explicitly tracking velocity and implicitly maintaining stability. We use contrastive learning to optimize the embedding to be close to the robot's successor state, which the response naturally embedded. HIM has several appealing benefits: It only needs the robot's proprioceptions, i.e., those from joint encoders and IMU as observations. It innovatively maintains consistent observations between simulation reference and reality that avoids in- formation loss in mimicking learning. It exploits batch-level information that more robust to noises and keeps better sample efficiency. It only requires 1 hour of training on RTX 4090 to enable quadruped robot to traverse any terrain un- der any disturbances. A wealth of real-world experiments demonstrates its agility, even in high- nigh-difficulyy tasks and cases never occurred during the training process, revealing remarkable open-world generalizability.
"Equal Contributions, X3 Corresponding Author. Project page at this URL

---

Published as a conference paper at ICLR 2024
1 INTRODUCTION
In recent years, dearning-based locomotion control methods for quadruped robots have achieved
remarkable results (Hwangbo et al., 2019 Peng et al.] 2020 Yang al.
2020 Xie et al. [02] 1 Imai
iet al.
2022 Yu et 2001 Yang et al. 2022, Margolis et al. 2022 Rudin et al. 2022 Agrawal
jtt al. 2022 Yang et al., 2023b Margolis & Agrawal 2023| Yang et al. 2023b). demonstrating
agility beyond traditional control methods in the wild (Lee et al., 2020 Miki et al. 2022b) or on
low-cost hardware platforms (Kumar et al., 202T, Wu et al., 2022). Powered by numerous and
diverse simulated data, these algorithms allow the quadruped robots to work in real environments directly, and perform not only simple movements but complex skills such as manipulation (Fu ct al.
2022 Ji et al., [023] Cheng et al. 2023) and parkour (Yng et al., 2023a Zhuang al., 2023
Caluwaerts et al. 2023).
To achieve substantial agillity, it is necessary provide the quadruped robots with accurate and com- plete external states (Hartley et al., 2018). However, most onboard sensors provide partial and noisy observations about proprioception and surroundings, impeding the development of robust robot sys- tems. As a remedy, a two-stage training strategy is usually adopted by recent learning-basse meth- ods. Firstly, an oracle policy is trained with access to external states such as terrain frictions and elevation maps in the simulator where domain randomization is applied to enhance generalizability across various terrain conditions. Subsequently, the knowledge this policy transferred to an- other policy through mimicking where external states are disabled (Miki et al. (022a) or inferred from other information (Kumar et al.] 202I). Although methods derived from this strategy yield promising results, their performance is constrained in two ways: (1) Despite the benefits of do- main randomization for simulation-to-real (sim2real) transfer, the introduced noises restrict further optimization of the model, particularly when they directly learn to regress absolute environmental parameters. (2) This two-phase training paradigm complicates the training process and inevitably induces information loss in the mimicking learning phase, no matter whether it is achieved by im- itation learning (Miki et al. 2022a) or adaptation methods (Kumar et al., 2021). Moreover, certain solutions require additional exteroceptive sensors, such cameras (Agarwal et al.) 2023) or li-
dar RRdii et al. (2022), whose configurations vary across different robot platforms. Therefore, it is
challenging train and deploy a policy on universal platforms.
In this work, we propose a method where the policy network does not require access to any external states during the entire training process. To be more specific, our method, termed Hybrid Internal Model (HIM), considers all external states such as elevation maps and ground friction as system dis- turbances. Inspired by the classical Internal Model Control (IMC) (Rivera et al. 1986) that simulates system response to estimate disturbances, we attempt to estimate these system disturbances accord- ing to the simulated response of the robot. Characterized by the velocity and stability, the response is termed as hybrid internal embedding, which is extracted from a sequence of historical observations. Considering the response is naturally embedded within the successor state of the robot, the hybrid internal embedding is optimized through contrastive learning, which pulls close the embedding and the future state. We use Proximal Policy Optimization (PPO) to train the policy network and feed it with both partial observations and the hybrid internal embedding. The embedding is optimized through Hybrid Internal Optimization (HIO) in each PPO iteration, allowing the policy to implicitly infer and distinguish the disturbance from the external environment. As result, the policy can be deployed in diverse real-world settings. Also, contrastive learning makes the policy more robust to noises, and the batch-level information further improves sample efficiency, On the other hand, our method is lightweight and universal on all legged-robot platforms, as long as they provide basic proprioceptive information from IMU and joint sensors.
We train our method in Isaac Gym (Makoviychuk et al. 2021 and deploy it on Unitree Aliengo, A1, and Gol robots. We evaluate and ablate its performance in both simulation and real-world regimes with carefully designed benchmarks and metrics. Experiments show that HIM can use minimal sensors, i.e., joint encoders and IMU, to drive a robot to traverse across any terrain under any disturbances. The policy converges with only 200 million samples and only costs I hour on an RTX 4090 GPU for deployment. We also observe that our method can perform very well with high-difficull tasks such as long-range stairs and the cases never occurred in the training process such as compositional terrains, and deformable slopes, revealing an excellent generalizability in the open world. We hope the simple and efficient method can bring new insights to the community.
2

---

Published as a conference paper at ICLR 2024
Table I: Comparisons between method and previous methods. Teacher-Student refers to (Miki et al. 2022a), MONO means (Agarwal et al. (023), AMP means Adversarial Motion Priors (Wu
al. 2022; Escontrela et al. 2022) and RMA means Rapid Motor Adaptation (Kumar al., 2021
Current Env. Parameters indicate the elevation map, friction, restitution, etc. in current frame.
External
Mimic
Reference
Samples
Methods
Representations
Sensors
Targets
Dataset
(millions)
Sencher-Student
Current Env. Parameters
LiDAR Depth Camera
Bchavior
Unknown Unknown
MONO
Current Env. Parameters
Behavior
AMP
Current Env. Parameters
Behavior
V
600
RMA
Current Env. Parameters
Latent
1,280
Ours
Simulated Robot Response
200
2 RELATED WORK
Historically, algorithms for legged robots have been largely rooted in traditional control- meth- ods, largely due their inherent stability and robustness. Most of these works use model-based con- trol to define controllers (Bosworth et al., 2015 Xiang et al. 2010 Yin et al.) 2007 Sreenath et al.) 2011 Hutter et al. 2016 Bledt al. 2018). However, is difficult for these controllers to adapt to situations with widely varying physical properties, such ice, rough terrains, or deformable ma- terials. At the same time, with the assistance of various physical simulators (Todorov et al. [012] Panerati et al. 2021), especially the Isaac Gym (Makoviychuk
al. 2021), which enables mas- sively parallel simulation, Deep Reinforcement Learning (DRL) for legged locomotion showcased its promising potential. This has motivated the of learned controllers trained with RL that can adapt to changes in dynamics (Tan et al. 2018: Lee et al., 2020: Kumar et al. 2021 Fu et al. 2021
Rudin al. 2022 in simulators, which be deployed the real-world robots.
While the sim2real framework exhibits attractive properties, there are gaps between simulation and real robots in two aspects. Firstly, most of the real robots cannot access external states such as elevation maps, contact forces, eíc. Secondly, the real-world observations are always noisy. To solve this problem, previous methods used mimic learning to compensate for the absence of environmental information. The mimic learning methods can be categorized into two main frameworks: adaptation and teacher-student. The methods using adaptation framework include Kumar et al., (202T), Agrawal et al. 2022), Margolis & Agrawal (2023) and the methods using seacher-student framework include Lee et al. (2020), Chen et al. (2019), 19), Margolis et al. (2022), Wu et al. (2022). While these methods go some way toward solving the sim-to-real problem, there are still huge performance reductions between the reference policy and the deployable policy.
Meanwhile, there is also different group of methods that incorporate dynamics learning im- prove legged locomotion. DayDreamer (Wu et al. 2023) followed the Model-based Reinforcement learning (MBRL) framework, which uses a learned "world model" to synthesize infinite interac- tions; Dream WaQ (Nahrendra et al. 2023) uses a leaned representations via VAE (?) to boost the performance of legged locomotion. What these methods have in common the of regression objective to learn a model or representation, which requires the neural network to fit the targets per- fectly. However, due the uncertainty brought by domain randomization, the network is willing to fit the random noise, eventually resulting in representation collapse.
While our method can fall into this group, significant difference from the works above is that our method neither follows the computationally expensive MBRL framework nor uses regression objectives, which can lead to representation collapse as auxiliary losses. In contrast, we follow the IMC principle and use prototypical contrastive learning (Caron et al.) 2020) Yarats et al. 2021 Deng et al.) (022) as the auxiliary loss, which only requires the encoder to distinguish how the current situation differs from other situations based on historical observations. This makes fuller use of the samples, and the sample-driven regularization leads to better robustness (LeCun) 2022).
3 METHODOLOGG
Our Hybrid Internal Model (HIM) can efficiently train a policy that not only promotes sim2real com- patibility but also significantly enhances robotic agility. It is inspired by the classical Internal Model Control principles and implemented within a learning- based framework. Next, we first present an overview and delve into details afterward.
3

---

Published as a conference paper at ICLR 2024
Proprioception
Hybrid Internal Optimization
Pull Closer
1
Target Encoder
Contrastive
H
Learning
12
Hybrid Internal
Model
Push Away
Policy
Contrastive Learning
Source Encoder
Hybrid Internal Embedding
Policy
Comparisons
Network
0 Hit
Mimic
of
External
12
Force
Critic Network
Policy Gradient
V
at
IN
Elevation Maps
Proximal Policy Optimization
Mimic Learning
Figure 2: Overview of our framework. The policy network receives partial observations and the hy- brid internal embedding, which is optimized to the robot's 's successor state with contrastive learning. The framework is alternatively optimized with HIO and PPO.
3.1 FRAMEWORK OVERVIEW
To control a legged robot, the algorithm needs to determine the movements of its actuators for desired velocity given its proprioceptive information. We follow the paradigm (Rudin et al.] 2022) that models legged locomotion task as a sequential decision problem. The entire framework depicted in Fig. 2 The optimization process encompasses two phases: Hybrid Internal Optimization (HIO), which trains the hybrid internal model, and Proximal Policy Optimization (PPO), which optimizes the policy network. In each iteration, first update the parameters of HIM, then freeze them and optimize the actor and critic modules.
State Space. The policy network takes partial observation of input, which includes the desired velocity, the proprioceptive information from its joint encoder and IMU, and the last action a The desired velocity c, [ug, eg, w indicates the linear velocity in longitudinal and lateral direc- tions, and the angular velocity in the horizontal direction, respectively. The joint encoder provides its joint position 0g and joint velocity ò,. The IMU provides its base angular velocity and gravity direction in robot frame g:. Our value network is allowed access privileged information the training stage and thus can provide a more accurate estimation of state values. Its input of contains two extra components to of: current external force f, and surrounding ground height h,
Action Space. The movement of each actuator is formulated as the bias between the target joint position Qurgge and nominal joint position 0n To reduce the instability the network output, we add scale & < 1 to the policy output thus the final target positions for joints are Ouaget = 0p + ka,- The dimension the action space A equals the number actuators. For example, quadrupedal robots such Unitree Al or ANYmal have 12 actuators, with 3 on each leg.
Reward Functions. We follow the reward functions from Rudin et al. (0022) and Agarwal et al. 2023 with default weights. The details are listed in Appendix A.I
3.2
HYBRID INTERNAL MODEI.
The classical Internal Model Control (IMC) (Rivera et al. 1986) suggests that we can perform robust control without directly modeling the disturbance. As shown Fig ( it uses internal model to simulate the system response and further estimate the system disturbance, increasing the closed-loop stability. The more accurate the internal model is, the more robust control it can perform.
4

---

Published as a conference paper at ICLR 2024
In legged locomotion, the external environmen- tal dynamics such as elevation maps, ground friction, and ground restitution disturbances to the system. However, accurately estimat- ing them is particularly challenging. Inspired by IMC, we propose using model to estimate the robot's response aforementioned, the robot is explicitly given desired velocity for locomotion. Meanwhile, is also implicitly commanded to maintain sta- bility. To this end, we introduce a hybrid in- ternal model that simultaneously estimates an explicit velocity and another implicit response. These estimations, along with robot observa- tions, are collectively fed into the policy net- work, forming a fully closed-loop control sys- tem. The framework is shown in Fig (b).
(a)
Disturbance
Reference
Control Signal
Output
Contraller
Simulated
Model
Response
viable alternative. As
Feedbback
(b)
Velocity (eoplicit)
Environmental Disturbances
Stability llmpliatt
Commands
Policy Network
Actions
Outputs
Velocity
Hybrid Internal Model
Implicit Response
Feedback
Addition
Concatenation
Figure 3: (a) IMC and (b) our implementations.
In practice, HIM uses sequential history observations oi t to extract the hybrid internal embed- ding that consists of the robot's velocity V, and the implicit response 1, H is set as 5 by default. The extractor is 3-layer Multi-Layer Perceptron (MLP) with hidden dimensions of 512. 256, and 128, respectively. The policy network takes the embedding and of as inputs and outputs actions.
3.3 HYBRID INTERNAL OPTIMIZATION
The most critical thing for the internal embedding (vr. 1,) is how to optimize it simulate the robot's response, which is naturally embedded within the robot's successor state of+1 In alignment with the principles of IMC, can directly estimate 0f+1 given of Iit" However, this is challenging due to the high dimension of robot states and the disparity between simulation and reality. Alternatively, given that we train the framework in simulation environment, we can directly learn regress the explicit velocity V, using its ground truth. For the implicit response It, we propose modeling into a latent space Z C R and optimizing it to be close to the successor state with contrastive learning.
In practice, in each iteration, we collect trajectories in each environment as batch. If pair of of and of belong to the same trajectory, they are positive pairs. Otherwise, they are negative pairs. The pairs are optimized by swapping assignments tasks similar to SwAV (Caron et al. 2020). Given sequence of proprioception observations sampled from the rollout trajectories, subsequent observation of from a transition as a target vector and view the concatenated historical observations of-
derive the
vector. We ensure that the augmentation is consistent across time steps. Source and target vectors are fed to the source encoder and target encoder respectively to obtain the latent feature 1. and 1 jarger which are mapped onto a unit sphere in nigh-dimensional space with a Ez-normalization. To predict the cluster assignment probability prouree and larget from pource and ptarget we first apply a Cy-normalization on the prototype to obtain normalized matrix E {éi, (ei,..., ēx); and then take sofumax over the dot products of source vectors or target vectors with all the prototypes:
exp( ! I ek.
exp( { Jarget
P
source
paurcee ex,) P
target
(1)
Σn exp( 1
Σ. exp( targer
CFr)
Here, yource and Pt individual cluster with index k. while 7 is a temperature parameter.
barget are the predicted probability that historical observations of Hat maps to
To obtain the targets (grouree ..... 9 goure) and (ql target ..... 9x targot, for the aforementioned predicted probabilities, while avoiding trivial solutions, the Sinkhorn-Knan plied to both encoders. Now that we have the cluster assignment predictions and targets, the repre- sentation learning objective is simply maximize the prediction accuracy:
algorithm (Cuturi) 2013) is ap-
H
SWAV
Σ
q
sounce
log Pt
turgel
+
targel
log pa Mource).
(2)
2H
5

---

Published as a conference paper at ICLR 2024
Tema Type Ster Stoge Steee Doccet
Teral Type
Star
Fough Sizee
Demersion
Dimension
Ours
(b) Regression Nahrendra
2023
Figure 4: Latent space visualizations of (a) our hybrid internal model and (b) Nahrendra et al. (023)
Analysis. Our method can maximize the similarity of latent features between historical obser- vations and the next observations, implicitly modeling external states without requirements for gression. This also improves the performance of locomotion policy by making use of the infor- mation in batch level, Le., the different environment properties in different kinds of terrain. We conduct t-SNE Van der Maaten & Hinton (2008) on the latent outputs of our methods and regression (Nahrendra et al. (023). The visualization shows that our hybrid internal model has more sepa- rable encoding of environments, which means that our method carries more precious environmental information, resulting stronger ability identify which kind of terrain the robot on.
3.4 TRAINING DETAILS
Simulation Setup. We use Isaac Gym (Rudin et al. 2022) with 4096 parallel environments and rollout length of 100 time steps. The training process needs 1000 rollouts which takes hour of wall clock time on NVIDIA RTX 4090. But its performance continues improving until 2000 rollouts.
Dynamics Randomization. To improve the robustness of the policy and facilitate the sim-to-real process, we randomize the mass of the robot body and links, the centre of mass (CoM) of the robot, the payload applied to the body of the robot, the ground friction and restitution coefficients, the motor strength, the joint-level PD gains, the system delay, the external force, and the initial joint po- sitions in each episode. The randomization ranges for each parameter are detailed in Appendix A.2
Training Curriculum. We use similar terrain curriculum Rudin et al. (2022) and Wu et al. 2022). We create a height field map with 200 terrains arranged in 20 x 10 grid, and each row has the same type of terrain arranged in increasing difficulty, while each grid is 10 x m0. The our training terrain is in Appendix A..3 We put all the robots in different types of terrains with the lowest difficulty at the beginning. the terrain level will increase if the robot reaches 80% of linear tracking reward, and will decrease they can not travel half of the terrains in one episode. We to hard. On complex terrains such as stairs or discrete obstacles, we sample longitudinal and lateral linear velocity commands in [-1.0, 1.0] m/s and horizontal angular velocity in [-2.0, 2.0] rad/s. On slopes and rough slopes, we increase the range of longitudinal linear velocity to [-3.0, 3.0] m/s and the range of horizontal angular velocity [.3.0, 3.0) rad/s. The commands are all sampled uniformly in the corresponding range and independently for each robot, we sample the commands every 25 time steps.
detailed description
simple command currieulum to help the robots learn from easy
4 EXPERIMENTS
In this section, we evaluate the performance of our method both simulation and real-world regimes and conduct ablation studies. The demo videos of method be found at the Project Page can be found in
Additional descriptions of the experimental setup, baselines, and the Appendix.

---

Published as a conference paper at ICLR 2024
Table 2: Main results with average performance 1 in the real-world benchmarks over 20 trials, where - captures a 95% confidence interval. The top results are highlighted.
Benchmarks
Environments
Metrics
Ours
RMA
MoB
Built-in MPC
Short-range
Success rate
100
0
Stairs
(A1)
(%)
Long-range (Aliengo)
Number of stairs
176.5=7.81
75.35=19.98
0.0+0.0
0.0+0.0
Compositional
Success rate
Unseen Terrains
Terrain (Aliengo)
(%)
Deformable
Success rate
55
10
0
Slope (A1)
(%)
Dragging Obstacle (AI)
Maximum weight
10
(Kg)
Anti- disturbance
Vertical Hit (AI)
Maximum weight
7.5
7
5
(Kg)
Payload
Maximum weight
(Al)
(Kg)
Missing steps (Aliengo)
Success
100
0
0
0
(%)
4.1 EVALUATION SETUPS
Compared Methods. We compare our method with the following methods:
Baseline: Train without HIM but keep the source encoder and optimize it with PPO.
Ours w/ velocity input: Set the velocity inputs to the policy network as
Ours w/o velocity loss: Direct remove the velocity estimation.
Ours w/o internal latent input: Set the internal latent inputs to the policy network as 0.
Ours w/o internal latent loss: Direct remove the internal latent estimation.
Regression: Optimize the hybrid internal model with regression methods.
Oracle: Train the policy with history of full observations.
Rapid Motor Adaptation (Kumar et al. 202T).
Multiplicity Behavior (Walk These Ways) (Margolis & Agrawal 2023).
Onboard MPC controller, which is only compared in real-world experiments.
Setups the Real World. We deploy our policy on Unitree A1, Gol, and Aliengo. The policy runs at 50 Hz, with PD controller running at 500 Hz to track the target. The parameters are kp 30.0, ka = 0.75 for Al and Gol, kp = 40.0, ka 2.0 for Aliengo. We compare methods with robust tests in the real world, with 3 benchmarks and 8 tasks. The details can be found in Appendix B.
Setups in Simulations. In simulations, we compare methods according to the tracking errors. The tracking errors for linear velocity and angular velocity are quantified using the norms ||U - 1 ||2
and || wyaw 12, respectively. We test the tracking performance under various terrain conditions and motion ranges. A detailed description be found in Appendix B.2
target
4.2 MAIN RESULTS
Real-world Results In our real-world benchmarks, we compare our method with RMA Kumar al., (02T), MoB (Margolis & Agrawal 2023). and Built-in MPC. The results in Table show our method outperforms the other methods on all tasks and maintains more natural gaits. It proves that our method can well inherit the policy in simulation and can deployed to the real world casily. We also observe that our method can perform very well with high-difficulty tasks such as long-range stairs and the cases never occurred in the training process such as compositional terrains, and deformable slopes, revealing excellent generalizability in the open world.
Simulation Results
In our simulation benchmarks, we compare our method with Baseline, Re- gression, RMA (Kumar al.) 202I) and MoB (Margolis & Agrawal 2023) with Unitree Aliengo. Results in Table show that our method outperforms the other methods on almost all tasks. Our method holds more performance improvements than the other methods and significantly outper-

---

Published as a conference paper at ICLR 2024
Table 3: Average tracking error &. in simulation over 4096 x 5 trials. Top results highlighted.
Terrain Types
Velocity Types
Ranges
Ours
Regression
RMA
Baseline
MoB (Trot)
[-1, 1] m/s
0.073
0.071
0.158
0.271
0.103
Lincar
[-2, 2]
m/s
0.075
0.077
0.270
0.305
0.126
[-1, I] rad/s
0.084
0.086
0.091
0.164
0.081
Slopes
Angular
[-2, 2] rad/s
0.124
0.130
0.134
0.202
0.114
[-1, 1] m/s
0.078
0.075
0.155
0.207
0.069
[1, I] rad/s
0.047
0.050
0.074
0.143
0.052
Combined
[-2, 2] m/s
0.094
0.107
0.203
0.224
0.103
[-2, 2] rad/s
0.067
0.071
0.102
0.157
0.069
[-1. 1] m/s
0.086
0.091
0.201
0.287
0.105
Linear
[-2, 2] m/s
0.085
0.088
0.294
0.331
0.124
Rough Slopes
[-1, 1] rad/s
0.097
0.108
0.142
0.160
0.119
Angular
[-2, 2] rad/s
0.143
0.150
0.147
0.187
0.135
[-,, I] m/s
0.088
0.095
0.167
0.220
0.147
[-1, 1] rad/s
0.054
0.070
0.086
0.169
0.092
Combined
[-2, 2] m/s
0.103
0.137
0.221
0,284
0.195
[-2, 2] rad/s
0.076
0.096
0.117
0.201
0.1 14
Linear
[-1, 1] m/s
0.147
0.194
0.306
0.377
1.108
Stairs
Angular
[-1. 1] rad/s
0.087
0.101
0.241
0.370
0.930
[-1. 1] m/s
0.135
0.217
0.337
0,405
1.270
Combined
[-1, I] rad/s
0.096
0.154
0.201
0.289
0.992
Linear
[-1, I] m/s
0.125
0.182
0.311
0.279
0.973
Discretre
Angular
[-1, 1] rad/s
0.083
0.097
0.175
0.253
0.841
Obstacles
[-], I] m/s
0.125
0.206
0.305
0.374
1.254
Combined
[-1, 1] rad/s
0.075
0.133
0.177
0.246
0.976
forms the Regression method. Besides, our method also performs better in 66.67% of the velocity range on easy terrain.
4.3 ABLATION STUDIES
Ablation Studies in Simulations. We mainly conduct ablation studies simulation with Aliengo. The metrics include the normalized linear velocity tracking score, the normalized angular velocity tracking score, and the maximum reachable terrain level. The task and command sampling are described in Section 3.4 The definition of terrain levels be found in Appendix A.3 Terrains include rough flats, pyramid slopes, rough pyramid slopes, wave terrain, stairs, and flats with discrete obstacles, and the corresponding proportion is 1..1. 0.2. 0.6, 0.1]. The normalized linear velocity tracking score (NLTS) and the normalized angular velocity tracking score (NATS) are:
NLTS = =xxp(- ||ar.y
Ve. 11,
target
NATS = exp( ||w wyyw 11
0.25
0.25
The results are shown in Fig. in which the curves are averaged over 10 seeds. The shaded area represents the standard deviation across seeds. Despite our method does not need terrain informa- tion. it exhibits the most similar results to the Oracle policy. We also conclude that:
In our hybrid internal model, both internal embedding and velocity estimation improve the policy. The absence of any of them can lead to performance degradation.
Our hybrid internal optimization outperforms the Regression method. proves that our method utilize batch-leve information to better understand the properties of different terrains.
RMA can not imitate all the great performances of Oracle. Our method maintains consistent observations between simulation and reality and results in better performance.
8

---

Published as a conference paper at ICLR 2024
30
MOLE LANNN
Dany
D. monin
Oara cuicc
Duts vulictty
Duz los valiclo
Dum with lacee
250
300
759
1800
1050
1906
2750
2000
250
323
150
3500
2290
1758
200
250
566
750
300
1290
3503
1750
2000
(a)
(b)
(c)
Figure 5: Ablation studies with learning curves of (a) normalized linear velocity tracking score, (b) normalized angular velocity tracking score, and (c) maximum reachable terrain level in Isaac Gym.
Table 4: Ablation studies with average performance in the real-world benchmarks over 20 trials, where captures a 95% confidence interval. Top performances are highlighted.
Benchmarks
Environments
Merrics
Ours
vol. inp.
vel. loss
w/o lat. inp.
lat. loss
Regression
Roseline
Stairs
(Al)
range
rate (5)
100
80
55
10
Long-range (Alienge)
Number
176.57777
51,45425.6
14.64224.0 68
13.3=13.42
10.699.66
155. 733.74333.4
0.060.0
Unseen Terrains
Terain (Aliengo)
Compositional
Success (%)
85
70
75
50
15
Slope (A1)
rate (%)
0
Dragging Obstacle (A1)
Maximum
weight (Kg)
Anti- disturbance
Vertical Hit (AI)
Maximum
weight
6.5
5.5
Payload (AI)
Maximum
weight (Kg)
3.5
3.5
7.5
3
Missing steps (Aliengo)
rate (%)
Ablation Studies in the Real World. We also conduct ablation studies real-world bench- marks. Results in Table that our hybrid internal model contributes to the performance, with the internal model playing critical role.
CONCLUSION
Our work presents promising framework and reveals the connection between learning burning-based loco- motion control and classical Internal Model Control principles. In addition, our experimental results show that joint encoders and an IMU provide sufficient information. We also demonstrate that com- pared to regression, contrastive learning provides better representation in terms of robustness to noise and high sample efficiency. To the best of our knowledge, our method is the first system to fully utilize batch-level information in locomotion tasks, leveraging massively parallel simulations of Isaac Gym. However, the lack external perceptions, e.g., visual information, leaves legged robots with only primitive skill set. We will incorporate method with external perception in the future to overcome more challenging environments and higher-level tasks.
Ac Acknowledgement. This work is supported by Shanghai Artificial Intelligence Laboratory.

---

Published as a conference paper at ICLR 2024
REFERENCES
Ananye Agarwal, Ashish Kumar, Jitendra Malik, and Deepak Pathak. Legged locomotion in chal- lenging terrains using egocentric vision. Conference on Robot Learning (CoRL), 2023.
Ayush Agrawal, Shuxiao Chen, Akshara Rai, and Koushil Sreenath. Vision-aided dynamic
quadrupedal locomotion on discrete terrain using motion libraries. In International Conference on Robotics and Automation (ICRA), 2022.
Gerardo Bledt, Matthew J Powell, Benjamin Katz, Jared Di Carlo, Patrick M Wensing, and Sangbae Kim. Mit cheetah 3: Design and control of robust, dynamic quadruped robot. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2018.
Will Bosworth, Sangbae Kim, and Neville Hogan. The mit super mini cheetah: A small, low-cost quadrupedal robot for dynamic locomotion. In 2015 IEEE International Symposium on Safery, Securify, and Rescue Robotics (SSRR), 2015.
Ken Caluwaerts, Atil Iscen, Chase Kew, Wenhao Yu, Tingnan Zhang, Daniel Freeman, Kuang- Huei Lee, Lisa Lee, Stefano Saliceti, Vincent Zhuang, al. Barkour: Benchmarking animal-level agility with quadruped robots. arXiv preprint arXivv23055146644 2023.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. Advances in neural information processing systems, 2020.
Dian Chen, Brady Zhou, Vladlen Koltun, and Philipp Krähenbühl. Learning by cheating. In Con- ference on Robot Learning (CoRI.), 2019.
Xuxin Cheng, Ashish Kumar, and Deepak Pathak. Legs as manipulator: Pushing quadrupedal agility beyond locomotion. In IEEE International Conference on Robotics and Automation (ICRA), 2023.
Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. Advances in neural information processing systems, 2013.
Fei Deng, Ingook Jang, and Sungjin Ahn. Dreamerpro: Reconstruction-Fi model-based rein- forcement learning with prototypical representations. In International Conference Machine Learning, 2022.
Alejandro Escontrela, Xue Bin Peng. Wenhao Yu, Tingnan Zhang, Atil Iscen, Ken Goldberg, and Pieter Abbeel. Adversarial motion priors make good substitutes for complex reward functions. In JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) 2022.
Zipeng Fu, Ashish Kumar, Jitendra Malik, and Deepak Pathak. Minimizing energy consumption leads the emergence of gaits in legged robots. In Conference on Robot Learning (CoRL), 2021.
Zipeng Fu, Xuxin Cheng, and Deepak Pathak. Deep whole-body control: Learning a unified policy for manipulation and locomotion. Conference on Robot Learning (CoRL), 2022.
Ross Hartley, Josh Mangelson, Lu Gan, Maani Ghaffari Jadidi, Jeffrey M Walls, Ryan M Eustice, and Jessy W Grizzle. Legged robot state-estimation through combined forward kinematic and preintegrated contact factors. In 20/8/0 IEEE International Conference on Roboties and Automation (ICRA), pp. 4422-4429. IEEE, 2018.
Marco Hutter, Christian Gehring, Dominic Jud, Andreas Lauber, C Dario Bellicoso, Vassilios Tsou- nis, Jemin Hwangbo, Karen Bodie, Peter Fankhauser, Michael Bloesch, et al. Anymal-a highly mobile and dynamic quadrupedal robot. In IEEE/RSJ international conference intelligent robots and systems (IROS), 2016.
Jemin Hwangbo. Joonho Lee, Alexey Dosovitskiy, Dario Bellicoso. Vassilios Tsounis. Vladlen Koltun, and Marco Hutter. Learning agile and dynamic motor skills for legged robots. Science Robotics, 2019.
10

---

Published as a conference paper at ICLR 2024
Chicko Imai, Minghao Zhang, Yuchen Zhang, Marcin Kierebinski, Ruihan Yang, Yuzhe Qin, and Xiaolong Wang. Vision- geionnguided quadrupedal locomotion in the wild with multi- delay ran- domization. In JEEE/RSI international conference intelligent robots and systems (IROS), 2022.
Yandong Ji, Gabriel B Margolis, and Pulkit Agrawal. Dribblebot: Dynamic legged manipulation in the wild. In IEEE International Conference on Robotics and Automation (ICRA), 2023.
Ashish Kumar, Zipeng Fu, Deepak Pathak, and Jitendra Malik. Rma: Rapid motor adaptation for legged robots. In Robotics: Science and Systems, 2021.
Yann LeCun. A path towards autonomous machine intelligence. Open Review, 2022.
Joonho Lee, Jemin Hwangbo, Lorenz Wellhausen, Vladlen Koltun, and Marco Hutter. Learning quadrupedal locomotion over challenging terrain. Science robotics, 2020.
Viktor Makoviychuk, Lukasz Wawrzyniak, Yunrong Guo, Michelle Lu, Kier Storey, Miles Macklin, David Hoeller, Nikita Rudin, Arthur Allshire, Ankur Handa, et al. Isaac gym: High performance gpu-based physics simulation for robot learning. Advances in neural information processing systems, 2021.
Gabriel B Margolis and Pulkit Agrawal. Walk these ways: Tuning robot control for generalization with multiplicity of behavior. In Conference on Robot Learning (CoRL), 2023.
Gabriel B Margolis, Ge Yang, Kartik Paigwar, Tao Chen, and Pulkit Agrawal. Rapid locomotion via reinforcement learning. In Robotics: Science and Systems, 2022.
Takahiro Miki, Joonho Lee, Jemin Hwangbo, Lorenz Wellhausen, Vladlen Koltun, and Marco Hut- ter. Learning robust perceptive locomotion for quadrupedal robots in the wild. Science Robotics, 2022a.
Takahiro Miki, Lorenz Wellhausen, Ruben Grandia, Fabian Jenelten, Timon Homberger, and Marco Hutter. Elevation mapping for locomotion and navigation using gpu. In JEEE/RSJ International Conference Intelligent Robots and Systems (IROS), 20226.
I Made Aswin Nahrendra, Bycongho Yu, and Hyun Myung. Dreamwaq: Learning robust quadrupedal locomotion with implicit terrain imagination via deep reinforcement learning. In IEEE International Conference Robotics and Automation (ICRA), 2023.
Jacopo Panerati, Hehui Zheng, SiQi Zhou, James Xu, Amanda Prorok, and Angela P Schoellig. Learning to fly- a gym environment with pybullet physics for reinforcement learning of multi- agent quadcopter control. In JEFF/RSJ International Conference on Intelligent Robots and Sys- tems (IROS), 2021.
Xue Bin Peng. Erwin Coumans, Tingnan Zhang, Tsang- Wei Lee, Jie Tan, and Sergey Levine. Learn- ing agile robotic locomotion skills by imitating animals. In Robotics: Science and Systems, 2020.
Daniel E Rivera, Manfred Morari, and Sigurd Skogestad. Internal model control: Pid controller design. Industrial & engineering chemistry process design and development, 1986.
Nikita Rudin, David Hoeller, Philipp Reist, and Marco Hutter. Learning to walk in minutes using massively parallel deep reinforcement learning. In Conference Robot Learning (CoRI.), 2022.
Koushil Sreenath, Hae-Won Park, Ioannis Poulakakis, and Jessy W Grizzle. A compliant hybrid zero dynamics controller for stable, efficient and fast bipedal walking on mabel. The International Journal of Robotics Research, 2011.
Jie Tan, Tingnan Zhang, Erwin Coumans, Atil Iscen, Yunfei Bai, Danijar Hafner, Steven Bohez, and Vincent Vanhoucke. Sim-to-real: Learning agile locomotion for quadruped robots. In Robotics: Science and Systems, 2018.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physies engine for model-based control. In JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2012.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 2008
11

---

Published as a conference paper at ICLR 2024
Jinze Wu, Guiyang Xin, Chenkun Qi, and Yufei Xue. Learning robust and agile legged locomotion using adversarial motion priors. IEEE Robotics and Automation Letters, 2022.
Philipp Wu, Alejandro Escontrela, Danijar Hafner, Pieter Abbeel, and Ken Goldberg. Daydreamer:
World models for physical robot learning. In Conference on Robot Learning (CoRL), 2023.
Yujiang Xiang, Jasbir S Arora, and Karim Abdel-Malek. Physics-based modeling and simulation of human walking: a review of optimization-base and other approaches. Structural and multidisci- plinary optimization, 2010.
Zhaoming Xie, Xingye Da, Michiel Van de Panne, Buck Babich, and Animesh Garg. Dynamics randomization revisited: case study for quadrupedal locomotion. In IEEE International Con- ference on Robotics and Automation (ICRA), 2021.
Chuanyu Yang, Kai Yuan, Qiuguo Zhu, Wanming Yu, and Zhibin Li. Multi-expert learning of adaptive legged locomotion. Science Robotics, 2020.
Ruihan Yang, Minghao Zhang, Nicklas Hansen, Huazhe Xu, and Xiaolong Wang. Learning vision- guided quadrupedal locomotion end-to-end with cross-modal transformers. In International Con- ference on Learning Reprexentations, 2022.
Ruihan Yang, Zhuoqun Chen, Jianhan Ma, Chongyi Zheng, Yiyu Chen, Quan Nguyen, and Xiaolong Wang. Generalized animal imitator: Agile locomotion with versatile motion prior. arXiv preprint arXiv:2310.01400, 2023a.
Ruihan Yang, Ge Yang, and Xiaolong Wang. Neural volumetric memory for visual locomotion control. In Conference on Computer Vision and Pattern Recognition, 2023b.
Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. totypical representations. In International Conference Machine Learning, 2021.
Reinforcement learning with pro-
KangKang Yin, Kevin Loken, and Michiel Van de Panne. Simbicon: Simple biped locomotion control. ACM Transactions on Graphics (TOG), 2007.
Wenhao Yu, Deepali Jain, Alejandro Escontrela. Atil Iscen, Peng Xu, Erwin Coumans, Sehoon Ha, Jie Tan, and Tingnan Zhang. Visual-locomotion: Learning walk on complex terrains with vision. In Conference on Robot Learning (CoRL), 2021.
Ziwen Zhuang, Zipeng Fu, Jianren Wang, Christopher Atkeson, Soeren Schwertfeger, Chelsea Finn, and Hang Zhao. Robot parkour learning. In Conference on Robot Learning (CoRL), 2023.
12

---

Published as a conference paper at ICLR 2024
A HYPER-PARAMETERS
A.1 REWARD FUNCTIONS
The reward functions we used during the training are shown in Table which come from Kumar et al. (2021) Agarwal et al. (2023); Margolis & Agrawal 2023); Fu et al. (0021)
Table 5: Rewards
Reward
Equation (r:)
Weight (w;)
l|verl Vxullé
Lin. velocity tracking
exp
1.0
(wi valyaw
Ang. velocity tracking
exp
0.5
Linear velocity (2)
-2.0
Angular velocity (zy)
-0.05
Orientation
-02
Joint accelcrations
0
225 x 10-7
Joint power
| 0 T
-2 x 10-5
Body height
(h (urge h)²
-1.0
Foot clearance
Ei=0
Iarget
0.01
By
Action rate
aa a
-0.01
Smoothness
aa 2a.
-0.01
In Table 5 is the tracking shaping scale and we use = 0.25, N target is the desired base height larget and p are the desired feet position and real feet position z-axis
corresponding to ground, ps of robot's frame and vby is the feet velocity in xy-plane of robot's frame.
A.2 DOMAIN RANDOMIZATIONS
For the training of our method and other methods, domain randomizations we use are shown in 6 which come from Wu al. (2022); Nahrendra et al. (2023).
Table 6: Domain Randomizations and their Respective Range
Parameters
Range[Min, Max]
Unit
Body Mass
00.8, 1.2 x nominal value
Kg
Link Mass
008, 1.2] nominal value
Kg
CoM
-0.1, x [-0.1, x [-0.1,0.1]
m
Payload Mass
-1,3]
Kg
Ground Friction
0.2, 2.75]
Ground Restitution
0.0,
Motor Strength
0.88, 1.2) x motor torque
Nm
Joint Kp
0.8, 1.2 20
Joint K2
0.8, 1.2 x 0.5
Initial Joint Positions
0.5, 1.5)
X nominal value
rad
System Delay
0,32,]
External Force
-30,30 30 x |-30, 30 x 1-30, 30]
13

---

Published as a conference paper at ICLR 2024
A.3 TRAINING TERRAINS
Our terrain comprises slopes, rough slopes, stairs, and discrete obstacles.
The slopes and rough slopes vary in inclination from 0' 40 with the addition of uniform noise ranging from + cm to 8 em on the rough slopes. The stairs have random widths ranging from 20 em 40 em and heights varying from 5 23 em. The discrete obstacles have heights ranging from + 5 em to + 15 cm. We assign difficulty levels to the terrains using integers from to 9. More specifically, for a terrain level indicated by ferrain the corresponding terrains are as follows:
Slopes: with inclination of 40 x Jovel
Rough slopes: with amplitude of x level em and inclination of 40 leve o,
9
Stairs: with step height of 5 18 level
cm;
Discrete obstacles: with height of 5 + 10 x level
cm. The proportion of the above terrains is
0.1, 0.2, 0.6, and 0.1 correspondingly.
A.4 HYPER PARAMETERS
Table 7: Hyper Parameters for Training
General
Batch Size
4096 x 200
Mini-batch Size
4096 x 50
Number of epochs
5
PPO
Clip range
0.2
Entropy coefficient
0.01
Discount factor
0.99
GAE discount factor
0.95
Desired KL-divergence
0.01
Learning rate
1 x 10
Adam epsilon
1 x 10
Gradient clipping
10
HIM
Number of prototypes Contrastive loss scale
16
1.0
Velocity estimation loss scale
1.0
Learning rate
1 x 10-3
Adam epsilon
1 x 10
Gradient clipping
10
14

---

Published as a conference paper at ICLR 2024
B EXPERIMENTAL DETAILS
B.1 REAL-WORLD BENCHMARK SETUP
Our benchmark consists of three parts: stairs, unseen terrains and anti-disturbance:
Stairs:
Short-ramge stairs: This scene includes two sets of stairs with 5 steps each. The height and width of the stairs are 16 em and 27 em, respectively. The Unitree Al robot used to test our policy in this scene, and the metric is the success rate of going through both stairs. The corresponding snapshots of this scene are displayed in the left column Fig. 6[a)
Long-range stairs: This scene consists of hundreds of stairs with height of 17 cm and a width of
25 cm. The Unitree Aliengo robot is equipped with policies that need to be tested, and the metric is
the number of stairs the robot can ascend without stopping. The corresponding photos of this scene are shown in the right column of Fig. ((a)
Unseen terrains:
Compositional terrain: This scene alternates between gravel and stairs, as depicted in the upper picture of Fig. [(b) The Unitree Aliengo robot is equipped with policies that need to be tested, and the metric is the success rate of going through three compositional alternations.
Deformable slope: This scene makes deformable slope created by two soft beds, as shown in the lower picture Fig. 6(b) The Unitree Al robot is used to test our policy, and the metric the
success rate of navigating through this slope.
Anti-disturbance:
Dragging obstacle: This task simulates the scenario where the robot's leg is dragged by obstacles. A wooden box is attached to the robot's back leg, preventing its movement. The metric for this task
is the maximum weight the robot can handle. The Unitree Al robot is used, and the illustration can
be found in the top-left of Fig. b(c)
Lateral hit: This task evaluates the robot's push-recovery ability. The robot is subjected to a lateral force from load swinging down from a height of 107 cm. The Unitree Al robot is used in this task,
and the metric is the maximum weight of the load that the robot can withstand after being hit. The illustration is shown the top-right of Fig. [
Payload: This task assesses the robot's 's carrying capacity. The Unitree A1 robot is used, and the
maximum weight that the robot can carry while walking is measured. The illustration can found in the bottom-left of Fig. 6(c)
Missing step: This task simulates the robot's reaction when it misses steps and falls from a high platform. The platform's 's height, shown in the bottom-right of Fig. [[c) is 42 em. The Unitree
Aliengo robot is used in this task, and the metric the success rate of maintaining balance after descending.
(a) Stairs
(b) Unseen terrains
(c) Anti-disturbance
Figure 6: Snapshots our real-world scene benchmarks for quantitative analysis in Section 4.1
15

---

Published as a conference paper at ICLR 2024
B.2 SIMULATION BENCHMARK SETUPS
We consider tracking tasks with 3 different ranges of commands for each type of terrain. The track-
ing enror slopes and rough slopes tested in the range [-1.0, 1.0] m/s x [-1.0, 1.0) m/s x [-1.0, 1.0) rad/s and |-2.0. 2.0] m/s [-1.0, 1.0] m/s x [-2.0, 2.0] rads/s. All other tests are con- ducted within the range [-1.0, 1.0) m/s [-1.0, 1.0) m/s [-1.0, 1.0) rad/s. These ranges corre- spond to linear velocity in the forward direction, lateral directions, and angular velocity about the vertical axis, respectively.
When testing the linear velocity tracking error, the angular velocity target is set to 0. For the an- gular velocity tracking error tests, the linear velocity target is O. The 'combined velocity' tests are conducted without setting either velocity component to O. We uniformly sample test commands within these ranges. The environmental conditions are set to level lerain 1,2,3, 4, with an equal distribution 25% for each, as explained in Appendix A.3
B.3 ABLATION ON NUMBER OF PROTOTYPES
We conducted ablation studies the number of prototypes in research. By default, set the number prototypes to 16, which corresponds to the square of terrain types.
The training curves for different numbers of prototypes shown in Fig. 7 It can be observed that the performance of our method steadily improves as the number of prototypes increases until reaches 64. After this, the performance starts decline.
This finding suggests that the representation of environmental dynamics does not require dense rep- resentation like pure regression, but overly sparse representation is also inadequate. Additionally, high number of prototypes accelerates training during the initial stages but compromises final per- formance. Conversely, a low number of prototypes may lead to slower initial training but ultimately achieves superior final performance.
Thus, determining an appropriate number of prototypes is vital for striking a balance between rapid training speed and optimal final performance. This matter warrants further investigation in future studies.
Geruge
sostors
Figure 7: Normalized Linear Tracking Score over the number of Prototypes
16

---

Published as a conference paper at ICLR 2024
B.4 OUTDOOR EXPERIMENTS
We further evaluated our policy in additional real-world scenes. Videos can be found at the following URL Illustrations of the scenes that have appeared during outdoor experiments shown in Fig. g
Figure 8: Snapshots of various outdoor scenes for the agility evaluation of our quadruped robot.
17


Reviews:
[{"review_id": "1WRSKOqaTp", "reviewer": "Reviewer_SBNA", "summary": "In this paper, the authors proposed a new auxiliary module, i.e. the hybrid internal model to help blind locomotion policy training. The hybrid internal model consumes past propiorceptive observation histories and produces a latent dynamic vector and estimated base body speed. Instead of using the regular regression loss, the latent dynamic vector is trained with a contrastive loss. The authors zero-shot sim2real transfer their policy to a Unitree robot and show that  their policy can outperform other blind baselines on a few difficult tasks including walking up/down large stairs and on soft terrains.", "questions": "Is the hybrid internal model co-trained with the policy or separately?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1) The paper shows great sim2real results. The learned gait looks very smooth, and more importantly, can solve difficult multi-terrains without using perception\n2) Some novelties in introducing a contrastively learned environment/internal dynamics model.", "weaknesses": "1) There isn't a big delta from previous works, i.e. the ETH 2020 science paper also demonstrates similar capabilities. \n2) While it is applaud to see good results on blind locomotion walking, without perception the policy is still fundamentally limited on difficult terrains. For example, the policy has to hit the vertical edges to sense the terrain before the hybrid internal model can \"sense\" the change in the environment. And there are already good perceptive locomotion works there. \n3) The experiment result section of the paper is thin. For example, there is no quantitative results on success rate etc. \n4) In the ablation study, the oracle policy seems learning slower than the proposed method, which is surprising. \n5) Consider publishing the code with the paper. Right now there is a supplementary pdf which is a duplication of the main paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1pdSrn2eG1", "reviewer": "Reviewer_FQfg", "summary": "The paper proposed a new legged locomotion method called HIM (Hybrid Internal Model). The method uses only joint encoders and IMU observations (no cameras or range sensors). HIM has 2 modules, the \"information extractor \" which learns the environment dynamics, and the \"policy network\" which learns the policy. The HIM model is trained in a simulator (IsaacGym) and successfully deployed in the real world (Unitree Go1 robot).", "questions": "How your work is related with \"world models\" that also learn the dynamics of the environment? For instance, [1] applies word models online and also addresses the legged locomotion problem. I know that in your work less sensor data is used, but in addition to this, how your approach is compared with world models? Can your approach be considered a world model?\n\n[1] Wu et al. \"Daydreamer: World models for physical robot learning\" applies word models online and address legged locomotion.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The main strength of the paper is the proposal of the HIM (Hybrid Internal Model), which is able to address the legged locomotion problem using only joint encoders and IMU observations (no cameras or range sensors). This is achieved using two components, the \"information extractor \" which learns the environment dynamics, and the \"policy network\" which learns the policy. This is a very interesting approach and from my point of view is related to the \"world models\" concept.\n\nThe method worked successfully in different simulated and real environments. In the real world, experiments were conducted on various environments, showing agile and robust locomotion.", "weaknesses": "From my point of view \"world models\" also learn the dynamics of the environment, in simulation or without using simulators, and it is required that authors compares its proposal with world models. For instance, Wu et al. \"Daydreamer: World models for physical robot learning\" applies word models online and address legged locomotion. \n\nThis is important in order to better assess the originality and contributions of the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ah15ZXhhJp", "reviewer": "Reviewer_Yqja", "summary": "This paper proposes to use a learned model to improve learning policies for a quadrupedal robot. In particular, the learned model is trained via contrastive learning instead of the typical regressive learning in existing literature.", "questions": "1. Seems to me the proposed model is more of a state estimation model (or a state compression model) instead of a dynamics model. For example, one would expect a dynamics model to be used to do rollouts instead of using a simulator during training. It will be nice to have some comments about it.\n\n2. A key missing ablations is to compare the proposed contrastive learning with a regression model. \n\n3. It would be nice to have more analysis of the learned latent model as this is the key contribution. For example, examining the effect of the number of prototypes,", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Using contrastive learning for learning the dynamics model seems interesting.\n\n2. Real robot experiments.", "weaknesses": "1. The paper seems to be written in a rush with lots of typos and incorrect references. \n\n2. The need of the velocity model is a little bit unsatisfying as one would expect a perfect internal model should be able to replace it.\n\n3. Need clarification for some questions that I will list below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "N9SCfZ4qCJ", "reviewer": "Reviewer_9L23", "summary": "This work proposees Hybrid Internal Model which uses minimal sensor input for legged locomotion aiming to address the limitation of existing learning-based locomotion control methods. The Hybrid Internal Model learns the latent representation of velocity and implicit dynamics of the environment via contrastive learning. The method is evaluated in simulation and deployed in the real world.", "questions": "Though this work provides a reasonable real-world demo, this work need to be improved in multiple ways to meet the standard of ICLR. \n1. Many technical details (including but not limited to the items mentioned in the weakness section) need to be cleared.\n2. More baseline needs to be compared.\n3. More ablation study need to be performed, for example regression or contrastive learning\n4. More specific performance analysis is required: performance across terrain or different physical properties.", "limitations": "", "rating": 5, "confidence": 5, "soundness": 1, "presentation": 1, "contribution": 2, "strengths": "1. The work provides a simple one-stage method for training blind locomotion controllers, with minimal sensory information.\n2. This work provides impressive real-world performance in the supplementary video.\n3. The t-SNE visualization shows that the proposed hybrid internal model learned to distinguish different terrain type in latent space.", "weaknesses": "1. The overall delivery of the work is not satisfactory in the following aspect:\n* Many technical details are poorly described or totally missing: what’s the exact task legged robot need to solve (follow forward velocity or follow angular velocity or anything else), what command is included in the user-command, what’s the linear velocity and angular velocity range for the input. \n* Some parts are quite confusing, for instance: the user command and trainable prototypes are using the same notation: c_{i}. There are many typos in the paper, for example: “police network” in section 3.\n\n2. The evaluation of the method is poor:\n* There is no quantitative comparison with any baseline in the paper (no result in simulation and no result in real world). \n* The only comparison result (The training curve in Figure 6) provides limited information, since there is no specific description of the task. It’s hard to see the actual performance gap of different methods in terms of the “task target” like velocity tracking and angular.0\n* There is no actual baseline compared in the experiment section, since the method use only proprioception observation of the robot, at least RMA, MoB should be compared, quantitatively, and provide corresponding training curve. Since this work claim the proposed method provide better performance and sample efficiency against two-stage methods, at least comparison is needed.\n* There is no performance analysis the method or baselines across different terrains, or across different command range or different command types, or across different command target.\n* This work claims contrastive learning is better than purely regression, but there is no experiment to support the claim.\n\n\nReference: \n[1] Kumar, et al. RMA: Rapid Motor Adaptation for Legged Robots", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)