Learning Vision-Guided Quadrupedal Locomotion End-to-End with Cross-Modal Transformers
Authors: Ruihan Yang
Minghao Zhang
Nicklas Hansen
Huazhe Xu
Xiaolong Wang
Venue: ICLR 2022 Spotlight (2022)
Keywords: Reinforcement Learning
Robotics
Locomotion Control
Multi-Modal Transformer

Abstract: We propose to address quadrupedal locomotion tasks using Reinforcement Learning (RL) with a Transformer-based model that learns to combine proprioceptive information and high-dimensional depth sensor inputs. While learning-based locomotion has made great advances using RL, most methods still rely on domain randomization for training blind agents that generalize to challenging terrains. Our key insight is that proprioceptive states only offer contact measurements for immediate reaction, whereas an agent equipped with visual sensory observations can learn to proactively maneuver environments with obstacles and uneven terrain by anticipating changes in the environment many steps ahead. In this paper, we introduce LocoTransformer, an end-to-end RL method that leverages both proprioceptive states and visual observations for locomotion control. We evaluate our method in challenging simulated environments with different obstacles and uneven terrain. We transfer our learned policy from simulation to a real robot by running it indoor and in-the-wild with unseen obstacles and terrain. Our method not only significantly improves over baselines, but also achieves far better generalization performance, especially when transferred to the real robot. Our project page with videos is at https://rchalyang.github.io/LocoTransformer/.

Published as a conference paper at ICLR 2022
LEARNING VISION-GUIDED QUADRUPEDAL LocO- MOTION END-TO-END WITH CROSS-MODAL TRANS-
FORMERS
Ruihan Yang" UC San Diego
Minghao Zhang* Tsinghua University
Nicklas Hansen UC San Diego
Huazhe Xu UC Berkeley
Xiaolong Wang UC San Diego
ABSTRACT
We propose to address quadrupedal locomotion tasks using Reinforcement Learn- ing (RL) with Transformer-basee model that learns combine proprioceptive information and high- -igh-dimensional depth sensor inputs. While learning-bast baannd-baeeddl lo- comotion has made great advances using RL, most methods still rely domain randomization for training blind agents that generalize to challenging terrains. Our key insight is that proprioceptive states only offer contact measurements for immediate reaction, whereas an agent equipped with visual sensory observations can learn proactively maneuver environments with obstacles and uneven terrain by anticipating changes in the environment many steps ahead. In this paper, we introduce LocoIransforner, an end-to-end RL method that leverages both propri- oceptive states and visual observations for locomotion control. We evaluate our method challenging simulated environments with different obstacles and uneven terrain. We transfer our learned policy from simulation to a real robot by running it indoors and in the wild with unseen obstacles and terrain. Our method not only significantly improves over baselines, but also achieves far better generalization performance, especially when transferred to the real robot. Our project page with videos is at https: settps://rchalannn git github.tt.. Locoo/oooooaaaaoooooooooooo foreer/.
INTRODUCTION
Legged locomotion is one of the core problems in robotics research. It expands the reach of robots and enables them to solve wide range of tasks, from daily life delivery planetary exploration in challenging, uneven terrain (Delcomyn & Nelson, 2000; Arena et al., 2006). Recently, besides the success of Deep Reinforcement Learning (RL) navigation (Mirowski et al., 2017: Gupta et al., 2019; Yang et al., 2019; Kahn al., 2021) and robotic manipulation (Levine et al., 2018; 2016; Tian et al., 2019; Jain et al., 2019b), we have also witnessed the tremendous improvement of locomotion skills for quadruped robots, allowing them to walk uneven terrain (Xie et al., 2020; 2021). and even generalize to real-world with mud, snow, and running water (Lee et al., 2020a).
While these results encouraging, most RL approaches focus on learning robust controller for blind quadrupedal locomotion, using only the proprioceptive state. For example, Lee et al. (2020a) utilize RL with domain randomization and large-scale training in simulation learn a robust quadrupedal locomotion policy, which can be applied to challenging terrains. However, is domain randomization with blind agents really sufficient for general legged locomotion?
By studying eye movement during human locomotion, Matthis et al. (2018) show that humans rely heavily on eye-body coordination when walking and that the gaze changes depending on characteristics of the environment, e.g., whether humans walk in flat or rough terrain. This finding motivates the use of visual sensory input to improve quadrupedal locomotion on uneven terrain. While handling uneven terrain is still possible without the vision, blind agent is unable to consistently avoid large obstacles as shown in Figure 1. To maneuver around such obstacles, the agent needs to perceive the obstacles at distance and dynamically make adjustments to its trajectory to avoid any collision. Likewise, an agent navigating rough terrain (mountain and forest Figure 1) may also
"Equal Contribution

---

Published as a conference paper at ICLR 2022
Thin Obelacle Sphere
Wide Otstacie Sphere
Moving Obslacle
Mountain
Figure 1: Overview of simulated enviromments & real robot trajectories. Top row shows the simulated environments. For each sample, the left image is the environment and the right image is the corresponding observation. Agents
tasked move forward while avoiding black obstacles and collecting red spheres. Following two rows show the deployment of the RL policy to real robot in an indoor hallway with boxes and a forest with trees. Our robot successfully utilizes visual information to traverse the complex environments. benefit from vision by anticipating changes in the terrain before contact, and visual observations can therefore play an important role improving locomotion skills.
In this paper, we propose to combine proprioceptive states and trst-person-view visual inputs with a cross-modal Transformer for learning locomotion RL policies. Our key insight that proprioceptive states (i.e. robot pose, Inertial Measurement Unit (IMU) readings, and local joint joint rotations) provide a precise measurement of the current robot status for immediate reaction, while visual inputs from depth camera help the agent plan to maneuver uneven terrain and large obstacles in the path. We fuse the proprioceptive state and depth image inputs using Transformers (Vaswani et al., 2017; Tsai et al., 2019) for RL, which enables the model to reason with complementary information from both modalities. Additionally, Transformers also offer a mechanism for agents to attend to specific visual regions (e.g. objects or uneven ground) that are critical for their long-term and short-term decision making, which may in turn lead to a more generalizable and interpretable policy.
Our Transformer- based model for locomotion, Lonofransformer, consists of two encoders for inputs (an MLP for proprioceptive states, a ConvNet for depth image inputs) and Transformer encoders for multi-modal fusion. We obtain feature embedding from the proprioceptive states and multiple image patch embeddings from the depth images, which are used jointly as token inputs for the Transformer encoders. Feature embeddings for both modalities are then updated with information propagation among all the tokens using self-attention. We combine both features for action prediction. The model trained end-to-end without hierarchical RI. (Peng et al., 2017: Jiang al., 2019; Jain et al., 2019a) nor pre-defined controllers (Da et al., 2020; Escontrela et al., 2020).
We experiment on both simulated and real environments as shown in Figure 1. Our tasks in simulation include maneuvering around obstacles of different sizes, dynamically moving obstacles, and rough mountainous terrain. With simulation-to-real (sim2real) transfer, we deploy the policies to the robot on indoor hallways with box obstacles and outdoor forests with trees and uneven terrain. We show that learning policies with both proprioceptive states and vision significantly improve locomotion control, and the policies further benefit from adopting cross-modal Transformer. We also show that Loco rrnsfforrrr generalizes much better to unseen environments, especially for sim2real transfer. We highlight our main contributions as follows:
Going beyond blind robots, we introduce visual information into end-to-end RL policies for quadrupedal locomotion to traverse complex terrain with different kinds of obstacles.
We propose LocoTransformer, which fuses proprioceptive states and visual inputs for better multi-modal reasoning in sequential decision making.
To the best of our knowledge, this is the first work which deploys vision- based RL policy on running real quadrupedal robot avoiding obstacles and trees in the wild.
2 RELATED WORK
Learning Legged Locomotion. Developing legged locomotion controllers has been a moan-itanding problem in robotics (Miura& & Shimoyama, 1984; Raibert, 1984; Torkos & van de Panne, 1998; Geyer
2

---

Published as a conference paper at ICLR 2022
et al., 2003; Yin al., 2007; Bledt al., 2018). While encouraging results have been achieved using Model Predictive Control (MPC) and trajectory optimization (Gehring et al., 2013; Carlo et al., 2018; Di Carlo al., 2018; Carius et al., 2019; Ding et al., 2019; Grandia et al., 2019; Bledt & Kim, 2020; Sun et al., 2021), these methods require in-depth knowledge of the environment and substantial manual parameter tuning, which makes these methods challenging apply complex environments. Alternativly, model-free RL can learn general policies on challenging terrain (Kohl & Stone, 2004; Zhang et al., 2018: Luo et al., 2020; Peng et al., 2018; Tan al., 2018; Hwangbo al., 2019; Lee et al., 2020a; Iscen et al., 2018; Jain et al., 2019: 9a; Xie et al., 2021; Kumar et al., 2021). Xie et al. (2020) use dynamies randomization to generalize RL locomotion policy in different environments, and Peng et al. (2020) use animal videos to provide demonstrations for imitation learning. However, most approaches currently rely only on proprioceptive states without other visual inputs. In this work, we propose to incorporate both visual and proprioceptive inputs using Transformer for RI. policy, which allows the quadruped robot to simultaneously move and plan its trajectory.
Vision-based Reinforcement Learning. To generalize RL to real-world applications beyond state inputs, a lot of effort has been made in RL with visual inputs (Sax et al., 2018; Jaderberg et al., Pathak et al., 2017; Jain et al., 2019b; Mnih et al., 2015a; Lin et al., 2019; Yarats et al., 2019; Laskin et al., 2020; Stooke et al., 2020; Schwarzer et al., 2020). For example, Srinivas al. (2020) propose to apply contrastive self-supervised representation learning (He et aal.2022) with the RI. objective in vision-based RI. Hansen & Wang (2021) further extend the joint representation learning and RL for better generalization to out-of-distribution environments. Researchers have also looked into combining multi-modalities with RL for manipulation tasks (Lee al., 2020b; Calandra et al., 2018) and locomotion control (Heess et al., 2017; Merel et al., 2020). Escontrela et al. (2020) propose to combine proprioceptive states and LiDAR inputs for learning quadrupedal locomotion with RL using MLPs. Jain et al. (2020) propose to use Hierarchical RL (HRL) for locomotion, which learns high-level policies under visual guidance and low-level motor control policies with IMU inputs. Different from previous work, we provide a simple approach to combine proprioceptive states and visual inputs with Transformer model in end-to- te--tt--endddd manner without HRL. Our Transfomeer not only performs better in challenging environments but also achieves better generalization results in unseen environments and with the real robot.
2017; Levine al., 2016; 2018;
Transformers and Multi-modal Learning, The Transformer model has been widely applied in the fields of language processing (Vaswani et al., 2017; Devlin et al., 2018; Brown et al., 2020) and visual recognition and synthesis (Wang et al., 2018; Parmar et al., 2018; Child et al., 2019; Dosovitskiy al., 2020; Carion al., 2020; Chen al., 2020a). Besides achieving impressive performance in a variety of language and vision tasks, the Transformer also provides an effective mechanism for multi-modal reasoning by taking different modality inputs as tokens for self-attention (Su et al., 2019; Tan & Bansal, 2019; Li et al., 2019; Sun et al., 2019; Chen et al., 2020b; Li et al., 2020; Prakash et al., 2021; Huang et al., 2021: : Hu & Singh, 2021; Akbari et al., 2021: Hendricks et al., 2021). For example, Sun et al. (2019) propose to use a Transformer to jointly model video frames and their corresponding captions from instructional videos for representation learning. Going beyond language and vision, we propose to utilize mrosl Transformers to fuse proprioceptive states and visual inputs. To our knowledge, this is the first work using cross-modal Transformers for locomotion.
REINFORCEMENT LEARNING BACKGROUND
We model the interaction between the robot and the environment as an MDP (Bellman, 1957) (S, A, P, R., H), where 8 E S are states, a E A are actions, P(s' as,a) is transition function, R is reward function, H is finite episode horizon, and is discount factor. The Agent learn a policy
To parameterized by 0 to output actions distribution conditioned on current state. The goal of agent is to learn that maximizes the discounted episode return: R = (E) r, R(s at) is the reward for time step t, T (T( is the trajectory.
H 7'r.). where
4
METHOD
We propose to incorporate both proprioceptive and visual information for locomotion tasks using a novel Transformer model, LocoTransformer. Figure 2 provides an overview of our architecture. Our model consists of the following two components: (i) Separate modality encoders for proprioceptive and visual inputs that project both modalities into latent feature space; (ii) A shared Transformer encoder that performs cross- modallty attention over proprioceptive features and visual features, as well as spatial attention over visual tokens to predict actions and predict values.
3

---

Published as a conference paper at ICLR 2022
4.1 SEPARATE MODALITY ENCODERS
In our setting, the agent utilizes both proprioceptive states and visual ob- servations for decision-making. Pro- prioceptive state and visual observa- tion are distinctively different modal- ities: the proprioceptive input a 93- D vector, and we use stacked first- person view depth images to encode the visual observations. To facili- tate domain-specinn characteristics of both modalities, we use two separate, domain sppeiii encoders for propri- oceptive and visual data respectively, and unify the representation in a latent space. We now introduce the architec- tural design of each encoder, and how features are converted into tokens for the Transformer encoder.
Output
Projection hoad
Propeisceptioee
Add & Narm
Shared Transformer Encoder
Forward
Attention
Linwer
Death
Propricceptive State
Image
Figure 2: Network Architecture. We process proprioceptive states with a MLP and depth images with ConvNet. We take proprioceptive embedding as single token, split the spatial visual feature representation into x N tokens and feed all tokens into the Transformer encoder. The output tokens are further processed by the projection head to predict value or action distribution.
We use an MLP to encode the propri- oceptive input into proprioceptive fea- tures Eruu E RC where CroU is the proprioceptive feature dimension. We encode additionally provided visual information using ConvNet. The ConvNet encoder forwards the stacked depth image inputs into a spatial representation Evisual with shape C x N x N, where C is the channel number, and N is the width and height of the representation. The depth images are from the first-person view from the frontal of the robot, which captures the obstacles and terrain from the perspective of the acting robot. However, for first-person view, the moving camera and limited field-of-ve view make learning visual policies significantly more challenging. For instance, changes in robot pose can result in changes in visual observations. This makes it essential to leverage propri- oceptive information to improve visual understanding. In the following, we present our proposed method for fusing the two modalities and improving their joint representation using Transformer.
4.2 TRANSFORMER ENCODER
We introduce the Transformer encoder to fuse the visual observations and the proprioceptive states for decision making. Given spatial visual features with shape C' N N from the ConvNet encoder, we split the spatial features into N x N different C-dimensional token embeddings trisual E RC (illustrated as yellow tokens in Figure 2). each corresponding to a local visual region. We use linear layer to project the proprioceptive features into a C-dimensional token embedding prop e RC (illustrated as a green token in Figure 2). Formally, we have N x N + tokens in total obtained by:
doa Wrrrop (EPuP) + doop
prop
E RCC
To = doop visual
visual 40,1
visual N-I.N-]
visual 1
E RCC
(2)
,
where Orisual is the token at spatial position (i,j) of the visual features Evisual and Wrpop grrp are the weights and biases, respectively, of the linear projection for proprioceptive token embedding. In the following, we denote T'm E [2(*++1)x0 as the sequence of tokens after m Transformer encoder layers, and To as the input token sequence from Eq. 2.
We adopt a stack of Transformer encoder layers (Vaswani et al., 2017) to fuse information from proprioceptive and visual tokens. Specifically, we formulate the Self-Attenton (SA) mechanism of the Transformer encoder as scaled dot- fot-proouct attention mechanism, omitting subscripts for brevity:
T. = TU4 T T
U9,UK,U U E BOxC
(3)
srum Softmax (T /VD)
a E (N(++1))((+++1
(4)
SA(T) = T
USA E RCC
(5)
4

---

Published as a conference paper at ICLR 2022
where D is the dimension of the self-attention layer. The SA mechanism first applies separate linear transformations on each input token T to produce embeddings T T" as defined in Eq. 3. We then compute weighted sum over input tokens, where the weight W for each token pair (t,, tz) is computed as the dot-product of elements 4 and L scaled by 1/vD and normalized by a Softmax operation. After matrix multiplication between weights W and values T", forward the result to linear layer with parameters USA as in Eq. 5, and denote this as the output SA(T).
Each Transformer encoder layer consists of a self-attention layer, two LayerNorm (LN) layers with residual connections, and 2-layer MLP as shown in Figure 2 (right). This is formally expressed as,
T' (N(S)((T)) + Tm,, Tm+1 (N(M(((TT)) + Tm), Tm. E [
(6)
where To is the normalized SA. Because SA is computed across visual tokens and single propriocep- tive token, proprioceptive information may gradually vanish in multi-layer Transformers; the residual connections make the propagation of proprioceptive information through the network easier.
We stack L Transformer encoder layers. Performing multi-layer self-attention on proprioceptive and visual features enables our model fuse tokens from both modalities at multiple levels of abstraction. Further, we emphasize that a Transformer-basee has a separate regional receptive field, and self-attention, therefore, enables the agent to explicitly attend to relevant visual regions. For modality-level fusion, direct application of pooling operation across all tokens would easily dilute proprioceptive information since the number visual tokens far exceed that of the proprioceptive token. To balance information from both modalities, we first pool information separately for each modality, compute the mean of all tokens from the same modality to get a single feature vector. We then concatenate the feature vectors of both modalities and project the concatenated vector into final output vector using an MLP, which we denote the projection head.
fusion allows spatial reasoning, each visual token
Space. In all environments, the agent receives both proprioceptive states and visual input as follows: (i) proprioceptive data: 93-D vector consists of IMU readings, local joint rotations, and actions taken by the agent for the last three time steps; and (ii) visual data: stacked the most recent 4 dense depth image of shape 64 x 64 from a depth camera mounted on the head of the robot, which provides the agent with both spatial and temporal visual information.
Observation
Implementation Details. For the proprioceptive encoder and the projection head, we use 2-layer MI. with hidden dimensions (256, 256). Our visual encoder encode visual inputs into 4 x spatial feature maps with 128 channels, following the architecture in Mnih et al. (2015b). Our shared Transformer consists of 2 Transformer encoder layers, each with a hidden feature dimension of 256.
EXPERIMENTS
We evaluate our method in simulation and the real world. In the simulation, we simulate quadruped robot in a of challenging and diverse environments. In the real world, we conduct experiments in indoor scenarios with obstacles and
with complex terrain and novel obstacles.
5.1 ENVIRONMENTS IN SIMULATION
We design 6 simulated environments with varying terrain, obstacles to avoid, and spheres collect for reward bonuses. Spheres are added to see whether agents are able to distinguish objects and their associated functions based on their appearance. All obstacles and spheres are randomly initialized and remain static throughout the episode unless stated otherwise. Specifically, our environment include: Wide Obstacle (Wide Obs.): wide cuboid obstacles on flat terrain, without spheres; Thin Obstacle (Thin Obs.): numerous thin cuboid obstacles on flat terrain, without spheres; Wide Obstacle & Sphere (Wide Obs.& Sph.): wide cuboid obstacles on flat terrain, including spheres that give a reward bonus when collected; Thin Obstacle & Sphere (Thin Obs.& Sph.): numerous thin cuboid obstacles and spheres on flat terrain; Moving Obstacle: similar to the Thin Obs. environment, but obstacles are now dynamically moving in random directions updated at a low frequeney. Mountain: a rugged mountain range with goal on the top of the mountain. We show 4 environments above in Figure 1, omitting Wide Obs. and Thin Obs. for simplicity. We provide further details the observation and action space, specific reward function, and relevant
in Appendix A.
Reward Function. For all environments, adopt the same reward function containing the following terms: (i) Forward reward incentivizing the robot to move forward along a task-specific direction, i.c. towards the goal position in the Mountain environment (visualized as the red sphere in Figure 1), or
5

---

Published as a conference paper at ICLR 2022
-05
000
(a) In environment with obstacles, the agent learns to automatically attend obstacles.
10
(b) On challenging terrain, the agent attends to the goal destination and the local terrain in an alternative manner. Figure 3: Self-attention from shared Transformer module. We visualize the self-attention between the proprioceptive token and all visual tokens in the last layer of our Transformer model. We plot the attention weight over visual input where warmer color represents larger attention weight.
the move along the axis in other environments (i.e. moving forward); (ii) Sphere reward for each sphere collected; (iii) Alive reward encouraging the agent to avoid unsafe situations, e.g. falling; and (iv) Energy usage penalty encouraging the agent to use motor torque of small magnitude.
5.2 BASELINE AND EXPERIMENTAL SETTING
To demonstrate the importance of visual information for locomotion in complex environments, as well as the effectiveness of our Transformer model, we compare our method to: State-Only baseline that only uses proprioceptive states; Depth-Oni Ouly baseline that only uses visual observations; State- Depth-Concat that uses both proprioceptive states and vision, but without our proposed Transformer. The State-Depthh Saate-Depth-Coneee Centh- baseline uses linear projection to project visual features into feature vector that has the same dimensions as the proprioceptive features. The State-Depth-Conca baseline then concatenates both features and feeds into the value and policy networks. We also introduce Hierarchical Reinforcement Learning (HRL) baseline described in Jain al. (2020), but without the use of the trajectory generator for a fair comparison (We follow Jain et al. (2020) faithfully and results indicate that it works expected). We train all agents using PPO (Schulman al., 2017) and share the same proprioceptive and visual encoder for the value and policy network.
Evaluation Metric and Training Samples. We evaluate policies by their mean episode return, and two sppecfcc evaluation metrics: (i) the distance (in meters) an agent moved along its target direction; and (ii) the number of collisions with obstacles per episode (with length of 1 steps). The collision is examined at every time step. and we only compute the collision when the robot pass by least one obstacle. We train all methods for 15M samples with 5 different random seeds and report the mean and standard deviation of the final policies.
5.3 ATTENTION MAPS
To gain insight into how Transformer model leverages spatial information and recognizes dominant visual regions for decision-making at different time steps, we visualize the attention map of our policy on the simulated environment in Figure 3. Specifically, we compute the attention weight Wi, between the proprioceptive token and all other visual tokens and visualize the attention weights on the corresponding visual region of each token. In the top row, we observe that the agent pays most attention to nearby obstacles in the front, i.e. objects that the agent needs to avoid to move forward.
The attention also evolves when new obstacles appear or get closer. In the Mountain environment (bottom row), the agent attends alternatively to two different types of regions: the close terrain immediately influencing the locomotion of the robot, and regions corresponding to the task-specific direction towards the target. The robot first attends to the terrain in front to step the ground (Ist & 3rd frame), once the agent is a relatively stable state, it attends to the goal far away to perform longer-term planning (2nd & 4th frame). The regions attended by the agent are highly task-related and this indicates that our model learns to recognize important visual regions for decision-making.
5.4 NAVIGATION ON FLAT TERRAIN WITH OBSTACLES
Static Obstacles without Spheres. We train all methods on navigation tasks with obstacles and flat terrain to evaluate the effectiveness of modal fusion and stability of locomotion. Results are shown in Figure 4 (a). Our method, the HRL baseline, and the State-Depth-CC Concct baseline significantly outperform the State-Only baseline in both the Thin Obstacle and Wide Obstacle environment, demonstrating clear benefit of vision for locomotion in complex environments. Interestingly, when

---

Published as a conference paper at ICLR 2022
Thin Obs
Thin Clbx,, on Wide Otty)
Thin Obs & 8ph.
Moving Otsilacle
2000
1900
1000
Wide Otss
Wide Obs (Trin Thin
Wide Obs Sph.
Mountsn
2000
2000
3000
two
2006
1900
Milio Framess
Number of Trainin Epochs
Milion Frames
Milion Framas (d)
(b)
(c)
Siate-Depth-Comcat
State-Only
Depth-Only
HRL
Ours
Figure 4: Training and evaluation curves on simulated environments (Concrete lines and shaded areas shows the mean and the std over 5 seeds, respectively). For environment without sphere (in (a)), our method achieve comparable training performance but much better evaluation performance unseen environments (in (b)). For more challenging environment (in (c) and (d)) our method achieve better performance and sample efficiency.
Table 1: Generalization. We evaluate the generalization ability of all methods by evaluating on unseen environments. Our method significantly outperform baselines on both metries (longer distance & less collision).
Distance Moved (m) f
Collision Happened 4
Thin (Train
Wide (Tsain
Thin OTs,nT
Wide (Tsan on Thin Obs.)
on Wide Obs.)
Thin Obs.)
Wide Obs.)
State-Only Depth-Only
3.6:1.3
5.910.9
456.31262.2
545.1
1.1
0.1
Mate--Deph-CCod
5,612.1
7.112.0
406.8189.5
$31.1:192.8
HRL
5.8 #2.2
11.521.8
527.9499.0
238.8+59.5
Ours
8. 8.2.12.5
14.212.8
310.4:151.3
82. 8232:103.8
the environment appearance is relatively simple (e.g., the Wide Obstacle environment), the Depth- Only baseline can learn a reasonable policy without using proprioceptive states. We surmise that the agent can infer part of the proprioceptive state from visual observations for policy learning. This phenomenon suggests that modeling the correlation between different modalities and better fusion techniques are essential for good policy. We also observe that the simpler State- Saate--Dpth-Cnncca baseline performs as well as our Transformer-based model these environments. We conjecture that this is because differentiating obstacles from flat terrain is not perceptually complex task, and simple concatenation, therefore proves sufficient for policy learning.
We further evaluate the generalization ability of methods by transferring methods trained with thin obstacles to environments with wide obstacles, and vice versa. Figure 4 (b) shows generalization measured by episode return, and Table 1 shows average the quantitative evaluation results. While the State-Depth-Conee Concat baseline is sufficient for training, we find that our Transformer-
method improves episode return in transfer by much 69% and 56% in the Wide and Thin obstacle environments, respectively, over the State-Depth- Concct baseline. Compared with the HRL baseline, the improvements of our method are 257.6% and 118. 2%, respectively. We observe that our method moves significantly farther on average, and reduces the number of collisions by 290.5% 402% and 663% over the HRL baseline, the State-Depth-Conl Concat and State-Only baselines when trained on thin obstacles and evaluated on wide obstacles. The Depth- Oppyh-Only baseline fails to generalize across environments and no collision occurs as the robot moves too little to even collide with obstacles. Interestingly, we observe that the generalization ability of the State-Depth-Cone training progresses, whereas for our method it either plateaus or increases over time. This indicates that our method is more effective at capturing essential information in the visual and proprioceptive information during training, and is less prone to overfit to training environments.
decreases as
Static Obstacles with Spheres. We now consider perceptually more challenging setting with the addition of spheres in the environment; results are shown in Figure 4 (c). We observe that with additional spheres, the sample efficiency of all methods decreases. While spheres with positive reward provide the possibility for higher episode return, spheres increase complexity in two ways: (i) spheres may lure agents into areas where it is prone to get stuck; and (ii) although spheres do not block the
7

---

Published as a conference paper at ICLR 2022
Table 2:
Evaluation on environments with spheres. We evaluate the final policy of all methods. Our method achieved the best performance on almost all environment for all metrics.
Distance Moved (m) t
Sphere Reward f
Collision Happened +
Thin Obs.
Wide Obs.
Thin Obs.
Wide Obs.
Thin Obs.
Wide Obs.
Sph.
& Sph.
Sph.
& Sph.
Sph. & Sph.
State-Only
5.61186
7.4:2.8
80.0143.2
80.0
450.2
556.51173.1
Depth-Only
0.010.1
5.2=3.9
0.0.0.0
33.3147.1
Sate--epth--oneee
13.1+2.3
11.4#3.3
206.0:41.1
193.31:24.9
229. 2 1.5.3
47.2
HRL.
10.8
11.3=2.9
166.7454.4
288.94154.8
256.8=87.4
423.3:170.0
Ours
15.2#1.8
14.5=0.7
233.3247.11
220.0
256.2=70.0
54.6220.8
agent physically, they may occlude the agent's vision and can be visually difficult to distinguish from obstacles in depth map. We observe that with increased environmental complexity, our method baseline in the final
consistently outperforms both the HRL baseline and the State-Depth-Conee performance and sample efficiency. We report the average distance moved, number of collisions, and the reward obtained from collecting spheres, in Table 2. Our method obtains a comparable sphere reward but longer moved distance, which indicates that our Loco'Transorree method is more capable of modeling complex environments using spatial and cross-modal attention.
Moving Obstacles. When the positions of obstacles are fixed within an episode, the agent may learn to only attend to the closest obstacle, instead of learning to plan long-term. To evaluate the ability of long-term planning. we conduct a comparison in an environment with moving obstacles to simulate real- world scenarios with moving objects like navigating in the human crowd. The top row of Figure 4 (d) and Table shows that the State-Only baseline and the Depth-Only baseline both perform poorly, and the HRL baseline performs worse than the State-Depth-Conn Concct baseline. These results indicate that the State-Only baseline lacks planning skills, which can be provided by visual observations, and the hierarchical policy can not fuse the information from different modalities effectively when the environment is sufficiently complex. While the Daee--Depthhoottee of distance, it collides more frequently than our method. This indicates that the baseline fails to recognize the moving obstacles, while our method predicts the movement of obstacles and takes a detour to avoid potential collisions. In this case, the conservative policy obtained by our method achieved better performance in terms of episode return though it did not move farther. We deduce that with only a compact visual feature vector, it is very hard for the Seate-Depth-CCe keep track of the movement of obstacles in the environment. On the other hand, is easier to learn and predict the movement of multiple obstacles with our method since the Transformer provides an attention mechanism to model the visual region relations.
Table 3: Evaluation results on the Moving Obsta- elc Environment. Method
Distance Moved (m) t Happened +
Collision
State-Only
6.0+1.3
129.4425.4
Depth-Only
1.1
State Sate-Depth-Concat HRL
16.3#1.7
88.4+34.0
7.142.6
75.8+110
Ours
11.3+2.
67.9+18.1
baseline performs better in terms
baseline to
Ablations. We evaluate the impor- tance of two components of our Trans- former model on the Thin Obs. & Sph. environment: (1) the number of Transformer encoder layers; and (2) the number of visual tokens (N/2 visual tokens). Results
Table 4: Ablation study on Thin Obs. & Sph.: We perform ablations on Thin Obs & Sph. environment and adopt the best setting (N 4, 2) for all environments, which includes 16 visual tokens and 2 Transformer encoder layers.
(a) On Number of Visual Tokens
(b) On Number of Layers
Method
Episode Return t
Method
Episode Return 1
shown in Table 4. From Table 4b, we observe that the performance of our model is relatively insensitive to the number of Transformer encoder layers. For ablation on the number of visual tokens, we change the kernel size and the stride of the last convolutional layer in our visual encoder to get visual features with different shapes and different numbers visual tokens. From Table 4a, we can see that the performance of our method is positively correlated with the number of the visual tokens. With a fixed size of the visual feature map, a higher number of tokens directly results in a smaller receptive field for each visual token. Because method performs spatial cross- modality attention across all tokens, our model benefits from richer low-level visual information. This indicates a potential for our model to work with high-resolution visual input and in more complicated environments and complex tasks.
Ours (N=1)
1204.8+243.6
Ours (L=1)
1509.74244.8
Ours (N=2)
1418.12167.
Ours (L=2)
1551.511200
Ours (N=4)
1551.5+120.4
Ours (L=3)
1423.5+110.7
8

---

Published as a conference paper at ICLR 2022
5.5 NAVIGATION ON SIMULATED UNEVEN TERRAIN
We also evaluate all methods on uneven, mountainous terrain. The bottom row of Figure 4 (d) and Table 5 shows training curves and the mean distance moved for each method, and our method improves over all baselines by large margin in episode
Table 5: Evaluation Result on the Mountain environment.
Method
3D Distance Moved (m) t
return. return. Despite having access to depth images, the State- Depth- baseline does not show any improvement over the State-Only baseline in episode return. We conjecture that naively projecting spatial-visual feature into vector and fusing multi-modality information with a simple concatenation can easily lose the spatial structure of visual information. Although the HRL baseline moves farther among baselines, it does not obtain higher episode return, indicating the HRL baseline
State-Onl Only
3.741.6
Depth-Only
3.020.5
State- State-Depth- See--eepthhhooee Concat
4.7 4:0.8
HRI.
6.320.3
Ours
6.8 1111
able utilize the visual guidance towards the target. Our Transformer- based method better captures spatial information such as both global and local characteristies of the terrain and more successfully fuses spatial and proprioceptive information than simple concatenation.
5.6 REAL-WORLD EXPERIMENTS
To validate our method in different real-world scenes beyond the simulation, we conduct real-world ex- periments in both indoor scenarios with obstacles (referred as Indoor & Obs.) and in-the-wild forest with complex terrain and trees (referred to For- est)) as shown in Figure 5. As the HRL baseline found to not generalize well unseen environ- ments as shown in Figure 4 (b) and Table 1, we only deploy policies learned in simulation using our Lo- co coTransformer and the Saae-Deeth-Concca baseline on Unitree Al Robot (Unitree, 2018). The poli- cies are trained with the Thin Obstacle environment randomized with uneven terrains. All the world deployment experiments are repeated 15 times across different seeds. Details about robot setup are pro- vided in Appendix A. Since it is challenging mea- sure the exact duration of collision with obstacles in the real world, we instead report the number of times that robot collides with obstacles (Collision Count) as a measure of performance.
(a) Indoor & Obs.
(b) Forest
Figure 5: Real World Samples We evaluate method real-world scenarios with different ob- stacles on complex terrain.
Figure 6: Experiment results real-world: We perform real-world experiment on Indoor & Obs. and Forest environments.
Method
Distance Moved (m) t
Collision
Times +
State- Sette-Depth-Concal
5.0=2.6
0.4:0.5
Ours
9.6=2.2
0.3
(a) Indoor & Obs.
Method
Distance Moved (m) f
Collision
Count
Saate-Deth--Couie
5.1
0.3
As shown in Table 6, our method outperforms the baseline by a large margin in both scenarios. In the Indoor & Obs environment, our method moves 92% % farther than the baseline and collides less. When facing complex terrain and unseen obstacles in the Forest environment, our method greatly improves over the baseline; our policy moved approximately 90% % farther without colliding into any obstacles, while the baseline frequently collides into trees and gets stuck in potholes. We generally observe that method is more robust than the baseline when deployed in the real world, indicating that our method better captures the object structure from visual observations, rather than overfitting the appearance of objects during training.
Ours
9.622.0
0.0
(b) Forest
CONCLUSION
We propose to incorporate the proprioceptive and visual information with the proposed LocoTrans- former model for locomotion control. By borrowing the visual inputs, we show that the robot can plan to walk through different sizes obstacles and even moving obstacles. The visual inputs also helps the locomotion in challenging terrain such as mountain. Beyond the training environment, we also show that our method with the mrods-todality Transformer achieves better generalization results when testing unseen environments and in the real world. This shows our Transformer model provides an effective fusion mechanism between proprioceptive and visual information and new possibilities reinforcement learning with information from multi-modality.

---

Published as a conference paper at ICLR 2022
7 REPRODUCIBILITY STATEMENT
To ensure the reproducibility of our work, we provide the following illustrations in our paper and appendix:
Environment: We provide the detailed description of the environment in Section 5.1, as well as the specific observation space, action space and reward function in Appendix A.2.
Implementation Details: We provide all implementation details and related hyperparame- ters for both our methods and baselines in Section 4.2 and Appendix B.
Real Robot Setup: We provide all relavant details about setting up the real robot and conduct wealdworld experiment in Appendix A.3.
We have released the code, environment and videos our project page: https::/rchhtt github. ic/ LocoTrans ietrranffrrre former/. We believe the open source of our code and environment will be important contribution to the community.
Acknowledgement: This work was supported, in part, by gifts from Meta, Qualcomm, and TuSim- ple.
REFERENCES
Hassan Akbari, Linagzhe Yuan, Rui Qian, Wei-Hong Chuang, Shih-Fu Chang, Yin Cui, and Boqing Gong, Vatt: Transformers for multimodal self-supervised learning from raw video, audio and text, 2021. 3
Arena, L. Fortuna, M. Frasca, L. Patané, and M. Pavone. Realization of a enn-drived cockroach- inspired robot. 2006 IEEE International Symposium on Circuits and Systems, pp. 4 pp.-, 2006.
Richard Bellman. A markovian decision process. Journal of Mathematics and Mechanics, 6(5): 679-684, 1957.
Gerardo Bledt and Sangbae Kim. Extracting legged locomotion heuristics with regularized predictive control. In 2020 IEEE International Conference Robotics and Automation (ICRA), pp. -126-112. IEEE, 2020.
Gerardo Bledt, Matthew J Powell, Benjamin Katz, Jared Di Carlo, Patrick M Wensing, and Sangbae Kim. Mit cheetah 3: Design and control of robust, dynamic quadruped robot. 2018 JEEE/RS. International Conference on Intelligent Robots and Systems (IROS), pp. 2245-2252, IEEE, 2018.
Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neclakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint wrXi::2005.1165. 2020. 3
R. Calandra, Andrew Owens, Dinesh Jayaraman, Justin Lin, Wenzhen Yuan, Jitendra Malik, E. Adel- son, and Sergey Levine. More than feeling: Learning to grasp and regrasp using vision and touch. IEEE Robotics and Automation Letters. 3:3300-3307, 2018.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European Conference Computer Vision, pp. 213-229. Springer, 2020.
Jan Carius, René Ranftl, Vladlen Koltun, and Marco Hutter. Trajectory optimization for legged robots with slipping motions. IEEE Robotics and Automation Letters, 4(3):3013-3022 2019. doi:
10.1 10.11009... 292.9997799999..... 3
Jared Di Carlo, Patrick M. Wensing, Benjamin Katz, Gerardo Bledt, and Sangbae Kim. Dynamic locomotion in the MIT cheetah through convex preectpredictive control. In 2018 IEEE/RSJ International Conference
Intelligent Robots and Systems, IROS 2018, Madrid, Spain, October 1-3, 2018, pp. 1-9. IEEE, 2018. doi: 10. 10.1110/ http:/RRS..01..5...000 2018.859448, URL https: ddoi.or// 10. 1109 TROS. 201 8. 85944 8. 3, 19, 20
10

---

Published as a conference paper at ICLR 2022
Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever. Generative pretraining from pixels. In International Conference on Machine Learning, pp. 1691 1703. PMLR, 2020a.
Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. Uniter: Universal image-text representation learning. In European Conference on Computer Vision, pp. 104-120. Springer, 2020b. 3
Rewon Child, Scott Gray, Alee Radford, and llya Sutskever. Generating long sequences with sparse transformers. arXiv preprint arXiv: 1904. 2019. 3
Xingye Da, Zhaoming Xie, David Hoeller, Byron Boots, Animashree Anandkumar, Yuke Zhu, Buck Babich, and Animesh Garg. Learning contact-adaptive controller for robust, efficient legged locomotion. ArXiv, abs/2009.10015 2020. 2
Delcomyn and M. Nelson. Architectures for a biomimetic hexapod robot. Robotics Auton. Syst., 30:5-15, 2000. 1
Jacob Devlin, Ming-Wor Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.048005 2018.
Jared Di Carlo, Patrick M Wensing, Benjamin Katz, Gerardo Bledt, and Sangbae Kim. Dynamic locomotion in the mit cheetah through convex model-predictive control. 2018 IEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pp. 1-9. IEEE, 2018. 3
Yanran Ding, Abhishek Pandala, and Hae- Won Park. Real-time model predictive control for versatile dynamic motions in quadrupedal robots. In 2019 International Conference on Robotics and Automation (ICRA), pp. 8484-8490. IEEE, 3
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint 1IXi::2010.11929, 2020. 3
Alejandro Escontrela, George Yu, Peng Xu, Atil Iscen, and Jie Tan. Zero-shot terrain generalization for visual locomotion policies, 2020. 2.
Christian Gehring, Stelian Coros, Marco Hutter, Michael Blösch, Mark A. Hoepflinger, and Roland Siegwart. Control of dynamic gaits for a quadrupedal robot. In 2013 IEEE International Conference Robotics and Automation, Karlsruhe, Germany, May 6-10, 2013, pp. 3287-3292. IEEE, 2013. doi: 1109/CCA.2011..6631 66310 0331035.3 3
URL htttps://///tt http//di.orgg...11/1 ICR.///CAA...000 2013..
Hartmut Geyer, Andre Seyfarth, and Reinhard Blickhan. Positive force feedback in bouncing gaits?
Proceedings of the Royal Society London. Series B: Biological Sciences, 270(1529):113--2183, 2003. 2
Ruben Grandia, Farbod Farshidian, Alexey Dosovitskiy, René Ranftl, and Marco Hutter. Frequency- aware model predictive control. IEEE Robotics and Automation Letters, 4(2):1517-1524, 2019. 2019,
Saurabh Gupta, Varun Tolani, James Davidson, Sergey Levine, R. Sukthankar, and J. Malik. Cognitive mapping and planning for visual navigation. International Journal of Computer Vision, 128:1311- 1330, 2019.
Nicklas Hansen and Xiaolong Wang. Generalization in reinforcement learning by soft data augmen- tation. International Conference on Robotics and Automation, 2021. 3
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross B. Girshick. Momentum contrast for unsupervised visual representation learning. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9726-9735, 2020. 3

---

Published as a conference paper at ICLR 2022
Nicolas Heess, Dhruva TB, Srinivasan Sriram, Jay Lemmon, Josh Merel, Greg Wayne, Yuval Tassa, Tom Erez, Ziyu Wang, S. M. Ali Eslami, Martin A. Riedmiller, and David Silver. Emergence of locomotion behaviours in rich environments. CoRR, abs/1707.02286, 2017. URL http: 02286.
///axivve wttp:///gg//aaaaaaa//o/aa//a/aaa
Lisa Anne Hendricks, John Mellor, Rosalia Schneider, Jean-Baptiste Alayrac, and Aida Nematzadeh. Decoupling the role of data, attention, and losses in multimodal transformers. arXiv preprint arXiv:2102.00529, 2021. 3
Ronghang Hu and Amanpreet Singh. Unit: Multimodal multitask learning with unified transformer, 2021. 3
Zhicheng Huang, Zhaoyang Zeng, Yupan Huang, Bei Liu, Dongmei Fu, and Jianlong Fu. Seeing out of the box: find-to-edd pre-training for vision-language representation learning, 2021. 3
Jemin Hwangbo, J. Lee, A. Dosovitskiy. Dario Bellicoso, Vassilios Tsounis, V. Koltun, and M. Hutter. Learning agile and dynamic motor skills for legged robots. Science Robotics, 4, 2019. 3
Atil Iscen, Ken Caluwaerts, Jie Tan, Tingnan Zhang, Erwin Coumans, Vikas Sindhwani, and Vincent Vanhoucke. Policies modulating trajectory generators. In 2nd Annual Conference Robot Learning. CoRL 2018. Zürich, Switzerland, 29-31 October 2018, Proceedings, vol- ume 87 of Proceedings of Machine Learning Research, pp. 916-926. PMLR, 2018. URL http: pttp///cccceeeeeseoe/ee ngs ml p.r..ppsssss..... /vrsss//v//// seciiscnnnni. 88 3
Max Jaderberg, Volodymyr Mnih, Wojeiech Marian Czarnecki, Tom Schaul, Joel Z. Leibo, David Silver, and Koray Kavukcuoglu. Reinforcement learning with unsupervised auxiliary tasks. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. URL https: http:::///peeeeviieeee net. met/fooune id= dd-SJ6ypDxxg.... 3
Deepali Jain, Atil Iscen, and Ken Caluwaerts. Hierarchical reinforcement learning for quadruped locomotion. In 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2019, Macau, SAR, China, November 3-8, 2019, pp. 7551-7557, IEEE, 2019a. doi: 10. 1109/RR0S40897.2219.8886 8967913. 2,
URL https: /ttpaps:://...oooo o 1109 10009/00000099720099999
Deepali Jain, Atil Iscen, and Ken Caluwaerts, From pixels to legs: Hierarchical learning of quadruped locomotion, 2020. 3, 6
Divye Jain, Andrew Li. Shivam Singhal, Aravind Rajeswaran, Vikash Kumar, and Emanuel Todorov. Learning deep visuomotor policies for dexterous hand manipulation. In 2019 International Conference on Robotics and Automation (ICRA), pp. 3636-3643 2019b. doi: 10. 209/ 8794033. 3.3
Yiding Jiang, Shixiang Gu, Kevin Murphy, and Chelsea Finn. Language as an abstraction for hierarchical deep reinforcement learning. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché-Buc, Emily B. Fox, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Pro- cessing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 9414-9426, 2019. URL https: http///pooeeeennsee/eecee. neurips cc/paper/201 9 2aw222556555-----aaaeeeee Abstract .tt.aaccctttt......m.m html. 2
G. Kahn, P. Abbeel, and Sergey Levine. Badgr: An autonomous self- supersupe learning-bass based navigation system. IEEE Roboties and Automation Letters, 111312-1319, 2021.
Nate Kohl and Peter Stone. Policy gradient reinforcement learning for fast quadrupedal locomotion. In Proceedings of the 2004 IEEE International Conference on Robotics and Automation, ICRA 2004, April 26 May 1, 2004, New Orleans, LA, USA, pp. 2619-2624. IEEE, 2004. doi: 10.1 ROB0T.2004.107456 URL https: //oii org 110 1009/011111 10109/ROOO ROBBOT 2004. 1307456.
Ashish Kumar, Zipeng Fu, Deepak Pathak, and Jitendra Malik. Rma: Rapid motor adaptation for legged robot. Robotics: Science and Systems, 2021. 3
12

---

Published as a conference paper at ICLR 2022
Michael Laskin, Kimin Lee, Adam Stooke, Lerrel Pinto, Pieter Abbeel, and Aravind Srinivas. Rein- forcement learning with augmented data. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tiee Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020. URL https: /propeeeee/p////eeeeepeeee ngs neuri ps..ppp... cass.ccpper cc/paper////cccccpppcc w1w.e666e1ddd2ddaaaaaaaaa
2020/hash/
Abstract htm1.
Joonho Lee, Jemin Hwangbo, Lorenz Wellhausen, Vladlen Koltun, and Marco Hutter. Learning quadrupedal locomotion over challenging terrain. Science robotics, 5(47), 2020a. 1,
Michelle A Lee, Yuke Zhu, Peter Zachares, Matthew Tan, Krishnan Srinivasan, Silvio Savarese, Li Fei-Fei, Animesh Garg, and Jeannette Bohg. Making sense of vision and touch: Learning multimodal representations for contact-rich tasks. IEEE Transactions on Robotics, 36(3):582-566, 2020b.
Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuomotor policies. J. Mach. Learn. Res., 17:39:1-39:40, 2016. URL http:/ / /wm...mmrooooo..oo org/ papers/v17/15 522. html. 1,3
Sergey Levine, Peter Pastor, Alex Krizhevsky, Julian Ibarz, and Deirdre Quillen. Learning hand- eye coordination for robotic grasping with deep learning and large-scale data collection. Int. J. Robotics Res., 37(4-5):411-366 2018. doi: 10. 00.1177/2273364911103188 URL https: 1,3
//doi, 1ttp:/1..111110 /ttp://.1117777774911111
Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. Visualbert: A simple and performant baseline for vision and language. arXiv preprint arXiv:1908.03555 2019. 3
Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang. Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, et al. Oscar: Object-semantics aligned pre-training for vision- -aiiune tasks. In European Conference on Computer Vision, pp. 121-137. Springer, 2020. 3
Xingyu Lin, Harjatin Singh Baweja, George Kantor, and David Held. Adaptive auxiliary task weighting for reinforcement learning. Advances in neural information processing systems, 32, 2019. 3
Y. Luo, Jonathan Hans Soeseno, Chen, and Wei-Chao Chen. Carl: Controllable agent with reinforcement learning for quadruped locomotion. ArXiv, abs/2005.03288, 2020. 3
Jonathan Samir Matthis, Jacob L Yates, and Mary M Hayhoe. Gaze and the control of foot placement when walking in natural terrain. Current Biology, 2828):1224-12333 2018.
Josh Merel, Saran Tunyasuvunakool, Arun Abuja, Yuval Tassa, Leonard Hasenclever, Vu Pham, Tom Erez, Greg Wayne, and Nicolas Heess. Catch carry: reusable neural controllers for vision-guided whole-body tasks. ACM Trans. Graph. 39(4):39, 2020. doi: 10.1 141114/38666699339477777 URL https://dot org/10 100001..0000 145/ /35.3333656.... 33924744
Mirowski, Razvan Pascanu, Fabio Viola, Hubert Soyer, Andy Ballard, Andrea Banino, Misha Denil, R. Goroshin, L. Sifre, K. Kavukcuoglu, D. Kumaran, and R. Hadsell. Learning navigate in complex environments. ArXiv, abs/161 1.03673, 2017.
Hirofumi Miura and Isao Shimoyama. Dynamic walk of biped. The International Journal of Robotics Research, 3(2))60-74, 1984. 2
Volodymyr Mnih, Koray Kavukeuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Pe- tersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep rein- forcement learning. Nature, 518(7540):529-5 533, February 2015a. ISSN 00280836. URL http://dx. doi. www.ddo....ooooooooooo 10t////tt//eeeeeeeee 4236.
13

---

Published as a conference paper at ICLR 2022
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Mare G. Bellemare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Pe- tersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep rein- forcement learning. Nature, 518(75400:529-533) February 2015b. ISSN 00280836. URL http://dxx wtw.::o............111111 http.//rrrr....///auuueee1...... htw.110//0//.uur11.......
Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran. Image transformer. In International Conference on Machine Learning, pp. 4055-4064. PMLR, 2018. 3
Deepak Pathak, Pulkit Agrawal, Alexei A. Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In ICML, 2017. 3
X. Peng, G. Berseth, KangKang Yin, and M. V. D. Panne. Deeploco: dynamic locomotion skills using hierarchical deep reinforcement learning. ACM Trans. Graph., 36:41:1-41::1: 2017.
Xue Bin Peng, Pieter Abbeel, Sergey Levine, and Michiel van de Panne. Deepmimic: Example- guided deep reinforcement learning of physics-based character skills. ACM Trans. Graph., (4):143:1-143:14, July 2018. ISSN 0730-0301. doi: 10.1 (0/1114//77 //doi. acm. org.rrggooooo. /10. 1et..1111/115511 3201311.
URL http:
Xue Bin Peng, Erwin Coumans, Tingnan Zhang, Tsang- Wei Edward Lee, Jie Tan, and Sergey Levine. Learning agile robotic locomotion skills by imitating animals. In Robotics: Science and Systems, 07 2020. doi: 10.15607/RSS 2020.002..00 3, 19
Aditya Prakash, Kashyap Chitta, and Andreas Geiger. Multi-modal fusion transformer for end-to-end autonomous driving. In Conference on Computer Vision and Pattern Recognition (CVPR), 2021. 3
Mare H Raibert. Hopping in legged modeling and simulation for the two-dimensional one-legged case. IEEE Transactions on Systems, Man, and Cybernetics, SMC-14(3)455 1984.
Alexander Sax, Bradley Emi, Amir R. Zamir, Leonidas Guibas, Silvio Savarese, and Jitendra Malik. Mid-level visual representations improve generalization and sample efficiency for learning visuomotor policies. 2018. 3
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 2017. 6
Max Schwarzer, Ankesh Anand, Rishab Goel, R Devon Hjelm, Aaron Courville, and Philip Bach- man. Data-efficient reinforcement learning with self seef-predietive representations. arXiv preprint arXiv:2007.05592, 2020. 3
Aravind Srinivas, Michael Laskin, and Pieter Abbeel. Curl: Contrastive unsupervised representations for reinforcement learning. arXiv preprint 3XXv:20004044366 2020. 3
Adam Stooke, Kimin Lee, Pieter Abbeel, and Michael Laskin. Decoupling representation learning from reinforcement learning. CoRR, abs/2009.08319, 2020. URL https: //tpxiv.ogg////////o
abs/2009 08319.
Weijie Su, Xizhou Zhu, Yue Cao. Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai. VI-bert: Pre-training of generic lingui-iin representations. arXiv preprint 19Xiv:1988.00 (et::00080000000 2019. 3
Chen Sun, Austin Myers, Carl Vondrick, Kevin Murphy, and Cordelia Schmid. Videobert: A joint model for video and language representation learning. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 7464-7473. 2019. 3
Yu Sun, Wyatt L. Ubellacker, Wen-Loong Ma, Xiang Zhang, Changhao Wang. Noel V. Csomay- Shanklin, Masayoshi Tomizuka, Koushil Sreenath, and Aaron D. Ames. Online learning of unknown dynamics for basel controllers in legged locomotion. IEEE Robotics and Automa- fion Letters (RA-I.), 2021. 3, 20
14

---

Published as a conference paper at ICLR 2022
Hao Tan and Mohit Bansal. Lxmert: Learning cross-modality encoder representations from trans- formers. arXiv preprint 19XXv:1908.074999 2019. 3
Jie Tan, Tingnan Zhang, Erwin Coumans, Atil Iscen, Yunfei Bai, Danijar Hafner, Steven Bohez, and Vincent Vanhoucke. Sim-to-real: Learning agile locomotion for quadruped robots. In Hadas Kress- Gazit, Siddhartha S. Srinivasa, Tom Howard, and Nikolay Atanasov (eds.), Robotics: Science and Systems XIV. Carnegie Mellon University, Pittsburgh, Pennsylvania, USA, June 26-30, 2018, 2018. doi: 10.15607/RSS 211.52221..1...0 XIV.10. URL oot.:///ooooo/////ooooooooooooot http://wwwooooiiioooceeeeeddddee org/ rss arg/rssll 4 4/p10. html. 3
Stephen Tian, Frederik Ebert, Dinesh Jayaraman, Mayur Mudigonda, Chelsea Finn, Roberto Calandra, and Sergey Levine. Manipulation by feel: Touch- based control with deep predictive models. In 2019 International Conference on Robotics and Automation (ICRA), pp. 818-824. IEEE, 2019. 1
Nick Torkos and Michiel van de Panne. Footprint-based quadruped motion synthesis. In Proceedings of the Graphics Interface 1998 Conference, June 18-20, 1998, Vancouver, BC, Canada, pp. 151- 160, June 1998. URL gi1998- pdf.
org/ wtt.//pconteet//pp...ooo
Yao- Muo-Hung Hubert Tsai, Shaojie Bai, Paul Pu Liang, J. Kolter, Louis-Philippe Morency, and R. Salakhutdinov. Multimodal transformer for unaligned multimodal language sequences. Pro- ceedings of the conference. Association for Computational Linguistics. Meeting, 2019:6558-6369, 2019. 2
Unitree. A1: More dexterity, more posibility, 2018. URL httpps://ww.wwt http://wwwww.ittee.coo products/al/ 9, 16
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need, 2017. 2, 3,
Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 7794-7803, 2018. 3
Zhaoming Xie, Xingye Da, Michiel van de Panne, Buck Babich, and Animesh Garg. Dynamics randomization revisited: A case study for quadrupedal locomotion. CoRR, abs/2011.02404, 2020. URL
absa 02404. 3,3
Zhaoming Xie, Xingye Da, Buck Babich, Animesh Garg, and Michiel van de Panne. Glide: Gen- eralizable quadrupedal locomotion diverse environments with a centroidal model. CORR, abs/2104.09771, 2021. URL https:/ /tta/aiivv.o./// hatp:/ab/2111 09771. 1.3
Wei Yang, X. Wang. Ali Farhadi, A. Gupta, and R. Mottaghi. Visual semantic navigation using scene priors. ArXiv, abs/18 10.00540.. 2019.
Yuxiang Yang, Tingnan Zhang, Erwin Coumans, Jie Tan, and Byron Boots. Fast and efficient locomotion via learned gait transitions. arXiv preprint arXiv:2104.046:4
2021. 20
Denis Yarats, Amy Zhang, Ilya Kostrikov, Brandon Amos, Joelle Pineau, and Rob Fergus. Improving sample efficiency in model-free reinforcement learning from images, 2019. 3
KangKang Yin. Kevin Loken, and Michiel Van de Panne. Simbicon: Simple biped locomotion control. ACM Transactions on Graphics (TOG), (6633)105-ee 2007. 3
Wenhao Yu, Greg Turk, and C. Karen Liu, Learning symmetric and low-energy locomotion, ACM Trans. Graph., 3744:144:1-144:12 2018. doi: 10.11473199511.32003997 URL https://doi. org/10 /0./10..0 1t//:/111111 1111/11199111 01111191117.721111111.. 3201397. 16
H. Zhang. S. Starke, T. Komura, and Jun Saito. Mode-adaptive neural networks for quadruped motion
control. ACM Transactions on Graphics (TOG), 37:1 11, 2018.
15

---

Published as a conference paper at ICLR 2022
A DETAILED EXPERIMENT SETUP
A.1 DETAILS ON PROPRIOCEPTION AND ACTION
Our Unitree Al robot has 12 Degrees of Freedom (DoF), and use position control set actions for the robot. Specifically, the proprioceptive input contains the following components:
Joint angle: 12-dimensional vector records the angle of each joint.
IMU information: a 4-dimensional vector records orientations and angular velocities.
Base displacement: 3-dimmensiona vector records the absolute base position of robot.
Last action: 12-dimensional vector records the angle change in the last step.
The full proprioceptive vector consists all these vectors over the last three steps retain historical state information. The action is also a 12-dimensional vector that controls the change of all the joint angles. We use 0.5 as the upper bound of action for locomotion stability. We use all default settings of A robot in the official repository.
A.2 REWARD DEFINITION
In all our experiments, we use the same reward function follow:
=
Offorward Rioeward + Otenergy Renergy + O'alive Ralive + Rupheres
(7)
where we Offowward 1, Ccenergy 0.005, O'aive 0.1 for all tasks.
Reorward stands for moving forward reward. In flat environments, it's defined by the moving speed of robot along the x-axis; in mountain environment, it's defined by that along the direction the mountain top (red sphere in Figure 1 Mountain in paper).
Rewergy ensures the robot is using minimal energy, which has been shown to improve the naturalness of motion, similar to Yu et al. (2018). Specifically, we penalize the actions resulting motor torques with large euclidean norm.:
Renergy - ||T|| T is the motor torques.
Ralive encourages the agent to live longer. gives a positive reward of 1.0 at each time step until termination. Dangerous behaviors like falling down and crashing into obstacles will call termination.
Rophere stands for sphere collection reward (whenever applicable) for each sphere collected, and K is the number of spheres collected the current time step.
A.3 REAL ROBOT SETUP
We use the Unitree Al Robot (Unitree, 2018), which has links and 12 degrees freedom for each leg). We mount an Intel RealSense camera at the head of the robot to capture the depth map, and use the robot sensors to get the joint states and IMU for the proprioceptive input. All computations are running with on- on-board resources. We set the control frequency to be 25 Hz and set the action repeat to be 16, so that the PD controller converts the target position commands to motor torques at 400 Hz. We set the KP and KD of PD controller to be 40 and 0.6 respectively. The base displacement removed from the observation for the real-world experiment. For the real-world experiment, we execute the policy on the real-robot for 20 seconds and measure the Euclidean distance between the start and end point the trajectory for evaluating the performance.
B HYPERPARAMETERS
In this section, detail the hyperparameters for each method used in experiments.
16

---

Published as a conference paper at ICLR 2022
Parameters
Range
KP
[40, 901
[0.4, 0.8]
Inertia (x default value)
[0.5, 1.5]
Lateral Friction (Ns m)
[0.5, 1.25]
Mass (x default value)
[0.8, 1.2]
Motor Friction (Nms/ rad)
[0.0, 0.05]
Motor Strength (x default value)
[0.8, 1.2]
Sensor Latency (S)
[0, 0.04]
Table 6: Variation Environment and Robot Parameters.
B.1 DOMAIN RANDOMIZATION
To narrow the reality gap. we leverage domain randomization during training phase. All methods conducted in real world experiments use same group of randomization setting to be fair. Specifically, we set the range of parameters as follow:
During training, besides the domain randomization for the proprioceptive state, we perform domain randomization for visual input. Specifically, each time-step. we randomly choose 3 to 30 values in (64, 64) depth input and set the depth reading to the maximum reading. In this case, we simulate the noisy visual observation the real-world.
B.2 HYPERPARAMETERS SHARED BY ALL METHODS
Here give the details of all hyperparameters that are related reinforcement learning and shared by all tested methods. "Horizon" denotes the episode length in both training and testing, and "Clip parameter" denotes the
of gradients in all trained networks.
Hyperparameter
Value
Horizon
1000
Non-lincarity
ReLU
Policy initialization
Standard Gaussian
# of samples per iteration
8192
Discount factor
.99
Batch size
256
Optimization epochs
Clip parameter
0.2
Policy network learning rate
le-4
Value network learning rate
1e-4
Optimizer
Adam
B.3 STATE-ONLY BASELINE
To keep the comparison fair, we use fully-comnected layers for state-only baseline to keep the network size large enough for higher learning capacity. We also try more layers but observe minor difference performance.
Hyperparameter Network size
Value FC layers with 256 units
B.4 STATE-DEPTHE DEPTH-CONAA BASELINE
Apart from the perception encoder, keep all other parts same for State-Depth-Concal baseline and our LocoTransformer.
Hyperparameter Proprioceptive encoder
Value
2 FC layers with 256 units
Projection Head
2 FC layers with 256 units
17

---

Published as a conference paper at ICLR 2022
B.5 OURS LOCOTRANSFORMER
Hyperparameter
Value
Token dimension
128
Proprioceptive encoder
2 FC layers with 256 units
Projection Head
2 FC layers with 256 units
# of transoformer encoder layers
C MORE ATTENTION VISUALIZATION RESULTS
We offer visualization of attention maps here to show that Traof consistently attends to reasonable parts of environment during an episode. In the Thin Obstacle, robot will be aware of the new appearing obstacles and give emergent reaction escape the threat. For example, in the last row, the robot attends to the closed obstacle and then turns left. Suddenly it attends to the wall, so it reorientates its body avoid risky. In the Mountain, robot not only pays attention the final goal position, but also attend to rugged terrains that are extremely hard to step on. This also shows the planning ability according to different visual regions learned by Transformer architecture. the second case, the robot first attends to goal position to ensure the forward direction, but the uneven rocks also draw its attention.
(a) Thin Obstacle
(b) Mountain
Figure Additional Attention Visualization We visualize more attention map visualization for better understanding of how our Transforrrrr works. Each row shows sequence of attention map to present how the attention of agent evolves.
D COMPARISON WITH MODEL PREDICTIVE CONTROL
To understand the advantages of learning- based locomotion, offer a detailed comparison with the classical quadrupedal robotic control pipeline, which commonly used in both classicalal approaches and recent visual locomotion learning works. With this comparison, understand the main difference between learning- based academic works and industrial solutions, like the famous Boston Dynamics Spot.
also help the community
18

---

Published as a conference paper at ICLR 2022
D.1 VISION-GUIDED WHOLE-BOD BODY CONTROLLER
We mainly follow Carlo et al. (2018) and the code for (Peng et al., 2020) to reproduce the vision- guided whole-body controller for A1. Our high-level vision controller trained with RL perceivinng the visual information same as other approaches and outputing the target linear and angular velocity for low-level controller. Our low-level motion controller provide the actual motor commands to the robot according to current target velocity.
High-level Visual Policy. High-level visual policy outputs the target linear velocity and angular velocity given the stacked depth maps and (optionally) CoM velocity and IMU information. If only given the depth maps, we remove the state encoder part in the original models and keep the Transformer and CNN encoders; if given both the depth maps and the body state (CoM velocity and IMU information), we just keep the two architectures same the main paper. We train the high-level controller with PPO to provide fair and consistent comparison with our method. The control frequency of the high-level controller is 20Hz, with the action repeat set as 10 to guide the low-level controller. We use the Unicycle Model to control the CoM velocity, i.e., specifying the absolute linear velocity and angular (rotating) velocity. In our experiments, the target linear velocity clipped to +0.44 and the target angular velocity is clipped to +-0.3.
Low-level Controller. The low-level controller uses position control for swing actions and torque control for stance actions. Specifically, decide when to swing each leg and how long stance each leg needs in a complete control cycle. The swing action is determined by fixed foot clearance height and controlled by PD-controller with the target foot position. The stance force (torques of each joint in a leg) is computed by model predictive control to track the desired CoM velocity. The whole-body controller outputs a new command in 200HZ and use another action repeat to control the body.
a finite state machine (FSM) based gait scheduler to
Training. We use PPO to train the control policy, similar to other methods in this paper. Here we only demonstrate some main modifications that might influence reproducing the experiments: remove the energy reward term, since the low-level motion control is not learnable; 2) Because the change of the control frequency (all other methods in our paper directly provide low-level command which requires a higher control frequency), we tune the following hyperparameters:
Hyperparameter
Value
Horizon
500
# of samples per iteration
4096
Otforward
0.5
We remove domain randomization terms which are only related to motion robustness.
D.2 RESULTS IN SIMULATION
We perform the training in simulation and answer the following two questions:
Can Transformer-basee architecture still outperform CNN-based one while combining with classical controller?
Is our RL-based framework better than visual policy + classical controller?
Our results show the advantage of our network design and that of our End-to-end RL pipeline.
Comparison between Network Architectures. We train the vision-guided whole body controller in two settings. The two settings are: (i) Stacked depth maps; (ii) Stacked depth maps with CoM velocity and IMU sensor input. The training curves are in Figure 9. Due to the difference of the reward settings and episode length, it's meaningless and unfair to compare the curves with that of RL-based methods. The training results show that for multi-modal setting. our LocoTransformer still outperforms the baseline. For vision-only setting, observe minor difference between transformer and CNN, and speculate that attention mechanism indeed improves the modal fusion rather than image representation learning.
19

---

Published as a conference paper at ICLR 2022
Note that it costs several times of samples to train the controller. We deduce the phenomenon is due to low-level controller is not agile enough to the varying high-level command to gain explicit information for RL optimization.
For customized traditional controllers, such as the MPC controller used by MIT cheetah (Carlo et al., 201 8), the inference speed is around 1 120Hz the Unitree A1 robot. To overcome the limitation of computation, an external laptop is used in (Sun et al., 2021; Yang et al., 2021). In contrast, our end-2-end learned policy is able to utilize the on-board GPU perform real-time inference.
Multi Modal
Vision Only
300
400
350
250
$300
202
220
2000
150
1500
100
50
10
20 lion Control Stops
30
40
10
20 Million Controller Stops
30
50
Baseline
Locotransformer
Figure 8: Training curves of vision-guided whole body controller. For multi-modal input setting, Loco- Transformer still outperforms the baseline.
D.3 RESULTS IN REAL WORLD
To further illustrate the advantage of end-to-end RL for locomotion, we deploy the vision-guided whole-body controller using only visual observation and baseline architecture in the real world. The vision- vision-guided guided whole-body controller controller generates natural and stable gaits, including larger amplitude of feet swing, lower frequency of stepping, and consistent body height. However, when facing obstacles like trees, the vision- rision-guided whole-body controller cannot plan well with the MPC controller for avoidance. It tries to avoid the tree, but since the motion not very flexible and the high-level transitions are not smooth, it collides with the side of the tree, and bounces away to come across the obstacle. While the agent is still moving forward, having more collisions makes the robot unsafe. On the other hand, when trained end-to-end with both vision and legged motion, our approach can flexibly change from a moving forward motion to a turning motion and adjust the speed at the same time. We generally observe much more diverse motions emerge and our policy is able to transition smoothly between these motions according to vision. For in-door environments where the obstacles are larger, there are less chances for the vision-guided whole-body controller to come across the obstacle when the robot is about to collide into it.
BA Finoa
A538
H
Figure 9: Some failure cases of rision-guided whole-body controller due to limitation of agility. The left figure shows that when walking through narrow path, the robot can not quickly turn around and collide into the wall. The right figure denotes that it may collide to the tree in the wild, due to the shape of the obstacles are out of the training distribution and the robot can't I adjust quickly enough avoid.
20

---

Published as a conference paper at ICLR 2022
D.4 QUANTITATIVE COMPARISON REAL WORLD
We also provide quantitative comparisons between our method and the vision-guided whole-body controller. The vision-guided whole-body controller create more collisions even though this helps walking longer distances.
Figure 10: Experiment results in the real-world: We perform real-world experiment on Indoor & Obs. and Forest environments to compare method and the usioon-guided whole-body controller
(a) Indoor Obs.
Method
Distance Moved (m) f
Collision
Times J
Vision-Guided Whole- Body Controller
7.5 9.6=2.2
3.4
Ours
0.3
(b) Forest
Method
Distance Moved (m) f
Collision
Count J
Whole Body Controller
11.613.5 5
1.920.8
Ours
9.6+22.0
0.0+0.0
For in-door environments where the obstacles are larger, there is less chance for the vision-guided whole-body controller to avoid the obstacle when the robot is about to collide into it. Thus the guided whole-body controller performs worse in both collision times and distance moved compared to our approach.
21


Reviews:
[{"review_id": "Q7AbqcdoDPy", "reviewer": "Reviewer_Dcnh", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper is very clear in its exposition, providing a detailed diagram of the method, clearly labeled inputs and outputs, and relevant implementation details. The experiments appear sound, with a number of baselines and ablations provided. I want to emphasize the value of real-world experiments in this space, as the 'sim-to-real' gap can be significant and invalidate otherwise good looking results. Scientifically, the paper proposes an architecture that is novel and can serve as a broader proof point that vision-based locomotion can be competitive and robust when trained with a sufficiently expressive model.\n\nStrengths:\n+ The paper tackles an important research problem: vision-guided locomotion and has made significant progress along this direction.\n+ Novel network architectures (such as Transformers) are under-explored in the legged robots community. This paper demonstrates that incorporating such architecture indeed makes a difference in performance.\n+ The proposed method is validated on a real robot. The evaluations are comprehensive and the conclusions are convincing.\n\nWeakness:\n- The technical novelty is lean. Neither of the key components of the paper are novel: RL for locomotion and Transformers. Although this can be considered as a weakness, it is not a deal breaker given that the combination of these two and the application to legged robots are novel and potentially influential.\n- The results are mostly obstacle avoidance on the flat ground, where legs are not essential. Most of the experiments can be done with a wheeled robot. To show the true value of this paper, more challenging terrain needs to be considered and tested on, such as stairs, stepstones, tall grasses, rocks, etc. In these terrains, both vision and legs are critical. From the accompanying video, it is a bit disappointing that the robot only learns steering for obstacle avoidance, but does not learn foot clearance or foothold location on different types of terrains. The robot always drags the foot, even on pebbles (5:25s in the video) where higher foot clearance is clearly a preferred choice. Since this paper trains end-to-end, I would expect that these behaviors would emerge automatically if trained in relevant environments. Showing these behaviors (foot clearance, foothold location, change of gait pattern) in addition to obstacle avoidance, would significantly strengthen the paper.\n\nAdditional questions:\n1) In addition to domain randomization, does the paper apply other techniques for sim-to-real transfer? For example, I would imagine that there will be a large sim-to-real gap in vision. The depth images from Intel realsense can be noisy and with holes, especially in outdoor environments. Do these sim-to-real gaps in vision cause any problems when deploying the policy on the robot?\n2) How much tuning is needed to learn natural (deployable) locomotion gaits? In the video, the learned locomotion gait is quite reasonable and deployable on the robot. The paper explicitly mentioned that it did not use trajectory generators. Does it purely rely on reward shaping? If so, how much tuning is needed? And what are the most important terms in the reward function that encourage the emergence of natural gaits?\n", "summary_of_the_paper": "This paper proposes an approach to legged locomotion which leverages a Transformer-based model and is trained via end-to-end reinforcement learning. It provides extensive experimental evaluation of the approach in terms of performance and safety metrics, both in simulation and using real-world experiments. The code is expected to be open-sourced.", "summary_of_the_review": "Good paper on relevance and experimental evidence, on a topic that is very much of interest to the robot learning community today.\nNovelty limited due to combination of known techniques.\nEDIT: bumped confidence to a 5 based on comments and rebuttal. This is a solid contribution.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "GniA3xh-2Tt", "reviewer": "Reviewer_XXAB", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The main strengths of the paper:\n\n(1) Proposed a novel transformer based architecture that can train visual-locomotion policies end-to-end, and demonstrated good navigation/obstacle avoidance/uneven terrain walking results in the simulation.\n\n(2)  Zero-shot real world transfer to a A1 robot and demonstrates walking + navigation behavior in various environments.\n\nThe main weakness of the paper:\n\nNot enough baselines to compare with. As the authors cited, there are many approaches to tackle visual locomotion + navigation problem besides end to end training. For example in the hierarchical approach one can combine: learned/optimization based navigation + pre-trained or hand tune walking (i.e. MPC) motions.  So in total even the hierarchical approach can have four different combinations to compare with. Yet I saw non of them here. I would say the authors should include at least one or two such baselines to compare with, and document the performances and cons and pros. \n\n\n", "summary_of_the_paper": "In this paper, the authors proposed a transformer based architecture that combines both visual (depth) and proprioceptive inputs (i.e. IMU and joint angles) to solve visual locomotion tasks. The authors demonstrated that their approach can solve challenging visual navigation tasks and locomotions task on uneven terrains. The proposed method out perform proprioceptive only, visual only, and HRL baselines. The sim trained policy has been demonstrated on the real A1 hardware.", "summary_of_the_review": "The authors proposed to use transformer architecture to solve visual locomotion + navigation tasks. The proposed approach is compared with a few end-to-end trained baselines including HRL and has demonstrated advantages. The authors also deployed the trained policy successfully to the real robot. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "7u6Ge5QNJe2", "reviewer": "Reviewer_GRoY", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Strengths and Weaknesses\n\n### Things I liked about this paper\n- **a powerful framework**: fusing visual and proprioceptive data for quadrupedal locomotion using transformer architectures is an interesting and also valuable approach that works well and sets an excellent opportunity for future work.\n- **a well written paper**: overall the paper is well written and all sections are broadly very clearly described.\n- **useful insight**: I like the provided key insight that proprioceptive states offer contact measurements for immediate reaction while visual sensory observations can help with longer-term planning.\n\n### Things that can be improved\n- **number of seeds is not great**: The adopted model-free approach is known to have very unstable learning process of the dynamics function which ideally requires 10 or more seeds to provide a solid results. Using only 5 seeds is not great. More details below.\n- **prose is not perfect**: There are some minor details and clarifications that may help further improve clarity.\n\nUsing 5 random seeds for a model free approach is rather small as a number. Ideally, the evaluation should be done on 10 or more seeds. In fact, I suspect that some of the results such as the moving obstacles from Table 3, would change if the approach was evaluated on more runs. Nevertheless, the provided training curves seem to have fairly small variance as illustrated in Figure 4 which makes me more inclined to agree that 5 seeds are sufficient to report on accurate results. In addition, I would expect that the variation in the learnt dynamics to primarily affect the performance of the learnt agent on the physical quadruped system.  However, this does not seem to be the case in the reported results, which is great as long as all 5 seeds were used to extract those results. It is great that the paper considers 15 runs per seed but I wonder if the results were acquired through cherry picking best n seeds. This is a detail that is not currently mentioned in the paper but would certainly improve clarity if it did.\n\nThere are a few additional minor comments. Currently, the distance measurement reported in meters is mentioned only in the text and not in the tables. Stating this there too would make it much clearer. Similarly, what exactly does the collision happened represent. Are these total number of collisions over 1000 steps? 'the number of time steps where collision happens between the robot and obstacles\nover the course of an episode' states the explanation seems a bit overly complicated. Why not just 'the number of collisions with obstacles per 1k step long episode' or something along those lines?\n\nThere is a typo in the contribution 'we the propose' should be 'we propose'. Another typo is '... whereas it for our method either plateaus' should be 'whereas for our method it either plateaus'", "summary_of_the_paper": "This work proposes a novel architecture for quadrupedal locomotion that fuses proprioceptive and visual information with a transformer-based model to enable an agent to proactively maneuver environments with obstacles and uneven terrain by anticipating changes in the environment many steps ahead. The method is extensively evaluated in simulation and on a sim to real transfer tasks. The method is shown to both achieve higher reward  but also better capacity to generalise in the context of sim to real. Overall, the paper is well written and the provided evaluation is conducted fairly and well.", "summary_of_the_review": "Overall this paper is written well and has a sound idea that is supported well by an extensive evaluation. There are some minor details that may further improve the quality of the paper but I see this as a strong submission which I can recommend for acceptance.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "2ltsIB0Ir4O", "reviewer": "Reviewer_scjr", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "[Strength]\n\nThis paper tackles an important question of how to incorporate visual information in learning policies for quadrupedal locomotion, where most existing learning-based control of quadruped robots in the published works only considered proprioceptive information, and the robots are essentially \"blind.\"\n\nThe use of visual information can allow the robots to be less conservative and plan their actions for a longer time horizon, as has been evident from the authors' comparison with a state-only baseline that only considers proprioceptive inputs.\n\nThis paper has extensive experiments and in-depth analysis in simulation, which provides a good reference for the readers to understand the benefits and limitations of different design choices.\n\nThe real-world demo from sim-to-real transfer also provides concrete empirical evidence on the practical use of the proposed method.\n\n\n[Weakness]\n\nWhile I like the direction this paper is going, I'm not entirely convinced that the real-world experiments in the paper fully demonstrate the necessity of visual information. For example, in [1, 2], the authors have shown working demos on terrains seemingly much more challenging than this paper. [1, 2] also showed examples of stair climbing, a task where vision is supposed to be extremely helpful: a blind robot may have to make a few failed trials before it knows the height of a stair. You could also imagine the benefit of vision in cases that require more precise footstep planning (e.g., https://youtu.be/k7s1sr4JdlI?t=176). The paper will be much stronger by including some more concrete comparisons with the current state-of-the-art learning approaches on what can be made possible via vision while previous blinds robots struggle.\n\nWhile I agree that vision is important for robots to make long-term plans and autonomously traverse around obstacles, I'm not sure whether this paper's approach is better than more classic robotic pipelines. For example, instead of treating the depth image as a 2D grid and processing it using CNN, one could use the depth camera to build a 3D map of the surrounding environment and blend in the explicit notion of what's traversable and what's not. You can then plan the trajectory based on the perception results. This seems to be how Boston Dynamics' Spot uses the visual information (https://www.youtube.com/watch?v=Ve9kWX_KXus) and has shown great generalization ability in real-world scenarios — showing examples of how this paper's way of using visual information is better than classic pipelines may be essential to claim improvements.\n\nContinuing my previous point, it would be better if the authors could include more discussion on the current state of quadrupedal locomotion both in academia and industry, where Boston Dynamics' Spot is seemingly better in terms of generalization and robustness than any of the reinforcement learning-based approaches. The authors may also shed light on under which scenarios we should choose RL-trained robots over Spot.\n\nHow does the method work in the real world if there are moving obstacles, e.g., humans and other animals?\n\nHow well does the method work compared with the built-in controller of the robot?\n\nThe paper may need a few passes of proofreading where the current manuscript includes a lot of typos, just to name a few:\nSection 1, contribution bulletin points: We the propose LocoTransformer, ...\nSection 3: a MDP --> an MDP\nSection 6: The visual inputs also inputs the locomotion ...\n\n\n[1] Joonho Lee, Jemin Hwangbo, Lorenz Wellhausen, Vladlen Koltun, Marco Hutter, \"Learning Quadrupedal Locomotion over Challenging Terrain\"\n[2] Ashish Kumar, Zipeng Fu, Deepak Pathak, Jitendra Malik, \"RMA: Rapid Motor Adaptation for Legged Robots\"\n\n=====================\n\n[Post Rebuttal]\n\nI thank the authors for the detailed feedback and additional experiments, which addressed most of my concerns. Great job! I have also read the reviews from other reviewers and decided to raise my score to 8: accept, good paper.", "summary_of_the_paper": "This paper proposes to incorporate the proprioceptive and visual information together for quadrupedal locomotion. The authors introduce a new model architecture named LocoTransformer that consists of separate modality encoders for proprioceptive and visual inputs, the output of which is fed through a shared Transformer encoder to predict actions and values.\n\nThrough experiments, the authors demonstrate that the robot, with the help of both the proprioceptive and visual inputs, can walk through different sizes of obstacles and even moving obstacles. They have also transferred the learned policy from simulation to a real robot by running it indoors and in the wild with unseen obstacles and terrain.", "summary_of_the_review": "I like the direction this paper is going: combining visual and proprioceptive information to train RL agents for quadrupedal locomotion. I also love that the authors include real-world demos of the learned policy on a physical robot. However, my main concern is that, although there are extensive evaluations in the simulation, the current set of real-world examples may not be sufficient to show the benefit of visual inputs. Examples like climbing stairs or scenarios that require more precise footstep planning would make the paper much stronger.\n\nI'm generally excited about the progress in this direction, thus I'm currently leaning towards the acceptance side, but I hope the authors can address the issues mentioned above.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper addresses vision-based and proprioception-based policies for learning quadrupedal locomotion, using simulation and real-robot experiments with the A1 robot dog. The reviewers agree on the significance of the algorithmic, simulation, and real-world results. Given that there are also real-robot evaluations, and an interesting sim-to-real transfer, the paper appears to be an important acceptance to ICLR.
Decision: Accept (Spotlight)