Learning Vision-Guided Quadrupedal Locomotion End-to-End with Cross-Modal Transformers

1 INTRODUCTION . Legged locomotion is one of the core problems in robotics research . It expands the reach of robots and enables them to solve a wide range of tasks , from daily life delivery to planetary exploration in challenging , uneven terrain ( Delcomyn & Nelson , 2000 ; Arena et al. , 2006 ) . Recently , besides the success of Deep Reinforcement Learning ( RL ) in navigation ( Mirowski et al. , 2017 ; Gupta et al. , 2019 ; Yang et al. , 2019 ; Kahn et al. , 2021 ) and robotic manipulation ( Levine et al. , 2018 ; 2016 ; Tian et al. , 2019 ; Jain et al. , 2019b ) , we have also witnessed the tremendous improvement of locomotion skills for quadruped robots , allowing them to walk on uneven terrain ( Xie et al. , 2020 ; 2021 ) , and even generalize to real-world with mud , snow , and running water ( Lee et al. , 2020a ) . While these results are encouraging , most RL approaches focus on learning a robust controller for blind quadrupedal locomotion , using only the proprioceptive state . For example , Lee et al . ( 2020a ) utilize RL with domain randomization and large-scale training in simulation to learn a robust quadrupedal locomotion policy , which can be applied to challenging terrains . However , is domain randomization with blind agents really sufficient for general legged locomotion ? By studying eye movement during human locomotion , Matthis et al . ( 2018 ) show that humans rely heavily on eye-body coordination when walking and that the gaze changes depending on characteristics of the environment , e.g. , whether humans walk in flat or rough terrain . This finding motivates the use of visual sensory input to improve quadrupedal locomotion on uneven terrain . While handling uneven terrain is still possible without the vision , a blind agent is unable to consistently avoid large obstacles as shown in Figure 1 . To maneuver around such obstacles , the agent needs to perceive the obstacles at a distance and dynamically make adjustments to its trajectory to avoid any collision . Likewise , an agent navigating rough terrain ( mountain and forest in Figure 1 ) may also benefit from vision by anticipating changes in the terrain before contact , and visual observations can therefore play an important role in improving locomotion skills . In this paper , we propose to combine proprioceptive states and first-person-view visual inputs with a cross-modal Transformer for learning locomotion RL policies . Our key insight is that proprioceptive states ( i.e . robot pose , Inertial Measurement Unit ( IMU ) readings , and local joint rotations ) provide a precise measurement of the current robot status for immediate reaction , while visual inputs from a depth camera help the agent plan to maneuver uneven terrain and large obstacles in the path . We fuse the proprioceptive state and depth image inputs using Transformers ( Vaswani et al. , 2017 ; Tsai et al. , 2019 ) for RL , which enables the model to reason with complementary information from both modalities . Additionally , Transformers also offer a mechanism for agents to attend to specific visual regions ( e.g . objects or uneven ground ) that are critical for their long-term and short-term decision making , which may in turn lead to a more generalizable and interpretable policy . Our Transformer-based model for locomotion , LocoTransformer , consists of two encoders for inputs ( an MLP for proprioceptive states , a ConvNet for depth image inputs ) and Transformer encoders for multi-modal fusion . We obtain a feature embedding from the proprioceptive states and multiple image patch embeddings from the depth images , which are used jointly as token inputs for the Transformer encoders . Feature embeddings for both modalities are then updated with information propagation among all the tokens using self-attention . We combine both features for action prediction . The model is trained end-to-end without hierarchical RL ( Peng et al. , 2017 ; Jiang et al. , 2019 ; Jain et al. , 2019a ) nor pre-defined controllers ( Da et al. , 2020 ; Escontrela et al. , 2020 ) . We experiment on both simulated and real environments as shown in Figure 1 . Our tasks in simulation include maneuvering around obstacles of different sizes , dynamically moving obstacles , and rough mountainous terrain . With simulation-to-real ( sim2real ) transfer , we deploy the policies to the robot on indoor hallways with box obstacles and outdoor forests with trees and uneven terrain . We show that learning policies with both proprioceptive states and vision significantly improve locomotion control , and the policies further benefit from adopting cross-modal Transformer . We also show that LocoTransformer generalizes much better to unseen environments , especially for sim2real transfer . We highlight our main contributions as follows : • Going beyond blind robots , we introduce visual information into end-to-end RL policies for quadrupedal locomotion to traverse complex terrain with different kinds of obstacles . • We propose LocoTransformer , which fuses proprioceptive states and visual inputs for better multi-modal reasoning in sequential decision making . • To the best of our knowledge , this is the first work which deploys vision-based RL policy on running real quadrupedal robot avoiding obstacles and trees in the wild . 2 RELATED WORK . Learning Legged Locomotion . Developing legged locomotion controllers has been a long-standing problem in robotics ( Miura & Shimoyama , 1984 ; Raibert , 1984 ; Torkos & van de Panne , 1998 ; Geyer et al. , 2003 ; Yin et al. , 2007 ; Bledt et al. , 2018 ) . While encouraging results have been achieved using Model Predictive Control ( MPC ) and trajectory optimization ( Gehring et al. , 2013 ; Carlo et al. , 2018 ; Di Carlo et al. , 2018 ; Carius et al. , 2019 ; Ding et al. , 2019 ; Grandia et al. , 2019 ; Bledt & Kim , 2020 ; Sun et al. , 2021 ) , these methods require in-depth knowledge of the environment and substantial manual parameter tuning , which makes these methods challenging to apply to complex environments . Alternatively , model-free RL can learn general policies on challenging terrain ( Kohl & Stone , 2004 ; Zhang et al. , 2018 ; Luo et al. , 2020 ; Peng et al. , 2018 ; Tan et al. , 2018 ; Hwangbo et al. , 2019 ; Lee et al. , 2020a ; Iscen et al. , 2018 ; Jain et al. , 2019a ; Xie et al. , 2021 ; Kumar et al. , 2021 ) . Xie et al . ( 2020 ) use dynamics randomization to generalize RL locomotion policy in different environments , and Peng et al . ( 2020 ) use animal videos to provide demonstrations for imitation learning . However , most approaches currently rely only on proprioceptive states without other visual inputs . In this work , we propose to incorporate both visual and proprioceptive inputs using a Transformer for RL policy , which allows the quadruped robot to simultaneously move and plan its trajectory . Vision-based Reinforcement Learning . To generalize RL to real-world applications beyond state inputs , a lot of effort has been made in RL with visual inputs ( Sax et al. , 2018 ; Jaderberg et al. , 2017 ; Levine et al. , 2016 ; 2018 ; Pathak et al. , 2017 ; Jain et al. , 2019b ; Mnih et al. , 2015a ; Lin et al. , 2019 ; Yarats et al. , 2019 ; Laskin et al. , 2020 ; Stooke et al. , 2020 ; Schwarzer et al. , 2020 ) . For example , Srinivas et al . ( 2020 ) propose to apply contrastive self-supervised representation learning ( He et al. , 2020 ) with the RL objective in vision-based RL . Hansen & Wang ( 2021 ) further extend the joint representation learning and RL for better generalization to out-of-distribution environments . Researchers have also looked into combining multi-modalities with RL for manipulation tasks ( Lee et al. , 2020b ; Calandra et al. , 2018 ) and locomotion control ( Heess et al. , 2017 ; Merel et al. , 2020 ) . Escontrela et al . ( 2020 ) propose to combine proprioceptive states and LiDAR inputs for learning quadrupedal locomotion with RL using MLPs . Jain et al . ( 2020 ) propose to use Hierarchical RL ( HRL ) for locomotion , which learns high-level policies under visual guidance and low-level motor control policies with IMU inputs . Different from previous work , we provide a simple approach to combine proprioceptive states and visual inputs with a Transformer model in an end-to-end manner without HRL . Our LocoTransformer not only performs better in challenging environments but also achieves better generalization results in unseen environments and with the real robot . Transformers and Multi-modal Learning . The Transformer model has been widely applied in the fields of language processing ( Vaswani et al. , 2017 ; Devlin et al. , 2018 ; Brown et al. , 2020 ) and visual recognition and synthesis ( Wang et al. , 2018 ; Parmar et al. , 2018 ; Child et al. , 2019 ; Dosovitskiy et al. , 2020 ; Carion et al. , 2020 ; Chen et al. , 2020a ) . Besides achieving impressive performance in a variety of language and vision tasks , the Transformer also provides an effective mechanism for multi-modal reasoning by taking different modality inputs as tokens for self-attention ( Su et al. , 2019 ; Tan & Bansal , 2019 ; Li et al. , 2019 ; Sun et al. , 2019 ; Chen et al. , 2020b ; Li et al. , 2020 ; Prakash et al. , 2021 ; Huang et al. , 2021 ; Hu & Singh , 2021 ; Akbari et al. , 2021 ; Hendricks et al. , 2021 ) . For example , Sun et al . ( 2019 ) propose to use a Transformer to jointly model video frames and their corresponding captions from instructional videos for representation learning . Going beyond language and vision , we propose to utilize cross-modal Transformers to fuse proprioceptive states and visual inputs . To our knowledge , this is the first work using cross-modal Transformers for locomotion . 3 REINFORCEMENT LEARNING BACKGROUND . We model the interaction between the robot and the environment as an MDP ( Bellman , 1957 ) ( S , A , P , R , H , γ ) , where s ∈ S are states , a ∈ A are actions , P ( s′|s , a ) is transition function , R is reward function , H is finite episode horizon , and γ is discount factor . The Agent learn a policy πθ parameterized by θ to output actions distribution conditioned on current state . The goal of agent is to learn θ that maximizes the discounted episode return : R = Eτ∼pθ ( τ ) [ ∑H t=0 γ trt ] , where rt ∼ R ( st , at ) is the reward for time step t , τ ∼ pθ ( τ ) is the trajectory . 4 METHOD . We propose to incorporate both proprioceptive and visual information for locomotion tasks using a novel Transformer model , LocoTransformer . Figure 2 provides an overview of our architecture . Our model consists of the following two components : ( i ) Separate modality encoders for proprioceptive and visual inputs that project both modalities into a latent feature space ; ( ii ) A shared Transformer encoder that performs cross-modality attention over proprioceptive features and visual features , as well as spatial attention over visual tokens to predict actions and predict values .

Summary: This paper proposes an approach to legged locomotion which leverages a Transformer-based model and is trained via end-to-end reinforcement learning. It provides extensive experimental evaluation of the approach in terms of performance and safety metrics, both in simulation and using real-world experiments. The code is expected to be open-sourced.