Learning Vision-Guided Quadrupedal Locomotion End-to-End with Cross-Modal Transformers

1 INTRODUCTION . Legged locomotion is one of the core problems in robotics research . It expands the reach of robots and enables them to solve a wide range of tasks , from daily life delivery to planetary exploration in challenging , uneven terrain ( Delcomyn & Nelson , 2000 ; Arena et al. , 2006 ) . Recently , besides the success of Deep Reinforcement Learning ( RL ) in navigation ( Mirowski et al. , 2017 ; Gupta et al. , 2019 ; Yang et al. , 2019 ; Kahn et al. , 2021 ) and robotic manipulation ( Levine et al. , 2018 ; 2016 ; Tian et al. , 2019 ; Jain et al. , 2019b ) , we have also witnessed the tremendous improvement of locomotion skills for quadruped robots , allowing them to walk on uneven terrain ( Xie et al. , 2020 ; 2021 ) , and even generalize to real-world with mud , snow , and running water ( Lee et al. , 2020a ) . While these results are encouraging , most RL approaches focus on learning a robust controller for blind quadrupedal locomotion , using only the proprioceptive state . For example , Lee et al . ( 2020a ) utilize RL with domain randomization and large-scale training in simulation to learn a robust quadrupedal locomotion policy , which can be applied to challenging terrains . However , is domain randomization with blind agents really sufficient for general legged locomotion ? By studying eye movement during human locomotion , Matthis et al . ( 2018 ) show that humans rely heavily on eye-body coordination when walking and that the gaze changes depending on characteristics of the environment , e.g. , whether humans walk in flat or rough terrain . This finding motivates the use of visual sensory input to improve quadrupedal locomotion on uneven terrain . While handling uneven terrain is still possible without the vision , a blind agent is unable to consistently avoid large obstacles as shown in Figure 1 . To maneuver around such obstacles , the agent needs to perceive the obstacles at a distance and dynamically make adjustments to its trajectory to avoid any collision . Likewise , an agent navigating rough terrain ( mountain and forest in Figure 1 ) may also benefit from vision by anticipating changes in the terrain before contact , and visual observations can therefore play an important role in improving locomotion skills . In this paper , we propose to combine proprioceptive states and first-person-view visual inputs with a cross-modal Transformer for learning locomotion RL policies . Our key insight is that proprioceptive states ( i.e . robot pose , Inertial Measurement Unit ( IMU ) readings , and local joint rotations ) provide a precise measurement of the current robot status for immediate reaction , while visual inputs from a depth camera help the agent plan to maneuver uneven terrain and large obstacles in the path . We fuse the proprioceptive state and depth image inputs using Transformers ( Vaswani et al. , 2017 ; Tsai et al. , 2019 ) for RL , which enables the model to reason with complementary information from both modalities . Additionally , Transformers also offer a mechanism for agents to attend to specific visual regions ( e.g . objects or uneven ground ) that are critical for their long-term and short-term decision making , which may in turn lead to a more generalizable and interpretable policy . Our Transformer-based model for locomotion , LocoTransformer , consists of two encoders for inputs ( an MLP for proprioceptive states , a ConvNet for depth image inputs ) and Transformer encoders for multi-modal fusion . We obtain a feature embedding from the proprioceptive states and multiple image patch embeddings from the depth images , which are used jointly as token inputs for the Transformer encoders . Feature embeddings for both modalities are then updated with information propagation among all the tokens using self-attention . We combine both features for action prediction . The model is trained end-to-end without hierarchical RL ( Peng et al. , 2017 ; Jiang et al. , 2019 ; Jain et al. , 2019a ) nor pre-defined controllers ( Da et al. , 2020 ; Escontrela et al. , 2020 ) . We experiment on both simulated and real environments as shown in Figure 1 . Our tasks in simulation include maneuvering around obstacles of different sizes , dynamically moving obstacles , and rough mountainous terrain . With simulation-to-real ( sim2real ) transfer , we deploy the policies to the robot on indoor hallways with box obstacles and outdoor forests with trees and uneven terrain . We show that learning policies with both proprioceptive states and vision significantly improve locomotion control , and the policies further benefit from adopting cross-modal Transformer . We also show that LocoTransformer generalizes much better to unseen environments , especially for sim2real transfer . We highlight our main contributions as follows : • Going beyond blind robots , we introduce visual information into end-to-end RL policies for quadrupedal locomotion to traverse complex terrain with different kinds of obstacles . • We propose LocoTransformer , which fuses proprioceptive states and visual inputs for better multi-modal reasoning in sequential decision making . • To the best of our knowledge , this is the first work which deploys vision-based RL policy on running real quadrupedal robot avoiding obstacles and trees in the wild . 2 RELATED WORK . Learning Legged Locomotion . Developing legged locomotion controllers has been a long-standing problem in robotics ( Miura & Shimoyama , 1984 ; Raibert , 1984 ; Torkos & van de Panne , 1998 ; Geyer et al. , 2003 ; Yin et al. , 2007 ; Bledt et al. , 2018 ) . While encouraging results have been achieved using Model Predictive Control ( MPC ) and trajectory optimization ( Gehring et al. , 2013 ; Carlo et al. , 2018 ; Di Carlo et al. , 2018 ; Carius et al. , 2019 ; Ding et al. , 2019 ; Grandia et al. , 2019 ; Bledt & Kim , 2020 ; Sun et al. , 2021 ) , these methods require in-depth knowledge of the environment and substantial manual parameter tuning , which makes these methods challenging to apply to complex environments . Alternatively , model-free RL can learn general policies on challenging terrain ( Kohl & Stone , 2004 ; Zhang et al. , 2018 ; Luo et al. , 2020 ; Peng et al. , 2018 ; Tan et al. , 2018 ; Hwangbo et al. , 2019 ; Lee et al. , 2020a ; Iscen et al. , 2018 ; Jain et al. , 2019a ; Xie et al. , 2021 ; Kumar et al. , 2021 ) . Xie et al . ( 2020 ) use dynamics randomization to generalize RL locomotion policy in different environments , and Peng et al . ( 2020 ) use animal videos to provide demonstrations for imitation learning . However , most approaches currently rely only on proprioceptive states without other visual inputs . In this work , we propose to incorporate both visual and proprioceptive inputs using a Transformer for RL policy , which allows the quadruped robot to simultaneously move and plan its trajectory . Vision-based Reinforcement Learning . To generalize RL to real-world applications beyond state inputs , a lot of effort has been made in RL with visual inputs ( Sax et al. , 2018 ; Jaderberg et al. , 2017 ; Levine et al. , 2016 ; 2018 ; Pathak et al. , 2017 ; Jain et al. , 2019b ; Mnih et al. , 2015a ; Lin et al. , 2019 ; Yarats et al. , 2019 ; Laskin et al. , 2020 ; Stooke et al. , 2020 ; Schwarzer et al. , 2020 ) . For example , Srinivas et al . ( 2020 ) propose to apply contrastive self-supervised representation learning ( He et al. , 2020 ) with the RL objective in vision-based RL . Hansen & Wang ( 2021 ) further extend the joint representation learning and RL for better generalization to out-of-distribution environments . Researchers have also looked into combining multi-modalities with RL for manipulation tasks ( Lee et al. , 2020b ; Calandra et al. , 2018 ) and locomotion control ( Heess et al. , 2017 ; Merel et al. , 2020 ) . Escontrela et al . ( 2020 ) propose to combine proprioceptive states and LiDAR inputs for learning quadrupedal locomotion with RL using MLPs . Jain et al . ( 2020 ) propose to use Hierarchical RL ( HRL ) for locomotion , which learns high-level policies under visual guidance and low-level motor control policies with IMU inputs . Different from previous work , we provide a simple approach to combine proprioceptive states and visual inputs with a Transformer model in an end-to-end manner without HRL . Our LocoTransformer not only performs better in challenging environments but also achieves better generalization results in unseen environments and with the real robot . Transformers and Multi-modal Learning . The Transformer model has been widely applied in the fields of language processing ( Vaswani et al. , 2017 ; Devlin et al. , 2018 ; Brown et al. , 2020 ) and visual recognition and synthesis ( Wang et al. , 2018 ; Parmar et al. , 2018 ; Child et al. , 2019 ; Dosovitskiy et al. , 2020 ; Carion et al. , 2020 ; Chen et al. , 2020a ) . Besides achieving impressive performance in a variety of language and vision tasks , the Transformer also provides an effective mechanism for multi-modal reasoning by taking different modality inputs as tokens for self-attention ( Su et al. , 2019 ; Tan & Bansal , 2019 ; Li et al. , 2019 ; Sun et al. , 2019 ; Chen et al. , 2020b ; Li et al. , 2020 ; Prakash et al. , 2021 ; Huang et al. , 2021 ; Hu & Singh , 2021 ; Akbari et al. , 2021 ; Hendricks et al. , 2021 ) . For example , Sun et al . ( 2019 ) propose to use a Transformer to jointly model video frames and their corresponding captions from instructional videos for representation learning . Going beyond language and vision , we propose to utilize cross-modal Transformers to fuse proprioceptive states and visual inputs . To our knowledge , this is the first work using cross-modal Transformers for locomotion . 3 REINFORCEMENT LEARNING BACKGROUND . We model the interaction between the robot and the environment as an MDP ( Bellman , 1957 ) ( S , A , P , R , H , γ ) , where s ∈ S are states , a ∈ A are actions , P ( s′|s , a ) is transition function , R is reward function , H is finite episode horizon , and γ is discount factor . The Agent learn a policy πθ parameterized by θ to output actions distribution conditioned on current state . The goal of agent is to learn θ that maximizes the discounted episode return : R = Eτ∼pθ ( τ ) [ ∑H t=0 γ trt ] , where rt ∼ R ( st , at ) is the reward for time step t , τ ∼ pθ ( τ ) is the trajectory . 4 METHOD . We propose to incorporate both proprioceptive and visual information for locomotion tasks using a novel Transformer model , LocoTransformer . Figure 2 provides an overview of our architecture . Our model consists of the following two components : ( i ) Separate modality encoders for proprioceptive and visual inputs that project both modalities into a latent feature space ; ( ii ) A shared Transformer encoder that performs cross-modality attention over proprioceptive features and visual features , as well as spatial attention over visual tokens to predict actions and predict values .

Summary: This work proposes a novel architecture for quadrupedal locomotion that fuses proprioceptive and visual information with a transformer-based model to enable an agent to proactively maneuver environments with obstacles and uneven terrain by anticipating changes in the environment many steps ahead. The method is extensively evaluated in simulation and on a sim to real transfer tasks. The method is shown to both achieve higher reward  but also better capacity to generalise in the context of sim to real. Overall, the paper is well written and the provided evaluation is conducted fairly and well.