title: Learning multi-skill locomotion on legged robots
Creators: name: Yu, Wanming
Publisher: name: The University of Edinburgh
Year: 2024
Subjects: subject: legged locomotion
subject: deep reinforcement learning
subject: quadrupedal robot
subject: robot learning

description: Robots with morphology similar to their biological counterparts provide unique opportunities to develop machines with motion capabilities comparable to those of animals. In particular, legged robots have advantages in tackling complex environments, such as rugged terrains, over other types of robots. To date, a variety of tasks have been demonstrated on legged robots through learning approaches. However, the successful learning of motor skills relies heavily on human knowledge and experiences without a systematic design approach or guideline to produce desired behaviors. This doctoral thesis investigates open research questions in two key aspects for the effective learning of versatile locomotion skills: What is the general understanding or guideline for designing learning framework for locomotion skills with a focus on state selection and learning efficiency? How to handle complex situations where multiple dynamic skills may be required? We first present a systematic saliency analysis that quantitatively evaluates the relative importance of different feedback states for motor skills acquired by deep reinforcement learning. Our approach can identify the most essential feedback states for various locomotion skills, including fall recovery, trotting, bounding, pacing and galloping. We demonstrate that a simulated quadruped robot can achieve robust performance in various test scenarios across these distinct skills using only the key states including joint positions, gravity vector, base linear and angular velocities. Through performance benchmark, we show that locomotion skills learned with key states can achieve comparable performance to those with more states. We find the task performance or learning success rate will drop significantly if key states are missing. This result provides quantitative insights into the relationship between state observations and specific types of motor skills, serving as a guideline for robot motor learning, which enables the learning of a wide range of motor skills with minimal sensing dependencies. To address the learning efficiency issue, we develop K-Access algorithm based on accessibility metrics for the automatic discovery of initial poses for effective exploration within the state space during the learning process. We formulate accessibility metrics to measure the difficulty of transitions between two arbitrary poses, and use the proposed K-Access algorithm for state-space clustering that automatically discovers the centroids of the static-pose clusters. By using the discovered centroidal static poses as the initial states, we can improve data efficiency by reducing redundant explorations, and enhance the robustness by more effective explorations from the centroids. We validate our method extensively by fall recovery and backflipping skills on a quadrupedal robot. Compared to the baselines, the learning curve of our proposed method converges much faster. We extend our research to study how to fuse multiple locomotion skills and enable automatic gait transitions through learning approaches with a particular focus on highly dynamic skills such as bounding and galloping. Different locomotion gaits can be activated according to the relative distance of the robot and the target which is discovered by evolution strategies through multiplicative composition of policies. The robot is demonstrated to be able to freely transition among fall recovery, trotting, bounding and galloping through the proposed approaches. Furthermore, this thesis addresses the sim-to-real discrepancies by proposing a state estimator network to effectively estimate robot velocity in real-world scenarios.