# Reinforcement Learning of Bipedal Walking Using a Simple Reference Motion
## 1 Introduction  
Humanoid robots are capable of human-like movements and can therefore be useful in assisting and/or replacing humans in environments designed for humans. However, enabling humanoid robots to walk on two legs is a challenging task, due to the complex dynamics. There has been a long history of research on the design of bipedal gait controllers using mathematical models and dynamics, such as the inverted pendulum [1, 2] and zero-moment point (ZMP) [3, 4], but these approaches require sophisticated controller designs involving accurate mathematical models of the robot and its environment. Recently, methods using reinforcement learning have been extensively studied (e.g., [5, 6, 7, 8, 9]) as a promising alternative approach. The reinforcement-learning-based methods are useful because they do not depend on a sophisticated controller designed by humans. The controller can be automatically acquired through trial and error. However, learning bipedal walking from scratch by reinforcement learning (e.g., [10, 11]) requires much time for the trial-and-error learning process and often results in unnatural gait patterns. In many studies, this problem has been addressed by providing the robot with a reference motion (e.g., [5, 7, 12, 13]), which is an example motion that helps the robot to quickly learn a natural gait pattern. Although the use of a reference motion enables the quick learning of a natural gait pattern, preparing the reference motion can be laborious or costly. For example, in some studies [7; 12], expert controllers were used to generate a reference motion, but the preparation of an expert controller requires significant human effort. In other studies [5; 13], reference motions were created by capturing human motion, but this approach requires costly motion capture equipment. Therefore, it is highly desirable in the reinforcement learning of bipedal walking to use a simple reference motion that can easily be prepared. Since the purpose of using a reference motion is to help a robot to learn to walk, the reference motion does not need to be so elaborate that the robot can walk using the reference motion alone (if such an elaborate reference motion is available, there are other useful approaches, such as tracking control methods that guarantee prescribed performance [14; 15]). However, little research has been conducted on the use of simple reference motions. An exception is the study by [PERSON] et al. [16] on a robot with bird-like legs (see Section 2 for details), but their method is not readily applicable to a humanoid robot. To address this issue, in this paper, we propose a novel reinforcement learning method that enables a humanoid robot to learn bipedal walking using only a simple reference motion. This is the first contribution of this paper. The reference motion that we use is a simple motion consisting of three sine waves, and the parameters that determine their waveforms are automatically set using Bayesian optimization. Thus, our reference motion can be prepared with minimal human involvement. To enable quick learning while using the simple reference motion, we introduce the following two means. First, we use inverse kinematics (IK). As will be described in Section 3, our simple reference motion only specifies the robot's positions in relation to the torso. Thus, we use IK to identify all the angles of the leg joints that realize the specified sole positions. We expect that the reference motion expressed by the joint angles will provide the robot with more direct clues in learning how to walk. Second, we use the reference motion as a bias for the action determined by reinforcement learning, rather than as an imitation target. In many previous studies [5; 7; 12; 13], the robot is given a reward function that encourages it to imitate a given reference motion. Although this approach is suitable when the goal of learning is to imitate the reference motion, such a reward function disturbs learning when the goal is not imitation itself.