# Reinforcement Learning of Bipedal Walking Using a Simple Reference Motion
## 2 Related Work  
Similar to the present study, [PERSON] et al. [16] proposed a reinforcement learning method that uses only a simple reference motion. In their study, the reference motion was represented by only two sine waves. They used a Cassie robot and showed that bipedal walking was successfully learned in a simulator. However, as described in Section 4, their method did not succeed in learning bipedal walking when applied on our humanoid robot. Thismay be due to the difference in the shape of the robots; the [PERSON] robot has bird-like legs and can walk without moving its hips to the left or right, whereas our robot is humanoid and needs to move its hips to the left or right while walking. In Section 4, we show that our proposed method is superior for our humanoid robot. Another difference is that our proposed method automatically searches for the sine wave parameters, whereas these must be provided by humans when using the method of [PERSON] et al. Additionally, we apply our method to a real robot, whereas their study was conducted with a simulator only.  
Various methods have been studied to assist the reinforcement learning of bipedal walking. Other than the reference-motion-based methods described in Section 1, the following methods have also been studied. [PERSON] et al. [17] proposed a reference-free reinforcement learning method in which the robot is helped by the application of external forces. However, it is necessary for humans to set up a well-designed learning curriculum, such as where to initially apply and then how to gradually reduce the external forces so that the robot can eventually walk without them. [PERSON] et al. [18] used a well-designed hand-crafted reward function and realized the learning of a natural bipedal walking pattern without using a reference motion. The reward function encourages the robot to realize, e.g., desirable foot-ground reaction forces depending on the phase of a gait cycle. However, it is necessary for humans to create such a reward function.  
The approach of using IK with reinforcement learning has been adopted in several existing studies; for example, [PERSON] et al. [8] proposed a gait controller that combined IK and reinforcement learning and applied it to DARwIn-OP, a small humanoid robot. However, in their study, some patterns of toe movements with parameters such as the stride length had to be prepared in advance, and the reinforcement learning method only optimized these parameters within a limited range. IK was used to convert the movement patterns into joint angles. In contrast, our proposed method optimizes the angle of every joint. [PERSON] et al. [19] proposed a method that involved learning to balance a NAO robot, using an approach similar to [8], where low-dimensional actions optimized by reinforcement learning are converted into joint angles using IK.  
In the present study, the KHR-3 HV robot (ver. 2) is used as the target robot. This robot is an inexpensive (less than USD 100), small humanoid robot and has been used for studies on various topics, including gait generation using an evolutionary algorithm [20], balancing on inclined surfaces [21], energy efficiency analysis for bipedal walking [22], and multimodal locomotion [23]. However, to the best of our knowledge, our study is the first in which a bipedal gait controller for a real robot is acquired using reinforcement learning. Compared to the Cassie robot, which has been extensively studied for bipedal walking by reinforcement learning [12; 16; 24; 25], enabling the KHR-3 HV robot to learn to walk might be more difficult. One reason is that, as mentioned above, Cassie has bird-like legs and the KHR-3 HV is humanoid. Another reason is that Cassie's servo motors are concentrated at the top of the legs, making the moments of the legs small, whereas the KHR-3 HV's servo motors are located inside the legs, making the leg moments large. We emphasize, however, that discussions of the superiority or inferiority of robots are beyond the scope of this paper.