# Reinforcement Learning of Bipedal Walking Using a Simple Reference Motion
## 6 Conclusions  
In this study, we proposed a reinforcement learning method to acquire a bipedal walking controller using only a simple reference motion. The reference motion was a simple motion consisting of three sine waves, and its parameters could be automatically set using Bayesian optimization. To facilitate reinforcement learning, we introduced two means: the use of inverse kinematics (IK) and the use of the reference motion as a bias. Numerical experiments showed that, with the proposed method, a bipedal walking controller can be learned using a smaller number of samples, compared with a method using reinforcement learning only and with methods using imitation rewards. Furthermore, experiments using a real robot, KHR-3 HV (ver. 2), showed that, using a sim-to-real technique using domain randomization, a real robot could walk with the controller acquired in a simulator using the proposed method.  
Since the present study was limited to controlling a small humanoid robot, an important future work would be to apply the proposed method to other robots, such as human-sized robots and robots of different shapes. In the present study, the previous method [16], which was successfully trained on a robot with bird-like legs, was not successful on our humanoid robot. Thus, it would be important to investigate the range of applicability of our proposed method.  
Another line of future work is to apply the proposed method to the learning of more complex gaits. The present study was limited to a case in which the robot was encouraged to walk as quickly as possible. Learning various walking patterns, such as stopping after a certain distance or walking slowly, remains an issue for future work. Using additional sensors may also be useful to achieve a variety of walking behaviors. For example, combinations with vision sensors utilizing advanced processing methods (e.g., [43]) may be used to enable walking on stairs or in places with various obstacles.  
Another future issue is to provide some guarantees for walking performance. There are recent studies on tracking control that can theoretically guarantee prescribed performance (e.g., [14; 15]). It would be desirable to provide such guarantees in the reinforcement learning of bipedal walking.  
Additionally, although the learning of bipedal walking was possible based on a relatively small number of samples when using our proposed method, the further acceleration of learning--for example, by parallelizing the learning with multiple simulations [44]--could be a future challenge.  
The following supporting information can be downloaded at [[https://www.mdpi.com/article/10.3390/app14051803/s1](https://www.mdpi.com/article/10.3390/app14051803/s1)]([https://www.mdpi.com/article/10.3390/app14051803/s1](https://www.mdpi.com/article/10.3390/app14051803/s1)). Video S1: Motions learned.  
Conceptualization, N.I. and H.I.; methodology, N.I.; software, N.I.; validation, N.I. and H.I.; formal analysis, N.I.; investigation, N.I.; resources, H.I. and H.F.; data curation, N.I. and H.I.; writing--original draft preparation, N.I.; writing--review and editing, H.I., H.F. and H.W.; visualization, N.I.; supervision, H.I., H.F. and H.W.; project administration, H.I. and H.W.; funding acquisition, H.I. All authors have read and agreed to the published version of the manuscript.  
This work was supported in part by JSPS KAKENHI Grant Number JP 22K1295.  
Figure 8: Gait of the real robot. The motion is shown in the video in the Supplementary Materials.