# Reinforcement Learning of Bipedal Walking Using a Simple Reference Motion
## 4 Numerical Experiments
### Experimental Results  
Figure 4 shows the learning curves of the proposed method (Section 3) and the RL-only method (Section 4.2.1). In the figure, the result of the proposed method without Bayesian optimization of the reference motion parameters is also plotted. In this case, we set the reference motion parameters to \(w=20\) [mm], \(\Delta w=5\) [mm], \(h=25\) [mm], and \(\Delta h=5\) [mm]. These parameter values were prepared for later use in the experiments of Imitation Learning 1 and 2 shown below. They are the parameter values at which the robot walks in place without falling down when controlled solely by the reference motion, and they were prepared through trial-and-error experiments performed by humans. The proposed method, both with and without Bayesian optimization, resulted in greater rewards than the RL-only method. Figure 5 shows the motions learned by \(5\times 10^{6}\) steps using the proposed method (with Bayesian optimization) and using the RL-only method. With the proposed method, the robot was able to walk forward, in contrast to the RL-only method. These results show that the proposed method enabled faster learning (i.e., learning based on a smaller number of samples) than the RL-only method. Figure 6 shows the results of the two methods that use imitation rewards, i.e., Imitation Learning 1 in Section 4.2.2 and Imitation Learning 2 in Section 4.2.3. The reference motion parameters were fixed at \(w=20\) [mm], \(\Delta w=5\) [mm], \(h=25\) [mm], and \(\Delta h=5\) [mm], which are the values that allowed learning bipedal walking using the proposed method, as shown in the video in the Supplementary Materials. Figure 6 shows the results in the following four conditions:  
* \(w^{\text{I}}=0.2\), \(w^{\text{T}}=0.8\);
* \(w^{\text{I}}=0.4\), \(w^{\text{T}}=0.6\);  
Figure 4: Learning curves of the proposed method and RL-only method. The vertical axis is the reward obtained in each episode, averaged over the most recent 100 episodes. For the proposed method, the results with and without Bayesian optimization of the reference motion parameters are plotted. For each method, the same experiment was repeated three times using different random seeds, and the means and standard deviations are plotted. \(w^{\rm I}=0.6\), \(w^{\rm T}=0.4\);
* \(w^{\rm I}=0.8\), \(w^{\rm T}=0.2\). Note that the case of \(w^{\rm I}=0.0\) and \(w^{\rm T}=1.0\) was excluded from the experimental conditions above because it was the same as the RL-only condition. The case of \(w^{\rm I}=1.0\) and \(w^{\rm T}=0.0\) was also excluded because, in this case, the learning was based only on the imitation reward term and the optimal policy was to simply imitate the reference motion that allowed the robot to walk in place. Figure 6a shows the results of Imitation Learning 1 (Section 4.2.2). It can be seen that the task reward, shown on the right side of the figure, reached a maximum when \(w^{\rm I}=0.2\) and \(w^{\rm T}=0.8\). However, the task reward was lower than that of the proposed method shown in Figure 4. The experiment was conducted up to \(1\times 10^{7}\) steps, twice as many as in Figure 4, but the robot could not walk, as shown in the video in the Supplementary Materials. Figure 6b shows the results of Imitation Learning 2 (Section 4.2.3). It can be seen that, as with Imitation Learning 1, the task reward, shown on the right side, reached a maximum when \(w^{\rm I}=0.2\) and \(w^{\rm T}=0.8\). Although the task reward was higher than that shown in Figure 6a, it was still lower than that of the proposed method shown in Figure 4. The robot could not walk after \(1\times 10^{7}\) steps, as shown in the video in the Supplementary Materials. These results indicate that the learning of bipedal walking occurred faster when using the proposed method rather than the imitation-reward-based methods. There are many parameters other than \(w^{\rm I}\) and \(w^{\rm T}\) in Imitation Learning 1 and 2, and it was not practical to examine all cases, but, at least in our experiments, bipedal walking was not learned when using these methods.