# Reinforcement Learning of Bipedal Walking Using a Simple Reference Motion
## 3 Proposed Method
### Reference Motion  
In the proposed method, a humanoid robot acquires a bipedal walking controller via reinforcement learning using a simple reference motion. To keep the reference motion simple, we consider only three variables: the heights of the left and right sides and the lateral displacement of the soles relative to the torso. As the reference motion, we use the motion of walking in place, rather than walking forward. In the reference motion, the left and right sides are alternately raised; the torso is displaced in the right direction when the left sole is raised, and vice versa. It is not necessary for the robot to actually walk in place without falling down when controlled by the reference motion alone. The reference motion is defined by three equations as  
\[y_{t}^{\text{ref}} = \text{sign}(\sin(2\pi\frac{t}{T}))\min(|w\sin(2\pi\frac{t}{T})|+ \Delta w,w), \tag{1}\] \[z_{t}^{\text{ref, left}} = \max(0,h\sin(2\pi\frac{t}{T})-\Delta h),\] (2) \[z_{t}^{\text{ref, right}} = \max(0,h\sin(2\pi\frac{t}{T}+\pi)-\Delta h), \tag{3}\]  
where \(t\,(=0,1,2,\ldots)\) is the time step after an episode has started and \(T\) is a given period of the reference motion. An episode starts with the robot in the initial posture shown in Figure 1 and ends either when the robot falls down or when a certain number of steps has elapsed (in this study, we set it to 500). Figure 3 shows example waveforms of these equations. The variable \(y_{t}^{\text{ref}}\) in Equation (1) is the lateral displacement of the soles of both feet relative to the torso. Equation (1) represents the movement of the torso to the left and right in accordance with the movement of the soles, as depicted in Figure 3. The variable \(z_{t}^{\text{ref, left}}\) in Equation (2) is the height at which the sole of the left foot is raised. Equation (2) represents the motion of raising the left foot during the first half of each walking cycle (Phase 3 in Figure 3). Similarly, \(z_{t}^{\text{ref, right}}\) in Equation (3) is the height at which the sole of the right foot is raised, and Equation (3) represents the motion of raising the right foot in the second half of each walking cycle (Phase 7 in Figure 3). Equation (1) has two parameters, \(w\) and \(\Delta w\). The parameter \(w\) determines the maximum amount of lateral displacement. The larger the fraction \(\frac{\Delta w}{w}\), the larger the proportion of time in each walking cycle during which the lateral displacement is at its maximum value. In addition, there are two parameters, \(h\) and \(\Delta h\), in Equations (2) and (3). The value of \(h-\Delta h\) determines the maximum height of the soles in the reference motion (see Phase 3 in Figure 3). The larger the fraction \(\frac{\Delta h}{h}\), the larger the proportion of time spent in the double support phase (i.e., the phase in which the two soles are touching the ground) in each walking cycle. This is because Equations (2) and (3) indicate that the heights of the soles, \(z_{t}^{\text{ref, left}}\) and \(z_{t}^{\text{ref, right}}\), are both zero when time \(t\) satisfies \(\sin(2\pi\frac{t}{T})\leq\frac{\Delta h}{h}\) and \(\sin(2\pi\frac{t}{T}+\pi)\leq\frac{\Delta h}{h}\). These four parameters, \(w\), \(\Delta w,h\), and \(\Delta h\), are automatically optimized using Bayesian optimization during learning and thus do not need to be set by humans. Details of the Bayesian optimization are explained in Section 3.6. \begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Joint** & \(a_{i}\) **[mm]** & \(\alpha_{i}\) **[rad]** & \(d_{i}\) **[mm]** & \(\varphi_{i}\) **[rad]** \\ \hline
1 & 0 & \(\frac{\pi}{4}\) & 23 & \(\varphi_{1}\)* \\
2 & 41 & \(-\frac{\pi}{2}\) & 0 & \(\varphi_{2}-\frac{\pi}{2}\) \\
3 & 65 & 0 & 0 & \(\varphi_{3}\) \\
4 & 65 & 0 & 0 & \(\varphi_{4}\) \\
5 & 41 & \(\frac{\pi}{2}\) & 0 & \(\varphi_{5}\) \\
6 & 25 & 0 & 0 & \(\varphi_{6}\) \\ \hline \hline \end{tabular}  
* In the present study, \(\varphi_{1}\) is fixed at 0 [rad]. Only the angles \(\varphi_{2}\) to \(\varphi_{6}\) are controlled.  
\end{table}