# Reinforcement Learning of Bipedal Walking Using a Simple Reference Motion
## 3 Proposed Method
### Action and Target Joint Angle  
At each time \(t\), we obtain output \(a_{t}^{\text{RL}}\) of the reinforcement learning and add it to the reference bias \(a_{t}^{\text{IK}}\) calculated from the reference motion as in the previous section. We let the result be the target joint angles \(u_{t}^{\text{target}}\). Specifically,  
\[u_{t}^{\text{target}}=a_{t}^{\text{IK}}+a_{t}^{\text{RL}} \tag{5}\]  
(Figure 1), where \(a_{t}^{\text{RL}}\) is also an \(N\)-dimensional vector like \(a_{t}^{\text{IK}}\). We assume that each joint of the robot has a servo motor that changes its angle by PID control so that the given target angle \(u_{t}^{\text{target}}\) is approached (Sections 4 and 5).  
Determining \(u_{t}^{\text{target}}\) in this way means that all the angles of the target servo motors can be directly optimized by reinforcement learning. The role of \(a_{t}^{\text{IK}}\) is simply to provide a bias for the reinforcement learning output. This is in contrast to the previous studies [8; 19] mentioned in Section 2, in which IK was used to calculate the target angles for all servo motors from low-dimensional parameters or actions optimized by reinforcement learning.  
We obtain the output of the reinforcement learning, \(a_{t}^{\text{RL}}\), using the following procedure. First, when the current state \(s_{t}\) (see Section 3.3 for details) is input to the neural network, the action \(a_{t}\) is determined according to a probabilistic policy \(\pi_{\theta}(a_{t}|s_{t})\) represented by the neural network. Here, \(\theta\) denotes the parameters of the neural network. The action \(a_{t}\) is an \(N\)-dimensional vector, and each element of \(a_{t}\) takes a value between \(-1\) and \(1\).  
Figure 3: The reference motion. The graph at the top shows the waveforms of Equations (1)–(3): \(y_{t}^{\text{ref}}\) (blue) is the lateral displacement of the soles of both feet, \(z_{t}^{\text{ref, left}}\) (red) is the sole height of the left foot, and \(z_{t}^{\text{ref, right}}\) (green) is the sole height of the right foot. The lower part of the figure shows the posture of the robot’s lower body at each time point. Our reference motion is a motion within the \(y\)–\(z\) plane. Note that Phases 1 to 9 in the figure are intended for descriptive purposes only and are not used in the proposed method.  
Next, \(a_{t}\) is passed through a low-pass filter (LPF) to improve the smoothness [27]. Specifically, \(a_{t}^{\prime}\) is calculated from \(a_{t}\) as  
\[a_{t}^{\prime}=b\,a_{t}+\left(1-b\right)a_{t-1}^{\prime}, \tag{6}\]  
where \(b\left(0<b\leq 1\right)\) is a smoothing factor, and \(a_{t-1}^{\prime}\) is the smoothed value one step ago. We set \(a_{-1}^{\prime}\), which is a variable that is necessary to calculate Equation (6) when \(t=0\), to 0. We set \(b=0.2\) in all the experiments in this study. The smoothed action \(a_{t}^{\prime}\) is also an \(N\)-dimensional vector, and each element of it takes a value between \(-1\) and 1.  
Finally, \(a_{t}^{\prime}\) is multiplied by a constant \(k\) to obtain the output of the reinforcement learning, \(a_{t}^{\text{RL}}\). Specifically,  
\[a_{t}^{\text{RL}}=ka_{t}^{\prime},\]  
where \(k\) is a parameter that determines the range of possible values of \(a_{t}^{\text{RL}}\), i.e., the range of search by the reinforcement learning (Figure 1). In our experiments, we set \(k=0.5\,\omega_{\text{max}}\,dt\), where \(\omega_{\text{max}}\) is the maximum angular velocity of the servo motors and \(dt\) is the time interval of one step.