# Reinforcement Learning of Bipedal Walking Using a Simple Reference Motion
## 4 Numerical Experiments
### Methods for Comparison
#### 4.2.2 Imitation Learning 1  
As mentioned in Section 2, [PERSON] et al. [16] also proposed a reinforcement learning method using a simple reference motion. They successfully trained a Cassie robot in a simulator, although they did not control a real robot. For comparison, we prepared a similar method. Here, we note that it was impossible to implement exactly the same method as theirs because of differences in the robots (e.g., their method uses information regarding shin springs and pelvis, which is unavailable in our robot). Thus, we replaced their task reward with our \(r^{\text{T}}\) in Equation (7) because \(r^{\text{T}}\) in Equation (7) worked well for our robot in our proposed method.  
\begin{table}
\begin{tabular}{c c} \hline \hline
**Hyperparameter** & **Value** \\ \hline Time steps per actor batch & 2048 \\ Batch size & 128 \\ Optimization epochs & 10 \\ Learning rate & \(3\times 10^{-4}\) \\ Discount factor \(\gamma\) & 0.99 \\ GAE \(\lambda\) & 0.95 \\ Clip parameter & 0.2 \\ Entropy coefficient & 0.0 \\ \hline \hline \end{tabular}
\end{table}
Table 2: PPO hyperparameters.  
Additionally, we modified the parameter values in Equations (15) and (16) below to maximize the reward obtained after learning. The method that we prepared is as follows.  
First, following their method [16], we used only the heights of the left and right soles for the reference motion. Specifically, \(z_{t}^{\text{ref, left}}\) and \(z_{t}^{\text{ref, right}}\) of Equations (2) and (3) were used, and \(y_{t}^{\text{ref}}\) in Equation (1) was not used. In this method, \(z_{t}^{\text{ref, left}}\) and \(z_{t}^{\text{ref, right}}\) were used as the targets for imitation.  
We did not perform Bayesian optimization of \(h\) and \(\Delta h\). This is because, as will be explained below, we used the reference motion to define an imitation reward term. By not performing Bayesian optimization, we avoided changing the imitation reward term so that we could plot and compare their values in later figures. \(h\) and \(\Delta h\) were fixed at \(h=25\) [mm] and \(\Delta h=5\) [mm]. These are the same parameter values that enable our proposed method to successfully learn bipedal gaits, as will be shown in Section 4.3.  
Second, the target angles \(u_{t}^{\text{target}}\) of the robot's joints were determined using Equation (13), without IK. Instead, as in [PERSON] et al. [16], an imitation reward term was added to the reward function so that the reward was higher when the robot's motion was closer to the reference motion. Specifically, the reward function \(r\) was defined as  
\[r=w^{\text{I}}r^{\text{I}}+w^{\text{T}}r^{\text{T}}, \tag{14}\] \[w^{\text{I}}+w^{\text{T}}=1,\]  
where \(r^{\text{I}}\) is the imitation reward defined below, \(r^{\text{T}}\) is the task reward defined in Equation (7), and \(w^{\text{I}}\) and \(w^{\text{T}}\) are the weights that balance these rewards.  
The imitation reward \(r^{\text{I}}\) was defined as in [16],  
\[r^{\text{I}_{*}} = \exp\left[-\frac{1}{0.015^{2}}\sum_{i=1}^{2}(h_{\text{foot}_{i}}^ {\text{ref}}-h_{\text{foot}_{i}})^{2}\right], \tag{15}\] \[r^{\text{I}} = \frac{r^{\text{I}_{*}}-B_{\text{lower}}}{B_{\text{upper}}-B_{ \text{lower}}}, \tag{16}\]  
where \(h_{\text{foot}_{1}}^{\text{ref}}\) and \(h_{\text{foot}_{2}}^{\text{ref}}\) are the heights of the soles in the reference motion and are equal to \(z_{t}^{\text{ref, left}}\) and \(z_{t}^{\text{ref, right}}\) in Equations (2) and (3), respectively. \(h_{\text{foot}1}\) and \(h_{\text{foot}2}\) are the current heights of the robot's left and right soles above the ground, respectively. Thus, this imitation reward encourages the robot to change the heights of its soles in the same way as \(z_{t}^{\text{ref, left}}\) and \(z_{t}^{\text{ref, right}}\) in Equations (2) and (3). The parameters in Equation (16) were set to \(B_{\text{lower}}=0.4\) and \(B_{\text{upper}}=1.0\). These values were empirically determined though several experiments.