## 2 Learning, Improving, and Generalizing Motor Skills Based on Imitation Learning and Self-Learning
### Improving and Generalizing Motor Skills through RL
The verification process in Step (ii) is necessary because the rollouts and their rewards are obtained at a location where no hole exists. As mentioned in Section 2.2, the purpose of improving the \"hole search\" motor skill is to enable robots to identify whether a hole is present using the tilting angle. The self-improvement of the \"peg insertion\" motor skill was performed at the hole location. This process did not require a specific verification step for the acquired rollouts because it was performed at the hole location. The robot was able to identify the optimal path for inserting the peg into the hole within a short time. \begin{table}
\begin{tabular}{c c c} \hline \hline  & **Hole Search** & **Peg Insertion** \\  & **(Tilt Search)** & **(Two-Point Contact)** \\ \hline
**HMMs** & Reaction force/moment & Reaction force/moment \\  & \(\mathbf{X}=\{f_{x},f_{y},f_{z},m_{x},m_{y},m_{z}\}\) & \(\mathbf{X}=\{f_{x},f_{y},f_{z},m_{x},m_{y},m_{z}\}\) \\
**DMPs** & Rotation (No change of position) & Force, rotation \\  & \(\mathbf{X}=\{r_{x},r_{y},r_{z}\}\) & \(\mathbf{X}=\{f_{x},f_{y},f_{z},m_{x},m_{y},m_{z}\}\) \\
**RL** & Reaction force/moment and rotation & Reaction force/moment and z-position \\  & \(\mathbf{X}=\{f_{x},f_{y},f_{z},m_{x},m_{y},m_{z},r_{x},r_{y},r_{z}\}\) & \(\mathbf{X}=\{f_{x},f_{y},f_{z},m_{x},m_{y},m_{z},P_{z}\}\) \\ \hline \hline \end{tabular}
\end{table} 
Table 1: States for HMMs, DMPs, and reward for RL for “hole search” and “peg insertion” motor skills. The return values increase with the number of iterations, and the number of execution time steps are reduced when using the reward functions and the iPoWER algorithm, as indicated in Table 2. In the \"hole search\" and \"peg insertion\" motor skills, the robot obtained the expected return value of 0.7412 during 64 steps of motion in the first iteration while it received the expected return value of 0.9611 during 9 steps of motion after 300 iterations and the expected return value of 0.6233 during 75 steps of motion in the first iteration while it received the expected return value of 0.9249 during 10 steps of motion after 300 iterations, respectively. This table illustrates that the reward functions and the iPoWER algorithm were effectively designed in terms of time and path optimization. Figure 8 illustrates the results of the iPoWER algorithm compared with the original PoWER algorithm. The iPoWER algorithm reduces the number of robot execution time steps. Fewer execution time steps were needed when the iPoWER algorithm were used, as indicated in Figure 8.

Figure 6: Four peg directions of a human demonstration of the “hole search” and “peg insertion” motor skills of the peg-in-hole task: (**a**) 30\({}^{\circ}\); (**b**) 120\({}^{\circ}\); (**c**) 210\({}^{\circ}\); and (**d**) 300\({}^{\circ}\). The upper and lower rows illustrate the motions of “hole search” and “peg insertion”, respectively. In the upper low, the black boxes (holes) and red lines (pegs) indicate relative postures. Figure 7: Human demonstrations and their clustered results. (**a**) Initial positions of the pegs (“x” symbols) with respect to the hole. The edge of the gray box indicates the boundary between the outside and inside of the hole. (**b**) Clustering results obtained by the k-means clustering algorithm using the reaction force/moment measured at the initial positions. We performed approximately 50 demonstrations to evaluate the clustering results. Figure 9 illustrates the RL performance of our framework for the following cases: (a) the initial motor skill was represented by a HMM only; (b) the initial motor skill was represented by a DMP only; and (c) the initial motor skill was represented by both a HMM and a DMP (our framework). To evaluate Case (a), we performed RL after only learning the initial parameters \(\mathbf{\lambda}\) for a HMM and target \(\mathbf{X}^{\mathbf{\delta}}\) from human demonstrations. However, the initial weight parameters \(\mathbf{\Omega}\) for a DMP were randomly assigned without imitation learning (yellow line in Figure 9).