## 2 Learning, Improving, and Generalizing Motor Skills Based on Imitation Learning and Self-Learning
### Improving and Generalizing Motor Skills through RL
Thus, such unintended signals should be eliminated from the human demonstrations through robot self-reproduction. That is, the robot acquires the targets of motor skills as well as the reward functions through this self-reproduction. Despite this self-reproduction, robots may still not be able to obtain an optimal solution. This can be improved through RL during the improvement process. A demonstration dataset must be modeled to enable reaction classification and motion generation. In this experiment, the robot should generate different motions depending on the relative positions/directions of the peg and the hole, as indicated in Figure 5. The states of both motor skills are defined as \(\mathbf{X}=\{\mathbf{F}_{(x,y,z)},\mathbf{M}_{(x,y,z)}\}\) to enable the HMMs to classify the reaction force/moment. In the DMPs, the state of the \"hole search\" motor skill is defined as \(\mathbf{X}=\{\mathbf{R}_{(x,y,z)}\}\), owing to the fact that a tilt search is performed without changing the peg's position. In contrast, the state of the \"peg insertion\" motor skill is defined as \(\mathbf{X}=\{\mathbf{F}_{(x,y,z)},\mathbf{R}_{(x,y,z)}\}\) to control the force and rotation. Finally, the state of the

Figure 5: Five pegs and five holes with: (**a**) rectangle; (**b**) hexagon; (**c**) pentagon; (**d**) triangle; and (**e**) star shapes. two reward functions for RL are defined as \(\mathbf{X}=\{\mathbf{F}_{(x,y,z)},\mathbf{M}_{(x,y,z)},\mathbf{R}_{(x,y,z)}\}\) (to calculate Equation (8)) and \(\mathbf{X}=\{\mathbf{F}_{(x,y,z)},\mathbf{M}_{(x,y,z)},P_{z}\}\) (to calculate Equation (9)). The states for HMMs, DMPs, and reward functions are summarized, as indicated in Figure 3.0

### Results

First, we performed human demonstrations on the rectangle shape (Figure 5a) to learn and acquire the initial motor skills based on imitation learning. The different hole and peg shapes (Figure 5b-e) were used to evaluate the generalization of the motor skills. The results of these experiments can be confirmed from supplemental video clip. Human performers provided demonstrations for the \"hole search\" and \"peg insertion\" demonstrations, as shown in Figure 3.1 Figure 7 illustrates an example of the different demonstrations that were performed according to the initial hole and peg positions. Figure 7a shows the initial positions of the pegs with respect to the hole, and Figure 7b shows their clustering results. In this case, the points were clustered using the reaction force/moment measured at the initial positions and the k-means clustering algorithm. The robot acquired four motor skills for the \"hole search\" and four motor skills for the \"peg insertion\". The motion trajectories of the robot were extracted at 50 Hz using the kinesthetic teaching method, following which the training data were acquired through self-reproduction. Eight motor skills (i.e., the parameter tuples for classifying reaction signals and generating motion trajectories for four types of \"hole search\" and four types of \"peg insertion\") were learned using the training dataset. We configured the reaction classification and motion generation processes to be independent for rational task execution in the RL implementation. In this case, the classification of the reaction force/moment was configured to use five pairs of robot and sensor signals, and the classification time was set to 50 Hz to ensure real-time performance. Next, the eight initial motor skills were improved using the RL process. For self-improvement of the \"hole search\" motor skills, the RL rollouts and their rewards were generated according to the following two steps: (i) acquiring rollouts and calculating rewards in a hole-free (blocked) location; and (ii) verifying rollouts and updating policy parameters at the hole position. The robot verified which of the rollouts actually found the hole at its location and changed the rewards of the rollouts that failed to find holes to zero. It then updated the policy parameters using all the rollouts.