## 2 Learning, Improving, and Generalizing Motor Skills Based on Imitation Learning and Self-Learning
### Improving and Generalizing Motor Skills through RL
Here, their motor skills and the parameters of \(\mathbf{X}^{g}\) and \(\mathbf{Y}^{s}\) can be modeled and extracted using various types of information (e.g., joints, positions, postures, velocities, forces, and/or torques) from human demonstrations depending on the purpose of motor skills. Finally, the robot can optimize or generalize motor skills through the improvement/generalization processes. However, it is necessary to determine the information to be modeled or used through human demonstrations in this process. The automation of this capability is not considered in this paper and it should be done with human. In general, a robot needs to perform a sequence of some motor skills to perform its task. In other words, motor skills must be selected from a library of multiple ones. However, many researchers have focused on dealing with a single motor skill [37; 38; 39; 40]. They suggested the ways to improve the performance of a motor skill. In addition, they did not consider the generalization of reusing the learned motor skills for other similar tasks (e.g., from the \"rectangle\" to the \"triangle\" peg-in-hole motor skills). In contrast, our proposed framework was able to handle the library of multiple motor skills based on the concatenated parameters. Furthermore, Algorithms 1-3 provide a way to improve existing motor skills (optimizing paths and reducing execution time steps) as well as add new motor skills. In the human demonstrations, we adopted a tilt search and two-point contact strategies for the \"hole search\" and \"peg insertion\". This is because humans tend to accomplish the peg-in-hole task by tilting a peg into a hole, as analyzed in [30]. The authors of [30] also determined the most appropriate tilting angle for inserting the peg. In this study, a suitable tilting angle was learned for the tilt search through RL. Moreover, our aim was to predict and select appropriate motor skills from current reaction signals using learned motor skills. In contrast, it is difficult to use reaction classification with different peg-in-hole task strategies, such as the spiral path and spray paint strategies, because ofthe uncertainties of the initial positions/poses of holes and pegs. These strategies tend to generate motions according to a set of predefined rules without any classification process. The reward functions include the terms of the force, moment, \(z\)-axis robot position, \(xyz\) robot rotation, and time step. Meta-parameters \(\alpha\), \(\beta\), and \(\gamma\) control the weights of the terms individually because they deal with different information types (see Equations (8) and (9)). We can change the importance of each term in the reward function by adjusting these meta-parameters. We assigned the largest weighting values to the force/moment terms and the z-position term in the \"hole search\" and the \"peg insertion\" motor skills, respectively. In other words, it makes sense to assign weighting values depending on terms that have a significant impact on the success of the task. ## 5 Conclusions and Future Work

We propose a framework for learning, improving, and generalizing the \"hole search\" and \"peg insertion\" motor skills for the peg-in-hole task. In this framework, motor skills are acquired using a mixture of imitation learning and RL. Reaction classification and motion generation are required in the peg-in-hole task owing to errors in the sensors and actuators. The robot learns the initial motor skills for classifying the reactions and generating the appropriate trajectories from human demonstrations. We designed a motor skill parameter tuple by concatenating the HMM parameters for reaction classification and the DMP parameters for motion generation. The initial motor skills learned using imitation learning are improved and generalized by means of RL. These motor skills are either improved for familiar reaction signals or generalized for unfamiliar reaction signals.