## 2 Learning, Improving, and Generalizing Motor Skills Based on Imitation Learning and Self-Learning
### Improving and Generalizing Motor Skills through RL
Figure 9: Expected returns of policy parameters with respect to number of iterations in the peg-in-hole task: (a) (yellow line) using only a HMM (without a DMP learned from human demonstrations); (b) (blue line) using only a DMP (without a HMM learned from human demonstrations); and (c) (red line) using both a HMM and a DMP. The average and variances of multiple RL trials from four different directions for the “hole search” and the “peg insertion” motor skills are shown. In these experiments, we generated the stop signal when the robot did not reach its target within \(\mathbf{X}^{\mathbf{g}}\pm 0.001\) or when the policy parameter did not converge within 300 iterations. When only the RL process was used without the imitation learning, it did not satisfy the targets of RL process, as shown in Figure 9c. We used several programs by modifying the open codes of DMP, HMM, and PoWER algorithms (these open codes can be downloaded in [31; 32; 33]) developed by the python language. In addition, the k-means and BIC algorithms performed using scikit-learn library. This library can be downloaded in [34]. Finally, the TM was manually created based on [24]. All of these experiments were performed on a PC (CPU: Intel i7-6700 3.40 GHz, RAM: 32.0 GB) with Windows 10 OS and Python 3.6 version. ## 4 Discussion

The proposed framework for the peg-in-hole task uses a mixture of imitation learning and RL. The peg-in-hole task requires the reaction force/moment classification and reaction motion generation because there is error in the sensors and robot actuators. Therefore, a robot should be able to continuously classify reactions and generate appropriate motion trajectories. To achieve this, motor skills are represented by concatenating the model parameters for reaction classification and motion generation. In general, a HMM and a DMP exhibit superior capabilities for time-varying classification and motion generation, respectively, as mentioned in [35]. Therefore, we used both models to consider their advantages in this framework. We refer to these concatenated parameter tuples as motor skills. We can use only HMMs or only DMPs for classification and motion generation, but they perform worse than the combination of HMMs and DMPs (refer Figure 9). The proposed framework was evaluated for the peg-in-hole task; however, it can be used for various robotic tasks. The algorithms presented in Algorithms 1-3 are task-independent and can be used without modification to improve and generalize the motor skills required for such tasks. In the algorithms, only two elements need to be prepared for various tasks: (i) the reward functions; and (ii) human demonstrations. Designing the reward functions is the most important and difficult process in RL design. The target configurations of robots and/or objects obtained by human demonstrations can be useful in designing reward functions. It is also necessary to acquire the initial parameters of the motor skills (that is, the parameters of the HMMs and DMPs) from the human demonstrations of a target task. The robot can acquire the initial parameters of the motor skill when at least one demonstration has been performed [36]. Thereafter, it can use the proposed algorithms to create various motor skills that are automatically optimized by the improvement and generalization processes. After these two elements have been provided, the robot can obtain motor skills for which the number of execution time step and path have been optimized over several iterations. For example, this framework can also be considered for the use in a variety of industrial applications such as polishing, machine tending, soldering, painting, cutting, grinding, deburring, and inspection. First, humans provide human demonstrations to a robot. Next, the robot learns initial motor skills and collects the parameters of \(\mathbf{X}^{g}\) and \(\mathbf{Y}^{s}\) for reward functions from human demonstrations.