## 1 Introduction
A general method is proposed that concatenates the parameters of two different models, one for classification (hidden Markov models (HMMs)) and one for motion generation (dynamic movement primitives (DMPs)), from human demonstrations. Robots are then able to select an appropriate motor skill from a library of motor skills. This method is used to classify various types of reaction force/moment signals and generate their corresponding reaction motion trajectories for the peg-in-hole task. 2. The policy learning by weighting exploration with the returns (PoWER) algorithm is used in the reinforcement learning (RL) process. Using this algorithm, the RL process improves and/or generalizes motor skills. It not only optimizes the parameters to reduce the execution time step and improve path of a DMP, but also re-estimates the parameters of its corresponding HMM to improve the motor skill. Furthermore, the RL process estimates new targets and initial parameters of new motor skills for generalization. The studies for obtaining motor skills can be broadly divided into three types: (i) predefined strategy; (ii) imitation learning; and (iii) RL methods. In Method (i), motor skills can be used immediately without any training costs. However, it is not easy to manually design motor skills and ensure their optimal solutions. Numerous researchers, such as Xu et al. [11], Park et al. [12], Zhang et al. [13], and Jokesch et al. [14], have proposed methods for solving the peg-in-hole problem, based on this method. These studies are not learning-based approaches; instead, they use predesigned strategies after analyzing the peg-in-hole task. In Method (ii), those can be learned using a few human demonstrations. However, it provides near-optimal solutions, not optimal ones. Calinon et al. [15], Ude et al. [16], and Kyrarini et al. [17] proposed methods to obtain motor skills based on imitation learning. Their motor skills are modeled from human demonstration dataset. In contrast, motor skills can be optimized based on the RL process, but a large trial-and-error iterations are required to obtain their optimal solutions in Method (iii). Yun [18] and Inoue et al. [19] proposed methods for learning the peg-in-hole task based on RL. Their aim was to enable robots to learn the motor skills for the peg-in-hole task through random explorations without human demonstration. To reduce training costs and obtain optimal solutions, many researchers have proposed ways to combine both imitation learning and RL process. These methods are able to learn the optimal solutions from initial motor skills learned by imitation learning through several trials-and-errors of the RL process. Various researchers have proposed methods to use imitation learning and RL. Kober et al. obtained DMPs based on imitation learning and improved their parameters using RL [20]. They dealt with the tasks of placing a ball into a cup and paddling a ball. Kormushev et al. used DMP-like dynamic models based on imitation learning and RL to enable a robot to learn and improve a pancake-flipping task [21]. Kroemer et al. used imitation learning and RL to divide a task into multiple phases and enable a robot to learn their transitions and motor skills [22]. The robots performed a bimanual grasping task with contact force. Levine et al. trained a neural network using a policy parameter optimization method--referred to as guided policy searching--to optimize the initial model parameters [23]. They verified their method for various tasks, including the peg-in-hole task, in virtual environments. None of these researchers considered including classification functions in their models for multiple motor skills. They only focused on improving model parameters through imitation learning and RL. This is slightly different from the definition of our motor skills. Moreover, they did not explicitly consider the generalization of motor skills, or both time step and path optimization. The remainder of this paper is organized as follows. Section 2 presents the details of the proposed framework.