## 2 Learning, Improving, and Generalizing Motor Skills Based on Imitation Learning and Self-Learning
### Learning (Initial) Motor Skills through Imitation Learning  
In this process, a human provides a demonstration dataset for learning initial motor skills for classification and motion generation; after this process, a robot can generate appropriate motion trajectories given its current situation. In this work, classification is also used to select a suitable motor skill from a library. The specific motor skill obtained by the classification result is used to generate motion trajectories. To achieve this, a motor skill is represented as a parameter tuple that concatenates the parameters of a HMM and a DMP. A HMM is a model that is suitable for classifying time-varying signals. The parameters of a HMM are defined as \(\mathbf{\lambda}=\{\pi_{i},a_{ij},b_{i}\}_{i,j=1}^{K}\), where \(\pi_{i}\), \(a_{ij}\), and \(b_{i}\) denote the initial probability distribution of the \(i\)th hidden state, the transition probability distribution from the \(i\)th hidden state to the \(j\)th hidden state, and the observation probability distribution of \(i\)th hidden state, respectively. In this case, the number of hidden states \(K\) is determined using the Bayesian information criterion [25]. Moreover, parameter \(b_{i}\) is modeled as a Gaussian mixture model to represent continuous non-linear trajectories. The parameters of a HMM are estimated by employing the Baum-Welch algorithm using training data \(\{\mathbf{X}_{m}^{m}\}\), with \(m=1,2,...,M\) and \(n=1,2,...,N\), where \(M\) and \(N\) denote the total number of demonstrations and the number of data points for classification, respectively [26]. A DMP is similar to a linear spring-damper system, which is dependent on the external force term and ensures convergence to the final goal (or target) [27]. The DMP is defined as  
\[\dot{\mathbf{V}}=K(\mathbf{X}^{g}-\mathbf{X})-D\mathbf{V}+(\mathbf{X}^{g}- \mathbf{X}^{0})\zeta \tag{1}\]  
and  
\[\tau\dot{\mathbf{X}}=\mathbf{V}, \tag{2}\]  
where \(\mathbf{X}\), \(\mathbf{V}\), \(\mathbf{X}^{0}\), and \(\mathbf{X}^{g}\) represent the position, velocity, initial position, and target position, respectively. All these variables are multidimensional vectors. Moreover, \(\tau\), \(K\), and \(D\) indicate the constants for adjusting the time-scale, spring, and damping terms. The external force term \(\zeta\) must be learned from demonstration dataset, and is defined using \(s\), which is expressed as  
\[\zeta(s)=\frac{\sum_{i=1}^{L}\omega_{i}\psi_{i}(s)s}{\sum_{i=1}^{L}\psi_{i}(s)}, \tag{3}\]  
Figure 1: The proposed framework consists of three processes: (**left**) learning initial motor skills through imitation learning; (**middle**) improving the (initial) motor skills through RL; and (**right**) generalizing them with RL. where \(\psi_{i}(s)=exp(-h_{i}(s-c_{i})^{2})\) is a Gaussian basis function with center \(c_{i}\) and width \(h_{i}\). Parameter \(\omega_{i}\) is its weighting value. Parameter \(L\) represents the number of Gaussian basis functions. Term \(\zeta\) is directly dependent on the phase variable \(s\), which is monotonically reduced from 1 to 0, independent of time, and is obtained by  
\[\tau\dot{s}=-\alpha s, \tag{4}\]  
where \(\alpha\) is a predefined constant, and this differential equation is referred to as a canonical system. To learn a DMP from demonstrations, the robot learns the average path of several demonstrations. The average path \(\mathbf{X}(t)\) is recorded and its derivatives \(\mathbf{V}(t)\) and \(\dot{\mathbf{V}}(t)\) are computed for each time step \(t=0,...,T\). Next, the canonical system \(s(t)\) is computed for an appropriately adjusted temporal scaling parameter \(\tau\). Based on Equations (1) and (2), \(\zeta_{\text{target}}(s)\) is computed according to  
\[\zeta_{\text{target}}(s)=\frac{-K(\mathbf{X}^{g}-\mathbf{X})+\tau D\dot{ \mathbf{X}}+\tau\dot{\mathbf{X}}}{\mathbf{X}^{g}-\mathbf{X}^{0}}, \tag{5}\]  
where \(\mathbf{X}^{0}\) and \(\mathbf{X}^{g}\) are set to \(\mathbf{X}(0)\) and \(\mathbf{X}(T)\), respectively.