## 2 Learning, Improving, and Generalizing Motor Skills Based on Imitation Learning and Self-Learning
### Improving and Generalizing Motor Skills through RL
```  
**Algorithm 1** Overall algorithm for improving and /or generalizing motor skills  
```
1: Input: initial parameters \(\mathbf{\Theta}_{0}=\{\mathbf{\lambda}_{0},\mathbf{\Omega}_{0},\mathbf{X}^{\mathcal{S}},T_{0}\}\) of a motor skill
2: Set \(\mathbf{\Theta}_{k}=\mathbf{\Theta}_{0}\)
3:while true do
4: Sampling: Using \(\mathbf{\Omega}_{k},\mathbf{X}^{\mathcal{S}}\), and \(T_{k}\), generate rollout (\(\mathbf{\



The iPoWER and gPoWER algorithms use a deterministic policy \(\bar{\mathbf{a}}=\mathbf{\Omega}^{T}\mathbf{\Psi}(\mathbf{X},t)\) with the weighting parameters \(\mathbf{\Omega}\) and basis functions \(\mathbf{\Psi}\) of a DMP [20]. When optimizing and generalizing a DMP, this policy is transformed into a stochastic policy using additive exploration \(\epsilon(\mathbf{X},t)\); to perform model-free RL, we always use a policy \(\pi(\mathbf{a}_{t}|\mathbf{X}_{t},t)\), which can be modified into the form \(\mathbf{a}=\mathbf{\Omega}^{T}\mathbf{\Psi}(\mathbf{X},t)+\epsilon(\Psi( \mathbf{X},t))\). Here, \(\epsilon(\Psi(\mathbf{X},t))=e_{t}^{T}\mathbf{\Psi}(\mathbf{X},t)\) with \(\lfloor\varepsilon_{t}\rfloor_{ij}\sim N(0,\sigma^{2})\) is used, where \(\sigma_{ij}\) is a meta-parameter of the exploration that can also be optimized in these algorithms. In the iPoWER algorithm, the target \(\mathbf{X}^{g}\) of a motor skill is used without any changes or updates. The length of a corresponding DMP can be reduced by the stop signal \(t_{stop}\) when \(\mathbf{X}^{g}=\mathbf{X}_{t}\) and \(t<T_{k}\). This means that during the RL process, robots can arrive more quickly at the target than the humans who demonstrated the motion. Stop signal \(t_{stop}\) is generated when the robot reaches the target within extremely small margins \(|\mathbf{X}^{g}\pm\epsilon_{x}|\) or when execution time steps reaches a pre-selected length \((T+\epsilon_{T_{k}})\). The policy parameters of a DMP are optimized, after which the parameters of its corresponding HMM are re-estimated while generating the motion trajectories from the optimized DMP. In the generalization process, the gPoWER algorithm finds a new target and the policy parameters for achieving it. To do this, parameter \(\hat{T}\) is set to be the time step (\(\arg\max_{t}r(t)\)) such that the reward value in all rollouts is higher than \(r_{\max}\) and the new target is set to be the robot configuration in time step \(\mathbf{X}(\hat{T})\). Parameters \(\mathbf{X}^{g}\) and \(t_{stop}\) are determined by the value of \(r_{\max}\), which indicates the highest reward in all rollouts and the time step in the rollout of the value of \(r_{\max}\). Parameters \(\mathbf{\Omega}_{new}\), \(\mathbf{X}^{g}_{new}\), and \(T_{new}\) are incrementally updated until the parameters converges: \(\mathbf{\Theta}_{k+1}=\mathbf{\Theta}_{k}\). Next, the parameters \(\mathbf{\lambda}_{new}\) of a HMM are estimated during motion generation with \(\mathbf{\Omega}_{new}\), \(\mathbf{X}^{g}_{new}\), and \(T_{new}\). In this algorithm, the length of the new motor skill may be increased from the original one of the initial motor skill. Therefore, the length of motor skill is changed using \(\epsilon_{T_{k}}\) while estimating the new target and the parameters of the new motor skill. The length of the new motor skill can be optimized during the improvement process after the generalization process is complete. To calculate the expected return values for the improvement and generalization processes, a reward function should be defined. Its general equation is defined as

\[r(t)=\exp\left(-\alpha\left(\mathbf{X}^{g}-\mathbf{X}(t)\right)-\beta\left( \frac{1}{\mathbf{Y}^{g}-\mathbf{Y}(t)}\right)\right), \tag{7}\]

where \(\mathbf{X}\) and \(\mathbf{Y}\) indicate the values that can be measured from the robot and/or sensors. Here, these are only used to represent different variables. The superscripts \(g\) and \(s\) denote the target and starting values of each variable, which depend on the given task.