## 2 Learning, Improving, and Generalizing Motor Skills Based on Imitation Learning and Self-Learning
### Improving and Generalizing Motor Skills through RL
Figure 3: Different demonstrations according to relative positions/directions of the peg and hole: pegs and holes at directions of: (**a**) 30\({}^{\circ}\); (**b**) 120\({}^{\circ}\); (**c**) 210\({}^{\circ}\); and (**d**) 300\({}^{\circ}\). Here, these directions represent the relative positions of the peg relative to the hole. The black and red boxes indicate the hole and the peg, respectively. The human needs to provide the corresponding demonstrations according to these configurations, and the robot learns different reaction force/moment signals and reaction motion trajectories from these demonstrations. Figure 2: “Hole search” and “peg insertion” motions in the peg-in-hole task: (**a**) tilt-search motion; (**b**) single-point contact motion, which pushes on one face of the peg to make contact with the hole; (**c**) two-point contact motion, which pushes two faces of the peg to make contact with the hole; (**d**) tilting motion to align that the peg and hole; and (**e**) pushing motion to push the peg down into the hole. ### Reward Functions of Imitation Learning and Self-Learning in the Peg-in-Hole Task

To calculate the expected return values for the \"hole search\" and \"peg insertion\" motor skills, two reward functions (i.e., \(r^{s}\) for \"hole search\" and \(r^{t}\) for \"peg insertion\") are, respectively, defined as

\[r^{s}(t)=\exp\left(-\alpha\mathbf{F}_{x,y,z}(t)-\beta\mathbf{M}_{x,y,z}(t)- \gamma\mathbf{R}_{x,y,z}(t)\right) \tag{8}\]

and

\[r^{t}(t)=\exp\left(-\alpha\mathbf{F}_{x,y}(t)-\beta\mathbf{M}_{x,y,z}(t)- \gamma\mathbf{P}_{z}(t)\right), \tag{9}\]

where \(\mathbf{F}\), \(\mathbf{M}\), \(\mathbf{R}\), and \(\mathbf{P}\) indicate the force, moment, rotation, and position measured from the robot, respectively. In particular, the variables \(\mathbf{R}\) and \(\mathbf{P}\) represent the robot configuration measured from the reference axis of the tool coordinate system. In addition, \(\mathbf{F}\) is calculated by \(\mathbf{F}_{(x,y,z)}(t)=|F_{x}^{g}-F_{x}(t)|+|F_{y}^{g}-F_{y}(t)|+|F_{z}^{g}- F_{z}(t)|\), and \(\mathbf{M}\) and \(\mathbf{P}\) are calculated using equations with a similar form. In contrast, \(\mathbf{R}\) is calculated using \(\mathbf{R}_{(x,y,z)}(t)=\frac{1}{|R_{x}^{g}-R_{x}(t)|}+\frac{1}{|R_{x}^{g}-R_ {y}(t)|}+\frac{1}{|R_{x}^{g}-R_{x}(t)|}\). Here, superscripts \(g\) and \(s\) indicate the target and starting values of each variable depending on the given task, respectively. Subscripts \(x\), \(y\), and \(z\) denote the variables of each measure. Further, parameters \(\alpha\), \(\beta\), and \(\gamma\) are constants, which are used to adjust the weight of each term. In Equation (8), the motor skills should be able to determine the minimum tilting angle needed to quickly distinguish whether or not a hole exists. This is determined using reward function \(r^{s}\) for the \"hole search\" motor skill. Here, it increases when all axes of \(\mathbf{F}_{(x,y,z)}(t)\) and \(\mathbf{M}_{(x,y,z)}(t)\) at every time step are closer to all targets \(\mathbf{F}^{g}\) and \(\mathbf{M}^{g}\). In contrast, the position of the z-axis is incorporated as a reward term for inserting the peg into the hole. The reward increases when the z-axis position \(\mathbf{P}_{z}(t)\) at every time step is closer to target \(\mathbf{P}^{g}\). It is possible for robots to determine the optimal motions. In these two equations, \(\mathbf{F}^{g}\), \(\mathbf{M}^{g}\), \(\mathbf{R}^{g}\), and \(\mathbf{P}^{g}\) are set to zero for the peg-in-hole task. In the imitation learning process, human demonstrations are provided by employing a kinesthetic teaching method. This is a method for easily and rapidly conveying the motor skills of human performers to robots. However, it is not suitable for the peg-in-hole task, in which reaction classification is important for achieving the goal because unintended reaction force/moment signals may be included in human demonstrations. When reproducing unintended reaction force/moment, the robot does not achieve the goal of the motor skill.