## 2 Learning, Improving, and Generalizing Motor Skills Based on Imitation Learning and Self-Learning
### Improving and Generalizing Motor Skills through RL
Here, the term \((\mathbf{X}^{g}-\mathbf{X}(t))\) is used to obtain a high return value when the robot configuration is close to the target values, and the term \(\frac{1}{\mathbf{Y}^{g}-\mathbf{Y}(t)}\) is used to obtain high return value when it is far from the starting value. Parameters \(\alpha\) and \(\beta\) are constants to adjust the importance of each term. Equation (7) is designed to take the form of \(\exp^{-(x)}\); therefore, a lower value of either term provides a higher return value. In this framework, we can easily obtain the value of \(\mathbf{X}^{g}\) that the robot must finally achieve from human demonstrations. The robot can use the first term in Equation (7) to obtain the optimal path and optimal execution time step while achieving the target \(\mathbf{X}^{g}\). In contrast, the robot may need to be as far from the initial value \(\mathbf{Y}^{s}\) as possible depending on the given tasks. We define this case as the second term in Equation (7). Here, the initial value \(\mathbf{Y}^{s}\) can be easily obtained from a robot or its sensors. ## 3 Experiment

### Description of the Peg-in-Hole Task

To evaluate the proposed method, we applied our framework to the peg-in-hole task. This task consists of the following two motor skills: \"Hole search\"--a parameter tuple that classifies the direction of a hole and generates the motion trajectories of searching a hole based on reaction force/moment signals in the inaccurate current positions/postures of the peg and hole--and \"peg insertion\"--the parameter tuple that classifies reactions according to the directions of the hole and generates the motion trajectories of inserting a peg into a hole. Although the \"hole search\" and \"peg insertion\" motions can be performed using various strategies [30], humans perform the \"hole search\" demonstrations by adopting a tilt-search strategy, as illustrated in Figure 2a, and they perform the \"peg insertion\" demonstrations by employing a two-point contact strategy, as indicated in Figure 2b-e. In the \"hole search\" motor skill, suitable tilting angles should be learned and then generated: if small tilting motion trajectories are generated, the hole may not be found because reaction force/moment signals cannot be measured; if large tilting motion trajectories are generated, the motion skill of \"peg insertion\" may not be performed because the peg is caught in the hole even though it has been found by \"hole search\" motor skill. Moreover, in the \"peg insertion\" motion, the robot must learn and generate suitable insertion motion trajectories for the peg according to reaction force/moment signals measured depending on the relative positions and directions of the hole and peg. To achieve this, it is important to classify the relative positional and directional relationship between the peg and hole and then to generate their appropriate motion trajectories from the selected motor skills, as shown in Figure 3.

For this purpose, the peg-in-hole task was performed using the experimental setup illustrated in Figure 4. We used the UR3 robotic arm (developed by Universal Robots, Denmark) and the FT300 F/T sensor, 2-finger gripper, and a wrist camera (all developed by Robotiq, Canada). We conducted the experiments using five pegs and five holes with triangle, rectangle, pentagon, hexagon, and star shapes, as indicated in Figure 5a-e. The clearance between the pegs and holes was approximately 200 \(\upmu\)m. The vision solution of the robot allows it to recognize the approximate peg and hole locations as well as their exact shapes. However, the peg cannot simply be inserted into the hole due to errors in the vision system of the wrist camera. The errors in the position and posture obtained in the experiments using this vision solution were approximately 5-10 mm and between 2\({}^{\circ}\) and 3\({}^{\circ}\), respectively. Therefore, both types of motor skills are needed to complete the peg-in-hole task despite these errors. Figure 4: Experimental setup for peg-in-hole task.