# Robotic Peg-in-Hole Assembly Strategy Research Based on Reinforcement Learning Algorithm
## 3 Results and Discussion
### Compliance Assembly Experiment Using DDPG  
To search the optimal policies, the DDPG algorithm explored all possible actions, which required a substantial amount of experimental data, and would probably generate some risky actions in realistic environments. Thus, we realized the assembly experiment based on the variable admittance control using DDPG in the simulation environment, where the simulation platform (as shown in Figure 10) was consistent with the realistic platform. Moreover, the gravity compensation algorithm for the simulation environment was verified to ensure that the force/torque data used were realistic. As shown in Figure 11, the compensated force data were approximately \(-0.001\) in the no-contact state. Obviously, the gravity compensation algorithm was effective, and the real contact force/torque could be obtained. The stiffness value of the network output was set to 0.1-4. The primary training parameters of the networks were decided through experimental testing, as described in Table 2. Additionally, the other experimental parameters were the same as in the former assembly experiment.  
Figure 10: Simulation platform.  
Figure 9: Online training results of variable stiffness compliance assembly using SARSA. (**a**) Cumulative reward in each episode; (**b**) Adjustment steps in each episode; (**c**) Insertion depth in each episode.  
The online training results of the variable stiffness compliance assembly using DDPG are shown in Figure 12. There were approximately 14 episodes needed to fill the memory pool, so the curves oscillated in the early training episodes. Then, the models in the DDPG began to learn and converge gradually. Additionally, the three curves finally converged approximately at the twenty-fifth episode, where the reward and steps stabilized at approximately \(-0.2\) and \(7\), respectively. Meanwhile, the assembly task of all episodes was successful. We noticed that three curves approximated a straight line after convergence due to fewer disturbance in the simulation environment. Compared with the compliance assembly experiment using SARSA, it could be found that the agent using the DDPG could learn a better assembly policy with a higher cumulative reward in each episode. Meanwhile, this method also had robustness against changes in the initial position and angle of the peg.  
For the quantitative analysis of the proposed variable admittance control method based on the RL algorithm, the results of the assembly experiments with different methods were compared, as shown in Table 3. The proposed methods in this article had fewer convergence steps and adjustment steps under similar success rates, as well as better comprehensive performance than other methods. In summary, the above experimental results verified the improvement of the assembly effects well for the robotic assembly task using the variable admittance control based on the RL algorithm.  
\begin{table}
\begin{tabular}{c c} \hline \hline
**Parameter Name** & **Value** \\ \hline Actor learning rate & 0.005 \\ Critic learning rate & 0.005 \\ Target update rate & 1 \\ Batch size & 50 \\ Memory pool size & 300 \\ Reward discount & 0.95 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Training parameters.  
Figure 11: The results of the gravity compensation. (**a**) Uncompensated force data; (**b**) Compensated force data.  
Figure 12: Online training results of variable stiffness compliance assembly using DDPG. (**a**) Cumulative reward in each episode; (**b**) Adjustment steps in each episode; (**c**) Insertion depth in each episode.