# Research on Robotic Peg-in-Hole Assembly Method Based on Variable Admittance
## 2 Materials and Methods
### TD3 Algorithm Design
#### 2.3.3 Design of Reward Reward Function  
A well-designed reward function can enhance the efficiency of learning algorithms and the effectiveness of trained models [26]. To account for as many factors affecting the assembly as possible, a nonlinear reward function was designed after balancing the impact of the single-step assembly process and the global assembly process. The reward function consisted of two types: regular rewards and terminal rewards.  
Regular rewards considered the effects of pose increment and contact force magnitude. If the pose increment was too small, no reward was granted; if the increment was too large, the system would cap the pose increment and grant the maximum reward.  
To prevent the system from getting stuck in a state of maximizing the single-step pose increment, the contact force reward was set as a penalty, where the greater the contact force, the greater the penalty. Upon completion of the assembly, the system granted a terminal reward based on the total assembly time; if the assembly failed, a negative terminal reward would be given directly.  
The specific reward function was as follows:  
\[\mathrm{r}_{PO}=\frac{1}{2}\left[\sin^{1/2}P_{t}+\sin^{1/2}O_{t}\right] \tag{11}\]  
\[r_{FT}=\left\|\frac{\mathrm{F}}{\mathrm{F}max},\frac{T}{T\mathrm{max}}\right\| \tag{12}\]  
\[r_{done}=\left\{\begin{array}{lr}1-\frac{t}{t_{\mathrm{max}}}&\text{, \emph{succcessful}}\\ -1&\text{,\emph{failed}}\end{array}\right. \tag{13}\]  
\[r=\omega_{1}\mathrm{r}_{PO}-\omega_{2}\mathrm{r}_{FT}+\omega_{3}\mathrm{r}_{done} \tag{14}\]  
\[P_{t}=\frac{\pi}{2}clip\left[\frac{H_{t}-H_{\mathrm{min}}}{H_{\mathrm{max}}-H _{\mathrm{min}}},0,1\right] \tag{15}\]  
\[O_{t}=\frac{\pi}{2}clip\left[\frac{\theta_{t}-\theta_{\mathrm{min}}}{\theta_{ \mathrm{max}}-\theta_{\mathrm{min}}},0,1\right] \tag{16}\]  
where \(r_{po}\) was a regular reward based on posture. \(P_{t}\) represented the single-step insertion depth process, and \(O_{t}\) represented the single-step rotation posture process. \(H_{min}\) was the minimum single-step displacement allowed, with a value of 0.04 mm. \(H_{max}\) was the maximum single-step displacement allowed in the Z-direction, with a value of 0.1 mm. \(\theta_{min}\) was the minimum single-step rotation angle around the X-axis, with a value of 0.04\({}^{\circ}\). \(\theta_{\mathrm{max}}\) was the maximum single-step rotation angle, with a value of 0.12\({}^{\circ}\). The function 'clip' was a clipping function that limited the input value to 1 if it was greater than 1, and to 0 if it was less than 0.  
Where \(r_{FT}\) was a regular reward based on force. \(F\) was the force acting along the Z-direction, \(T\) was the moment acting around the \(X\)-axis direction of the tool, \(F_{maxz}\) was the maximum allowable force expected on the Z-axis, its value was 1 N. \(T_{max}\) was the maximum allowable torque expected around the \(X\)-axis of the tool, its value was 0.35 N\(\cdot\)m.  
Where \(r_{done}\) was terminate reward. \(t\) was the time taken to complete the assembly, and \(t_{max}\) was the maximum assembly time allowed, set to 24 s.  
Where \(r\) was the overall reward. \(\omega_{1}\), \(\omega_{2}\), \(\omega_{3}\) were the weight of each reward. The values set for this experiment were as follows: 1.0, 0.5, and 1000.