# Research on Robotic Peg-in-Hole Assembly Method Based on Variable Admittance
## 2 Materials and Methods
### TD3 Algorithm Design
#### 2.3.1 Introduction to TD3 Algorithm  
The purpose of training for peg-in-hole assembly was to develop an agent capable of learning the adjustment strategy for the admittance control parameter B by interacting with the assembly environment. The robotic peg-in-hole assembly process was treated as a Markov decision process. The TD3 algorithm employed two Actor networks to model behavioral strategies and four Critic networks to approximate the value function, with the ultimate goal of maximizing the value function. The algorithm network diagram is shown in Figure 7 below, where \(\Phi\) represents the Actor network parameters, and \(\theta\) represents the Critic network parameters.  
The Actor network output an action \(\pi_{\Phi}(s)\) based on the environmental states. To prevent the action from falling into local optima, a perturbation \(\varepsilon\) (following a normal distribution) is added to the action.  
\[a=\pi_{\Phi}(s)+\varepsilon \tag{6}\]  
Action a was applied to the environment, and the agent received the subsequent state s' and obtained a reward r. The tuple (\(s\), \(a\), \(s\)', and \(r\)) was stored in the experience buffer, known as the Buffer. When the amount of data in the experience buffer reached a predetermined amount, the network would be updated.  
Figure 6: Rotation angle around the \(X\)-axis.  
The Critic network provided a value \(Q_{\theta}(s,a)\) based on action a and state s, representing the agent's estimated action reward value and used to evaluate the effectiveness of action a on the environment. The target network of the Critic was employed to estimate the return value for the next time step. The return value was determined by the target network of the Actor and the added perturbation, with the inclusion of perturbation serving to smooth and regularize the target policy.  
The target network of Critic is used to estimate the return value \(Q_{\theta\prime}(s,a)\) for the next moment. Among them, \(a\prime=\pi_{\Phi\prime}(s\prime)+\varepsilon\) is given by the target network and perturbation of the Actor. The addition of perturbation is for smooth regularization of the target policy.  
We obtained the target value \(y=r+\gamma Q_{\theta\prime}(s\prime,a\prime)\). However, due to the increase in perturbation, the Q value may be overestimated, which, in turn, leads to an overestimation of y. Such overestimation negatively affects the decision-making of intelligent agents. Therefore, the TD3 network introduces two Q networks and two target Q networks. During training, the smaller value between the two is selected to calculate the target value y, thus solving the problem of overestimation.  
\[\left\{\begin{array}{l}y_{1}=r+\gamma Q_{\theta_{1}^{\prime}}(s\prime,a \prime)\\ y_{2}=r+\gamma Q_{\theta_{2}^{\prime}}(s\prime,a\prime)\\ y=\min(y_{1},y_{2})\end{array}\right. \tag{7}\]  
When performing network updated, the Critic network minimized the loss Li to update parameters \(\theta_{1}\) and \(\theta_{2}\) in the Critic1 and Critic2 networks, respectively.  
\[Loss_{i}=N^{-1}\sum{(y-Q_{\theta_{i}}(s,a))^{2}}\ \ \ i=1,2 \tag{8}\]  
The parameter update of Actor policy network was not synchronized with those of the of Critic network. To ensure the stability of policy network training, updates were typically delayed. The gradient rise method could be used to update parameter \(\Phi\) of the Actor policy network.  
\[\
abla_{\Phi}J(\Phi)=N^{-1}\sum{\
abla_{a}Q_{\theta_{1}}(s,a)}\Big{|}_{a=\pi_{ \Phi}(s)}\
abla_{\Phi}\pi_{\Phi}(s) \tag{9}\]  
Figure 7: TD3 network framework diagram.  
After updating the Actor network, the Actor target network and the two Critic target networks were also updated. A soft update method was employed, as shown in the following equation:  
\[\left\{\begin{array}{l}\theta_{i}^{\prime}\leftarrow\tau\theta_{i}+(1-\tau) \theta_{i}^{\prime}\\ \Phi^{\prime}\leftarrow\tau\Phi+(1-\tau)\Phi^{\prime}\end{array}\right. \tag{10}\]  
where \(\tau\) represents learning efficiency, which is typically set to around 0.01 for slow updates to ensure training stability.