# A Collaborative Control Method of Dual-Arm Robots Based on Deep Reinforcement Learning
## 3 Dual-Arm Deep Deterministic Policy Gradient Algorithm
Execute actions \(a^{t}=(a_{1}^{t},a_{2}^{t})\) and observe new state \(\mathbf{x}^{t+1}\) 9. _endfor_ 10. _for \(t\) = 1 to \(T\)do_ 11. For each agent \(i\), calculate reward \(r_{i}^{t}:=r(\mathbf{x}^{t},a_{i}^{t},g)\) 12. Store the transition \((\mathbf{x}^{t}\|g,a^{t},r^{t},\mathbf{x}^{t+1}\|g)\) in \(R\) 13. Sample a set of additional goals for replay G: = \(X\) (**current episode**) 14. _for \(g\prime\in G\)do_ 15. For each agent \(i\), \(r_{i}^{t}:=r(\mathbf{x}^{t},a_{i}^{t},g)\) 16. Store the transition \((\mathbf{x}^{t}\|g,a^{t},r^{t},\mathbf{x}^{t+1}\|g)\) in \(R\) 17. _endfor_ 18. _endfor_ 19. _for \(t\) = 1 to \(T\)do_ 20. _for agent \(i\) = 1 to 2 do_ 21. Sample a random minibatch of \(N\) transitions \((\mathbf{x}^{j},a^{j},r^{j},\mathbf{x}^{t+1,j})\) from \(R\) 22. Set \(y^{j}=r_{i}^{j}+\gamma Q_{i}(\mathbf{x}^{t+1,j},a_{1}^{t},a_{2}^{t}\big{|} \theta_{i}^{Q})\big{|}_{a_{i}^{t}=\mu_{i}(o_{i}^{t}|g_{i}^{t})}\) 23. Update critic by minimizing the loss: \[L=1/N\!\!\sum_{j}\!\left(y^{j}-Q_{i}(\mathbf{x}^{j},a_{1}^{t},a_{2}^{t}\big{|} \theta_{i}^{Q})\right)^{2}\] 24. Update the actor policy using the sampled policy gradient \[\
abla_{\theta^{j}}\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!To improve the convergence of the algorithm, the samples are processed using the HER algorithm. Mini-batch size groups of data are randomly extracted from the replay buffer. The original goal is recorded as \(g\), and the goal replacement is performed sequentially. Take a sample \((x^{i},a^{i},g^{i})\) from \(\{x^{1},a^{1},S^{1},ag^{1},\cdots,x^{t},a^{t},g^{t},ag^{t}\}\). In this paper, the reward was not given when the sample was saved from the episode the first time, because the reward needs to be calculated based on the reward function. This solution dramatically increases efficiency and saves memory. The method to create a new goal is the future strategy. Let the sampling time be step \(i\); the original goal \(g\) would be replayed by \(ag\) in \(k(k=4)\) steps, which comes from the same episode and was observed after step \(i\). That means \(g^{\prime}\gets ag^{\prime},j\in\left[i+1,T\right],\) where \(T\) was the set length of time. After the new goal \(g^{\prime}\), the reward \(r\prime\) is recalculated and the new tuple \((X^{t}||g^{\prime},a^{t},r^{\prime},X^{t+1}||g^{\prime})\) is stored in the replay buffer. The replay buffer at this time records the experience of all agents. At each timestep, the actor and critic are updated by sampling a minibatch uniformly from the buffer. The pseudo-code of the DADDPG algorithm can be described as follows:  
In practice, the data generated in an episode are saved in the replay buffer. The data include the states \(x^{i}\) and actions \(a^{t}\) at each moment, and are stored in chronological order. The set length of time is \(T=50\) and the number of elements in the sequence is \(N=\) 1,000,000. The replay buffer can be large, allowing the algorithm to benefit from learning across a set of uncorrelated transitions. When the replay buffer overflows, the oldest transitions will be discarded. If the mission has been completed within time \(T\), the round episode will be terminated early. If the mission is not completed within 50 timesteps, this round will also be forcibly terminated. This is to reduce the difficulty of exploring space.