# A Collaborative Control Method of Dual-Arm Robots Based on Deep Reinforcement Learning
## 3 Dual-Arm Deep Deterministic Policy Gradient Algorithm  
The problem setup is as follows. Given a cooperative task, the dual-arm robot must learn how to collaborate to complete the task through a sequence of actions. This process is controlled by two agents in the continuous action space. The policy of each agent changes during training, resulting in a non-stationary environment. Agents not only adapt to the state of the environment but also cooperate with other agents' action strategies. Based on HER and MADDPG, a dual-arm deep deterministic policy gradient algorithm (DADDPG) was proposed for collaborative control of dual-arm robots. Figure 1 shows the block diagram of the DADDPG algorithm. The DADDPG algorithm uses two agents to control two robotic arms, and each agent is trained using the DDPG algorithm. In order to encourage agents to reduce competition and increase collaboration in overlapping action spaces, the two agents share observations and actions with each other. Each agent adjusts its own action strategy according to the output of the other one and finally achieves cooperation. More specifically, each of the two mechanical arms has an agent, and they interact within the same environment. The training algorithm for each agent is DDPG. At each timestep, each agent \(i\) takes an action \(a_{i}\) to interact with the environment; the joint action leads to a new state, and each agent \(i\) receives their own separate observations \(o_{i}\). However, in order to achieve cooperation between the two arms, a collaborative mechanism of rewarding coordination and punishing competition is introduced. The agents can share the states and actions of each other. Namely, each agent's actor can obtain a global observation vector \(\mathbf{x}=(o_{1},o_{2})\). For each agent \(i\), the algorithm maintains two actor and critic functions. One of the actor functions \(\mu_{i}(\mathbf{x}_{i}\big{|}\theta_{i}^{t})\) is the current policy to select an action based on the observation \(o_{i}\), and one of the critics \(Q(\mathbf{x},a\big{|}\theta_{i}^{Q})\) is learned using the Bellman equation as in MADDPG. The other actor and critic networks, \(\mu_{i}(\mathbf{x}_{i}\big{|}\theta_{i}^{\mu_{i}})\) and \(Q^{\prime}(\mathbf{x},a\big{|}\theta_{i}^{Q^{\prime}})\), respectively, are a copy of them. They are used to calculating the target values. The weights of these target networks are updated by slowly tracking the learned networks. At the same time, samples from a noise serial \(N\) are added to the actor policy to treat the problem of exploration. Agent \(i\) makes an action \(a_{i}^{t}\) based on the output of its actor network and random  
Figure 1: The block diagram for the dual-arm deep deterministic policy gradient (DADDPG) algorithm. noise, and obtains observation \(o_{i}^{t+1}\) which is \(ag_{i}^{t}\) generated by \(a_{i}^{t}\) (\(ag\) represents the state that the agent has reached, which can also be called the achieved local goal). A replay buffer is used to ensure the samples are independently and identically distributed. The replay buffer is a cache \(R\), which has a finite size. At each step, the actor and critic are updated by sampling a minibatch uniformly from the buffer. 1. For each agent i: 1. randomly initialize critic network \(Q_{i}(\mathbf{x},a_{1},a_{2}\big{|}\theta_{i}^{Q})\) and actor \(\mu_{i}(o_{i}\big{|}\theta_{i}^{\mu})\) with weights \(\theta_{i}^{Q}\) and \(\theta_{i}^{\mu}\) 2. Initialize target network \(Q_{i}\prime\) and \(\mu_{i}\prime\) with weights \(\theta_{i}^{Q\prime}\leftarrow\theta_{i}^{Q}\), \(\theta_{i}^{\mu\prime}\leftarrow\theta_{i}^{\mu}\) 2. Initialize replay buffer: \(R\) 3. _for episode = 1 to \(M\)do_ 4. Initialize a random process \(N\) for action exploration 5. Sample a goal \(g\) and an initial state \(\mathbf{x}^{1}\) 6. _for \(t\) = 1 to \(T\)do_ 7. For each agent \(i\), select action \(a_{i}^{t}=\mu_{i}(o_{i}\big{|}\theta_{i}^{\mu})+N\) according to the current policy and exploration noise 8.