# A Collaborative Control Method of Dual-Arm Robots Based on Deep Reinforcement Learning
## 2 Preliminary Knowledge
### Multi-Agent Deep Deterministic Policy Gradient  
Traditional reinforcement learning algorithms are poorly suited to multi-agent environments. One issue is that during the training process, each agent's policy is changing, and the environment becomes non-stationary from the perspective of any individual agent. This presents learning stability challenges and prevents the use of past experience replay.  
On the other hand, policy gradient methods usually have very high variance when multiple agents need efficient coordination.  
The MADDPG algorithm aims to solve the multi-agent control strategy in a variety of cooperative and competitive multi-agent environments, which is an extension of the Deep Deterministic Policy Gradient (DDPG) [29]. DDPG uses the Actor-Critic [30] framework, which includes two neural networks: a target policy (the actor) \(\mu:S\to A\) and an action value function approximator (the critic) \(Q:S\times A\to R\). The actor select actions based on the current observations. The critic's job is to approximate the actor's action-value function \(Q^{\mu}\). MADDPG adopts the framework of centralized training with decentralized execution. The critic can obtain extra information about the policies of other agents, while the actor only obtains local information. During the execution phase after training is completed, only the local actors are used to act in a decentralized manner.  
More specifically, suppose that there are \(N\) agents in the system. The set of all policies is \(\mu=\{\mu_{1},\cdots,\mu_{N}\}\), which is parameterized by \(\theta=\{\theta_{1},\cdots,\theta_{N}\}\). Every policy is represented by one neural network. Each agent's policy only depends on its own observations, which means \(a_{i}=\mu_{i}(o_{i})\), and no structural assumptions are made on the communication methods between agents. The action value function \(Q^{\mu}_{i}(\mathbf{x},a_{1},\cdots,a_{N})\) describes the expected return after taking the actions, \(a_{1},\cdots,a_{N}\), in state \(\mathbf{x}\). In the simplest case, \(\mathbf{x}\) could consist of the observations of all agents, \(\mathbf{x}=(o_{1},\cdots,o_{N})\). The critic network parameters are updated by minimizing the following loss:  
\[\begin{split} L(\theta_{i})=\mathrm{E}_{\mathbf{x},\theta,r, \mathbf{x}}[(Q^{\mu}_{i}(\mathbf{x},a_{1},\cdots,a_{N})-y)^{2}],\\ y=r_{i}+\gamma Q^{\mu}_{i}(\mathbf{x}^{\prime},a^{\prime}_{1}, \cdots,a^{\prime}_{N})\big{|}_{a^{\prime}_{i}=\mu^{\prime}_{i}(o_{i})}\end{split} \tag{4}\]  
where, \(y\) is the target value, \(\gamma\in[0,1]\) is a discount factor, and \(\mu^{\prime}=\big{\{}\mu_{\theta_{1},r},\cdots,\mu_{\theta_{N},r}\big{\}}\) is the set of target policies with delayed parameters \(\theta_{i}\prime\). The parameters of the action network are updated based on the gradient descent method. Using the \(Q\) function defined previously, the gradient of the policy can be written as:  
\[\
abla_{\theta_{i}}J(\mu_{i})=\mathrm{E}_{\mathbf{x},a\sim D}[\
abla_{\theta ^{\mu}}\mu_{i}(a_{i}\big{|}o_{i})\
abla_{a_{i}}Q^{\mu}_{i}(\mathbf{x},a_{1}, \cdots,a_{N})\big{|}_{a_{i}=\mu_{i}(o_{i})}]. \tag{5}\]  
where, \(D\) is a distribution over transitions \((\mathbf{x},a,r,\mathbf{x}^{\prime})\) contained in the replay buffer.