# A Collaborative Control Method of Dual-Arm Robots Based on Deep Reinforcement Learning
## 2 Preliminary Knowledge  
### Hindsight Experience Replay  
The problem of sparse reward has always been the core issue that deep reinforcement learning faces in solving practical tasks [21; 22]. In supervised learning, supervised signals are provided by training data. In reinforcement learning, rewards play the role of super-vising signals, and agents optimize strategies based on rewards. The problem of reward sparseness will cause the algorithm to iterate slowly and even converge difficultly [23]. In the collaborative task of the robotic arm, the robot can only obtain the reward after successfully completing a series of complex posture control to grab the target. Failure to succeed in any step in the training progress will lead to failure to obtain the reward [24; 25].  
Hindsight Experience Replay has been introduced to learn policies from sparse rewards, especially for robot manipulation tasks [26]. The rationale of HER is to replay each trajectory with an arbitrary goal. In this way, when the agent fails to achieve the desired goal in the episode, the additional goal can be used instead of the original desired goal, and then the reward is recalculated according to the new goal to provide experience for the agent. HER improves the sample efficiency in the sparse reward DRL algorithm and can be used in combination with any off-policy reinforcement learning algorithm [27]. The key of the HER algorithm is how to construct the replay buffer [28], which includes two steps. Firstly, a fixed size queue where each entry is a 5-tuple and is randomly selected from \(M\) sequences, and the initial state \(s_{0}\) and goal \(g\) are defined. The agent produces an action based on the current state and goal \(g\). The action formula is shown as follows:  
\[a_{t}\leftarrow\pi(s_{t}||g) \tag{1}\]  
where \(s_{t}\) and \(g\) are the current state and goal, \(a_{t}\) is the selected action, and \(\pi\) is the behavior strategy. Then, the reward can be calculated as follows:  
\[r_{t}\gets r(a_{t},s_{t}||g) \tag{2}\]  
The sample \((s_{t},a_{t},r_{t},s_{t+1},g)\) will be stored in the replay buffer. Then, a complete agent's experience sequence at each time is sampled based on the actual goal \(g\). Secondly, the reward is recalculated using a new goal \(g^{\prime}\). The reward formula is as follows:  
\[r^{\prime}\gets r(a_{t},s_{t}||g^{\prime}) \tag{3}\]  
The same data structure as the normal training transitions is constructed. The obtained transitions \((s_{t},\ a_{t},\ r^{\prime},s_{t+1},\ g^{\prime})\) are also stored in the replay buffer. Then, the new experience transitions are obtained by changing the goal. After the replay buffer is constructed, the agent can be trained through some off-policy reinforcement learning algorithms.  
There are four main methods to obtain new goals, including: final, future, episode, and random. The final strategy is to take the final state in each episode as the new goal. The future strategy is to replay the successive states with \(k\) random states, which come from the same episode after the current time. The episode strategy is to replay the special states with \(k\) random states, which come from the same episode as the transition being replayed. The random strategy is to replay the special states with \(k\) random states that appeared so far in the whole training procedure. A new experience sample can be obtained based on the first goal generation strategy. The last three strategies have a hyper-parameter \(k\) which controls the ratio of HER data to those coming from normal experience replay in the replay buffer, and \(k\) new experience transitions can be generated from one original sample.