## 2 Deep Reinforcement Learning Method
### Hindsight Experience Replay for Collision Avoidance  
Hindsight Experience Replay (HER) is an experience pooling mechanism for improving the training efficiency of agents used in sparse and binary reward situations. The basic idea is that every experience of an agent still contains valuable information even if it fails to reach the goal. By treating a hindsight goal from failed experiences as the original goal, HER enables the agent to receive rewards frequently.  
To improve HER for the collision problem, the path points are considered as target points in the HER algorithm while the obstacle points in the collision experience are moved. The location of the new obstacle point is at a random location, thus there is a high probability that the experience will become a collision-free experience. In this way, agent learns the collision avoidance ability from the experience of failed collision avoidance, and achieves the purpose of accelerating deep reinforcement learning training and improving the training effect, as shown in Figure 3.  
There are two ways to keep obstacles away when using HER-CA, obstacle reset and obstacle move away. Obstacle reset is when an obstacle is randomly placed into the space as if resetting the environment, and obstacle move away is to move the obstacle away by \(x_{random}\), \(y_{random}\) distance, where \(x_{random}\), \(y_{random}\) are random numbers. We will perform a comparison of the two later in the course of the experiment.  
\begin{table}
\begin{tabular}{c c} \hline \hline
**Symbol** & **Representation** \\ \hline \(x\) & Current position x-coordinate \\ \(y\) & Current position y-coordinate \\ \(x_{goal}\) & Target position x-coordinate \\ \(y_{goal}\) & Target position y-coordinate \\ \(x_{obs1}\) & Obstacle 1 location x-coordinate \\ \(y_{obs1}\) & Obstacle 1 location y-coordinate \\ \(x_{obs2}\) & Obstacle 2 location x-coordinate \\ \(y_{obs2}\) & Obstacle 2 location x-coordinate \\ \hline \hline \end{tabular}
\end{table}
Table 2: Description of state space elements.  
In HER-CA we save the data recorded in each game as OP, save the data of one of the steps as OS, and use the future strategy [17] for goal conversion, i.e., converting a random step after the current position to the goal position, and using obstacle resets to generate possible collision-free experiences. See Algorithm 1 for a more formal description of the algorithm.  
```
1:Initialize: TD3
2:for\(epoch\leftarrow\)1 to\(MAX_{e}\)do
3: Play a number of games and add the data from each game to the data pool PL.
4:for\(trun\leftarrow\)1 to\(MAX_{t}\)do
5: Randomly select a game OL from PL.
6: Randomly select a step OS from OL.
7: Get the future step coordinates \(x_{future},y_{future}\).
8:\(x_{goal}\gets x_{future},y_{goal}\gets y_{future}\)
9:if OS collision then
10:\(x_{obsi}\gets x_{obsi}+x_{random},x_{random}\in[-40,40]\)
11:\(y_{obsi}\gets y_{obsi}+y_{random}\cdot y_{random}\in[-40,40]\)
12:Limit \(x_{obsi}\) and \(y_{obsi}\) to 0 to 90.
13:endif
14: Recalculating OS's reward value.
15: One network training using OS's data.
16:endfor
17: Soft update of the network model.
18: Save Action Models.
19:endfor
```  
**Algorithm 1** Hindsight Experience Replay for Collision Avoidance