## 3 TD3 Based Motion Planning for Smoother Paths
### Hindsight Experience Replay  
Since the agent in reinforcement learning is trained from samples, it is utmost important to have helpful samples which mean that the action state pair improve the action value function. On the other hand, many unhelpful samples are generated in RAMDP since many episodes end without reaching the goal. In other words, RAMDP is an MDP with sparse reward. For the purpose of enhancing sample efficiency, HER (Hindsight Experience Replay) is employed in this paper. For the episode \(e=[(q_{0},a_{0}),(q_{1},r_{1},a_{1}),\cdots,(q_{T},r_{T})]\) in the memory where \(q_{T}\) is not the goal state, HER resets \(q_{T}\) as \(q_{\textit{goal}}\). This means that even though the episode does not end the goal state, the episode becomes a ended state at the goal after modification by HER. Then, the failed episodes can be use to train the agent since the modified episodes are a goal achieved episodes. Algorithm 1 summarizes the proposed algorithm introduced in this section.  
Figure 3: Details of critic and actor deep neural network for RAMDP.  
```
1:Initialize critic networks \(Q_{\theta_{1}},Q_{\theta_{2}}\), and actor network \(\pi_{\phi}\) with \(\theta_{1},\theta_{2},\phi\)
2:Initialize target networks \(\theta^{\prime}_{1}\leftarrow\theta_{1},\theta^{\prime}_{2}\leftarrow\theta_{2}, \phi^{\prime}\leftarrow\phi\)
3:Initialize replay buffer \(\mathcal{R}\)
4:for\(e=1\)to\(M\)do
5: Initialize local buffer \(\mathcal{L}\)\(\triangleright\) memory for HER
6:for\(t=0\)to\(T-1\)do
7: Randomly choose goal point \(q_{\text{goal}}\in\mathbf{Q}_{\text{\tiny{init}}}\)
8: Select action with noise:
9:\(a_{t}\sim\mu_{\phi}(q_{t}||q_{\text{goal}})+\epsilon,\ \epsilon\sim\mathcal{N}(0,\sigma)\)\(\triangleright\)\(||\) denotes concatenation
10:\(q_{t+1}=q_{t}+\alpha*a_{t}\)
11:
12:if\(q_{t}+\alpha(\mu_{\phi}(q_{t}||q_{\text{goal}})+\epsilon)\
ot\in\mathbf{Q}\)then
13:\(q_{t+1}\gets q_{t}\)
14:else if\(q_{t+1}\in\mathbf{Q}_{\text{\tiny{init}}}\)then
15:\(q_{t+1}\gets q_{t}\)
16:else if\(|q_{t+1}-q_{\text{goal}}|\leq 0.2*\alpha\)then
17: Terminal by goal arrival
18:endif
19:
20:\(r_{t}:=r(q_{t},a_{t},q_{\text{goal}})\)
21: Store the transition \((q_{t}||q_{\text{goal}},a_{t},r_{t},q_{t+1}||q_{\text{goal}})\) in \(\mathcal{R},\mathcal{L}\)
22:
23: Sample mini-batch of \(n\) transitions \((q_{j}||q_{\text{goal}},a_{j},r_{j},q_{j+1}||q_{\text{goal}})\) from \(\mathcal{R}\)
24:\(\bar{q}_{j}\leftarrow\mu_{\phi^{\prime}}(q_{j+1}||q_{\text{goal}})+\epsilon,\ \epsilon\sim\text{clip}(\mathcal{N}(0,\sigma),\ c,-c)\)
25:\(y_{j}\gets r_{j}+\gamma\min_{i=1,2}\mathcal{Q}_{\theta^{\prime}_{i}}(q_{j+ 1},\bar{q}_{j})\)
26: Update critics \(\theta_{i}\) with temporal difference error:
27:\(\
abla_{\theta_{j}}I(\theta_{i})=\frac{1}{n}\sum_{j=1}^{n}\
abla_{\theta_{i}} (y_{j}-Q_{\theta_{i}}(q_{j},a_{j}))^{2}\)
28:
29:if\(t\)mod\(d\)then\(\triangleright\) delayed update with \(d\)
30: Update actor \(\phi\) by the deterministic policy gradient:
31:\(\
abla_{\phi}I(\phi)=\frac{1}{n}\sum_{j=1}^{n}\
abla_{\phi}\mathcal{Q}_{ \theta_{1}}(q_{j},\mu_{\phi}(q_{j}||q_{\text{goal}}))\)
32: Update target networks:
33:\(\theta^{\prime}_{i}\leftarrow\tau\theta_{i}+(1-\tau)\theta^{\prime}_{i}\)
34:\(\phi^{\prime}\leftarrow\tau\phi+(1-\tau)\phi^{\prime}\)
35:endif
36:endfor
37:if\(q_{T}\
eq q_{\text{goal}}\)then
38: Set additional goal \(q^{\prime}_{\text{goal}}\gets q_{T}\)
39:for\(t=0\)to\(T-1\)do
40: Sample a transition \((q_{t}||q_{\text{goal}},a_{t},r_{t},q_{t+1}||q_{\text{goal}})\) from \(\mathcal{L}\)
41:\(r^{\prime}_{t}:=r(q_{t},a_{t},q^{\prime}_{\text{goal}})\)
42: Store the transition \((q_{t}||q^{\prime}_{\text{goal}},a_{t},r_{t},q_{t+1}||q^{\prime}_{\text{goal}})\) in \(\mathcal{R}\)
43:endfor
44:endif
45:endfor
```  
**Algorithm 1** Training procedure for motion planning by TD3 with HER. The red part is for a robot manipulator, and the blue part is for HER.