## 3 Soft Actor-Critic Based Path Planning for Moving Obstacles with LSTM
### Soft Actor-Critic (SAC)-Based Path Planning with Prediction of Moving Obstacle Position via LSTM
```
1:for\(e=1\)to\(M\)do\(\triangleright\) Train LSTM network
2: Initialize local buffer \(N\) and 5-size buffer \(D_{t}\)
3:\(d_{t}\), \(D_{target,t}\) = env.step()\(\triangleright\) Collect the position data
4: Stack \(d_{t}\) in \(D_{t}\)
5:\(\hat{D}_{t}\)= LSTM(\(D_{t}\))
6: Store (\(\hat{D}_{t}\),\(D_{target,t}\)) in \(N\)
7: Sample mini-batch of m (\(\hat{D}_{t}\),\(D_{target,t}\)) from \(N\)
8:\(J=\frac{1}{m}\sum_{i=1}^{m}(\hat{D}_{i}-D_{target,i})^{2}\)
9: LSTM network is updated by gradient descent using \(\
abla J\)
10:endfor
11:
12: Define \(\text{MAMMDP}_{\text{mo}}\) and the current state \(s_{t}=(q_{t}\cup q_{\text{goal}}\cup D_{t})\) and \(q_{\text{goal}}\)
13: Initialize network parameters \(\psi,\phi_{1,2},\theta\)
14: Initialize the parameter values of the target network \(\tilde{\psi}\leftarrow\psi\)
15: Initialize global replay memory \(\mathcal{K}\)
16: Initialize environment and get \(D_{t}\)
17:
18:for\(e=1\)to\(M\)do
19: Initialize local buffer \(\mathcal{L}\)\(\triangleright\) Memory for an episode
20:for\(t=0\)to\(T-1\)do
21: Get future position of obstacle with LSTM and the goal and initial positions \(q_{\text{goal}}\), \(q_{\text{goal}}\), \(q_{\text{goal}}\)\(\in\)\(\mathbf{Q}_{\text{mo}}^{a}\)
22:\(\hat{D}_{t}\)= LSTM(\(D_{t}\)), \(s_{t}=(q_{t}\cup q_{\text{goal}}\cup\hat{D}_{t})\), \(a_{t}=f_{\phi}(e_{t},s_{t})\)\(e_{t}\sim\mathcal{N}(0,\sigma_{t})\)
23:\(q_{t+1},d_{t+1},r_{t+1}\) = env.step(\(a_{t}\))
24: Stack \(d_{t+1}\) in \(D_{t+1}\)
25:\(\hat{D}_{t+1}\)= LSTM(\(D_{t+1}\))
26:\(s_{t+1}=(q_{t+1}\cup q_{\text{goal}}\cup\hat{D}_{t+1})\)
27:
28: Store the transition \((s_{t},a_{t},r_{t+1},s_{t+1})\) in \(\mathcal{K}\), \(\mathcal{L}\)
29:
30: Sample mini-batch of \(m\) transitions \((s_{t},a_{t},r_{t+1},s_{t+1})\) from \(\mathcal{K}\)
31:\(I_{V}(\psi)=\mathbb{E}_{s_{1}}[\frac{1}{2}(V_{\psi}(s_{1})-\mathbb{E}_{a_{1}}[ \min_{k=1,2}Q_{\phi_{k}}(s_{t},a_{t})-\beta\log\pi_{\theta}(a_{t}|s_{t})])^{2}]\)
32:\(I_{Q}(\phi_{k=1,2})=\mathbb{E}_{s_{t},a_{t}}[\frac{1}{2}(Q_{\phi_{k-1,2}}(s_{t },a_{t})-(r_{t+1}+V_{\tilde{\psi}}(s_{t+1})))^{2}]\)
33:\(J_{\pi}(\theta)=\mathbb{E}_{s_{t},a_{t}}[\beta\log\pi_{\theta}(a_{t}|s_{t})- \min_{k=1,2}Q_{\phi_{k}}(s_{t},a_{t})]\)
34:
35: Each network parameters \(\psi,\theta_{1,2},\phi\) are updated by gradient descent
36: using \(\
abla_{\psi}J_{V}(\psi),\
abla_{\phi_{t}}I_{Q}(\phi_{1}),\
abla_{\phi_{t}}I_ {Q}(\phi_{2}),\
abla_{\theta}J_{\pi}(\theta)\)
37: Update state value target \(\tilde{\psi}\leftarrow\tau\psi+(1-\tau)\tilde{\psi}\)
39:endfor
40:
41:if\(q_{T}\
eq q_{\text{goal}}\)then\(\triangleright\) HER
42:for\(t=0\)to\(T-1\)do
43: Sample a transition \((s_{t},a_{t},r_{t+1},s_{t+1})\) from \(\mathcal{L}\)
44: Change goal point \(q^{\prime}_{\text{goal}}=q_{t+1}\)
45:if\(q_{t+1}\in Q^{a}_{\text{collide}}\)then
46: continue
47:elseif\(|q_{t+1}-q^{\prime}_{\text{goal}}|\leq\eta\cdot\alpha\)then
48:\(r^{\prime}_{t+1}=0\)
49:endif
50: Store the transition \((s_{t},a_{t},r^{\prime}_{t+1},s_{t+1})\) in \(\mathcal{K}\)
51:endfor
52:endif
53:endfor
```  
**Algorithm 1** Proposed SAC-based path planning algorithm for multi-arm manipulators.