## 3 Soft Actor-Critic Based Path Planning for Periodically Moving Obstacles
### Soft Actor-Critic (SAC) Based Path Planning Considering Periodically Moving Obstacles
```
1:Define MAMMDP and the current state \(s_{t}=(q_{t}||D_{t})\) and the initial and goal point \(q_{\text{ini}}\) and \(q_{\text{goal}}\)
2:Initialize network parameters \(\psi,\theta_{1,2},\phi\)
3:Initialize the parameter values of the target network \(\bar{\psi}\leftarrow\psi\)
4:Initialize global replay memory \(\mathcal{D}\)
5:for\(e=1\) to\(M\)do
6: Initialize local buffer \(\mathcal{L}\)\(\triangleright\) Memory for an episode
7:for\(t=0\) to\(T-1\)do
8: Randomly choose obstacle position \(D_{t}\) and the goal and initial positions \(q_{\text{goal}},q_{\text{ini}}\in\mathbf{Q}_{\text{free}}^{a}\)
9:\(a_{t}=f_{\emptyset}(e_{t},s_{t}||q_{\text{goal}}),s_{t}=(q_{t}||D_{t}),\;e_{t} \sim\mathcal{N}(0,\sigma_{t})\)
10:\(\hat{q}_{t+1}=q_{t}+\alpha\cdot a_{t}+e_{e},\;\;e_{e}\sim\mathcal{N}(0,\sigma_ {e})\)
11:if\(\hat{q}_{t+1}\in\mathbf{Q}_{\text{free}}^{a}\)then\(\triangleright\) Get next state and reward
12:\(q_{t+1}\leftarrow\hat{q}_{t+1}\)
13:\(r_{t+1}=-1\)
14:elseif\(\hat{q}_{t+1}\in\mathbf{Q}_{\text{free}}^{a}\)then
15:\(q_{t+1}\gets q_{t}\)
16:\(r_{t+1}=-1\)
17:elseif\(|q_{t+1}-q_{\text{goal}}|\leq\eta\cdot\alpha\)then
18:\(r_{t+1}=0\)
19: Terminate due to goal arrival
20:endif
21: Store the transition \((s_{t}||q_{\text{goal}},a_{t},r_{t+1},s_{t+1}||q_{\text{goal}})\) in \(\mathcal{D},\mathcal{L}\)\(\triangleright\) Parameters update
22: Sample mini-batch of \(m\) transitions \((s_{t}||q_{\text{goal}},a_{t},r_{t+1},s_{t+1}||q_{\text{goal}})\) from \(\mathcal{D}\)
23:\(J_{V}(\psi)=\mathbb{E}_{s_{t}}[\frac{1}{2}(V_{\psi}(s_{t}||q_{\text{goal}})- \mathbb{E}_{a_{t}}[\min_{k=1,2}Q_{\theta_{k}}(s_{t}||q_{\text{goal}},a_{t})- \beta\log\pi_{\phi}(a_{t}|\;s_{t}||q_{\text{goal}}))]^{2}]\)
24:\(J_{Q}(\theta_{k=1,2})=\mathbb{E}_{s_{t},a_{t}}[\frac{1}{2}(Q_{\theta_{k=1,2}}(s_ {t}||q_{\text{goal}},a_{t})-(r_{t+1}+V_{\psi}(s_{t+1}||q_{\text{goal}})))^{2}]\)
25:\(J_{\pi}(\phi)=\mathbb{E}_{s_{t},a_{t}}[\beta\log\pi_{\phi}(a_{t}|s_{t}||q_{ \text{goal}})-\min_{k=1,2}Q_{\theta_{k}}(s_{t}||q_{\text{goal}},a_{t})]\)
26: Each network parameter \(\psi,\theta_{1,2},\phi\) is updated by gradient descent
27: using \(\
abla_{\psi}J_{V}(\psi),\
abla_{\theta_{1}}J_{Q}(\theta_{1}),\
abla_{\theta_ {2}}J_{Q}(\theta_{2}),\
abla_{\phi}J_{\pi}(\phi)\)
28: Update state value target \(\bar{\psi}-\tau\psi+(1-\tau)\bar{\psi}\)
29:endfor
30:if\(q_{T}\
eq q_{\text{goal}}\)then\(\triangleright\) HER
31: Set additional goal \(q^{\prime}_{\text{goal}}\in\{q_{1},q_{2},\cdots,q_{\mathcal{T}}\}\)
32:for\(t=0\) to\(T-1\)do
33: Sample a transition \((s_{t}||q_{\text{goal}},a_{t},r_{t},s_{t+1}||q_{\text{goal}})\) from \(\mathcal{L}\)
34:if\(|q_{t+1}-q^{\prime}_{\text{goal}}|\leq\eta\cdot\alpha\)then
35:\(r^{\prime}_{t+1}=0\)
36:else\(r^{\prime}_{t+1}=-1\)
37:endif
38: Store the transition \((s_{t}||q^{\prime}_{\text{goal}},a_{t},r^{\prime}_{t+1},s_{t+1}||q^{\prime}_{ \text{goal}})\) in \(\mathcal{D}\)
39:endfor
40:endif
41:endfor
```  
**Algorithm 1** Proposed SAC-based path planning algorithm for multi-arm manipulators.