## 3 Soft Actor-Critic Based Path Planning for Moving Obstacles with LSTM
### Soft Actor-Critic (SAC)-Based Path Planning with Prediction of Moving Obstacle Position via LSTM  
This section briefly reviews how the SAC is used for the proposed SAC-based path planning. The proposed \(\text{MAMMDP}_{\text{mo}}\) has a continuous action space. There are many algorithms for MDP with continuous action in reinforcement learning [38; 41; 46; 53]. This paper employs the SAC, which is known to have good performance for high dimensional MDP with continuous action such as robotics. While other algorithms usually maximize only the sum of the expected rewards, SAC maximizes the sum of the value function and entropy. If the existing algorithms are updated in a way that maximizes the sum of the expected rewards, the policy \(\pi\) is made such that a specific action is increased. However, considering the policy entropy in the SAC helps the agent to explore a wide range of various actions. In the SAC, soft state value function \(V\) and soft action value function \(Q\) are defined as follows. \[V(s_{t})=E_{a_{t}}[Q(s_{t},a_{t})-\beta log\pi(a_{t}|s_{t})], \tag{4}\]\[Q(s_{t},a_{t})=r(s_{t},a_{t})+\gamma E_{s_{t+1}}[V(s_{t+1})], \tag{5}\]  
where \(\beta\) means a temperature parameter that controls the weight for \(Q\) function and the entropy term \(log\pi(a_{t}|s_{t})\). The Equation (4) is called a soft state function used in the SAC. Since the entropy term needs to be maximized, the gradient ascent should be used. So the sign is negative. The Equation (5) is a soft state-action function used in SAC. If the action is assumed to be finite, repeatedly executing the soft Bellman equation, Equation (5), with a given policy leads to a soft Q-function. In order to estimate the functions, the SAC uses neural networks. To create training data for the SAC networks, \(\hat{D}_{t}\) is obtained using the trained LSTM network using \(D_{t}\) as an input. Using this, we created a state \(s_{t}\), obtained \(a_{t}\), and then interacted with the environment to store training data \(\{s_{t},a_{t},s_{t+1},r_{t+1}\}\) in replay buffer at each step. Specifically, the SAC utilizes five neural networks to estimate the policy, state value, and state-action value functions; see Figure 3.  
In order to estimate the state value function \(V\), in the proposed algorithm, the value network \(V_{\psi}\) and its target network \(V_{\psi^{-}}\) are used. The target network \(V_{\psi^{-}}\) is used to improve training performance for the \(Q\)-network. The subscripts \(\psi\) and \(\psi^{-}\) indicate the parameters of the neural networks that approximate the functions. The cost function (6) is used to update the parameters \(\psi\) of the neural network for the value function. \[J_{V}(\psi)=E_{s_{t}}[\frac{1}{2}(V_{\psi}(s_{t})-[min_{k=1,2}Q_{\phi_{k}}(s_{ t},a_{t})-\beta log\pi_{\theta}(a_{t}|s_{t}))^{2}] \tag{6}\]  
To prevent sudden change, the target network is used. Moreover, the target network has to change smoothly. Hence, parameter update is performed based on (7)  
\[\psi^{-}=\tau\psi+(1-\tau)\psi^{-} \tag{7}\]  
where the parameter \(\tau\in[0,1]\) controls the learning speed. Two neural networks \(Q_{\phi_{1}}\) and \(Q_{\phi_{2}}\) approximate the \(Q\)-function and the minimum value of the two values is selected for learning. This helps to prevent \(Q\) values from being inappropriately overestimated [53]. The cost function (8) is minimized to update the parameters of \(Q_{\phi_{1}}\) and \(Q_{\phi_{2}}\). \[J_{Q}(\phi_{k=1,2})=E_{(s_{t},a_{t})}[\frac{1}{2}(Q_{\phi_{k=1,2}}(s_{t},a_{t} )-(r(s_{t},a_{t})+\gamma[V_{\psi^{-}}(s_{t+1})]))^{2}] \tag{8}\]  
Figure 3: Architecture of the proposed SAC-based path planning algorithm with HER. Similarly, the neural network parameterized by \(\theta\) approximates the policy by minimizing the cost function (9).