## 3 Soft Actor-Critic Based Path Planning for Periodically Moving Obstacles
### Soft Actor-Critic (SAC) Based Path Planning Considering Periodically Moving Obstacles  
Since our \(\text{MAMMDP}_{\text{m0}}\) has a continuous action value, there are several reinforcement learning methods for continuous action MDP such as those in [32; 36; 37]. Soft actor-critic (SAC), especially, is the best known for its performance for the MDP with the continuous action and high-dimensional state. SAC is based on maximum entropy reinforcement learning that maximizes not only the expected sum of rewards but also the entropy of the policy of action [36; 38]. When the policy \(\pi\) is updated by maximizing only the total reward, the center of the distribution of the policy is formed around a specific action. However, by maximizing policy entropy together with the reward, the SAC method can make the distribution spread more and encourage the agent to explore a wider range of the actions. The soft state-value function and soft \(Q\)-value function are defined as follows:  
\[V(s_{t})=\mathbb{E}_{a_{t}\sim\pi}[Q(s_{t},a_{t})-\beta\log\pi(a_{t}|s_{t})], \tag{4}\]  
\[Q(s_{t},a_{t})=r(s_{t},a_{t})+\gamma\mathbb{E}_{s_{t+1}\sim p}[V(s_{t+1})], \tag{5}\]  
where \(\beta\) denotes the temperature parameter of the entropy function. The soft state value function \(V(s_{t})\) defined in (4) is the expected sum of the entropy augmented reward for the given current state \(s_{t}\). In addition, the soft \(Q\)-value function or soft action-value function \(Q(s_{t},a_{t})\) are defined in (5) and is the expected sum of augmented reward for a given pair of state \(s_{t}\) and action \(a_{t}\). For the purpose of estimating the soft value and \(Q\)-value functions, and finding the optimal policy, the SAC-based algorithm consists of five neural networks: two for the soft value function, two for the soft \(Q\)-functions, and one for the policy. The proposed SAC-based path planning utilizes the structure described in Figure 2. The structure is quite similar to that in [20] except for the information of the moving obstacle. As the proposed path planning has to take the periodically moving obstacle into account, it has to be decided properly how to use the obstacle information \(D_{t}\) for the path planning. Since information \(D_{t}\) has to be considered when the policy is trained at a given state in order for the manipulator to enable to avoid the obstacle, information \(D_{t}\) is also used as the input to the network for the policy. In addition, since the policy network is affected by the value functions, as a result, all the neural networks for the proposed SAC-based path planning have to have information \(D_{t}\) as an input. Figure 2: Architecture of the proposed SAC-based with the HER path planning algorithm. Figure 3 illustrates how the deep neural network (DNN) estimates the state value function. The DNNs N1 and N2 are parameterized by \(\psi\) and \(\bar{\psi}\). The network parameterized by \(\bar{\psi}\) is the target network which makes the training stable and enhances the learning performance of DNNs N3 and N4 for estimating the \(Q\)-value function. The inputs to DNN N1 are the configuration \(q_{t}\), goal state \(q_{\text{goal}}\), and \(D_{t}\). In order to train DNN N1, the following objective function and its gradient are used:  
\[J_{V}(\psi)=\mathbb{E}_{s_{t}}[\frac{1}{2}(V_{\bar{\psi}}(s_{t})-\mathbb{E}_{ a_{t}}[\min_{k=1,2}Q_{\theta_{k}}(s_{t},a_{t})-\beta\log\pi_{\phi}(a_{t}|s_{t}) ])^{2}], \tag{6}\]  
where the minimum of the soft \(Q\)-function is determined between \(Q_{\theta_{1}}\) and \(Q_{\theta_{2}}\), which are the output of the DNN N3 and N4. It is reported that the use of such a minimum makes it possible to avoid overestimation of \(Q\)-value [37]. The parameterized policy \(\pi_{\phi}\) comes from DNN N5. To find the optimal parameter \(\psi\) of DNN N1, the network N1 for the soft state value is trained by minimizing \(J_{V}(\psi)\) using \(\
abla J_{V}\).