## 3 Soft Actor-Critic Based Path Planning for Moving Obstacles with LSTM
### Multi-Arm Manipulator MDP with Position Estimation of Moving Obstacles via LSTM  
Since the path planning algorithm of the multi-arm manipulator using LSTM (MAMMDP\(\mathbf{m}_{\text{O}}\)) proposed in this paper uses deep reinforcement learning, it is necessary to define the tuple \(\{S,A,P,r,\gamma\}\) for the multi-arm manipulator MDP. Figure 1 describes the structure of MAMMDP\(\mathbf{m}_{\text{O}}\). The MAMMDP\(\mathbf{m}_{\text{O}}\) proposed in this paper is similar to the MAMMDP of [22]. The difference is that in [22], information about the obstacles (i.e., past and current positions) is obtained directly from the environment and it is a part of the state. On the other hand, in the proposed method, the agent separately predicts the future position of moving obstacles using the trained LSTM with past and current positions and uses the predicted position for SAC-based path planning. To this end, the LSTM is trained using the measured position data of the moving obstacle in advance considering the work environment and moving obstacles before the  
Figure 1: Multi -Arm Manipulator MDP with Position Estimation of Moving Obstacles via LSTM (MAMMDP\(\mathbf{m}_{\text{O}}\)). training of the SAC-based path planning. In practice, position information can be obtained through various sensors (e.g., cameras). At every sampling time, the position data of the obstacle are acquired and stored using machine vision technology. For the training of the LSTM, the input of the LSTM is the obstacle position over the past \(n_{p}\) sampling times, and the output of the LSTM is the prediction of the obstacle position for the future \(n_{f}\) sampling times. \(n_{p}\) denotes the number of time steps for the obstacle's past position information, and \(n_{f}\) indicates how far the LSTM predicts the position of the obstacle after the current time point. As \(n_{f}\) increases, the LSTM's output dimension, i.e., the agent's state dimension becomes large, which can make the training of SAC-based path planning computationally demanding. Hence, an appropriate tradeoff is required. Figure 2 shows the I/O (input/output) structure of the LSTM, and the I/O variables of the LSTM are defined as follows. \[D_{t}=\left[ \begin{array}{c}d_{t}\\ d_{t-1}\\ \vdots\\ d_{t-(n_{p}-1)}\end{array} \right]\in\mathbb{R}^{3\cdot n_{p}},\ \ \hat{D}_{t}=\left[ \begin{array}{c}\hat{d}_{t+1}\\ \hat{d}_{t+2}\\ \vdots\\ \hat{d}_{t+(n_{f})}\end{array} \right]\in\mathbb{R}^{3\cdot n_{f}},\ D_{target,t}= \left[ \begin{array}{c}d_{t+1}\\ d_{t+2}\\ \vdots\\ d_{t+(n_{f})}\end{array} \right]\in\mathbb{R}^{3\cdot n_{f}}\]  
\[d_{t}=\left[ \begin{array}{c}x_{t}^{d}\\ y_{t}^{d}\\ z_{t}^{d}\end{array} \right]\in\mathbb{R}^{3},\ \ \hat{d}_{t}=\left[ \begin{array}{c}\hat{x}_{t}^{d}\\ \hat{y}_{t}^{d}\\ \hat{z}_{t}^{d}\end{array} \right]\in\mathbb{R}^{3},\]  
where \(d_{t}=\left[ \begin{array}{cc}x_{t}^{d}&y_{t}^{d}&z_{t}^{d}\end{array} \right]^ {T}\) and \(\hat{d}_{t}=\left[ \begin{array}{cc}\hat{x}_{t}^{d}&\hat{y}_{t}^{d}&z_{t}^{d} \end{array} \right]^{T}\) denote the position of the obstacle in the workspace and its prediction at the sampling time \(t\), respectively. Therefore, the state at sampling time \(t\) is defined as \(s_{t}=(q_{t}\cup q_{goal}\cup\hat{D}_{t})\). In other words, for accurate path planning, the state includes not only the current joint information of the multi-arm manipulator but also the prediction of the future position of the obstacle using \(\hat{D}_{t}\). The LSTM is trained using the following cost function  
\[Loss=\frac{1}{N}\Sigma_{t=1}^{N}(D_{target,t}-\hat{D}_{t})^{2}, \tag{2}\]  
where \(D_{target,t}\) denotes the true position of the moving obstacle, and \(\hat{D}_{t}\) is the output of the LSTM, meaning the prediction of the position of the moving obstacle. As a matter of fact, there are two possibilities to obtain the current position \(d_{t}\) of the moving obstacle for the LSTM training. The first is that the position of the moving obstacle can be obtained as a time series using a sensor such as a camera.