## 3 Soft Actor-Critic Based Path Planning for Moving Obstacles with LSTM
### Multi-Arm Manipulator MDP with Position Estimation of Moving Obstacles via LSTM
To be specific, after running only the moving obstacle without robot manipulators, the camera gets the position information and saves them in a buffer. Then, the LSTM can be trained using the time series. The second is that the position \(d_{t}\) can be made using simulation. If the trajectory followed by the moving obstacle is known, that information can be used as the time series for the LSTM training. To obtain the training data of the LSTM in this paper, the robot arm does not move in the simulation, but only the obstacle moves within the workspace. The SAC-based path planning agent receives the position of the obstacle from the environment at every step. The  
Figure 2: Architecture of LSTM network for position prediction. received obstacle position data are stored in a buffer and the training data \(D_{t}\) and \(D_{target,t}\) is generated from the buffer. The action is defined as \(a_{t}=f_{\theta}(\epsilon_{t},s_{t})\). Here, \(\epsilon_{t}\) is an added noise in order to model the uncertainty of the environment and assumed to follow the Gaussian distribution \(\mathcal{N}(0,\sigma_{t})\) where \(\sigma_{t}\) represents variance. \(f_{\theta}(\epsilon_{t},s_{t})\) denotes the output of the policy network when the input to the policy network is \(s_{t}\). Consequently, \(a_{t}\) can be viewed as a sampled value from a Gaussian distribution. Then, the state at the next time step is calculated using the current state value and the relation \(\hat{q}_{t+1}=q_{t}+\alpha a_{t}+\epsilon_{e}\), where the tuning parameter \(\alpha\) is defined as the geometric mean of the maximum variation of each joint and \(\epsilon_{e}\) is the environmental noise following \(\epsilon_{e}\sim\mathcal{N}(0,\sigma_{e})\). If the next state value causes a collision after applying the action, that is, if it is \(\hat{q}_{t+1}\in\mathbf{Q}_{\text{sub}}^{a}\), the state does not move and maintains the current state value. The following summarizes how the joint values are updated during path planning. \[q_{t+1}= \begin{cases}\hat{q}_{t+1},&\text{if }\hat{q}_{t+1}\in\mathbf{Q}_{ \text{inv}}^{a},\\ q_{t},&\text{if }\hat{q}_{t+1}\
otin\mathbf{Q}_{\text{inv}}^{a}.\end{cases} \]  
Using this, the next state is expressed as \(s_{t+1}=(q_{t+1}\cup\hat{D}_{t+1})\), where \(\hat{D}_{t+1}\) is the prediction of the obstacle position by the LSTM after receiving \(q_{t+1}\) from the environment. Until the joint value included in the next state reaches goal point \(q_{\text{goal}}\), the process of applying the action according to the policy and obtaining the state and reward is repeated. The final goal is given by \(q_{t+1}=q_{\text{goal}}\). However, this is very difficult due to practical limitations such as numerical errors. Therefore, using the norm of the vector, the final goal is changed as \(\|q_{t+1}-q_{\text{goal}}\|\leq\eta\cdot\alpha\). Here, \(\|\cdot\|\) means the norm of the vector, and \(\alpha\) is the maximum action step such that the manipulator does not move much at once. \(\eta\) is a tuning variable that satisfies \(0<\eta<1\). That is, the goal of learning is set to arrive near the goal point, rather than finding the perfect goal point such as \(q_{t+1}=q_{\text{goal}}\). The following is a reward function \(r(s_{t},a_{t})\) used for learning. \[r_{t+1}= \begin{cases}0,&\text{if }|q_{t+1}-q_{\text{goal}}|\leq\eta\cdot \alpha\\ -1,&\text{if }q_{t+1}\in\mathbf{Q}_{\text{inv}}^{a}\\ -3,&\text{if }q_{t+1}\in\mathbf{Q}_{\text{sub}}^{a}\end{cases}  \tag{3}\]  
This gives a greater penalty to collisions with obstacles, helping the agent learn to avoid collisions as much as possible. Therefore, if the agent does not reach the goal point by the maximum number of iterations \(T\), then the minimum total reward received is \(-3\times T\). However, if it arrives at the goal point in a specific iteration \(T_{done}<T\), the reward is better than that. In the next section, we present a method to update the policy during training using a deep neural network.