## 4 Case Study
### Simulation  
For the simulation-based case study, a moving obstacle is considered such that it follows a zigzag-shaped trajectory along a vertical line in the workspace repeatedly. Figure 5 depicts the considered workspace implemented with GAZEBO. In the GAZEBO simulation, dynamic objects such as obstacles and robot manipulators communicate with each other in the ROS (robot operating system). The moving obstacle is depicted in the form of green boxes, and the gray box indicates the fixed obstacle in Figure 5. Figure 5,b are the initial setups of the simulation environment before running the proposed algorithm. Figure 5,d show the snapshot of the simulation environment when the proposed algorithm is working. For the LSTM training, at first, without running the robot manipulators, by operating the moving obstacle only in GAZEBO, its position data are saved as a time series. Using the \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Name** & **Value** & **Notation** & **Unit** \\ \hline The number of manipulator joints & 3 & \(n\) & pc. (Piece) \\ The number of manipulators & 2 & \(m\) & pc. (Piece) \\ Joint maximum & (140, \(-\)45, 150, 140, \(-\)45, 150) & & \({}^{\circ}\) (Degree) \\ Joint minimum & (\(-\)140, \(-\)180, 45, \(-\)140, \(-\)180, 45) & & \({}^{\circ}\) (Degree) \\ Dimension of \(\mathbf{Q}_{\text{free}}^{t}\) & 6 & \(n\cdot m\) & Dimension \\ The number of dynamic obstacle & 1 & & pc. (Piece) \\ The number of past features & 5 & \(n_{p}\) & Feature \\ The number of future features & 3 & \(n_{f}\) & Feature \\ The dynamic obstacle axis & \(x,\,y,\,z\) & & \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Parameters of the 3-DOF manipulator and environment. Figure 5: The GAZEBO workspace. time series data, the LSTM is trained using (2). Figure 6 demonstrates the loss function of the LSTM to show the training performance. In view of the convergence of the loss function, the LSTM is trained successfully. Using the trained LSTM, which acts as the predictor of the position of the moving obstacle, the proposed SAC-based path planning agent is trained. It is assumed in the simulation-based case study that \(D_{t}\) is obtained from the environment and it is given to the trained LSTM in order to compute \(\hat{D}_{t}\). The five neural networks for the proposed SAC-based path planning in Figure 3 are trained through 100,000 episodes using a Nvidia GeForce RTX 3080 Ti GPU and AMD Ryzen 5 5600X CPU. The networks are implemented using Tensorflow 2.5. It takes 19 h for the training. In addition, for 100,000 episodes, random starting and goal points are used at the beginning of each episode. Figure 7 shows the success rate of path creation in the course of learning 100,000 episodes. Success rates are calculated for every 30 episodes. It can be seen that the success rate converges to about 95% as the learning progress is going on until 100,000 episodes. For the evaluation of training performance, not only the success rate but also the reward matters. Figure 8 shows the reward while learning. One of the goals of this paper is to compute the shortest route. As the training goes on, the SAC agent finds shorter paths due to the definition of the reward function in (3). This is why the reward gets better as the training goes on in Figure 8.  
Figure 6: Loss function of simulation LSTM training. Figure 7: Simulation: success ratio of the proposed path planning in training. In addition to the faster training speed in Figure 9, the reward also converges more quickly, as shown in Figure 10.  
Figure 11: Experiment: success ratio of the proposed path planning in training. Figure 12: Experiment: reward of the proposed path planning in training. Figure 10: Comparison of reward. Since the SAC-based path planning is trained, its testing result can be obtained for arbitrary starting and goal points according to the procedure in Figure 4. The testing results are demonstrated on the web page [[https://url.kr/4m562t](https://url.kr/4m562t)]([https://url.kr/4m562t](https://url.kr/4m562t)) (accessed on 25 September 2022).