## 4 Case Study: Two 3 DOF Manipulators with a Periodically Moving Obstacle on a Line
### Simulation
As seen in Figure 5(b), the PRM views entire \(D_{\omega}\) as a static obstacle to deal with the moving obstacle. In the learning process, the five neural networks in the proposed SAC-based path planning are trained by 70,000 episodes using GPU Nvidia GeForce RTX 2080 Ti and CPU Intel i7-9700F ([[https://www.nvidia.com](https://www.nvidia.com)]([https://www.nvidia.com](https://www.nvidia.com)), [[https://www.intel.com](https://www.intel.com)]([https://www.intel.com](https://www.intel.com)) (accessed on 14 March 2021). For the training, it took 36 h and the proposed algorithm was implemented using Tensorflow 2.1. In each episode, the agent explores and learns the environment based on the previous experience that was stored in the memory. To visualize the success of the agent, the success ratio from every 10 episodes is calculated and recorded. Figure 7 depicts the success ratio from 70,000 episodes. In Figure 7, the green chart describes the actual success ratio. However, for better visualization, the moving average of the success ratio is calculated and shown as the dark green line. At the end of the training, the success ratio reaches 81.76 %. Figure 6: The Matlab workspace. Figure 7: Success ratio of the proposed method for two open manipulators with moving obstacles. Not only the success ratio but also the reward value is monitored during the training. Because the agent receives \(-1\) for the reward at every step if the next configuration is not the goal, the total reward value can be interpreted as the required total time step for the agent to complete the task. The total reward of every episode is shown in Figure 8. The light blue line tells the total reward in each episode and the dark blue line describes the moving average of the total reward. From the repeated learning, similar convergent results on the success ratio and total reward are obtained. In view of Figures 7 and 8, it can be concluded that the agent is well trained, which means the agent can find the optimal path for arbitrary starting and goal configurations. In view of the learning results, the success rate of the training is not satisfactory if \(n_{d}\) is set to a value less than 5. On the other hand, when it is higher than 5, the success rates of most learning reach a similar level to that with \(n_{d}=5\). This is why \(n_{d}\) is set to 5. To support this, Figure 9 presents the learning performance depending on various values of \(n_{d}\). Figure 8: Reward of the proposed SAC-based path planning for two open manipulators with moving obstacles. Figure 9: Comparison of the training performance between different numbers of features. When the training is over, the actor-network is used to generate the optimal path for an arbitrary starting and goal configurations. Figure 10 shows an example of initial and goal configurations. Figure 11 shows how the trained actor generates the path by interacting MAMMDPmo and measuring the location of the moving obstacle. To be specific, \((q_{\text{goal}},q_{0},D_{0})\) is given to the trained actor-network in the beginning and the actor-network calculates the variations of each joint denoted by \(f_{\phi}(s_{t}||q_{\text{goal}})\) in Figure 11. Then, each joint changes its angle by the variations and the resulting new configuration \(q_{1}\) is generated. This is one iteration. In the second iteration, \((q_{\text{goal}},q_{1},D_{1})\) is injected into the trained actor-network with the updated obstacle information \(D_{1}\), and the goal configuration is reached by repeating the same procedure. One testing result is shown in Figure 12. In Figure 12, the areas in dark green denote \(\mathbf{Q}_{\text{goal}}\). In this result, it is important to note that the path generated by the proposed SCA-based planning can penetrate \(D_{\text{safe}}\) without collision if necessary. On the other hand, the paths by other methods go around \(D_{\text{safe}}\). This shows that the proposed method generates indeed an efficient path when there is a moving obstacle.