## 4 Case Study: Two 3 DOF Manipulators with a Periodically Moving Obstacle on a Line
### Simulation  
For the implementation of the proposed SAC-based path planning algorithm, two 3 DOF open-manipulators are used. The detailed information about the manipulator can be seen at [[http://en.robotis.com/model/page.php?co_id=prd_openmanipulator](http://en.robotis.com/model/page.php?co_id=prd_openmanipulator)]([http://en.robotis.com/model/page.php?co_id=prd_openmanipulator](http://en.robotis.com/model/page.php?co_id=prd_openmanipulator)) (accessed on 14 March 2021). The entire workspace is 70 cm \(\times\) 120 cm \(\times\) 100 cm. For the moving obstacle, a hexagon is assumed to move along a line segment back and forth periodically in the workspace, the length of the line segment is 40 cm, and the average speed of the moving obstacle is 9 cm/sec. In the simulation, it is always possible for the agent to know the location information of the obstacle. For the training, \(n_{d}\) is set to 5 meaning that the location of the obstacle over the past five sampling times are given to the networks. Moreover, a square prism is located in the workspace as a static obstacle. The two manipulators are located such that the intersection of the workspace of both is not empty. The parameters of the robot manipulators and environment are summarized in Table 1.  
Figure 6 show the workspace under consideration. In Figure 6a, obstacle 1 moves back and forth along the line segment defined by \(D_{\text{\tiny{eb}}}=\{(x,y,z)|x=150\text{ mm},-25\text{ mm}\leq y\leq 375\text{ mm},\text{ and }z=300\text{ mm}\}\). The obstacle 2 depicts the static obstacle. It is assumed that the manipulator 1 can detect the obstacle only when it is in \(\{x=150\text{ mm},0\text{ mm}\leq\text{mm}\leq\text{mm}\leq\text{mm}\leq \text{mm}\}\). \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Name** & **Value** & **Notation** & **Unit** \\ \hline The number of manipulator joints & 3 & \(n\) & pc. (Piece) \\ The number of manipulators & 2 & \(m\) & pc. (Piece) \\ Joint maximum & (140, \(-\)45, 150, 140, \(-\)45, 150) & \({}^{\circ}\) (Degree) \\ Joint minimum & (\(-\)140, \(-\)180, 45, \(-\)140, \(-\)180, 45) & \({}^{\circ}\) (Degree) \\ Dimension of \(\mathbf{Q}_{\text{free}}^{n}\) & 6 & \(n\cdot m\) & Dimension \\ The number of dynamic obstacle & 1 & & pc. (Piece) \\ The number of features & 5 & \(n_{d}\) & Feature \\ The dynamic obstacle axis & y & & \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Parameters of the 3 DOF manipulator and environment. \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Name** & **Value** & **Notation** & **Unit** \\ \hline Policy network size & \(12\times 800\times 500\times 400\times 400\times 300\times 6\) & \(\phi\) & Node \\ Soft Q network size & \(18\times 800\times 500\times 400\times 400\times 300\times 1\) & \(\theta_{1,2}\) & Node \\ Soft value network size & \(12\times 800\times 500\times 400\times 400\times 300\times 1\) & \(\psi\), \(\bar{\psi}\) & Node \\ Learning rate & 0.0001 & & \\ Replay memory size & \(10^{6}\) & \(\mathcal{D}\) & Buffer \\ Episode maximum step & 100 & \(T\) & Step \\ Soft value target copy rate & 0.005 & \(\tau\) & \\ Mini batch size & 512 & \(m\) & Batch \\ Environment noise deviation & 0.002 & \(\epsilon_{e}\) & \\ Action step size & 0.3813 & \(\alpha\) & \\ Goal boundary & 0.2 & \(\eta\) & \\ Discount factor & 0.98 & \(\gamma\) & \\ Entropy temperature parameter & 0.2 & \(\beta\) & \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Tuning parameters for the designed SAC with HER. \(y\leq 175\) mm, and \(z=300\) mm\()\), and the manipulator 2 can do only when it is in \(\{x=150\) mm, \(175\) mm\(\leq y\leq 350\) mm, and \(z=300\) mm\(\}\). Namely, the manipulators can detect the moving obstacle only when it is in a predefined area. In this simulation, the location of the moving obstacle is directly extracted as the input of the network. Due to that, the network can detect the obstacle location based on the input value of the network. Figure 5(b) describes how the PRM deals with the moving obstacle.