## 4 Case Study for 2-DOF and 3-DOF Manipulators  
In order to show the effectiveness of the proposed method, it is applied to robot manipulators. Table 1 shows the information of the used 2-DOF and 3-DOF and manipulators. For easy visualization, the proposed algorithm is applied to the 2-DOF manipulator first. Table 2 summarizes the tuning parameters for the designed TD3 with HER. In order to train TD3 with HER for the 2-DOF manipulator, 8100 episodes are used. Figure 4 describes the success ratio of each episode with HER when the network is training. In other words, when the network is learning with arbitrary starting and goal points, sometimes the episodes end at the given goal point but sometimes the episodes end before reaching the given goal point. In Figure 4, the green lines denote the success ratio of every 10 episodes and the thick lines stand for the moving average of the gray  
When this is applied to the 2-DOF manipulator, the resulting paths are shown in Figure 7. In Figure 7, the green areas represent obstacles in the configuration space, and the rhombus denote the starting point and the circles means the goal point. For comparison, PRM is also applied to generate the paths with the same starting and goal points. As shown in Figure 7, the proposed method generates smoother paths in general. This is confirmed from many other testing data as well. In Figure 7, the red lines are the resulting paths by PRM and the blue lines by the proposed method. In average, the resulting path by the proposed method is shortened by 3.45% compared with the path by PRM. In order to test the proposed method for a real robot manipulator, the 3-DOF open manipulator. For details, see [[http://en.robotis.com/model/page.php?co_id=prd_openmanipulator](http://en.robotis.com/model/page.php?co_id=prd_openmanipulator)]([http://en.robotis.com/model/page.php?co_id=prd_openmanipulator](http://en.robotis.com/model/page.php?co_id=prd_openmanipulator)) is considered. For easy understanding, Figure 8 shows the workspaces in Matlab and Gazebo in ROS (Robot Operating System) respectively, and configuration space of the open manipulator with four arbitrary obstacles. The tuning parameters for the TD3 with HER are also shown in Table 2.  
Figure 5: 2-DOF manipulator: reward from learning. Figure 6: Path generation using the trained actor [PERSON]. For training, 140000 episodes are used. In view of the converged reward and success ratio in Figures 9 and 10, we can see that the learning is also over successfully. With this result, random starting and goal points are given to the trained network in order to obtain a feasible path between them. Figure 11 shows several examples of generated paths by the trained actor [PERSON] when arbitrary starting points and goal points are given. The red lines are resulting paths by PRM and the blue lines by the proposed method. As seen in the figure, the proposed method results in smoother and shorter paths in general. For the sake of between comparison, 100 arbitrary starting and gold points are used to generate paths using PRM and the proposed method. Figure 12 shows the lengths of the resulting 100 paths. In light of Figure 12, it is obvious that the proposed method generates smoother and shorter paths in general. Note that, in average, the resulting path by the proposed method is shortened by 2.69% compared with the path by PRM. When the proposed method is also implemented to the open manipulator, the same experimental result as the simulation was obtained. The experimental result is presented in the [[https://sites.google.com/site/cdslweb/publication/td2-demo](https://sites.google.com/site/cdslweb/publication/td2-demo)]([https://sites.google.com/site/cdslweb/publication/td2-demo](https://sites.google.com/site/cdslweb/publication/td2-demo)). Figure 8: Workspace and configuration space. (**a**) Workspace in Matlab; (**b**) Workspace in Gazebo (in ROS); (**c**) Configuration space. Figure 7: DDPG based path generation for arbitrary starting and goal points in C-space. Figure 11: Cont.