## 3 TD3 Based Motion Planning for Smoother Paths  
### RAMDP (Robot Arm Markov Decision Process) for Path Planning  
In order to develop a reinforcement learning (RL) based path planning, the robot arm MDP (RAMDP) needs to be defined properly first [17]. The state \(q_{t}\) of the RAMDP is the angle value of each joint of the manipulator where the joint angle belongs to the configuration space \(\mathbf{Q}\). Hence, for collision-free operation of a robot manipulator, in the motion planning, the state \(q_{t}\) belongs to only \(\mathbf{Q}_{\mathit{mw}}\in\mathbb{R}^{n}\) where \(n\) is the number of the joint in the manipulator. In the RAMDP, the action is joint angle variation. Unlike MDP with discrete state and action such as frozen-lake ([17]), the RAMDP under consideration has continuous state (i.e., joint angle) and continuous action (i.e., joint angle variation). Due to this, DDPG or its variants are fit to find the optimal policy for the agent in the RAMDP. In this paper, TD3 (twin delayed DDPG) is employed to find an optimal policy for the continuous action with deterministic policy \(\mu\) in the RAMDP. Figure 1 describes how to generate the sample \((q_{t},\ a_{t},\ q_{t+1},\ r_{t+1})\) in the RAMDP. Suppose that arbitrary initial state \((q_{0})\) and goal state \((q_{\mathit{goal}})\) are given, and that the maximum length of the episode is \(T\). At state \(q_{t}\), if the action \(a_{t}\) is applied to the RAMDP, \(\hat{q}_{t+1}\) is produced from the RAMDP. Then, according to the reward function in (1), the next state \(q_{t+1}\) and \(r_{t+1}\) are determined. In view of (1), if \(\hat{q}_{t+1}\) does not belong to \(\mathbf{Q}_{\mathit{mw}}\), then the next state is set as the current state. Furthermore, if the next state is the goal point, then the reward is \(0\), and otherwise the reward is \(-1\). Then, the whole procedure is repeated until the episode ends. Note that this reward function leads to as short as possible path since TD3 tries to maximize the sum of reward and a longer path implies more \(-1\) reward.  
\[q_{t+1}=\begin{cases}\hat{q}_{t+1},&\text{if }\hat{q}_{t+1}\in\mathbf{Q}_{ \mathit{mw}}\\ q_{t},&\text{if }\hat{q}_{t+1}\
ot\in\mathbf{Q}_{\mathit{mw}}\end{cases},\ \ \ \ \ r_{t}= \begin{cases}0,&\text{if }\ q_{t+1}==q_{\mathit{goal}}\\ -1,&\text{if }q_{t+1}\in\mathbf{Q}_{\mathit{mw}}\\ -1,&\text{if }q_{t+1}\in\mathbf{Q}_{\mathit{mw}}\end{cases} \tag{1}\]  
There are two possibilities for the episode to end: (1) the next state becomes the goal state, i.e., \(e=[(q_{0},a_{0}),(q_{1},r_{1},a_{1}),\cdots,(q_{\mathit{goal}},r_{g})]\), and (2) the length of the episode is \(T\), i.e., \(e=[(q_{0},a_{0}),(q_{1},r_{1},a_{1}),\cdots,(q_{T},r_{T})]\) and \(q_{T}\
eq q_{\mathit{goal}}\). For both cases, every sample in the form of \((q_{t},a_{t},r_{t},q_{t+1})\) is saved in the memory. Note that when \(\hat{q}_{t+1}\
otin\mathbf{Q}_{\mathit{mw}}\), the next state \(q_{t+1}\) is set to the current state \(q_{t}\) in order to avoid collision. In the worst case, if this happens repeatedly, the agent is  
Figure 1: Robot Arm Markov Decision Process (RAMDP).  
trained such that the corresponding episode does not occur. Based on these samples in the memory, the optimal policy is determined in accordance with TD3, which is explained in the next subsection.