## 3 TD3 Based Motion Planning for Smoother Paths
### TD3 (Twin Delayed Deep Deterministic Policy Gradient)  
In this section, TD3 is introduced [24], which is used to search the optimal policy in the proposed algorithm [17; 22; 35; 36]. To this end, it is assumed that a sufficient number of samples (\(s_{t},a_{t},r_{t},s_{t+1}\)) are stored in the memory. Figure 2 describes the structure of TD3. The basic structure of TD3 is the actor-critic network. However, unlike the actor critic network, there are two critic deep neural networks (DNN), one actor DNN, and three target DNNs for each critic and actor DNNs. Hence, there are 6 DNNs (2 critic DNNs and their target DNNs, 1 actor DNN, and its target DNN) in TD3. As seen in Figure 3, the critic DNNs generate an optimal estimate \(Q_{\theta_{i}}(s_{t},a_{t})\) of the state-action value function. The input of the critic DNN is a mini-batch from the memory and its output is \(Q_{\theta_{i}}(s_{t},a_{t}),\ i=1,2\). The mini-batch is a finite set of samples (\(s_{t},a_{t},r_{t},s_{t+1}\)) from the memory. In TD3, it is important that the two critic DNNs are used in order to remove overestimation bias in \(Q_{\theta_{i}}(s_{t},a_{t})\) function approximation. The overestimation bias can take place when bad states due to accumulated noises are overestimated. In order to cope with this, TD3 chooses the smaller \(Q_{\theta_{i}}(s_{t},a_{t})\) value out of two critic DNN outputs critic DNN as the target value. In Figure 3, \(\theta_{1}\) and \(\theta_{2}\) are the parameters of the two critic DNNs, and \(\theta_{1}^{\prime}\) and \(\theta_{2}^{\prime}\) those of corresponding target DNNs. In order to train the critic DNN, as the cost function, the following quadratic function of temporal difference error \(\delta:=Q_{\theta}(q,a)-y^{\text{target}}\) is minimized,  
\[J_{\delta}(\theta):=1/2(Q_{\theta}(q,a)-y^{\text{target}})^{2}, \tag{2}\]  
where \(Q_{\theta}(s,a)\) stands for the parameterized state-action value function \(Q\) with parameter \(\theta\),  
\[y^{\text{target}}=r+\gamma\min_{i=1,2}Q_{\theta_{i}^{\prime}}(q,\tilde{a}) \tag{3}\]  
is the target value of the function \(Q_{\theta}(s,a)\), and the target action (i.e., the action used in the critic target DNN) is defined as,  
\[\tilde{a}=\mu_{\tilde{\theta}}(q)+\tilde{\epsilon}, \tag{4}\]  
where noise \(\tilde{\epsilon}\) follows a clipped normal distribution \(\text{clip}[(\mathcal{N}(0,\sigma),-c,c],\ c>0\). This implies that \(\tilde{\epsilon}\) is a random variable with \(\mathcal{N}(0,\sigma)\) and belongs to the interval \([-c,c]\). Figure 2: Structure of TD3 (Twin Delayed Deep Deterministic Policy Gradient) with RAMDP. The inputs of the actor DNN are both \(Q_{\theta_{1}}(q_{t},a_{t})\) from the critic DNN and the mini-batch from the memory, and the output is the action. Precisely, the action is given by \(a_{t}=\mu_{\phi}(q_{t})+\epsilon\) where \(\phi\) is the parameter of the actor DNN, \(\mu_{\phi}\) is the output from the actor DNN, and a deterministic and continuous value. Noise \(\epsilon\) follows the normal distribution \(\mathcal{N}(0,\sigma)\), and is added for exploration. In order to tune the parameter \(\phi\), the following cost function is minimized. \[J_{\mu}(\phi)=\sum_{q}d^{\mu}(q)Q_{\theta}(q,\mu_{\phi}(q)+\epsilon), \tag{5}\]  
where \(d^{\mu}(q)\) denotes the distribution of the state. Note that the gradient \(\
abla J_{\mu}(\theta)\) (equivalently \(\
abla Q_{\theta}(q,a)\)) is used to update the parameter \(\phi\). This is why the method is called the policy gradient. Between the two outputs from the two critic target DNN, the common target value in (3) is used to update the two critic DNNs. Also, TD3 updates the actor DNN and all three target DNNs every \(d\) steps periodically in order to avoid too fast convergence. Note that the policy \(\mu_{\phi}(q)\) is updated proportionally to only \(Q_{\theta_{1}}(q,\mu_{\phi}(q))\)[24].