## 2 Improved TD3 Algorithm
### Algorithm Structure  
The actor-critic structure DRL algorithm TD3 [23] for continuous control is advanced. Similar to popular DRL such as soft actor critical (SAC) [34], TD3 uses the double network structure of actor and critic. Both the current actor network \(\mu_{\theta}(s)\) and the current critic network \(Q_{\omega_{k}}(s,a)\) (\(k=1,2\)) have a corresponding target network, during the implementation, only \(\mu_{\theta}(s)\) and \(Q_{\omega_{k}}(s,a)\) participate in parameter update while the target actor network \(\mu_{\theta^{\prime}}(s)\) and the target critic network \(Q_{\omega^{\prime}_{k}}(s_{i+1},a^{\prime})\) (\(k=1,2\)) are used to store the parameters of the corresponding target network at the previous time. \(\mu_{\theta^{\prime}}(s)\) and \(Q_{\omega^{\prime}_{k}}(s_{i+1},a^{\prime})\) do not completely copy the parameters of their corresponding original network when storing them; instead, a soft update method shown in Equation (2) is adopted. \[\left\{ \begin{array}{l}\theta^{\prime}=\tau\theta+(1-\tau)\theta^{\prime} \\ \omega^{\prime}{}_{k}=\tau\omega_{k}+(1-\tau)\omega^{\prime}{}_{k}\end{array} \right. \tag{2}\]  
where \(\tau\in(0,1)\). \(\theta\) and \(\theta^{\prime}\) represent the network parameters of \(\mu_{\theta}(s)\) and \(\mu_{\theta^{\prime}}(s)\), respectively. \(\omega_{k}\) and \(\omega^{\prime}{}_{k}\) correspond to \(Q_{\omega_{k}}(s,a)\) and \(Q_{\omega^{\prime}_{k}}(s_{i+1},a^{\prime})\), respectively. TD3 select an action based on policy \(\mu_{\theta}(s_{i})\) in the \(s_{i}\). Due to the deterministic policy adopted by TD3, a certain proportion of random noise is added in the exploration phase to improve the exploration ability of the agent. The output of \(a_{i}\) is shown in Equation (3). \[a_{i}=\mu_{\theta}(s_{i})+\delta \tag{3}\]  
where \(\delta\sim\mathcal{N}(0\text{,}\sigma)\) represents a Gaussian distribution with a mean of 0 and variance \(\sigma\). To limit the action, \(a\) is cropped as \(a_{low}<a<a_{high}\). The concept of off-policy DRL is to fully utilize previous experience memory, these algorithms usually have a large experience replay buffer \(\mathcal{B}\), which is used to store information such as state, action, reward, and next state for each step. When updating the actor network and critic network, the required parameter sequence \(\Gamma=(s_{i},a_{i},r_{i},s_{i+1})\) in the update equation will be obtained by sampling from the \(\mathcal{B}\), and the objective function can be calculated using Equations (4) and (5). \[y_{i}=r_{i}+\gamma\underset{k=1,2}{\text{min}}Q_{\omega^{\prime}_{k}}(s_{i+1},a^{\prime}) \tag{4}\]  
\[a^{\prime}=\mu_{\theta^{\prime}}(s_{i+1})+\xi \tag{5}\]  
where \(r_{i}\) is thetwo target networks \(Q_{\omega^{\prime}_{k}}(s_{i+1},a^{\prime})\) is minimized to reduce bias; otherwise, the update of the current critic network can be performed with Equation (6) to minimize the loss \(\mathcal{L}_{\omega}\). \[\mathcal{L}_{\omega}=\frac{1}{N}\sum_{i=0}^{N}\left[y_{i}-Q_{\omega_{k}}(s_{i},a_{i})\right]^{2} \tag{6}\]  
where \(N\) is the number of batches sampled from \(\mathcal{B}\). In deterministic policy gradient RL [35], the policy parameters are updated by calculating the sampled policy gradient. Equation (7) can be used to update the current actor network. \[\mathcal{L}_{\theta}=\frac{1}{N}\sum_{i=0}^{N}\left[\
abla_{a}Q_{\omega_{1}}(s _{i},a)\right|_{a=\mu_{\theta}(s_{i})}\
abla_{\theta}\mu_{\theta}(s_{i})] \tag{7}\]  
As shown in Equation (3), although adding a certain proportion of noise in the exploration stage of TD3 can increase a certain exploration ability, the deterministic policy plays a dominant role in the policy learning exploration, which still limits the early exploration of the agent. In order to enhance the exploration ability of the agent, \(\epsilon\)-greedy is a commonly used policy for balancing exploitation and exploration.