## 2 Improved TD3 Algorithm
### Algorithm Structure
The \(\epsilon\)-greedy policy is shown in Equation (8), which shows that when the agent makes a decision, there is a small probability of positive \(\epsilon\) will randomly selecting an unknown action, and a probability of \((1-\epsilon)\) will selecting the action with the largest action value among the existing actions. \[\mu_{\theta}(s_{i})=\left\{ \begin{array}{l}\frac{\epsilon}{|A|}+1-\epsilon \,\ if\ a=\text{argmax}Q(s_{i},a_{i})\\ \frac{\epsilon}{|A|}\,\ \ if\ a\
eq\text{argmax}Q(s_{i},a_{i})\end{array} \right. \tag{8}\]  
A standard RL algorithm must include exploration and exploitation. Exploration helps the agent fully understand state space and select the other unknown action, and exploitation helps the agent find the optimal action to maximize the expected return at the present moment. Inspired by these methods, TD3 is combined with \(\epsilon\)-greedy policy. In order to enhance the exploration ability of agent in the early stage and make more stable use of the exploration in the later stage, the decay of \(\epsilon\)-greedy policy is used and combined with TD3. The \(\epsilon\)-greedy policy proposed based on cosine decay as shown in Equation (9). \[\epsilon=\lambda(c+\cos(\frac{\pi t}{i})) \tag{9}\]  
where \(\lambda\in(0,0.5)\), \(c\) is a constant greater than 1, \(i\) is training steps, and \(t\) is the current step value. After combining TD3 with the \(\epsilon\)-greedy policy of cosine decay, the method of action selection shown in Equation (3) can be changed to the form shown in Equation (10). \[a_{i}=\left\{ \begin{array}{l l}\mu_{\theta}(s_{i})+\delta,&if\ \epsilon\leq X \\ \widetilde{\delta},&if\ \epsilon>X\end{array} \right. \tag{10}\]  
where X is the uniform distribution on [0,1); and \(\widetilde{\delta}\sim(0,\sigma^{\prime})\) is a random number conforming to a Gaussian distribution with a mean of 0 and a standard deviation of \(\sigma^{\prime}\); it is common to set \(\sigma^{\prime}\) to a number associated with the maximum action. The flow of the cosine decay TD3 (CDTD3) algorithm is shown in Figure 1.  
The performance of CDTD3 was tested in MuJoCo through the OpenAI Gym interface. The original task set was used during testing without modifying the environment and reward function. Except for the individual characteristics of the algorithms, other settings remained the same. Each algorithm was run 10 times under different random seeds. The number of training steps in each task was two million steps. The results are shown in Figure 2, including the results of three different DRL algorithms running on three different robot tasks in MuJoCo. The dimensions of the robot's joints in Figure 2a-c from more to less, and the specified task difficulty ranges from hard to easy. The \(x\)-axis is the number of steps, while the \(y\)-axis is the average return of ten evaluations per five thousand steps for the current task under two million training steps. The shaded area in the Figure 2 represents the maximum and minimum value intervals with test data smoothed by convolution, while the solid line represents the average of ten experimental results. In Figure 2c, the robust exploratory ability of CDTD3 does not significantly contribute to relatively simple tasks. However, in the scenarios illustrated in Figure 2a,b, CDTD3 exhibits a superior ability to explore actions with higher rewards and to rapidly incorporate them into decision-making process compared to the TD3. CDTD3 has strong exploration ability and adaptability to tasks, which makes policy learning more efficient. Figure 1: Flowchart of the CDTD3 algorithm. Figure 2: Simulation results of CDTD3 in MuJoCo: (**a**) Walker2d-v4, (**b**) Ant-v4, and (**c**) Hopper-v4.