## 3 Design and Analysis of Algorithm  
### TD3 Algorithm  
The TD3 algorithm is the chosen training method in the environmental framework. Within reinforcement learning, several classical training algorithms merit consideration, such as the Deep Q-Network (DQN) algorithm and the Deep Deterministic Policy Gradient (DDPG) algorithm. The DQN algorithm relies on a learned value function, known as the Q-function, and integrates crucial techniques such as sample pooling and target networks [38; 39]. However, DQN is primarily effective for tasks with a limited range of discrete actions and faces limitations in scenarios requiring real-time decisions, such as car-following on roads. To effectively address the reinforcement learning problem with continuous actions, the DDPG algorithm is introduced [40]. DDPG merges the value function and the policy gradient algorithm, improving parameter updates and facilitating seamless decision-making in continuous action spaces. Nonetheless, certain algorithm-level issues persist in DDPG, such as overestimation bias and susceptibility to overfitting of narrow peaks in the value estimate. As a remedy, TD3 (Twin Delayed DDPG) is introduced to address these shortcomings in the DDPG algorithm [41]. Compared with DDPG algorithm, the TD3 algorithm uses a double critic network to calculate the target value of Q-functions, opting for the lesser of the two values, thus suppressing the problem of network overestimation. As illustrated in Figure 2, the network structure of the TD3 algorithm is composed of the actor policy network and a critical value network. The actor network is responsible for determining optimal actions, while the critic network evaluates the desirability of these actions by estimating values of Q-functions. To enhance training stability, the target network periodically updates its parameters by copying them from the current network. Additionally, the delayed policy update is employed to ensure that the actor network is updated after the critic network undergoes multiple updates. TD3 employs a parameterized actor neural network, which takes the state (\(S_{t}\)) as input and generates a continuous action (\(A_{t}\)) as output. Simultaneously, a parameterized critic neural network is utilized to take both the state (\(S_{t}\)) and action (\(A_{t}\)) as inputs and estimate the Q-value function. The parameters of the algorithm are manually adjusted through extensive simulations. Both the actor and critic neural networks utilize a two-hidden-layer architecture employing a multi-layer perceptron (MLP). The first layer of both the actor and critic networks consists of 400 neurons, while the second layer comprises 300 neurons. The network structure of the TD3 algorithm is intricately designed with distinct roles for each component. The actor network serves as the interface with the external intersection environment, managing the input and output of data. Concurrently, the set of transitions \((S_{t},A_{t},R_{t},S_{t+1})\) is systematically added to the experience replay pool as samples for future training iterations. Action parameters (\(A_{t}\)) are transferred from the actor network to the critic network. The double critic networks are employed, selecting the minimum value between \(Q^{\theta_{t}\prime}(S_{t+1},A_{t+1})\) and \(Q^{\theta_{t}\prime}(S_{t+1},A_{t+1})\) when calculating the target value, to mitigate the issue of network overestimation in the algorithm. For a set of data from the sample pool, the dual critic-target network calculates the target value (\(y\)) according to (11). \[y=R_{t}+\gamma\underset{i=1,2}{min}Q^{\theta_{t}\prime}(S_{t+1},A_{t+1}), \tag{11}\]  
where \(\gamma\) is the discount factor for reward \(R_{t+1}\), \(\theta_{t}\prime\) denotes the parameter of critic-target networks I and II, and \(Q^{\theta_{t}\prime}(S_{t+1},A_{t+1})\) represents the estimated Q-value according to the state and action. Target policy smoothing is implemented in TD3 as a regularization technique.