## 3 Methodology
### Rainbow DQN Algorithm Process  
Figure 3 shows the framework of the intersection signal control algorithm based on Rainbow DQN_AM, which consists of two parts: the intersection signal control strategy and neural network parameter updates. The Rainbow DQN intersection signal control algorithm is based on DQN [47], so its algorithmic flow closely resembles that of the DQN algorithm. The specific steps are as follows:  
Step 1: parameter settings, including \(\theta\)--initialization of the main \(Q\)-network parameters; \(\theta^{\prime}\)--initialization of the target \(Q\)-network parameters; \(D\)--experience replay pool; \(\gamma\)--discount factor; \(\tau\)--smoothing coefficient; \(\lambda\)--learning rate of the main \(Q\)-network; \(\epsilon\)--exploration rate; \(M\)--maximum capacity of the experience replay pool; \(B\)--training batch size; \(\mu\)--target \(Q\)-network update frequency; \(I\)--number of iterations; \(N\)--total number of training episodes; and \(T\)--simulation episode time. Step 2: initialize the experience replay pool with a capacity of M; initialize the Rainbow DQN main network \(Q(s,a;\theta)\) and its corresponding parameters \(\theta\); initialize the target network \(Q_{target}(s,a;\theta^{\prime})\) and its corresponding parameters \(\theta^{\prime}\). Step 3: start the iteration, generate the traffic scenario, and obtain the initial intersection environment state \(s_{0}\). Step 4: Generate a random probability value \(r\). If \(r<\!\epsilon\), randomly select a control action for the traffic signal \(a_{t}\). If \(r>\!\epsilon\), the network outputs a control action \(a_{t}\) based on the current intersection environment state \(s_{t+1}\), \(a_{t}=\underset{a}{max}Q(s_{t},a;\theta)\). Step 5: Execute the action \(a_{t}\), obtain the new intersection state \(s_{t+1}\) and the corresponding reward \(r_{t}\). Store the tuple \((s_{t},a_{t},r_{t},s_{t+1},d)\) as a sample in the experience replay buffer, with the sample priority calculated based on the TD error \(|\delta|\). If the experience buffer is full, delete the oldest sample record. Step 6: Network update: The neural network parameters are updated using prioritized experience replay. After the parameters are updated, if the iteration count has not reached the predefined value, proceed to Step 4. Otherwise, proceed to Step 7. The process for updating the neural network parameters is as follows:  
(1) The prioritized experience replay strategy is used to sample tuples \((s_{t},a_{t},r_{t},s_{t+1},d)\) from the experience pool. It is important to note that, due to the multi-step learning strategy, samples from the current tuple to the next N steps in the future need to be collected. (2) The optimal action \(a*\) is calculated using the main \(Q\)-network in the Double DQN network, as shown in the following formula:  
\[a*=argmax_{a}Q(s_{t+N},a;\theta) \tag{16}\]  
where \(Q(s_{t+N},a;\theta)\) represents the expected value of the action \(a\) taken by the main \(Q\)-network at the state \(s_{t+N}\). The expected value of the main network is calculated through the distributional output of the Distributional DQN, as shown in the following formula:  
\[Q(s_{t+N},a;\theta)=\sum_{i=0}^{M}p(z_{i}|s_{t+N-1},a_{t+N-1};\theta)z_{i} \tag{17}\]  
where \(z_{i}\) represents the value of the \(i\)-th sample point; \(p(z_{i}|s_{t+N-1},a_{t+N-1};\theta)\) denotes the probability of the \(i\)-th sample point when the main \(Q\)-network takes action \(a_{t+N-1}\) at state \(s_{t+N-1}\). (3) The target \(Q\)-value distribution is computed using the target network, based on the multi-step learning strategy. The formula is as follows:  
Figure 3: Framework of the Rainbow DQN_AM algorithm. \[z_{itarget}^{\prime}=\sum_{k=0}^{N-1}\gamma^{k}r_{t+k}+\gamma^{N}(1-d)z_{itarget} \tag{18}\]  
where \(z_{itarget}\) is the \(Q\)-value distribution of the optimal action \(a*\) from the target Q network, \(\gamma\) is the discount factor, and \(z_{itarget}^{\prime}\) represents the true target value of the agent at state \(s_{t}\).