## 3 Methodology
### Algorithm Improvements Based on Rainbow DQN  
Based on the Rainbow DQN framework, we designed an efficient and comprehensive Rainbow DQN_AM model by introducing the ACmix module to address the discretized state space. Traditional CNN models excel at extracting local data features but struggle to capture sufficient global information. On the other hand, while LSTM models can capture temporal patterns, they lack effective spatial modeling capabilities. In intersection signal optimization, traffic flow is complex and dynamically changing, which requires models to have strong state information capturing abilities. Moreover, real-time signal optimization demands high computational efficiency to quickly process large amounts of state data. Combining sufficient state information extraction with rapid computation efficiency helps to enhance the model's decision-making capabilities. [PERSON] (2018) explored the relationship between self-attention mechanisms and convolution and proposed the ACmix module, which integrates both. This module retains the advantages of convolutional layers, such as parameter sharing, sparse interactions, and multi-channel processing, allowing it to efficiently learn features, reduce computational load, and enhance the model's expressiveness. Additionally, by incorporating the self-attention mechanism, the ACmix module can handle long-range dependencies and global information, enabling a more comprehensive capture of global features in intersection traffic flow. The integration of these two modules compensates for CNN's limitations in capturing global features while maintaining computational efficiency, thereby improving the model's performance in complex traffic flow environments. The network structure is shown in Figure 4.  
The pseudocode for the improved Rainbow DQN Algorithm 1 is as follows:  
```
0: Parameters
1: For \(e\in\{1,2,3,\dots,N\}\) do;
2: Initialize the intersection road network environment \(s_{0}\);
3: for \(t\in\{1,2,\dots,T\}\) do;
4: Observe the intersection road network environment \(s_{t}\), Select action \(a_{t}\) based on the greedy policy and noisy network. Execute action \(a_{t}\), obtain new state \(s_{t+1}\), and receive a reward \(r_{t}\);
5: Store the tuple \((s_{t},a_{t},r_{t},s_{t+1},d)\) as an experienced sample in the experience replay buffer \(D\);
6: If \(|D|>M\) exceeds capacity, delete the sample with the lowest priority;
7: Sample \((s,a,r,s^{\prime},d)\) batch samples of size \(B\) from the experience replay buffer \(D\) using a prioritized experience replay strategy;
8: for each transition \((s,a,r,s^{\prime},d)\) do. Primary \(Q\)-network determines the optimal action \(a*=argmax\ Q\ (s_{t+N},a;\theta)\);
9: Compute the target value \(z^{\prime}_{target}=\sum_{k=0}^{N-1}\gamma^{k}r_{t+k}+\gamma^{N}(1-d)z_{i_{ target}}\);
10: Determine the target value distribution \(p_{i_{target}}\) based on the target value;
11: Update parameters \(\theta\) using the cross-entropy loss function;
12:\(J(\theta)=-\sum_{i=0}^{M}p_{i_{target}}(s,a;\theta^{\prime})log\ p_{i}(s_{t},a _{t};\theta)\);
13: Update the sample priorities in the experience replay buffer;
14: At every \(I\) step, update the target network parameters \(\theta^{\prime}=\tau\theta+(1-\tau)\theta^{\prime}\).
15:end
16:end
17:end
```  
**Algorithm 1**Rainbow DQN algorithm