# Power Grid Load Forecasting Using a CNN-LSTM Network Based on a Multi-Modal Attention Mechanism
## 3 Methodology
### Channel Attention Model  
In the global attention mechanism phase, the model captures the long-range temporal dependencies of input features using the self-attention mechanism. This process helps the model understand the temporal dynamics and trends of load data. However, while the global attention mechanism effectively captures sequential information, it cannot directly focus on the importance of each feature channel. Therefore, to further optimize feature representation and enhance the model's sensitivity to key predictive information under the comprehensive modeling of both temporal and spatial features, we designed a channel attention mechanism based on the SE (Squeeze-and-Excitation) module. This mechanism applies weighted processing to the feature channels, automatically identifying and reinforcing the channels most relevant to the task, while suppressing redundant and  
Figure 4: The structure diagram of global attention. irrelevant channel features. The structure of the SE module is shown in Figure 1. Specifically, the SE module first performs a \"Squeeze\" operation to compress the global information of each channel, resulting in a channel-level description. Then, through the \"Excitation\" operation, a weight coefficient is assigned to each channel, determining its contribution to the final output. Finally, through the \"Scale\" operation, the channel weight coefficients obtained in the \"Excitation\" step are applied to each channel of the original feature map, adjusting the features of each channel accordingly. In detail, the channel weights obtained after the \"Excitation\" step are element-wise-multiplied with the features of each channel in the original feature map, thereby scaling the features of different channels. This process highlights the channels that are most important for the task and suppresses those that are less relevant. This ensures that the model can dynamically focus on the most predictive channels while suppressing redundant or irrelevant information, thereby improving the model's performance in tasks like grid load forecasting. The specific operations of Squeeze, Excitation, and Scale are as follows:  
The Squeeze operation compresses each feature channel into a real number, called a descriptor, by applying global pooling to the input features. This descriptor represents the statistical information of the corresponding channel, which is typically achieved through global average pooling. The unweighted feature map \(U\) is summarized and averaged, resulting in a multi-dimensional statistic \(z\in\mathbb{R}^{C}\). The \(n\)-th element of \(z\) is calculated as follows:  
\[z_{n}=F_{squeeze}(x_{n})=\frac{1}{H\times W}\sum_{i=1}^{H}\sum_{j=1}^{W}x_{n}(i, j), \tag{14}\]  
where \(W\) and \(H\) represent the width and height of the feature map, \(x_{n}\) is the feature vector of the \(n\)-th channel in the input features, and \(x_{n}(i,j)\) represents the element in the \(i\)-th row and \(j\)-th column of \(x_{n}\). The Excitation operation uses two fully connected layers to fully learn the descriptors of each channel and explore the interactions between different feature channels. Under the excitation operation, each feature channel is dynamically weighted by its corresponding activation function. The calculation process is as follows:  
\[s=F_{excitation}(z,W)=\sigma(W_{2}\lambda(W_{1}z)), \tag{15}\]  
where \(\sigma\) and \(\lambda\) represent the Sigmoid and ReLU activation functions, \(W_{1}\in\mathbb{R}^{(C/r)\times C}\) and \(W_{2}\in\mathbb{R}^{C\times(C/r)}\) are the weight parameters of the two fully connected layers, \(r\) is the ratio function that balances performance and computational complexity, and \(s\) is the vector representing the channel weights of the feature map \(U\). The final output of the SE module is achieved through the scaling operation, where the weight vector \(s\) and the unweighted input feature map \(U\) are element-wise-multiplied by the weight of each channel.