## 3 Proposed Methodology
### Proposed GATransformer
#### 3.2.4 Transformer Module  
The limitations of exclusively relying on attention weights from GAT for channel pruning are identified in this study, which also recognizes the necessity of a more comprehensive methodology. The GAT's primary emphasis is on the attention weights of individual layers, and it does not establish connections between them. It is widely acknowledged that the CNN layer at the preceding stage substantially impacts the results of the succeeding CNN layer. This article recommends integrating the GAT and Transformer models to optimize the efficacy of channel pruning and improve interlayer connections. The Transformer paradigm's multi-head attention mechanism is essential to this goal. The multi-head attention mechanism, initially developed for natural language processing, has been applied to several fields.  
The Transformer module has four parts, as shown in Figure 9. The Transformer encoder block sends GAT module output to the classification and attention blocks for classification and parallel attention map generation. The attention restructuring block comprises three mechanisms. The initial step is to divide the attention extracted from GAT into a series of identical segments. The Transformer uses the segment count to compute the input sequence length. A trainable linear embedding compresses the patches into a sequence of identical tokens via a linear projection. This is necessary because Transformers require a consistent latent vector size across all layers. Finally, the embedded tokens are positionally embedded to retain the patch position. Standard, learnable 1D position embeddings are used. The Transformer encoder block will receive the output of these operations as input.  
The Transformer encoder has \(L\) iterative normalization layers with two sublayers. The first sublayer is multi-head self-attention. Sublayer two is a complete feed-forward network. All sublayers receive normalization [37] and residual connection [38]. Each sublayer outputs \(I+SL(Norm(I))\), where \(SL\) is its function over normalized input \(I\). First,  
Figure 9: Architecture of the proposed Transformer module.  
investigate an attention function to understand multi-head self-attention. An attention function maps a query and key-value pairs to an output. People call attention to scaling dot-product attention. The attention function receives vectors with a query and key-value. A compatibility function evaluates the query-key connection to determine the weights of the items in the weighted sum. Attention, or self-attention, associates points in an input sequence to represent it.  
Consider the inputs \(Q\), \(K\), and \(V\) for the queries, keys, and values of dimensionality dim. The weight matrices \(\omega_{Q}\), \(\omega_{K}\), and \(\omega_{V}\) are multiplied with the embeddings of the input vector \(I\) acquired through training. The final calculation involves the combination of \(Q\) and \(K\), which are then divided by the square root of the dimensionality of the key vectors. The softmax function defines normalization such that each output vector sums to one. The attention of the input is multiplied by \(V\).  
\[Attn(K,Q,V)=\rho\bigg{(}\frac{QK^{T}}{\sqrt{dim_{K}}}\bigg{)}V \tag{7}\]  
Multi-head self-attention uses a concurrent attention function to focus on information from different representation subspaces at different locations. The operation results are concatenated and converted into linear units of the expected dimension:  
\[mHSA(K,Q,V)=\bigoplus(h_{1},\dots,h_{g})\omega^{out} \tag{8}\]  
\[h_{i}=Attn(K\omega_{K}^{i},Q\omega_{Q}^{i},V\omega_{V}^{i}) \tag{9}\]  
The projections are parameter matrices \(\omega_{K}^{i}\in\mathbb{R}^{dim_{m}\times dim_{K}}\), \(\omega_{Q}^{i}\in\mathbb{R}^{dim_{m}\times dim_{Q}}\), and \(\omega_{V}^{i}\in\mathbb{R}^{dim_{m}\times dim_{V}}\). The GAT module output dimension is represented by \(dim_{m}\). Additional output processing is possible using the MLP sublayer. It has two wholly connected layers and a GeLU nonlinearity layer [39].