## 3 Proposed Methodology
### Proposed GATransformer
#### 3.2.3 GAT Module  
This study employs the graph attention network (GAT) to compute attention scores among channel weight nodes and eliminate unnecessary channels, facilitating efficient channel pruning. GAT is a neural network architecture for graph-structured data, which learns node representations by considering structural linkages and nearby node attributes. Each channel weight node has a feature vector describing its features in channel pruning. The attention mechanism in GAT uses a learnable weight matrix to calculate similarity scores between the feature vectors of neighboring nodes. These similarity scores are normalized into attention coefficients, which determine the influence of each neighboring node on the target node. The final representation of a node is derived by combining the weighted sum of its neighbors' features, using the attention coefficients, with its feature vector. In this study, GAT identifies meaningful channels across the network by calculating attention scores for each channel weight node. These scores rank the importance of channels, enabling the pruning of less significant ones. This progressive pruning method processes layers sequentially, from the first to the last, ensuring that channel importance is consistently evaluated and optimized. By exploring intra-layer and inter-layer relationships, the GAT module enhances the pruning process, improving the model's efficiency while retaining critical channels that contribute to performance. The flattened channel weights from each convolutional layer serve as the input to the GAT module, as illustrated in Figure 7. Through this approach, GAT enables precise and effective channel pruning for large neural networks. We will explain the only graph attentional layer employed in our studies across all GAT topologies. Our attentional arrangement closely follows [36] but it is agnostic to the attention mechanism. Our layer receives node characteristics, \(\xi=\{\xi_{6},\ldots,\xi_{y}\},\quad\xi_{x}\in\mathbb{R}^{d}\), where \(y\) is the number of nodes and \(d\) is their features. The layer generates additional node features (of perhaps varied cardinality \(d^{\prime}\)) as \(\xi^{\prime}=\{\xi^{\prime}_{1},\ldots,\xi^{\prime}_{y}\},\quad\xi^{\prime} _{x}\in\mathbb{R}^{d}\) as its output. At least one learnable linear transformation is needed to turn input features into higher-level features expressively. First, a weight matrix-parameterized shared linear transformation \(\omega\in\mathbb{R}^{d}\times\mathbb{R}^{d^{\prime}}\) is applied to each node. A shared attentional mechanism \(\alpha:\mathbb{R}^{d^{\prime}}\times\mathbb{R}^{d^{\prime}}\rightarrow\mathbb{R}\) computes attention coefficients to perform self-attention on nodes. \[c_{j,k}=\alpha(\omega\xi_{j};\omega\xi_{k}) \tag{1}\]  
This equation shows how vital the node's attributes are to node \(j\). In the most general model, any node can attend any other node, eliminating all structural information. To incorporate graph structure into the mechanism, we use masked attention to compute \(c_{j,k}\) solely  
Figure 7: Input to GAT module through flattened channel weights. for nodes \(k\in\eta_{j}\), where \(\eta_{j}\) is a neighborhood of node \(j\) in the network. All our trials will make them the first-order neighbors of \(j\), including \(j\). To ensure coefficient comparability across nodes, we normalize them using the softmax function \(\rho\) for all \(k\) choices:  
\[\varsigma_{j,k}=\rho_{k}(c_{j,k})=\frac{\exp(c_{j,k})}{\sum_{i\in\eta_{j}}\exp (c_{j,k})} \tag{2}\]  
Our experiments use a single-layer feedforward neural network (a) with a weight vector \(\in\mathbb{R}^{2H^{\prime}}\) and LeakyReLU nonlinearity \(L\) (with negative input slope \(\varsigma=0.2\)) for the attention mechanism.