## 2 Graph Neural Networks
### Graph Attention Networks  
Graph attention networks (GATs) [2; 4] mark a significant theoretical progression from graph convolutional networks (GCNs). At the heart of GATs is the principle that certain nodes are more crucial than others. This idea, while not entirely novel and somewhat reflected in GCNs, is advanced in GATs. In GCNs, nodes with fewer neighbors gain more importance due to a normalization coefficient that depends primarily on node degrees. However, the limitation of this GCN approach is its sole dependence on node degrees for determining node importance.  
In contrast, GATs aim to create weighting factors that consider not just the node degrees but also the significance of node features. This is where GATs diverge from GCNs: their method of assigning scale factors during the aggregation of neighborhood information. GCNs use a non-parametric scaling factor derived from a normalization function, whereas GATs employ an attention mechanism to allocate scaling factors. This attention-based approach allows GATs to assign greater weights to more important nodes during neighborhood aggregation. This key difference grants GATs a finer degree of control over how information flows within intricate graph structures, making them more adaptable for handling complex data. On the other hand, GCNs are generally more effective in scenarios where the graph's structure is clearly defined, and the significance of each node is more or less uniform.  
To realize this functionality, the graph attention layer in GATs performs several operations on graph-structured data. Initially, each node undergoes a joint linear transformation through a weight parameter matrix \(\mathbf{W}\), setting the stage for further processing and analysis.  
Following the initial transformation in graph attention networks (GATs), the process continues with the computation of attention coefficients. These coefficients represent the non-normalized attention weights calculated pairwise between neighboring nodes. At this stage, the z embeddings of two adjacent nodes are concatenated, forming a combined vector. This concatenated vector is then subjected to a dot product operation with a learnable weight vector, effectively integrating the node features into the attention mechanism.  
Subsequently, to introduce nonlinearity into the model, the LeakyReLU activation function is applied to the result of this dot product operation. The next step involves normalizing the attention coefficients to maintain consistency across all nodes. This is achieved through the application of the softmax function, ensuring that the coefficients are comparable and appropriately scaled.  
Figure 3: Visualization of graph convolutional network (GCN) layers. Unlike classical graph neural networks (GNNs), GCNs incorporate the degree of each node to enable effective normalization during the aggregation process.  
In the aggregation phase, the model combines embeddings from various neighbors, guided by the calculated attention weights. However, an important consideration in GATs is the potential instability of self-attention. To address this issue, the concept of multi-head attention is employed. This approach involves creating multiple attention mechanisms or 'heads,' each with its distinct parameters. These multiple heads operate in parallel, enhancing the model's capacity and stability. Each head computes its own output, which is subsequently integrated to form the final output. Typically, the outputs of these heads are concatenated in intermediate layers of the network, while averaging is used in the final layer to consolidate the information gleaned from different perspectives. In GATs, attentional weights are determined implicitly by comparing inputs to each other (a process known as self-attention) as can be seen in Figure 4. The above mathematical relation concerning the computation of vector representations of nodes is transformed as follows:  
\[h_{i}=\sum_{j\in N_{i}}a_{ij}Wx_{j}\]  
where \(a_{ij}\) are the attention weights calculated dynamically by the network.