## 2 Literature Survey
### GCN to GAT Transition  
GCNs, the simplest formulation of Graph Neural Networks, define model training based on the concept of isotropic aggregation. Isotropic aggregation means that the contribution of each neighbor to the node under consideration is considered equal. On this basis, the simple neighborhood aggregation function is defined as follows [24]:  
\[h_{v}^{k}=\sigma(W_{k}[\Sigma_{ucN(v)}\frac{h_{u}^{k-1}}{N(v)}]+B_{k}.h_{v}^{k-1}) \tag{7}\]  
Equation (7) is a more formalized way of expressing a GCN model. Here, \(h_{v}^{k}\) represents the resulting node embedding after \(k\) layers of neighborhood aggregation, \(\sigma\) represents the nonlinear function (e.g., ReLU), \(h_{u}^{k-1}\) represents the node embedding of the previous layer, \(\Sigma_{ucN(v)}\frac{h_{v}^{k-1}}{N(v)}\) is the average neighborhood aggregation over \(u\) nodes such that \(u\epsilon N(v)\), and \(B_{k}\) is the bias.  
The Graph Attention Model varies the concept of GCN by defining the importance of neighborhood connections. It defines the importance of the message from node \(u\) to node \(v\) such that \((u,v)e\). Let \(a\) define the attention coefficient for the edge \(e_{vu}\) for the pair of nodes \((u,v)\); \(e_{vu}\) is defined as [24]:  
\[e_{vu}=a(W_{k}h_{u}^{k-1},W_{k}h_{v}^{k-1}) \tag{8}\]  
Furthermore, normalizing the expression in Equation (8) using the Softmax function, the equation turns out to be:  
\[\alpha_{vu}=\frac{exp(e_{vu})}{\sum_{k\epsilon N(v)}exp(e_{vk})} \tag{9}\]  
Finally, the node embedding at \(K^{th}\) layer is defined as:  
\[h_{v}^{k}=\sigma(\sum_{ucN(v)}\alpha_{vu}W_{k}h_{u}^{k-1}) \tag{10}\]  
Equation (10) defines the learning model according to GAT. The parameters of the attention mechanism \(a\) are trained using a neural network. These parameters are learned together with the weight matrices. To stabilize the learning process, a multi-head attention mechanism is used. Here, the attention operations for each layer are repeated \(R\) times independently. Finally, the output is aggregated or summed. This equation-based analysis of the process justifies why GAT has an advantage over GCNs. GAT considers aggregation of neighbors based on their importance with respect to the node. This further improves the accuracy of the results and the quality of the learning process imparted to the model.  
In the world of neural network architectures running on graph structure data, [PERSON] et al. [29] introduced a new approach called GATs, i.e., Graph Attention Networks. The authors try to strengthen GCN approximation techniques and remove weaknesses by using mask self-attention layers. In this model, nodes can pick up information about neighboring nodes by superimposing the layers. Interestingly, no prior knowledge of graph structure is required to implement layering, and this process is computationally fast.  
[PERSON] et al. [30] present a supervised graph attention network (super- GAT). This model considers both implicit and explicit weights for the nodes. Therefore, the model provides better results than the traditional GAT. In another robust variant of GAT, [PERSON] et al. [31] claim that the performance of semi-supervised learning is improved. In refining GAT, [PERSON] et al. [32] focused on overfitting during training and improved the performance of their model.