## 3 Background
### Graph Attention Network  
Graph attention networks (GATs) are a type of graph neural network in which the edges of the graph are weighted differently, to indicate the importance of nodes with respect to other nodes. The weighting is performed through a self-attention mechanism [37]. Initially, the node representations (node embeddings) are the raw features \(\mathbf{h}^{0}=\mathbf{X}^{T}\) with a size \(N\times F\). Then, a series of linear transformations can be applied through weight matrix multiplication, as follows in Equation (1):  
\[\mathbf{h}^{(l)}=\sigma\left(\mathbf{W}^{(l)}\times\mathbf{h}^{(l-1)}+b^{(l)}\right) \tag{1}\]  
where \(\mathbf{W}^{(l)}\) is the weight matrix of the \(l^{th}\) linear layer, with a size of \(F^{\prime}\times F\). Consider \(F\) as the number of inputs of the \(l^{th}\) layer, and \(F^{\prime}\) as the number of outputs of the \(l^{th}\) layer. \(b^{(l)}\) is the additive bias vector of the \(l^{th}\) layer, and of size \(F^{\prime}\). The activation function to add non-linearity is \(\sigma(.)\). The linear layer can be applied separately for numerical features \(Nu_{i}\) and categorical features \(C_{i}\) of each node. To calculate the weight of each edge (including the self-loops), we used Equation (2):  
\[e_{ij}=a\,\left(\mathbf{W}\times h_{i}^{(l-1)}\,\mathbf{W}\times h_{j}^{(l-1)}\right) \tag{2}\]  
where \(a(\,.\,)\) was the self-attention mechanism. These weights were not normalized, and therefore a Softmax function was utilized for normalization, as in Equation (3):  
\[\alpha_{ij}=\text{softmax}\,\left(e_{ij}\right)=\frac{\exp(LeakyReLU\left(e_{ ij}\right))}{\sum_{k\in\mathcal{N}_{i}}\exp(LeakyReLU\left(e_{ij}\right))} \tag{3}\]  
where \(\alpha_{ij}\) was the attention coefficient; \(\mathcal{N}_{i}\) represented the neighbors of node \(i\); LeakyReLU was an activation function, defined as follows in Equation (4):  
\[LeakyReLU\left(x\right)=\left\{ \begin{array}{ll}x,&x\geq 0\\ \mu\times x,&\text{otherwise}\end{array} \right. \tag{4}\]  
where \(\mu\) was the negative slope. The attention mechanism was performed by concatenating the representation of node \(i\) and node \(j\) (after linear transformation), and multiplying this by a learnable weighting  
Figure 1: Semi-supervised learning uses nodes’ embedding to predict the node class. Both labeled and unlabeled nodes are used in SSL. vector \(z\) of size \(2F^{\prime}\). The result was passed through the LeakyReLU function. Thus, the attention coefficient in Equation (3) could be re-written as shown in Equation (5):  
\[\alpha_{ij}=\frac{\exp\Bigl{(}LeakyReLU\left(z^{T}\times\left[\mathbf{W}\times h_{i }^{(l-1)}\big{|}\big{|}\mathbf{W}\times h_{j}^{(l-1)}\big{|}\big{)}\right)\right)}{ \sum_{k\in\mathcal{N}_{i}}\exp\Bigl{(}LeakyReLU\left(z^{T}\times\left[\mathbf{W} \times h_{i}^{(l-1)}\big{|}\big{|}\mathbf{W}\times h_{k}^{(l-1)}\big{|}\right) \right)\Bigr{)}} \tag{5}\]  
where \(\left(\.\ \right)^{T}\) represented the transpose operator and \(\ |\ |\ \) was the concatenation operator. In [38], the authors discovered that the implementation of the GAT had a static attention nature. The reason for this was the application of the \(\mathbf{W}\) and \(z^{T}\) in the above equation sequentially. The authors fixed this by altering the operations, in which multiplication by \(\mathbf{W}\) was performed after concatenating the nodes' representations, a LeakyReLU was applied, and multiplication by \(z^{T}\) was carried out as a final step. This is shown in Equation (6):  
\[\alpha_{ij}=\frac{\exp\Bigl{(}z^{T}\times LeakyReLU\left(\mathbf{W}\times\left[h_ {i}^{(l-1)}\big{|}\big{|}h_{j}^{(l-1)}\big{|}\big{)}\right)\Bigr{)}}{\sum_{k \in\mathcal{N}_{i}}\exp\Bigl{(}z^{T}\times LeakyReLU\left(\mathbf{W}\times\left[h _{i}^{(l-1)}\big{|}\big{|}h_{k}^{(l-1)}\big{|}\big{|}h_{k}^{(l-1)}\big{|} \big{)}\right)\Bigr{)}} \tag{6}\]  
This is called modified GAT (or GATv2), and we used it in our framework.