# Data Mining of Students' Consumption Behaviour Pattern Based on Self-Attention Graph Neural Network
## 3 Methods
### Model Description  
We proposed an improved self-attention GNN based on previous research. The corresponding Algorithm 2 pseudo-code of improved self-attention model is as follows.  
```
0: Adjacency matrix of constructed matrix \(W_{adj}\), node feature vector \(V\), graph indicators \(P\) and degree matrix \(D\)
0: The prediction labels on the test set
1: Initialize: initialize weight matrix \(W\) and bias \(\epsilon\)
2: Normalize adjacency matrix by \(L=D^{-\frac{1}{2}}(W_{adj}+I)D^{-\frac{1}{2}}\), in which \(D\) refers to the degree matrix
3: Build a graph convolutional layer to obtain the attention scores by calculating \(Z=GCN(L,V,W,\epsilon)\)
4: Perform convolution twice, use activation function ReLu to process the scores, and concatenate the scores \(S\)
5: Perform self-attention pooling by function pooling (\(W_{adj}\), \(S\), \(P\)) and update the graph structure and adjacency matrix \(W_{adj}\) according to the mask
6: Perform maximum pooling and average pooling, and concatenate the results
7: Predict the labels after the three fully connected layers
```  
**Algorithm 2** Graph classification based on improved self-attention GNN  
In the standard self-attention mechanism, given a group of nodes \((x_{1},x_{2},\cdots,x_{k})\) and weight matrices \(W_{Q},W_{K},W_{V}\) that represent different linear transformations of features, the attention coefficients are computed to reflect the pair-wise importance of the nodes, as shown in Equation (2):  
\[e_{ij}=\left(W_{Q}^{T}x_{i}\right)^{T}\left(W_{K}^{T}x_{j}\right),\forall 1 \leq i,j\leq k \tag{2}\]  
Then \(e_{ij}\) is normalized by all possible values of \(j\) using the Softmax function as shown in Equation (3):  
\[\alpha_{ij}=\frac{\exp\left(e_{ij}\right)}{\sum_{1\leq l\leq k}\exp(e_{il})} \tag{3}\]  
Finally, a weighted sum of transformed features is calculated as shown in Equation (4):  
\[\vec{d}_{i}=\tanh\left(\sum_{1\leq j\leq k}\alpha_{ij}W_{V}^{T}x_{j}\right) \tag{4}\]  
Additionally, a new node embedding vector set can be obtained when using a multi-head graph attention layer. In graph attention neural networks, the node used for the attention mechanism generally only aggregates the information of the first-order neighbours to update the information. In the improved model, due to the characteristics of the extracted features, and to utilize the neighbouring information of node \(v_{i}\) better, we applied two graph convolution layers. In the convolution layers, we calculated the node embedding using a linear transformer \(W\) and used the activation function ReLu to calculate the raw attention score between pair-wise nodes using Equations (5) and (6):  
\[Z=GCN(L,V,W,\epsilon) \tag{5}\]  
\[e_{ij}=\text{ReLU}\left(\vec{a}^{T}\left(z_{i}^{(l)}\|z_{j}^{(l)}\right)\right) \tag{6}\]where \(\vec{a}\) is a weight vector for learning and \(\parallel\) refers to concatenation. Next, we applied the SoftMax function to normalize \(e_{ij}\), and a weighted sum based on attention on the features of all the neighbour nodes, as shown in Equation (7):  
\[h_{i}^{(l+1)}=\text{ReLu}\left(\sum_{j\in\mathcal{N}(i)}\alpha_{ij}^{(l)}z_{j}^ {(l)}\right) \tag{7}\]  
\(\mathcal{N}(i)\) refers to all the neighbours of node i. We also used the pooling method to handle redundant information and reduce the amount of calculations. The extracted features denote the occurrence frequency of consumption behaviour sequence patterns and the ReLu function shows the best performance on our dataset.