SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Donglin Xia $^{1}$ , Xiao Wang $^{2*}$ , Nian Liu $^{1}$ , Chuan Shi $^{1*}$
<sup>1</sup>Beijing University of Posts and Telecommunications
2Beihang University
{donglin.xia, nianliu, shichuan}@bupt.edu.cn, xiao_wang@buaa.edu.cn
Graph neural networks (GNNs) have become increasingly popular in modeling graph-structured data due to their ability to learn node representations by aggregating local structure information. However, it is widely acknowledged that the test graph structure may differ from the training graph structure, resulting in a structure shift. In this paper, we experimentally find that the performance of GNNs drops significantly when the structure shift happens, suggesting that the learned models may be biased towards specific structure patterns. To address this challenge, we propose the Cluster Information Transfer (CIT) mechanism<sup>2</sup>, which can learn invariant representations for GNNs, thereby improving their generalization ability to various and unknown test graphs with structure shift. The CIT mechanism achieves this by combining different cluster information with the nodes while preserving their cluster-independent information. By generating nodes across different clusters, the mechanism significantly enhances the diversity of the nodes and helps GNNs learn the invariant representations. We provide a theoretical analysis of the CIT mechanism, showing that the impact of changing clusters during structure shift can be mitigated after transfer. Additionally, the proposed mechanism is a plug-in that can be easily used to improve existing GNNs. We comprehensively evaluate our proposed method on three typical structure shift scenarios, demonstrating its effectiveness in enhancing GNNs' performance.
Graphs are often easily used to model individual properties and inter-individual relationships, which are ubiquitous in the real world, including social networks, e-commerce networks, citation networks. Recently, graph neural networks (GNNs), which are able to effectively employ deep learning on graphs to learn the node representations, have attracted considerable attention in dealing with graph data [13, 24, 32, 36, 9, 5]. So far, GNNs have been applied to various applications and achieved remarkable performance, e.g., node classification [13, 32], link prediction [33, 14] and graph classification [6, 34].
Message-passing mechanism forms the basis of most graph neural networks (GNNs) [13, 24, 5, 9]. That is, the node representations are learned by aggregating feature information from the neighbors in each convolutional layer. So it can be seen that the trained GNNs are highly dependent on the local structure. However, it is well known that the graph structure in the real world always changes. For instance, the paper citations and areas would go through significant change as time goes by in citation network [11]. In social networks, nodes represent users and edges represent activity between users, because they will be changed dynamically, the test graph structure may also change [7]. This change in the graph structure is a form of distribution shift, which we refer to as graph structure shift. So
$^{2}$ Code available at https://github.com/BUPT-GAMMA/CITGNN
the question naturally arises: when the test graph structure shift happens, can GNNs still maintain stability in performance?
We present experiments to explore this question. We generate graph structure and node features. Subsequently we train GNNs on the initially generated graph structure and gradually change the structures to test the generalization of GNNs (more details are in Section 2). Clearly, the performance consistently declines with changes (shown in Figure 1), implying that the trained GNNs are severely biased to one typical graph structure and cannot effectively address the structure shift problem.
This can also be considered as Out-Of-Distribution problem (OOD) on graph. To ensure good performance, most GNNs require the training and test graphs to have identically distributed data. However, this requirement cannot always hold in practice. Very recently, there are some works on graph OOD problem for node classification. One idea assumes knowledge of the graph generation process and derives regularization terms to extract hidden stable variables [28, 8]. However, these methods heavily rely on the graph generation process, which is unknown and may be very complex in reality. The other way involves sampling unbiased test data to make their distribution similar to that of the training data [37]. However, it requires sampling test data beforehand, which cannot be directly applied to the whole graph-level structure shift, e.g., training the GNNs on a graph while the test graph is another new one, which is a very typical inductive learning scenario [24, 17].
Therefore, it is still technically challenging to learn the invariant representations which is robust to the structure shift for GNNs. Usually, the invariant information can be discovered from multiple structure environments [28], while we can only obtain one local structure environment given a graph. To avoid GNNs being biased to one structure pattern, directly changing the graph structure, e.g., adding or removing edges, may create different environments. However, because the graph structure is very complex in the real world and the underlying data generation mechanism is usually unknown, it is very difficult to get the knowledge on how to generate new graph structures.
In this paper, we propose to learn the invariant representations of GNNs by transferring the cluster information of the nodes. First, we apply GNNs to learn the node representations, and then combine it with spectral clustering to obtain the cluster information in this graph. Here, we propose a novel Cluster Information Transfer (CIT) mechanism, because the cluster information usually captures the local properties of nodes and can be used to generate multiple local environments. Specifically, we characterize the cluster information using two statistics: the mean of cluster and the variance of cluster, and transfer the nodes to new clusters based on these two statistics while keeping the cluster-independent information. After training GNNs on these newly generated node representations, we aim to enhance their generalization ability across various test graph structures by improving the generalization ability on different clusters. Additionally, we provide insights into the transfer process from the embedding space and theoretically analyze the impact of changing clusters during structure shift can be mitigated after transfer. The contributions of our work are summarized as follows:
- We study the problem of structure shifts for GNNs, and propose a novel CIT mechanism to improve the generalization ability of GNNs. Our proposed mechanism is a friendly plug-in, which can be easily used to improve most of the current GNNs.
- Our proposed CIT mechanism enables that the cluster information can be transferred while preserving the cluster-independent information of the nodes, and we theoretically analyze that the impact of changing clusters during structure shift can be mitigated after transfer.
- We conduct extensive experiments on three typical structure shift tasks. The results well demonstrate that our proposed model consistently improves generalization ability of GNNs on structure shift.
2 Effect of structure distribution shift on GNN performance
In this section, we aim to explore the effect of structure shift on GNN performance. In the real world, the tendency of structure shift is usually unknown and complex, so we assume a relatively simple scenario to investigate this issue. For instance, we train the GNNs on a graph with two community structures, and then test the GNNs by gradually changing the graph structures. If the performance of GNNs drops sharply, it indicates that the trained GNNs are biased to the original graph, and cannot be well generalized to the new test graphs with structure shift.
We generate a network with 1000 nodes and divide all nodes into two categories on average, that is, 500 nodes are assigned label 0 and the other 500 nodes are assigned label 1. Meanwhile, node features, each of 50 dimensions, are generated by two Multivariate Gaussian distributions. The node features with same labels are generated by same Gaussian distribution. We employ the Stochastic Blockmodel (SBM) [12] to generate the graph structures. We set two community and set the generation edges probability on inter-community is  $0.5\%$  and intro-community is  $0.05\%$ . We randomly sample 20 nodes per class for training, and the rest are used for testing. Then we train GCN [13] and GAT [24] on this graph. The test graph structures are generated as follows: we decrease the inter-community edge probability from  $0.5\%$
Figure 1: The node classification accuracy of GCN, GAT, APPNP and GCNII on generated data with structure shift. The x-axis is probability of edges  $(\%)$ .
to  $0.25\%$  and increase the intra-community probability from  $0.05\%$  to  $0.3\%$ .
The accuracy is shown in Figure 1. The x-axis is the probability of edges, where the first number is the edge probability of inter-community and the second number is the edge probability of introcommunity. As we can see, because of the structure shift, the performance of GNN declines significantly. It shows that once the test graph pattern shifts, the reliability of GNNs becomes compromised.
Notations: Let  $G = (\mathbf{A}, \mathbf{X})$  represent a training attributed graph, where  $\mathbf{A} \in \mathbb{R}^{n \times n}$  is the symmetric adjacency matrix with  $n$  nodes and  $\mathbf{X} \in \mathbb{R}^{n \times d}$  is the node feature matrix, and  $d$  is the dimension of node features. Specifically,  $A_{ij} = 1$  represents there is an edge between nodes  $i$  and  $j$ , otherwise,  $A_{ij} = 0$ . We suppose each node belongs to one of  $C$  classes and focus on semi-supervised node classification. Considering that the graphs always change in reality, here we aim to learn the invariant representations for GNNs, where the overall framework is shown in Figure 2 and the whole algorithm is shown in A.1.
3.1 Clustering process
We first obtain the node representations through GNN layers:
$$
\mathbf {Z} ^ {(l)} = \sigma \left(\tilde {\mathbf {D}} ^ {- 1 / 2} \tilde {\mathbf {A}} \tilde {\mathbf {D}} ^ {- 1 / 2} \mathbf {Z} ^ {(l - 1)} \mathbf {W} _ {G N N} ^ {(l - 1)}\right), \tag {1}
$$
where  $\mathbf{Z}^{(l)}$  is the node representations from the  $l$ -th layer,  $\mathbf{Z}^{(0)} = \mathbf{X}$ ,  $\tilde{\mathbf{A}} = \mathbf{A} + \mathbf{I}$ ,  $\tilde{\mathbf{D}}$  is the degree matrix of  $\tilde{\mathbf{A}}$ ,  $\sigma$  is a non-linear activation and  $\mathbf{W}_{GNN}^{(l-1)}$  is the trainable parameters of GNNs. Eq. (1) implies that the node representations will aggregate the information from its neighbors in each layer, so the learned GNNs are essentially dependent on the local structure of each node. Apparently, if the structure shift happens in the test graph, the current GNNs may provide unsatisfactory results. Therefore, it is highly desired that the GNNs should learn the invariant representations while structure changing, so as to handle the structure shift in test graphs.
One alternative way is to simulate the structure change and then learn the invariant representations from different structures, which is similar as domain generalization [26, 35]. Motivated by this, we consider that the local properties of a node represents its domain information. Meanwhile, different clusters can capture different local properties in a graph, so we can consider cluster information as domain information of nodes. Based on this idea, we first aim to obtain the cluster information using spectral clustering [2]. Specifically, we compute the cluster assignment matrix  $\mathbf{S}$  of node using a multi-layer perceptron (MLP) with softmax on the output layer:
$$
\mathbf {S} = \operatorname {S o f t m a x} \left(\mathbf {W} _ {M L P} \mathbf {Z} ^ {(l)} + \boldsymbol {b}\right), \tag {2}
$$
Figure 2: The overall framework of our proposed CIT mechanism on GNNs consists two parts: the traditional GNNs and Cluster Information Transfer (CIT) mechanism. After getting node representations from GNNs, we conduct CIT mechanism on node representations before the last layer of GNNs, which transfers the node to another cluster to generate new representations for training.
where  $\mathbf{W}_{MLP}$  and  $\pmb{b}$  are trainable parameters of MLP. For assignment matrix  $\mathbf{S} \in \mathbb{R}^{n \times m}$ , and  $m$  is the number of clusters.  $s_{ij}$  represents the weight that node  $i$  belongs to cluster  $j$ . The softmax activation of the MLP guarantees that the weight  $s_{ij} \in [0,1]$  and  $\mathbf{S}^T \mathbf{1}_M = \mathbf{1}_N$ .
For S, we want to cluster strongly connected nodes. So we adopt the cut loss which evaluates the mincut given by S:
$$
\mathcal {L} _ {c} = - \frac {\operatorname {T r} \left(\mathbf {S} ^ {T} \tilde {\mathbf {A}} \mathbf {S}\right)}{\operatorname {T r} \left(\mathbf {S} ^ {T} \tilde {\mathbf {D}} \mathbf {S}\right)}, \tag {3}
$$
where  $Tr$  is the trace of matrix. Minimizing  $\mathcal{L}_c$  encourages the nodes which are strongly connected to be together.
However, directly minimizing  $\mathcal{L}_c$  will make the cluster assignments are equal for all nodes, which means all nodes will be in the same cluster. So in order to avoid the clustering collapse, we can make the clusters to be of similar size, and we use orthogonality loss to realize it:
$$
\mathcal {L} _ {o} = \left\| \frac {\mathbf {S} ^ {T} \mathbf {S}}{\| \mathbf {S} ^ {T} \mathbf {S} \| _ {F}} - \frac {\mathbf {I} _ {M}}{\sqrt {M}} \right\| _ {F}, \tag {4}
$$
where  $\| \cdot \| _F$  indicates the Frobenius norm,  $M$  is the number of clusters. Notably, when  $\mathbf{S}^T\mathbf{S} = \mathbf{I}_M$ , the orthogonality loss  $\mathcal{L}_o$  will reach 0. Therefore, minimizing it can encourage the cluster assignments to be orthogonal and thus the clusters can be of similar size.
Overall, we optimize the clustering process by these two losses:
$$
\mathcal {L} _ {u} = \mathcal {L} _ {c} + \lambda_ {1} \mathcal {L} _ {o}, \tag {5}
$$
where  $\lambda_{1}$  is a coefficient to balance the mincut process and orthogonality process.
After obtaining the assignment matrix  $\mathbf{S}$ , we can calculate cluster representations  $\mathbf{H}^c$  by average the node representations  $\mathbf{Z}^{(l)}$ :
$$
\mathbf {H} ^ {c} = \mathbf {S} ^ {T} \mathbf {Z} ^ {(l)}. \tag {6}
$$
3.2 Transfer cluster information
Now we obtain the cluster representations  $\mathbf{H}^c$ , and each cluster captures the information of local properties. Originally, given a graph, a node can be only in one domain, i.e., one cluster. Next, we aim to transfer the node to different domains. Specifically, the cluster information can be characterized by two statistics, i.e., the center of the cluster  $(\mathbf{H}^c)$  and the standard deviation of the cluster  $(\sigma(\mathbf{H}_k^c))$ :
$$
\sigma \left(\mathbf {H} _ {k} ^ {c}\right) = \sqrt {\sum_ {i = 1} ^ {n} s _ {i k} \left(\mathbf {Z} _ {i} ^ {(l)} - \mathbf {H} _ {k} ^ {c}\right) ^ {2}}, \tag {7}
$$
where  $\mathbf{H}_k^c$  is the representation of the  $k$ -th cluster and  $\mathbf{Z}_i^{(l)}$  is the representation of node  $i$ .
Then we can transfer the node  $i$  in the  $k$ -th cluster to the  $j$ -th cluster as follows:
$$
\mathbf {Z} _ {i} ^ {\prime (l)} = \sigma \left(\mathbf {H} _ {j} ^ {c}\right) \frac {\mathbf {Z} _ {i} ^ {(l)} - \mathbf {H} _ {k} ^ {c}}{\sigma \left(\mathbf {H} _ {k} ^ {c}\right)} + \mathbf {H} _ {j} ^ {c}, \tag {8}
$$
where  $k$  is the cluster that node  $i$  belongs to, and  $j$  is the cluster which is randomly selected from the remaining clusters.
Here, we explain the Eq. (8) in more details. As shown in Figure 3, the center of cluster  $k$  is  $\mathbf{H}_k^c$ , and node  $i$  belongs to this cluster, where the representation of node  $i$  is  $\mathbf{Z}_i^{(l)}$ . It can be seen that  $\mathbf{H}_k^c$  is obtained by aggregating and averaging the local structure information, which captures the cluster information. Therefore,  $\mathbf{Z}_i^{(l)} - \mathbf{H}_k^c$  represents the cluster-independent information. Then, the CIT mechanism can be seen as we transfer the node to a new position from embedding space. The standard deviation is the weighted distance of nodes from the center, which is the aggregating scope of the clusters. After the transfer, the target node surrounds a new cluster with new cluster information, while keeping the cluster-independent information.
Figure 3: We show two parts of one graph. The orange and green points represent two clusters in one graph. The circle is aggregating scope of cluster. And the red point represents the target node we transfer from orange cluster to green cluster.
The above process only transfers the nodes on the original domain. Moreover, in order to improve the robustness of the model for unknown domain, we increase the uncertainty and diversity of model [16, 21]. Based on this, we add Gaussian perturbations to this process. The whole transfer becomes:
$$
\mathbf {Z} _ {i} ^ {\prime (l)} = \left(\sigma \left(\mathbf {H} _ {j} ^ {c}\right) + \epsilon_ {\sigma} \Sigma_ {\sigma}\right) \frac {\mathbf {Z} _ {i} ^ {(l)} - \mathbf {H} _ {k} ^ {c}}{\sigma \left(\mathbf {H} _ {k} ^ {c}\right)} + \left(\mathbf {H} _ {j} ^ {c} + \epsilon_ {\mu} \Sigma_ {\mu}\right). \tag {9}
$$
The statistics of Gaussian is determined by the whole features:
$$
\epsilon_ {\sigma} \sim \mathcal {N} (0, 1), \epsilon_ {\mu} \sim \mathcal {N} (0, 1), \tag {10}
$$
$$
\Sigma_ {\sigma} ^ {2} = \sigma \left(\sigma \left(\mathbf {H} ^ {c}\right) ^ {2}\right) ^ {2}, \Sigma_ {\mu} ^ {2} = \sigma \left(\mathbf {H} ^ {c}\right) ^ {2}. \tag {11}
$$
Here, with the Gaussian perturbations, we can generate new clusters based on the original one and the nodes can be further transferred to more diverse domains.
The proposed transfer process offers a solution to overcome the limitations of changing graph structures when generating nodes in different domains. Traditional approaches require changing the graph structure by adding or deleting edges, which can be challenging due to the lack of prior knowledge on how the changes in graph structure may affect the domain. Here, our proposed CIT mechanism addresses this challenge by directly transferring a node to another domain through the manipulation of cluster properties of nodes in the embedding space. The CIT method is implemented before the classifier in GNNs training, making it backbone agnostic and compatible with any GNN last layer.
3.3 Objective function
With CIT mechanism, we have the generated representation  $\mathbf{Z}_i^{(l)}$  in other clusters for node  $i$ . In traditional GNNs training, we input the representation  $\mathbf{Z}^{(l)}$ , where the  $i$ -th row vector represents the representation of node  $i$ , to the cross-entropy loss [13]. Here, we randomly select a part of nodes, and replace original representation  $\mathbf{Z}_i^{(l)}$  with the generated representation  $\mathbf{Z}_i^{(l)}$ . Then we can obtain the new representation matrix  $\mathbf{Z}^{(l)}$ , and optimize the cross-entropy loss based on it as follows:
$$
\mathcal {L} _ {f} = \mathcal {L} _ {e} \left(f _ {\theta} \left(\mathbf {Z} ^ {\prime (l)}\right), \mathbf {Y}\right), \tag {12}
$$
where  $\mathcal{L}_e$  is cross-entropy loss function,  $f_{\theta}$  is the classifier, and  $\mathbf{Y}$  is the labels.
Finally, the overall optimization function is as follows:
$$
\mathcal {L} = \mathcal {L} _ {f} + \lambda_ {2} \mathcal {L} _ {u}, \tag {13}
$$
where  $\lambda_{2}$  is a coefficient to balance the classification process and clustering process. In the optimization process, we randomly select  $n\times p$  nodes and then conduct Eq. (9) every  $k$  epochs.
3.4 Theoretical analysis
In this section, we theoretically analyze our CIT mechanism from the perspective of domain adoption theory. Since the Eq. (8) is the cornerstone of our CIT mechanism and Eq. (9) is an extending from it, we use Eq. (8) to represent our CIT mechanism in the following proof for convenience.
We analyze that the classifier using the new generated representation  $\mathbf{Z}^{\prime(l)}$  in Eq. (8) has better generalization on the structure shift. As mentioned before, cluster information can capture local properties of nodes, so we convert this problem to analyzing the classifier has better generalization on cluster shift. To achieve this, following [31], we take the binary classification task as the example, and analyze the relationship between the decision boundary with cluster information. For analysis, we follow [31] using a fisher classifier and analyze the relationship between them. According to Fisher's linear discriminant analysis [1, 23, 3], the decision boundary of fisher classifier depends on two statistics  $Var(Z)$  and  $Cov(Z,Y)$ .
Theorem 1. The decision boundary of fisher classifier is affected by the cluster information.
The proof is given in Appendix A.3, where we calculate the mathematical expression of the classification boundary. Theorem 1 indicates that the cluster information will "help" the classifier to make decisions. But when the graph structure changes, the cluster information of the nodes also changes. Therefore, the classifier using cluster information to make decisions is not reliable.
Theorem 2. Let  $Z_{R}$  represent the node representations in cluster  $R$ . Assume that there are  $p$  percent of nodes are transferred from cluster  $R$  to cluster  $D$  by  $\Sigma_{D}\frac{Z_{R} - \mu_{R}}{\Sigma_{R}} +\mu_{D}$ . After the transfer, the impact of changing clusters during structure shift can be mitigated.
The proof is given in Appendix A.4. Comparing the expression of classifier before and after our CIT mechanism, we can find that we mitigate the impact of changing clusters during structure shift, enhancing the robustness of the model against such changes.
Datasets and baselines. To comprehensively evaluate the proposed CIT mechanism, we use six diverse graph datasets. Cora, Citeseer, Pubmed [20], ACM, IMDB [27] and Twitch-Explicit [17]. Details of datasets are in Appendix B.1. Our CIT mechanism can be used for any other GNN backbones. We plug it in four well-known graph neural network methods, GCN [13], GAT [24], APPNP [9] and GCNII [5]. Meanwhile, we compare it with two graph OOD methods which are also used for node-classification, SR-GNN [37] and EERM [28]. We combine them with four methods mentioned before. Meanwhile, we use CIT-GNN(w/o) to indicate that no Gaussian perturbations are added.
Experimental setup. For Cora, CiteSeer and Pubmed, we construct structure shift by making perturbations on the original graph structure. For ACM and IMDB, we test them using two relation structures. For Twitch-Explicit, we use six different networks. We take one network to train, one network to validate and the rest of networks to test. So we divide these datasets into three categories: perturbation on graph structures, Multiplex networks and Multigraph. The implementation details are given in Appendix B.1.
4.1 Perturbation on graph structures data
In this task, we use Cora, Citeseer and Pubmed datasets. We train the model on original graph. We create new graph structures by randomly adding  $50\%$ ,  $75\%$  and deleting  $20\%$ ,  $50\%$  edges of original graph. To comprehensively evaluate our model on random structure shift, we test our model on the
new graphs. We follow the original node-classification settings [13] and use the common evaluation metrics, including Macro-F1 and classification accuracy. For brief presentation, we show results of deleting edges in Appendix B.2.