SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Divin Yan†, Gengchen Wei†, Chen Yang†, Shengzhong Zhang†, Zengfeng Huang†*
†Fudan University, {yanl21, gcwei22, yanc22}@m.fudan.edu.cn {szzhang17, huangzf}@fudan.edu.cn
This paper introduces a new approach to address the issue of class imbalance in graph neural networks (GNNs) for learning on graph-structured data. Our approach integrates imbalanced node classification and Bias-Variance Decomposition, establishing a theoretical framework that closely relates data imbalance to model variance. We also leverage graph augmentation technique to estimate the variance, and design a regularization term to alleviate the impact of imbalance. Exhaustive tests are conducted on multiple benchmarks, including naturally imbalanced datasets and public-split class-imbalanced datasets, demonstrating that our approach outperforms state-of-the-art methods in various imbalanced scenarios. This work provides a novel theoretical perspective for addressing the problem of imbalanced node classification in GNNs.
Graphs are ubiquitous in the real world, encompassing social networks, financial networks, chemical molecules [22, 46, 9] and so on. Recently, graph neural networks (GNNs) [16, 35, 9] have shown exceptional proficiency in representation learning on graphs, facilitating their broad application in various fields. Nevertheless, the process of learning on graphs, analogous to its counterparts in computer vision and natural language processing, frequently encounters significant obstacles in the form of skewed or insufficient data, leading to the prevalent issue of class imbalance. It is undeniable that real-world data often exhibits biases and numerous categories, which entail data limitations and distribution shifts to be common. Furthermore, coupled with the fact that GNNs often feature shallow architectures to prevent "over-smoothing" or "information loss", training with such datasets leads to severe over-fitting problems in minor classes.
Designing GNN imbalanced learning schemes for graph-structured data poses unique challenges, unlike image data. Graph-structured data often requires consideration of the data's topology and environment, and empirical evidence [6, 24, 31] shows that topological asymmetries can also affect the model's performance. However, due to the extremely irregular and structured nature of the data, quantifying and solving topological asymmetry is difficult and computationally intensive. As a result, existing methods such as oversampling [28, 50, 24] and loss function engineering [6, 31] are problematic for achieving satisfactory outcomes. In depth, the modeling approach for data personalization exhibits very poor scalability and generalization capability. Furthermore, the prevalence of this problem has prompted the community to seek a more effective framework for imbalanced learning. Thus, a more fundamental and theoretical perspective is urgently needed when considering the imbalance node classification problem.
Following the idea, in this work, we propose a novel viewpoint to understand graph imbalance through the lens of Bias-Variance Decomposition. The Bias-Variance Decomposition [1, 2, 23, 45] has long been applied in terms of model complexity and fitting capacity. Our theoretical analyses have confirmed the relationship between model variance and the degree of dataset imbalance, whereby an imbalanced training set can lead to poor prediction accuracy due to the resulting increase in variance. Furthermore, we conducted some experiments on real-world dataset and the results demonstrate a significant relationship between the variance and the imbalance ratio of the dataset. As far as we know, we were the first to establish a connection between imbalanced node classification and model variance, which conducts theoretical analysis in this field.
Moreover, we have devised a regularization term for approximating the variance of model, drawing on our theoretical analysis. The main challenge of this idea lies in estimating the expectation across different training sets. Our key insight is to leverage graph data augmentation to model the different training sets in the distribution, which has not been explored before. Our empirical evaluations have consistently demonstrated that our algorithm, leveraging three basic GNN models [16, 35, 9], yields markedly superior performance in diverse imbalanced data settings, surpassing the current state-of-the-art methods by a substantial margin. Notably, we have conducted meticulous experiments on two naturally imbalanced datasets, which serve as the realistic and representative benchmarks for real-world scenarios.
Our contribution can be succinctly summarized as follows: (i) We are the first to integrate imbalanced node classification and Bias-Variance Decomposition. Our work establishes the close relationship between data imbalance and model variance, based on theoretical analysis. (ii) Moreover, we are the first to conduct a detailed theoretical analysis for imbalanced node classification domain. (iii) Our principal insight is to leverage graph data augmentation as a means of representing the varied training sets that lie within the distribution, whilst simultaneously designing a regularization term that approximates the model's variance. (iv) We conduct exhaustive tests on multiple benchmarks, such as the naturally imbalanced datasets and the public-split class-imbalanced datasets. The numerical results demonstrate that our model consistently outperforms state-of-the-art machine learning methods and significantly outperforms them in heavily-imbalanced scenarios.
We concentrates on the task of semi-supervised imbalanced node classification within an undirected and unweighted graph, denoted as  $\mathbf{G} = (\mathbf{V},\mathbf{E},\mathbf{V}_{\mathbf{L}})$ .  $\mathbf{V}$  represents the node set,  $\mathbf{E}$  stands for the edge set, and  $\mathbf{V}_{\mathbf{L}}\subset \mathbf{V}$  denotes the set of labeled nodes. The set of unlabeled nodes is denoted as  $\mathbf{V}_{\mathbf{U}}\coloneqq \mathbf{V}\setminus \mathbf{V}_{\mathbf{L}}$ . The feature matrix is  $\mathbf{X}\in \mathbb{R}^{n\times f}$ , where  $n$  is the number of nodes and  $f$  is the feature dimension. The adjacency matrix is denoted by  $\mathbf{A}\in \{0,1\}^{n\times n}$ . Let  $\mathbf{N}(v)$  represent set of adjacent nodes to node  $v$ . The labeled sets for each class are denoted as  $(\mathbf{C}_1,\mathbf{C}_2,\dots ,\mathbf{C}_k)$ , where  $\mathbf{C}_i$  represents the labeled set for class  $i$ . The imbalance ratio  $\rho$ , is defined as  $\rho = \max_{i}|\mathbf{C}_{i}| / \min_{i}|\mathbf{C}_{i}|$ .
2.2 Graph Dataset Augmentations
To craft diversified perspectives of the graph dataset, we deploy advanced graph augmentation methodologies, leading to the formation of  $\tilde{\mathbf{G}} = (\tilde{\mathbf{A}},\tilde{\mathbf{X}})$  and  $\tilde{\mathbf{G}}^{\prime} = (\tilde{\mathbf{A}}^{\prime},\tilde{\mathbf{X}}^{\prime})$ . Elements such as node attributes and connections within the foundational graph are selectively obfuscated. These augmented perspectives are subsequently channeled through a common Graph Neural Network (GNN) encoder, symbolized as  $f_{\theta}:\mathbb{R}^{n\times n}\times \mathbb{R}^{n\times f}\to \mathbb{R}^{n\times d}$ , in order to distill compact node-centric embeddings:  $f_{\theta}(\tilde{\mathbf{A}},\tilde{\mathbf{X}}) = \mathrm{h}\in \mathbb{R}^{n\times d}$  and  $f_{\theta}(\tilde{\mathbf{A}}',\tilde{\mathbf{X}}') = \mathrm{h}'\in \mathbb{R}^{n\times d}$ .
Semi-Supervised Imbalanced Node Classification. Numerous innovative approaches[28, 40, 50, 19, 25, 6, 24, 31, 43] have been proposed to tackle the difficulties arising from imbalanced node classification on graph data. GraphSMOTE [50] employs the SMOTE [5] algorithm to perform interpolation at the low dimensional embedding space to synthesize the minority nodes, while ImGAGN [25] introduces GAN [8] for the same purpose. Another work GraphENS [24] synthesizes
mixed nodes by combining the ego network of minor nodes with target nodes. To address topology imbalance in imbalanced node classification task, ReNode [6] adjusts node weights based on their proximity to class boundaries. TAM [31] leverages local topology that automatically adapts the margins of minor nodes. Due to spatial constraints, we provide a comprehensive exposition of other relevant literature in Appendix A.
3.1 Theoretical Motivation
This section presents a succinct overview of the classical Bias-variance Decomposition [1, 2, 23, 45] and its application to semi-supervised node classification. Furthermore, we illustrate the influence of imbalance on classification results, which subsequently amplifies the variance.
Bias-variance Decomposition. Let  $x \in X$  and  $y \in Y$  denote the input and label, respectively. Consider an underlying mapping  $f: X \to Y$  and the label can be expressed as  $y = f(x) + \epsilon$ , where  $\epsilon$  is some noise. Given a training set  $D = (x_i, y_i)$ , we train a model  $\hat{f}(x) = \hat{f}(x; D)$  using the samples in  $D$ .
Definition 1 Bias-Variance Decomposition is that the expected predicted error of  $f$  can be decomposed into Eq(1),
$$
\mathbb {E} _ {D, x} \left[ (y - \hat {f} (x; D)) ^ {2} \right] = \left(\operatorname {B i a s} [ \hat {f} (x) ]\right) ^ {2} + \operatorname {V a r} _ {D} [ \hat {f} (x; D) ] + \text {I r r e d u c i b l e e r r o r}, \tag {1}
$$
where  $\mathrm{Bias}[\hat{f}(x)] = \mathbb{E}_x[\hat{f}(x;D) - f(x)] = \mathbb{E}_D[\hat{f}(x;D)] - \mathrm{E}[y(x)]$ , and  $\mathrm{Var}_D[\hat{f}(x;D)] = \mathbb{E}_D\left[\left(\mathbb{E}_D[\hat{f}(x;D)] - \hat{f}(x;D)\right)^2\right]$ .
It is important to note that this expectation is taken with respect to both the training sets  $D$  and the predicted data  $x$ . The bias term reflects the model's ability to fit the given data, while the variance term indicates the stability of the model's results with different training sets. In other words, the variance describes the model's generalization performance.
Next, we consider the specific formulation of variance in the setting of semi-supervised node classification on graph. First, we make some assumptions to simplify the analysis.
Assumptions. We make two assumptions in our approach. Firstly, we assume that the node embeddings  $h^i$  of node  $x^i$  extracted by a graph neural network for nodes belonging to class  $i$  follow a multivariate normal distribution  $h^i \sim N(\mu^i, \Lambda^i)$ , where  $\Lambda^i$  is a diagonal matrix for all  $i = 1, 2, \ldots, c$ . Here, let  $\epsilon^i = h^i - \mu^i$  which follows the distribution  $\epsilon^i \sim N(0, \Lambda^i)$ .
Secondly, we consider a simple classifier that estimates the probability of a node belonging to a particular class based the distance  $h^T C^i$ . Here,  $C^i = C^i(D) = \frac{1}{n_i} \left( h_1^i + \dots + h_{n_i}^i \right)$  is the average of labeled node embedding in training set  $D$ , where  $n_i$  is the number of nodes in class  $i$ .  $C^i$  follows the distribution  $C^i \sim N\left(\mu^i, \frac{1}{n_i} \Lambda^i\right)$ . Similarly, we denote  $e^i = C^i - \mu^i$  and it follows that  $e^i \sim N\left(0, \frac{1}{n_i} \Lambda^i\right)$ .
Variance for Semi-supervised Node Classification. Under the above assumption, we can write the variance generated by different sampling training set explicitly. For a node  $x$  that belongs to class  $j$ , the variance can be written:
$$
\begin{array}{l} \operatorname {V a r} (x) = \sum_ {i = 1} ^ {c} \mathbb {E} _ {D} \left[ \left(h ^ {T} (x) C ^ {i} - \mathbb {E} _ {D} \left[ h ^ {T} (x) C ^ {i} \right]\right) ^ {2} \right] \\ = \sum_ {i = 1} ^ {c} \mathbb {E} _ {e ^ {i}} \left[ \left(h ^ {T} (x) \left(\mu^ {i} + e ^ {i}\right) - h ^ {T} (x) \mu^ {i}\right) ^ {2} \right] \tag {2} \\ = \sum_ {i = 1} ^ {c} \mathbb {E} _ {e ^ {i}} \left[ \left(h ^ {T} (x) e ^ {i}\right) ^ {2} \right] = \sum_ {i = 1} ^ {c} \frac {1}{n _ {i}} h ^ {T} (x) \Lambda^ {i} h (x) \\ \end{array}
$$
The last equation is derived from the variance of multivariate normal distribution. The variance over the whole graph  $\sum_{i=1}^{c} \mathbb{E}_x[\operatorname{Var}(x)] = \sum_{i=1}^{c} \mathbb{E}_x\left[\frac{1}{n_i} h^T(x) \Lambda^i h(x)\right]$  is the expectation of Equation 2 on node  $x$ .
Variance and Imbalance. In Equation 2, we notice that the variance of a specific class  $i$  is proportional to  $\frac{1}{n_i}$ . This reveals a relation between imbalance and variance. If we assume  $\mathbb{E}_x[h^T(x)\Lambda^i h(x)]$  is the same for different  $i$ , then the following theorem holds:
Theorem 1 Under the condition that  $\sum_{i}n_{i}$  is a constant, the variance  $\sum_{i = 1}^{c}\mathbb{E}_{x}\left[\frac{1}{n_{i}} h^{T}(x)\Lambda^{i}h(x)\right]$  reach its minimum when all  $n_i$  equal.
The proof is include in Appendix B.1. As the ratio of imbalance increases, the minority class exhibits a smaller sample size  $n_i$ , which consequently makes a greater contribution to the overall variance. This result gives an new perspective of explanation about why the model shows poor performance under imbalance training, that variance will increase as the training set becomes imbalanced. To substantiate our analysis, we conducted an experimental study to show the relation between imbalance and the variance. The result is shown in Figure 1.
(a) CiteSeer-GCN
(b)CiteSeer-GAT
Figure 1: We examine the alteration in variance concerning node classification as the imbalance ratio increases on and plot the regression curves for variance and imbalance ratio. We conduct this experiment using a fixed number of training set nodes but different ones, to mitigate the influence of the number of training set nodes on variance. Detailed experimental setup is in Appendix D.2.
(c) CiteSeer-SAGE
3.2 Variance Regularization
Optimize Variance during Training. Both our analysis and the experimental results show that the increase of variance is a reason for the deterioration of performance for imbalance training set. Therefore, We propose to estimate the variance and use it as a regularization during training. We decompose the model into two parts. A GNN  $E$  is utilized as a feature extractor, and a classifier  $P$  predict the probability based on the feature extracted by GNN. We define the variance of  $P$  conditioning on  $E$  of as Equation 2 evaluated for a fixed GNN  $E$ . Then we can take this as regularization. By optimizing this term, we allow the model to automatically search for the feature extractor that yields the smallest variance during the optimization process.
Estimate the Expectation with Labeled Nodes. The most challenging aspect lies in the estimation of variance, as we lack access to other training sets. Therefore, we propose Lemma 1 to estimate the variance on training set with the variance on labeled data. The proof is included in Appendix B.2.1.
Lemma 1 Under the above assumption for  $h^i \sim N(\mu^i, \Lambda^i)$ ,  $C^i \sim N\left(\mu^i, \frac{1}{n_i} \Lambda^i\right)$ , minimizing the  $\sum_{i=1}^{c} \mathbb{E}_x[\operatorname{Var}(x)]$  is equivalent to minimizing Equation 3:
$$
\frac {1}{N} \sum_ {x \in G} \sum_ {i = 1} ^ {c} \left(h (x) ^ {T} \frac {1}{\sqrt {2 n _ {i}}} \left(h _ {1} ^ {i} - h _ {2} ^ {i}\right)\right) ^ {2} = \frac {1}{N} \sum_ {k = 1} ^ {n _ {j}} \sum_ {j = 1} ^ {c} \sum_ {i = 1} ^ {c} \left(\left(\mu_ {k} ^ {j} + \epsilon_ {k} ^ {j}\right) ^ {T} \frac {1}{\sqrt {2 n _ {i}}} \left(\epsilon_ {1} ^ {i} - \epsilon_ {2} ^ {i}\right)\right) ^ {2}. \tag {3}
$$
Estimate the Expectation with Unlabeled Nodes. Lemma 1 allows us to replace the sampling on training set with sampling on labeled node pairs  $(h_1^i, h_2^i)$  of class  $i$ . However, to evaluate Equation 3 it still needs to have access to embedding pair  $(h_1^i, h_2^i)$  that belong to the same class  $i$ . Since labels are scarce, such node pairs is difficult to obtain. To make our algorithm more practical, a way of utilizing unlabeled nodes for estimating Equation 3 is required. We accomplish this goal through two steps. In the first step, we use graph augmentation to obtain pseudo embedding pair  $(h_1, h_2)$ . Graph augmentation is a typical used technique in graph contrastive learning. It generate new view  $G'$  of a graph  $G$  by adding noise to the graph structure. For a node  $x$  in two different views  $G, G'$ , the resulting embedding  $h, h'$  can be seen as that of two different nodes. Thus, they can be used to replace true embedding pair in Equation 3.
However, these pseudo node pairs give no information about which class it belongs to. Thus it prohibits us from assign proper  $\frac{1}{\sqrt{2n_i}}$  for different  $i$  in Equation 3. It is such coefficients that compensate on the minority class, so it's crucial to reintroduce the message about class number for constructing variance regularization. Therefore, in the second step, we use the class center  $C^i$  to replace  $h$  in Equation 3, as shown in Equation 4. Since  $C^i = \mu^i + e^i$  and the variance of  $e^i$  is proportional to  $\frac{1}{n_i}$ , using Equation 4 to estimate Lemma 1 can apply the similar compensatory for the minority class. More mathematical details are presented in Appendix B.2.1.
$$
\begin{array}{l} \frac {1}{N} \sum_ {x \in G} \sum_ {i = 1} ^ {c} \left(\left(C ^ {i}\right) ^ {T} h (x) - \left(C ^ {i ^ {\prime}}\right) ^ {T} h ^ {\prime} (x)\right) ^ {2} \\ = \frac {1}{N} \sum_ {k = 1} ^ {n _ {j}} \sum_ {j = 1} ^ {c} \sum_ {i = 1} ^ {c} \left(\left(\mu^ {i}\right) ^ {T} \left(\epsilon_ {k} ^ {j} - \epsilon_ {k} ^ {j ^ {\prime}}\right) + \left(e ^ {i}\right) ^ {T} \epsilon_ {k} ^ {j} - \left(e ^ {i ^ {\prime}}\right) ^ {T} \epsilon_ {k} ^ {j ^ {\prime}}\right) ^ {2} \tag {4} \\ \end{array}
$$
From the Viewpoint of Graph Sampling. We notice that an alternative perspective can be adopted to interpret this regularization term. Given a graph data  $G$ , we should acknowledge that there exists some noise in its structure or feature, as a result of measuring. This randomness also attribute to the variance of classification. Assume each graph  $G$  in our dataset is a sample drawn from an underlying true graph  $\tilde{G}$ . Denote the classification result trained on  $G$  as  $f(x; G)$ . Then the variance on  $G$  can be written as
$$
\operatorname {V a r} _ {G} = \mathbb {E} _ {x \in G} \left[ \mathbb {E} _ {G} \left[ [ f (x; G) - \mathbb {E} _ {G} [ f (x; G) ] ^ {2} ] \right] \right] \tag {5}
$$
Taking  $f(x) = h^{T}(x)C$ , it is straightforward to show that Equation 4 and Equation 5 are same.
Despite the algorithm obtained through this new interpretation being identical to the previous one, it is worth noting that the two interpretations differ significantly. The former incorporates information from unlabeled nodes and assumes the variance originates from selecting different nodes as the training set. Graph augmentation facilitates the sampling of node pairs belonging to the same class. The latter, on the other hand, considers the same training set, but the variance originates from the entire graph's information. Graph augmentation is used to simulate the process of sampling the training graph from the underlying true graph.
4 The Final Algorithm
In this section, we introduce our ReVar (Regularize Variance) framework which is based on previous theoretical frameworks for optimizing model variance. We explain our innovative approach to approximating the model's variance and using it as a regularization term in our algorithm in Section 4.1. In Section 4.2, we discuss how we integrate the GCL framework with a new contrastive learning term designed for semi-supervised learning tasks. Lastly, we present the final formulation of our optimization objective in Section 4.3, which optimizes both the cross-entropy loss function and our variance-constrained regularization term, providing a comprehensive approach to improve the performance of semi-supervised imbalanced node classification tasks.
4.1 Variance-constrained Optimization with Adaptive Regularization
To initiate, we elucidate the methodology underpinning the computation of Equation 4. Initially, we calculate the class center  $C_i$  corresponding to each class. Following this, we systematically construct
Figure 2: Overall pipeline of ReVar. (a) Two different views of the graph  $\tilde{\mathbf{G}},\tilde{\mathbf{G}}^{\prime}$  are obtained by graph augmentation transform, and are subsequently fed into GNN encoder  $f_{\theta}$ . (b) Intra-class and inter-class representations are aggregated, which means, for labeled nodes, it's positive samples not only belong to the same class in both view but also in the other view. (c) Variance is estimated by Equation 4. Specifically, the label probability distribution is computed for each node in two views based on its similarity with each class center. And the difference between two probability distributions is used to approximate the model's variance and also optimized as one term in the loss function.
a probability distribution reference matrix, denoted by  $\mathcal{S} \coloneqq [C_1; C_2; \ldots; C_k]$ . It's worth noting that the assemblage of class centers is represented by  $C \coloneqq (C_1; C_2; \ldots; C_k)$ .
Given a node  $i$  with its associated embedding  $h_i$ , the label probability distribution  $\pi_i$  pertaining to node  $i$  is ascertainable via the subsequent equation:
$$
\pi_ {i} ^ {j} = \frac {\exp (\sin \left(\mathrm {h} _ {i} , C _ {j}\right) / \tau)}{\sum_ {l = 1} ^ {k} \exp (\sin \left(\mathrm {h} _ {i} , C _ {l}\right) / \tau)} \tag {6}
$$
In the aforementioned equation,  $\pi_i^j$  signifies the  $j$ -th dimensional component of  $\pi_i$ ,  $\mathrm{sim}(\cdot, \cdot)$  denotes the function to compute the cosine similarity between a pair of vectors, and  $\tau$  is a specified temperature hyperparameter.
Subsequent to this, for every node  $i \in V$ , we determine the label probability distribution. Distinctively,  $\tilde{\pi}_i$  and  $\tilde{\pi}_i'$  are derived from  $\tilde{G}$  and  $\tilde{G}'$ , respectively. The former acts as the prediction distribution, whereas the latter represents the target distribution. Our approach then aims to minimize the cross-entropy between these two distributions.
It's paramount to underscore that for each labeled node within  $\tilde{G}^{\prime}$ , its one-hot label vector is directly assigned as  $\tilde{\pi}_i' = y_i$  for all  $i \in V_L$ . This diverges from the procedure of deducing the predicted class distribution as described in Equation 6. This methodology is pursued to optimally leverage the extant label information.
However, a simplistic minimization of the aforementioned loss might engender confirmation bias. This is attributed to the potentially inaccurate  $\tilde{\pi}_i^{\prime}$  estimated for the unlabeled nodes (i.e.,  $V_{U}$ ), an aspect that can deleteriously impact the efficacy of pseudo-labeling-oriented semi-supervised methodologies. To address this concern, we introduce a confidence-based label-guided consistency regularization [41, 30, 48], as detailed below:
$$
\mathcal {L} _ {\mathrm {V R}} = \frac {1}{| V _ {\text {c o n f}} |} \sum_ {i \in V _ {\text {c o n f}}} C E \left(\tilde {\pi} _ {i} ^ {\prime}, \tilde {\pi} _ {i}\right) + \frac {1}{| V _ {L} |} \sum_ {i \in V _ {L}} C E \left(y _ {i}, \tilde {\pi} _ {i}\right) \tag {7}
$$
where  $V_{\mathrm{conf}} = \{v_i \mid \mathbf{1}_{\{\max(\tilde{\pi}_i') > v\}} = 1, \forall i \in V_U\}$  represents the set of nodes with confident predictions,  $v$  is the threshold for determining whether a node has a confident prediction, and  $\mathbf{1}_{\{\dots\}}$  is an indicator function.
In the context of confidence determination for  $\tilde{\pi}_i'$ , we employ a criterion wherein a value is appraised as confident if its maximum constituent surpasses a stipulated threshold. We postulate that a heightened threshold, represented by  $v$ , acts as a safeguard against confirmation bias. This stratagem ensures that only high-caliber target distributions, specifically  $\tilde{\pi}_i'$ , have a pivotal influence on Equation 7.
4.2 Intra-Class Aggregation Regularization