Title: Rethinking Semi-Supervised Imbalanced Node Classification from Bias-Variance Decomposition
Venue: NeurIPS 2023
Accepted: yes
Keywords: Imbalanced Node Classification

Bias-Variance Decomposition

Graph Neural Networks
Abstract: This paper introduces a new approach to address the issue of class imbalance in graph neural networks (GNNs) for learning on graph-structured data. Our approach integrates imbalanced node classification and Bias-Variance Decomposition, establishing a theoretical framework that closely relates data imbalance to model variance. We also leverage graph augmentation technique to estimate the variance and design a regularization term to alleviate the impact of imbalance. Exhaustive tests are conducted on multiple benchmarks, including naturally imbalanced datasets and public-split class-imbalanced datasets, demonstrating that our approach outperforms state-of-the-art methods in various imbalanced scenarios. This work provides a novel theoretical perspective for addressing the problem of imbalanced node classification in GNNs.

Rethinking Semi-Supervised Imbalanced Node Classification from Bias-Variance Decomposition

Abstract

This paper introduces a new approach to address the issue of class imbalance in graph neural networks (GNNs) for learning on graph-structured data. Our approach integrates imbalanced node classification and Bias-Variance Decomposition, establishing a theoretical framework that closely relates data imbalance to model variance. We also leverage graph augmentation technique to estimate the variance, and design a regularization term to alleviate the impact of imbalance. Exhaustive tests are conducted on multiple benchmarks, including naturally imbalanced datasets and public-split class-imbalanced datasets, demonstrating that our approach outperforms state-of-the-art methods in various imbalanced scenarios. This work provides a novel theoretical perspective for addressing the problem of imbalanced node classification in GNNs.

1 Introduction

Graphs are ubiquitous in the real world, encompassing social networks, financial networks, chemical molecules [22, 46, 9] and so on. Recently, graph neural networks (GNNs) [16, 35, 9] have shown exceptional proficiency in representation learning on graphs, facilitating their broad application in various fields. Nevertheless, the process of learning on graphs, analogous to its counterparts in computer vision and natural language processing, frequently encounters significant obstacles in the form of skewed or insufficient data, leading to the prevalent issue of class imbalance. It is undeniable that real-world data often exhibits biases and numerous categories, which entail data limitations and distribution shifts to be common. Furthermore, coupled with the fact that GNNs often feature shallow architectures to prevent "over-smoothing" or "information loss", training with such datasets leads to severe over-fitting problems in minor classes.

Designing GNN imbalanced learning schemes for graph-structured data poses unique challenges, unlike image data. Graph-structured data often requires consideration of the data's topology and environment, and empirical evidence [6, 24, 31] shows that topological asymmetries can also affect the model's performance. However, due to the extremely irregular and structured nature of the data, quantifying and solving topological asymmetry is difficult and computationally intensive. As a result, existing methods such as oversampling [28, 50, 24] and loss function engineering [6, 31] are problematic for achieving satisfactory outcomes. In depth, the modeling approach for data personalization exhibits very poor scalability and generalization capability. Furthermore, the prevalence of this problem has prompted the community to seek a more effective framework for imbalanced learning. Thus, a more fundamental and theoretical perspective is urgently needed when considering the imbalance node classification problem.

Following the idea, in this work, we propose a novel viewpoint to understand graph imbalance through the lens of Bias-Variance Decomposition . The Bias-Variance Decomposition [1, 2, 23, 45] has long been applied in terms of model complexity and fitting capacity. Our theoretical analyses have confirmed the relationship between model variance and the degree of dataset imbalance, whereby an imbalanced training set can lead to poor prediction accuracy due to the resulting increase in variance. Furthermore, we conducted some experiments on real-world dataset and the results demonstrate a significant relationship between the variance and the imbalance ratio of the dataset. As far as we know, we were the first to establish a connection between imbalanced node classification and model variance, which conducts theoretical analysis in this field.

Moreover, we have devised a regularization term for approximating the variance of model, drawing on our theoretical analysis. The main challenge of this idea lies in estimating the expectation across different training sets. Our key insight is to leverage graph data augmentation to model the different training sets in the distribution, which has not been explored before. Our empirical evaluations have consistently demonstrated that our algorithm, leveraging three basic GNN models [16, 35, 9], yields markedly superior performance in diverse imbalanced data settings, surpassing the current state-of-the-art methods by a substantial margin. Notably, we have conducted meticulous experiments on two naturally imbalanced datasets, which serve as the realistic and representative benchmarks for real-world scenarios.

Our contribution can be succinctly summarized as follows: (i) We are the first to integrate imbalanced node classification and Bias-Variance Decomposition . Our work establishes the close relationship between data imbalance and model variance, based on theoretical analysis. (ii) Moreover, we are the first to conduct a detailed theoretical analysis for imbalanced node classification domain. (iii) Our principal insight is to leverage graph data augmentation as a means of representing the varied training sets that lie within the distribution, whilst simultaneously designing a regularization term that approximates the model's variance. (iv) We conduct exhaustive tests on multiple benchmarks, such as the naturally imbalanced datasets and the public-split class-imbalanced datasets. The numerical results demonstrate that our model consistently outperforms state-of-the-art machine learning methods and significantly outperforms them in heavily-imbalanced scenarios.

2 Preliminaries

2.1 Notations

We concentrates on the task of semi-supervised imbalanced node classification within an undirected and unweighted graph, denoted as G = ( V , E , V L ) . V represents the node set, E stands for the edge set, and V L ⊂ V denotes the set of labeled nodes. The set of unlabeled nodes is denoted as V U := V \ V L . The feature matrix is X ∈ R n × f , where n is the number of nodes and f is the feature dimension. The adjacency matrix is denoted by A ∈ { 0 , 1 } n × n . Let N ( v ) represent set of adjacent nodes to node v . The labeled sets for each class are denoted as ( C 1 , C 2 , . . . , C k ) , where C i represents the labeled set for class i . The imbalance ratio ρ , is defined as ρ = max i | C i | / min i | C i | .

2.2 Graph Dataset Augmentations

To craft diversified perspectives of the graph dataset, we deploy advanced graph augmentation methodologies, leading to the formation of ˜ G = ( ˜ A , ˜ X ) and ˜ G ′ = ( ˜ A ′ , ˜ X ′ ) . Elements such as node attributes and connections within the foundational graph are selectively obfuscated. These augmented perspectives are subsequently channeled through a common Graph Neural Network (GNN) encoder, symbolized as f θ : R n × n × R n × f → R n × d , in order to distill compact node-centric embeddings: f θ ( ˜ A , ˜ X ) = h ∈ R n × d and f θ ( ˜ A ′ , ˜ X ′ ) = h ′ ∈ R n × d .

2.3 Related Work

Semi-Supervised Imbalanced Node Classification. Numerous innovative approaches[28, 40, 50, 19, 25, 6, 24, 31, 43] have been proposed to tackle the difficulties arising from imbalanced node classification on graph data. GraphSMOTE [50] employs the SMOTE [5] algorithm to perform interpolation at the low dimensional embedding space to synthesize the minority nodes, while ImGAGN [25] introduces GAN [8] for the same purpose. Another work GraphENS [24] synthesizes

mixed nodes by combining the ego network of minor nodes with target nodes. To address topology imbalance in imbalanced node classification task, ReNode [6] adjusts node weights based on their proximity to class boundaries. TAM [31] leverages local topology that automatically adapts the margins of minor nodes. Due to spatial constraints, we provide a comprehensive exposition of other relevant literature in Appendix A.

3 Theory

3.1 Theoretical Motivation

This section presents a succinct overview of the classical Bias-variance Decomposition [1, 2, 23, 45] and its application to semi-supervised node classification. Furthermore, we illustrate the influence of imbalance on classification results, which subsequently amplifies the variance.

Bias-variance Decomposition. Let x ∈ X and y ∈ Y denote the input and label, respectively. Consider an underlying mapping f : X → Y and the label can be expressed as y = f ( x ) + ϵ , where ϵ is some noise. Given a training set D = ( x i , y i ) , we train a model ˆ f ( x ) = ˆ f ( x ; D ) using the samples in D .

Definition 1 Bias-Variance Decomposition is that the expected predicted error of f can be decomposed into Eq(1),

It is important to note that this expectation is taken with respect to both the training sets D and the predicted data x . The bias term reflects the model's ability to fit the given data, while the variance term indicates the stability of the model's results with different training sets. In other words, the variance describes the model's generalization performance.

Next, we consider the specific formulation of variance in the setting of semi-supervised node classification on graph. First, we make some assumptions to simplify the analysis.

Assumptions. We make two assumptions in our approach. Firstly, we assume that the node embeddings h i of node x i extracted by a graph neural network for nodes belonging to class i follow a multivariate normal distribution h i ∼ N ( µ i , Λ i ) , where Λ i is a diagonal matrix for all i = 1 , 2 , . . . , c . Here, let ϵ i = h i -µ i which follows the distribution ϵ i ∼ N (0 , Λ i ) .

Secondly, we consider a simple classifier that estimates the probability of a node belonging to a particular class based the distance h T C i . Here, C i = C i ( D ) = 1 n i ( h i 1 + · · · + h i n i ) is the average of labeled node embedding in training set D , where n i is the number of nodes in class i . C i follows the distribution C i ∼ N ( µ i , 1 n i Λ i ) . Similarly, we denote e i = C i -µ i and it follows that e i ∼ N ( 0 , 1 n i Λ i ) .

Variance for Semi-supervised Node Classification. Under the above assumption, we can write the variance generated by different sampling training set explicitly. For a node x that belongs to class j , the variance can be written:

The last equation is derived from the variance of multivariate normal distribution. The variance over the whole graph ∑ c i =1 E x [Var( x )] = ∑ c i =1 E x [ 1 n i h T ( x )Λ i h ( x ) ] is the expectation of Equation 2 on node x .

Variance and Imbalance. In Equation 2, we notice that the variance of a specific class i is proportional to 1 n i . This reveals a relation between imbalance and variance. If we assume E x [ h T ( x )Λ i h ( x )] is the same for different i , then the following theorem holds:

Theorem 1 Under the condition that ∑ i n i is a constant, the variance ∑ c i =1 E x [ 1 n i h T ( x )Λ i h ( x ) ] reach its minimum when all n i equal.

The proof is include in Appendix B.1. As the ratio of imbalance increases, the minority class exhibits a smaller sample size n i , which consequently makes a greater contribution to the overall variance. This result gives an new perspective of explanation about why the model shows poor performance under imbalance training, that variance will increase as the training set becomes imbalanced. To substantiate our analysis, we conducted an experimental study to show the relation between imbalance and the variance. The result is shown in Figure 1.

1

10 1

10 2

0.016

0.018

0.020

0.022

(a)

CiteSeer-GCN

1

10 1

10 2

0.0150

0.0175

0.0200

0.0225

0.0250

0.0275

0.0300

(b)

CiteSeer-GAT

1

10 1

10 2

0.022

0.024

0.026

0.028

0.030

0.032

0.034

(c)

CiteSeer-SAGE

3.2 Variance Regularization

Optimize Variance during Training. Both our analysis and the experimental results show that the increase of variance is a reason for the deterioration of performance for imbalance training set. Therefore, We propose to estimate the variance and use it as a regularization during training. We decompose the model into two parts. A GNN E is utilized as a feature extractor, and a classifier P predict the probability based on the feature extracted by GNN. We define the variance of P conditioning on E of as Equation 2 evaluated for a fixed GNN E . Then we can take this as regularization. By optimizing this term, we allow the model to automatically search for the feature extractor that yields the smallest variance during the optimization process.

Estimate the Expectation with Labeled Nodes. The most challenging aspect lies in the estimation of variance, as we lack access to other training sets. Therefore, we propose Lemma 1 to estimate the variance on training set with the variance on labeled data. The proof is included in Appendix B.2.1.

Lemma 1 Under the above assumption for h i ∼ N ( µ i , Λ i ) , C i ∼ N ( µ i , 1 n i Λ i ) , minimizing the ∑ c i =1 E x [Var( x )] is equivalent to minimizing Equation 3:

Estimate the Expectation with Unlabeled Nodes. Lemma 1 allows us to replace the sampling on training set with sampling on labeled node pairs ( h i 1 , h i 2 ) of class i . However, to evaluate Equation 3 it still needs to have access to embedding pair ( h i 1 , h i 2 ) that belong to the same class i . Since labels are scarce, such node pairs is difficult to obtain. To make our algorithm more practical, a way of utilizing unlabeled nodes for estimating Equation 3 is required. We accomplish this goal through two steps. In the first step, we use graph augmentation to obtain pseudo embedding pair ( h 1 , h 2 ) . Graph augmentation is a typical used technique in graph contrastive learning. It generate new view G ′ of a graph G by adding noise to the graph structure. For a node x in two different views G,G ′ , the resulting embedding h, h ′ can be seen as that of two different nodes. Thus, they can be used to replace true embedding pair in Equation 3.

However, these pseudo node pairs give no information about which class it belongs to. Thus it prohibits us from assign proper 1 √ 2 n i for different i in Equation 3. It is such coefficients that compensate on the minority class, so it's crucial to reintroduce the message about class number for constructing variance regularization. Therefore, in the second step, we use the class center C i to replace h in Equation 3, as shown in Equation 4. Since C i = µ i + e i and the variance of e i is proportional to 1 n i , using Equation 4 to estimate Lemma 1 can apply the similar compensatory for the minority class. More mathematical details are presented in Appendix B.2.1.

From the Viewpoint of Graph Sampling. We notice that an alternative perspective can be adopted to interpret this regularization term. Given a graph data G , we should acknowledge that there exists some noise in its structure or feature, as a result of measuring. This randomness also attribute to the variance of classification. Assume each graph G in our dataset is a sample drawn from an underlying true graph ˜ G . Denote the classification result trained on G as f ( x ; G ) . Then the variance on G can be written as

Taking f ( x ) = h T ( x ) C , it is straitforward to show that Equation 4 and Equation 5 are same.

Despite the algorithm obtained through this new interpretation being identical to the previous one, it is worth noting that the two interpretations differ significantly. The former incorporates information from unlabeled nodes and assumes the variance originates from selecting different nodes as the training set. Graph augmentation facilitates the sampling of node pairs belonging to the same class. The latter, on the other hand, considers the same training set, but the variance originates from the entire graph's information. Graph augmentation is used to simulate the process of sampling the training graph from the underlying true graph.

4 The Final Algorithm

In this section, we introduce our ReVar ( Re gularize Var iance) framework which is based on previous theoretical frameworks for optimizing model variance. We explain our innovative approach to approximating the model's variance and using it as a regularization term in our algorithm in Section 4.1. In Section 4.2, we discuss how we integrate the GCL framework with a new contrastive learning term designed for semi-supervised learning tasks. Lastly, we present the final formulation of our optimization objective in Section 4.3, which optimizes both the cross-entropy loss function and our variance-constrained regularization term, providing a comprehensive approach to improve the performance of semi-supervised imbalanced node classification tasks.

4.1 Variance-constrained Optimization with Adaptive Regularization

To initiate, we elucidate the methodology underpinning the computation of Equation 4. Initially, we calculate the class center C i corresponding to each class. Following this, we systematically construct

𝒕𝒓𝒂𝒏𝒔𝒇𝒐𝒓𝒎 ′

𝒕𝒓𝒂𝒏𝒔𝒇𝒐𝒓𝒎

𝐺 =

𝑋, 𝐴

෨

𝐺 =

෨

𝑋,

ሚ

𝐴

෩

𝑃𝑖

෨

𝑃 𝑖

′

෨

𝐺 ′ =

෨

𝑋 ′

,

ሚ

𝐴 ′

෨

𝐺 ′ =

෨

𝑋 ′

,

ሚ

𝐴 ′

Intra-Class Aggregation

Regularization

Class center

dimension 1

dimension 2

Unlabeled nodes

Labeled nodes

Inter-view  contrast

(b) Intra-view and inter-view regularization

(Corresponding to

𝓛𝑰𝑹

)

label probability

distribution

𝑓 𝜃

෨

𝐺 =

෨

𝑋,

ሚ

𝐴

shared

Variance-constrained Optimization

with Adaptive Regularization

𝑓 𝜃

𝑦𝑖

dimension 2

Class center

dimension 1

Unlabeled nodes

Labeled nodes

label probability

distribution

CE

(a) Graph augmentations

(c) Variance regularization

(Corresponding to

𝓛𝑽𝑹

)

Unlabeled nodes

Labeled nodes

𝑓 𝜃

GNN encoder

a probability distribution reference matrix, denoted by S := [ C 1 ; C 2 ; . . . ; C k ] . It's worth noting that the assemblage of class centers is represented by C := ( C 1 ; C 2 ; . . . ; C k ) .

Given a node i with its associated embedding h i , the label probability distribution π i pertaining to node i is ascertainable via the subsequent equation:

In the aforementioned equation, π j i signifies the j -th dimensional component of π i , sim( · , · ) denotes the function to compute the cosine similarity between a pair of vectors, and τ is a specified temperature hyperparameter.

Subsequent to this, for every node i ∈ V , we determine the label probability distribution. Distinctively, ˜ π i and ˜ π ′ i are derived from ˜ G and ˜ G ′ , respectively. The former acts as the prediction distribution, whereas the latter represents the target distribution. Our approach then aims to minimize the crossentropy between these two distributions.

It's paramount to underscore that for each labeled node within ˜ G ′ , its one-hot label vector is directly assigned as ˜ π ′ i = y i for all i ∈ V L . This diverges from the procedure of deducing the predicted class distribution as described in Equation 6. This methodology is purposed to optimally leverage the extant label information.

However, a simplistic minimization of the aforementioned loss might engender confirmation bias. This is attributed to the potentially inaccurate ˜ π ′ i estimated for the unlabeled nodes (i.e., V U ), an aspect that can deleteriously impact the efficacy of pseudo-labeling-oriented semi-supervised methodologies. To address this concern, we introduce a confidence-based label-guided consistency regularization [41, 30, 48], as detailed below:

where V conf = { v i | 1 { max(˜ π ′ i ) >v } = 1 , ∀ i ∈ V U } represents the set of nodes with confident predictions, v is the threshold for determining whether a node has a confident prediction, and 1 { ... } is an indicator function.

In the context of confidence determination for ˜ π ′ i , we employ a criterion wherein a value is appraised as confident if its maximum constituent surpasses a stipulated threshold. We postulate that a heightened threshold, represented by v , acts as a safeguard against confirmation bias. This stratagem ensures that only high-caliber target distributions, specifically ˜ π ′ i , have a pivotal influence on Equation 7.

4.2 Intra-Class Aggregation Regularization

In this section, we propose an extension to the concept of graph contrastive learning that emphasizes the invariance of node representations in semi-supervised scenarios. To achieve this, we partition the nodes into two groups: labeled and unlabeled. Specifically, for the unlabeled nodes, we learn node representation invariance through the use of positive examples exclusively. For each labeled node in both views, its positive sample no longer includes itself in the other view, but instead encompasses all labeled nodes in both views belonging to the same class. Through this approach, labeled nodes belonging to the same class can be aggregated, enabling the model to learn that representation invariance now applies not to a single node, but to the complete class representation.

̸

where N all = ∑ k i =1 | C i | ( | C i | -1) , h i and h ′ i represent feature embedding in ˜ G and ˜ G ′ respectively for node i . To the best of our knowledge, we are the first to introduce label information into the contrastive learning paradigm to obtain invariant representations within the context of intra-class variation and address the challenge of semi-supervised imbalanced node classification. Notably, our approach only utilizes positive examples, which significantly reduces the model's training complexity and enhances its scalability and generalization capabilities.

4.3 Objective Function

To derive our ultimate objective function, we incorporate both L VR and L IR . These are weighted by coefficients λ 1 and λ 2 , respectively. Formally, the amalgamation can be represented as:

Additionally, we introduce the cross-entropy loss denoted by L sup , which is established over a collection of labeled nodes, denominated as V L .

5 Experiment

5.1 Experimental Setups

Datasets and Baselines. We have demonstrated the efficacy of our method on five commonly used benchmark datasets across various imbalance scenarios. For the conventional setting ( ρ =10) of imbalanced node classification in [50, 24, 31], we conducted experiments on Cora, CiteSeer, Pubmed, and Amazon-Computers. More precisely, we select half of the classes as the minority classes and randomly convert labeled nodes into unlabeled ones until the training set reaches an imbalance ratio of ρ . To be more precise, for the three citation networks, we adopt the standard splits proposed by [44] as our initial splits, with an initial imbalance ratio of ρ . In order to better reflect real-world scenarios, we conducted representative experiments on the naturally imbalanced datasets Amazon-Computers and Coauthor-CS. For this setting, we utilize random sampling to construct a training set that adheres to the true label distribution of the entire graph. The comprehensive experimental settings, including the evaluation protocol and implementation details of our algorithm, are explicated in Appendix E. For baselines, we evaluate our method against classic techniques, including cross-entropy loss with re-weighting [13], PC Softmax [12], and Balanced Softmax [26], as well as state-of-the-art methods for imbalanced node classification, such as GraphSMOTE [50], GraphENS [24], ReNode [6], and TAM [31]. See Appendix E for implementation details of the baselines.

GNN Encoder Selection. As the foundational structures for our GNN encoder, we harness architectures such as GCN [16], GAT [35], and GraphSAGE [9].

5.2 Main Results

Experimental Results Under Traditional Semi-supervised Settings. Table 1 presents the averaged balanced accuracy (bAcc.) and F1 score, along with their standard errors, obtained by the baselines and ReVar on three class-imbalanced node classification benchmark datasets, where ρ = 10 . The outcomes showcase the superiority of ReVar in comparison to the existing methods. Our proposed technique exhibits a consistent and significant outperformance of the state-of-the-art approaches across all three datasets and three base models. In all cases, ReVar achieves a decisive advantage that underscores its efficacy in addressing the challenge of class imbalance in node classification. Due to space limitations, the experimental results and analysis on Cora are presented in the Appendix C.

Experimental Results On Naturally Imbalanced Datasets. Our model was evaluated on two naturally imbalanced datasets, Coauthor-CS ( ρ ≈ 41 . 0 ) and Amazon-Computers ( ρ ≈ 17 . 7 ), both of which exhibit imbalanced unlabeled data (as indicated in Table 7). The construction of the training, validation, and testing sets is detailed in Appendix E. Experimental results are presented in Table

4 and Table 2. Notably, ReVar outperformed other methods consistently on these two datasets, a significant finding given the challenging nature of the imbalanced data in these datasets.

5.3 Further Analysis of ReVar

Analysis of Each Component in the Loss Function. As shown in Figure 3(a), each component of our loss function can bring performance improvements. In particular, in all three settings in the figure, our full loss achieves the best F1 scores. In all cases, VR loss improves the performance of the model, proving our hypothesis that can approximate the variance of the model to improve the performance of the model. More analysis is presented in Appendix D.

SAGE

GAT

GCN

Model

60

65

70

75

80

F1-score(%)

Sup

Sup+IR

Sup+VR

Sup+IR+VR

(a)

Computers-Semi

0.25 1.25 2.25 3.25 4.25 5.25 6.25

1

60

63

66

69

72

75

F1-score(%)

SAGE

GAT

GCN

(b)

Pubmed-

λ

1

0.5

1.5

2.5

3.5

4.5

5.5

6.5

2

60

63

66

69

72

75

78

F1-score(%)

SAGE

GAT

GCN

(c)

PubMed-

λ

2

0

50

100

150

200

Epoch

6

4

2

0

2

4

6

8

Loss

Our SAGE

Our GAT

Our GCN

Vanilla SAGE

Vanilla GCN

Vanilla SAGE

Vanilla GCN

(d)

CiteSeer-Loss

Sensitive to Hyperparameters λ 1 and λ 2 . The two intensity terms, λ 1 and λ 2 , corresponding to the L VR and L IR , respectively, is depicted in Figure 3(b), 3(c). We explore the impact of perturbations on model performance when one of the two intensity terms is held constant while the other is varied. We believe that a smaller value of λ is inadequate to reflect the regularization of ReVar. Instead, suitable values of λ can significantly enhance the performance of ReVar. More analysis is presented in Appendix D.

Analysis of Test Loss and F1 Score Curves. The present study includes a comparison of ReVar and Vanilla models(only use cross-entropy) in terms of their loss and F1 score. As shown in Figure 3(d), various models were evaluated. The ReVar model, which incorporates additional L VR and L IR components, demonstrated a faster convergence of loss, as evidenced by the larger slope in Figure 3(d). More analysis is presented in Appendix D.

6 Conclusion and Future Work

This paper presents a novel approach to address imbalanced node classification. Our method integrates imbalanced node classification and the Bias-Variance Decomposition framework, establishing a theoretical foundation that closely links data imbalance to model variance. Additionally, we employ graph augmentation techniques to estimate model variance and design a regularization term to mitigate the impact of class imbalance. We conduct exhaustive testing to demonstrate superior performance compared to state-of-the-art methods in various imbalanced scenarios. Future work includes extending ReVar and its theories to the fields of computer vision and natural language processing. Moreover, we anticipate the emergence of more effective theories and algorithms to be developed and flourish.

Reproducibility Statements

The model implementation and data is released at https://github.com/yanliang3612/ReVar.

References

Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal. Reconciling modern machinelearning practice and the classical bias-variance trade-off. Proceedings of the National Academy of Sciences , 116(32):15849-15854, 2019.

Erica Briscoe and Jacob Feldman. Conceptual complexity and the bias/variance tradeoff. Cognition , 118(1):2-16, 2011.

Jiarui Cai, Yizhou Wang, and Jenq-Neng Hwang. Ace: Ally complementary experts for solving long-tailed recognition in one-shot. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 112-121, 2021.

Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. Learning imbalanced datasets with label-distribution-aware margin loss. Advances in neural information processing systems , 32, 2019.

Nitesh V Chawla, Kevin W Bowyer, Lawrence O Hall, and W Philip Kegelmeyer. Smote: synthetic minority over-sampling technique. Journal of artificial intelligence research , 16:321357, 2002.

Deli Chen, Yankai Lin, Guangxiang Zhao, Xuancheng Ren, Peng Li, Jie Zhou, and Xu Sun. Topology-imbalance learning for semi-supervised node classification. Advances in Neural Information Processing Systems , 34:29885-29897, 2021.

Yoav Freund and Robert E Schapire. A decision-theoretic generalization of on-line learning and an application to boosting. Journal of computer and system sciences , 55(1):119-139, 1997.

Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM , 63(11):139-144, 2020.

Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. Advances in neural information processing systems , 30, 2017.

Hui Han, Wen-Yuan Wang, and Bing-Huan Mao. Borderline-smote: a new over-sampling method in imbalanced data sets learning. In International conference on intelligent computing , pages 878-887. Springer, 2005.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision , pages 1026-1034, 2015.

Youngkyu Hong, Seungju Han, Kwanghee Choi, Seokjun Seo, Beomsu Kim, and Buru Chang. Disentangling label distribution for long-tailed visual recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 6626-6636, 2021.

Nathalie Japkowicz and Shaju Stephen. The class imbalance problem: A systematic study. Intelligent data analysis , 6(5):429-449, 2002.

Bingyi Kang, Saining Xie, Marcus Rohrbach, Zhicheng Yan, Albert Gordo, Jiashi Feng, and Yannis Kalantidis. Decoupling representation and classifier for long-tailed recognition. In International Conference on Learning Representations , 2019.

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014.

Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907 , 2016.

Bolian Li, Baoyu Jing, and Hanghang Tong. Graph communal contrastive learning. In The Web Conference , 2022.

Xu-Ying Liu, Jianxin Wu, and Zhi-Hua Zhou. Exploratory undersampling for class-imbalance learning. IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics) , 39(2):539550, 2008.

Yang Liu, Xiang Ao, Zidi Qin, Jianfeng Chi, Jinghua Feng, Hao Yang, and Qing He. Pick and choose: a gnn-based imbalanced learning approach for fraud detection. In Proceedings of the Web Conference 2021 , pages 3168-3177, 2021.

Zhining Liu, Pengfei Wei, Jing Jiang, Wei Cao, Jiang Bian, and Yi Chang. Mesa: boost ensemble imbalanced learning with meta-sampler. Advances in Neural Information Processing Systems , 33:14463-14474, 2020.

Aditya Krishna Menon, Sadeep Jayasumana, Ankit Singh Rawat, Himanshu Jain, Andreas Veit, and Sanjiv Kumar. Long-tail learning via logit adjustment. arXiv preprint arXiv:2007.07314 , 2020.

Mohammadreza Mohammadrezaei, Mohammad Ebrahim Shiri, and Amir Masoud Rahmani. Identifying fake accounts on social networks based on graph analysis and classification algorithms. Security and Communication Networks , 2018, 2018.

Brady Neal, Sarthak Mittal, Aristide Baratin, Vinayak Tantia, Matthew Scicluna, Simon LacosteJulien, and Ioannis Mitliagkas. A modern take on the bias-variance tradeoff in neural networks. arXiv preprint arXiv:1810.08591 , 2018.

Joonhyung Park, Jaeyun Song, and Eunho Yang. Graphens: Neighbor-aware ego network synthesis for class-imbalanced node classification. In International Conference on Learning Representations , 2021.

Liang Qu, Huaisheng Zhu, Ruiqi Zheng, Yuhui Shi, and Hongzhi Yin. Imgagn: Imbalanced network embedding via generative adversarial graph networks. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining , pages 1390-1398, 2021.

Jiawei Ren, Cunjun Yu, Xiao Ma, Haiyu Zhao, Shuai Yi, et al. Balanced meta-softmax for longtailed visual recognition. Advances in neural information processing systems , 33:4175-4186, 2020.

José A Sáez, Julián Luengo, Jerzy Stefanowski, and Francisco Herrera. Smote-ipf: Addressing the noisy and borderline examples problem in imbalanced classification by a re-sampling method with filtering. Information Sciences , 291:184-203, 2015.

Min Shi, Yufei Tang, Xingquan Zhu, David Wilson, and Jianxun Liu. Multi-class imbalanced graph convolutional network learning. In Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence (IJCAI-20) , 2020.

Michael R Smith, Tony Martinez, and Christophe Giraud-Carrier. An instance level analysis of data complexity. Machine learning , 95(2):225-256, 2014.

Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raffel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li. Fixmatch: Simplifying semisupervised learning with consistency and confidence. Advances in neural information processing systems , 33:596-608, 2020.

Jaeyun Song, Joonhyung Park, and Eunho Yang. Tam: Topology-aware margin loss for classimbalanced node classification. In International Conference on Machine Learning , pages 20369-20383. PMLR, 2022.

Kaihua Tang, Jianqiang Huang, and Hanwang Zhang. Long-tailed classification by keeping the good and removing the bad momentum causal effect. Advances in Neural Information Processing Systems , 33:1513-1524, 2020.

Yuchun Tang, Yan-Qing Zhang, Nitesh V Chawla, and Sven Krasser. Svms modeling for highly imbalanced classification. IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics) , 39(1):281-288, 2008.

Junjiao Tian, Yen-Cheng Liu, Nathaniel Glaser, Yen-Chang Hsu, and Zsolt Kira. Posterior re-calibration for imbalanced datasets. Advances in Neural Information Processing Systems , 33:8101-8113, 2020.

Petar Veliˇ ckovi´ c, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. arXiv preprint arXiv:1710.10903 , 2017.

Petar Veliˇ ckovi´ c, William Fedus, William L Hamilton, Pietro Liò, Yoshua Bengio, and R Devon Hjelm. Deep graph infomax. 2018.

Jianfeng Wang, Thomas Lukasiewicz, Xiaolin Hu, Jianfei Cai, and Zhenghua Xu. Rsg: A simple but effective module for learning imbalanced datasets. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 3784-3793, 2021.

Tong Wang, Yousong Zhu, Chaoyang Zhao, Wei Zeng, Jinqiao Wang, and Ming Tang. Adaptive class suppression loss for long-tail object detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 3103-3112, 2021.

Xudong Wang, Long Lian, Zhongqi Miao, Ziwei Liu, and Stella X Yu. Long-tailed recognition by routing diverse distribution-aware experts. arXiv preprint arXiv:2010.01809 , 2020.

Zheng Wang, Xiaojun Ye, Chaokun Wang, Jian Cui, and S Yu Philip. Network embedding with completely-imbalanced labels. IEEE Transactions on Knowledge and Data Engineering , 33(11):3634-3647, 2020.

Qizhe Xie, Zihang Dai, Eduard Hovy, Thang Luong, and Quoc Le. Unsupervised data augmentation for consistency training. Advances in neural information processing systems , 33:6256-6268, 2020.

Ziyu Xu, Chen Dan, Justin Khim, and Pradeep Ravikumar. Class-weighted classification: Trade-offs and robust approaches. In International Conference on Machine Learning , pages 10544-10554. PMLR, 2020.

Liang Yan, Shengzhong Zhang, Bisheng Li, min zhou, and Zengfeng Huang. UNREAL: Unlabeled nodes retrieval and labeling for heavily-imbalanced node classification, 2023.

Zhilin Yang, William Cohen, and Ruslan Salakhudinov. Revisiting semi-supervised learning with graph embeddings. In International conference on machine learning , pages 40-48. PMLR, 2016.

Zitong Yang, Yaodong Yu, Chong You, Jacob Steinhardt, and Yi Ma. Rethinking bias-variance trade-off for generalization of neural networks. In International Conference on Machine Learning , pages 10767-10777. PMLR, 2020.

Rex Ying, Ruining He, Kaifeng Chen, Pong Eksombatchai, William L Hamilton, and Jure Leskovec. Graph convolutional neural networks for web-scale recommender systems. In Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining , pages 974-983, 2018.

Yuning You, Tianlong Chen, Yongduo Sui, Ting Chen, Zhangyang Wang, and Yang Shen. Graph contrastive learning with augmentations. Advances in neural information processing systems , 33:5812-5823, 2020.

Bowen Zhang, Yidong Wang, Wenxin Hou, Hao Wu, Jindong Wang, Manabu Okumura, and Takahiro Shinozaki. Flexmatch: Boosting semi-supervised learning with curriculum pseudo labeling. Advances in Neural Information Processing Systems , 34:18408-18419, 2021.

Hengrui Zhang, Qitian Wu, Junchi Yan, David Wipf, and Philip S. Yu. From canonical correlation analysis to self-supervised graph neural networks. In Advances in Neural Information Processing Systems , 2021.

Tianxiang Zhao, Xiang Zhang, and Suhang Wang. Graphsmote: Imbalanced node classification on graphs with graph neural networks. In Proceedings of the 14th ACM international conference on web search and data mining , pages 833-841, 2021.

Yizhen Zheng, Shirui Pan, Vincent C. S. Lee, Yu Zheng, and Philip S. Yu. Rethinking and scaling up graph contrastive learning: An extremely efficient approach with group discrimination. In Advances in Neural Information Processing Systems , 2022.

Boyan Zhou, Quan Cui, Xiu-Shen Wei, and Zhao-Min Chen. Bbn: Bilateral-branch network with cumulative learning for long-tailed visual recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 9719-9728, 2020.

Zhi-Hua Zhou and Xu-Ying Liu. Training cost-sensitive neural networks with methods addressing the class imbalance problem. IEEE Transactions on knowledge and data engineering , 18(1):63-77, 2005.

Yanqiao Zhu, Yichen Xu, Feng Yu, Qiang Liu, Shu Wu, and Liang Wang. Deep graph contrastive representation learning. In International Conference on Machine Learning Workshop on Graph Representation Learning and Beyond , 2020.

Yanqiao Zhu, Yichen Xu, Feng Yu, Qiang Liu, Shu Wu, and Liang Wang. Graph contrastive learning with adaptive augmentation. In The Web Conference , 2021.

A Other related work

Imbalanced Learning in Traditional ML. Most real-world data is naturally imbalanced, presenting a significant challenge in training fair models that are not biased towards majority classes. To address this problem, various approaches have been commonly utilized. Ensemble learning [7, 18, 52, 39, 20, 3] combines the outputs of multiple weak classifiers. Data re-sampling methods [5, 10, 29, 27, 14, 37] aim to adjust the label distribution in the training set by synthesizing or duplicating samples from the minority class. Another approach tackles the imbalance issue by modifying the loss function, assigning larger weights to minority classes or adjusting the margins between different classes [53, 33, 4, 32, 42, 26, 38]. Post-hoc correction methods compensate for the imbalanced classes during the inference step, after completing the model training [14, 34, 21, 12]. Although these techniques have been extensively applied to i.i.d. data, extending them to graph-structured data poses non-trivial challenges.

Graph Contrastive Learning. Contrastive methods, which have proven effective for unsupervised learning in vision, have also been adapted for graph data. One notable approach is DGI [36], which presents a framework for unsupervised node-level representation learning that maximizes global mutual information. Other approaches, such as GRACE [54], GCA [55], and GraphCL [47], utilize augmented graphs to optimize the similarity between positive node pairs and minimize negative pairs. CCA-SSG [49] introduces an efficient loss function based on canonical correlation analysis, eliminating the need for negative samples. Incorporating community information, gCooL [17] enhances node representations and downstream task performance. GGD [51] simplifies the mutual information loss function by directly discriminating between two sets of node samples, resulting in faster computation and lower memory usage. These contrastive methods exhibit potential for improving unsupervised learning on graph data. However, it is important to note that our model operates within the context of semi-supervised learning, which significantly differs from the mechanisms employed by these models.

B Proofs

B.1 Proofs of Theorem 1

Theorem 1 Under the condition that ∑ c i =1 n i is a constant, the variance ∑ c i =1 E x [ 1 n i h T ( x )Λ i h ( x ) ] reach its minimum when all n i equal.

Proof The expression ∑ c i =1 E x [ 1 n i h T ( x )Λ i h ( x ) ] can be equivalently expressed as ∑ c i =1 1 n i E x [ h T ( x )Λ i h ( x ) ] . As previously assumed, the E x [ h T ( x )Λ i h ( x )] is the same for different i , which implies that our goal is to demonstrate that ∑ c i =1 1 n i is minimized when all n i are equal.

Let m be ∑ c i =1 n i . We wish to find the extremum of the sum of their reciprocals, which is given by

Using the inequality of arithmetic and harmonic means, we have

with equality if and only if n 1 = n 2 = · · · = n c . Rearranging, we get

with equality if and only if n 1 = n 2 = · · · = n c . Since m = n 1 + n 2 + · · · + n c , we have

with equality if and only if n 1 = n 2 = · · · = n c = m c . Therefore, when all the c numbers are equal, the sum of their reciprocals is minimized and given by S = c 2 m .

We can also use the method of Lagrange multipliers. Let f ( n 1 , n 2 , . . . , a c ) = 1 n 1 + 1 n 2 + . . . + 1 n c be the function that we want to extremize. Then, the Lagrangian is:

Taking the partial derivatives of L with respect to a i and λ , we get:

Setting these partial derivatives to zero, we get:

Thus, when the c numbers are equal, their reciprocal sum is minimized and is equal to c 2 m . Moreover, since S is a continuously differentiable function of n 1 , n 2 , . . . , n c , this extremum is a minimum.

Therefore, we have shown that the reciprocal sum of c numbers with a sum of m is minimized and equal to c 2 m when the c numbers are equal.

□

B.2 Proofs of Lemma 1 and More Details for Estimating the Expectation with Unlabeled Nodes (Section 3.2)

B.2.1 Proofs of Lemma 1

Lemma 1 Under the above assumption for h i ∼ N ( µ i , Λ i ) , C i ∼ N ( µ i , 1 n i Λ i ) , minimizing the ∑ c i =1 E x [Var( x )] is equivalent to minimizing Equation 17:

Proof Let ϵ i denote h i -µ i , which follows the distribution ϵ i ∼ N (0 , Λ i ) . Similarly, we denote e i = C i -µ i and it follows that e i ∼ N ( 0 , 1 n i Λ i ) .

We know so we get

Motivated by stochastic gradient descent, we opt to sample an e i on each occasion and compute, as opposed to directly calculating the expectation E D ⊂ G .

At present, we remain uncertain about how to sample the noise term e i associated with the class center C i . We put forth a proposition to estimate the class center noise e utilizing the feature noise ϵ . Under the supposition that when v ∈ C k , the j th element of the feature f ( v ) adheres to a Gaussian distribution:

Consequently, multiplying the noise term ϵ i in Equation 20 by 1 √ n i yields a random variable that exhibits an identical distribution to e i :

Moreover, how might we compute ε i ? In practical scenarios, we merely possess the feature h . However, we are able to calculate the disparity between two features, h i 1 and h i 2 , originating from the same class i :

Incorporating this into Equation 19, the loss is expressed as:

□

B.2.2 More Details for Estimating the Expectation with Unlabeled Nodes (Section 3.2)

Lemma 1 suggests that ∑ c i =1 E x [Var( x )] can be estimated by sampling labeled node pairs ( h i 1 , h i 2 ) from the same class i . However, due to the scarcity of data in the minority class, this can often be challenging. In Section 3.2, we address this issue by utilizing random data augmentation and defining nodes from different views as members of the same class, enabling us to sample node pairs for Equation 3 and perform embedding subtraction.

However, as we mentioned, the pseudo node pairs lack information regarding their class membership, making it impossible to assign appropriate 1 √ 2 n i values for different i in Equation 3. These coefficients play a crucial role in compensating for the minority class, making it imperative to reintroduce information about the class number when constructing variance regularization. Therefore, in the second step, we replace h in Equation 3 with the class center C i , as demonstrated in Equation 4. As C i = µ i + e i and the variance of e i is proportional to 1 n i , using Equation 4 to estimate Lemma 1 can provide similar compensatory effects for the minority class. We present the proof for the aforementioned propositions below.

Proof Firstly, the ∑ c i =1 E x [Var( x )] can be decomposed as follows:

(24)

The above Equation can be decomposed into three parts, namely T 1 , T 2 , and T 3 . Notably, each of these parts is associated with weight 1 n i . This observation supports Theorem 1 and Lemma 1, which suggest that the variance of a model is highly dependent on the distribution of the dataset's sample size, and that the extent of sample imbalance can significantly increase the model's variance.

Note that ϵ i ∼ N (0 , Λ i ) , e i ∼ N ( 0 , 1 n i Λ i ) , and so 1 √ n i ϵ i ∼ N ( 0 , 1 n i Λ i ) . So, if we use the class center C i to replace h in Equation 3, then we can get the following expression,

Like the previous Equation 24, Equation 25 can be decomposed into three parts: L 1 , L 2 , and L 3 . If we substitute the class center C i for h , we can observe that even though L 1 is insensitive to class imbalance, variables L 2 and L 3 , which respectively incorporate weights 1 n j and 1 √ n j , can still provide the following support: when optimizing the variance of the model, more attention can be given to the variance introduced by the minority classes, which also provides an innovative perspective for understanding class imbalance on graphs. □

C More Results

C.1 More results Under Traditional Semi-supervised Settings

In Table 3, we present the results of ReVar and other algorithms on Cora-Semi ( ρ =10). The experimental results demonstrate that TAM has a significant gain effect on all models (BalancedSoftmax, Renode, GraphENS), while GraphENS achieved the best performance among all current baselines on various models. By comparing with GraphSMOTE, we can conclude that addressing the node classification imbalance issue should focus on the topological characteristics of the graph. Finally, and most importantly, ReVar achieved state-of-the-art results under all basic models, which also verifies the superiority of our model in traditional imbalanced segmentation.

C.2 More results On Naturally Imbalanced Datasets

One of our major highlights is testing our algorithms on naturally imbalanced datasets, which is more representative of real-world scenarios. This means that in a semi-supervised setting, our training set and unlabeled data are both imbalanced, which poses a significant challenge for oversampling methods. In Table 4, we present the results of ReVar and all baselines on Computers-Random ,

demonstrating that our model still achieves state-of-the-art performance. In contrast, oversampling methods such as GraphSMOTE perform poorly on this dataset due to overfitting limitations. We believe that our model's ability to regularize variance from a bottom-up perspective contributes to its superior performance on naturally imbalanced graphs, while also demonstrating its powerful generalization capabilities.

D More Analysis

D.1 More Ablation Analysis for the Loss Function

SAGE

GAT

GCN

Model

20

30

40

50

60

70

F1-score(%)

Sup

Sup+IR

Sup+VR

Sup+IR+VR

(a)

CiteSeer-Semi

SAGE

GAT

GCN

Model

20

30

40

50

60

70

80

F1-score(%)

Sup

Sup+IR

Sup+VR

Sup+IR+VR

(b)

PubMed-Semi

Details of Experimental Setup. We conduct more experiments on imbalance ratio 10 ( ρ = 10 ) for the datasets CiteSeer and PubMed. For each GNN network, we add one loss regularization at a time, i.e. for GCN, we gradually add L IR , L VR from the initial L sup . The architecture we employed consisted of a 2-layers graph neural network (GNN) with 128 hidden dimensions, using GCN [16], GAT [35], and GraphSAGE [9]. The models were trained for 2000 epochs.

Analysis. Figure 4 reports more experiments for each component in the loss function. As we have seen, each component of the loss function can bring an improvement in the training effect. It is worth noting that in most cases L VR has a larger effect boost than L IR . We believe that L VR plays a more important role in the loss function.

D.2 More Experiments for Variance and Imbalance Ratio Correlation in Theorem 1

1

10 1

10 2

0.04

0.06

0.08

0.10

0.12

(a)

PubMed-GCN

1

10 1

10 2

0.04

0.06

0.08

0.10

0.12

0.14

0.16

(b)

PubMed-GAT

1

10 1

10 2

0.06

0.08

0.10

0.12

0.14

0.16

(c)

PubMed-SAGE

Details of Experimental Setup. In this experiment, the classifier is not MLP, which means that the probability of a node v being classified into class i depends on the distance of this node from the center of class i . We classify half of the classes as majority classes and the other half as minority classes. Initially, each class in majority classes and minority classes has 200 labeled nodes. To generate different imbalance scenarios ( ρ ), we reduce the number of labeled nodes in the minority class by 1 and add 1 to the number of labeled nodes in the majority class each time, so as to keep the number of samples in the training set consistent and eliminate the effect of the sample size variance in the training set. To calculate the variance of the model under a certain imbalance ratio ( ρ ), we repeat 20 times to randomly select different but the same number of training sets, and train 2000 epochs for each fixed training set. The architecture we employed consisted of a 2-layers graph neural network (GNN) with 128 hidden dimensions, using GCN [16], GAT [35], and GraphSAGE [9].

̸

Hypothesis testing. To test the correlation between the variance and imbalance ratio, we computed the Pearson correlation coefficient ( ρ ) between variance and imbalance ratio (log), as presented in the table below. The Pearson correlation coefficient, denoted as r , is a prevalent metric for gauging linear correlations. This coefficient lies between (-1) and (1) ,and reflects both the magnitude and direction of the correlation between two variables. An ( r ) value greater than (0.5) indicates a strong positive correlation. Furthermore, the p-value results from a hypothesis test with the null hypothesis ( H 0 : ρ = 0 ) and the alternative hypothesis ( H 1 : ρ = 0 ) , where represents the population correlation coefficient.

Given that the Pearson correlation coefficient between variance and imbalance ratio exceeds (0.5), and the p-value is below (0.01), we deduce that there is a robust correlation between variance and imbalance ratio. This relationship is statistically significant at the (0.01) significance level.

Analysis. Figure 5 reveals an intriguing pattern, as the majority of data points closely align along a regression curve exhibiting a positive slope. This observation provides substantial evidence that establishes a strong and direct linear relationship between the imbalance ratio ( ρ ) and the associated variance. Consequently, our findings provide compelling support for the hypothesis postulated in Theorem 1.

D.3 More Experiments for Hyperparameter Sensitivity Analysis of ReVar

0.25 1.25 2.25 3.25 4.25 5.25 6.25

1

40

43

46

49

52

55

58

61

64

67

F1-score(%)

SAGE

GAT

GCN

(a)

CiteSeer-

λ

1

0.5

1.5

2.5

3.5

4.5

5.5

6.5

2

40

43

46

49

52

55

58

61

64

67

F1-score(%)

SAGE

GAT

GCN

(b)

CiteSeer-

λ

2

0.25 1.25 2.25 3.25 4.25 5.25 6.25

1

60

63

66

69

72

75

78

81

F1-score(%)

SAGE

GAT

GCN

(c)

Computers-

λ

1

0.5

1.5

2.5

3.5

4.5

5.5

6.5

2

60

63

66

69

72

75

78

81

F1-score(%)

SAGE

GAT

GCN

(d)

Computers-

λ

2

Details of Experimental Setup. We implemented experiments on the two datasets, CiteSeer and Amazon-Computers. In this experiment, we evaluate the F1 score when we fix one λ while the other changes. The architecture we employed consisted of a 2-layer graph neural network (GNN) with 128 hidden dimensions, using GCN [16], GAT [35], and GraphSAGE [9]. The models were trained for 2000 epochs.

Analysis. In Figure 6, we present the sensitivity analysis of ReVar to two weight hyperparameters ( λ 1 , λ 2 ) in the loss function on the CiteSeer and Amazon-Computers. It is evident that ReVar exhibits varying degrees of sensitivity to hyperparameters on different datasets. Specifically, the model demonstrates lower sensitivity to a and b on CiteSeer, while the opposite is observed on Amazon-Computers. We postulate that the number of nodes (i.e., dataset size) may be a crucial factor in this discrepancy. Nevertheless, the optimal range of hyperparameter selection appears to be relatively narrow, indicating that significant efforts are not be required for model fine-tuning.

D.4 More Results of loss curve and F1 score

Details of Experimental Setup. We implemented experiments on three datasets PubMed, CiteSeer and Amazon-Computers. We compare our method ReVar and other vanilla models (only using cross-entropy and not using graph augmentation). The architecture we employed for ReVar and

0

50

100

150

200

Epoch

7.5

5.0

2.5

0.0

2.5

5.0

7.5

10.0

Loss

Our SAGE

Our GAT

Our GCN

Vanilla SAGE

Vanilla GCN

(a)

PubMed-Loss

0

50

100

150

200

Epoch

5

0

5

10

15

Loss

Our SAGE

Our GAT

Our GCN

Vanilla SAGE

Vanilla GCN

(b)

Computers-Loss

0

20

40

60

80

100

Epoch

0

50

100

F1-score(%)

Our SAGE

Our GAT

Our GCN

Vanilla SAGE

Vanilla GCN

Vanilla SAGE

Vanilla GCN

(c)

PubMed-F1 Score

0

20

40

60

80

100

Epoch

0

50

100

F1-score(%)

Our SAGE

Our GAT

Our GCN

Vanilla SAGE

Vanilla GCN

Vanilla SAGE

Vanilla GCN

(d)

Computers-F1 score

vanilla model consisted of a 2-layer graph neural network (GNN) with 128 hidden dimensions, using GCN [16], GAT [35], and GraphSAGE [9]. The models were trained for 2000 epochs.

Analysis. We plot the loss curve and F1 score with epoch on the PubMed and Computers datasets. By analyzing the plotted loss curves, we observe that our model demonstrates notable advantages, including faster convergence of the loss curves and improved performance across multiple datasets. For the F1 score, our model consistently outperforms alternative approaches, showcasing its superior capability in effectively capturing the complex relationships within the data and making accurate predictions.

E Elaboration of the experimental setup

In this section, we present our approach for constructing imbalanced datasets, describe our evaluation protocol, and provide comprehensive details on our algorithm as well as the baseline methods utilized in our study. To achieve this, we leverage sophisticated techniques and utilize advanced metrics to ensure the reliability and relevance of our results.

E.1 Construction for Imbalanced datasets

The detailed descriptions of the datasets are shown in Table 5. The details of label distribution in the training set of the five imbalanced benchmark datasets are in Table 7, and the label distribution of the full graph is provided in Table 7.

Imbalanced datasets construction for Traditional Semi-supervised Settings. For each citation dataset, we adopt the "public" split and apply a random undersampling technique to make the class distribution imbalanced until the target imbalance ratio ρ is achieved. Specifically, we convert the minority class nodes to unlabeled nodes in a random manner. Regarding the co-purchased networks Amazon-Computers, we conduct replicated experiments by randomly selecting nodes as the training set in each trial. We create a random validation set that contains 30 nodes in each class, and the remaining nodes are used as the testing set.

Imbalanced datasets construction for Naturally Imbalanced Datasets. For Computers-Random and CS-Random , we constructed a training set with equal proportions based on the label distribution of the complete graph (Amazon-Computers). The label distributions for the training sets of ComputersRandom and CS-Random are presented in Table 7. To achieve this, we employed a stratified sampling approach that ensured an unbiased representation of all labels in the training set.

E.2 Architecture of GNNs

We conducted evaluations using three classic GNN architectures, namely GCN [16], GAT [35], and GraphSAGE [9]. The GNN models were constructed with different numbers of layers, namely L = 1 , 2 , 3 . To enhance the learning process, each GNN layer was accompanied by a BatchNorm layer with a momentum of 0.99, followed by a PRelu activation function [11]. For the GAT architecture, we employed multi-head attention with 8 heads. We search for the best model on the validation set. The available choices for the hidden unit sizes were 64, 128, and 256.

E.3 Evaluation Protocol

We utilized the Adam optimizer [15] with an initial learning rate of 0.01 or 0.005. To manage the learning rate, we employed a scheduler based on the approach outlined in [31], which reduced the learning rate by half when there was no decrease in validation loss for 100 consecutive epochs. Weight decay with a rate of 0.0005 was applied to all learnable parameters in the model. In the initial training iteration, we trained the model for 200 epochs using the original training set for Cora, CiteSeer, PubMed, or Amazon-Computers. However, for Flickr, the training was extended to 2000 epochs in the first iteration. Subsequently, in the remaining iterations, we trained the models for 2000 epochs using the aforementioned optimizer and scheduler. The best models were selected based on validation accuracy, and we employed early stopping with a patience of 300 epochs.

E.4 Technical Details of ReVar

For all datasets Cora-Semi , CiteSeer-Semi , PubMed-Semi , Computers-Semi , Computers-Random and CS-Random , the learning rate η is chosen from { 0 . 0001 , 0 . 0005 , 0 . 001 , 0 . 005 , 0 . 01 , 0 . 1 } . The temperature hyperparameter τ is chosen from { 0 . 05 , 0 . 08 , 0 . 13 , 0 . 16 , 0 . 21 , 0 . 23 , 0 . 26 } . The threshold v is chosen from { 0 . 6 , 0 . 63 , 0 . 66 , 0 . 7 , 0 . 8 , 0 . 83 , 0 . 9 , 0 . 93 , 0 . 96 , 0 . 99 } . The factor λ 1 of L VR is chosen from { 0 . 25 , 0 . 35 , 0 . 5 , 0 . 85 , 1 , 1 . 5 , 2 , 2 . 15 , 2 . 65 , 3 } . The factor λ 2 of L IR is chosen from { 0 . 35 , 0 . 5 , 1 , 1 . 25 , 1 . 5 , 2 . 85 , 3 } . The factors of mask node properties and edges in ˜ G are chosen from { 0 . 4 , 0 . 45 , 0 . 5 , 0 . 6 , 0 . 65 , 0 . 7 } and { 0 . 4 , 0 . 45 , 0 . 5 , 0 . 55 , 0 . 6 , 0 . 65 , 0 . 7 } . The factors of mask node properties and edges in ˜ G ′ are chosen from { 0 . 1 , 0 . 15 , 0 . 2 , 0 . 3 , 0 . 4 , 0 . 45 } and { 0 . 1 , 0 . 15 , 0 . 2 , 0 . 3 , 0 . 35 , 0 . 4 , 0 . 45 } .

E.5 Technical Details of Baselines

For the GraphSMOTE method [50], we employed the branched algorithms whose edge predictions are discrete-valued, which have achieved superior performance over other variants in most experiments. Regarding the ReNode method [6], we search hyperparameters within the lower bound of cosine annealing, w min ∈ 0 . 25 , 0 . 5 , 0 . 75 , and the upper bound range, w max ∈ 1 . 25 , 1 . 5 , 1 . 75 , as suggested in [6]. The PageRank teleport probability was fixed at α = 0 . 15 , which is the default setting in the released codes. As for TAM [31], we performed a hyperparameter search for the coefficient of the ACM term, α ∈ 1 . 25 , 1 . 5 , 1 . 75 , the coefficient of the ADM term, β ∈ 0 . 125 , 0 . 25 , 0 . 5 , and the minimum temperature of the class-wise temperature, ϕ ∈ 0 . 8 , 1 . 2 , following the approach described in [31]. The sensitivity to the imbalance ratio of the class-wise temperature, δ , was fixed at 0.4 for all the main experiments. Additionally, consistent with [31], we implemented a warmup phase for 5 iterations, as we utilized model predictions for unlabeled nodes.

E.6 Configuration

All the algorithms and models are implemented in Python and PyTorch Geometric. Experiments are conducted on a server with an NVIDIA 3090 GPU (24 GB memory) and an Intel(R) Xeon(R) Silver 4210R CPU @ 2.40GHz.

F Algorithm

Algorithm 1 ReVar

Input: Imbalanced Graph G = ( V , E , V L ) , feature matrix X , adjacency matrix A , unlabeled set V U = V -V L , label matrix Y , feature matrix of labeled nodes and unlabeled nodes X L and X U , the number of classes k , the threshold v for determining whether a node has confident prediction, temperature hyperparameter τ , GNN model f θ , the classifier h θ , learning rate η , The total number T of epochs for model training. The sim( · , · ) computes the cosine similarity between two vectors, CE represents the cross-entropy loss function, and I ( · ) is an indicator function.

1: for t = 0 , 1 , . . . , T do

2: Generate two differently augmented views ˜ G = ( ˜ A , ˜ X) and ˜ G ′ = ( ˜ A ′ , ˜ X ′ ) from original graph G .

4: ˜ O ′ L ← f θ ( ˜ A ′ , ˜ X ′ L ) , ˜ O ′ U ← f θ ( ˜ A ′ , ˜ X ′ U )

5: %Component 1: Supervised Loss

6: L sup ← 1 2 CE ( h θ ( ˜ O L ) , Y) + 1 2 CE ( h θ ( ˜ O ′ U ) , Y)

7: %Component 2: Intra-Class Aggregation Regularization

̸

9: %Component 3: Variance-constrained Optimization with Adaptive Regularization

10: for i = 1, 2, . . . , k do

11: Compute the class centers ˜ C i and ˜ C ′ i for class i

12: end for

13: ▷ Obtain the Label Probability for Each Node

14: for i = 1 , 2 , . . . , | V | do

16:

17: end for

18: ▷ Eliminate Nodes with Low Confidence.

19: V conf ←{ i | I (max(˜ p ′ i ) > v ) = 1 , ∀ i ∈ V U }

20: ▷ Replace Labeled Node Prediction

21: for i = 1 , 2 , . . . , | V L | do

22: ˜ p ′ i ← Y i

23: end for

26: θ ← θ -η ( ∇L Training )

27: end for

G Notations

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies imbalanced node classification problem and propose  a novel perspective to understand graph imbalance via bias-variance decomposition.By leveraging graph data augmentation, the paper develops a regularization technique to approximate the model's variance. The effectiveness of the method is evaluated across multiple settings and datasets, where the proposed method largely outperforms the compared baselines.\n\nSTRENGTHS:\n- The idea is novel and has solid theoretical motivation.\n- Strong performance improvement + extensive experiments.\n- The paper is clearly written and well-structured.\n\nWEAKNESSES:\n- As someone not working directly in the same field, I think it would be very helpful if the related work section could be more comprehensive than its current shape. Plus, the method design in Sec. 4 shares similarity with many existing techniques. For example, the \"confidence-based label-guided consistency regularization\" in Eq. 8 has been widely used in standard semi-supervised learning (UDA, FixMatch, FlexMatch) for a long time. And the idea of intra-class aggregation for contrastive learning is also studied in Supervised Contrastive Learning paper. IMHO, it would be better if the paper could discuss the difference with them and tune down accordingly, which does not harm the originality of this paper.\n\nUnconfident comments:\n- I appreciate the analysis on that overall variance increases with the increase of imbalance ratio. But it is a bit surprising there was not any analysis like this before even in the field of long-tailed recognition. Could the authors please comment on this?\n\nQUESTIONS:\n- What value does \"v\" at Line 208 take? Is it a fixed value across the training or it is dynamically adapted? \n- What does it mean in Sec. 2.2 that \"We randomly mask node properties\"? Does it mean dropping nodes and the corresponding edges of the original graph at random? Is this a standard way of performing data augmentation on graph? What are other alternative data augmentation methods? It would be great if the authors could give more information on this either in Sec. 2.2 or in related work.\n\nLIMITATIONS:\nNone", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 1, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper is majoring on the imbalance problem in graph node classification. The authors confirmed the relationship between model variance and the degree of dataset imbalance by adopting the Bias-Variance Decomposition. Furthermore, they diverted a regularization term for approximating the variance of the model from the above theoretical analysis.\n\nSTRENGTHS:\nThis paper discovers the relation between imbalance and variance, estimates the variance with graph argumentation, and solves the imbalance problem by adopting two proposed regularization terms, which is an important problem in node classification.\nThe regularization term is theoretically derived from the Bias-variance Decomposition, and the author, for the first time, fits this theory into the field of imbalance by making two weak assumptions, which is a good contribution to this field.\n\nWEAKNESSES:\n- Overall, this paper contains many typos and abuse of the notation that may make the reader uncomfortable. I list those points in the limitations.\n- In this work, the authors make a strong assumption that all embeddings follow the multivariate normal distribution with the same sample variance ($h(x)^T\\Lambda^ih(x)$ in the paper) between each class. And the authors use this assumption to derive the core relation between variance and imbalance, then the regularization term. \nHowever, this assumption is strongly related to the quality of the extracted embeddings. \nOne important guarantee for this assumption is a well-trained feature extractor with a clear boundary between each class. \nHowever, at the initial stage, the model is not well-trained, meaning that a large variance of $h(x)^T\\Lambda^ih(x)$ in each class. \nIt is unclear how this will affect the power of regularization though the overall results look good, and the authors did not make a discussion of the above problem in the paper.\n\nQUESTIONS:\n1. Why do the authors assume node embeddings follow the multivariate normal distribution?\n2. Could the authors provide the significance of the results in Fig1 (e.g., using hypothesis testing)?\n\nLIMITATIONS:\nThe author could improve their manuscript by solving the below points and typos:\n- line 25-27: sentence should be shortened.\n- line 76: no definition for $N$, $F$ and $D$.\n- Section 3.1: Is there a lack of citations of Bias-variance Decomposition?\n- The term between Eq(2) and line 111, line 114 is different.\n- line 140: \"condition on\" should be replaced by \"conditioning on\".\n- line 208: indicator function is often denoted as $\\\\mathbf{1}_{ \\\\{ \\\\cdots \\\\} }$.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-01"}

comments: SUMMARY:
This paper focuses on semi-supervised imbalanced node
classification tasks. Specifically, the authors first establish a theoretical result that connects the imbalance ratio with the model variance and then propose a new regularization term related to the variance based on the graph augmentation technique. Experimental results show the effectiveness of the proposal.

STRENGTHS:
1. The authors give theoretical results that connect the imbalance ratio to the model variance. The results are insightful.
2. The experimental results show the proposal can achieve the SOTA performance on various class-imbalanced datasets.

WEAKNESSES:
1. The theoretical results rely on strong assumptions, which may be difficult to satisfy in real-world tasks.
2.  From Figure 3, it looks like the performance is highly influenced by the hyper-parameters. So how to determine the hyper-parameter should be discussed in the paper.
3. Moreover, I doubt that the performance of the proposal is not related to the theoretical results, it may rely on graph augmentation. Thus, more ablation studies should be conducted.

QUESTIONS:
1. How to determine the hyper-parameters in different datasets?
2. How to demonstrate the superiority of algorithm performance is related to theoretical results?

LIMITATIONS:
Yes.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-05

comments: SUMMARY:
This paper introduces a new approach to address the issue of class imbalance in graph neural networks (GNNs) for learning on graph-structured data. It also provides a novel theoretical perspective for addressing the problem of imbalanced node classification in GNNs.

STRENGTHS:
1 The article is well written and easy to understand.

3 The authors prove their claim by theoretical derivation.

4 The experimental results further corroborate the authors' view.

WEAKNESSES:
1 The use of L to represent the set of labeled nodes is still relatively rare in the definition of graphs, and the authors are advised to describe the definition involved further in the opening paragraph of the method.

2 Is the authors considering restructuring the article? Putting the theoretical derivation before the introduction of the method may not give the reader a good reading experience.

3 I am curious about this one: the authors point out that the full graph noise distribution is taken into account when sampling the graph in order to sample the training set that has approximate variance. However, if one considers graph learning in an open world, then the noise from the added data might be quite different from the original graph data.

4 In Table 1, the authors can consider further reporting the percentage of performance improvement compared to SOTA.

5 In Fig 3(a), it seems that the performance improvement from VR loss is not significant. Are you considering the Sup+VR combination? Also, can you further diet why VR loss gives a smaller boost to GAT?

QUESTIONS:
No more questions.

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-06

comments: SUMMARY:
This paper proposes a theory that relates data imbalance to model variance and designs a method to mitigate the bias of class imbalance. 
For the theory, this paper finds that the variance of each class is proportional to the invert of the number of samples in that class. 
For the method, this paper uses a regularization term to approximate the model variance and construct a varied training distribution with graph augmentation. The regularization term is added to the original loss function.
 Results on public-split datasets and naturally imbalanced datasets verify the proposal.

STRENGTHS:
This paper first connects the model variance and class imbalance in graph learning.

WEAKNESSES:
1. L_IR objective is more closely related to variance than contrastive learning. It would help to clarify the difference between your work and contrastive graph learning.

2. This paper does not discuss previous work on variance and imbalance in traditional imbalanced learning. Since the variance calculated in this paper does not use graph-specific settings, it is not unique to graph learning.

QUESTIONS:
1. Why can you assume that node embeddings follow a multivariate normal distribution?

2. Many elements of Figure 2 need descriptions. E.g.,  What do yellow and blue circles mean?

LIMITATIONS:
This paper does not discuss limitations. But I think the novelty and unique contribution is limited.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-09