IGL-Bench: Establishing the Comprehensive Benchmark for Imbalanced Graph Learning

Authors: Jiawen Qin, Haonan Yuan, Qingyun Sun, Lyujin Xu, Jiaqi Yuan, Pengfeng Huang, Zhaonan Wang, Xingcheng Fu, Hao Peng, Jianxin Li, Philip S. Yu
Primary area: datasets and benchmarks
Keywords: imbalanced graph learning, graph class-imbalance, graph topology-imbalance, comprehensive benchmark

Abstract:
Deep graph learning has gained grand popularity over the past years due to its versatility and success in representing graph data across a wide range of domains. However, the pervasive issue of imbalanced graph data distributions, where certain parts exhibit disproportionally abundant data while others remain sparse, undermines the efficacy of conventional graph learning algorithms, leading to biased outcomes. To address this challenge, Imbalanced Graph Learning (IGL) has garnered substantial attention, enabling more balanced data distributions and better task performance. Despite the proliferation of IGL algorithms, the absence of consistent experimental protocols and fair performance comparisons pose a significant barrier to comprehending advancements in this field. To bridge this gap, we introduce **IGL-Bench**, a foundational comprehensive benchmark for imbalanced graph learning, embarking on **17** diverse graph datasets and **24** distinct IGL algorithms with uniform data processing and splitting strategies. Specifically, IGL-Bench systematically investigates state-of-the-art IGL algorithms in terms of **effectiveness**, **robustness**, and **efficiency** on node-level and graph-level tasks, with the scope of class-imbalance and topology-imbalance. Extensive experiments demonstrate the potential benefits of IGL algorithms on various imbalanced conditions, offering insights and opportunities in the IGL field. Further, we have developed an open-sourced and unified package to facilitate reproducible evaluation and inspire further innovative research, available at: https://github.com/RingBDStack/IGL-Bench.

Decision: accept (spotlight)
Reviewer scores: 8, 8, 6

Reviews and discussion:
<thread>
creation date:
1729184750892

thread_type:
official_review

<thread_object>
creator: reviewer_mzgm

<content>
summary:
This paper introduces a benchmark for Imbalanced Graph Learning (IGL). IGL-Bench offers a unified framework for evaluating 24 IGL algorithms across 17 datasets, addressing both class- and topology-imbalance at node and graph levels. The authors conduct a rigorous evaluation on effectiveness, robustness, and efficiency, providing a comparative analysis of existing methods. Additionally, the paper presents an open-source package to facilitate reproducible research and encourage further development in the IGL field.

strengths:
-	The proposed IGL-Bench is well-constructed, it can provide a standardized testbed for future work, encouraging a more consistent evaluation process for IGL.
-	The paper is well-written and easy to follow. The authors' thorough analysis of the benchmarking results leads to interesting insights.

weaknesses:
Recently, several more general IGL methods have been proposed, which can be used independently or in combination with the IGL techniques discussed in this paper. For instance, [1] based on topological augmentation and automated loss engineering approach [2], both presented at ICML '24, are notable examples. It would be beneficial if IGL-bench could include these recent booster-type methods and provide related discussion.
[1] Liu, Zhining, et al. "Class-Imbalanced Graph Learning without Class Rebalancing." Forty-first International Conference on Machine Learning.
[2] Guo, Xinyu, et al. "Automated Loss function Search for Class-imbalanced Node Classification." Forty-first International Conference on Machine Learning.

questions:
1.	 How does the benchmark incorporate, or plan to incorporate, recent advances in IGL (see weaknesses), and evaluate whether these general approaches improve performance, robustness, and other metrics?
2.	Are there any plans to provide an API reference for IGL-bench? This would significantly enhance usability.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730512815366

thread_type:
official_review

<thread_object>
creator: reviewer_t4gw

<content>
summary:
This work presents a benchmark for imbalanced graph learning, encompassing 17 graph datasets and 24 imbalance-handling algorithms, along with a unified data processing pipeline, splitting strategy, and evaluation protocols. The benchmark addresses questions related to effectiveness in handling imbalance, robustness across different imbalance ratios, and efficiency in terms of time and space for both node-level and graph-level tasks. Additionally, an open-source codebase is provided to facilitate reproducible evaluations.

strengths:
1. Benchmarking imbalanced graph learning is important to track progress in the research community.
2. The four research questions posed are relevant and of interest to the community.
3. The paper is well-written and easy to follow.
4. The experiments are thorough and well-designed. The availability of open-source code enhances reproducibility and accessibility.

weaknesses:
1. For node-level class-imbalanced algorithms, it would be beneficial to include traditional methods for addressing class imbalance, such as reweighting or PC Softmax [1], to provide a comprehensive comparison. Additionally, reporting per-class F1 scores could offer more insights into model performance across majority and minority classes, enabling a more detailed analysis of effectiveness.

2. Some findings lack depth and do not offer clear insights. For instance, the observation that most algorithms perform poorly on the large-scale ogbn-arXiv dataset is presented without an explanation. It would be valuable to investigate and discuss the reasons for this poor performance, especially through additional experiments.

3. In Section 4.4, which assesses the efficiency of different models, the experiments are conducted on the Cora dataset, the smallest of the datasets used. This setup makes the time and space usage of the algorithms highly dependent on implementation details rather than their inherent algorithmic complexity. While the Appendix includes results for the ogbn-arXiv dataset, they cover only a subset of models, as other models encountered memory limitations. A theoretical analysis of the algorithms' complexities would provide a more meaningful comparison of their efficiency. Relying solely on runtime measurements can be misleading, as they may be influenced by implementation optimizations and do not necessarily reflect the algorithms' underlying efficiency.

4. On the ogbn-arXiv dataset, certain resampling algorithms, such as GraphENS and GraphSHA, run successfully, whereas others, like GraphSMOTE, encounter memory issues. It is unclear whether this is due to differences in algorithmic complexity or specific implementation choices. Given the similarity of these resampling approaches, clarifying whether the issue arises from inherent algorithm characteristics or particular implementation choices would be helpful.

[1] Disentangling label distribution for long-tailed visual recognition. CVPR. 2021.

questions:
See Weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730734970963

thread_type:
official_review

<thread_object>
creator: reviewer_6gyc

<content>
summary:
This paper addresses the gap in graph learning by introducing a benchmark named IGL-Bench for imbalanced graph learning. The benchmark covers 24 algorithms across 17 datasets, tackling node-level and graph-level tasks that are affected by class and topology imbalances. The benchmark includes standardized data processing, evaluation protocols, and metrics across different datasets. Evaluation on effectiveness, robustness, boundary clarity, and efficiency of IGL methods under imbalanced conditions are further carried out. The benchmark findings reveal key insights into the strengths and limitations of current IGL algorithms, particularly around scalability, robustness, and their dependency on graph topology properties.

strengths:
- Originality: The paper introduces a comprehensive benchmark for IGL to fill the gap in graph learning.
- Quality: Experiments valiadate the quality of the benchmark.
- Clarity: The paper is well written and organized. Easy for general audience to follow.
- Significance: The benchmark provides an easy platform for developing or comparing IGL methods, especially as graph learning expands into real-world, imbalanced datasets.

weaknesses:
- Complex visualizations: Certain visualizations, such as multi-metric plots across multiple imbalance ratios, could benefit from clearer explanations to aid interpretability for readers less familiar with the metrics.
- New dataset: No new dataset is introduced in the paper.

questions:
- Can the authors elaborate on the specific challenges encountered in standardizing data splits and evaluation protocols across such a diverse set of IGL methods?
- How does IGL-Bench handle algorithm hyperparameter tuning, particularly for methods that might be sensitive to different types of imbalance?
- Are there future plans to study the scalability within IGL-Bench, e.g., for larger, more complex graph datasets?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731960726521

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Global Response

comment:
We sincerely thank all reviewers for their thoughtful and constructive feedback on our paper. We are delighted that the reviewers recognize the significance of our benchmark to the Imbalanced Graph Learning (IGL) research community. Specifically, reviewers acknowledged:

- **Pioneering Benchmark.** The introduction of IGL-Bench as the first comprehensive benchmark for imbalanced graph learning (IGL)  fills a critical gap in the field. Establishing such a benchmark is essential for tracking progress within the research community. (Reviewer `6GyC`, Reviewer `t4gw`)
- **Thorough  and High-Quality Experimental Design.** The reviewers believe that the experimental design is thorough and high-quality, validating the effectiveness of the benchmark. The research questions and the authors’ insights posed are highly relevant and of interest to the research community, offering a comprehensive evaluation of IGL methods. (Reviewer `6GyC`, Reviewer `t4gw`, Reviewer `mZgm`)
- **Reproducibility and Accessibility.** The provision of an open-source package and  a standardized  platform facilitate reproducible evaluation,  encouraging a more consistent evaluation process for future work. (Reviewer `6GyC`, Reviewer `t4gw`, Reviewer `mZgm`)
- **Clarity and Presentation.** The reviewers believe that the paper is well-written, well-organized, and easy to follow, making it suitable for both general and specialized audiences. (Reviewer `6GyC`, Reviewer `t4gw`, Reviewer `mZgm`)

We have provided detailed responses to each reviewer’s comments. Additional experimental results and further clarifications are also included in the **[revised PDF (Appendix G)](https://openreview.net/pdf?id=uTqnyF0JNR#page=59)**.

Once again, we are grateful for the reviewers’ valuable insights, which have greatly helped us improve our work. We have made every effort to address all concerns thoroughly. Thank you for considering our rebuttal, and we kindly request that you reconsider the ratings of our paper, if possible.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732372164647

thread_type:
official_comment

<thread_object>
creator: area_chair_jumt

<content>
comment:
Dear Reviewers,

The authors have uploaded their rebuttal. Please take this opportunity to discuss any concerns you may have with the authors.

AC
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper introduces IGL-Bench, a benchmark for Imbalanced Graph Learning (IGL) that evaluates 24 algorithms across 17 datasets, addressing both class and topology imbalances in node- and graph-level tasks. The benchmark provides a unified data processing pipeline, evaluation protocols, and metrics, allowing researchers to assess the effectiveness, robustness, and efficiency of IGL methods. Additionally, the paper offers an open-source codebase to promote reproducibility and further research in the field.

The paper addresses a significant gap in graph learning by introducing a comprehensive benchmark for Imbalanced Graph Learning (IGL), essential for standardized evaluation in this rapidly evolving field. Through thorough experiments and analysis, it provides valuable insights into the strengths and limitations of current IGL algorithms, laying a solid foundation for future research. The open-source codebase and standardized evaluation framework enhance reproducibility, ensuring that the benchmark remains accessible and impactful for the broader research community. As a result, the reviewers unanimously voted to accept this paper.