Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper aims to solve both graph data imbalance and unnecessary model-level computation burden in a unified framework. Specifically, the authors first examine the challenges from theoretical and empirical perspectives. Then, the authors propose GraphDec, a novel data-model dynamic sparsity framework to address the challenges. Extensive experiments on multiple benchmark datasets demonstrate that GraphDec outperforms state-of-the-art methods.
strength_and_weaknesses: +This work is solid and insightful. The proposed framework investigates a new perspective in which sparsification can be utilized in both graph datasets and graph neural networks, which possess many advantages for real-world applications. For example, this work can be leveraged to detect important data points and conserve computational time.

+The authors conduct comprehensive ablation studies to evaluate each model components, which clearly show the contributions of every module and the impact of rebalancing strategies.
The introduced components such as dynamically downsampling important subsets, and pruning contrastive model are empirically proven to contribute distinct positive impacts to the full framework. Despite employing a subset of the data, the proposed method achieves similar or better performance compared to the methods that utilize the entire dataset. 

+ The overall structure is based on an intriguing hypothesis with proof. The authors provide a hypothesis to explain why downsample a subset can estimate the entire graph dataset; they infer that the subset with the gradient closest to the gradient of the entire dataset is the most important. They then apply this inspiration to develop GraphDec. In addition, the inspiration provided by this hypothesis is also relevant for tasks in other domains (e.g., computer vision, natural language processing).

Weakness/questions:
-What is the purpose of training the contrastive learning system? Does this method exclude other learning schemes and supervised loss? The paper can be improved by highlighting the unique effect, such as a concise summary of the motivation and empirical ablation experiments.

-Certain technical descriptions lack clarity for readers unfamiliar with coresets. In the experiment section, for instance, the word "prune" is sometimes used in a rather ambiguous manner. In the previous method section, it refers to the network, whereas in the experiment section, prune appears to be the "sampling" step of the dataset. The authors should provide additional information to explain the definitions of prune and sample on data and model or clarify that the terms are used in different sentences.


clarity,_quality,_novelty_and_reproducibility: The paper is well-written and the methodology is well-motivated. The designed model is novel and solid. The availability of code and data facilitates reproducibility.
summary_of_the_review: The paper address two significant challenges of imbalanced graph learning by designing a novel and effective model. The technical contribution is solid. The empirical explanations are interesting, and the experimental results are remarkable.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper addresses the class-imbalance problem in graph learning. The authors propose Graph Decantation (GraphDec) framework for learning balanced graph representation in a self-supervised manner. The key procedure of GraphDec is to select some informative samples from unbalanced node data or graph data during training based on the gradient norm. The idea is interesting and novel, and the method is generally applicable to either node-level or graph-level class-imbalance situations.
strength_and_weaknesses: Strength:
1. The idea of unifying dynamic informative subset selection and subnetwork selection during training is a first exploration and compelling. In addition, as a rebalancing method in this paper, data and model co-pruning do not require additional learnable weights or human-crafts hyperparameters. It can be used as a plug-and-play tool for other GNN models.

2. The model design is well-driven by an intriguing hypothesis. It theoretically examines the extent to which a subset of the training data can approximate the learning efficiency of the entire dataset. Based on the hypothesis, the methods and experiments also demonstrate the rebalanced subset's gradients can effectively approximate the entire imbalanced dataset. 

3. The authors suggest Graph Decantation (GraphDec) as a solution to the difficulties posed by training on huge class-imbalanced graph data. Utilizing the self-pruned contrastive framework's sensitivity on informative samples to gradually refine a more balanced subset is an innovative approach. Consequently, this subset can reduce learning computation and train the model contributing more balanced predictions for classifying graph data.

4. The experiments are sufficient and the improvement of GraphDec is significant. Particularly, the experiments studying the "Evolution of Sparse Subset by Scoring All Samples" are interesting, which supports the paper's motivation and inspires readers.


Weakness:
1. Does the idea of dynamically mining informative samples be applicable to some supervised frameworks? It will be nice to discuss the effect of applying the self-supervised learning (SSL) framework since the motivation for using SSL needs to be strengthened. Also, I think the problem of imbalance classification on the graph is very general among supervised and self-supervised settings. It is better to analyze the generality of GraphDec.

2. When I see the GraphDec sorting neural weights and data samples for pruning, I am expecting the exact computation consumption of this model-data joint optimization can be clearly shown. I am not sure whether the method can actually trim down the computational cost.
clarity,_quality,_novelty_and_reproducibility: Core resources (e.g., code, data) are available. The submission contains sufficient details (e.g., proofs, experimental setups) to be useful and clear to other researchers. The overall method is novel and ingenious.

summary_of_the_review: This is a technically sound paper with novel ideas on dynamic graph learning model and data sparsity, and its impact on the community is not restricted to the graph area. I believe that the experimental evaluation results are encouraging and sufficient. I would like to accept this novel and solid work.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper argues that current Graph Neural Networks (GNNs) suffer from high time cost when trained on massive data, and class-imbalance issue. The authors then propose GraphDec to address the problems by pruning both GNN and the training data.
strength_and_weaknesses: ## Strength
(+) The authors provide plentiful empirical results to show the effectiveness of GraphDec.

## Weakness
(-) Motivation is unclear. 
- Does the imbalance class problem really exist when massive training data is available? 
- Why do the authors choose self-supervised graph contrastive learning? If the model is learned in a self-supervised fashion, does it suffer from class-imbalance?

(-) Key theoretical results seem to be irrelevant with GNNs, graph contrastive learning. Furthermore, I can not see the connections between the proposed methods and the theorem. 

(-) Moreover, class-imbalance issue seems not to be considered in method design, making the readers confusing about the improvements in experiments. 

(-) The main method seems not to consider any characteristics of graphs and GNNs, but a bag of tricks when applying Data Diet to graph contrastive learning.

(-) The main method also introduces additional computation and memory overheads, which makes readers confusing about its advantages compared to existing graph contrastive learning, sparsity and class-imbalanced learning methods. Besides, the authors did not provide any efficiency checks such as complexity or the running time cost in experiments.

(-) The authors did not compare other pruning methods and the state-of-the-art graph contrastive learning methods. The node classification experiments did not include graph contrastive learning methods either.

(-) There are too many hyperparameters are introduced, but the authors did not test the sensitivity to these hyperparameters.

clarity,_quality,_novelty_and_reproducibility: The problem seems to be new and interesting, but the writing and the organization of the paper is hard for readers to get the idea that the authors try to convey. There are unclarified points, concepts and typos. Here are some examples:

- In Theorem 1, what’s $L^i_{train}$? 
- In Theorem 1, why the validation loss in involved?
- In Theorem 1, what’s $\sigma_T$?
- In Theorem 1, what’s the point to upper bound a minimum number?
- Eq. 4: the input is $x^{(t)}$?
- In Eq. 4-5, what’s $\theta_{pruned}$? Is it irrelevant with $t$?

summary_of_the_review: I vote for a reject. As pointed out in Weakness, the motivation of the paper is unclear, the proposed methods along with the theory seems to be irrelevant to the problems (the massiveness and class-imbalance in graphs) that the authors try to address, making the main methodology is full of tricks and the paper too technical.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper first examines the challenges in graph data imbalance and model-level computational burden theoretically and empirically. A GraphDec model has been proposed to address these challenges and evaluated on multiple benchmark datasets. The GraphDec encompasses dynamically choosing a subset of samples in the graph data and also dynamically prune the GNN during the training process.

Solving the class-imbalanced graph learning problem through a dynamic data-model co-sparsity approach is interesting and novel. There are sufficient ablation studies to emphasize the significance of each algorithmic component.

The theorem 1, which is a key pivot for the entire paper seems flawed with several steps of the theorem being unclear and incorrect. 




justification_for_why_not_higher_score: The SAC and AC found serious issues with the key theorem in the paper (Theorem 1) during the discussion phase. It seems to be a rehashing of a standard result about using a stochastic gradient in place of the true gradient, and it is not clear what (if any) specialized properties are leveraged here. Further, several steps in the theorem are incorrect or unclear. For example, the term Err^t includes L_{G_F}, but L_{G_F} does not appear in the proof. Also, the theorem is quite unrelated to graphs, and there are several strong assumptions. 
justification_for_why_not_lower_score: NA
Decision: Reject