Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper proposes a training strategy for GNN, TuneUp, that applies curriculum learning with node synthesis and label generation. It is evaluated on multiple modalities and outperforms multiple data augmentation baselines.
strength_and_weaknesses: ### Strengths
1. The paper introduces a training strategy, TuneUp, that organically combines multiple data augmentation methods for GNN.
2. The proposed TuneUp strategy is intuitive and performs well on multiple types of prediction tasks.
3. It also considers an interesting cold-start scenario that may be useful for real applications.

### Weaknesses
1. The paper claims the TuneUp strategy can improve the performance of the GNN training. However, it is only tested on the basic GNN models, GraphSage and GCN. It would be interesting to see other GNN architectures that appear in the ogbn leaderboard to show the generality of the method.
2. It would be interesting to see some examples of nodes and targets that are generated from the synthesis stage to give more insights into the method.
3. The method adopts pseudo labels to improve performance. It would be also great to have some label-smoothing baselines.
4. The paper made several assumptions for the augmentation techniques used in the algorithm. It can be more convincing to show some statistics of the graphs in different datasets, such as how many pseudo-labels are correct, and the accuracy for the head nodes vs tail nodes for the base model
clarity,_quality,_novelty_and_reproducibility: The writing conveys the ideas of the authors, but the organization of the paper is not very good. There are a few assumptions not well supported by experiments.
summary_of_the_review: The paper proposes a training strategy for GNN that combines multiple data augmentation methods together and improves the performance of two basic GNN models. It is not well-written and several assumptions are not well supported.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper proposes a two-stage curriculum learning process for GNNs to ensure better performane in a variety of tasks. Specifically, the paper argues that low-degree nodes are the hardest to make predictions for and focus on improving this.
strength_and_weaknesses: Strengths:
- Clear communication, easy to understand
- Discusses the assumptions that enable the method to perform better.
- General method, i.e. applicable in many areas
- Good dataset selection to evaluate empirical performance
- Covers both transductive and inductive settings, as well as cold-start nodes
- Ablation studies done are meaningful and reinforce the points of the paper
- Many other graph augmentation methods are compared against

Weaknesses:
- The GNN architecture used in evaluation is old and a narrow choice. The most standard choice would be an MPNN and then checking with the 3 most common aggregators, which I suspect would have a large impact on how the node degrees affect prediction. So I think the information provided by the empirical evaluation may not be useful to the community as GraphSAGE is simply not the default choice anymore. GCN is even older. The authors should rerun at least the largest dataset in each task (node classification, link prediction, recommender system) with the three MPNN (max, mean, sum aggregators).
- The evaluation does not do enough to convince the reader that this is useful across GNN architectures, for that MPNN with different aggregators are the minimum.
- The deviation in the evaluation protocol for node classification, while sensible, is unfortunate in that it prevents easy comparison for past, concurrent, and future work as it is not standard.
- For link prediction there is no test set, which I find problematic and is not an acceptable protocol for a top machine learning conference.

Minor:
- The standard deviation numbers in a lot of tables (especially Table 2.) are too small to be readable, the authors would be better off reporting them in the appendix and removing them from the main paper. It harms the readability too significantly.

Questions:
- You claim in section 2.4 that links in heterophilieous graphs would necessarily harm predictive performance in node classification. I can't imagine that is true, consider graph colouring, where adjacent nodes needs to have different colours and we are given an initial colouring for some of the nodes. This is a node classification problem in a heterophelic graph, link information is informative, in fact necessary for prediction. Am I missing something or are heterophileous graphs not a good example of the point you are trying make?

clarity,_quality,_novelty_and_reproducibility: Clarity:
- Clear.

Quality:
- Good

Novelty:
- Small

Reproducibility:
- Good
summary_of_the_review: The paper is clear and well written. The method is a small technical innovation that seems applicable across a wide variety of tasks. However, the empirical evaluation needs to be improved, in particular the GNN architectures considered as well as using a test set for link prediction.

If these concerns are addressed, I am willing to raise my score.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper proposes a two-stage curriculum learning strategy -- TuneUp --  to improve GNN's performance on tail nodes of small node degrees. The paper is motivated by the observation that GNNs perform worse on nodes with smaller degrees. 

In this first stage, TuneUp conducts standard training without augmentations. In the second stage, TuneUp trains GNNs with DropEdge augmentation. The paper argues that the DropEdge augmentation in the second stage can synthesize tail nodes from head nodes, thus providing more tail nodes for training.

TuneUp is evaluated in three tasks (semi-supervised node classification, link prediction, and recommendation). When the task is semi-supervised node classification, TuneUp also uses pseudo labels in the second stage of training.
strength_and_weaknesses: **Strengths:**

* The paper is clearly written.
* The studied problem of generalization to tail nodes is important for recommender systems and social network analysis.
* The proposed model is tested on extensive benchmarks from various domains. 



**Weakness**:
-  The technical novelty is weak. 
   1. The methods of DropEdge and Pseudo Labeling are well explored in previous works, as already cited in this paper. 
   2. The proposed TuneUp method is a straightforward adaptation of the "from easy to difficulty" idea from Curriculum Learning. 

- The experimental results are insignificant. 
   1. In Table1, *TuneUp w/o pseudo-labels* underperforms *DropEdge*. The result shows that the proposed two-stage training method is no better than a single-stage training with augmentation. Although using pseudo-labels can improve performance, pseudo labeling is proposed by previous works and does not align with the motivation of this paper. 
   2. In Table2, TuneUp's improvement over DropEdge is marginal. The result shows that the improvement over the base model is mainly attributed to the DropEdge augmentation rather than the proposed method.

- The experimental settings need more careful treatment. 
   1. In sec4.2, the experimental setting of the link prediction task follows that of two papers from a different domain -- recommender system -- instead of link prediction. The authors are encouraged to follow the experimental setting of papers from the same domain for appropriate evaluation. 
   2. It is weird to use an ogb**n** (**n** for node) dataset for link prediction evaluation, since the ogb benchmark has multiple ogb**l** (**l** for link) datasets designed specifically for link prediction evaluation.
   3. I also notice that the experimental settings of semi-supervised node classification and recommendation are new and different from existing works. The authors are encouraged to report the performances under the same evaluation protocols of existing works. 
clarity,_quality,_novelty_and_reproducibility: * **Clarity:** The presentation is clear. 
* **Novelty:** The nolvety is weak since the employed DropEdge, Pseudo Label and Curricum Learning method are well explored in previous works. See the 1st weakness in the previous section.
* **Reproducibility:** The reproducibility is in concern due to the abnormal experimental setting. See the 3rd weakness in the previous section.
summary_of_the_review: The paper presents a two-stage curriculum learning method to improve performance on tail nodes of small degrees. The novelty is weak and the experimental results are insignificant. The experimental settings need more careful treatments. The paper is not ready for publication at the current stage. Hence, I am leaning on the negative side.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This work proposes a curriculum learning strategy, named TUNEUP for GNN, which aims to improve the performance of GNNs at the tail nodes. TUNEUP first trains a GNN that produces good performance on head nodes, then synthesizes tail nodes by dropping the edges of head nodes, and then fine-tunes on these synthesized tail nodes. The authors conduct three types of experiments: semi-supervised node classification, link prediction and recommender systems. Extensive experiments demonstrate the effectiveness of this method.
strength_and_weaknesses: Strength:
1. The authors performed three different types of GNN applications, all with improved results.
2. It is significant to improve the performance of the tail node.

Weakness:

The author's approach is based on three very strong assumptions, which I disagree with, as follows:
1. The first one is "Tail nodes can be synthesized by dropping edges from head nodes".
Why the DropEdge can synthesize the tail node from the head node?
I think in most cases， dropping edges from head nodes would give synthetic tail nodes that have different characteristics from real tail nodes
The authors did not prove this experimentally or theoretically, so I don't think this assumption is convincing enough.

2. The second one is "Target supervision on head nodes can be reused for synthetic tail nodes".
The author points out that the degree of a node is not related to its properties, which I think is incorrect.
In most cases, the degree of a node is strongly related to its own properties.
For example, in a social network, celebrities will have many people to follow, thus resulting in a high degree.
I think the authors should also give explanations from an experimental or theoretical point of view.

3. The third one is "Edge information helps for GNNs to make prediction".
In many cases, more edges are not always better.
Real-life graph data has many heterogeneous connections, and these edges can lead to performance degradation.

clarity,_quality,_novelty_and_reproducibility: This paper is built on three suspicious assumptions, please refer to “weakness” above.
summary_of_the_review: I think the authors should give theoretical or experimental explanations and clarifications for these three assumptions.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper proposes a training strategy for GNN, TuneUp, a two-stage curriculum learning strategy that applies curriculum learning with node synthesis and label generation. In this first stage, TuneUp conducts standard training without augmentations. In the second stage, TuneUp trains GNNs with DropEdge augmentation. Specifically, the paper argues that low-degree nodes are the hardest to make predictions for and focus on improving this. TuneUp is evaluated in three tasks (semi-supervised node classification, link prediction, and recommender systems). When the task is semi-supervised node classification, TuneUp also uses pseudo labels in the second stage of training. The reviewer raised concerns including need for testing on extensive set of GNNs, using synthetic examples of nodes and targets to give more insights of algorithms, need for some label-smoothing baselines, need for test set for link prediction, limited novelty. The authors addressed some of the commented issues during response period. However, overall, there still seems to be a lack of enough enthusiasm among the reviewers after the author responses.
justification_for_why_not_higher_score: The reviewer raised concerns including need for testing on extensive set of GNNs, using synthetic examples of nodes and targets to give more insights of algorithms, need for some label-smoothing baselines, need for test set for link prediction, limited novelty. The authors addressed some of the commented issues during response period. However, overall, there still seems to be a lack of enough enthusiasm among the reviewers after the author responses.
Decision: Reject