TuneUp: A Training Strategy for Improving Generalization of Graph Neural Networks
Authors: Weihua Hu
Kaidi Cao
Kexin Huang
Edward W Huang
Karthik Subbian
Jure Leskovec
Venue: Submitted to ICLR 2023 (2023)
Keywords: Graph Neural Networks
Curriculum learning
Tail nodes

Abstract: Despite many advances in Graph Neural Networks (GNNs), their training strategies simply focus on minimizing a loss over nodes in a graph. However, such simplistic training strategies may be sub-optimal as they neglect that certain nodes are much harder to make accurate predictions on than others. Here we present TuneUp, a curriculum learning strategy for better training GNNs. Crucially, TuneUp trains a GNN in two stages. The first stage aims to produce a strong base GNN. Such base GNNs tend to perform well on head nodes (nodes with large degrees) but less so on tail nodes (nodes with small degrees). So, the second stage of TuneUp specifically focuses on improving prediction on tail nodes. Concretely, TuneUp synthesizes many additional supervised tail node data by dropping edges from head nodes and reusing the supervision on the original head nodes. TuneUp then minimizes the loss over the synthetic tail nodes to finetune the base GNN. TuneUp is a general training strategy that can be used with any GNN architecture and any loss, making TuneUp applicable to a wide range of prediction tasks. Extensive evaluation of TuneUp on two GNN architectures, three types of prediction tasks, and both inductive and transductive settings shows that TuneUp significantly improves the performance of the base GNN on tail nodes, while often even improving the performance on head nodes, which together leads up to 58.5% relative improvement in GNN predictive performance. Moreover, TuneUp significantly outperforms its variants without the two-stage curriculum learning, existing graph data augmentation techniques, as well as other specialized methods for tail nodes.

Under review as a conference paper at ICLR 2023
TUNEUP: A TRAINING STRATEGY FOR IMPROVING GENERALIZATION OF GRAPH NEURAL NETWORKS
Anonymous authors Paper under double-blind review
ABSTRACT
Despite many advances in Graph Neural Networks (GNNs), their training strategies simply focus on minimizing a loss over nodes in graph. However, such simplistic training strategies may be sub-optimal as they neglect that certain nodes are much harder to make accurate predictions on than others. Here we present TUNEUP, a curriculum learning strategy for better training GNNs. Crucially, TUNEUP trains a GNN in two stages. The first stage aims to produce a strong base GNN. Such base GNNs tend to perform well on head nodes (nodes with large degrees) but less so on tail nodes (nodes with small degrees). So, the second stage of TUNEUP specifically focuses on improving prediction on tail nodes. Concretely, TUNEUP synthesizes many additional supervised tail node data by dropping edges from head nodes and reusing the supervision on the original head nodes. TUNEUP then minimizes the loss over the synthetic tail nodes to finetune the base GNN. TUNEUP is a general training strategy that can be used with any GNN architecture and any loss, making TUNEUP applicable to a wide range of prediction tasks. Extensive evaluation of UPNEUP on five diverse GNN architectures, three types of prediction tasks, and both inductive and transductive settings shows that TUNEUP significantly improves the performance of the base GNN on tail nodes, while often even improving the performance on head nodes, which together leads up to 58.5% relative improvement in GNN predictive performance. Moreover, TUNEUP significantly outperforms its variants without the two-stage curriculum learning, existing graph data augmentation techniques, as well other specialized methods for tail nodes.
INTRODUCTION
Graph Neural Networks (GNNs) are one of the most successful and widely used paradigms for representation learning on graphs, achieving state-of-the-ar performance a variety of prediction
tasks, such as semi-supervised node classification (Kipf & Welling) 2017 Velickovic al.) 2018), link
prediction (Hamilton et al. [017]] Kipf& & Welling (016)), and recommender systems YYing et al. 2018)
He et 2020). There has been surge of work on improving GNN model architectures (Velickovic
et al. 2018 Xu et al.) 2019 2018 Shi et al. 2020 Klicpera et al. 2019 Wu et al.) 2019 Zhao &
Akoglu
2019 Li al. 2019 Chen et al. 2020; Li al.] 2021 and task-specific losses (Kipf &
Welling 2016 Rendle et al. 2012 Verma et al. 2022 Huang et al. 2021). Despite all these advances,
strategies for training a GNN on a given supervised loss remain largely simplistic. Existing work has focused on simply minimizing the given loss over nodes in a graph. While such simplistic default strategy already gives a strong performance, the strategy may still be sub-optimal as neglects that some nodes are much harder to make accurate predictions on than others. Consequently, GNN trained with the default strategy may significantly under-perform on those hard nodes, resulting in overall sub-optimal predictive performance.
Here we present TUNEUP to better train a GNN on a given supervised loss. The key motivation behind TP is that GNNs tend to under-perform of neighbors (Liu et al. 2021). In practice, performing well on tail nodes is important since they prevalent in real-world scale-free graphs (Clauset et al. 2009) and newly-rriviing cold-star start nodes (Lika et al. 2014). To better train a GNN on those hard-to- tord-to- tail nodes, the key idea of TUNEUP is to curriculum learning strategy (Bengio al. 2009): : TUNEUP first trains GNN
tail nodes, i.e., nodes with small number

---

Under review as a conference paper at ICLR 2023
Sem node classification
Link prediction
Recommender systems
Dutaset: products Medel. GrachSWGE
Detaoet: arehe MMddll: GraphtAGE
Darasett gowall Model: Grap-
2.40
Tune-pp
Tunsip
3.00-
9:25
0.30
4.75
9:20
0.25
0.00
0:20
0:15
140
0.19
335
8:19
350
3250
0:93
0.00
307
107 Noce: degrne (Copacale)
Rede deginse day Skalo)
No:88 dog + 0.ep Aippicckk)
Figure I: Spgree-ppecific generalization performance of the base GNN and TUNEUP in the trans- ductive setting. The r-axis represents the node degrees in the training graph, and the y-axis is the generalization performance averaged over nodes with the specific degrees. We see from the dotted blue eurves that the base GNN tends to perform poorly tail nodes, f.e., nodes with small degrees. Our TUNEUP (denoted by the solid orange curves) improves or at least maintains the base GNN performance on almost all node degrees. The improvement is more significant tail nodes.
to perform well on relatively easy head nodes, L.e,, nodes with large number of neighbors. It then proceeds to improve the performance on the hard tail nodes.
Specifically, TUNEUP uses the two-stage strategy train a GNN. In the first stage, TUNEUP employs the default training strategy, i.e., simply minimizing the given supervised loss, to produce a strong base GNN start with. The base GNN tends perform well head nodes, but poorly on tail nodes (see the dotted blue curves in Figure DT. To mitigate this issue, the second stage of TUNEUP focuses on improving the performance on the tail nodes. Specifically, TUNEUP synthesizes many additional tail node inputs by dropping edges from head nodes. TUNEUP then adds target supervision (e.g., class labels for node classification, edges for link prediction) on the synthetic tail nodes by reusing the original head nodes (before dropping edges). Finally, TUNEUP finetunes the
the supervision base GNN by minimizing the loss over the increased supervised tail node data. The dedicated training on the synthetic tail nodes allows the resulting GNN to perform much better the real tail nodes, while often even improving the performance on head nodes. TUNEUP is simple to implement on top Morcover, TUNEUP can be used
of the default training pipeline of GNNs, shown in Algorithm [I to train any GNN model with any supervised loss, making it generally applicable to a broad range of node and edge-level prediction tasks.
We extensively evaluate TUNEUP on a wide range of settings. We consider five diverse GNN architectures, three types of key prediction tasks (semi-supervised node classification, link prediction, and recommender systems) with a total of eight datasets, as well as both transductive (i.e., prediction on nodes seen during training) and inductive (i.e., prediction on new nodes never seen during training) settings. For the inductive setting, we additionally consider the challenging cold-start scenario (i.e., limited edge connectivity from new nodes) by randomly removing certain portions of edges from new nodes.
Across all settings, TUNEUP produces consistent improvement on the generalization performance of GNNs. In the transductive setting, TUNEUP significantly improves the performance of base GNNs on tail nodes, while oftentimes even improving the performance on head nodes (see Figure [T). Moreover, our ablation study shows that the two-stage curriculum training strategy of TUNE UP is critical and gives significantly improved performance over its variant strategy without curriculum learning. Finally, we extensively compare our TUNEUP against recent graph augmentation techniques (Rong et al. 2020 Liu et al. 2022) and specialized methods for tail nodes (Liu et al.) 2021 ZZeng et al. 2022 Zhang al. 2022 Kang al.] 2022). Our TUNEUP outperforms all these methods in all
settings, while being simpler and more general. Overall, our work demonstrates that training strategies
can play important role in improving generalization performance of GNNs.
GENERAL SETUP AND TUNEUP
TUNEUP is curriculum learning strategy to train any GNN model with any supervised loss to solve node or edge-level prediction tasks over graphs. We first provide a general task setup for machine learning on graphs and review the default training strategy of GNNs solve the task. We then present
2

---

Under review as a conference paper at ICLR 2023
TUNEUP, which adds few simple components the default training strategy. Finally, discuss assumptions TUNEUP exploits to improve generalization performance of GNNs and why TUNEUP even improves the performance on head nodes.
2.1 GENERAL TASK SETUP
We are given a graph G = EV) with a set of nodes V and edges E' with potentially some features associated with them. GNN Fe. parameterized by 8, takes the graph G as input and makes prediction Y for the task of interest. The loss function L measures the discrepancy between the GNN's prediction Y and the target supervision Y. In the default training. GNN parameter 0 is learned to minimize the loss L( Y) using gradient descent. The setup is general to cover most graph machine learning scenarios. Below, describe three representative scenarios under the general task setup, which we also consider in our experiments.
Semi-supervised node classification. The task is to predict class labels of unlabeled nodes given small set of labeled nodes in graph, which can be formalized as follows.
Graph G: A graph with input node features.
Supervison Y: Class labels of labeled nodes Viabeled C V.
GNN Fo: model that takes G as input and predicts class probabilities over V.
Prediction Y: The GNN's prediction over Vabeled.
Loss L: Cross-cntropy loss.
Since input node features are available, the GNN Fg can make not only transductive predictions, ie., prediction over Vunlabeled V Viabeled, but also inductive predictions (Hamilton al. 2017). iee. prediction over new nodes Vnew that are not in V but connected to V via new edges Enew.
Link prediction. The task is predict new links in a graph given existing links. We consider the node-centric formulation (You et al. 2021): : given source node, predict target nodes that the source node linked to.
Graph G: A graph with input node features.
Supervison Y: Whether node E is linked to node E V G (positive) or not (negative).
GNN Fg: A model that takes G as input and predicts the score for pair of nodes (s, t) E V V. Specifically, the model generates embedding 2, for each node in v E V and uses an MLP over the concatenation of 2, and z, to predict the score for the pair (s, t) (He et al. 2017).
Prediction Y: The GNN's s predicted scores over V x V.
Loss L: The Bayesian Personalized Ranking (BPR) loss (Rendle et al., 2012), which encourages the predicted score for the positive pair (s, (pss) to be higher than that for negative pair (s, Ineg) for each source node E V.
As input node features are available, the GNN F9 can naturally make inductive link prediction by generating node embeddings on a new graph with new nodes and edges.
Recommender systems. recommender system can be modeled as a bipartite graph between user nodes Vuser and item nodes Vitem- where edges represent user-item interactions. The task is essentially link prediction, i.e., given a user node E Vuser. predict a set of item nodes that u is likely to interact with. recommender systems, the most successful paradigm is collaborative filtering (Schafer et al., 2007), where shallow embeddings (learnable embeddings for each node) instead of input node features are used to achieve state-of state-of-the-and -tht-art performance (Wang et al., 2019 (He) al. 2020). As input node features are not available in many public recommender system datasets anyway, we focus on the feature-less setting.
Graph G: User-item bipartite graph without input node features.
Supervison Y: Whether a user node u has interacted with an item node e in G (positive) or not (negative).
GNN Fo: A model that takes G as input and predicts the score for pair of nodes (u, v) e Vuser X Vitem- Following Wang et al. (2019). GNN parameter 0 contains the input shallow embeddings in addition to the original message passing GNN parameter. To produce the score for the pair of nodes (u, v),, we generate the user and item embeddings, 2n and 2,, and take the inner product _ to compute the score (Wang et al. 2019).
Prediction Y: The GNN's predicted scores over
Vuser X Vitem-
3

---

Under review as a conference paper at ICLR 2023
Algorithm 1 TUNEUP. Compared to the default training of a GNN (L2-5), TUNEUP introduces the two-stage training and only adds two components (L8 and L12) that are straightforward to implement.
Given: GNN Fs. graph G. loss L, supervision Y. DropEdge ratio a.
1: # First stage: Default training to obtain base GNN.
2: while not converged do
Make prediction Y = [o(
4: Compute loss L(Y,, Y) compute gradient VoL. and update parameter 0.
5: end while
6: # Set up for the second stage.
7: if task is semi-supervised node classification then
8: Use Fi to predict pseudo labels non-isolated, unlabeled nodes. Add the pseudo labels into Y.
9: end
10: Second stage: Fine-tuning the base GNN with increased tail supervision.
11: while 0 not converged do
12: Synthesize tail nodes, Le., randomly drop of edges: G DropEdgge G.
13: Make prediction Y = (G)).
14: Compute loss L(Y, Y), compute gradient VoL, and update parameter 0.
15:
end while
Loss L: The BPR loss (Rendle et al. 2012).
As we learn the shallow embedding for each node, it is non-trivial to make inductive predictions on new nodes. Therefore, we only consider the transductive setting for recommender systems.
2.2 DEFAULT GNN TRAINING
Given the graph G. supervision Y, GNN Fe, its prediction - FG(),, and the loss function L(Ý, Y), it is straightforward to train the GNN Fg using gradient descent to minimize the loss. The default training procedure of a GNN is described in L2-5 of Algorithn
Remark on mini-batch training. In practice, the prediction Ý and the loss computation L(Ý, be made in mini-batch manner for scalable training. For instance, in the case of semi-supervised node classification, we can predict and compute the loss on mini-batch of labeled nodes (Hamilton) ct al. 2017 Zeng et al.) 2020). In the case of link prediction and recommender systems, to compute the BPR loss, the score prediction only needs to be made over positive links and randomly sampled negative links. Moreover, the BPR loss can be computed in mini-batch manner by subsampling the source nodes and keeping only one positive link per source node. These mini-batch training tricks are hidden in Aggorithm[Ter for simplicity, but should be implemented in practice. Our TUNEUP, which we explain next, is fully compatible with mini-batching training.
2.3 TUNEUP
We are ready to present TUNEUP that uses a two-stage curriculum learning strategy (Bengio et al. 2009) to better train GNN. At high level, TUNEUP first trains GNN to perform well on relatively easy head nodes and then proceeds to finetune the GNN to also perform well on hard tail nodes. Specifically, in the first stage (1.2-5 in Algorithm T]. TUNEUP uses the default training of GNNs to obtain a strong base GNN model. The base GNN model tends to perform well on head nodes, but poorly on tail nodes. To remedy this issue, in the second training stage, TUNEUP finetunes the base GNN with increased supervision on tail nodes (L7-L15 in Algorithm I]. TUNEUP increases the supervised tail node data in two steps: (1) synthesizing additional tail node inputs and (2) adding target supervision on the synthetic tail nodes, which we detail below.
(1) Synthesizing tail node inputs. TUNEUP synthesizes many additional tail nodes by removing edges from the head nodes. Specifically, in this work, we directly adopt DropEdge (Rong et al. 2020) for simplicity, where a certain portion (given by hyperparameter a.) of edges are randomly removed from the original graph G to obtain & (L12 in Algorithm 1p. The resulting contains more nodes with low degrees, i.e,, tail nodes, than the original graph G does. Hence, the GNN sees more (synthetic) tail nodes as input during training. More advanced strategies to synthesize tail nodes (e.g., dropping more ratio of edges from head nodes) are left for future work.
4

---

Under review as a conference paper at ICLR 2023
(2) Adding supervision on the synthetic tail nodes. After synthesizing the tail node inputs, TUNEUP adds target supervision (e.g., class labels for node classification, edges for link prediction) on them so that the supervised loss can be computed over the synthetic tail nodes.
For link prediction tasks, TUNEUP directly reuses the original edges E in G (before dropping) as the the synthetic tail nodes. To describe the effectiveness of the approach, suppose we have a node v with six neighbors in the original training graph G. After dropping a = 0.5 of
target supervision
edges in L12 of Algorithm | this node becomes synthetic tail node e with three neighbors in G. Nevertheless, in the loss computation of L.14, TUNFUP still reuses all original six edges from e in G as target supervision on this synthetic node ü. Therefore, this synthetic tail node i has twice as much edge supervision as any tegree-three real tail node (in the original graph G) has.
Similarly. for semi-supervised node classification, TUNEUP can also reuse the target labels of labeled nodes in G for synthetic tail nodes in G. Specifically, for labeled node E Viablled with ground- truth class label Vu. TUNEUP can reuse yv for the corresponding synthetic tail node in G. However, in the semi-supervised setting, the number of labeled nodes Viabeled is often very small, e.g., 1%-5% of all nodes V, limiting the amount of target label supervision TUNEUP can reuse.
To resolve this issue, TUNEUP utilizes pseudo labels (Lee et al. 2013) in addition to the limited ground-truth labels on Viabeled. Specifically, TUNEUP applies the base GNN (obtained in the first training stage) over G to predict pseudo labels on non-isolated (i.e., positive-degree) nodes in Vunlabeled. practice, the pseudo labels do not need to be directly predicted by the base GNN, e.g., we can apply post-processing, such as label smoothing and C&S, to refine the pseudo labels. We leave the investigation to the future work.
TUNEUP then includes the pseudo labels as supervision Y in the second stage (L8 in Algorithm [ This would significantly increase the size of the supervision Y, e.g., by a factor of 1100 if only 1% of nodes are labeled. While the predicted pscudo labels are noisy in general, they are "best guesses" in the sense that the base GNN uses the full graph information G to predict the labels. In the second stage. TUNEUP essentially forces the base GNN to maintain its best guesses given sparser graph G with limited neighborhood as input. This in turn allows the resulting GNN to make more accurate prediction on real tail nodes with limited neighborhood in the original graph G.
2.4 ASSUMPTIONS TUNEUP EXPLOITS TO IMPROVE GENERALIZATION PERFORMANCE
As the no-free no-free-lunch theorem suggests (Wolpert 1996), improving generalization performance in- volves exploiting additional assumptions real-werth world prediction tasks, which may not be satisfied by all possible tasks. Here we discuss three key assumptions TUNEUP exploits to improve general- ization performance, which are (approximately) satisfied by many real-world tasks, including our experimented benchmark datasets across three different task types in Section
Tail nodes can be synthesized by dropping edges from head nodes. This assumption holds for many real-world graph datasets, as head nodes often start off as tail nodes, e.g., well-cited paper nodes are not cited at the beginning in a paper citation network, and warm users (users with many item interactions) start off as cold-start users in recommender systems.
Target supervision on head nodes can be reused for synthetic tail nodes. This assumption holds for tasks where prediction to be made on given node is more or less static property of the node. For instance, papers' subject areas in paper citation network, products' categories in a product co-purchasing network, and users' taste in recommender systems stay (mostly) the same regardless of the number of edges we observe on the nodes.
More edges benefit GNNs to make accurate predictions. TUNEUP assumes that more edges are useful for GNNs to make accurate predictions, and tail nodes are harder to predict due to the lack of edges. This assumption is likely to hold for many tasks as GNNs can aggregate more neighboring information with more edges, and is experimentally verified in Figure
2.5
WHY TUNBUP IMPROVES PERFORMANCE ON HEAD NODES
is counter-intuitive that TUNEUP improves performance not only on tail nodes but also on head nodes, seen in Figure One reason may be that best perffrrnng GNNs for nodeledge-level tasks, including our experimented ones, use (roughly) average-bass based schemes to aggregate neighboring
node features (Hamilton al. 2017 Kipf & Welling 2017 Wu et al., 2019: Velickovic al.
5

---

Under review as a conference paper at ICLR 2023
With the average-basee GNNs, node embeddings obtained on the sparsified graph G can be thought of as the noisy version of the node embeddings obtained on the full graph G. If the base GNN is finetuned to perform well many realizations of the noisy embeddings (with different realizations of G in L12 of Algorithm l). then the resulting GNN would most likely still perform well on the noise-free free embeddings (computed over the full graph G). Moreover, training with the noisy embeddings can even improve the generalization performance on head nodes. We leave in-depth empirical/theoreticall investigation for future work.
2018).
3 RELATED WORK
Data augmentation for GNNs. The second stage of TUNEUP can be regarded as data augmentation
over graphs, on which there has been rich body of work (Ding et al. 2022 Some are specifically designed for semi-supervised node classification (Zhao et al. 202 I| Feng et al. 2020 Verma et al.) 2021), while others are designed for recommender systems (Verma al. 2021) and graphs with input node features (Liu et al. 2022). Different from these methods, TUNEUP is generally applicable to any prediction tasks over nodes and edges. The general nature of TUNEUP also allows it to be combined with any of the task-specific data augmentation techniques. As a general graph augmentation technique, Kong al. (2020) proposed FLAG, which adversarially perturbs input node features (Shafahi et al. 2019). This is complementary to TUNEUP, which perturbs the edge connectivity of the graph. DropEdge (Rong al. 2020) randomly drops edges from graphs. was originally developed to overcome the over-smoothing issue of GNNs (Li et al. 2018) in semi- supervised node classification. In contrast, in this work, we adopt DropEdge as a way to synthesize additional tail node inputs for wide range of prediction tasks over graphs. Methodologically. TUNEUP is distinct from DropEdge in that employs the two-stage curriculum learning strategy and uses pseudo labels to add supervision on the synthetic tail nodes, both of which are important to yield substantially better performance than the original DropEdge.
Curriculum learning for GNNs. A few works have explored curriculum learning for GNNs. Wang ett al. (202T) developed a curriculum learning approach for graph classification, while our work focuses on node/edge-leve prediction tasks. |Ying et al. (2018) presented curriculum learning for negative sampling in link prediction, and Li et al. (2022) developed a curriculum learning for tackling imbalanced class labels in node classífication. TUNFUP is complementary both of these approaches while being more broadly applicable to any node/edge-level prediction tasks.
Pre-training GNNs. Pre-training GNNs has attracted huge attention (Veličković et al., 2019 Hu
et al. 20200b Qiu et al. [2020] Hu et al., [2020c] You et al.] 2020b). This line of work develops task- agnostic strategies pre-train a GNN such that the resulting GNN can be finetuned with task-specific supervised losses to improve performance on diverse downstream tasks. Our work focuses on the downstream stage and presents a strategy for training GNN on task-specific supervised loss.
Specialized methods for tail nodes. Recently, many methods have been developed for improving generalization performance of GNNs on tail nodes (Liu et al. 202T. Zheng et al., 2022] Kang et al. 2022 Zhang et al. (022). These methods require augmenting a GNN with tail-node-spede architectural components, while our work does not require any architectural modification and focuses purely on strategy for training GNN that performs well on both tail and head nodes.
4 EXPERIMENTS
Here we extensively evaluate TUNEUP under wide range of settings. We consider five diverse GNN models and test them on the three prediction tasks described in Section 2.r three different predictive settings: transuctive, inductive, and cold-start inductive predictions.
4.1 EXPERIMENTAL SETTINGS
Here we describe our experimental settings and datasets for evaluating TUNEUP. We noticed that the
standardized experimental protocols by Hu et al. (2020a); Wang et al. 2019) are not suitable for evaluating TUNEUP because (1) inductive prediction scold-start) settings are not provided and (2) datasets are heavily pre-pre propprceesedd to eliminate tail nodes (e.g.. recommender system benchmarks processsed with the 10-core algorithm to eliminate the cold-start users and items (Wang et al., (2019),
which is the focus of this work. We therefore take their original realistic graph datasets and split

---

Under review as a conference paper at ICLR 2023
them ourselves to create the realistic inductive (cold-start) prediction setting well as the realistic transductive setting with tail nodes. The dataset statistics are summarized in Table B in Appendix. Below, we describe the split and the datasets for each task type.
Semi-supervised node classification. Given the entire nodes in the original dataset, we randomly selected 95% of the nodes and used their subgraph induced as the graph G (V, E) to train GNNs. The remaining 5% of the entire nodes, Vnew- are used for inductive prediction. Within V. 10% and 2% of the nodes are used as labeled nodes Viabeled for arxiv and products, respectively. A half of Viaboled is used for computing the loss for supervised training, and another half is used as the transductive validation set for tuning pyper-paramedt the standard classification accuracy. For the transductive performance, we report the accuracy on the unlabeled nodes Vunlabeled V Viabeled, while for the inductive performance, we report the accuracy on Vnew- For the inductive prediction, we also consider the cold-start scenario, where certain portions (30%, 60%, and 90%) of edges are randomly removed from the new nodes. We used the following two datasets in our experiments.
For the evaluation metric, we used
arxiv (Hu et al. 2020a): Given a paper citation network, the task is to predict the subject areas of the papers. Each paper has abstract words as its feature.
products (Hu et al., 2020a): Given a product co-purchasing network, the task is to predict the categories of the products. Each product has the product description as its feature.
Link prediction. We follow the standard link prediction evaluation (Zhang & Chen 201 8 You ctt al. (02I) and randomly split the edges in the original graph into training and validation edges with the ratio of 50%/50%. We follow the same protocol
semi-supervised node classification to obtain nodes for transductive and inductive settings. For the evaluation metric, we used the recall 50 averaged over nodes (Wang et al. 2019), where the positive target nodes scored among all negative nodes. For the transductive performance, we report the recall @ 50 computed over validation edges within V. while for inductive setting, we report the recall @ 50 over validation edges from Vnew. For the inductive setting, we also consider the cold-start scenario. We used the following three datasets in our experiments.
flickr (Zeng et al. 2020); Given an incomplete image-image common-property (e.g., same geo- graphic location, same gallery, comments by the same user, etc.) network, the task is to predict the new common-poe common-property links between images. Each image has its description has its feature.
ppi (Chandak et al.) 2022):: Given an incomplete protein-proted interaction network, the task is to predict new interactions. Each protein feature is generated with ESM protein language model (Rives) et al. 2021) applied to the protein sequence.
arxiv (Hu al.) (020a)): Given an incomplete paper citation network, the task is to predict the additional citation links. Each paper has words in its abstract as its feature.
Recommender systems. For recommender systems, we notice that widely-used benchmark datasets heavily processed to climinate all tail nodes, e.g., via the 10-core algorithm (Wang et al. 2019). For example, with the conventional 80%/20% train/validation split, the median training interactions per user is 17, 26, and 27 for gowalla, yelp2018. and amazon-book, respectively, which clearly do not reflect the realistic use case that involves many cold-start users and items (Lika et al. 2014). To reflect the realistic case, use the small training edge ratio top of the existing benchmark datasets. Specifically, we randomly split the edges in the original graph into training and validation edges with 10%/90% ratio. For the evaluation metric, used the recall @ 50 averaged over users (Wang et al. 2019 He et al.) 2020). For the transductive performance, we report the recall @ 50 computed over the validation edges. We do not consider the inductive setting for recommender systems. We used the following three datasets in our experiments.
gowalla (Liang et al., 2016 Wang et al. 2019): Given an user-location check-in bipartite graph,
the task is to predict new check-in of users.
yelp2018 (Wang et al. 2019): : Given user-restaurant review graph, the task is to predict new reviews
by users.
amazon-book (He & McAuley 2016 Wang et al. 2019 : Given user-product reviews, the task is
predict new reviews by users.

---

Under review as a conference paper at ICLR 2023
Table 1: Semi-supervised node classification performance with GraphSAGE as the backbone archi-
tecture. The metric is classification accuracy. For the "Inductive (cold)", , 90% of edges are randomly removed from new nodes. For the results with other edge removal ratios, refer Table [ Appendix. Refer to Table in Appendix for the performance with GCN, where similar trend is observed.
Method
arxiv
products
Transductive
Inductive
Inductive (cold)
Transductive
Inductive
Inductive (cold)
Base
0.6738:00006
0.6689-000g
0,4748
0.8408=:0.00m
0.8424
0.7226:00012
DropEdge
0.6756=0003
0.6692:0ma.
0,5446:0ms0
0.8463-00000
0.847 #:00005
0.7710
LocalAug
0.6824=00004
0.6773-00023
0.4976=a00s
0.8445 +00000
0.846 +:00000
0.7256+00000
ColdBrew
0.6726:00000
0.6480
0.5080
0.8378
=0.0007
0.8385
0.7417 +0.0025
GraphLessNNN
0.6074
0,5457
0.5457=00006
0.6670=a0001
0.6640
0.6640=0.00
Tail-GNN
0.6613:00006
0.6557=00022
0.5389
0OM
ODM
00M
TU w/o curriculum
a.6753
0.6688
0.5474.nor19 19
0.8456200000
0.8466
0.7573
TUNEUP wo syn-tails
0.6785:00001
0.6758
000008
0.4900
0.8434
0.8410+00.000
0.7255 +0.0011
TUNEUP w/o pseudo lahels
0.6747:00000
0.6681 100006
0.5332:00016
0.8461 10.0000
0.8471 1:00007
0.7630:20.065
TUNEUP (ours)
0.6867
0.6784=0008
0.6006:0.0000
0.8554
=0,0002
0,8564
0,8054
Rol. gain over base
+1.9%
+1.4%
+26.5%
+1.7%
+1.7%
+11.5%
Table 2: Link prediction performance with GraphSAGE the backbone architecture. The metric is recall@ @ 50. For the "Inductive (cold)". 60% of edges are randomly removed from new nodes. For other edge removal ratios, refer to Table T0 in Appendix, where TUNEUP consistently outperforms the baselines. Refer to Table 9 Appendix for the performance with GCN, where we see a similar trend.
Mathod
ppl
Tranductive
Inductivv
Inductive (cold)
Transductive
Inductive
Indoctive (cald)
Transducthve
Inductive
Inductiee (col))
Base
0,1030 0.1000
0,0606 20023
0,1165 iunics
0,1061 Laaas
ILOE35 .ancc
0.1326
0.1255 LC364
0,0722 (005)
Deppfoge
0 201297
0. 0.1750gsmm1
0. 0.1492g11m
B: 0.1387
0.201 Saroon
B. 0.1789-0-m0
01 @1154gt -1
LucalAng
0.0619-mms
0. 0.1255amm
0. 6.1186-mmm
0.0950
0.1 0.1365..mmmm
8. 011204---145
010750-1-444
Coodiree
0,0015 0.061
0.1011
0,0834 LozM
0U0761 L0000
0.1161 L0006
BL1095 Lc300)
0.0630000000
TNNN-CNN
0160000
0.08
0,0266:30014
0.11502
0. 0.1002-arms
0.8229.00000
0. Lonees
010006-0
01
TUSEUP we cureullu
0.3333 LE303
13001s
0.1315
0.1995 LAM00
U.1760-0800
0.0603
0.1172
0.0057-44m
0.082--11100
0.0758
Tuxet's fours)
0.1379
0.1819
0,8577
0.1328
0.1637
0.1252
Rel. gain over base
+37.2%
+33,8%
180.1%
156.1%
+48.7%
+59.1%
+58.5%
+46.4%
+73.39%
4.2 BASELINES AND ABLATIONS
We compare our TUNEUP against the following strong baselines.
Base:
Trains a GNN with the default strategy, i.e., L2-5 of Algorithm I The accuracy of pseudo labels coincides with the accuracy of the base GNN.
DropEdge (Wang et al. 2019): Randomly drops edges during training, i.e., L11-15 of Algorithm [
Local augentation (LocalAug) (Liu et al. 2022): Uses conditional generative model to generate neighboring node features and use them as additional input to a GNN.
ColdBrew (Zheng et al., 2022): Distills head node embeddings computed by the base GNN into an MLP. Uses the resulting MLP to obtain nigher-qualing tail node embeddings.
GraphLessNNN (Zhang et al.) (2022): Distills the pseudo labels predicted by the base GNN into an ML.P. Uses the resulting MIP to make prediction.
TTil--NNN (Liu et al. (02T): Adds tail-node specific component inside the original GNN.
RAWLS-GCN (Kang et al. 2022): Modifies the GCN's adjacency matrix to be doubly-stochastic
(i.e., all rows and columns sum 1).
Note that GraphLessNN is only applicable for node classification. LocalAug and ColdBrew require input node features to be available; hence, not applicable to recommender systems. RAWLS-GCN is only applicable to the GCN architecture.
In addition to the existing baselines, we consider the following three direct ablations of TUNEUP.
TUNEUP w/o curriculum: Interleaves the first stage prediction (L3 in Algorithm I and the second stage prediction (L12-13 in Algorithm [] in every parameter update. It is close to TUNEUP except that it does not follow the two-stage curriculum learning strategy.
TUUEEU w/o syn-tails: No L12 in Algorithm| []
TUNEUP w/o pseudo-labels: No L8 in Algorithm
8

---

Under review as a conference paper at ICLR 2023
Table Transductive performance on the recommender systems datasets. The metric is recall @ 50.
Method
gowalla
yelp2018
amazon-book
SAGE
GCN
SAGE
GCN
SAGE
GCN
Base
0.085610,0000
0.091110000
0.0864 1000005
0.0091
0.0559 1000000
0.0540.0.000
DropEdge
0. 0,0835 0000004
0.0827.000012
0.0814 100000
0.0719
0.0543-4a.00
0.055
Tail-GNN
0.0806.00.00
0.0815.000m
0.0713
100004
0.07131:0000
0.0562.100002
0.0542-0.000
RAWS-GCN
0.00632.000006
0.0684.000004
0,0476
TUNEUP w/o curriculum
1.0849=0.000
0.0874:0.000
0.0837
0,0755
0.0549+0.000
0.0539x0.000
TUNEUP w/o syn-tails
0.0859+0.000 +0.0006
0.0917 0:000005
0.0866=00004
0.0797=0.000
+000003
0.0542-0.000
TUNEUP (ours)
0. 0.1043 2.002
0.1102:000013
0.0866:10004
0,0885 :e.00s
0.0:57 ::00005
0.0630=:00004
Relative gain over base
+21.8%
+21.0%
+0.2%
+11.9%
+3.1%
+16.7%
Table 4: The improvement with TUNEUP over the base GNNs for diverse GNN model architectures. For semi-supervised node classification, GAT gave Out-Or-Memory (OOM) on the products dataset; so we report the performance arxiv instead.
We used the same set datasets as Figure 1
Task
Conlig
Setting
SAGE(-meen)
GCN
SAGE-mux
SAGEE-sm
GAT
Transductive
Base
0.8408=00007
0.8432-0.00
0.8134:0200
0.7607
OOM/ / 0.6864+00x01
Semi-sup
TUNEUP
0.8554
0.8523.0.007
0.7615:20006
OOM/ / 0.6968
node
Inductive
Base
0.842410080
0,8447 .0003
0.8133 100011
0.7613 Lov0829
OOM/ 1.600)
classification (producis)
TUNEUP
0.8564
0,8535
0883992.000
0.7619
:90084
OOM/ / 0.6936
Inductive (cold)
Base
0.7226:00012
0,7461
20000
0.6908 1000007
0.5315 Leuss59
OOM/ / 0,5411 1:00sss
TUNEUP
0.8054:90007
0.7925-..094
0.7873
0.5490=au0119
OOM/ / 0.5932
Transduutive
Base
0.1326 100077
0,2152
0,1618
116181000m
0.07 30 L00017
0.2295
Link
TUNEUP
0,2101 =000411
0.2417
0.2397 8.0001
0.1168290mme
0.2569 20.091
prodiction (arxiv)
Inductive
Base
0.1255
0,2077=0000
0.1503=0000
0.0703=00023
0.2080=000029
TUNEUP
0,1837 880331
0.2254 0.0022
0.2140::..00p
0.1031 H9u0ss
0.2360
Inductive (cold)
Base
0.0722-00001
0.1192=00009
0,0998
0.0547
0.13:66:0.003
TUNEUP
0.1252:900844
0.1440=0.000
0,1559: e..600
0.0788
0.1627::0.00
Recsys (gowalla)
Transductive
Base
0.0856:00005
00.000
0.0870
0.0896=0000
0.0814:10.000
TUNCUP
0.1043=:0000
0.1102
0.1063:0.006
0,1129 :40022
0.0834:900m
Another possible ablation, TUNEUP w/o the first stage training (i.e., only performing the second stage training of L2-5 in Algorithm D is essentially covered as DropEdge in our experiments.
4.3 GNN MODEL ARCHITECTURES
We mainly experimented with two classical yet strong GNN models: the mean-pooling variant of
GraphSAGE (or SAGE for short) (Hamilton et al. 2017) and GCN (Kipf & Welling 2017). In Table we additionally experimented with the max- and sum-pooling variants of GraphSAGE
well the Graph Attention Network (GAT) (Velickovic et al.) 2018) to demonstrate the applicability of TuneUP on diverse GNN architectures to improve their performance. In total, we have five diverse GNN architectures that cover representative aggregation schemes (i.e., mean, renormalized-mest
(Kipf & Welling 2017), max, sum, and attention) that many recent advanced GNN architectures are based on (Corso et al. 2020; (Shi et al. 2021 You et al. 2020b Wu et al. 2019
Rossi et al., 2020 Li et al.
2018
You et al. 2020a).
4.4
HYPER-PARAMEER PARAMETERS
We used 3-layer GNNs and the Adam optimizer (Kingma & Ba 2015) for all GNN models and datasets, which we found to perform well in our preliminary experiments. For all methods, we performed the early stopping and tuned their dyyer-parametert based on the transductive validation performance. We used the resulting models for both transductive and inductive prediction. For the drop edge ratio a, we tuned it from [0.25, 0.5, 0.75] for all the datasets. We repeated all the experiments with 5 different training seeds report the mean and the standard deviation. More details are described in Appendix A
4.5 RESULTS
We first compare TUNEUP against the base GNNs that are trained with the default strategy. The last rows of Tables [ 3 highlight the relative improvement of TUNEUP over the base GNNs. TUNEUP improves over the base GNNs across the transductive settings, giving up to 1 1.9%, 58.5%.

---

Under review as a conference paper at ICLR 2023
and 21.8% relative improvement in the semi-supervised node classification, link prediction, and recommender systems, respectively. TUNEUP gives even larger improvement on the challenging cold-start inductive prediction setting, yielding up to 26.5% and 80.1% % relative improvement on the node classification and link prediction, respectively. In Appendix, we provide Tables 7 to show the full results on the cold-start inductive prediction with the different edge removal ratios from new nodes. The larger the ratio is, the more cold-start the setting becomes. We observe that TUNEUP provides larger relative gain on larger edge removal ratios, demonstrating its effectiveness on the highly cold-start prediction setting.
g 10 and [1
We also analyze the pegree-specifir generalization improvement and highlight the results in Figure The full results (two GNN architectures times the eight datasets) are available in Figures 2 and in Appendix. Across all datasets, architectures, and node degrees, TUNEUP produces consistent improvement over the base GNNs. Not surprisingly, improvement is most significant on tail nodes.
Finally, we compare TUNEUP against the strong baselines and ablation methods described in Section 4.2 Tables and show the results. We summarize our findings below.
TUNEUP outperforms the graph augmentation methods (DropEdge and LocalAug) as well as the specialized methods for tail nodes (ColdBrew, GraphLeesSNN, and Tail-GNN), establishing its superior performance against the existing strong baseline methods.
TUNEUP outperforms TPNEUP w/o curriculum, which highlights the importance of the two-stage curriculum learning strategy in TUNEUP.
TUNEUP also outperforms TUNEUP w/o syn-tails and TUNEUP w/o pseudo-labels, which suggests that both of the ablated components are necessary for TUNEUP to achieve the high performance. On semi-super node classification (Table T). TUNEUP w/o syn-tails, i.e. conventional semi- supervised training with the pseudo labels (Lee et al.
2013). gives limited improvement over the base GNN. In contrast, TUNEUP trains the GNN to predict pseudo labels with limited neighbor- hood, which gives sigmificant improvement over the base GNN. Moreover, TUNEUP significantly outperforms DropEdge, suggesting the importance of using DropEdge together with pseudo labels. On link prediction (Table DropEdge (TUNEUP UP without the first stage) already gives significant performance improvement over the base GNN, implying the unrealized potential of DropEdge on this task, beyond node classification (Rong et al., 2020). Nonetheless, TUNEUP still gives consistent improvement over DropEdge, suggesting the benefit of the two-stage training.
On recommender systems (Table 3. TUNEUP is the only method that produced the significantly better performance than the base GNN. DropEdge and TUNEUP w/ curriculum even gave worse performance than the base GNN. This is possibly because jointly learning the GNN and shallow embeddings is hard without the two-stage training.
Overall, TUNEUP, despite its simplicity, is the only method that yielded consistent improvement
across the three prediction tasks.
From Table we see that TUNEUP improves the performance on the five diverse GNN architectures. Although the performance improvement with the sum aggregation is limited for semi-supervised node classification, the sum aggregation gave the poor base GNN performance anyway due the poor inductive bias and unstable training (Wu et al. 2019 Hamilton et al. 2017).
CONCLUSIONS
In this paper, we presented TUNEUP, curriculum learning strategy train a GNN improve its generalization performance. TUNEUP first trains a GNN to produce a strong base GNN that performs well on easy head nodes. It then proceeds to improve the prediction over hard tail nodes by finetuning the base GNN with additional synthetic tail nodes. TUNEUP is a general strategy that can be used to train any GNN model with any supervised loss. Through extensive experiments, we demonstrated the effectiveness of TUNEUP on wide range of settings, including five GNN architectures, three types of prediction tasks, and both transductive and inductive settings. Overall, our work suggests that training strategies matter in improving generalization of GNNs and can be complementary to advances in model architectures and task-specific losses.
REFERENCES
Yoshua Bengio. Jérôme Louradour, Ronan Collobert, and Jason Weston, Curriculum learning. In International Conference Machine Learning (ICMI.), pp. 41-48, 2009.
10

---

Under review as a conference paper at ICLR 2023
Payal Chandak, Kexin Huang, and Marinka Zitnik. Building knowledge graph to enable precision medicine. bioRxiv, 2022.
Ming Chen, Zhewei Wei, Zengfeng Huang, Bolin Ding, and Yaliang Li. Simple and deep graph convolutional networks. In International Conference on Machine Learning (ICML), pp. 1725-1735. PMLR, 2020.
Aaron Clauset, Cosma Rohilla Shalizi, and Mark EJ Newman. Power-law distributions in empirical data. SIAM review, 51(4):661-733 2009.
Gabriele Corso, Luca Cavalleri, Dominique Beaini, Pietro Liò, and Petar Veličković. Principal neighbourhood aggregation for graph nets. volume 33, pp. 13260-13271, , 2020.
Kaize Ding, Zhe Xu, Hanghang Tong, and Huan Liu. Data augmentation for deep graph learning: A survey. arXiv preprini arXiv:2202.08235. 2022.
Wenzheng Feng, Jie Zhang, Yuxiao Dong, Yu Han, Huanbo Luan, Qian Xu, Qiang Yang, Evgeny Kharlamov, and Jie Tang. Graph random neural networks for semi-supervised learning on graphs Advances in Neural Information Processing Systems (NeurIPS), 33:22092-223 -22003, 2020.
William L Hamilton, Rex Ying, and Jure Leskovec. Inductive representation learning on large graphs. In Advances Neural Information Processing Systems (NeurIPS), pp. 1025-1035, 2017.
Ruining He and Julian McAuley. Ups and downs: Modeling the visual evolution of fashion trends with one-class collaborative filtering. In Proceedings of the International World Wide Web Conference (WWW), pp. 507-517, 2016.
Xiangnan He, Lizi Liao, Hanwang Zhang, Liqiang Nie, Xia Hu, and Tat-Seng Chua. Neural collaborative filtering. In Proceedings of the International World Wide Web Conference (WWW), pp. 173-182, 2017.
Xiangnan He, Kuan Deng, Xiang Wang, Yan Li, Yongdong Zhang, and Meng Wang. Lightgen: Simplifying and powering graph convolution network for recommendation. ACM SIGIR conference on Research and development in Information Retrieval (SIGIR), pp. 639-648, 2020.
Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovee. Open graph benchmark: Datasets for machine learning on graphs In Advances in Neural Information Processing Systems (NeurIPS), 2020a.
Weihua Hu, Bowen Liu, Joseph Gomes, Marinka Zitnik, Percy Liang, Vijay Pande, and Jure Leskovec. Strategies for pre-training graph neural networks. In International Conference on Learning Representations (ICLR), 2020b.
Ziniu Hu, Yuxiao Dong, Kuansan Wang. Kai- Kai-Wei Chang, and Yizhou Sun. Gpt-gnn: Generative pre-training of graph neural networks. In ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) pp. 1857-1867, 2020c.
Tinglin Huang, Yuxiao Dong, Ming Ding, Zhen Yang, Wenzheng Feng, Xinyu Wang, and Jie Tang. Mixgef: An improved training method for graph neural network-based recommender systems. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pp. 665-674, 2021.
Jian Kang, Yan Zhu, Yinglong Xia, Jiebo Luo, and Hanghang Tong. Rawlsgen: Towards rawlsian difference principle on graph convolutional network. In Proceedings of the International World Wide Web Conference (WWW), pp. 1214-1225 2022.
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. In International Conference on Learning Representations (ICLR), 2015.
Thomas N. Kipf and Max Welling. Variational graph auto-encoders. arXiv preprint 17X:001611.07308 2016.
Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In International Conference on Learning Representations (ICLR), 2017.
11

---

Under review as a conference paper at ICLR 2023
Johannes Klicpera, Aleksandar Bojchevski, and Stephan Günnemann. Predict then propagate: Graph neural networks meet personalized pagerank. In International Conference on Learning Representations (ICLR), 2019.
Kezhi Kong, Guohao Li, Mucong Ding, Zuxuan Wu, Chen Zhu, Bernard Ghanem, Gavin Taylor, and Tom Goldstein. Flag: Adversarial data augmentation for graph neural networks. arXiv preprint arXiv:2010.09891, 2020.
Dong-Hyun Lee al. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In Workshop on challenges in representation learning, ICMI, volume 3, pp. 896, 2013.
Guohao Li, Matthias Muller, Ali Thabet, and Bernard Ghanem. Deepgens: Can gens go as deep as cnns? In IEEE Conference Computer Vision and Pattern Recognition (CVPR), pp. 9267-9276, 2019.
Guohao Li, Matthias Müller, Bernard Ghanem, and Vladlen Koltun. Training graph neural networks with 1000 layers. In International Conference on Machine Learning (ICML), pp. 6437-6449, PMLR, 2021.
Qimai Li, Zhichao Han, and Xiao-Ming Wu. Deeper insights into graph convolutional networks for semi-supervised learning. arXiv preprint arXiv:1801.07666, 2018
Xiaohe Li, Lijie Wen, Yawen Deng, Fuli Feng, Xuming Hu, Lei Wang, and Zide Fan. Graph neural network with curriculum learning for imbalanced node classification. arXiv preprint arXiv:2202.02559, 2022.
Dawen Liang, Laurent Charlin, James McInerney, and David M Blei. Modeling user exposure in recommendation. In Proceedings of the International World Wide Web Conference (WWW), pp. 951-961, 2016.
Blerina Lika, Kostas Kolomvatsos, and Stathes Hadjiefthymiades. Facing the cold start problem in recommender systems. Expert systems with applications, 41(4):2065-200 2014.
Songtao Liu, Rex Ying, Hanze Dong, Lanqing Li, Tingyang Xu, Yu Rong, Peilin Zhao, Junzhou Huang, and Dinghao Wu. Local augmentation for graph neural networks. International Conference Machine Learning (ICML), pp. 14054-14072. PMLR, 2022.
Zemin Liu, Trung-Kie Kien Nguyen, and Yuan Fang. Tail-gnn: Tail-node graph neural networks. In ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD), pp. 1109-1119, 2021.
Jiezhong Qiu, Qibin Chen, Yuxiao Dong, Jing Zhang, Hongxia Yang, Ming Ding, Kuansan Wang, and Jie Tang. Gee: Graph contrastive coding for graph neural network pre-training, In ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD), pp. 1150-1160, 2020.
Steffen Rendle, Christoph Freudenthaler, Zeno Gantner, and Lars SchhiddtThhemme Bpr: Bayesian personalized ranking from implicit feedback. arXiv preprint arXiv: 2618, 2012.
Alexander Rives, Joshua Meier, Tom Sercu, Siddharth Goyal, Zeming Lin, Jason Liu, Demi Guo, Myle Ott, Lawrence Zitnick, Jerry Ma, et al. Biological structure and function emerge from scaling unsupervised learning 250 million protein sequences. Proceedings of the National Academy of Sciences, 1 (18(15)c220122911
2021.
Yu Rong, Wenbing Huang, Tingyang Xu, and Junzhou Huang. Dropedge: Towards deep graph convo- lutional networks on node classification. In International Conference on Learning Representations (ICLR), 2020.
Emanuele Rossi, Fabrizio Frasca, Ben Chamberlain, Davide Eynard, Michael Bronstein, and Federico Monti. Sign: Scalable inception graph neural networks. arXiv preprint arXiv:2004. / 2020.
J Ben Schafer, Dan Frankowski, Jon Herlocker, and Shilad Sen. Collaborative filtering recommender systems. In The adaptive web, pp. 291-324, Springer, 2007.
12

---

Under review as a conference paper at ICLR 2023
Ali Shafahi, Mahyar Najibi, Mohammad Amin Ghiasi, Zheng Xu, John Dickerson, Christoph Studer, Larry S Davis, Gavin Taylor, and Tom Goldstein. Adversarial training for free! Advances in Neural Information Processing Systems (NeurIPS), 32, 2019.
Yunsheng Shi, Zhengjie Huang. Wenjin Wang. Hui Zhong. Shikun Feng, and Yu Sun. Masked label prediction: Unified message passing model for semi-supervised classification. arXiv preprint WwXiv:2009003509 2020.
Yunsheng Shi, PGL Team, Zhengjie Huang, Weibin Li, Weiyue Su, and Shikun Feng. Runimp: So- lution for kddcup 2021 mag240m-Ise. 2021. URL
kcdcup2021/maan mag24 BD-PGL. påf
Petar Velickovic, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. In International Conference on Learning Representations (ICLR), 2018.
Petar Veličković, William Fedus, William L Hamilton, Pietro Liò, Yoshua Bengio, and R Devon Hjelm. Deep graph infomax. In International Conference on Learning Representations (ICLR), 2019.
Vikas Verma, Meng Qu, Kenji Kawaguchi, Alex Lamb, Yoshua Bengio, Juho Kannala, and Jian Tang. Graphmix: Improved training of gnns for semi-supervised learning. In AAAI Conference on Artificial Intelligence, volume 35, pp. 10024-10032, 2021.
Xiang Wang, Xiangnan He, Meng Wang, Fuli Feng, and Tat-Seng Chua. Neural graph collaborative filtering. In ACM SIGIR conference on Research and development in Information Retrieval (SIGIR), pp. 165-174, 2019.
Yiwei Wang, Wei Wang, Yuxuan Liang. Yujun Cai, and Bryan Hooi. Curgraph: Curriculum learning for graph classification. In Proceedings of the International World Wide Web Conference (WWW),
pp. 1238-1248 2021.
David H Wolpert. The lack of a priori distinctions between learning algorithms. Neural computation, 8(7):1441-3300, 1996.
Felix Wu, Tianyi Zhang, Amauri Holanda de Souza Jr, Christopher Fifty, Tao Yu, and Kilian Q Weinberger. Simplifying graph convolutional networks. In International Conference on Machine Learning (ICML). 2019.
Keyulu Xu, Chengtao Li, Yonglong Tian, Tomohiro Sonobe, Ken-ichi Kawarabayashi, and Stefanic Jegelka. Representation learning on graphs with jumping knowledge networks. In International Conference on Machine Learning (ICML), pp. 5453-5462, 2018.
Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? In International Conference on Learning Representations (ICLR), 2019.
Rex Ying, Ruining He, Kaifeng Chen, Pong Eksombatchai, William I. Hamilton, and Jure Leskovec. Graph convolutional neural networks for web-scale recommender systems. In ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD), pp. 974-983, 2018.
Jiaxuan You, Zhitao Ying, and Jure Leskovec. Design space for graph neural networks. volume 33, pp. 17009-17021, 2020a.
Jiaxuan You, Jonathan M Gomes-Selman, Rex Ying, and Jure Leskovec. Identity-aware graph neural networks. In AAAI Conference on Artificial Intelligence, volume 35, pp. 10737-10745, 2021.
Yuning You, Tianlong Chen, Yongduo Sui, Ting Chen, Zhangyang Wang, and Yang Shen. Graph contrastive learning with augmentations. Advances in Neural Information Processing Systems (NeurIPS), 33:5812-5823, 2020b.
Hanqing Zeng, Hongkuan Zhou, Ajitesh Srivastava, Rajgopal Kannan, and Viktor Prasanna. Graph- Saint: Graph sampling based inductive learning method. In International Conference on Learning Representations (ICI.), 2020.
13

---

Under review as a conference paper at ICLR 2023
Muhan Zhang and Yixin Chen. Link prediction based graph neural networks. In Advances in Neural Information Processing Systems (NeurIPS), pp. 5165-5175, 2018.
Shichang Zhang, Yozen Liu, Yizhou Sun, and Neil Shah. Graph-less neural networks: Teaching old mlps new tricks via distillation. In International Conference on Learning Representations (ICLR), 2022.
Lingxiao Zhao and Leman Akoglu. Pairnorm Tackling oversmoothing in gnns. 2019.
Tong Zhao, Yozen Liu, Leonardo Neves, Oliver Woodford, Meng Jiang, and Neil Shah. Data augmentation for graph neural networks. AAAI Conference on Artificial Intelligence, volume 35, pp. 11015-11002,, 2021.
Wenqing Zheng, Edward W Huang, Nikhil Rao, Sumeet Katariya, Zhangyang Wang, and Karthik Sub- bian. Cold brew: Distilling graph node representations with incomplete or missing neighborhoods. In International Conference on Learning Representations (ICLR), 2022.
14

---

Under review as a conference paper at ICLR 2023
Table 5: Statistics of nodes used for the transductive evaluation. For link prediction and the recom- mender system graphs (user-item bipartite graphs), we only evaluate on nodes/users that have at least one edge in the validation set.
Task
Dataset
#Nodes
Avg deg.
Feat. dim
arxiv
143,941
12.93
Node cls.
products
2.277.597
48.01
100
flickr
82,981
4.81
500
ppi
15,390
17.30
1,280
Link pred.
arxiv
141,917
7.20
128
gowalla
29,858
3.44
Recsys
yelp2018
31,668
4.93
amazon-book
52,643
5.67
Table 6: Semi-supervised node classification performance with GCN the backbone architecture. The evaluation metric is classification accuracy. For the "Inductive (cold)", 90% of edges are randomly removed from new nodes. For the results with other edge removal ratios, refer to Table & in Appendix.
Method
arxiv
products
Transductive
Inductive
Inductive (cold)
Transductive
Inductive
Inductive (cold)
Base
0.6922.0.000
0.6893
0.5497:0008
0.8432-0000
0.8447=0000
0,7461 #00004
DropEdge
10.00s
0.6939:00001
0.5630
0.8486_0008
0.8497
0.7671 10:0000
LocalAug
0.6987 29.012
0.6962 :4.002
0.5660=00017
0.8488
0.8501 -00005
0.7526#0000
ColdBrew
0.6867
0.6732:0.000
0.5369 100009
0,8399
0.8406-000m
0.737
GraphL.es:NNN NN
0.6129 100007
0.5462
1.54622.00ms
0.6669_.0oo
0.6648 20000
0.66488 10.000
RAWLS-GOE GCN
0.6708:s.00m
0.6698 s0.002
0.5324:00025
0.8209-0000
0.8220=00011
0.7096:60006
Tail-GNN
0.6437.0ooss
0.642310.00m
290045
0OM
0OM
0OM
TUNEUP w/o curriculum
0.696] 1000000
0.6924
0.5593=0.002
0,8481 =00000
0.8490=00006
0.7624=00000
TUNEUP wlo syn-tails
0.69400s..00s
0.6924
1.6924--00000
0.5607:=0.007
0.8452.00006
0.8468-n.000
0.7549
TUNEUP w/o pseudo-labels
0.6967
0.69362s00005
0.5534=00009
0.8488-00007
0.8497=0000
0.7676=00:10
TUNEUP (ours)
0.6988 :9,000s
0.6973:9.001
0.5923:000053
0.8523
0.7925
Relative gain over base
+1.0%
+1.1% 1%
+7.8%
+1.1%
+1.0%%
+6.2%
A DETAILS OF
Here present the details of hyper-parameters used in experiments.
Semi-supervised node classification. We used the hidden dimensionality of 256 and 64 for arxiv and products, respectively. We trained GNNs in a full-batch manner, and for products, reduced dimensionality of 64 so that the entire graph can be fit into the limited GPU memory of 45GB. Mini-batch training is left for future work. We used 1500 epochs for both the default training and finetuning. The learning rate is set to 0.001.
used the
Link prediction. We used the hidden dimensionality of 256 for all datasets. We added the L2 regularization on the node embeddings and tuned its weight for each dataset and GNN architecture. For both the default training and finetuning, used 1000 epochs and the learning rate of 0.0001.
Recommender systems. We used the shallow embedding dimensionality of 64 and the hidden emhedding dimensionality of 256. Similar the link prediction, added the 1.2 regularization to the node embeddings and tuned its weight for each dataset and GNN architecture. For the default training, we trained the model for 2000 epochs with the initial learning rate of 0.001, which is multiplied by 0.1 at the 1000th and 1500th epoch. For finetuning, we used 500 epochs with the learning rate of 0.0001.
For training strategies without curriculum learning, we used the same configuration as the default training.
15

---

Under review as a conference paper at ICLR 2023
Dataset: Model: GraphSAGE
Dataset: products Model: GraphSAGE
0.8
TunetIp
-4-
Base
0.80-
0.75
ACCURIC
0.6
Accurlry
0.70
0.5
0.65
0.60
0.55
TuneUp
0.3
0.50
Base
degree (Log-scale)
degree [Log-scale]
Dataset: anviv/ Model: GCN
Dataset: products Model: GCN
0.8
TuncUp
0.85
TuneUip
0.7
0.75
Rccurscy
0.6
0.65
0.60
0.50
10a
10'
Node degree 1 (Logescale)
degree- (Logescale)
Figure 2: Degree-specific generalization performance of the base GNN and TUNEUP in transductive semi-supervised node classification. The evaluation metric is classification accuracy.
Datanet andy/ Model: Graph
Turstlo Boo
0000
0.13
0.30
012
Tundil Dane
0:38
0.00
10
18"
degroe
rngree sokid
Mode (aspeceel
Datacet: fickr / Mocel GCN
Dataset: pol/ Model GON
Dutasit: ondy Made: GCN
6.20
Turelz
0.20
0.15
0035
1022
0.58
0.65
0.00
0.02
10
36'
Mode degres 2.2 flog scalo)
Node segree 1log oc
seprec
Figure 3: Degree-specific generalization performance of the base GNN and TUNEUP in transductive link prediction. The evaluation metric is recall @50.
16

---

Under review as a conference paper at ICLR 2023
Dataset gowall
Daraset: yolp20L8
amazon book/
1:35
0:30
0.33
Scoe
Anerage Bocatg00
6.25
Suse
0:23
Aeerage Bocal
0.73
5.03
0:00
6:38
10
12
107
157
12
Mode degriee d.pg stalo
Node degins 0.0g 0.ogsaal)) sold)
Node degiee Queesaaa)
Oatacet: gowalla / Modell GON
Dataset: yelp2018/ Model: GCN
Dataset: amazon book Model: GCN
0.35
Tarell)
Tunelip
3.33
Recalito
0.25
Aeenaa Recaingoo
0.25
0.23
8,55
5.05
8.00
Mede
Node (.og-etaa)
Figure 4: Degree-specific generalization performance of the base GNN and TUNEUP in transductive recommender systems. The evaluation metric is recall 50.
Table 7: Cold-start inductive node classification performance with GraphS the backbone architecture. The larger the edge removal ratio is, the more cold-start the prediction task becomes. The evaluation metric is classification accuracy.
arxiv
products
Method
Edge removal ratio
Edge removal ratio 60%
30%
60%
90%
30%
90%
Base
0.6452
0.6000
0.833410.0000
0.7226.10001
DropEdge
0.6532g0009
0.6247=200000
0.5446200000
0.8409 20.0005
0.7710 0.8
LocalAug
0.6552 1000044
0.6158 10.0014
0.4976.0.000
0.8370 10.0000
0.8166 100008
0.7256 1000077
ColdBrew
0,6289100022
0..043..0.042
0.5080.00.085
10,000
0.7417.10002 0.8142
GraphLessNNN
0.0057
0,5457
0.5457=0.006
0.6640=0.000
0.6640=0.006
0.6640
Tail-GNN
0.6131
0.5089
OOM
OOM
OOM
TUNEUP w/o curriculum
0.8395 0.5474.0.011 0.6270=00000
0.7573 2.8243
TUNEUP w/o syn-tails
0.6523100009
0.6103
2000004
0.49000...00
0.8361
1000004
0,8161 1.0.000
0.7255 1000011
TUNEUP w/o pseudo-labels
0.6505 1000015
0.6198 1000051
0.5332.:0.016
10,000
0. 0.8260
0.7630
TUNEUP (ours)
0.6673::00015
0.6507=0.002
0.,00600.0010
0.8523 0.0000
1.8434:00000
0.8054
Relative gain over buse
+3.4%
+8.4%%
+26.5%
+2.3%
+3.8%
+11.5%
Table 8: Cold-start inductive node classification performance with GCN the backbone architecture. The larger the edge removal ratio is, the more cold-start the prediction task becomes. The evaluation metric classification accuracy.
arxiv
products
Method
Edge removal ratio 60%
Edge removal ratio
30%
90%
30%
60%
90%
Base
0.6400=so.003 0.6015
0.8375=0.000 0.5497=0.002
0.8210=anom
0.7461 #:00004
DropEdge
0.6753+00001
0.6475=0.003
0.5630=0.009
0.8438
0.8301
0.767
LocalAug
0.6785
0.6491 =0.002
0.5660=00017
0.8429 =0.0000
0.8267
0.7526=80011
ColdBrew
0.6505
0.6180=0.000
0.5369=0.009
0.8330
0.815620000
0.7371 L:000S
GeaphLessNNN
0.5462.10002
0.5462.2..0p
0.5462..0.012
0. 0.6648 1000000
0. 6.6688 L00000
0.6648 1000077
RAWLS-GCN
0.6488110005
0.6115 20.015
0.5324.0.002
0.812410.001
0.7916.00002
0,7096100000
Tail-GNN
0.0083
0.6032-s00013
00.001
OOM
OOM
0OM
TUNEUP w/o curriculum
000055
0.6448 =0.0023
0.5593=0.00m2
0.8427=00.0m
0.7624 0.8285=
TUNEUP w/o syn-tails
0.6741 +0.0015
0.6468
=0.0025
0.5607=00.007
0.8398
0.8246
0.7549::0000
TUNEUP w/o pscudo-labels
0.6750 =000000
0.6439=0.001
0.5534=0.0029
0.8439=000mm
=000005
0.8300=n.com
0.
TUNEUP (ours)
0.6808
0.6604-4.omm
0.5923-0..053
0.8489 =0.000
0.8385
0.8385
0.7925=au000
Relative gain over buse
+1.4%
+3.2%
+7.8%
+1.4%
+2.1%
+6.2%
17

---

Under review as a conference paper at ICLR 2023
Table 9: Link prediction performance with the GCN as the backbone architecture. The evaluation metric is recall 50. For the "Inductive (cold)", 60% of edges are randomly removed from new nodes. For the results with other edge removal ratios, refer to Table 1I in Appendix.
Method
flickr
PM
arxis
Tranductive
Inductive
Indurtive (celd)
Transduction
Inductive (cald)
Inductive
(cold)
0.130 0.145aara01
0008
2.1891
D
Depe
0,1425 0.1563acam0
0.1036 20039
0.1903
0.8712
0.1364 .ame
0.250%
0.2 0.2116.ca441
0.1346 1000
Loca/Aug
0.1445
0,0936.200
0,1450
0.1321 Laaae
0.6920 .a00
0.2134 0.2200
0.1216 LE00s
Coddfern
2.06
0.11 R
0.1 PXN
81 1122
g1
Tail-ONN
a 8.1282x--m
0.0810-2100
0. 1.1564amss
0. m.
0. 0.125
B. 1144
0.08 sa-
RAWLS OCN
0,8488
0,0030
0.1280
0.0653
0.8453
0.8110cca00
0.0418 c0004
TuxeUU we curticllun
0.1466 113011
0.182788002
0.1511
0.1341 .000
JAINET
0.1195
TuNeU's
(.1637:am0
6,1517
0.1050-40mz
0.1857
0L1430-sam
0.2417
0.2254:
6,1440-amm
Rebative pin here
-14.4%
+10.8%
+22.5% 5%
+54 872
+57 77.7%%
+57 45%.4%
+12 35
5%
+:01 9S
Table 10: Cold-start inductive link prediction performance with GraphSAGE. The evaluation metric recall @ 50. The larger the edge removal ratio is, the more cold-start the prediction task becomes.
flickr
arsiv
Method
Edige removval ratio
Edge removal ratio
Edge ressoval ratio
30%
60%
90%
30%/
60%
90%
30%
60%
90%
Base
0.0630.uum
0.0186 0.050
0.0835:naes 0.0900
0,0708:000)
0,1021
0.0722
0.0240
Drop
0,1007 0.1
0.0902
0.1401:aam2
0.1282:0:94
0,6999
0,153
0.1204
0.0579
LocalAug
0,0855
0,0619
0.0185
0.3110
0,0950
0.0717.0212
0,1081
0.0750
ColdBeew
0.0556-0re
00046-3-002
0.0261-e-09>
0.0857-mmem
0.6318-30055
Tail-GNN
0.0767.00011
0,0521
0.0939-3.02
0.0029.6:003
0.0642460033
0,077
0.0410-9004
TENNIi curricluu
0.119225n0is
1004
0:1974:3300
0.131520000
0.0005
TUNEUP calls
0.0184:002>
0.0988:2.440
0.0824:1:4::
0.0710.00m:
0.0273
TUNEUR (ours)
0.1247 c0a03
0.10M2 coE
0,0715 .00441
0.1513 1.00
0.1328 6.043
0.1045
0.159
4.044
gain over
+255.1%
+52.8%
+73,3%
+44.55
Table 11: Cold-start inductive link prediction performance with GCN. The evaluation metric is recall @ 50, The larger the edge removal ratio is, the more cold-start the prediction task becomes.
ppi
arxiv
Method
Fdgg removal ratio 60S
Edge removal ratio
Edge remeraa ratio 60%
30%
90%
50%
0.11442
0.0361 gores
0.1083:3000
2.00
0.1777
0.110220000
0.0376-9005
Droplidge
0.100
0.0001-00e
0.1600
0.1364:
0.1822-0ox) 1.976
0.1346-00000
0.0997 aaccs
LocalAug
0.0926.uue
0.0282.mme24
0.115:33a7
0.0920
0.0400
0.1772-00011
0.1216
0.0378-axx2
ColdBlow
0,0937. e003
0.0086 oues
0.0888 iaass
0,0328 0.0376643223
0.1399 L00e5
0.9927
L0005
010415 -000
Tail-INN
n. 1.166940ms
0.0830
0.0541
0.800::amm
D. 1.000 0.0560400008
0.811
0.081
RAW1 S-GON
0.04375noon
0.8430
0.0421
0.0678-3.0m
0.0453:e-003
0.099410000
0.0623-noomt
0.048-0000
0.0230
TINEEte wio curriulum
0.1272
D. 1018
0,0661 0.0661:0ue soeet
0.145723a
0.1241:0-00
0.8754:0a07
0.1852_nees
0.1356-000
00057723000
TUNEUP wio syn calls
0.1166
0.0833
2002
0.8110:3am5
0.0916.8338
0,0047.0005
0.1727 20020
0.1195 00014
0.0368.11
-0000
TUNEUP (ours)
0.1308
0
0.6661-000
0.1714
0.3430
0.1933-6/44
0.1440-00617
0U8627
Relazive gain over base
+14.4
+22.5%
+153.2%
+58.25%
+57.4%
+106.4%
+12.6%
+20.9%
+67.0%
18


Reviews:
[{"review_id": "rpA8omftfk", "reviewer": "Reviewer_Yj9L", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a two-stage curriculum learning strategy -- TuneUp --  to improve GNN's performance on tail nodes of small node degrees. The paper is motivated by the observation that GNNs perform worse on nodes with smaller degrees. \n\nIn this first stage, TuneUp conducts standard training without augmentations. In the second stage, TuneUp trains GNNs with DropEdge augmentation. The paper argues that the DropEdge augmentation in the second stage can synthesize tail nodes from head nodes, thus providing more tail nodes for training.\n\nTuneUp is evaluated in three tasks (semi-supervised node classification, link prediction, and recommendation). When the task is semi-supervised node classification, TuneUp also uses pseudo labels in the second stage of training.", "summary_of_the_review": "The paper presents a two-stage curriculum learning method to improve performance on tail nodes of small degrees. The novelty is weak and the experimental results are insignificant. The experimental settings need more careful treatments. The paper is not ready for publication at the current stage. Hence, I am leaning on the negative side.", "strength_and_weaknesses": "**Strengths:**\n\n* The paper is clearly written.\n* The studied problem of generalization to tail nodes is important for recommender systems and social network analysis.\n* The proposed model is tested on extensive benchmarks from various domains. \n\n\n\n**Weakness**:\n-  The technical novelty is weak. \n   1. The methods of DropEdge and Pseudo Labeling are well explored in previous works, as already cited in this paper. \n   2. The proposed TuneUp method is a straightforward adaptation of the \"from easy to difficulty\" idea from Curriculum Learning. \n\n- The experimental results are insignificant. \n   1. In Table1, *TuneUp w/o pseudo-labels* underperforms *DropEdge*. The result shows that the proposed two-stage training method is no better than a single-stage training with augmentation. Although using pseudo-labels can improve performance, pseudo labeling is proposed by previous works and does not align with the motivation of this paper. \n   2. In Table2, TuneUp's improvement over DropEdge is marginal. The result shows that the improvement over the base model is mainly attributed to the DropEdge augmentation rather than the proposed method.\n\n- The experimental settings need more careful treatment. \n   1. In sec4.2, the experimental setting of the link prediction task follows that of two papers from a different domain -- recommender system -- instead of link prediction. The authors are encouraged to follow the experimental setting of papers from the same domain for appropriate evaluation. \n   2. It is weird to use an ogb**n** (**n** for node) dataset for link prediction evaluation, since the ogb benchmark has multiple ogb**l** (**l** for link) datasets designed specifically for link prediction evaluation.\n   3. I also notice that the experimental settings of semi-supervised node classification and recommendation are new and different from existing works. The authors are encouraged to report the performances under the same evaluation protocols of existing works. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "gU6jTbH3gL", "reviewer": "Reviewer_1BhX", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work proposes a curriculum learning strategy, named TUNEUP for GNN, which aims to improve the performance of GNNs at the tail nodes. TUNEUP first trains a GNN that produces good performance on head nodes, then synthesizes tail nodes by dropping the edges of head nodes, and then fine-tunes on these synthesized tail nodes. The authors conduct three types of experiments: semi-supervised node classification, link prediction and recommender systems. Extensive experiments demonstrate the effectiveness of this method.", "summary_of_the_review": "I think the authors should give theoretical or experimental explanations and clarifications for these three assumptions.", "strength_and_weaknesses": "Strength:\n1. The authors performed three different types of GNN applications, all with improved results.\n2. It is significant to improve the performance of the tail node.\n\nWeakness:\n\nThe author's approach is based on three very strong assumptions, which I disagree with, as follows:\n1. The first one is \"Tail nodes can be synthesized by dropping edges from head nodes\".\nWhy the DropEdge can synthesize the tail node from the head node?\nI think in most cases， dropping edges from head nodes would give synthetic tail nodes that have different characteristics from real tail nodes\nThe authors did not prove this experimentally or theoretically, so I don't think this assumption is convincing enough.\n\n2. The second one is \"Target supervision on head nodes can be reused for synthetic tail nodes\".\nThe author points out that the degree of a node is not related to its properties, which I think is incorrect.\nIn most cases, the degree of a node is strongly related to its own properties.\nFor example, in a social network, celebrities will have many people to follow, thus resulting in a high degree.\nI think the authors should also give explanations from an experimental or theoretical point of view.\n\n3. The third one is \"Edge information helps for GNNs to make prediction\".\nIn many cases, more edges are not always better.\nReal-life graph data has many heterogeneous connections, and these edges can lead to performance degradation.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "fGr7rqUOs4", "reviewer": "Reviewer_xBXx", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a two-stage curriculum learning process for GNNs to ensure better performane in a variety of tasks. Specifically, the paper argues that low-degree nodes are the hardest to make predictions for and focus on improving this.", "summary_of_the_review": "The paper is clear and well written. The method is a small technical innovation that seems applicable across a wide variety of tasks. However, the empirical evaluation needs to be improved, in particular the GNN architectures considered as well as using a test set for link prediction.\n\nIf these concerns are addressed, I am willing to raise my score.", "strength_and_weaknesses": "Strengths:\n- Clear communication, easy to understand\n- Discusses the assumptions that enable the method to perform better.\n- General method, i.e. applicable in many areas\n- Good dataset selection to evaluate empirical performance\n- Covers both transductive and inductive settings, as well as cold-start nodes\n- Ablation studies done are meaningful and reinforce the points of the paper\n- Many other graph augmentation methods are compared against\n\nWeaknesses:\n- The GNN architecture used in evaluation is old and a narrow choice. The most standard choice would be an MPNN and then checking with the 3 most common aggregators, which I suspect would have a large impact on how the node degrees affect prediction. So I think the information provided by the empirical evaluation may not be useful to the community as GraphSAGE is simply not the default choice anymore. GCN is even older. The authors should rerun at least the largest dataset in each task (node classification, link prediction, recommender system) with the three MPNN (max, mean, sum aggregators).\n- The evaluation does not do enough to convince the reader that this is useful across GNN architectures, for that MPNN with different aggregators are the minimum.\n- The deviation in the evaluation protocol for node classification, while sensible, is unfortunate in that it prevents easy comparison for past, concurrent, and future work as it is not standard.\n- For link prediction there is no test set, which I find problematic and is not an acceptable protocol for a top machine learning conference.\n\nMinor:\n- The standard deviation numbers in a lot of tables (especially Table 2.) are too small to be readable, the authors would be better off reporting them in the appendix and removing them from the main paper. It harms the readability too significantly.\n\nQuestions:\n- You claim in section 2.4 that links in heterophilieous graphs would necessarily harm predictive performance in node classification. I can't imagine that is true, consider graph colouring, where adjacent nodes needs to have different colours and we are given an initial colouring for some of the nodes. This is a node classification problem in a heterophelic graph, link information is informative, in fact necessary for prediction. Am I missing something or are heterophileous graphs not a good example of the point you are trying make?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "HCShzSheg1", "reviewer": "Reviewer_khNG", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a training strategy for GNN, TuneUp, that applies curriculum learning with node synthesis and label generation. It is evaluated on multiple modalities and outperforms multiple data augmentation baselines.", "summary_of_the_review": "The paper proposes a training strategy for GNN that combines multiple data augmentation methods together and improves the performance of two basic GNN models. It is not well-written and several assumptions are not well supported.", "strength_and_weaknesses": "### Strengths\n1. The paper introduces a training strategy, TuneUp, that organically combines multiple data augmentation methods for GNN.\n2. The proposed TuneUp strategy is intuitive and performs well on multiple types of prediction tasks.\n3. It also considers an interesting cold-start scenario that may be useful for real applications.\n\n### Weaknesses\n1. The paper claims the TuneUp strategy can improve the performance of the GNN training. However, it is only tested on the basic GNN models, GraphSage and GCN. It would be interesting to see other GNN architectures that appear in the ogbn leaderboard to show the generality of the method.\n2. It would be interesting to see some examples of nodes and targets that are generated from the synthesis stage to give more insights into the method.\n3. The method adopts pseudo labels to improve performance. It would be also great to have some label-smoothing baselines.\n4. The paper made several assumptions for the augmentation techniques used in the algorithm. It can be more convincing to show some statistics of the graphs in different datasets, such as how many pseudo-labels are correct, and the accuracy for the head nodes vs tail nodes for the base model", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject