MLPInit: Embarrassingly Simple GNN Training Acceleration with MLP Initialization
Authors: Xiaotian Han
Tong Zhao
Yozen Liu
Xia Hu
Neil Shah
Venue: ICLR 2023 poster (2023)
Keywords: Graph Neural Network
Large-scale Graph
Accleration

Abstract: Training graph neural networks (GNNs) on large graphs is complex and extremely time consuming. This is attributed to overheads caused by sparse matrix multiplication, which are sidestepped when training multi-layer perceptrons (MLPs) with only node features. MLPs, by ignoring graph context, are simple and faster for graph data, however they usually sacrifice prediction accuracy, limiting their applications for graph data. We observe that for most message passing-based GNNs, we can trivially derive an analog MLP (we call this a PeerMLP) with an equivalent weight space, by setting the trainable parameters with the same shapes, making us curious about how do GNNs using weights from a fully trained PeerMLP perform? Surprisingly, we find that GNNs initialized with such weights significantly outperform their PeerMLPs, motivating us to use PeerMLP training as a precursor, initialization step to GNN training. To this end, we propose an embarrassingly simple, yet hugely effective initialization method for GNN training acceleration, called \mlpinit. Our extensive experiments on multiple large-scale graph datasets with diverse GNN architectures validate that MLPInit can accelerate the training of GNNs (up to 33× speedup on OGB-Products) and often improve prediction performance (e.g., up to $7.97\%$ improvement for GraphSAGE across $7$ datasets for node classification, and up to $17.81\%$ improvement across $4$ datasets for link prediction on metric Hits@10). The code is available at https://github.com/snap-research/MLPInit-for-GNNs.

Published as a conference paper at ICLR 2023
MLPINIT: EMBARRASSINGLY SIMPLE GNN TRAIN- ING ACCELERATION WITH MLP INITIALIZATION
Xiaotian Han
Tong Zhao²
Yozen Liu
Xia Hu
Neil Shah²
'Texas A&M University SSap Inc. 'Rice University
han@tamu.cm edu
(tzhao, yliu2, nshah @snap...p com
xia. hu@rice.ed
ABSTRACT
Training graph neural networks (GNNs) on large graphs is complex and extremely time consuming. This is attributed to overheads caused by sparse matrix multipli- cation, which are sidestepped when training multi-layer perceptrons (MLPs) with only node features. MLPs, by ignoring graph context, are simple and faster for graph data, however they usually sacrifice prediction accuracy, limiting their ap- plications for graph data. We observe that for most message passing- based GNNs, we can trivially derive analog MLP (we call this PeerMLP) with equivalent weight space, by setting the trainable parameters with the same shapes, making curious about how do GNNs using weights from a fully trained PeerMLP per- form? Surprisingly, we find that GNNs initialized with such weights significantly outperform their PeerMLPs, motivating to use PeerMLP training as precursor, initialization step to GNN training. To this end, we propose embarrassingly simple, yet hugely effective initialization method for GNN training acceleration, called MLPInit. Our extensive experiments multiple large-scale graph datasets with diverse GNN architectures validate that MLPInit can accelerate the training OGB OGB-products) and often improve prediction performance (c.g., up to 7.97% improvement for GraphSAGE across datasets for node classification, and up 17.81% improvement across 4 datasets for link prediction on metric Hits @ 10). The code is available at http://github.comssap Paaerrh/MMPInnn-ffo----MMM
of GNNs (up to 33x speedup
INTRODUCTION
Graph Neural Networks (GNNs) (Zhang et al., 2018; Zhou al., 2020; Wu et al., 2020) have attracted considerable attention from both academic and industrial researchers and have shown promising results on various practical tasks, e.g., recommendation (Fan et al., 2019; Sankar al., 2021; Ying et al., 2018; Tang et al., 2022), knowledge graph analysis (Arora, 2020; Park et al., 2019; Wang et al., 2021), forecasting (Tang et al., 2020; Zhao et al., 2021; Jiang & Luo, 2022) and chemistry analysis (Li et al., 2018b; You al., 2018; De Cao & Kipf, 2018; Liu et al., 2022). How- ever, training GNN on large-scale graphs is extremely mone-connuming and costly in practice thus spurring considerable work dedicated to scaling up the training of GNNs, even necessitating new massive graph learning libraries (Zhang et al., 2020; Ferludin et al., 2022) for large-scale graphs.
Recently, several approaches for more efficient GNNs training have been proposed, including novel architecture design (Wu et al., 2019; You et al., 2020d; Li et al., 2021), data reuse and partitioning paradigms (Wan et al., 2022: Fey et al., 2021: Yu et al., 2022) and graph sparsification (Cai al., 2020; Jin et al., 2021b). However, these kinds of methods often sacrifice prediction accuracy and increase modeling complexity, while sometimes meriting significant additional engineering efforts.
MLPs are used to accelerate GNNs (Zhang et al., 2021b; Frasca et al., 2020; Hu et al., 2021) by decoupling GNNs to node features learning and graph structure learning. Our work also leverages MI.s but adopts a distinct perspective. Notably, we observe that the weight space of MLPs and GNNs can be identical, which enables us to transfer weights between MLP and GNN models. Hav- ing the fact that MLPs train faster than GNNs, this observation inspired us to raise the question:
"This work done while the first author was an intern Snap Inc.

---

Published as a conference paper at ICLR 2023
023-arTis (GINN
DCB-arXty CCC)terCCN)
DGB (CmplSAGE
00B-artiv (OCN)
0673
0.4
Spendup
0.
Speadup
-030
Spendop
0.50
Agundep
Razdoom
Mii5
Epectins
Figure 1: The training speed comparison of the GNNs with Random initialization and MLPInit. indicates the best performance that GNNs with random initialization can achieve. * indicates the comparable performance of the GNN with MLPInit. Speedup indicates the training time reduced by our proposed MLPInit compared to random initialization. This experimental result shows that MLPInit is able to accelerate the training of GNNs significantly.
Can we train GNNs more efficiently by leveraging the weights of converged MILPs?
To answer this question, we first pioneer a thorough investigation to reveal the relationship between the MLPs and GNNs in terms of trainable weight space. For ease of presentation, we define the PeerMLP of GNN that GNN and its PeerMLP share the same weights 2. We find that interest- ingly, GNNs can be optimized by training the weights of their PeerMLP. Based on this observation, adopt weights of converged PeerMLP the weights of corresponding GNNs and find that these GNNs perform even better than converged PeerMLP on node classification tasks (results in Table 2).
Motivated by this, propose embarrassingly simple, yet remarkably effective method to accel- erate GNNs training by initializing GNN with the weights of its converged PeerMLP. Specificalll to train target GNN, we first train its PeerMLP and then initialize the GNN with the optimal weights of converged PeerMLP. We present the experimental results in Figure to show the training speed comparison of GNNs with random initialization and with MLPInit. In Figure 1, Speedup shows the training time reduced by our proposed MLPInit compared to random initialized GNN, while achiev- ing the same test performance. This experimental result shows that MLPinit is able the accelerate the training of GNNs significantly: for example, we speed up the training of GraphSAGE, GraphSAINT, ClusterGCN, GCN by 2.48x, x,94x, 2.06x, 1.91 x on OGB-arXiv dataset, indicating the superior- ity of our method in GNNs training acceleration. Moreover, we speed up GraphS AGE training more than 14x on OGB- DGB-products. We highlight our contributions as follows:
We pioneer thorough investigation to reveal the relationship between MLPs and GNNs in terms the trainable weight space through the following observations: (i) GNNs and MLPs have the same weight space. (ii) GNNs be optimized by training the weights of their PeerMLPs. (iii) GNN with weights from its converged PeerMLP surprisingly performs better than the perfor- mance of its converged PeerMLP on node classification tasks.
Based on the above observations, we proposed an embarrassingly simple yet surprisingly effective initialization method to accelerate the GNNs training. Our method, called MLPinit, initializes the weights of GNNs with the weight of their converged PeerMLP. After initialization, we observe that GNN training takes less than half epochs to converge than those with random initialization. Thus, MLPInit is able to accelerate the training of GNNs since training MLPs is cheaper and faster than training GNNs.
Comprehensive experimental results on multiple large-scale graphs with diverse GNNs validate that MLPinit is able to accelerate the training of GNNs (up to 33x speedup OGB (CB-prrductss while often improving the model performance (c.g., 7.97% improvement for node classification GraphSAGE and 17.81% improvement for link prediction on Hits 10).
MLPInit extremely easy implement and has virtually negligible computational overhead com- pared to the conventional GNN training schemes. In addition, it orthogonal to other GNN accel- eration methods, such as weight quantization and graph coarsening, further increasing headroom for GNN training acceleration in practice.
'The formal definition of PeerMLP is in Section 3.
BB share the same weight, we mean that the trainable weights of GNN and its PeerMLP the in terms of size, dimension, and values.
By performance, refer to the model prediction quality metric of the downstream task the correspond- ing test data throughout the discussion.
2

---

Published as a conference paper at ICLR 2023
2 PRELIMINARIES
xN) E RND is the node feature matrix and A {0, 1 is the binary adjacency matrix. N is the number of nodes, and D is the dimension of node feature. For the node classification task, we denote the prediction targets by Y E {0, 10.1....4 C 1}.², where C' is the number of classes. We denote GNN model as ggnn (X, A; Wggn), and an MLP as fmip (X; Wmlp), where Ugnn and Wmip denote the trainable weights in the GNN and MLP, respectively. Moreover, Wgnn and Wmip denote the fixed weights of optimal (or converged) GNN and MLP, respectively.
Notations. We denote an attributed graph G = (X, where X X2,
Graph Neural Networks. Although various forms of graph neural networks (GNNs) exist, our work refers to the conventional message passing flavor (Gilmer al., 2017). These models work by learning a node's 's representation by aggregating information from the node's neighbors recursively. One simple yet widely popular GNN instantiation is the graph convolutional network (GCN), whose multi-layer form can be written concisely: the representation vectors of all nodes at the I-th layer where o--) denotes activation function, W is the trainable weights of
are H' = "AH'-I the 1-th layer, and H ' is the node representations ouiput by the previous layer. Denoting the output of the last layer of GNN by H, for a node classification task, the prediction of node label Y = so). For a link prediction task, one can predict the edge probabilities with any suitable decoder, e.g., commonly used inner-product decoder as Á = sigmoid H (Kipf & Welling, 2016b).
3 MOTIVATING ANALYSES
In this section, we reveal that MLPs and GNNs share the same weight space, which facilitates the transferability of weights between the two architectures. Through this section, we use GCN (Kipf & Welling, 60) as prototypical example for GNNs for notational simplicity, but we note that our discussion is generalizable to other message-passing GNN architectures.
GNNs share the same weight space with MLPs. To show the weight space of GNNs and MIPs, we present the mathematical expression of one layer of MLP and GCN (Kipf & Welling, 201 as follows:
Motivation 1:
GNN: H' = (AH (
MLP: H' "H W'in)):
(1)
where W gnn and W mlp are the trainable weights of 1-th layer of MLP and GCN, respectively. If we set the hidden layer dimensions of GNNs and MLPs to be the same, then W mip and W gan will naturally have the same size. Thus, although the GNN and MLP are different models, their weight spaces can be identical. Moreover, for any GNN model, we can trivially derive a corresponding MLP whose weight space can be made identical. For brevity, and when the context of a GNN model is made clear, we can write such an MLP which shares the same weight space as PeerMLP, i.c., their trainable weights can be transferred to each other.
Motivation 2: MLPs train faster than GNNs, GNNs train slower than MLPs, owing to their non-trivial relational data dependency. We empirically validate that training MLPs is much faster than training GNNs in Table 1. Specifically, this is because MLPs do not involve sparse matrix multiplication for neighbor aggregation. GNN layer (here we consider simple GCN layer, as defined in Equation (1)) can be broken down into two operations: feature transformation (Z H'-lW') and neighbor aggregation (H' = AZ) (Ma et al., 2021). The neighbor aggregation and feature transformation are typically sparse and dense matrix multiplications, respectively. Table shows the time usage for these different operations on several real-world graphs. As expected, neighbor aggregation in GNNs consumes the large majority of computation time. For example, on the Yelp dataset, the neighbor aggregation operation induces a 3199x time overhead.
Given that the weights of GNNs and their PeerMLP can be transferred to each other, but the PeerMLP can be trained much faster, we raise the following questions:
1. What will happen if we directly adopt the weights of a converged PeerMLP to GNN?
2. To what extent can PeerMLP speed up GNN training and improve GNN performance?
In this paper, we try to answer these questions with a comprehensive empirical analysis.
3

---

Published as a conference paper at ICLR 2023
Table I: Comparison of the running time of forward and backward for different operations (i.e., feature transformation and neighbor aggregation) in GNNs. The time unit is milliseconds (ms).
Operation
0GB-arXiv
Flickr
Yelp
#Nodes #Edges
169343
89250
716847
1166243
899756
13954819
Forward
Backward
Total
Forward
Backward
Total
Forward
Backward
Total
Z XW = AZ
0.32
1.09
1.42
0.28
0.97
1.26
1.58
4,41
5.99
1028.08
1029.11 17
1.01
836.95
837.97
9.74
19157.17
19166.90
721x
665x
3199x
3.5
smip (X; Wmmp)
76
7.5
Croaeetropp Lose
fgnn [X, A; 18 1mmp)
3.0
25
25
mlp
2.5
2.0
-000
Epoch
Figure 2: The relation of GNN and MLP during the training of PeerMLP. Left: Cross-Entropy loss of f ynnn X, A: Wmip) (GNN) and fmip(X: (mpp)) (PeerMLP) on training set over training epochs of PeerMLP. In this experiment, GNN and PeerMLP share the same weight Wmlpp which are trained by the PeerMLP. Middle: training trajectory of PeerMLP on its own loss landscape. Right: training trajectory of GNN with weights from PeerMI. on GNN's loss landscape. 'he figures show that training loss of GNN with weights trained from MLP will decrease. The details are presented in Ap- pendix D.2. We also present loss curves validation/test sets and accuracy curves in Appendix A.4.
WHAT WILL HAPPEN IF WE DIRECTLY ADOPT THE WEIGHTS OF A CONVERGED PEERMLP TO GNN?
To answer this question, we conducted comprehensive preliminary experiments to investigate weight transferability between MLPs and GNNs. We made the following interesting and inspiring findings:
The training loss of GNN will decrease by optimizing the weights of its PeerMLP. We conducted a verification experiment to investigate the loss changes of the GNNs with the weights trained from its PeerMLP and present the results in Figure 2. In this experiment, we have two models, a GNN and its corresponding PeerMLP, who share the same weights Wmlp. That is, the PeerMLP is m X: Wmtip) and the GNN is ggon (X, A; Wmip)) We optimize the weights Willp by training the PeerMLP, and the loss curve of Fmipp [X; (Wmmp) is the blue line in the left figure in Fig- ure 2. We also compute the loss of GNN fgnn (X, A; Wmip)) with the weights from PeerMLP. The loss curve of [X,n(X A; Wmip) is shown in the red line. Figure 2 shows the surprising phenomenon that the training loss of GNN with weights trained from PeerMLP decreases consistently. Impres- sively, these weights (Wmip) were derived without employing neighbor aggregation in training.
Observation 1:
Converged weights from PeerMLP provide a good GNN initialization. As PeerMLP and GNN have the same weight spaces, a natural follow-up question is whether GNN can directly adopt the weights of the con- verged PeerMLP and perform well. We next aim to understand this question empirically. Specifi- cally, we first trained PeerMLP for target GNN and obtained the optimal weights w mip. Next, we run inference on test data using GNN with whnp of PeerMLP, i.e., applying fenn A; mmip) Table shows the results of Fmip (X; (mmpp)) and
Observation 2:
Table 2: The performance of GNNs and its PeerMLP with the weights of converged PeerMLP on test data.
Methods
PeerMLP
GNN
Improv.
Ml.iint
GrahhSAGE
56.0440.7
62.87:0955
12.18%
72.25-0.30
GraphSAINT 53.881041 63.262071
11.41%%
68 0.20
ClusterGCN
54.471041
60.81:1300
111633
69.53-0.50
GCN 56.31::021
$6.28
0.0.%
0.34
10G8- producta
GrapSSAGE
63.4322.14
74.322104
17.16%
80.04=0062
GraphSAINT 57.29-0.32
69.00=1344
20.44%
74.02-0.19
ClusterGCN
59.53-046
71.74
20.51%
78.48-0.64
GCN
62.6320.15
71.1120.10
13.55%
76.85_0.54
fonn (X, A; wins))) We can observe that the
GNNs with the optimal weights of PeerMLP consistently outperform PeerMLP, indicating that the
weights from converged PeerMLP can serve as good enough initialization of the weights of GNNs.
4

---

Published as a conference paper at ICLR 2023
4.1 THE PROPOSED METHOD: MLPINIT
The above findings show that MLPs can help the training of GNNs. In this section, we formally present our method MLPInit, which is an embarrassingly simple, yet extremely effective approach to accelerating GNN training.
The basic idea of MLPInit is straightforward: we adopt the weights of a converged PeerMLP to initial- ize the GNN, subsequently, fine-tune the GNN. Specifically, for a target GNN (fgnn (X, A; Wggn we first construct a PeerMLP (fmpp (X, A; Wmtp)), with matching target weights. Next, we optimize the weight of the PeerMLP model by training the PeerMLP solely with the node features X for m epochs. Upon training the PeerMLP to convergence and obtaining the optimal weights (wintp). we initialize the GNN with diip and then fine-tune the GNN with n epochs. We present PyTorch-style pseudo-code of MLPInit in node classification setting in Algorithm 1.
Training Acceleration. Since training of the PeerMLP is comparatively cheap, and the weights of the converged PeerMLP can pro- vide good initialization for the corresponding GNN, the end result is that we can significantly reduce the training time of the GNN. Assum- ing that the training of GNN from random initialization needs N epochs to converge, and N >> n, the total training time be largely reduced given that MLP training time is neg- ligible compared to GNN training time. The experimental results in Table 3 show that is generally much larger than n.
Algorithm PyTorch-style Pseudocode of MLPInit
i_gon: graph neural network model f f_mlp: PeerHLP of f_gnn
Train PeerHLP for Il epochs for X, p f_mlp(X)
dataloader_mlp
loss nn. San.Crreeeaatrepyoosssc Y) loss. backvarr))) Divimizer_mlp.stoe
Initialize GNV with MLPInit
(tt_allpttatteeiiee...ooo siee_dditt,,. -gnn.load_state attaatt_dittt"__ll________
" "H_mip.pt")
Train GXN n epochs for X, A, Y in datalondor_gmm p f_gnn(X, A)
Ease of Implementation. MLPInit is extremely easy to implement as shown in Algorithm 1. First, we construct an MLP (PeerMLP), which has the same weights with the target GNN. Next, we use the node features X and node labels Y to train the PeerMLP to converge. Then, we adopt the weights of converged PeerMLP to the GNN, and fine-tune the GNN while additionally leveraging the adjacency A. In addition, our method can also directly serve as the final, deployed GNN model, in resource-constrained settings: assuming n = O, we can simply train the PeerMLP and adopt diti directly. This reduces training cost further, while enabling us to serve a likely higher performance model in deployment or test settings, as Table 2 shows.
loss nn. CrasCrressaatrryyooss?? Y) loss. backvar())
4.2 DISCUSSION
In this section, discuss the relation between MLPInit and existing methods. Since we position MLPInit as an acceleration method involved MLP, we first compare it with ML.--ased GNN accel- eration methods, and we also compare with GNN Pre-training methods.
Recently, several works aim to sim- plify GNN to MLP-based constructs during training or inference (Zhang al., 2022: Wu et al., 2019; Frasca et al., 2020; Sun et al., 2021; Huang et al., 2020; Hu et al., 2021). Our method is proposed accelerate the message passing based GNN for large-scale graphs. Thus, MLP-based GNN acceleration is a completely different line of work compared to ours since it removes the mes- sage passing in the GNNs and uses MLP to model graph structure instead. Thus, MLP-bbaed GNN acceleration methods are out of the scope of the discussion in this work.
Comparison to MLP-based GNN Acceleration Methods.
Comparison to GNN Pre-training Methods. Our proposed MLPinit are orthogonal to the GNN pre-training methods( You et al., 2020b; Zhu et al., 2020b; Veličković et al., 201 8b; You et al., 2021; Qiu et al., 2020; Zhu et al., 2021; Hu et al., 2019). GNN pre-training typically leverages graph augmentation to pretrain weights of GNNs or obtain the node representation for downstream tasks. Compared with the pre-training methods, MLPInit has two main differences (or advantages) that significantly contribute to the speed up: (i) the training of PeerMLP does not involve using the graph structure data, while pre- pre-training methods rely on it. (ii) Pre-training methods usually involve graph data augmentation (Qiu et al., 2020; Zhao et al., 2022a), which requires additional training time.
5

---

Published as a conference paper at ICLR 2023
Table 3: Speed improvement when MLPInit achieves comparable performance with random initial-
ized GNN. The number reported is the training epochs needed. (-) means our method can not reach
comparable performance. The epoch used by Random/ML PInit is denoted as * Figure 1. The detailed speedup computation method are presented in Appendix D.3.
Methods
Flickr
Yelp
Reddit
Reddit2
A-producta
OCB-arXiv
DGB aoB-prrdacts
Avg.
GCN C-GCN SAINT SAGE
Random
45.6
44.7
36.0
48.0
48.9
46.7
43.0
44.7
MLPInit
39,9
20.3
7.3
7.7
40.8
22.7
2.9
20.22
Improv.
1.14x
2.20x
4.93x
6,23
1,20
2.06x
14.83
2.21
Random
31.0
35.8
40.6
28.3
50.0
48.3
44.9
40.51
MLPInit
14.1
0.0
21.8
6.1
9.1
19.5
16.9
14.58
Improv.
2.20x
1.86x
4.64
5,49
2.48x
2.66
2.77x
Random
15.7
40.3
46.2
47.0
37.4
42.9
42.8
38.9
MLPInit
7.3
18.0
12.8
17.0
1.0
10.9
15.0
11.7
Improv.
2.15x
2.24x
2.76x
37.40x
3.94x
2.85
3.32
Random
46.4
44.5
42,4
2.4
47.7
46.7
43.8
45.35
MLPInit
30.5
23.3
0.0
0.0
0.0
24.5
1.3
19.9
Improv.
1,52
1.91
1.91
33.69
2.27
OGB-a:Xiv (GCN)
028-aritiv (GmpLSSGG))
0G5-artiv (CippSAANTT)
DOB-acKiv (ChusterCCN)
1.2
16-
1.2
2,4
3.1
1.55
1.0
Epoch
Epocs
0.00
0.0
4.100
0.0
Random-sni
0.35
6.0
Rardom--ost
MLPlnit-v.
6.0
Figure 3: The training curves of different GNNs on DGB-arXiv. GNN with MLPInit generally obtains lower loss and higher accuracy than those with the random initialization and converges faster. The training curves are depicted based on ten runs. More experiment results are in Appendix A.
5 EXPERIMENTS
In the next subsections, we conduct and discuss experiments to understand MLPinit from the follow- ing aspects: (i) training speedup, (ii) performance improvements, (iii) hyperparameter sensitivity, (iv) robustness and loss landscape. For node classification,
consider Flickr, Yelp. Reddit, Reddit2, A A-products, and two OGB datasets (Hu et al., 2020), OGB-arXiv and OGB-products as benchmark datasets. We adopt GCN (w/ mini-batch) (Kipf & Welling, 2016a), GraphSAGE (Hamil- ton et al., 2017), GraphSAINT(Zen) backbones. The details of datasets and baselines in Appendices C.1 and C.2, respectively. For the link prediction task, we consider Cora, Ci CiteSeer, PubMed, CoraFull, CS, Physics, A-Photo, datasets. Our link prediction setup is using as GCN an encoder which
et al., 2019) and ClusterGCN (Chiang et al., 2019) as GNN
and A A-Computers transforms graph to node representation H and an inner-product decoder A sigmoid(H H²) to predict the probability of the link existence, which is discussed in Section 2.
5.1 How MUCH CAN MLPINIT ACCELERATE GNN TRAINING?
In this section, we compared the training speed of GNNs with random initialization and MLPInit. We computed training epochs needed by GNNs with random initialization to achieve the best test performance. We also compute the running epochs needed by GNNs with MLPInit to achieve com- parable test performance. We present the results Table 3. We also plotted the loss and accuracy curves of different GNNs OGB-arXa in Figure 3. We made the following major observations:

---

Published as a conference paper at ICLR 2023
Table 4: Performance improvement when GNN with random initialization and with MLPInit achieve
best test performance, respectively. Mean and standard deviation are calculated based on ten runs.
The best test performance for the two methods is independently selected based on validation data.
Methods
Flickr
Yelp
Beddit
Reddit2
A-producta
DGB-arXiv
0GB-producta
Avg.
GCN C-GCN SAINT SAGE
Random 53.7240.16
63.03:to.0
96.50+a0a
77.58:t005 $1.76+2553
12.00
80.05
70.66
MLPInit 53.82.10.3
63,93
96.661100
77.74 89.60
10,30
80
76.29
Improv. 0.19%
t 1.43%
0.16%
1 73.09%
' 0.21%
0.36%
0.00%%
+ 7.97%
Random 51.37=
29.42
67.95 59.31 36.45+4m 05.58
25
73.80
59.12
MLPInit 51.35
13.10
41.71++125 1006
68.80 68.24
74.02
63.26
Improv. 4 0.05%
46.47%
0.06%
t 14.45%
15.06%
1.25%
0.30%
t 7.00%
Random 49.954015
56.394404
53.79 0007
52.74
68.00+0.99
78. 7 +:0.99
65,04
MLPInit
49.964020
58.05#0.66
96.02+004
77.77+198
55.61
49.53
78.48
69.34
Improv. + 0.02%
t 2.94%
1 0.33%
1 444.60%
t 5.45%
2.266
0.30%
t 6.61%
Random 50.904a11
40.08+015
92.7840.11
36.354a15 27.87+345
70.25
77.08
56.47
MLPInit 11.16
40.83+027
39.70 20.37 91.40.:o00
70.35
76.85 10034
64.38
Improv. 1 0.51%
t 1.87%
4 1.49%
188.8 42%
1 9.22%
0.14%
4. 0.29%
14.00%
Table 5: The performance of link prediction task. The results are based on ten runs. The experiments on other datasets are presented in Table 6. More experiments are presented in Appendix A.I.
Methods
AUC
AP
Hits@ 10
Hits@20
Hits@50
Hits@ 100
MLP
94.7620.
94.28
14. 12.68
24.01
40.02
24
Pubbedd
GNNandom
96.66+0.29
96.78
28.38+6.11
42.55
60.62+4.29
75. 15.14
GNNmipinit
17,31
97.53
37,58
51,83
70.57
81,42:1.52
Improvement
t 0.68%
1 0.77%
t 32.43%%
1 21.80%
1 16.42%
1 8.36%
MLP mindoo
95.20+0.18
+5.5
28.70+3.73
39.22-+4.13
53.36+3381
64.83+
DELP
GNN random
96.29.10.0
96.641002
36.55
43.13:1285
59.981244
71.57:11.00
GNN mipinit
96.67=0.13
97.09+0.14
40.84
53.72#425
67.99
77.76+1.20
Improvement
1 0.39%
1 0.47%
11.73%
1 21.57%
t 13.31%
8.65%
A-Photo
MLPmandon
86.1811.4
85.37
4.3621.14
6.9621.88
12.20
17.91
GNN randoom
92.07+214
91.52++208
9,63+1.58
12.82+1.72
20.90++100
29.08
GNN
93.99
93.32:1000
9.17+2.11
13.12:1211
22.93:12.66
32.37
Improvement
1 2.08%
1 1.97%
1. 4..75%
1 2.28%
9.73%
1 11.32%
MLP MLPrandom
96.2620.11
95.63
5.38
8.76
15.861081
24.70=
Phyaics
GNN randomo
95.84+0.13
95.38++.15
6.62+1.00
10.39
18.55+1.00
26.88
miNmipiiit
96.8910.07
96.55
8.05
13.06
22.38:1191
32.31
Improvement
t 11.00
1 1.22%
21.63%
1 25.76%
1 20.63%
1 20.20%
Avg.
1.05%
1 1.10%
17.81%
20.97%
14.88%
1 10.46%
Observation 3: MLPinit can significantly reduce the training time of GNNs. In this experiment, we summarize the epochs needed by GNN with random initialization to obtain the best performance, and then we calculate the epochs needed by GNN with MLPInit to reach comparable performance on par with the randomly initialized GNN. We present the time speedup of MLPIni in Table 3. Table 3 shows MLPInit speed up the training of GNNs by 2-5 times generally and in some cases even more than 30 times. The consistent reduction training epochs on different datasets demonstrates that MLPInit can generally speed up GNN training quite significantly.
5.2 How WELL DOES MLPINIT PERFORM ON NODE CLASSIFICATION AND LINK PREDICTION TASKS?
In this section, we conducted experiments to show the superiority of the proposed method in terms of final, converged GNN model performance node classification and link prediction tasks. The reported test performances of both random initialization and MLPInit are selected based on the vali- dation data. We present the performance improvement of MLPInit compared random initialization in Tables 4 and for node classification and link prediction, respectively.
Observation 4: MLPInit improves the prediction performance for both node classification and link prediction task in most cases. Table 4 shows our proposed method gains 7.97%, 7.00%, 6.61 % and 14.00% improvements for GraphSAGE, GraphSAINT,, ClusterGCN, and GCN on average cross

---

Published as a conference paper at ICLR 2023
OGB-arXiv
CGBBurroducts SCBBaaa
OGB- OGB-prodiee pro.poodcccss
Random Init
MLPInit
Random Init
MLPPlit
Figure 5: The loss landscape of GNN trained with random initialization (left) and MLPInit (right). The low-loss area of GNNs with MLPInit larger than that with random initialization.
Figure 6: The training trajectory of the GNN with random initialization (left) and MLPInit (right). The first-phase training of GNNs can be taken over by lightweight MLPs.
all the datasets for the node classification task. The results Table 5 and Table show our proposed method gains 1.05%. 1.10%, 17.81%, 20.97%, 14.88,110.465.. on average cross various metrics for the link prediction task.
5.3 MLPINIT ROBUST UNDER DIFFERENT HYPERPARAMETERS:
In practice, of the most time-consuming parts of training large-scale GNNs is hyperparameter tuning (You et al., 2020a). Here, we perform experiments to investigate the sensitivity of MLPInit to various hyperparameters, including the architecture hyperparameters and training hyperparameters.
Observation 5: MLPInit makes GNNs less sensi- tive to hyperparameters and improves the overall performance across various hyperparameters. In this experiment, trained PeerMLP and GNN with
antoy (INT
Training HPP
different "Training HP" (Learning rate, weight de- cay, and batch size) and "Architecture HP" (i.e., lay- ers, number of hidden neurons), and we presented the learning curves GNN with different hyperpa- rameters in Figure 4. One can see from the results that GNNs trained with MLPInit have a much smaller standard deviation than those trained with random initialization. Moreover, MLPInit consistently out- performs random initialization in task performance. This advantage allows our approach to saving time searching for architectural hypreaarrmettrs. In prac- tice, different datasets require different hyperparam- eters. Using our proposed method, we generally choose random hyperparameters and obtain reason- able and relatively stable performance owing the PeerMLP's lower sensitivity to hyperparameters.
Architecturo HP
SE.Phiit-d
Figure 4: Training curves of GNNs with var- ious hyperparameters. GNNs with MLPInit consistently outperform random initializa- tion and have smaller standard deviation.
5.4 WILL MLPINIT FACILITATE BETTER CONVERGENCE FOR GNNs?
In this experiment, we performed experiments to analyze the convergence of fine-tuned GNN model. other words, does this pre-training actually help find better local minima?
Observation 6: MLPinit finds larger low-loss area in loss landscape for GNNs. The geometry of loss surfaces reflects the properties of the learned model, which provides various insights to assess the generalization ability (Fort et al., 2019; Huang et al., 2019; Wen et al., 2018) and robustness (Fort et al., 2019; Liu et al., 2020; Engstrom et al., 2019) of a neural network. In this experiment, we plot the loss landscape of GNN (GraphSAGE) with random weight initialization and MLPInit using the visualization tool introduced training loss and with OGB-arX: and OGB- OCB-products datasets. The loss landscapes shown in
et al. (2018a). The loss landscapes are plotted based on the
8

---

Published as a conference paper at ICLR 2023
Figure 5. For the fair comparison, two random directions of the loss landscape are the same and the lowest values of losses in the landscape are the same within one dataset, 1.360 for OGB-arXiv and 0.340 for OGB -GBprruccts.. From the loss landscape, we can that the low-loss area of MLPInit is larger in terms of the same level of loss in each dataset's (row's) plots, indicating the loss land- scape of the model trained with MLPInit results in larger low-loss area than the model with random initialization. In summary, MLPInit helps larger low-loss areas in loss landscape for GNNs.
Observation 7: MLPinit speeds up the optimization process for GNNs. To further understand the training procedure of MLPInit, we visualize the training trajectories along with loss landscapes through tools (Li al., 20182) Sa) in Figure 6. In this experiment, we use GraphSAGE on OGB-arXiv and DGB-products datasets, we first train a PeerMLP and then use the weights of the converged PeerMLP to initialize the GraphSAGE. and fine-tune it. Then we plot the training trajectories of PeerMLP and GraphSAGE on the loss landscape of GraphSAGE. The red line indicates the training trajectories of the training of PeerMLP and the blue line indicates the fine-tuning of GNNs. We can see that the end point of the training of MLP (red line) is close to the minima area in the loss landscape. The training trajectory clearly shows the reason why MLPInit works, i.e., the first-phase training of GNNs can be taken over by lightweight MLPs.
6 RELATED WORK
In this section, present several lines of related work and discuss their difference and relation to our work. Also appearing at the same conference as this work, (Yang et al., 2023) concurrently found similar phonnomenia our findings and provided theoretical analysis of it.
Message passing-based GNNs. Graph neural networks typically follow the message passing mech- anism, which aggregates the information from node's 's neighbors and learns node representation for downstream tasks, Following the pioneering work GCN (Kipf & Welling, 6a), several other works (Veličković al., 201 80; Xu al., 2018: Balcilar al., 2021; Thorpe et al.. 2021; Brody et al., 2021; Tailor et al., 2021) seek to improve or better understand message passing-based GNNs.
Efficient GNNs. In order to scale GNNs to large-scale graphs, the efficiency of GNNs has attracted considerable recent attention. Subgraph sampling technique (Hamilton et al., 2017: Chiang et al., 2019; Zeng al., 2019) has been proposed for efficient mini-batch training for large-scale graphs. All of these methods follow the message passing mechanism for efficient GNN training. There is another line of work on scalable GNNs, which uses MLPs to simplify GNNs (Zhang al., 2022: Wu et al., 2019; Frasca et al., 2020; Sun et al., 2021; Huang et al., 2020; Hu et al., 2021). These methods aim to decouple the feature aggregation and transformation operations to avoid excessive, expensive aggregation operations. We compared our work with this line of work in Section 4.2. There also other acceleration methods, which leverage weight quantization and graph sparsification (Cai et al., 2020). However, these kinds of methods often sacrifice prediction accuracy and increase modeling complexity, while sometimes meriting significant additional engineering efforts.
GNN Pre-training. The recent GNN pretraining methods mainly adopt contrastive learning (Has- sani & Khasahmadi, 2020; Qiu et al., 2020; Zhu et al., 2020b; 2021; You et al., 2021; 2020c; Jin al., 2021a; la; Han et al., 2022: Zhu al., 2020a). GNN pretraining methods typically leverage graph augmentation to pretrain weights of GNNs or obtain the node representation for downstream tasks. For example, Zhu et al. (2020b) maximizes the agreement between two views of one graph. GNN pre-training methods not only use graph information for model training but also involve extra graph data augmentation operations, which require additional training time and engineering effort.
CONCLUSION
This work presents simple yet effective initialization method, MLPInit, to accelerate the training of GNNs, which adopts the weights from their converged PeerMLP initialize GNN and then fine-tune GNNs. With comprehensive experimental evidence, we demonstrate the superiority of our proposed method on training speedup (up to 33x speedup on OGB (GB-prroucct), downstream task performance improvements(up to 7.97% performance improvement for GraphSAGE), and robustness improve- ments (larger minimal area in loss landscape) on the resulting GNNs. Notably, our proposed method is easy to implement and employ in real applications to speed up the training of GNNs.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGEMENT
We thank the anonymous reviewers for their constructive suggestions and fruitful discussion, Xiao- tian would like thank Zirui Liu from Rice University for the discussion about the training time for MLP and GNN. He would also like to thank Kaixiong Zhou from Rice University and Keyu Duan from National University of Singapore for the discussion about the GNN training on large-scale graphs. Xiaotian would also like to thank Hanqing Zen from Meta Inc., for his valuable feedback and suggestions on the manuscript of this work. We thank Jingyuan Li from the Department of Electrical and Computer Engineering at the University of Washington for identifying typo in paper. Portions of this research were conducted with the advanced computing resources provided by Texas A&M High Performance Research Computing. This work is, part, supported by NSF IIS-1750074 and I1S-1900990. The views and conclusions contained this paper are those of the authors and should not be interpreted representing any funding agencies.
REPRODUCIBILITY STATEMENT
To ensure the reproducibility of our experiments and benefit the research community, we provide the source code at https//ggihuu..oo/saaap.pp wwwrccc/MMiiiiitfo-------- The other variables required to reproduce our experiments are described Appendix
and
REFERENCES
Siddhant Arora. A survey on graph neural networks for knowledge graph completion. arXiv preprint arXiv:2007. /2374, 2020.
Muhammet Balcilar, Renton Guillaume, Pierre Héroux, Benoit Gaüzère, Sébastien Adam, and Paul Honeine. Analyzing the expressive power of graph neural networks in spectral perspective. In Proceedings of the International Conference on Learning Representations (ICLR), 2021.
Aleksandar Bojchevski and Stephan Günnemann. Deep gaussian embedding of graphs: Unsuper- vised inductive learning via ranking. In ICLR, 2018.
Shaked Brody, Uri Alon, and Eran Yahav. How attentive are graph attention networks? In Interna- tional Conference on Learning Representations, 2021.
Chen Cai, Dingkang Wang, and Yusu Wang. Graph coarsening with neural networks. In Interna- fional Conference on Learning Representations, 2020.
Wei-Lin Chiang, Xuanqing Liu, Si Si, Yang Li, Samy Bengio, and Cho-Jui Hsich. Cluster-gen: An efficient algorithm for training deep and large graph convolutional networks. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining, pp- 257-266, 2019.
Gabriele Corso, Luca Cavalleri, Dominique Beaini, Pietro Liò, and Petar Veličković. Principal neighbourhood aggregation for graph nets. Advances in Neural Information Processing Systems, 011112260-1221.,1 2020.
Nicola De Cao and Thomas Kipf. Molgan: An implicit generative model for small molecular graphs. arXiv preprint /805. 1973, 2018
Keyu Duan, Zirui Liu, Peihao Wang, Wenqing Zheng, Kaixiong Zhou, Tianlong Chen, Xia Hu, and Zhangyang Wang. A comprehensive study on large-scale graph training: Benchmarking and rethinking. In Neural Information Processing Systems Datasets and Benchmarks Track, 2022.
Logan Engstrom, Brandon Tran, Dimitris Tsipras, Ludwig Schmidt, and Aleksander Madry. Ex- ploring the landscape of spatial robustness. In International conference on machine learning, pp. 1802-1811. PMLR. 2019.
Wengi Fan, Yao Ma. Qing Li, Yuan He, Eric Zhao, Jiliang Tang, and Dawei Yin. Graph neural networks for social recommendation. In The world wide web conference, pp. 417-426, 2019.
10

---

Published as a conference paper at ICLR 2023
Oleksandr Ferludin, Arno Eigenwillig, Martin Blais, Dustin Zelle, Jan Pfeifer, Alvaro Sanchez- Gonzalez, Sibon Li, Sami Abu-El-Haija, Peter Battaglia, Neslihan Bulut, et al. Tf-gnn: Graph neural networks in tensorflow. arXiv preprint arXiv:2207.03522 2022.
Matthias Fey and Jan E. Lenssen. Fast graph representation learning with PyTorch Geometric. In ICLR Workshop on Representation Learning on Graphs and Manifolds, 2019.
Matthias Fey, Jan E Lenssen, Frank Weichert, and Jure Leskovec. Gnnautoscale: Scalable and expressive graph neural networks via historical embeddings. In International Conference Machine Learning, pp. 3294-3304. PMLR, 2021.
Stanislav Fort, Huiyi Hu, and Balaji Lakshminarayanan. Deep ensembles: A loss landscape per- spective. arXiv preprint arXiv: 1912.02777, 2019.
Fabrizio Frasca, Emanuele Rossi, Davide Eynard, Ben Chamberlain, Michael Bronstein, and Fed- erico Monti. Sign: Scalable inception graph neural networks. arXiv preprint arXiv:2004.11199 2020.
Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl. Neural message passing for quantum chemistry. In International conference on machine learning, pp- 1263-1272. PMLR, 2017.
Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning large graphs. In NewrlPS, pp. 1024-1034, 2017.
Xiaotian Han, Zhimeng Jiang. Ninghao Liu, Qingquan Song. Jundong Li, and Xia Hu. Geometric graph representation learning via maximizing rate reduction. In Proceedings of the ACM Web Conference 2022, pp. 1226-1237, 2022.
Kaveh Hassani and Amir Hosein Khasahmadi. Contrastive multi-view representation learning on graphs. In International Conference Machine Learning, pp. 4116-4126. PMLR, 2020.
Weihua Hu, Bowen Liu, Joseph Gomes, Marinka Zitnik, Percy Liang, Vijay Pande, and Jure Leskovec. Strategies for pre-training graph neural networks. arXiv preprint arXiv: 12265, 2019.
Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning on graphs. Advances in neural information processing systems, 15:22118-22133 2020.
Yang Hu, Haoxuan You, Zhecan Wang, Zhicheng Wang, Erjin Zhou, and Yue Gao. Graph-mlp: node classification without message passing in graph. arXiv preprint arXiv:2 166004051, 2021.
Qian Huang, Horace He, Abhay Singh, Ser-Nam Lim, and Austin R Benson. Combining label propagation and simple models out-performs graph neural networks. arXiv preprint arXiv:2010. 13993, 2020.
W Ronny Huang, Zeyad Emam, Micah Goldblum, Liam Fowl, Justin K Terry, Furong Huang, and Tom Goldstein. Understanding generalization through visualizations. arXiv preprint 190.029906.03291, 2019.
Weiwei Jiang and Jiayun Luo. Graph neural network for traffic forecasting: A survey. Expert Systems with Applications, pp. 117921, 2022.
Wei Jin, Tyler Derr, Haochen Liu, Yiqi Wang, Suhang Wang, Zitao Liu, and Jiliang Tang. Self- supervised learning on graphs: Deep insights and new direction. In WWW Workshop. 2021a.
Wei Jin, Lingxiao Zhao, Shichang Zhang, Yozen Liu, Jiliang Tang, and Neil Shah. Graph con- densation for graph neural networks. In International Conference on Learning Representations, 2021b.
Diederík Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional net- works. arXiv preprint arXiv: 1609.02977 2016a.

---

Published as a conference paper at ICLR 2023
Thomas N Kipf and Max Welling. arXiv:1611.0738.. 2016b.
Variational
graph auto-encoders.
arXiv
preprint
Guohao Li, Chenxin Xiong, Ali Thabet, and Bernard Ghanem. Deepergen: All you need to train deeper gens. arXiv preprint arXiv:2006.07733 2020.
Guohao Li, Matthias Müller, Bernard Ghanem, and Vladlen Koltun. Training graph neural networks with 1000 layers. In International conference on machine learning, pp. 6437-6449. PMLR, 2021.
Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss land- scape of neural nets. Advances neural information processing systems, 31, 2018a.
Yujia Li, Oriol Vinyals, Chris Dyer. Razvan Pascanu, and Peter Battaglia. Learning deep generative 2018b.
models of graphs. arXiv preprint
Chen Liu, Mathieu Salzmann, Tao Lin, Ryota Tomioka, and Sabine Stisstrunk. On the loss landscape of adversarial training: Identifying challenges and how to overcome them. Advances in Neural Information Processing Systems, 33:21476-21487, 2020.
Gang Liu, Tong Zhao, Jiaxin Xu, Tengfei Luo, and Meng Jiang. Graph rationalization with environment- based augmentations. In Proceedings of the 28th ACM SIGKDD international con- ference on knowledge discovery & data mining. 2022.
Yao Ma, Xiaorui Liu, Tong Zhao, Yozen Liu, Jiliang Tang, and Neil Shah. A unified view on graph neural networks as graph signal denoising. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management, pp. 1202-1211, 2021.
Julian McAuley, Christopher Targett, Qinfeng Shi, and Anton Van Den Hengel. Image-based rec- ommendations styles and substitutes. In Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 43-52, 2015.
Namyong Park, Andrey Kan, Xin Luna Dong. Tong Zhao, and Christos Faloutsos. Estimating node importance in knowledge graphs using graph neural networks. Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining, pp. 596-606, 2019.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. In NeurIPS, 2019.
Jiezhong Qiu, Qibin Chen, Yuxiao Dong, Jing Zhang, Hongxia Yang, Ming Ding, Kuansan Wang. and Jie Tang. Gee: Graph contrastive coding for graph neural network pre-training. In Pro- ceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 1150-1160, 2020.
Aravind Sankar, Yozen Liu, Jun Yu, and Neil Shah. Graph neural networks for friend ranking in large-scale social platforms. In Proceedings of the Web Conference 2021, pp. 2535-2546, 2021.
Oleksandr Shchur, Maximilian Mumme, Aleksandar Bojchevski, and Stephan Günnemann. Pitfalls of graph neural network evaluation. arXiv preprint arXiv: 1.000688, 2018.
Chuxiong Sun, Hongming Gu, and Jie Hu. Scalable and adaptive graph neural networks with self- label-enhanced training. arXiv preprint arXiv:2104.09376, 2021.
Shyam A Tailor, Felix Opolka, Pietro Lio, and Nicholas Donald Lane. Do we need anisotropic graph neural networks? In International Conference on Learning Representations, 2021.
Xianfeng Tang, Yozen Liu. Neil Shah, Xiaolin Shi, Prasenjit Mitra, and Suhang Wang. Knowing your fate: Friendship, action and temporal explanations for user engagement prediction social apps. Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery 2020.
& data mining, pp.
Xianfeng Tang, Yozen Liu, Xinran He, Suhang Wang, and Neil Shah. Friend story ranking with edge-contextual local graph convolutions. In Proceedings of the Fifteenth ACM International Conference Web Search and Data Mining, pp. 1007-1015, 2022.
12

---

Published as a conference paper at ICLR 2023
Matthew Thorpe, Tan Minh Nguyen, Hedi Xia. Thomas Strohmer, Andrea Bertozzi, Stanley Osher, and Bao Wang. Grand++: Graph neural diffusion with source term. In International Conference Learning Representations, 2021.
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. Graph attention networks. In International Conference on Learning Representations, 2018a.
Petar Veličković, William Fedus, William L Hamilton, Pietro Liò, Yoshua Bengio, and R Devon Hjelm. Deep graph infomax. In International Conference Learning Representations, 2018b.
Cheng Wan, Youjie Li, Ang Li, Nam Sung Kim, and Yingyan Lin. Bns-gen: Efficient full-graph training of graph convolutional networks with partition-paraln parallelism and random boundary node sampling. Proceedings of Machine Learning and Systems, 4:673-693 2022.
Shen Wang, Xiaokai Wei, Cicero Nogueira Nogueira dos Santos, Zhiguo Wang, Ramesh Nallapati, Andrew Arnold, Bing Xiang, Philip Yu, and Isabel F Cruz. Mixed-curvature multi-relational graph neural network for knowledge graph completion. In Proceedings of the Web Conference 202/, pp. 1761-1771, 2021.
Wei Wen, Yandan Wang, Feng Yan, Cong Xu, Chumpeng Wu, Yiran Chen, and Hai Li. Smoothout: Smoothing sharp minima to improve generalization in deep learning. arXiv preprint arXiv: 2018.
Felix Wu, Amauri Souza, Tianyi Zhang, Christopher Fifty, Tao Yu, and Kilian Weinberger. Simpli- fying graph convolutional networks. In ICML, pp. 6861-6871. PMLR, 2019.
Zonghan Wu, Shirui Pan, Fengwen Chen, Guodong Long, Chengqi Zhang, and S Yu Philip. A comprehensive survey on graph neural networks. IEEE transactions on neural networks and learning systems, 2020.
Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful graph neural networks? In International Conference on Learning Representations, 2018.
Chenxiao Yang, Qitian Wu, Jiahua Wang, and Junchi Yan. Graph neural networks are inherently good generalizers: Insights by bridging gnns and mlps. International Conference on Learning Representations, 2023.
Zhilin Yang, William W Cohen, and Ruslan Salakhutdinov. Revisiting semi-supervised learning with graph embeddings. In Proceedings of the 33rd ICML, 2016.
Rex Ying. Ruining He, Kaifeng Chen, Pong Eksombatchai, William L Hamilton, and Jure Leskovec. Graph convolutional neural networks for web-scale recommender systems. In Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining, pp. 974- 983, 2018
Jiaxuan You, Rex Ying, Xiang Ren, William Hamilton, and Jure Leskovec. Graphrnn: Generat- ing realistic graphs with deep auto-regressive models. In International conference on machine learning, pp. 5708-5717. PMLR, 2018.
Jixxuan You, Zhitao Ying, and Jure Leskovec. Design space for graph neural networks, Advances in Neural Information Processing Systems, 33: 13:11009-17021, 2020a.
Yuning You, Tianlong Chen, Yongduo Sui, Ting Chen, Zhangyang Wang, and Yang Shen. Graph contrastive learning with augmentations. Advances in Neural Information Processing Systems, 33:5812-5822 2020b.
Yuning You. Tianlong Chen, Zhangyang Wang, and Yang Shen. When does self-supervision help graph convolutional networks? In ICML, pp. 10871-10880, PMLR, 2020c.
Yuning You, Tianlong Chen, Zhangyang Wang, and Yang Shen. L2-gen: Layer-wise and learned efficient training of graph convolutional networks. Proceedings of the IÉEE/CVF Conference Computer Vision and Pattern Recognition, pp. 2127-2135, 2020d.
13

---

Published as a conference paper at ICLR 2023
Yuning You, Tianlong Chen, Yang Shen, and Zhangyang Wang. Graph contrastive learning auto- mated. In International Conference on Machine Learning, pp. 12121-12132. PMLR, 2021.
Haiyang Yu, Limei Wang, Bokun Wang, Meng Liu, Tianbao Yang, and Shuiwang Ji. Graphfim: Im- proving large-scale gnn training via feature momentum. In International Conference on Machine Learning, pp. 25684-25701. PMLR, 2022.
Hanqing Zeng, Hongkuan Zhou, Ajitesh Srivastava, Rajgopal Kannan, and Viktor Prasanna. Graph- saint: Graph sampling based inductive learning method. International Conference on Learning Representations, 2019.
Dalong Zhang, Xin Huang, Ziqi Liu, Zhiyang Hu, Xianzheng Song, Zhibang Ge, Zhiqiang Zhang, Lin Wang, Jun Zhou, Yang Shuang, et al. Agl: a scalable system for industrial-purpose graph machine learning. arXiv preprint arXiv:2003 20030440,,,,444, 2020.
Muhan Zhang and Yixin Chen, Link prediction based on graph neural networks. Advances in neural information processing systems, 31, 2018.
Muhan Zhang, Pan Li, Yinglong Xia, Kai Wang, and Long Jin. Labeling trick: A theory of using graph neural networks for multi-node representation learning. Advances in Neural Information Processing Systems, 34:9061-9073, 2021a.
Shichang Zhang, Yozen Liu, Yizhou Sun, and Neil Shah. Graph-less neural networks: Teaching old mlps new tricks via distillation. In International Conference on Learning Representations, 2021b.
Si Zhang, Hanghang Tong, Jiejun Xu, and Ross Maciejewski. Graph convolutional networks: Algo- rithms, applications and open challenges. In International Conference on Computational Social Networks, pp. 79-91. Springer, 2018.
Wentao Zhang, Ziqi Yin, Zeang Sheng, Yang Li, Wen Ouyang, Xiaosen Li, Yangyu Tao, Zhi Yang, and Bin Cui. Graph attention multi-layer perceptron. arXiv preprint arXiv:2206.0m 04355, 2022.
Tong Zhao, Bo Ni, Wenhao Yu, Zhichun Guo, Neil Shah, and Meng Jiang. Action sequence augmen- tation for early graph-based anomaly detection. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management, pp. 2668-2678, 2021.
Tong Zhao, Wei Jin, Yozen Liu, Yingheng Wang, Gang Liu, Stephan Günneman, Neil Shah, and Meng Jiang. Graph data augmentation for graph machine learning: A survey. arXiv preprint 20Xiv:2202.08871, 2022a.
Tong Zhao, Gang Liu, Daheng Wang, Wenhao Yu, and Meng Jiang. Learning from counterfactual links for link prediction. In International Conference on Machine Learning, pp. 26911 PMLR, 2022b.
Jie Zhou, Ganqu Cui, Shengding Hu, Zhengyan Zhang, Cheng Yang. Zhiyuan Liu, Lifeng Wang. Changeheng Li, and Maosong Sun. Graph neural networks: A review of methods and applica- tions. Al Open, 1:57-81, 2020.
Qikui Zhu, Bo Du, and Pingkun Yan. Self-supervised training of graph convolutional networks. NXX:220060003880 2020a.
Yanqiao Zhu, Yichen Xu, Feng Yu, Qiang Liu, Shu Wu, and Liang Wang. Deep graph contrastive representation learning. arXiv preprint arXiv:2006.041 04131, 2020b.
Yanqiao Zhu, Yichen Xu, Feng Yu, Qiang Liu, Shu Wu, and Liang Wang. Graph contrastive learning with adattive augmentation. In Proceedings of the Web Conference 2021, pp. 2069-2080, 2021.
14

---

Published as a conference paper at ICLR 2023
CONTENTS OF APPENDIX
Additional Experiments
16
A.I
Additional experimental link prediction
16
A.2 Additional hyperparameter sensitivity
16
A.3
Additional training curves
16
A.4
Additional loss/accuracy curves of PeerMLP and GNN
16
A.5
Training eurves of link prediction task
17
More Experiments
21
B.1
The Performance of GCN with the weight of PeerMLP
B.2
Weight difference of GNNs with random initialization and MLPInit
B.3
Running time comparison of MLP and GNN
22
B.4
Comparison to GNN pre-training methods
22
B.5
Experiments more complicated aggregators..
23
B.6
Results on graph sampling methods and GNN architectures
24
B.7 Experiments datasets where node features less important
24
B.8 Deriving the PeerMLP
25
B.8.1
Two Methods to Derive the PeerMLP
25
B.8.2
Discussion about the efficiency of PeerMLP deriving
Datasets and Baselines
28
C.I Datasets for node classification
28
C.2 Baselines for node classification
28
C33 Datasets for link prediciton
29
C.4 Baselines for link prediciton
29
D Implementation Details
30
D.I Running environment
30
D.2 Experiment setting for Figure
D.3 Experiment setting for Figures 3, 8 and 9 and Tables 3 and ................................
30
D.4 Experiment setting for Table 2
30
D.5 Experiment setting for Tables 5 and 6
30
D.6 Experiment setting for Figure
30
15

---

Published as a conference paper at ICLR 2023
A ADDITIONAL EXPERIMENTS
In this appendix, we present additional experiments to show the superiority of our proposed method, MLPInit, including additional results for link prediction and training curves. We also present more analysis on the weights distribution changes of MLPInit.
A.1 ADDITIONAL EXPERIMENTAL LINK PREDICTION
Here, present the additional experiment results the link prediction task in Table 6, which is similar to our results in Table 5, but on more datasets. In general, we observe that GNN with MLPInit outperforms that with random initialization MLPInit does not gain better performance on A-products dataset. We conjecture that the reason for this is that node features may contain less task-relevant information on A A-products.
the link prediction tasks on these additional datasets.
Table 6: The performance of link prediction task. The results are based ten runs.
Methods
AUC
AP
Hits @ 10
Hits @ 20
Hits 550
Hits@ 100
MLP random
91.87+125
92.16 02.16#0.99
46.98
58.05
76.03
86.28
Cora
GNNrandom
91. 91.802139
92.6821.22
51.48
63.04_6.12
78. 12
86.07+2.7
GNN mipinit
12.9
93.36
53.72.743
68.66 4.22
80.3012.5
89.37-221
Improvement
t 1.23%
1 0.74%
1 4.35%
t 8.8.1% %
1 2.79%
1 3.84%
MLP random
90.10
90.65
46.6445.1
57.21
71.45+315
81.93+204
CitaSear
GNN
89.8611111
90.88 0..9
49.98
58.02
71.65
82. 11284
GNN
90.51
90.96=0.00
50.02
58.77=4.8
71.78
82.66+258
Improvement
1 0.73%
1 0.08%
1 0.09%
+ 1.29%
1 0.18%
1 0.67%
MLP random
96.2911012
95.79
13.36+149
19.67
33.46+2.1
46.82+191
GNNeandom
96.11 +:00s
95.75
14.27+2m7
22.57
35.40+201
48.21+200
CS
GNNmipinit
96.7210.10
96.49
16.964337
25.44_3.00
40.69+2.9
53.78+200
Improvement
1 0.63%
0.77%
1 18.81%
12.70%
1 14.96%
1 11.55%
A-Compators
MLP random
81.85
82.41 20.69
2.10
4.13-0.85
7.83
12.18+101
GaNrandom
91.78
91.94
7.60+147
11.10-1.74
18.644194
25,42+215
GNNmlpinit
90.76+1.11
91.06
6.76+332
11.11=1.22
17.40
24.59
Improvement
4 1.11%
0.96%
4 11.04%
1 0.16%
4. 6.65%
4 3.26%
MLP random
95.72+t0.8
95.55
19.38
27.83
42.98
57.20+127
CoraPull
GNNeandom
95.87
95.77
21.331477
30.57
45.08
59.58+253
GNNmipinit
96.71
96.73
25.78
36.68
53,8112M
66.731196
Improvement
0.87%
f 1.01%
1 20.87%
19.98%
t 19.37%
12.01% %
A.2 ADDITIONAL HYPERPARAMETER SENSITIVITY
In this appendix, we present the additional results to explore the sensitivity the various hyperpa- rameters. The results are the full version of Figure 7.
A.3 ADDITIONAL TRAINING CURVES
In this appendix, we present the additional training curves other datasets in Figure 8 and Figure 9, which are additional experimental results of Figure 3. The results comprehensively show the training curves on various datasets. As we can see from Figure 8 and Figure 9 that MLPInit consistently outperforms the random initialization and is able to accelerate the training of GNNs.
A.4 ADDITIONAL LOSS/ CURVES OF PEERMLP AND GNN
In this appendix, we plotted the loss and accuracy curves of PeerMLP and GNN on training/validaa- tion/test set and presented the results in Figure 10, which the additional experimental results to
16

---

Published as a conference paper at ICLR 2023
023-artiv (GUN)
00B (SmnSSGGS
CoB-arXLy (CraplAANTT)
0GB-a:Ziv (Caale
Training HP
5
0.4
57
003-artiv (OUN)
003-arTly (Smn
0:25 22.30 (ChEAISAIN))
008-a:219 (CasattiN))
Arhitetture HPP
0.24
Bandom-tort
MiLP tost
Figure 7: The training curves of the GNNs with different hyperparameters on DGB-arXiv dataset. The training curves of GNN with MLPInit generally obtain lower loss and higher accuracy than those with the random initialization and converge faster. The training curves depicted based on ten runs.
Figure 2. The results surprisingly show that GNN using the weight from trained PeerMLP has worse cross-entropy loss but better prediction accuracy than PeerMLP. The reason would be that GNN can smooth the prediction logit, making loss worse but accuracy better.
A.5 TRAINING CURVES OF LINK PREDICTION TASK
To further investigate the training process of the link prediction task, present the training curves for the link prediction task for each metric we used. The metrics we used are AUC, AP, Hits @ K, which are commonly used evaluate the performance of link prediction (Zhang & Chen, 2018; Zhang et al., 2021a; Zhao et al., 2022b). AUC and AP measure binary classification, reflecting the value loss for link prediction, and Hits the count of how many positive samples are ranked in the top-K K positions against a bunch of negative samples.
The results show that the AUC and are easy to train, while the Hits@K harder to train. The GNN with random initialization needs much more time than MLPInit to obtain good Hits@K. K. Since Hits @ K more realistic metric, the results demonstrate the superiority our method in the link prediction task. These experimental results also show that node features are important for link prediction task since we can obtain a good performance only with MLP, therefore, MLPinit is beneficial for link prediction since the MLPInit used node feature information only to train PeerMLP.
17

---

Published as a conference paper at ICLR 2023
Tolp (GUO))
Yeip (GrapSAAGE)
Telp (ChopSAANT)
Yelp (ClataCCCN)
3.559
0.2
11.24
Choceda SANE Lo0
230
10.20
1620
9.36
0.235
0.15
0.23
0:30
Bpodi
Epods
Tp (CRTT))
Yelp (GGG))
Yelp (SiqpISAANTT)
Yw1p (Clns
0.20
Randkm-vi
+0.20
Rardon text
13
0.18
Epeda
Flicke (GUNN
Flickr (Cop6SAGE)
Flicke (GonASAANNE)
(Chater(CCN)
1.50
mree Lum
1.30
1,30
1,0
Egodi
(GCN)
(GDASACE)
(GmpLSAINT
(ClsterGCN)
0.5
2.30
2.00
Randlmmeaa
Bandon-tees
MI.liitei
MI. Phrit-ott
0:11
22
22
27
Spuds
Spadi
Egudi
Seddit (GCN)
Reddit: (GmpLSAGE)
Bedd1t (CapASAINT)
Baddit ( (CheerCCC))
0.27
Chios Medernnn Len0
Lent
0:22
0.10
0.55
32
22
Roddit
(GmpASAINT)
3.0
2.00
0.06
Bonkkm-el Rondim--ted
0.80
Mi.Phittol
3.70
22
10
72
89
12
Figure 8: The training curves of GNN with random initialization and MLPInit on Yelp. Flickr, Reddit datasets.
18

---

Published as a conference paper at ICLR 2023
(GCNN)
(GraphSAAG))
(GmpESAINT)
(CheterGCN)
3.5
C Papeottriig strrage lowe
RESTE: Lan
Bed0112 (GCN)
Beddit2 (CrphSSAGiE)
Beddi:2 (GINI
Beddie2 ChusterCCN
Handonesed
Baunoon-les
0.5
MLPlui
0.1
MLPluiteees
250
Egesd
Epedd
028-prodact (OON!
D00 32-gcoduct (GmpSAAG)
008 pcoduct (GaplSANN
000 Se-prrduct (ChsteGCN)
2.5
Crowettnp
0.50
Flech
Rpshh
USB-grodoct
0oP-product (GriphSAGI))
000-groduce (GrapSSAINT)
0GR-product
0.00
Randomeol
Ramdamstost
M.Plnit-al
0,70
Mr. Pinittttt
3.70
10
1
10
12
22
27
Rpadi
JesocnProdtcta
(CuemnGNN)
0.000
AcazosProduccee (OCAN
A AessonProdocco (GrpusAIITT)
11.5000
01..30
Randon-vrl
0.53
Randon-tel
MLLPha-vil
Mi.Phai-ito
12
Figure 9: The training curves of GNN with random initialization and with MLPInit on Reddit2. -C8-products, A-products datasets.
19

---

Published as a conference paper at ICLR 2023
0GB-arXiv (Triin)
DGB D08-arXiv (Valdationn)
0GB-arXiv (Test)
Chocoettogg eeteegg 100
Croeveeteog Jane
5.0
3.11
2.0
2.5m
2.0
2.0
1.34
1.54
8
Epusds
Bjach
0.5
0.4
Aenmmee:
0.3
Smitl X: Wmil))
Sonel X. A Wmlp)
0.0
0.00
Rooth
Epoch
Epoch
Figure 10: The relation of GNN and MLP during the training of PeerMLP. We report the cross- entropy loss and accuracy on the training/validationnesst sets.
Donaa (AP)
(AUC)
(Hitvit100)
0.2
Ori sthugee
6:00
07
Oltedieer (ATO))
CitoBeur (AP)
C1Ceeeerr (FlreO 1877
2.70
180
Pobbfidd (Rite 201
Publiee (iit))
Santigg Lecs
50
Ramdknn-d
Eani Fand--nnddon
2.0
ML.Laitou
Pave
Figure 11: The training curves for link prediction task on Cora, CiteSeer, PubMed datasets. GNN with MLPini generally obtains higher metrics for link prediction task than those with random ini- tialization and converge faster.
20

---

Published as a conference paper at ICLR 2023
B MORE EXPERIMENTS
In this appendix, we conducted additional experiments to further analyze our proposed method MLPInit. We conducted experiments to show that training MLP is much cheaper than training GNN. The results show that the running time MLP can be negligible compared to that of GNNs. We compare the efficiency of our method to GNN pre-training methods. The comparison GNN pre- training methods demonstrates the superiority of MLPInit in effectiveness and efficiency. We conduct experiments to investigate the effectiveness of MLPInit on GNN with more complicated aggregators. We provided the new form of our results in Table 4 to show the performance improvements for graph sampling methods and GNN architectures separately.
B. THE PERFORMANCE OF GCN WITH THE WEIGHT OF PEERMLP
In this experiment, aimed to verify Observation 2: Converged weights from PeerMLP provide good GNN initialization from Section 4 by evaluating the performance of GCN with the weight of PeerMLP on the Cora, CiteSeer, and PubMed datasets. We trained the PeerMLP for GCN, and then calculated the accuracy of the GCN using the well-trained PeerMLP weights (without fine-tuning). We used the public split for these three datasets in semi-supervised setting. The evaluated models are described in detail below and the results are presented in Table 7.
PeerMLP: the well-trained PeerMLP for GCN on different datasets.
GCN w/ Wipeerlp: the GCN with the weight of well-trained PeerMLP, without fine-tuning.
GCN: the well-trained graph convolutional neural network.
Table The performance GNNs and its PeerMLP with the weights of converged PeerMLP on test data. The accuracy are in percentage (%).
PeerMLP
GCN w/ dpeermbo
Improv.
GCN
Cora
58.50
77.60
t 32.64%
82.60
CiteSeer
60.50
69.70
1 15.20%
71.60
PubMed
73.60
78.10
t 6.11%
79.80
The results show that the GNN w/ Wpeermip significantly outperforms the PeerMLP, even though the weight wpeermp is trained on PeerMLP. The performance improvements notable, with increases of 32.64%. 15. 15.20%. and 6.11% on Cora, CiteSeer, and PubKed datasets, respectively. The results would be additional evidence for Observation 2: Converged weights from PeerMLP provide a good GNN initialization.
Moreover, this intriguing phenomenon implies relationship between MLPs and GNNs that could potentially shed light on the generalization capabilities of GNNs. We believe that our findings will be of significant interest to researchers and practitioners the field of graph neural networks, and we hope that our work will inspire follow-up research to further explore the relationship between ML.ss and GNNs.
B.2 WEIGHT DIFFERENCE OF GNNs WITH RANDOM INITIALIZATION AND MLPINIT
Prior work (Li al., 201 8a) suggests that "small weights still appear more sensitive to perturbations, and produce sharper looking minimizers." To this end, we explore the distribution of weights of GNNs with both random initialization and MLPinit, and present the results in Figure 12. We can observe that with the same number of training epochs, the weights of GraphSAGE with MLPInit produce more high-man magnitude (both positive and negative) weights, indicating the MLPInit can help the optimization of GNN. This difference stems from straightforward reason: MLPInit provides a good initialization for GNNs since the weights are trained by the PeerMLP before (also aligning with our observations in Section 5.4).
21

---

Published as a conference paper at ICLR 2023
X10*
Random MLPlnit
Number
Number
3
0
-0.2
0.0 Weight
0.2
0.2
0.0 Weight
0.2
(a) DCB-arXiv
(b) DGB DGB-products
Weight histograms of GraphSAGE model weights with random initialization (blue) and MLPInit (red) on OGB-arXiv (left) and OCB-producto (right). The results are plotted based on 20 epochs after training GraphSAGE with weight decay. Note that MLPInit produces more weights of higher magnitude.
Figure 12:
B.3 RUNNING TIME COMPARISON OF MLP AND GNN
We conducted experiments to compare the rumning time of MLP and GNN (GraphSAGE in this experiment) and presented the running time needed by training MLP and GraphSAGE for one epoch in Table 8. In this experiment, MLP is trained in full-batch way, and training node features are stored in GPU memory. We adopted the official example code of GraphSAGE The running time of MLP only needs 1/147 and 1 1/2303 of that of GraphSAGE datasets. The results show that training MLP is much cheaper than training GNN. In practice, MLP usually only needs be trained less than 50 epochs converge. Thus, the training time of MLP in MLPInit is negligible compared to the training time of GNNs.
AGB-arXivg and OGB OCB-products
Table 8: The comparison of the rumning time of MLP and GraphSAGE. We report the running time of one epoch for MIP, GraphSAGE, and their ratio. The results show that training MIP is much cheaper than training GrapSAGE, especialy for the large graph.
Dataset
MLP GraphSAGE
MLP/GraphSAGE Ratio
OGB-arXiv
0.035
5.17020311 175.758 +93500
1/147
OGB-products
0.076+00000
1/2303
B.4 COMPARISON TO GNN PRE-TRAINING METHODS
In this appendix, we compare the efficiency of our method to GNN pre-training methods. In this experiment, we adopt DGI (Veličković et al., 201 8b) as the pre-training method to pretrain the weight of GNN. DGI maximizes the mutual information between patch representations and corresponding high-level summaries of graphs. Since the output of DGI is a hidden representation, DGI to pretrain weights of the GNN except for the last layer (classification head). We report the final prediction performance of GNN with MLPInit and DGI in Table 9 and reported the running time of MIPInit and DGI in Table 10. The experimental results show that MLPInit obtains rank 1. 1.28 and 1.42 GraphSAGE and GCN, demonstrating MLPInit outperforms DGI slightly. This might be because the additional classification head for DGI is not pretrained. It is worth noting that DGI much more time-consuming than MLPInit, Table 10 show that MLPInit only needs 1/6.59 and 1 1/1017.41 running time of DGI. The comparison to GNN pre-training methods demonstrates the superiority of MLPInit effectiveness and efficiency.
leverage
hhttps://githae http://gittuub.co/pye..ee
O. www.///lle//ogg.pppaeeoeeeeeeeee
sage.py
22

---

Published as a conference paper at ICLR 2023
Table The comparison of the performance of MLPInit and DGI. The best performance is in bold- face. The Rank is the average performance rank over 7 datasets.
Methods
Flickr
Yelp
Reddit
Reddit2
A-groducte
0GB OGB-arXáv 0GB 00B-products
Avg. Rank
GCN SAGE
Random
53.72#+016
63.03
96,50
51.76+2253
77.58
72.00
80.05
3.00
DGI
53.97=0.15 62.53
96.57=0.0s
54.82++1.22
77.11 +0.08
71.86+0.3
80.24
1.71
MLPInit
53,82
63,93
96.66+0..4
89.60
77.74
72.25
80.04
1.28
Random 50.90=a.12
40.08=00.15
92.78=0.11
27.87=3445
36.35
70.25
77.08=00:6
3.00
DGI
51.23
38,24-004
94.14
6..98
35. a5.5
69 49.400s.3
77.15
1.57
MLPInit 51.16=0020 40.83 91.40=+0.0
80.37
39.70=a.11
70.35
76.85=0.34
1.42
The comparison of the running time of MLPInit and DGI. We report the running time of one epoch for MLPInit and DGI in the pretraining stage and their ratio. The results show that training MLP is much cheaper than DGI, especially for the large graph.
Table 10:
Dataset
MI MlPPnti(ours)
DGI
MIPInit/DGI Ratio
OGB-arXiv
0.035
4.794
1/137
OGB-products
0.076
1892.386
1 1/24798
B.5 EXPERIMENTS ON MORE COMPLICATED AGGREGATORS
Information aggregators play a vital role in graph neural networks, and recent work proposed com- plicated aggregators to improve the performance of graph neural networks. In this appendix, conducted experiments to investigate the effectiveness of MLPInit on GNN with more complicated aggregators. We explore the acceleration effect and prediction accuracy improvement of MLPInit on GNN with more complicated aggregators. The adopted aggregators include Mean, Sum, Max, Median, and Softmax. Their details are presented follows:
Mean is commonly used aggregation operator that averages features across neighbors.
Max, Median (Corso al., 2020) aggregation operators that take the feature-wist maximum/median across neighbors. The mathematical expressions of Max, Median are max X,. median,, ex
Softmax (Li et al., 2020) is learnable aggregation operator, which normalizes the features neighbors based on a learnable temperature term, as softmax(X11) exp(txx,) Σx,ex Ex,ex exp(1xx, where / controls the softness of the softmax when aggregating neighbors' features.
We reported the performance improvement, training speedup, and training curves in Table 11, Ta- ble 12 and Figure 13. Generally, MLPInit is effective for other aggregators. MLPInit speed up the training of GNNs by 2.06x over four aggregators. Moreover, MLPInit improves the prediction per- formance by 0.75% over four aggregators. The training curves in Figure 13 show that GNN with MLPInit generally obtain lower loss and higher accuracy than those with random initialization and converge faster.
Table 12: Speed improvement when MLPInit achieves comparable performance with randomly initialized GNN using various information aggregators. The number reported is the number of training epochs needed.
Methods
Mean
Max
Median
Softmax
Avg.
Random
46.7
37.1
40.9
42.0
41.6
MLPInit
22.7
22.4
27.2
8.8
20.2
028
Improv.
2.06x
1.66x
1.50x
4.77x
2.06x
23

---

Published as a conference paper at ICLR 2023
OGB-arXiv (Max)
aiVvarXiv (Meadian)
0GB-arXiy (Softmax)
2.25
Croeetizor Loe
Choce entropy Loos
2,70
1.2
0.50
1.24
1.25
1.0
1,0
1.001
P Upoch
(Max)
aavvvi (Mendiau)
(Sofmax))
0.70
0.771
0.65
0.700
Acnurayy
Apcurccy
Random-val
0.000
Randomstest
MLPPhit val
0.60
11,20
MI.Phit test
11.000
80 Epoch
Eporh
Eguch
Figure 13: The training curves for GraphSAGE using different aggregators with random initializa- tion and MLPInit.
Table 11: Performance improvement when GNN (GraphSAGE) uses different aggregators with ran- dom initialization. Mean, and standard deviation are calculated based on ten runs.
Methods
Mean
Max
Median
Softmax
Avg.
Random
72.00
68. 1::00
69.97
71.05 71.94
70.33
MLPInit
72.25
69.30
69.95
70.86
Improv.
1 0.36%
1 1.44%
0.02%
1 1.25%
1 0.75%
B.6 RESULTS ON GRAPH SAMPLING METHODS AND GNN ARCHITECTURES
In this appendix, we provided the new form our results Table 4 to show the performance improvements for graph sampling methods and GNN architectures separately in Tables 13 and 14.
From the new form of our results, we observed that 1) MLPinit improves the performance of dif- ferent graph sampling methods as it improves the performance of GraphSAGE, GraphSAINT, and ClusterGCN by 7.96%, 7.00% and 6. 6.62%. 2) MLPInit improves the performance of different graph neural network layers, as it improves the performance of SAGEConv, GCNConv by 6.62%, 14.00%.
B.7 EXPERIMENTS ON DATASETS WHERE NODE FEATURES ARE LESS IMPORTANT
Our proposed method does depend on a tendency towards node feature-label correlation. Thus, would likely suffer if features provided less conducted experiments on synthetic graphs. The synthetic graphs have differing degrees of correla- tion between node features and labels. We generated the synthetic graph node features X agynthetic by mixing the original features and random features for each node in the
information about labels. In this appendix,
graph as follows:
X Ssynthetic AXoriginal + (1 X)X Xrandom
where Xoriginal and Xrandom are the original features of OGB-arXiv and random features, and mediates the two. shows different levels of association between node features and labels. When the synthetic node features will be the original features of DGB-an DGB-arXiv. When À 1, the synthetic node features will be completely random features, which are totally uncorrelated to the node labels. We change the value of À to explore the behavior of MLPInit. Note that initially
24

---

Published as a conference paper at ICLR 2023
Table 13: Performance improvements for various graph sampling methods. The improvement the overall accuracy performance improvement across all the datasets.
Sampling
Methods Flickr Yelp Reddit Reddit2 A d-products DGB-arXiv DGB-products Improv.
Random 53.72
63.03
96.50
51.76
77.58
72.00
80.05
GraphSAGE
7.96%
MLPInit 53.82
63.93
96.66
89.60
77.74
72.25
80.04
Random
51.37
29.
95.58
36.45
59.31
67.95
73.80
GraphSAINT
7.00%
MLPInit
51.35
43.10
95.64
41.71
68.24
68.80
74.02
Random 49.95
56.39
95.70
53.79
52.74
68.00
78.71
Cluster-GCN
1 6.62%
MLPInit 49.96
58.05
96.02
77.77
55.61
69.53
78.48
Table 14: Performance improvements for various GNN architectures. The improvement is the over-
all accuracy performance improvement across all the datasets.
GNN layers Methods Flickr Yelp
Reddit Reddit2 A A-products OCB-arXiv OGB NCB-products Improv.
Random 49.95
56.39
95.70
53.79
52.74
68.00
78.71
SAGEConV MLPInit 49.96
6.62%
58.05
96.02
77.77
55.61
69.53
78.48
Random
50.90
40.08
92.78
27.87
36.35
70.25
77.08
GCNConV
t 14.00%
MLPinit 51.16
10.83
91.40
80.37
39.70
70.35
76.85
conducted the experiments on À = 0.1. and we observed that performance on 0.1 much lower than other values, thus we conducted more on = [0.05, 0. 0.15) around 0.1.
The results shows that
If node features are uncorrelated to the node labels (A < 0.2), GNN with the weights of PeerMLP will not outperform the PeerMLP.
If the node features are correlated to the node labels (A > 0.2), GNN with the weights of PeerMLP will consistently outperform the PeerMLP.
Overall, MLPInit obtains a better (0.74% average improvement) final accuracy than Random Init over 13 different Äs.
B.8 DERIVING THE PEERMLP
In this appendix, we discuss two potential methods to derive the PeerMLP, and discuss their advan- tages and disadvantages.
B.8.1 Two METHODS TO DERIVE THE PEERMLP
The two potential methods are as follows:
Remove the information aggregation operation in GNN. In this way, we construct a new neural network (PeerMLP, which may contain ikip-connections or other complexities of the GNN layer) by entirely removing the neighbor aggregation operation; hence, the trainable weights of PeerMLP will be the same as GNN by design. We need to build a dataloader for it (Algorithm 1). The advantage of this strategy that it is efficient, since the PeerMLP is "pure" MLP (no aggregation required by design).
2. Change the adjacency matrix to an identity matrix In this way, we use the original GNN architecture, but pretend the set of edges is a set of self- each of the nodes. The advantage of this strategy is that the same dataloader and model structure for GNN can be used for MLP- we don't need to change the input of PeerMLP, which are node features and adjacency matrix (changed to an identity matrix). This facilitates code reuse and ease of engineering/devellpmeen associated model forward operations, we pay for some more training time owing to these operations (graph sampling and identity aggregation).
However, since also must the GNN dataloader and
25

---

Published as a conference paper at ICLR 2023
Performance of PeerMLP and GNN with converged weights mip) ferent À on dataset OGB-arXiv. The accuracy in percentage is based on 5 runs.
Table 15:
) of PeerMLP on dif-
(0.05)
(0.15)
0.2
0.5
0.6
0.8
0,9
PearMLP 5.95
6.63
11.88
16.39
19.82
23.50
25.63
31.19
40.65
48.74
53.40
55.67
56. 23
GNN w/ Windpl
5.86
5.86
5.89
8.82
24.10
32.87
40.67
51.41
55.99
59.05
61.30
62.14
62.43
Improv.
| 1.53%
11.59%
50.43%
46.16%
21.5611186
21.58%
39.37%
58.71%
64.81%
37.73%
21.15%
14.79%
11.63%
11.03%
Table 16: Performance of GNN trained with Random Init and MLPInit with different on DCB-arXiv. The accuracy in percentage is the best performance of the two methods and is based on runs.
0.0
(0.05)
0.1
(0.15)
0.2
0.3
0.4
0.5
0.6
0.7
0.9
1.0
Avg.
GNN w/ Random Init]
60.03
59.22
60.71
63:24
64.71
67.46
69.02
70.05
70.72
71.31
71.76
71.70
72.00
67.07
GNN MLPinit
59.78
60.47
62.53
64.49
65.94
67.84
69.11
70.47
70.70
71.45
71.50
71.92
72.25
67.57
Improv.
0.44%
2.12% 3.01%
1.98%
1.91% 0.57%
0.13%
0.59%
-0.03%
0.20%
-0.96%
0.29%
0.34% 0.74%
Table 17: Comparison of the running time of forward and backward for different operations (i.e., X W, and Z) in GNNs. The time unit is milliseconds (ms).
OGB-arXiv
Flickr
Yelp
#Nodes #Edges
169343
89250
716847
1662443
899756
13954819
Forward
Backward
Total Forward Backward
Total Forward Backward
Total
Z
X
W
0.56
1.31
1.87
0.51
1.80
2.31
3.10
9.14
12:24
H
A Z
1.47
1285.86
1287.33
1.16
842.49
843.65
13.19
16956. 19
16969.38
H 1.Z
1.32
112.82
114.13
0.70
63.55
64.25
5.61
440.51
446.12
Table 18: Comparison of the running time of forward and backward for different methods (i.e.,
W. W.X,A. A A.W.W.W.. W..Xan and I. W. X). The time unit is milliseconds (ms).
DGB-arX1v
Flickr
Yelp
Forward
Backward
Total Forward Backward
Total Forward Backward
Total
W
0.56
1.31
1.87
0.51
1.80
2.31
3.10
9.14
12.24
A-X.
2.03
1287.17
1289.2
1.67
844.29
845.96
16.29
16965.33
16981.62
I-X- W
1.88
114.13
116
1.21
65.35
66.56
8.71
449.65
458.36
ratio (X WFAA X. W)
1:689
1:366
1:1387
ratio WEd X W)
1:28
For more complex graph convolution layers, we take a layer with skip-connection in GNN, H' a (AH''' W! gan +H' +H an example. To derive the PeerMLP for the layer with skip-connection, Method directly removes the adjacency matrix A yield H' (H'-1W2 gan H Thus, the PeerMLP will also contain a skip-connection operation. Method can be casily and directly adopted since it just alters the adjacency matrix trivial way. H' o "H w!
gmn H1-1).
B.8.2 DISCUSSION ABOUT THE EFFICIENCY OF PEERMLP DERIVING
Firstly, Method and Method are mathematically equivalent.
Secondly, in the sense of producing computational graphs, they are different. In the next, we only consider the terms 1 w and I H W gmn. in Method and since the rest terms are the same. For operation H gmm
W gan in Method it only has one dense matrix multiplication. For operation H W gan in Method 2, it has two steps, one is dense matrix multiplication (Z H W ) ) the other is sparse matrix multiplication (I Z). Thus they will produce different computational graphs (using torch geometric). Typically, sparse matrix multiplication needs much
26

---

Published as a conference paper at ICLR 2023
more time than dense matrix multiplication (in the that the sparse matrix is an identity matrix, this still holds if the computation package has no special optimization for the identity matrix).
To investigate the efficiency of these two methods in production environments, conducted exper- iments to present the running time of different operations (X W, A and I Z) and different methods (X W, A X W and I X W). X W is dense matrix multiplication, A Z and I Z are sparse matrix multiplication. The experiments are conducted with an NVIDIA RTX A5000. The softwares and their version in this experiment are cudatoolkit (11.0.221), PyTorch (1.9.1), torch- sparse (0.6.12). The results are presented in Tables 17 and 18. From the results, we have the following observation:
Table shows that I Z (sparse matrix multiplication) takes much more time than X W (dense matrix multiplication).
Table 18 shows that I X W takes much more time than X W. where the ratio of the rumning time for (X W):(1 X W) are 1 62, 1 : 28, and 1 for OGB-arXiv, Flickr and Yelp, respectively. The results indicate that the I X W is not equivalent to X W in the sense of producing computational graph (at least for the sparse matrix multiplication in torch-sparse package).
The choice between the two ultimately boils down to the intended setting. Method 2 is easier for development, and Method more optimized for speed (and thus useful in aesource-constraine settings, or for efficiency in production environments).
27

---

Published as a conference paper at ICLR 2023
C DATASETS AND BASELINES
In the appendix, we present the details of datasets and baselines for node classification and link prediction tasks.
C.I DATASETS FOR NODE CLASSIFICATION
The details of datasets used for node classification are listed as follows:
Yelp (Zeng et al., 2019) contains customer reviewers as nodes and their friendship as edges. The node features are the low-dimensional review representations for their reviews.
Flickr (Zeng et al., 2019) contains customer reviewers as nodes and their common prop- erties edges. The node features the 500-dimensional bag-of-word representation of the images.
Reddit, Reddit2 (Hamilton et al., 2017) is constructed by Reddit posts. The node in this dataset is a post belonging to different communities. Reddit2 the sparser version of Reddit by deleting some edges.
A A-products (Zeng al., 2019) contains products and its categories.
DGB-arXiv (Hu et al., 2020) the citation network between all arXiv papers. Each node denotes paper and each edge denotes citation between two papers. The node features the average 228-dimensional word vector its title and abstract.
OGB-products (Hu et al., 2020; Chiang et al., 2019) is Amazon product co-purchasing network. Nodes represent products in Amazon, and edges between two products indicate that the products are purchased together. Node features in this dataset are low-dimensional representations of the product description text.
We present the statistics of datasets used for node classification task in Table 19.
Table 19: Statistics for datasets used for node classification.
Dataset
# Nodes.
Edges
Classes
# Feat
Density
Flickr
89,250
899,756
0.11%
Yelp 716,847
13,954,819
100
300
0.03%
Reddit 232,965
114,615,892
41
602
2.1 2.1e%
Reddit2 232,965
23,213,838
41
602
0.43%e
A d-products
1,569,960
264,339,468
107
200
0.1
DGB-arXiv 169,343
1,166,243
40
0.04%
OGB -cB-producta 2,449,029
61,859,140
47
218
0.01%
C.2 BASELINES FOR NODE CLASSIFICATION
We present the details of GNN models as follows:
GCN (Kipf & Welling, 2016a; Hamilton et al., 2017) is the original graph convolution network, which aggregates neighbor's information to obtain node representation. In our experiment, we train GCN in a mini-batch fashion by adopting subgraph sampler from Hamilton et al. (2017).
GraphSAGE (Hamilton al., 2017) proposes graph sampling- based training strategy to scale up graph neural networks. It samples a fixed number of neighbors per node and trains the GNNs in mini-batch fashion.
GraphSAINT
(Zeng et al., 2019) is a graph sampling saapling-based based method to train GNNs on large-scale graphs, which proposes set of graph sampling algorithms to partition graph data into subgraphs. This method also presents normalization techniques to eliminate ases during graph sampling.
28

---

Published as a conference paper at ICLR 2023
ClusterGCN (Chiang et al., 2019) is proposed to train GCNs in small batches by using the graph clustering structure. This approach samples the nodes associated with the dense subgraphs identified by the graph clustering algorithm. Then the GCN is trained by the subgraphs.
C.3 DATASETS FOR LINK PREDICITON
For link prediction task, consider Cora, CiteSeer, PubMed, CoraFull, Physics, A-Photo. and A A-Computers our baselines. The details of the datasets used for node classification are listed as follows:
Cora, CiteSeer, PubMed (Yang et al., 2016) are representative citation network datasets. These datasets contain number of research papers, where nodes and edges denote docu- ments and citation relationships, respectively. Node features are low-dimension represen- tations for papers. Labels indicate the research field of documents.
CoraFull (Bojchevski & Günnnemann, 2018) is citation network that contains papers and their citation relationship. Labels are generated based on topics. This dataset is the original data of the entire network of Cora, and Cora dataset in Planetoid is its subset.
CS, Physics (Shchur et al., 2018) are from Co-author dataset, which is co-authorship graph based on the Microsoft Academic Graph from the KDD Cup 2016 challenge. Nodes this dataset are authors, and edges indicate co-author relationships. Node features represent paper keywords. Labels indicate the research field of the authors.
A -PPoto, A- A-Computers (Shchur et al., 2018) dataset (McAuley et al., 2015). Nodes in this dataset represent products, while edges repre- sent the co- co-purchase relationship between two products. Node features are low-dimension representations of product reviews. Labels are categories of products.
datasets from Amazon co-purchase
We also present the statistics of datasets used for link prediction task in Table 20.
Table 20: Statistics for datasets used for link prediction.
Dataset
# Nodes
# Edges
# Feat
Density
Cora
2,708
5,278
1,433
0.72%o
CiteSeer
3.327
4,552
3,703
0.41%o
PubMed
19.717
44,324
500
0.1 1.1%%
DBLP
17,716
105,734
1,639
0,34%
CoraFull
19,793
126,842
8.710
0.32%
A-Photo
7,650
238,162
745
4.07%
A A-Computers
13,752
491,722
767
2.60%
CS
18,333
163,788
6,805
0.49%
Physics 34,493
495,924
8,415
0.14%
C.4 BASELINES FOR LINK PREDICITON
Our link prediction setup is consistent with our discussion in Section in which we use an inner- product decoder À sigmoid H H to predict the probability of the link existence. We presented the results in Tables 5 and 6. Following standard benchmarks (Hu et al., 2020), the evaluation metrics adopted are AUC, Average Precision (AP), are hits ratio (Hit@#), The experimental details of link prediction are presented in Appendix D.5.
29

---

Published as a conference paper at ICLR 2023
D IMPLEMENTATION DETAILS
In this appendix, we present the hyperparameters used for the node classification task and link prediction task for all models and datasets.
D.1 RUNNING ENVIRONMENT
We run experiments the machine with NVIDIA Tesla T4 GPU (16GB memory) and 60GB DDR4 memory to train the models. For A-producta and -GB-products datasets, we run the experiments with one NVIDIA A100 GPU (40GB memory). The code is implemented based on PyTorch 1.9.0 (Paszke et al., 2019) and PyTorch Geometric 2.0.4 (Fey & Lenssen, 2019). The optimizer is Adam (Kingma & Ba, 2015) train all GNNs and their PeerMLPs.
D.2 EXPERIMENT SETTING FIGURE 2
In this experiment, we use GraphSAGE as GNN on OCB-arXiv dataset. We construct that PeerMLP(fmip (X: wmip) ) for GraphSAGE (f (X, A; Wmip)) and train it for epochs. From the mathematical expression, GraphSAGE and its PeerMLP share the same weights Wolpp and the weights Wmip are only trained by PeerMLP. We use the trained weights for GraphSAGE to make inference along the training procedure. For the landscape, suppose "gn and "hi for GraphSAGE and its PeerMLP, the middle one the loss landscape based on the PeerMLP with optimal with Whip while the right one is the loss landscape based the GraphSAGE with optimal weights w Wgan
have two optimal weight
D.3 EXPERIMENT SETTING FOR FIGURES 3, AND 9 AND TABLES AND
In this appendix, present the detailed experiment setting for our main result Figures 3, and and Tables and 4. We construct the PeerMLP for each GNN. We first train the PeerMLP for 50 epochs and the best model with the best validation performance. And then, we use the weight trained by PeerMLP to initialize the GNNs, then fine-tune the GNNs. To investigate the performance of GNNs, we fine-tune the GNNs for 50 epochs. We list the hyperparameters used in our experiments. We borrow the optimal hyperparameters from paper (Duan et al., 2022). And our code also based the official code of paper (Duan al., 2022). For datasets not included in paper (Duan et al., 2022), we use the heuristic hyperparameter setting for them.
D.4 EXPERIMENT SETTING FOR TABLE
The GNN used Table 2 is GraphSAGE. We construct the PeerMLP for GraphSAGE. on OGB-arXiv and DGB-products datasets. We first train the PeerMLP for 50 epochs and save the best model with the best validation performance. And then we infer the test performance with PeerMLP and GraphSAGE with the weight trained by PeerMLP and we report the test performance in Table 2.
D.5 EXPERIMENT SETTING FOR TABLES AND 6
In this appendix, we present the detailed experiment setting for the link prediction task. We adopt the default setting for the official examples of PyTorch Geometric 2.0.4. The GNN used for the link prediction task is 2-layer GCN, and the decoder is the commonly used inner-product decoder Å sigmoid(H H H (Kipf& & Welling, 2016b).
D.6 EXPERIMENT SETTING FOR FIGURE 4
We explore two kinds of hyperparameters "Training HP" (Learning rate, weight decay, batch size, and dropout) and "Architecture HP" (i.e., layers, number of hidden neurons), this experiment.
Training Hyperparameter Training HP ), total combinations: x 2 x 2 16
30

---

Published as a conference paper at ICLR 2023
Table 21: Training configuration for GNNs training in Figures 3, and and Tables and 4.
Model|
Dataset
#Layers
#Hidden Learning Batch size Dropout Weight decay Epoch
Flickr
4
512
0.0001
1000
0.5
0.0001
50
GraphSAGE
Yelp
4
512
0.0001
1000
0.2
0
50
Reddit
4
512
0.0001
1000
0.2
50
Reddit2
4
512
0.0001
1000
0.2
0
50
A-products
4
512
0.001
1000
0.5
50
OGB-arXiv
4
512
0.001
1000
0.5
0
50
OGB -rr-rurducts
4
512
0.001
1000
0.5
50
Flickr
4
512
0.001
5000
0.2
0.0004
50
GraphSAINT
Yelp
2
128
0.01
5000
0.7
0.0002
50
Reddit
2
128
0.01
5000
0.7
0.0002
50
Reddit2
2
128
0.01
5000
0.7
0.0002
50
A-products
2
128
0.01
5000
0.2
50
OGB-arXiv
2
128
0.01
5000
0.2
50
OGB -GB-prod
2
128
0.01
5000
0.2
0
50
Flickr
2
256
0.001
5000
0.2
0.0002
50
ClusterGCN
Yelp
4
256
0.0001
2000
0.5
50
Reddit
4
256
0.0001
2000
0.5
50
Reddit2
4
256
0.0001
2000
0.5
50
A-products
4
128
0.001
2000
0.2
0.0001
50
OCB-arXiv
4
128
0.001
2000
0.2
0.0001
50
OGB OCB-prodscts
4
128
0.001
2000
0.2
0.0001
50
Flickr
2
512
0.0001
1000
0.5
0.0001
50
Yelp
2
512
0.0001
1000
0.2
50
GCN
Reddit
2
512
0.0001
1000
0.2
50
Reddit2
2
512
0.0001
1000
0.2
50
A-products
2
512
0.001
1000
0.5
50
OGB-arXiv
2
512
0.001
1000
0.5
50
OGB products
2
512
0.001
1000
0.5
50
Learning rate: {0.001, 0.0001}
Weight decay: {le 4, 4e 4}
Batch size: (500, 1000)
Dropout: {0.2, 0.5}
Architecture Hyperparameter ( Architecture HP ), total combinations: 3 x 5 = 15
Number of layers: {2, 3, 4}
number of hidden neurons: {32, 64, 128, 256, 512)
In Figures 4 and 7, we plotted the learning curves based on the mean and standard deviation over all the hyperparameters combinations.
31


Reviews:
[{"review_id": "eVA37yLUYz", "reviewer": "Reviewer_uoW4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new method for initializing GNN, by utilizing the trained weights from MLP. The method can lead to much faster convergence.", "summary_of_the_review": "Overall, this paper reveals an interesting observation for GNN initialization. However, I think this is just a good start for a research paper. I highly encourage the authors to derive theoretical analysis based on the finding and include more convincing experiment settings and ablation studies to verify the idea.", "strength_and_weaknesses": "Strength:\n- The proposed method has a strong empirical performance.\n\n\nWeakness:\n- The paper lacks technical novelty. There is no theoretical analysis of the findings. \n- The observation is not surprising. All the evaluated datasets have rich feature information, where MLP alone can perform well. Therefore, it's not surprising that a trained MLP weight is a good initialization for GNN. The paper should include more experiments on datasets where node features are less important, e.g., molecule classification.\n- More ablation study is needed. To verify the proposed idea is valuable, it will be useful to investigate using shallow-GNN (e.g., 1-layer) weights to initialize GNN.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "X-mE9yhlTl", "reviewer": "Reviewer_gkVw", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper leverages the pre-training of a corresponding/analog MLP to speed up the training of GNNs. For most message-passing GNNs, this paper proposes to use the corresponding MLP training as a precursor initialization step to GNN training, which is called the MLPInit. Extensive experiments on multiple large graphs with various GNN architectures validate that MLPInit can accelerate the training of GNNs and also improve the prediction performance for node classification and link prediction tasks.", "summary_of_the_review": "Overall I would recommend rejection for this current manuscript. This paper explores an interesting direction, initialize some message-passing GNNs' weights using a fully-trained analog MLP, to speed up the convergence and ease the training. However, the discussion is limited to just a few types of fixed-aggregation/convolution GNNs. The convergence speed-up is not reported in a very fair manner. The performance improvement (especially for link prediction) seems promising, but we lack critical understanding, cross-validation, and analysis of the results. Based on these, I think there is still considerable room for improving the manuscript, which could significantly enlarge the potential impact of this work.", "strength_and_weaknesses": "#### Strengths\n1. The idea is easy to describe, understand, and implement. Not only because of these, but the authors also devote much effort to improving the clarity of the writing and the completeness of the experiments.\n2. I like the visualizations in Figure 2 (although it is one single case), and the corresponding analysis (i.e., Observation 1). Although not really surprising, these detailed experimental results and visualizations are new to me.\n3. I think it is a good idea to consider MLPInit for link-prediction tasks, which may be significantly harder to train due to the stochasticity of the loss. And indeed, it is demonstrated that the performance improvements for link prediction are usually larger. \n\n\n#### Weaknesses\n1. I appreciate the idea that the author wants to make the description of the algorithm simple and leave more space for extensive empirical evaluations. However, the exploration of MLPInit is still a bit limited in the following (but not limited to) aspects:\n    1. This paper only considers message-passing GNNs with a fixed and not learnable aggregation operator (or, equivalently, fixed convolution matrix). Some popular MPNNs with learnable, e.g., GAT and GIN-$\\epsilon$, or MPNNs with a feature-dependent aggregator, e.g., min/max-aggregations in PNA, are omitted. From the intuition of this paper, it is possible, and likely, MLPInit could also be helpful to speed up the training of those MPNNs with slightly more complexed aggregators, but these discussions or experiments are mostly missing.\n    2. Different GNN architectures (e.g., GCN and SAGE) and different graph mini-batch sampling strategies (e.g., GraphSAINT and Cluster-GCN) are actually two separate concepts. I would suggest the authors treat them more separately and discuss them one by one. That is, the generalizability to different GNN architectures and the comparison or combination of different mini-batch sampling strategies should be discussed separately. From the current design of experiment tables, it is unclear what are the two effects independently.\n    3. We lack understanding of why MLPInit can improve the final performance of GNNs, even if when GNN is shallow (e.g., 2-layer used in experiments) and the loss/task is simple (e.g., mini-batch trained for node classification). Does it mean without MLPInit, training those GNNs from random initialization will almost always be trapped by some local minima (with higher loss)? Since the authors mostly show the test accuracy but not the training loss, it is a bit unclear whether MLPInit is helpful for convergence or generalization. Also, there are some extreme numbers like node classification on Reddit2 in Table 4, which make people curious to ask if the reported random-initialized GNNs' performance is similar to the other publicly available results. And why could MLPInit simply improve the performance by such a large margin?\n    4. The performance improvement for link prediction tasks looks more promising. But again, we lack intuition and understanding of the results. For example, why is the performance in terms of AUC and AP scores similar but in terms of Hits@XX very different?\n2. The two observations to motivate MLPInit (section 4 but before section 4.1) seem a bit redundant to me. MLPInit could be a good initialization simply may because the GNNs initialized with MLP weights already have high performance, and the subsequent convergence is much faster than random initialization. I do not quite get the logic/necessity of a separate Observation 2. Why is it necessary that the GNNs with the optimal weights of PeerMLP consistently outperform PeerMLP? Even if it does not outperform PeerMLP, as long as it is better than random initialization, it is already a good initialization to some extent. This redundancy makes this part also a bit hard to understand.\n3. The convergence speed comparison (Table 3) compared the number of training epochs (of random initialization and MLPInit) to reach a certain performance. However, it seems in Table 3; we do not consider the number of epochs needed for MLP training, right? So this may make the comparison of Table 3 a bit unfair. Although the epoch-training time of MLP is shorter than the epoch-time of GNN training, since it is not free, we should not simply omit that. To me, the fast convergence is definitely the major claim/contribution of MLPInit, and I suggest the authors be more careful about those experiments.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "TnDyEuEAMbC", "reviewer": "Reviewer_pot2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes to accelerate the training of GNNs. This is done by defining an MLP limited to just the node features with a matching parameter count/shapes, training that MLP, and then using the resulting weights to initialize the GNN. Surprisingly, this proves to be very effective, as the MLP weights are close to optimal for the GNN. ", "summary_of_the_review": "Overall, while I have a few reservations about one of the results (W1) and the paper is a bit hard to parse at times (W2), the general idea presented in this work is interesting and practical, and the evaluation is thorough. Hence I lean towards acceptance, assuming the authors clean up the text and clarify the things I asked about in the main part of my review.\n\n=== Update after author response ===\n\nI'm happy with the author response, which cleared up many of my concerns. The authors also added several useful results to further strengthen the work empirically. Consequently, I raise my scores on correctness (3 -> 4), empirical novelty (3 -> 4), overall (6 -> 8) and also confidence (3 -> 4).", "strength_and_weaknesses": "=== Strengths === \n\n(S1): The paper proposes a very simple trick to speed up GNN training, and then does a thorough empirical evaluation to show that it actually works. It's very surprising to me that the weights of the MLP actually transfer, and thus I wouldn't be inclined to believe in this method a priori, but the experimental results look sound. \n\n(S2): The paper presents good visualizations to explain why MLPInit could work; e.g. Figure 6 shows an interesting phenomena where the first part of GNN training trajectory is very similar no matter if one trains in GNN-space or MLP-space. \n\n(S3): I imagine the results presented in this work can also lead to some further insights about the inner workings of GNNs. Perhaps GNNs are more similar to MLPs-on-node-features than we think, or (as suggested by Figure 6), a large initial part of their training is approximately equivalent to fitting node features in isolation, and connectivity is taken into account reluctantly, when needed. \n\n \n\n=== Weaknesses === \n\n(W1): I am not convinced by the loss landscape comparison. As far as I understand, for Figure 5 the authors selected the contour lines so that the lowest line (1.360 for OGB-arXiv) is the same in both plots. However, isn't it the case that the MLPInit-based GNN achieves lower loss than the plain one, and thus there would be more contour lines in its plot, but these are just not shown? I think this plot may be conflating flatness (large area with similar/equivalent loss) with having a low minimum loss in general (which also means there is a large area with loss smaller than a given constant, but that area is not flat). I don't think this is a very major result in the paper, yet I think some clarification is needed here. \n\n(W2): The reading experience is a bit bumpy, with many typos or small errors. While this doesn't prevent understanding of the work, it would be nice to clean up these issues. I list some of them in the \"Other comments\" section, and defer the (many) minor ones to the \"Nitpicks\" section. \n\n \n\n=== Other comments === \n\n(O1): Comparing Figure 2 with Table 2, is it the case that plugging in the MLP weights into the GNN makes loss worse but accuracy better? If so then that's an interesting finding that could be highlighted. \n\n(O2): \"MLP whose weights can be made identical\" in the abstract can sound a bit confusing before one reads the paper; maybe expand that part a bit, or mention that this is about the shape of the weights?\n\n(O3): The contribution section mentions that training MLPs is cheaper than training GNNs, but gives no quantitative value for that, leaving the reader wondering (until they get to the actual experiments). Maybe mention the rough ratio there already. \n\n(O4): Do the 0's in Table 4 denote cases where fine-tuning could not improve on top of the MLPInit-initialized weights? Is that not taken into account in the reported averages (which would make sense as technically it's \"infinite speedup\")? \n\n(O5): The title of Observations 6 and 7 is basically the same? \n\n \n\n=== Nitpicks === \n\nHere I include some final nitpicks; they are here to help improve and polish the paper. \n\n- Figure 1 is hard to read, partially because the grid lines are very thick and interfere with the text. \n\n- Missing dots in a few places (after citations in the into, in Observation 1), missing space (in Observation 2), or extra comma + space (Observation 5). \n\n- \"With that MLPs train faster than GNNs in mind\" - maybe \"Having the fact that MLPs train faster...\" \n\n- \"we denote the prediction targets denoted by\" - repetition\n\n- \"with comprehensive empirical analysis\" - I would add \"a\" before \"comprehensive\" \n\n- In caption of Table 1 \"We compute the time used for forward and backward of two operation.\" - what does \"two operation\" mean here? \n\n- In Section 4.2, \"In the section\" -> \"In this section\". \n\n- Overall there are many typos, grammar errors, or broken sentences, in Section 4.2, beginning of Section 5, and also Sections 6-7. I will not list all of them, but I encourage the authors to carefully proof-read their paper, especially the second half. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "OrQm2EuvQq", "reviewer": "Reviewer_hojB", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work proposes to accelerate GNN training by initializing GNN model with a converged MLP of the same parameter size (referred to as peerMLP in the paper).  The authors first point out their empirical observations that 1) a GNN and its peerMLP have same convergence trend and 2) a converged MLP is not good enough and the performance can be further improved by taking the advantage of GNN's message passing scheme. Then, the authors propose the MLPInit based on their observations and demonstrate its great performance in terms of both efficiency and efficacy with rich experiments.", "summary_of_the_review": "I really like this paper, it is neat but strong. The observation that a fully-converged MLP can be a perfect initialization for GNN training is very interesting and meaningful, and the experimental results are very impressive. Though there is no theoretical analysis as a mathematical justification, enough empirical results are given to demonstrate the rationale of the proposed MLPInit. Therefore, I think it deserves a clear acceptance.", "strength_and_weaknesses": "Strength:\n\n1. The writing of this paper is very clear, the motivation is strong, and the paper is organized in an easy-to-follow structure. \n2. The observation of this paper that a fully-converged MLP can be a perfect initialization for GNN training is very interesting and meaningful, and the explanation for this makes sense to me.\n3. The proposed MLPInit makes sense and is orthogonal to other GNN acceleration techniques.\n4. Experimental results are very impressive, the performance improvement in terms of both acceleration and efficacy is very clear. \n5. Example code is provided and the experimental setting is explained, so I think there is no reproducibility concern.\n\n\nWeakness:\n\nIt would be better if the major observation can be explained with some theoretical analysis. I know this might be very challenging, and I agree it is ok if this paper is without any theories, it's just how to further enhance the solidity of this work.\n\nSome questions:\n1. In table 2, if I understand correctly, the results in the \"GNN\" column are without any fine-tuning right? So it is not exactly MLPinit, right? Then comparing these two columns give us the intuition that, the parameters learned with MLP can get further improved by taking the advantage of graph structure with message passing. Then, can we add one more column to show the results of MLPinit in the same table, so that people can get a more straightforward intuition that this MLP-learned parameter can get further improved so we still need to do some training on the GNN?\n2. Compare to the line of GNN pre-training work, in terms of just time, then MLPinit has a clear advantage. But if we only consider the efficacy, can MLPInit get comparable (or even better) performance than other GNN pre-training work?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster