Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper leverages the pre-training of a corresponding/analog MLP to speed up the training of GNNs. For most message-passing GNNs, this paper proposes to use the corresponding MLP training as a precursor initialization step to GNN training, which is called the MLPInit. Extensive experiments on multiple large graphs with various GNN architectures validate that MLPInit can accelerate the training of GNNs and also improve the prediction performance for node classification and link prediction tasks.
strength_and_weaknesses: #### Strengths
1. The idea is easy to describe, understand, and implement. Not only because of these, but the authors also devote much effort to improving the clarity of the writing and the completeness of the experiments.
2. I like the visualizations in Figure 2 (although it is one single case), and the corresponding analysis (i.e., Observation 1). Although not really surprising, these detailed experimental results and visualizations are new to me.
3. I think it is a good idea to consider MLPInit for link-prediction tasks, which may be significantly harder to train due to the stochasticity of the loss. And indeed, it is demonstrated that the performance improvements for link prediction are usually larger. 


#### Weaknesses
1. I appreciate the idea that the author wants to make the description of the algorithm simple and leave more space for extensive empirical evaluations. However, the exploration of MLPInit is still a bit limited in the following (but not limited to) aspects:
    1. This paper only considers message-passing GNNs with a fixed and not learnable aggregation operator (or, equivalently, fixed convolution matrix). Some popular MPNNs with learnable, e.g., GAT and GIN-$\epsilon$, or MPNNs with a feature-dependent aggregator, e.g., min/max-aggregations in PNA, are omitted. From the intuition of this paper, it is possible, and likely, MLPInit could also be helpful to speed up the training of those MPNNs with slightly more complexed aggregators, but these discussions or experiments are mostly missing.
    2. Different GNN architectures (e.g., GCN and SAGE) and different graph mini-batch sampling strategies (e.g., GraphSAINT and Cluster-GCN) are actually two separate concepts. I would suggest the authors treat them more separately and discuss them one by one. That is, the generalizability to different GNN architectures and the comparison or combination of different mini-batch sampling strategies should be discussed separately. From the current design of experiment tables, it is unclear what are the two effects independently.
    3. We lack understanding of why MLPInit can improve the final performance of GNNs, even if when GNN is shallow (e.g., 2-layer used in experiments) and the loss/task is simple (e.g., mini-batch trained for node classification). Does it mean without MLPInit, training those GNNs from random initialization will almost always be trapped by some local minima (with higher loss)? Since the authors mostly show the test accuracy but not the training loss, it is a bit unclear whether MLPInit is helpful for convergence or generalization. Also, there are some extreme numbers like node classification on Reddit2 in Table 4, which make people curious to ask if the reported random-initialized GNNs' performance is similar to the other publicly available results. And why could MLPInit simply improve the performance by such a large margin?
    4. The performance improvement for link prediction tasks looks more promising. But again, we lack intuition and understanding of the results. For example, why is the performance in terms of AUC and AP scores similar but in terms of Hits@XX very different?
2. The two observations to motivate MLPInit (section 4 but before section 4.1) seem a bit redundant to me. MLPInit could be a good initialization simply may because the GNNs initialized with MLP weights already have high performance, and the subsequent convergence is much faster than random initialization. I do not quite get the logic/necessity of a separate Observation 2. Why is it necessary that the GNNs with the optimal weights of PeerMLP consistently outperform PeerMLP? Even if it does not outperform PeerMLP, as long as it is better than random initialization, it is already a good initialization to some extent. This redundancy makes this part also a bit hard to understand.
3. The convergence speed comparison (Table 3) compared the number of training epochs (of random initialization and MLPInit) to reach a certain performance. However, it seems in Table 3; we do not consider the number of epochs needed for MLP training, right? So this may make the comparison of Table 3 a bit unfair. Although the epoch-training time of MLP is shorter than the epoch-time of GNN training, since it is not free, we should not simply omit that. To me, the fast convergence is definitely the major claim/contribution of MLPInit, and I suggest the authors be more careful about those experiments.
clarity,_quality,_novelty_and_reproducibility: 1. I appreciate the clarity and simplicity and agree with the claimed novelty (to the best of my knowledge).
2. The quality is reflected by the extensive experiments. However, the lack of some important discussion and experimental understanding/analysis limits the soundness of the results. Also, from the theoretical/methodological side, the limited discussion and setup also limit its potential contribution.
3. The reproducibility depends on whether the authors will release the code (conditioned on the acceptance) and cannot be judged now.
summary_of_the_review: Overall I would recommend rejection for this current manuscript. This paper explores an interesting direction, initialize some message-passing GNNs' weights using a fully-trained analog MLP, to speed up the convergence and ease the training. However, the discussion is limited to just a few types of fixed-aggregation/convolution GNNs. The convergence speed-up is not reported in a very fair manner. The performance improvement (especially for link prediction) seems promising, but we lack critical understanding, cross-validation, and analysis of the results. Based on these, I think there is still considerable room for improving the manuscript, which could significantly enlarge the potential impact of this work.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a new method for initializing GNN, by utilizing the trained weights from MLP. The method can lead to much faster convergence.
strength_and_weaknesses: Strength:
- The proposed method has a strong empirical performance.


Weakness:
- The paper lacks technical novelty. There is no theoretical analysis of the findings. 
- The observation is not surprising. All the evaluated datasets have rich feature information, where MLP alone can perform well. Therefore, it's not surprising that a trained MLP weight is a good initialization for GNN. The paper should include more experiments on datasets where node features are less important, e.g., molecule classification.
- More ablation study is needed. To verify the proposed idea is valuable, it will be useful to investigate using shallow-GNN (e.g., 1-layer) weights to initialize GNN.
clarity,_quality,_novelty_and_reproducibility: The paper is clearly written.
summary_of_the_review: Overall, this paper reveals an interesting observation for GNN initialization. However, I think this is just a good start for a research paper. I highly encourage the authors to derive theoretical analysis based on the finding and include more convincing experiment settings and ablation studies to verify the idea.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes to accelerate the training of GNNs. This is done by defining an MLP limited to just the node features with a matching parameter count/shapes, training that MLP, and then using the resulting weights to initialize the GNN. Surprisingly, this proves to be very effective, as the MLP weights are close to optimal for the GNN. 
strength_and_weaknesses: === Strengths === 

(S1): The paper proposes a very simple trick to speed up GNN training, and then does a thorough empirical evaluation to show that it actually works. It's very surprising to me that the weights of the MLP actually transfer, and thus I wouldn't be inclined to believe in this method a priori, but the experimental results look sound. 

(S2): The paper presents good visualizations to explain why MLPInit could work; e.g. Figure 6 shows an interesting phenomena where the first part of GNN training trajectory is very similar no matter if one trains in GNN-space or MLP-space. 

(S3): I imagine the results presented in this work can also lead to some further insights about the inner workings of GNNs. Perhaps GNNs are more similar to MLPs-on-node-features than we think, or (as suggested by Figure 6), a large initial part of their training is approximately equivalent to fitting node features in isolation, and connectivity is taken into account reluctantly, when needed. 

 

=== Weaknesses === 

(W1): I am not convinced by the loss landscape comparison. As far as I understand, for Figure 5 the authors selected the contour lines so that the lowest line (1.360 for OGB-arXiv) is the same in both plots. However, isn't it the case that the MLPInit-based GNN achieves lower loss than the plain one, and thus there would be more contour lines in its plot, but these are just not shown? I think this plot may be conflating flatness (large area with similar/equivalent loss) with having a low minimum loss in general (which also means there is a large area with loss smaller than a given constant, but that area is not flat). I don't think this is a very major result in the paper, yet I think some clarification is needed here. 

(W2): The reading experience is a bit bumpy, with many typos or small errors. While this doesn't prevent understanding of the work, it would be nice to clean up these issues. I list some of them in the "Other comments" section, and defer the (many) minor ones to the "Nitpicks" section. 

 

=== Other comments === 

(O1): Comparing Figure 2 with Table 2, is it the case that plugging in the MLP weights into the GNN makes loss worse but accuracy better? If so then that's an interesting finding that could be highlighted. 

(O2): "MLP whose weights can be made identical" in the abstract can sound a bit confusing before one reads the paper; maybe expand that part a bit, or mention that this is about the shape of the weights?

(O3): The contribution section mentions that training MLPs is cheaper than training GNNs, but gives no quantitative value for that, leaving the reader wondering (until they get to the actual experiments). Maybe mention the rough ratio there already. 

(O4): Do the 0's in Table 4 denote cases where fine-tuning could not improve on top of the MLPInit-initialized weights? Is that not taken into account in the reported averages (which would make sense as technically it's "infinite speedup")? 

(O5): The title of Observations 6 and 7 is basically the same? 

 

=== Nitpicks === 

Here I include some final nitpicks; they are here to help improve and polish the paper. 

- Figure 1 is hard to read, partially because the grid lines are very thick and interfere with the text. 

- Missing dots in a few places (after citations in the into, in Observation 1), missing space (in Observation 2), or extra comma + space (Observation 5). 

- "With that MLPs train faster than GNNs in mind" - maybe "Having the fact that MLPs train faster..." 

- "we denote the prediction targets denoted by" - repetition

- "with comprehensive empirical analysis" - I would add "a" before "comprehensive" 

- In caption of Table 1 "We compute the time used for forward and backward of two operation." - what does "two operation" mean here? 

- In Section 4.2, "In the section" -> "In this section". 

- Overall there are many typos, grammar errors, or broken sentences, in Section 4.2, beginning of Section 5, and also Sections 6-7. I will not list all of them, but I encourage the authors to carefully proof-read their paper, especially the second half. 
clarity,_quality,_novelty_and_reproducibility: Clarity: The paper is sometimes a bit hard to read (see my questions and nitpicks), but overall the ideas are clear. 

 

Quality: The quality of the experimental evaluation seems sufficient. 

 

Reproducibility: The results look reproducible. 
summary_of_the_review: Overall, while I have a few reservations about one of the results (W1) and the paper is a bit hard to parse at times (W2), the general idea presented in this work is interesting and practical, and the evaluation is thorough. Hence I lean towards acceptance, assuming the authors clean up the text and clarify the things I asked about in the main part of my review.

=== Update after author response ===

I'm happy with the author response, which cleared up many of my concerns. The authors also added several useful results to further strengthen the work empirically. Consequently, I raise my scores on correctness (3 -> 4), empirical novelty (3 -> 4), overall (6 -> 8) and also confidence (3 -> 4).
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This work proposes to accelerate GNN training by initializing GNN model with a converged MLP of the same parameter size (referred to as peerMLP in the paper).  The authors first point out their empirical observations that 1) a GNN and its peerMLP have same convergence trend and 2) a converged MLP is not good enough and the performance can be further improved by taking the advantage of GNN's message passing scheme. Then, the authors propose the MLPInit based on their observations and demonstrate its great performance in terms of both efficiency and efficacy with rich experiments.
strength_and_weaknesses: Strength:

1. The writing of this paper is very clear, the motivation is strong, and the paper is organized in an easy-to-follow structure. 
2. The observation of this paper that a fully-converged MLP can be a perfect initialization for GNN training is very interesting and meaningful, and the explanation for this makes sense to me.
3. The proposed MLPInit makes sense and is orthogonal to other GNN acceleration techniques.
4. Experimental results are very impressive, the performance improvement in terms of both acceleration and efficacy is very clear. 
5. Example code is provided and the experimental setting is explained, so I think there is no reproducibility concern.


Weakness:

It would be better if the major observation can be explained with some theoretical analysis. I know this might be very challenging, and I agree it is ok if this paper is without any theories, it's just how to further enhance the solidity of this work.

Some questions:
1. In table 2, if I understand correctly, the results in the "GNN" column are without any fine-tuning right? So it is not exactly MLPinit, right? Then comparing these two columns give us the intuition that, the parameters learned with MLP can get further improved by taking the advantage of graph structure with message passing. Then, can we add one more column to show the results of MLPinit in the same table, so that people can get a more straightforward intuition that this MLP-learned parameter can get further improved so we still need to do some training on the GNN?
2. Compare to the line of GNN pre-training work, in terms of just time, then MLPinit has a clear advantage. But if we only consider the efficacy, can MLPInit get comparable (or even better) performance than other GNN pre-training work?
clarity,_quality,_novelty_and_reproducibility: Clarity: Good

Quality: Good

Novelty: Good

Reproducibility: Good

summary_of_the_review: I really like this paper, it is neat but strong. The observation that a fully-converged MLP can be a perfect initialization for GNN training is very interesting and meaningful, and the experimental results are very impressive. Though there is no theoretical analysis as a mathematical justification, enough empirical results are given to demonstrate the rationale of the proposed MLPInit. Therefore, I think it deserves a clear acceptance.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Summary: This paper proposes to accelerate the training of GNNs. This is done by defining an MLP limited to just the node features with a matching parameter count/shapes, training that MLP, and then using the resulting weights to initialize the GNN. Surprisingly, this proves to be very effective, as the MLP weights are close to optimal for the GNN.

The original scores for this paper were 3,3,6,8. After extensive discussion with the authors and between the reviewers, the reviewers updated their score to 5,5,8,8. All reviewers appreciated the simple proposed trick to speed up GNN training. The reviewers who thought the paper was marginally below the acceptance threshold (5) were questioning whether there were sufficient provided evidence for the "generalizability" of MLPInit to other GNN scenarios (see the response to the rebuttal from reviewer gkVw). On the other hand, reviewer pot2 was particularly enthusiastic about the paper in discussion and appreciated the revision with further empirical evaluation. While this AC agrees that further setups could be tested like gkVw suggested, they decided to side with pot2's enthusiasm for the paper as highlighting an interesting line of enquiry to test more and justify this method which already works well. The authors are encouraged to take the discussion in consideration in their camera ready version.
justification_for_why_not_higher_score: Some reviewers questioned how generalizable are the observations made in the paper.
justification_for_why_not_lower_score: The reviewers agreed this paper makes an interesting contribution; and one reviewer was particularly enthusiastic about it.
Decision: Accept: poster