Diffusing to the Top: Boost Graph Neural Networks with Minimal Hyperparameter Tuning

Authors: Lequan Lin, Dai Shi, Andi Han, Zhiyong Wang, Junbin Gao
Primary area: learning on graphs and other geometries & topologies
Keywords: graph neural networks, generative diffusion models, network generation, hyperparameter tuning, node classification, link prediction

Abstract:
Graph Neural Networks (GNNs) are proficient in graph representation learning and achieve promising performance on versatile tasks such as node classification and link prediction.
Usually, a comprehensive hyperparameter tuning is essential for fully unlocking GNN's top performance, especially for complicated tasks such as node classification on large graphs and long-range graphs. This is usually associated with high computational and time costs and careful design of appropriate search spaces. 
This work introduces a graph-conditioned latent diffusion framework (GNN-Diff) to generate high-performing GNNs based on the model checkpoints of sub-optimal hyperparameters selected by a light-tuning coarse search. We validate our method through 166 experiments across four graph tasks: node classification on small, large, and long-range graphs, as well as link prediction. Our experiments involve 10 classic and state-of-the-art target models and 20 publicly available datasets. The results consistently demonstrate that GNN-Diff: (1) boosts the performance of GNNs with efficient hyperparameter tuning; and (2) presents high stability and generalizability on unseen data across multiple generation runs. The code is available at https://github.com/lequanlin/GNN-Diff.

Decision: accept (poster)
Reviewer scores: 6, 5, 6, 6

Reviews and discussion:
<thread>
creation date:
1730330348250

thread_type:
official_review

<thread_object>
creator: reviewer_ti8e

<content>
summary:
This paper proposes GNN-Diff, a graph-conditioned latent diffusion model designed to enhance Graph Neural Networks (GNNs) by generating effective parameters with minimal hyperparameter tuning. This is achieved by utilizing a coarse search strategy that identifies a subset of viable hyperparameters, which then guides the model’s parameter generation. GNN-Diff claims to reduce tuning time and computational cost by creating high-performance GNN parameters through diffusion in a latent space conditioned on graph data. The authors validate GNN-Diff through extensive experiments on various tasks including node classification and link prediction with numerous model backbones, showing improved efficiency and stability.

strengths:
- The idea of using the diffusion framework to help with hyperparameter tuning seems novel and reasonable.
- The experiments are thorough, spanning a diverse set of tasks and backbone models, with results that are both convincing and reproducible.
- The proposed method effectively enhances both speed and performance across most cases, showcasing its practical value.

weaknesses:
- The heterophily datasets used in this paper have faced criticism for extensive duplicate nodes as noted in [1]. It would be beneficial for the authors to consider experimenting with the datasets proposed in [1] for a more robust evaluation of heterophilic cases.

- Table 5 shows that, compared to p-Diff, the improvements with GNN-Diff are relatively marginal and often inconsistent, with many results falling within one standard deviation. This brings into question the effectiveness of the GNN-Encoder design (Eq. 2).

- There is a lack of theoretical analysis. 


[1] A critical look at the evaluation of GNNs under heterophily: Are we really making progress?

questions:
Have the authors tried other forms of G-Encoders other than Eq. (2) and conducted ablation studies about it?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730463120526

thread_type:
official_review

<thread_object>
creator: reviewer_gdji

<content>
summary:
This article introduces an interesting hyperparameter search architecture that learns graph embedding information for specific graph tasks through a designed GNN architecture. This information is then integrated into conditional latent denoising diffusion probabilistic models, using a parameter encoding module to flexibly convert the input and output dimensions of the parameters.

strengths:
1. The overall expression of the article is clear and easy to understand.
2. The experiments are comprehensive, and the performance of the algorithm is impressive.
3. It connects model parameter tuning with the data itself.

weaknesses:
Simply using GNNs for embedding learning on graph-structured data and adding it to the G-LDM module does not analyze why this part of the graph embedding works for the method.

questions:
1. There is a spelling error at line 126 of the article, "P-Eecoder()".
2. Why use such a GNN architecture? Would using classical graph embedding frameworks like GAT or GCN be effective?
3. Clearly, graph data structures are diverse, and using just one type of GNN makes it difficult to effectively embed all types of graph data. When GNNs become ineffective, or when increasing the number of GNN layers causes the graph embeddings to become too smooth, what impact will that have on the method? Since this paper proposes a parameter tuning method, which can be considered a fundamental approach, we need to focus more on when it will fail or the lower limits of the method’s capabilities.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730568004735

thread_type:
official_review

<thread_object>
creator: reviewer_truh

<content>
summary:
In summary， this paper proposes a strategy for tuning hyper-parameters of GNNs to maximize their potential. It is achieved by collecting a set of model parameters, encoding graphs, and then learning to generate parameters for GNNs. Extensive methods have been conducted in this paper to verify the designed method.

strengths:
- This paper is well-organized.
- Extensive experiments are conducted in this paper.

weaknesses:
1. The technical contribution is limited given the related works.
It seems that GNN-diff is a combination of LDM and graph techniques. The sample collection method described in Section 4.1 is trivial, and the designed encoder is also widely used in existing methods. It would be better to highlight the technical contributions in comparison with the different methods mentioned in the related work section.

2. The experiments seem somewhat unrigorous and fail to adequately highlight the efficiency and effectiveness of GNN-diff. 
- Baseline selection: While numerous target GNNs and datasets are utilized, the paper employs only a few comparable baselines. It appears that methods from "Advanced Methods for Hyperparameter Tuning" should be incorporated to validate the designed parameter tuning strategy more effectively. 
-  Evaluations of the effectiveness of Coarse search: In Tables 1-4, the "Coarse" variant struggles to outperform the "Random" and "Grid" baselines. With these results, justifying the effectiveness of the coarse search approach is challenging.

3. The transferability of GNN-Diff is limited due to its hyper-parameter tuning being tailored for a specific dataset, relying on parameter collection and the GAE model.

questions:
The upper bound of the proposed method remains unclear. It is uncertain how a well fine-tuned GCN compares to lightly fine-tuned, more expressive models.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730677716156

thread_type:
official_review

<thread_object>
creator: reviewer_8awd

<content>
summary:
The paper introduces GNN-Diff, a latent diffusion model designed to enhance GNN training while minimizing the need for hyperparameter tuning. The author claim that this approach not only improves GNN performance but also demonstrates significant stability across various tasks.

strengths:
1.	The overall motivation is clear: improving hyperparameter search strategies can save time and enhance performance. This is an area that has not been explored too much in GNN training so far.

2.	The presentation is clear, with well-organized results and analysis provided for better interpretability.

weaknesses:
1.	The reported performance is a significant concern, as many results are noticeably lower than expected. For instance, nearly all results in Table 2 fall substantially short of those reported in existing literature and on the OGBN leaderboard: for example, I believe that GCN on OGBN-Arxiv can achieve 71.6% (as opposed to the reported 69.2%), APPNP can reache 71.8% (versus the reported 55.05%), and GraphSAGE can easily achieve around 79% on OGBN-Products, compared to the reported 75.6%. This discrepancy raises questions about whether the authors adhered to standard settings, whether the reported accuracy is reliable, and, if not, how the experimental setup was configured. Clearer details on these aspects would help clarify the validity of the reported results.

2.	The motivation for certain components is unclear. For instance, why do the authors choose to incorporate GAE in the training process?

3.	An ablation study illustrating the impact of the proposed techniques would strengthen the findings.

4.	The authors use classical GNNs for downstream tasks, but employing SOTA GNNs are necessary. For example, RevGAT[1], SAGN[2] and LazyGNN[3]. Many advanced architectures perform well without extensive parameter tuning, and using these models could better showcase the contributions of this work.

5.	Figure 7 reports only running time, without any indication of memory cost. Additionally, if three different models are trained in this work, I am wondering how the runtime remains lower than that of a random search? A comparison with coarse search would also be useful.

6.	Can the proposed techniques be applied to graph classification?

[1] Training Graph Neural Networks with 1000 Layers

[2] Scalable and Adaptive Graph Neural Networks with Self-Label-Enhanced training

[3] LazyGNN: Large-Scale Graph Neural Networks via Lazy Propagation

questions:
1. Could you explain the reason of the claim that concatenating different receptive fields in message passing can handle both homophilic and heterophilic graphs?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732318865057

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Response to Area Chair and All reviewers

comment:
We would like to express our sincere gratitude to the area chair and all reviewers for their efforts and time in evaluating our paper. Your constructive comments and suggestions have been invaluable in improving the quality and clarity of our work.

Our paper introduces GNN-Diff, a generative diffusion framework designed to boost the performance of GNNs through efficient hyperparameter tuning. Through extensive experiments, we demonstrate that GNN-Diff consistently enhances GNN performance, achieves high stability, and generalizes effectively to unseen data.

In response to the reviewers’ insightful comments, we have made revisions to the paper. Some major changes include:

- Extension to graph classification tasks with pilot experiment results (Appendix I.3, pages 30-31)

- Comparison with Bayesian optimization in 120 experiments (Appendix I.4, page 32)

- Further ablation and analyses on GAE to show the effectiveness of the current architecture and explore other potential architectures (Appendix I.5, pages 33-34)

We hope these updates address the concerns raised and further highlight the significance of our contributions. We sincerely hope the reviewers will reconsider increasing their ratings in light of these revisions, as we believe this work provides a meaningful advancement in making GNNs more efficient and robust for real-world applications. Your support would be greatly appreciated in promoting this work to the broader research community.

Thank you again for your efforts and consideration.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732786854330

thread_type:
official_comment

<thread_object>
creator: area_chair_pfap

<content>
comment:
I would like to encourage the reviewers to engage with the author's replies if they have not already done so. At the very least, please 
acknowledge that you have read the rebuttal.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732794293764

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Clarification on "GAE" and "GFE"

comment:
Dear Area Chair and Reviewers,

The name "Graph Autoencoder (GAE)" has been updated to "Graph Feature Encoder (GFE)" in the latest version of the revised paper, following the suggestion of Reviewer 8AwD. This change aims to better distinguish the module from traditional Graph Autoencoders that use reconstruction loss. However, please feel free to use "GAE" and "GFE" interchangeably in the discussion.

Best regards,

The Authors
</content>
</thread_object>

</thread>
<thread>
creation date:
1733216846302

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
A Sincere Request to Reviewers for Kindly Increasing Your Score

comment:
Dear Reviewers 8AwD, TruH, and ti8E,

We are writing to kindly seek your continued support for the acceptance of our work. As you may have noticed, one reviewer has recently decreased the score and confidence due to limited familiarity with this research area, though the merit of our work is still acknowledged. While we greatly respect the opinions and decisions of all reviewers, we feel deeply sorry to see this situation, as it may have an unintended adverse impact on the final decision.

**We sincerely hope you could kindly  revisit and potentially increase your score considering the efforts we have devoted during the discussion period.** We remain fully committed to addressing any further questions or suggestions you may have within the remaining discussion time and are grateful for your engagement throughout this process.

Thank you for your consideration.

Best regards,

The Authors
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper introduces a method for improving the performance of GNNs with minimal hyperparameter tuning.  It propose a graph-conditioned generative model to generate (better) hyperparameter configurations based on lightly-tuned suboptimal parameters. This approach to parameter tuning is novel as far as I know. While the combination of techniques is compelling, the lack of a fundamentally new contribution limits the overall impact. The experimental evaluation is decent. Although some strongly-relevant baselines were missing, the authors added one such baseline in the rebuttal. 

The biggest weakness of the paper is that the majority of the results are shown with "cluster" training, due to a lack of computational resources. Since, cluster training yields significantly worse performance than full training, the hyperparameters matter more and thus the proposed method looks better than it is. In response to the reviewers' request the authors included results with full training, where the gap to the baselines here is smaller. In my opinion this weakness is almost enough for a rejection, but I still recommend acceptance because the idea is interesting. However, I strongly advise the authors to include a prominent limitation section to highlight the issue.