Diffusing to the Top: Boost Graph Neural Networks with Minimal Hyperparameter Tuning
Authors: Lequan Lin
Dai Shi
Andi Han
Zhiyong Wang
Junbin Gao
Venue: ICLR 2025 Poster (2025)
Keywords: graph neural networks
generative diffusion models
network generation
hyperparameter tuning
node classification
link prediction
Primary area: learning on graphs and other geometries & topologies

Abstract: Graph Neural Networks (GNNs) are proficient in graph representation learning and achieve promising performance on versatile tasks such as node classification and link prediction.
Usually, a comprehensive hyperparameter tuning is essential for fully unlocking GNN's top performance, especially for complicated tasks such as node classification on large graphs and long-range graphs. This is usually associated with high computational and time costs and careful design of appropriate search spaces. 
This work introduces a graph-conditioned latent diffusion framework (GNN-Diff) to generate high-performing GNNs based on the model checkpoints of sub-optimal hyperparameters selected by a light-tuning coarse search. We validate our method through 166 experiments across four graph tasks: node classification on small, large, and long-range graphs, as well as link prediction. Our experiments involve 10 classic and state-of-the-art target models and 20 publicly available datasets. The results consistently demonstrate that GNN-Diff: (1) boosts the performance of GNNs with efficient hyperparameter tuning; and (2) presents high stability and generalizability on unseen data across multiple generation runs. The code is available at https://github.com/lequanlin/GNN-Diff.

Published as a conference paper at ICLR 2025
DIFFUSING TO THE TOP: BOOST GRAPH NEURAL NET- WORKS WITH MINIMAL HYPERPARAMETER TUNING
Lequan Lin University of Sydney
Dai Shi"
Andi Han Riken AIP
Zhiyong Wang University of Sydney
University of Sydney
Junbin Gao University of Sydney
ABSTRACT
Graph Neural Networks (GNNs) are proficient in graph representation learning and achieve promising performance versatile tasks such as node classification and link prediction. Usually, a comprehensive hyperparameter tuning is essential for fully unlocking GNN's s top performance, especially for complicated tasks such node classification on large graphs and long-range graphs. This is usually associated with high computational and time costs and careful design of appropriate search spaces. This work introduces a graph-conditions (GNN-Diff) to generate gerf-rerooring GNNs based on the model checkpoints of sub-optimal hyperparameters selected by light-tuning coarse search. We validate our method through 166 experiments across four graph tasks: node classification small, large, and long-range graphs, well as link prediction. Our experiments involve 10 classic and state-of-the-arr target models and 20 publicly available
latent diffusion framework
datasets. The results consistently demonstrate that GNN-Diff: (1) boosts the
performance of GNNs with efficient hyperparameter tuning: and (2) presents high
stability and generalizability unseen data across multiple generation runs. The code is available at httpt::////gttt thwb. wttp:ttt.cmmmeeeuaannnnnnnnneeee DNNGINDGGGNNNN
INTRODUCTION
Graph neural networks (GNNs) are deep learning models that excel in capturing relationships within graph-structured data, making them highly effective for a variety of graph tasks such as node classification and link prediction (Wu al. 2020; Zhou et al., 2020). Although GNNs achieve good performance simple graph tasks without sophisticated hyperparameter tuning (Kipf & Welling 2017), fully unlocking their potential to achieve top performance or handle complicated graph tasks still requires extensive hyperparameter tuning (Luo et al.) 2024: Tönshoff al. [024]. This is often associated with high computational and time costs and careful design of appropriate search spaces. For example, in Figure [ for node classification typically increases with the hyperparameter search space. Nevertheless.
show that the average test accuracy of GCN (Kipf & Welling 2017)
Avorge Acounay (N
92.3
90:56
110 scaton rung)
GCN
50.13
GCu-hoto
75.11
15112
73.00
2134
2245
4226
43.00
4000
Number Hyperparameter Configuratinns Bearchad
Figure 1: Effeet of hyperparameter search space on GCN performance for node classification. The test accuracy is averaged over 10 search runs. We also show the results of our method, GNN-Diff.
"Equal contribution. Lequan Lin is the corresponding author. Lequan. 1ing mann@syddy..ee. edu au. In memory of MeiMei H. whose love and companionship will always be remembered.

---

Published as a conference paper at ICLR 2025
population GNis
d'feeer hypurpina corigunaiions
The optnnal GNN
GMBs found by gril eearch
GM& found by random search
GNG found by coanse sea
GNMs pererated GANDD
(b)
(ff
Figure 2: GNNs with parameters found by (a) grid search with the full search space, (b) random search with the sub-search space, (c) coarse search with the minimal search space, and (d) GNN-Diff generation based on sub-optimal hyperparameters selected by the coarse search. Top performance may be achieved by large search space at a cost of time, generated with GNN-Diff efficiently.
may cost over 20x of tuning time to gain an accuracy increase of less than 1% This motivates develop method that boosts the performance of GNNs while minimizing search space and tuning time. Furthermore, this method will be particularly beneficial for complex tasks such as node classification in large and long-range graphs (Hu al. 2020a Dwivedi et al. 2022). where hyperparameter tuning often plays a critical role.
In this paper, we propose a graph-conditioned latent diffusion framework (GNN-Diff) to generate high-performing GNN parameters with sub-optimal hyperparametens selected by a coarse search, which random search with a relevantly small search space (Figure 2 Our method assumes that the performance of GNNs is primarily determined by the quality of the learned parameters. In addition, high-quality parameters exist the underlying parameter population produced by sub-optimal but sufficiently good configuration, which can be found by the coarse search. Based on our experiments, the coarse search space only needs be 10% of the grid search space for most graph tasks to find promising configuration. The parameter generation is fulfilled by a latent denoising diffusion probabilistic model (Ho et al. 2020 Rombach et al. 2022). While parameter generation have been extensively studied in previous research (Erkoç al. 2023 Peebles al. 2022 [Schürholt
et al. 2022 Soro et al. 2024] Wang et al. (024), the intrinsic re relationship between data characteristics
and network parameters remains underexplored. To fill this gap, adopt a task-oriented graph feature encoder to integrate data and graph structural information as a condition for GNN parameter generation. Lastly, it noteworthy that GNN-Diff does not require additional hyperparameter tuning for parameter generation, reducing tuning costs to only those incurred by the coarse search.
Results from 166 experiments across 4 graph tasks, 10 target models, and 20 datasets consistently show that GNN-Diff: (1) efficiently boosts GNN performance with minimal hyperparameter tuning; and (2) demonstrates high stability and generalizability over multiple generation runs on unseen data.
2
PRELIMINARIES
2.1 GRAPH NEURAL NETWORKS
Graph Neural Networks (GNNs) are deep-learning architectures designed for graph data. Typically, there are two types of GNNs: spatial GNNs that aggregate neighboring nodes with message passing
(Hamilton al. 2019 Kipl Welling 2017 Gasteiger et al., 2019) and spectral GNNs developed
from the graph spectral theory (Defferrard et al. | Lin & Gao 2023 Zou et al. 2023). The key
Lin
component of GNNs is the convolutional layer, which usually consists of graph convolution, linear transformation, and non-linear activation function.
We denote an undirected graph with N nodes as G(V,E, E,A) A), where and È are the node set and the edge set, respectively, and A IR N is the adjacency matrix containing information of relationships. A can be either weighted or unweighted, normalized or unnormalized. The graph signals are stored in matrix X E RD where Dj is the number of features. Then, the convolutional layer in GCN can be written as o (AXW). involving graph convolution by node aggregation with the adjacency A, feature linear transformation operator W, and the activation Usually, learnable parameters of GNNs exist in the linear operator and sometimes in the parameterized graph convolution as well (Defferrard et al. 2016 Zheng et al. 2021)
2

---

Published as a conference paper at ICLR 2025
2.2 LATENT DIFFUSION MODELS
Diffusion models generate a step-by-step denoising process that recovers data in the target distribution from random white noises. A typical diffusion model adopts a pair of forward-backward Markov chains, where the forward chain perturbs the observed samples eventually to white noises, and then the backward chain learns how remove the noises and recover the original data. Let Wo qw (wo) be the original data (c.g., vectorized network parameters) from the target distribution gw (wo). Then, the forward-backward chains are formulated as follows.
Forward chain. For diffusion steps t = 0.1...., T, Gaussian noises e ~ NoO. I) are injected to wt until qw (wr)
9w (wr/wo) (Wo)dwy = (wr:: O.I). By the Markov property, one may jump to any diffusion steps via w, = Vo:wo + v1 - are, where ã₁ = II (1 3.) with
{3,, B2, .... 37 is pre-defined noise schedule.
Backward chain. The backward chain removes noises from WT gradually via a backward transition kemne gw (W,-i (W,-1/W.), [w,),
which is usually approximated by neural network po (W/-i//w with learnable parameters 6. In this paper, we adopt the framework of denoising diffusion probabilistic models
(DDPMs), which alternatively uses a denoising network Eg to predict the noise injected at each diffusion step with the loss function
LDDPM (w))) -EE/ -e-avooaro + VI -1-
(I)
Latent diffusion for parameter generation. Generating GNN parameters directly with diffusion models may lead to slow training and long inference time, especially when the model is heavily- parameterized. As a common solution to such problems, latent diffusion models (LDM) (Rombach) al. 2022, (Sroo et al., 2024 Wang et al. 2024) first learn a parameter autoencoder (PAE) to convert the target parameters Wo Dh to a dimeninnnioonl latent space 20 P- P-Encoder(ww (wo) ) E RD, with Dp < Dw. then generate in the latent space before reconstructing the original target with a sufficiently precise decoder. The loss function of PAE is usually formulated as the mean squared error (MSE) between original parameters Wo and reconstructed parameters P-Decoder(zg).
3 RELATED WORKS
This summary related works, and we discuss more details in Appendix A The related works cover three topics: network parameter generation, GNN training. and advanced methods for hyperparameter tuning. In network parameter generation, we start with hypernetwork, an early
concept of predicting parameters of the target network (Ha et al. 2017 (Stanlyy et al. 2009). followed
by a review on generative models for parameter generation (Schürholt et al. 2021 2022 Peebles
et al. 2022 Wang et al.) 2024 [Soro al. 2024). p-diff (Wang et al. 2024) the most relevant
method to our approach, which collects checkpoints from the training process of the target network
and adopts an unconditional latent diffusion model to generate high-performing parameters. Next, in
GNN training, we discuss two major approaches to boost GNN performance, including pre-training
(uu et al., 2020b cj Lu al., (02T) and architecture search (Gao et al. 20200] You et al., 2020). However, both approaches inevitably rely on hyperparameter tuning to succeed. Lastly, we consider
some advanced methods of hyperparameter tuning, such as Bayesian optimization (Snoek et al. 2012
2015) and coarse-to-fine search (Moshkelgosha et al., 2017: Payrosangari et al. 2020).
4 GRAPH NEURAL NETWORK DIFFUSION (GNN-DIFF)
GNN-Diff is a graph-conodeee conpitoondiiioned diffusion framework that generates high-performing parameters for a target GNN to match or even surpass the results of time-consuming hyperparameter tuning with much more efficient process. This is achieved via a pipeline of four steps: (1) input graph data, (2) parameter collection of the target GNN with a light-tuning coarse search, (3) training, and (4) inference and prediction. GNN-Diff has three trainable components: a parameter autoencoder (PAE), latent diffusion model (G-LDM). We provide the overview of the GNN-Diff pipeline in Figure 3 The input data and PAE have been previously discussed in Section 2 thus the rest of this section focuses on the remaining details in the GNN-Diff pipeline. The pseudo codes of training and inference algorithms provided in Appendix
a graph feature encoder (GFE), and a graph-conditione,
3

---

Published as a conference paper at ICLR 2025
Inpul Graph Data
Parameler Collaction
GNN- Diff Training
GNN DIT Inference
Graph Bignals
Coarse Search
Parameter Autoencoder (PAE)
Graph Conditional Latent DDPM LG-LDM)
Saampiigg and Prediction
X
Chamgaann
G:LDM
Graph Structure
Wegi:ddogg
Leareigg monn
parametirs
Sampping
Tain with coetgs
Latent
porurader
Target CHN
Graph Feature Encoder (GPE)
P. P-Cecodar
Parameter somples
Pick ago.d corlg
Gaph
WLF
Nosses
TrainWall Labels Youis. Yuu
Checkpolnts
condition
G-LOMd
Figure 3: How GNN-Diff works for node classification. (1) Input graph data: input graph signals, adjacency matrix, and train/validation ground truth labels. (2) Parameter collection: we use a coarse search with small search space to select an appropriate hyperparameter configuration for the target GNN, and then collect model checkpoints with this configuration.
(3) Training: PAE and GFE are firstly trained to produce latent parameters and the graph condition, and then G-LDM learns how to recover latent parameters from white noises with the graph condition as a guidance. (4) Inference: after sampling latent parameters from G-LDM, GNN parameters are reconstructed with the PAE decoder and returned to the target GNN for prediction.
4.1 PARAMETER COLLECTION WITH COARSE SEARCH
To produce nigh-performing GNN parameters, we need a set of good parameter samples such that the diffusion model can generate from their underlying population distribution. Intuitively, the quality of parameter samples relies on the hyperparameter selection. Hence, we apply coarse search with relevantly small search space to determine the suitable configuration. Two types of hyperparameters considered: training-related such learning rate and weight decay, and model-related such as hidden size and the teleport probability a in APPNP (Gasteiger et al.] 2019). We train the target model with multiple runs for each configuration. Then, we select the configuration with the best validation results on average. After the coarse search, model checkpoints trained with the selected configuration are saved as parameter samples. Parameter samples are reorganized and vectorized before they are further processed by the PAE.
4.2 GRAPH FEATURE ENCODER (GFE)
Previous works have discussed using datasets as conditions of parameter generation, but only for the purpose of transfcrability (Nava et al. 2022; Soro et al. 2024 Zhang al. 2024).. The role of specific data characteristics in parameter generation still remains underexplored. To fill this gap, GNN-Diff leverages the graph data and structural information as the generative condition. The GFE designed to encode graph information in graph signals X and structure A to form the condition. Our experiments involve node classification and link prediction tasks, thus the GFE encoder aims producing a graph representation that works well on the specific task. Straightforwardly. we consider adopting a GNN- based architecture for the encoder to capture and process both data and graph information. This architecture is determined by the nature of each task. For example, multiple GNN layers may be well-suited for node classification on homophilic graphs, where connected nodes are normally from the same classes, but inappropriate for heterophilic graphs, where nodes with different labels are prone to linked (Zheng et al.) 2022 Han et al. 2024). Hence, enlightened by some previous works (Shao et al., 2023) Thorpe et al. 2022 Zhu et al. 2020), we introduce structure that is capable of handling node classification on both homophilic and heterophilic graphs. The GFE encoder (without activation) can be denoted as
(2) which is composed of the concatenation of a two-layer GCN, a one-layer GCN, and an MLP, followed by linear transformation W4 E P where Dg and Dp are the dimensions of the concatenation and the latent parameter representation, respectively. Next, the GFE decoder for general node classification is single linear layer G-Decoder(n) w: E RD,, with D. being the
G-Encoder(X,, A) Concat (A3xW AXW,, XW:) W RxD,,
4

---

Published as a conference paper at ICLR 2025
number of node classes. This ensures that graph information is fully encapsulated in the output of the encoder. Finally, the loss function is the cross entropy on node labels. We use this GFE for node classification small homophilic and heterophilic graphs. The GFEs for node classification on large and long-range graphs and link prediction are discussed in Appendix C.T The graph condition is constructed via mean-pooling the nodes as Cg N of graph representation. The purpose of applying mean pooling on nodes construct an overall graph condition regardless of the graph size, which will later be combined with latent representation 2o Pr as input of diffusion denoising network.
1 Σ (1) E RR where 17(2) is the i-th row
4.3 GRAPIICONDDIIOOED LATENT DDPM (G-LDM)
G-LDM is developed from LDM with the graph condition as a consolidated guidance on latent GNN parameter generation. The forward pass of G-LDM is the same as traditional DDPM, while the backward pass uses graph-conditioned backward kernel to recover data from white noises, Given the latent parameters Z, from the PAE encoder and the graph condition cg from the GFE encoder, the corresponding G-LDM loss function is written
LG.LDM (x))) <- (V V1 Vre, cc) co)
For both PAE and denoising network es, we adopt the Convl d-based architecture in (Wagg et al. 2024). We include the graph condition by taking the sum of 2t. cg, and the time embedding input to 6g. We observe in experiments that the most important factor in high-quality generation is to reduce parameter dimension sufficiently while ensuring reconstruction tightness. This can be easily done by setting the appropriate kernel size and stride for Conv aavvld layers in PAE based on the parameter size, which does not require hyperparameter tuning. We discuss more details about the PAE and co architectures and how to ensure high-qualig generation in Appendices C.2 and C.3
4.4 INFERENCE AND PREDICTION
The last step to obtain parameters and return them to the target GNN for prediction. G-LDM generates samples of latent parameters with the learned ca. Then, the learned PAE decoder is applied to reconstruct parameters back to the original parameter space. These parameters are returned the target GNN, thus we have group of GNNs with generated parameters. Lastly, the model with the best validation performance will used for final evaluation.
EXPERIMENTS AND DISCUSSIONS
5.1 EXPERIMENTAL SETUP
Tasks and metrics. We evaluate our method with graph tasks: (1) basic node classification on small homophilic and heterophilic graphs; (2) node classification on large graphs with huge amount of nodes; (3) node classification on long-range graphs with very deep GNNs; (4) link existence prediction on undirected graphs. We accuracy the metric for all tasks except for the node classification on long-range graphs. Since node classes are highly imbalanced in long-range range graphs, we employ macro Fl-score the metric. We will report 100x Fl-scores for presentation purpose.
Baseline methods. We compare GNN-Diff with (1) grid search with full search space; (2) random search with 20% of full search space; (3) coarse search with 10% of full search space; (4) coarse-to- fine (C2F) that narrows down search space based on the coarse search to optimize hyperparameters.
Datasets. We conduct the experiments on 20 publicly available datasets. For basic node clas- sification, we consider 6 homophilic graphs: Cora. Citeseer, and Pubmed (Yang et al.) 2016), as well as Computers, Photo, and Cs (Shchur et al. 2018); and 6 heterophilic graphs: Actor and Wisconsin (Pei et al. (2020). along with Roman-Empizz Minesweeper, and Tolokers (Platonov et al. (023). We also test on large graphs with 89,250 to 2,339,029 nodes, including Flickr (Zeng et al.] 2020). Reddit (Hamilton et al.) 2019), and OCB-arXiv and OGB-Products (Hu et al. 2020a). For long-range graphs, we Pascalvoc-S0 and COCO-SP (Dwivedi et al. (022). Additionally, our link prediction exper- iments involve 4 datasets: Cora and Citeseer (Yang et al. 2016), as well as Chame 1 leon and Squirrel (Rozemberczki et al. 2021
Amazon-Ratinne Lngs,
5

---

Published as a conference paper at ICLR 2025
Target models. 9 classic and state-of-the-an GNNs are selected as target models: GCN (Kipf & Welling 2017). SAGE (Hamilton et al., 2019). APPNP (Gasteiger et al. 2019), GAT (Veličkovid
et 2l.1100 ChebNee Defferrard et al. 2016). H2GCN (Zhu al. 2020). SGC (Wu et al. 2019)).
GRPGNN (Chien et al., 1. and Mix (Abu-HI-Haija et al. 2019). We also include multilayer perceptron (MIP) to represent baseline performance.
Experiment details. All experiments are run on NVIDIA 4090 GPUs with 24GB memory. GNN- Difi's is trained with a consecutive training flow of its three components (200 epochs for GFE, 9000 epochs for PAE and 6000 epochs for G-LDM). We may generate either all or part of the target model's parameters. The partial generation focuses on the last learnable layer, because earlier layers typically perform the role of feature extraction and representation learning, while the last layer is more task-specific and crucial for the final classification (Yosinski et al., 2014). For the sake of efficiency, GNN-Diff performs partial generation in all experiments. To collect partial parameters, we fix the parameters in all layers except the last layer after 190 epochs, save the checkpoint with the best validation result, and further train it with a small learning rate 5e-4 in another 10 epochs for 10 runs. The parameters collected in each run are different due to randomness caused by dropout, but they are assumed to be from the same population, they In evaluation, we generate 100 samples and test the sample with the best validation performance. For the baseline search methods, over 10 runs (3 runs for some large and long-range graphs in consideration of time), then select the configuration with the best average validation performance. To ensure fair comparison with GNN-Diff, which fine-tunes the last layer for parameter collection, we also train the target model for 190 epochs and fine-tune the last layer for 10 more epochs using the best configuration from the baseline search. We then report the better test accuracy from either the full 200-epoch training or the fine-tuning scheme.
obtained by slightly altering the saved checkpoint.
first train the target model with each configuration for 200 epochs
Reproducibility and comparability. The random seed was set as 42 to mitigate randomness in experiments. Our results may be different from some related works such as (Luo et al,, [024]] Tönshoff et al. 2024) due to different data split and model architecture. For example, we only use the first split of Actor and Wi Disconsin (Pei et al. 2020) because all the experiments are conducted based on one train/validationntest split. Also, the target GNNs mostly adopt simple architectures without tricks such as residual connection and layer or batch normalization.
Please refer to Appendices D and for more experiment settings and details.
Table 1: Results of basic node classification on homophilic and heterophilic graphs. We report the average test accuracy (%) and the standard deviation of models selected by validation over 10 training or generation runs. The best results are marked in bold.
Datraets
Cora (Homophily)
Cátesser (Homophily)
Models
Grid
Random
Course
C2F
GNN Duf
Grid
Random
Course
C2F
GNN
MI.P
50.41
56.33
38.28
0.43
58:76
57.61
1.45
$7.51
1.30
57.63
1.44
58.72
0.54
GCN
$2.54 +: 0.96
81.52 = 0.71
$1.80 + 0.45
81.99
0.95
82.33
: 0.17
71.92
1.10
72.04
0.56
71.97
= 0.67
72.13
0:23
72.37
= 0.29
SAGE
30.58
1.04
80.49
30,60
0.15
70.56
0.58
09.50
6831
0.56
70.39
70.45
0.14
APPNP
$1.91 1 0.90
81.69
0.62
$1.47 1.551
81.91
0.50
82.51
1: 0.29
70.82
1.40
69.87
69.68
0.63
69.50
1.:01100 1.32
71.44 1 0.17
GAT
$1.10-t0.52
81:05
0.82
30,13 1.18
80.52
1.29
$1.09
0.10
70.83
11 0.58
70.81
0:06
70.45
1 1.13
69.8
1,10
71.50
= 0.89
ChebNer
$1,83 0.45
81.51
81.30 0.85
81.73
82.05
0.55
71.14
0.13
0.72
71.02
1.12
0.114.0.13
71.65
0.27
H2OCN
$2.05 0.8.8
$1.67 1 0.71
$1.63 1 0.88
62.11
0.72
82.17
1. 0.12
71.49
0.89
71.30
1.31
71:39
1 0.19
71,12
71.76 : 0.25
SGC
$1.80
$1,60++0.0
0.24
71.94
71.82++0.3 0.36
71.37
0.23
72.10
GPRGNN
$1.03
: 0.65
81.34 = 0.63
51.21 4 0.64
81.41 0.54
81.79 1: 0.20
70.93
1.39
70.44
0.71
70005
0.94
70,93 4: 1.19
71.56 = 0.19
MisHop
50.08
79.39
0.77
78,81
79.50
0.71
70.38
0.94
1.27
70.15 0.68
70.81
1.16
71.50 0.49
Datsets
Acbor (Hotrephiit)
Wisconain (Hatmophity)
Models
Grid
Random
Course
C2F
GNN Duf
Grid
Random
Coarse
C2F
CNN Dift
MI.P
37.36
0.71
1781
37.20 +
+.3.3
78663
1.72
78.43
2.36
78:43
4.13
70.61
3.09
1.07
GCN
30.94 0.45
30:43 1 0.47
30.79 1 0.65
30.99
0.46
31.24
0.26
59.50
2.96
59,61
2.62
5902 1 1.72
60.39
3.40
61.17 1 0.78
SAGE
35.54 0.84
35.53
0.84
35.35
+
36.70
0.51
36.11
0.000
75.68
4.26
74.00
4.60
73.3
4:15
73.73
+ 5.48
76.47 + 2.32
APPNP
35.32
35.09
0.53
35.11
0.75
35:53
35.92
0.21
80.98
3.50
79.61
7960
3.50
50.78 3,80
81.16
1.53
GAT
29.84 0.61
29.79 =: 0.86
29.61
0:73
29:87 0.74
29.38
2 0.23
53.92
3.73
$1.95 4.26
52.35
2 2.62
51.57
1 4.72
$2.96 2 1105
37.29
36.48
37.49
80.00
79.61
2.95
7961
2.48
79,80
80.59
HIGEN
34,05 : 0.76
34.05
= 0.76
34,01 1 0.59
33.63
066
34.16
0.21
85.14
2.65
3.39 82.55
81.% 2 3.93
82.55
4: 3.39
N3.72
= 0.90
SGC
30.32 0.39
30.32
0.39
29.13
28.87
30.38
0.17
57.25
4.41
58.24 3.34
57.34
4.06
57.25
4.41
60.32 1.19
GPRONN
36.30 1 0.43
36.56
1.21
0.61 36.36
36.68
0.57
36,84
0.41
80.20
3.63
79.61
3.83
79.11
3.49
80.75
1: 4.41
81.53 = 0.96
MisHop
37.76 1.00
37.57
+ 0.9
37.06 +
67
0.60
38.15
0.38
3.10 77.45
+..41+2.331
79.61 + 2.95
79.80 + 3.07
30.30
0.87

---

Published as a conference paper at ICLR 2025
Table Results of node classification on large graphs. We report the average test accuracy (%) and the standard deviation of models selected by validation over 10 (F1 1 c and OGB-arxive or 3 Reddit t and SGB-Producte s) training or generation runs. The best results are marked in bold.
Datioets
Flickr
Reddit
Models
Grid
Random
Coune
C2F
GNN-Dit Diff
Grid
Random
Coarse
C2P
GNN Diff
GCN
$2.54
0.89
$2.65
$2.64
0.6
0.7
52.84 0U94
91.01 0.00
91.07
0.05
91.12
0.00
91.12
0.7
91,15 0U04
SAGE
53.60 a: 0.29
53.53 a 0.39
53.48 a: 0.47
53.63 4 0.9
53.70 1. 0.02
93.26 0.02
93.19
0.06
93.28
4: 0.03
93.34 1. 0.06
93.45 1: 0.03
APPNP
52.37 061
5L78
0.64
51.92 0.46
51.99 1
52.23 1 0.03
K7.96 0.00
86.44
0.18
86,00
0.12
87,93
0.8
88,07 1 0.45
Dataets
OGB-arKiv
OGB-Products
Models
Grd
Random
Cousse
C2F
GNN-DiI
Grd
Randon
Coarse
C2F
GNN-Dil
QCN
73.29
73.04 0:33
0.16
SAGR
59.91 + 0.38
60.85 +
0.28
60.64 + 0.17
70.03 + 0.26
70.34 + 0.07
75.68 - 0.37
75.0 + 0.33
75.6 + 0.34
76.01 + 0.42
76.82 + 0.14
APPNP
55.05
1.28
1.12
1.02
$6.09 0:24
53.38
54.22
53.59 0.38
Table 3: Results of node classification on long-range graphs. We report the average test Fl-score and the standard deviation of models selected by validation over 3 training or generation runs. We show 100x the original values for presentation purpose. The best results are marked in bold.
Datasets
Pasca1VOC-SP
COCO-SP
Models
Gri
Random
Course
C2F
GNN Didf
Grid
Random
Course
C2F
GNN Duff
MLP
11,82 =: 0.16
11.79 1 0.12
11.77 a 0.14
11.39
1.27
11.97 1 0.04
3:15 4: 0.31
3.13 a 0.16
3.05 1: 0:226
3.17 &: 0.42
3,20 1. 0.05
GCN
23.31 0.14
23,20
0.177
23.18 0:111
23.33 0.26
23,52
1 0.08
7:94:1.0
0:211.0.21 0:21
7 0.26
7.92 1 0.088
7.90
807 1 0.03
SAGE
27.98 _ 0.23
28.08
0.32
27.42.1.22
27.98
0.22
28.24
0.06
8.65 4: 0.61
9.19
9.19:.0.3
8.99 1 1.064
1.552..66 0.61
9.28 1: 0.08
APPNP
17.24 0.12
15.82
0.111
1:114
17.24.0.11 0.12
15.57
1 0,05
4.28.10000 1.00 0.07
4.27 0.08
4.06 0.15
4.27
1.278.0.08 0.08
4.43 1 0.02
SGC
21,65 0.23
21.55
0.221
21.05
0.26
21.32
1
21,80
1 0.12
5.56
1 0.39
5.82 1 0.29
5.53
0:40
2: 0.33
5.84 1: 0.11
GPRGNN
15,56
0.19
15.21
1:000
15.03
0.09
14.91
1. 0.12
15.26
1 0.09
4.40
0.07
4.33
1 0.12
4:27
1 0.08
4.39.1
119.1..00 0.08
4.34
1. 0.07
MixHlp
21.79
0.18
22.37
0.12
22
4 0.16
22.60 à 0.16
22,75
1 0.03
7.06
1 0.19
7.01
a 0.15
6.55 4 0.25
6.87
0.21
6.74 1 0.03
Table 4: Results of link existence prediction. We report the average test accuracy (%) and the standard deviation of models selected by validation over 10 training or generation runs. The best results are marked in bold.
Datisess
Coza
Citeauur:
Models
Random
Random
Counse
C2
GNN Dif
MLP
75.04 # 1.12
74.61 + 1.72
74.5 # 1:25
74.79 + 0.97
75.42 :: 0.19
78.36 # 0.80
78:23 = 1:23
76.79
1.28
78.25 + 1.79
78.57
0.67
GCN
76,92 0.83
76.33
76.99
76,47 4
77.04 0.12
77,90
75.93
76,90
77.25
77.94 0.14
SAGE
75.09 0.39
75.22
1 1.14
7487
1 0.43
74.11 2:26
76.33 1. 0.31
76.25 1 1.15
76.32
1.32
75.84
191
76.43 1.8.5
76.31 0.08
APPNP
75.95 0.3
75.50
1.07
75.64
+ 0.91
05.95
76,94 : 0.15
75,98 + 1.06
75.73= 0.81
75.68
+ 0.2
75.92
76.18
0.15
Che-Net
74.96 0.79
74.53
0.70
74.17 0.48
74.31 + 0.6
74.96
78.31 + 1.88
77.36=2.7
76.76
76,82 0.:
78.35
0.33
Danasess
Chaneloon
Squirre1
Modcl
Grid
Randam
GNN-D.I
Runden
Couss
CBF
GNN-Dil
MLP
76.88
0.39
76.56 1 0.36
75.27 1 0.56
36.1 1. 0.50
36.23 1 0,100
13.24 1.16
73:13_ 0.07
73.13 23131:000
73.28
1.0.2
73.33 0.64
GCN
78.71 1: 0.54
78.50
0.38
77.69
+: 0.43
77.91
0:34
78.26 0:00
74.72
0.12
24:52-045
7466
14,0 1:117
74.73 0,65
SAGE
82.06 0.41
81.50
0.35
81.8 0.40
62.02+
82.68
75.34
0.32
-4.40-0511
0400
74.37+00.7
74.66
APPNP
78.87 :: 0.75
77.67
0:34
77.66
= 0.29
77.74 0.65
77:95 a 0.04
73.61
à 0.22
73.54
13.51
73,61 0:222
73.76
:: 0,09
ChebNet
0.54
77.52
0.49
77.25
77.25
0.18
15.48
1.34
-531
062
75.32
75.54
5.2
MAIN EXPERIMENT RESULTS
Results of basic node classification. We show results of node classification on Cora, Citeseer, Actor, and Wi risconsin n in Table[] More node classification results on homophilic and heterophilic graphs can be found in Appendix II In general, models generated by GNN-Diff achieve the best average test accuracy for 36 out of 40 experiments in Table For the rest of 4 experiments, GNN-Dif produces accuracy that still sufficiently matches the grid search performance. Both GNN-Diff and C2F show clear improvement on the coarse search results, while GNN-Diff outperforms C2F for most target models and datasets. Moreover, it is apparent that models generated by GNN-Diff exhibit more stable performance in terms of standard deviation.
Results of node classification on large graphs. The challenge of large graphs is often associated with the large number of nodes, which makes the computation of graph convolution considerably slow. In addition, GNNs often need a comparably larger hidden size (e.g., 256) to achieve good performance on large graphs (Hu et al. (2020a), which increases the volume of parameters to learn. The experiment results in Table 2 show that GNN-Diff can produce high-performing GNNs large
7

---

Published as a conference paper at ICLR 2025
graphs by generating only the last layer parameters efficiently. In consideration of time and memory efficiency, our experiments are conducted with the clustering training (Chiang et al., 2019). We also discuss the potential of combining GNN-Diff with other state-of-the-art large-scale training algorithms in Appendix 1.7
Results of node classification long-range graphs. In long-range graphs, nodes need exchange information over long distances (i.e., information is passed through many edges to form useful graph representations), which usually relies on very deep GNNs with many graph convolutional layers (Dwivedi et al. 2022). Partial generation is even more helpful in this case because a well-informed graph representation formed just before the last layer, making it an important mapping to final classification. We observe in Table 3 that GNN-Diff generates top-performing GNNs for most long-range graph tasks as expected.
Results of link prediction. To validate GNN-Diff on various graph tasks, we conduct link existence prediction following the experiment design in (Kipf & Welling 2016). We use accuracy as the metric instead of AUC-ROC because we adopt equal-propottion sampling for positive and negative edges. The results are presented in Table Models generated by GNN-Diff show outstanding performance in Cora, Citeseer, and Squirrel. On Chamel however, random search shows more promising results, though GNN-Diff generally better enhances the original performance from coarse search compared to C2F.
5.3 FURTHER DISCUSSIONS
Visualizations
We use visualizations of node classification with GCN on Citeseer as examples to answer the following three questions. Please note that "input samples" refers to the parameter samples collected for GNN-Diff training.
Velldation Acquracy Test Accuracy
Bubble isomap of Generated Parameters and Input Samples
Distribution of Test Accuracy
72.75
Roosed paree.
Racica parci gna
Neyoor paree (nd 2w-15
Text
72.75
Fandir Search
Coutse Search
Validaaioo en Accuraty (N)
Text Acrunsry [%)
(8)
(5)
(6)
Figure 4: (a) Scatter plot between validation and test accuracy for grid. random, and coarse search. The black circle indicates the final models selected by validation. (b) Visualization of generated and input sample parameters with isomap. Each bubble represents one parameter sample, and the bubble size shows the corresponding test accuracy. (c) Kernel density estimation plot of test accuracy distributions corresponding to input samples, noised sample parameters, and GNN-Diff parameters.
Q1. Why are some random or coarse search results even better than grid search? We observe that some random or coarse search results may be even better than the comprehensive grid search. For example, with GCN node classification on Citeseer, results from coarse, random, and grid search are 71.97%. 97%.7 72.04%, and 71.92% respectively. This is different from our previous observation in Figure I In Figure 4 (a), we note that models that achieve higher validation accuracy may not perform better on the test set. We recognize this phenomenon overly extensive grid search. Random search with a smaller search space may solve the problem. But how do we know the appropriate search space size? In addition, more than 80% of experiment results show that grid search can find better generalized models than random search. Thus GNN-Diff, which learns from samples with sub-optimal validation performance, is considered as a better solution.
"overfitting on validation set" with
Q2. How does GNN-Diff generate better performance based on the input samples? In short, GNN-Diff approximates the population of input samples and explores into better performir regions. In Figure (b), we visualize the input parameters (orange) and 100 parameters generated by GNN- Diff (red) via isometric mapping dimensionality reduction (Tenenbaum et al.) 2000). Additionally, we plot each model point bubbles with bubble size representing the corresponding test accuracy. We
8

---

Published as a conference paper at ICLR 2025
observe that parameters generated by GNN-Diff spread around the imput sample distribution. Notably, parameters in the extended distribution may lead to better accuracy than the best-performing input sample (72.70% vs. 72.30%). This may verify our assumption that better-performing parameters exist in the population of sample parameters from a sub-optimal configuration, and can be found by GNN-Diff.
Q3. Can we find better-performing models by simply adding random noises to input samples? There is a slight possibility, but GNN-Diff clearly shows more promising outcomes. To answer this question, we conduct an experiment in which we add random Gaussian noises to the input samples of GNN-Diff. We set the mean noises as 0 and tune the standard deviation (std) in [le-3, Se-3, 1e-2, 2e-2]. All distributions are plotted based on 100 results. The model performance with lightly noised parameters (std = Ie-3) almost follows the sample distribution, while high-level noises (std = 2e-2) significantly diminish the model accuracy. We observe that when noise std = 5e-3, the corresponding accuracy distribution presents slightly better results than the original samples. Nevertheless, parameters from GNN-Diff consistently outperform the noised ones, indicating that GNN-Diff enhances model performance beyond random guessing.
Full Generation vs. Partial Generation
We found that GNN-Diff significantly enhances GNNs by generating only the last learnable layer of the target model. To confirm the effectiveness of this approach, compared GNN- Diff with three different generation and sampling strategies: (a) partial generation as in the main experiments; (b) full generation (best checkpoint) with samples collected by training the best validation checkpoint from 190 epochs with another 10 epochs and a very small learning rate for 10 runs: (c) full generation (all checkpoints) with samples collected by sav- ing all checkpoints over 200 epochs and discarding the first 10 that have not yet converged. In Figure 5] we show the distribution
of the GNN-Diff parameters and the input parameters and their test accu-
(0) rull geeeneen 14) martel Coamenation
DKN off qurtst
ONA Of Inputseroros
Irout aumppa
racy. The test. accuracy distribution of GNN-Diff is based on the results of 30 generation runs of 100 sam- ples. We only show the test accu- racy corresponding to the best vali- dation sample in each run. Overall, GNN-Diff effectively captures and ex- pands the sample distribution across all three strategies. Moreover, the generated samples usually yield better test performance than the input sam- ples. Partial generation and full gener- ation (best checkpoint) produce sim- ilar results, while full generation (all checkpoints) shows noticeably infe- rior performance. This difference may be caused by the low-quality param- eters in the input samples, the test accuracy from the input samples col- lected with the all-checkpoint strategy exhibits a much wider range of lower values (see Figure (c)).
Dons
-0.2
thest checkgoutt
:1
Denitty
103
-0.
$1.5
$2.0
TUE generation (heciporntti
ON TMF) Irzu sample
ONI-2 lepal sempres
Censity
81
conv2 bios param
coTv2 innv2 peram
Test Accuracy (N6)
Figure 5: GNN-Diff for GCN node classification Cora with three generation and sampling strategies. Each row contains visualisations of (1) generated and input sample distributions of the first parameter in last layer bias; (2) gen- erated and input sample distributions of the first parameter in last layer weights; (3) test accuracy of generated and input sample parameters.
Ablation Study on the Graph Condition
An ablation study is conducted to evaluate GNN parameter generation with and without the graph condition (more analyses related to the GFE and graph condition are in Appendix L.5). Here we compare GNN-Diff with p-diff (Wang al. 2024). an unconditional diffusion model for parameter generation. GNN-Diff adopts nearly identical PAE and as p-diff, minimizing the impact of model architecture differences. We apply the same training and hyperparameter settings for both methods to ensure that the only difference is the graph condition. The results are exhibited in Table[5] GNN-Diff outperforms p-diff in almost all experiments, and generally presents more stable generation outcomes

---

Published as a conference paper at ICLR 2025
with lower standard deviation. Therefore, conclude that the graph condition is indeed helpful with GNN parameter generation by providing insightful guidance based on graph and data information.
Table 5: Comparison between GNN parameter generation with and without the graph condi- tion (GNN-Diff vs. p-diff). We consider node classification on 4 types of graphs, Cora (ho- mophily), Actor (heterophily), Flickr (large), and P Pascal (long-range), and two tar-
(a) OCN Core
GCM--ore
Gnd DINCI!
Cild search Laa
Test F1--5oore "est
20.0 n0.3 n0.3
750 500 1000
%
bbi Pescehsch
24
161 DCNFescalvvCcce
get models, GCN and SAGE.
22
10
100
750 500 7h 1000
Vel. rec. of Coniiguntion Selected
Number of Samplec Gererated
Models
SAGE
Mechods
dddif
GNN
p
Figure 6: Sensitivity analysis. How average test results varies with coarse search results (left column) and number of samples gener-
Coca
81.96
0.31
M0.60
IJU5
Actor
31.24 0.36
30.96
0.30
36.11
0.09
36.25
0.31
Flickr
52.84
0.04
52,70 = 0.11
$3.70
0.12
53.35
+ 0.23
PascaLVOC-SP
23.52 0.08
23,49 + 0.09
28.24
0.06
28.12
0.02
ated (right column).
Sensitivity Analyses
(1) Sensitivity analysis on coarse search results. We would like to know whether coarse search with 10% of full search space is able to find sufficiently good configuration for GNN-Diff to generate high-performing parameters. So, we repeat the coarse search for GCN node classification on Cora and Pasca 1 VOC-SP without fixed random seed times and obtain 7 configurations selected for parameter collection. Then, we train and generate with GNN-Diff based on these configurations. The results show that GNN-Diff consistently produces better test outcomes than grid search (see Figure 6 (a) and (b)), which confirms the feasibility of adopting the 10% coarse search space.
(2) Sensitivity analysis on the number of samples generated by GNN-Diff. Since overfitting the validation data may lead to inferior results unseen data, we wonder if generating too many samples from GNN-Diff will encounter this issue, Hence, we set the number of samples generated by GNN-Diff from 10 1000. In general, GNN-Diff's performance to the number of samples, though generating 750 to 1000 samples indeed diminishes the test accuracy on Cora (see Figure (c) and (d)). In consideration of sampling efficiency, we suggest that 100 samples are sufficient for GNN-Diff to produce good performance in a reasonable sampling time.
unseen data is not very sensitive
Time Efficiency
rodes)
rodel
Recabvocas 15,4483433
X0:
300
04.3
Salas
64.5
S4-PEP
14.95
64.54
S.C.4
69990
Figure 7: Time costs of grid search, random search, coarse search, C2F search, and GNN-Diff.
Finally, we compare the time costs of GNN-Diff and other search methods in Figure The coarse search is the most time-consuning component in GNN-Diff (see more details in Appendix [6). GNN-Diff shows a clear time advantage, though it may not be faster than C2F on small graphs such as Cora and Actor. This is acceptable, as GNN-Diff often yields netter-performing models than C2F and is more efficient on larger graphs such as Flickr and fascalvoc-$s which offers greater practical value, especially for complicated tasks with large and long-range graphs.
6 CONCLUSION
In this paper, we proposed a graph-conditioned latent diffusion framework, GNN-Diff, to generate hop-performing GNN parameters by learning from checkpoints saved with sub-optimal configuration selected by a light-tuning coarse search. We validate with 166 empirical experiments that our method is an efficient alternative to costly search methods and is able to generate better prediction outcomes than the comprehensive grid search on unseen data. Future works may involve combining GNN-Diff with efficient large-scale training algorithms, and more details will be discussed in 1.7
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Sami Abu-EI-Haija, Bryan Perozzi, Amol Kapoor, Nazanin Alipourfard. Kristina Lerman, Hrayr Harutyunyan, Greg Ver Steeg, and Aram Galstyan. Mixhop: Higher-order graph convolutional ar- chitectures via sparsified neighborhood mixing. In International Conference on Machine Learning. pp. 21-29, 2019.
Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. Optuna: next-generation hyperparameter optimization framework. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 2019.
Wei-Lin Chiang, Xuanqing Liu, Si Si, Yang Li, Samy Bengio, and Cho-Jui Hsich. Cluster-gen: An efficient algorithm for training deep and large graph convolutional networks. In Proceedings the 25th ACM International Conference on Knowledge Discovery & Data Mining, pp. 257-266, 2019.
Eli Chien, Zhichun Pei, Suhang Lin, Arya Jain, Ashish Krishnamurthy, and Alfred O. Hero. Adaptive universal generalized pagerank graph neural network. In International Conference on Learning Representations, 2021.
Aaron Defazio and Konstantin Mishchenko. Learning-rate-free learning by d-adaptation. In Interna- tional Conference on Machine Learning, pp. 7449-7479, 2023.
Michaël Defferrard, Xavier Bresson, and Pierre Vandergheynst. Convolutional neural networks on graphs with fast localized spectral filtering. In Advances in Neural Information Processing Systems, 2016.
Lior Deutsch. Generating neural networks with neural networks. arXiv:1801.01990 2018.
Yingtong Dou, Kai Shu, Congying Xia, Philip S Yu, and Lichao Sun. User preference-aware fake news detection. In International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 2051-2055, 2021.
Vijay Prakash Dwivedi, Ladislav Rampášek, Michael Galkin, Ali Parviz, Guy Wolf, Anh Tuan Luu, and Dominique Beaini. Long range graph benchmark. Advances in Neural Information Processing Systems, pp. 22326, 2022.
Ziya Erkoç, Fangchang Ma, Qi Shan, Matthias Nießner, and Angela Dai. Hyperdiffusion: Generating implicit neural fields with weight-space diffusion. Proceedings of the International Conference Computer Vision, pp. 14300-14310, 2023.
Hongyang Gao and Shuiwang Graph u-nets. In International Conference on Machine Learning. pp. 2083-2002 2019.
Yang Gao, Hong Yang, Peng Zhang, Chuan Zhou, and Yue Hu. Graph neural architecture search. In Proceedings of the 29th International Joint Conference on Artificial Intelligence, 1403-1409, 2020.
Johannes Gasteiger, Aleksandar Bojchevski, and Stephan Günnnemann. Predict then propagate: Graph neural networks meet personalized pagerank. In International Conference Learning Representations, 2019.
David Ha, Andrew M. Dai, and Quoc V. Le. Hypemetworks.. In International Conference Learning Representations, 2017.
Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. In Advances in Neural Information Processing Systems, 2019.
Andi Han, Dai Shi, Lequan Lin, and Junbin Gao. From continuous dynamics to graph neural networks: Neural diffusion and beyond. Transactions on Machine Learning Research. 2024.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in
Neural Information Processing Systems, 33:6840-6851, 2020.
11

---

Published as a conference paper at ICLR 2025
Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning on graphs. Advances in Neural Information Processing Systems, 33:221 1122118-22333 2020a.
Weihua Hu, Bowen Liu, Joseph Gomes, Marinka Zitnik, Percy Liang, Vijay Pande, and Jure Leskovec. Strategies for pre-training graph neural networks. In International Conference on Learning Representations, 2020b.
Ziniu Hu, Yuxiao Dong, Kuansan Wang, Kai-Wei Chang, and Yizhou Sun. GPT-GNN: Generative pre-training of graph neural networks, In Proceedings of the 26th ACM International Conference on Knowledge Discovery & Data Mining, pp. 1857-1866 2020c.
Maor Ivgi, Oliver Hinder, and Yair Carmon. DoG is SGD's best friend: A parameter-free dynamic step size schedule. In International Conference on Machine Learning, pp. 14465-14499, PMLR, 2023.
Thomas N Kipf and Max Welling, Variational graph auto-encoders, /oXiv:/6/1.073000 2016.
Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In International Conference on Learning Representations, 2017.
Guohao Li, Matthias Müller, Bernard Ghanem, and Vladlen Koltun. Training graph neural networks with 1000 layers. In International Conference on Machine Learning, pp. 6437-6449, 2021.
Lisha Li, Kevin Jamieson, Giulia DeSalvo, Afshin Rostamizadeh, and Ameet Talwalkar. Hyperband: A novel bandit-based approach to hyperparameter optimization. Journal of Machine Learning Research, 18(185): 2018.
Lequan Lin and Junbin Gao. A magnetic framelet-based convolutional neural network for directed graphs. In International Conference on Acoustics, Speech and Signal Processing, 2023.
Chuang Liu, Yibing Zhan, Jia Wu, Chang Li, Bo Du, Wenbin Hu, Tongliang Liu, and Dacheng Tao. Graph pooling for graph neural networks: Progress, challenges, and opportunities. arXiv preprint arXiv:2204.073311 2022.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization, In International Confer- on Learning Representations, 2019.
Yuanfu Lu, Xunqiang Jiang, Yuan Fang, and Chuan Shi. Learning pre-train graph neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 4276-4284, 2021.
Yuankai Luo, Lei Shi, and Xiao-Ming Wu. Classic gnns are strong baselines: Reassessing gnns for node classification. arXiv preprint arXiv:2406.00 08993, 2024.
Konstantin Mishchenko and Aaron Defazio. Prodigy: An expeditiously adaptive parameter-fie leamerr In Proceedings of the 41st International Conference Machine Learning (ICML), 2024.
Vahid Moshkelgosha, Hamed Behzadi-Khormouuj and Mahdi Yazdian- DechirnnDehkordi. Coarse-to-fine parameter tuning for content- based object categorization. In 2017 3rd International Conference on Pattern Recognition and Image Analysis, pp. 160-165, 2017.
Elvis Nava, Seijin Kobayashi, Yifei Yin, Robert K. Katzschmann, and Benjamin Grewe. Meta- learning via classifier(-fres (free) guidance. In 6th Workshop on Meta-Learning at the Advances in Neural Information Processing Systems, 2022.
Samin Payrosangari, Afshin Sadeghi, Damien Graux, and Jens Lehmann. Meta-hyperband: Hyper- parameter optimization with meta-learning and coarse-to-fine. International Conference on Intelligent Data Engineering and Automated Learning, pp. 335-347, 2020.
William Peebles, Ilija Radosavovic, Tim Brooks, Alexei A Efros, and Jitendra Malik. Learning to learn with generative models of neural network checkpoints. arXiv:2209. 128922 2022.
Hongbin Pei, Bingzhe Wei, Kevin Chen-Chuan Chang, Yu Lei, and Bo Yang. Geom-gen: Geometric graph convolutional networks. In International Conference Learning Representations, 2020.
12

---

Published as a conference paper at ICLR 2025
Oleg Platonov, Denis Kuznedelev, Michael Diskin, Artem Babenko, and Liudmila Prokhorenkova. A critical look at the evaluation of gnns under heterophily: Are we really making progress? International Conference Learning Representations, 2023.
Neale Ratzlaff and Li Fuxin. Hypergan: A generative model for diverse, performant neural networks. In International Conference on Machine Learning, pp. 5361-5369. PMLR, 2019.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the Conference Computer Vision and Pattern Recognition, pp. 10684-10695, 2022.
Benedek Rozemberezki, Carl Allen, and Rik Sarkar. Multi-scale attributed node embedding. Journal of Complex Networks, 9(2):cnab014, 2021.
Konstantin Schürholt, Dimche Kostadinov, and Damian Borth. Self-supervised representation learning on neural network weights for model characteristic prediction. Advances in Neural Information Processing Systems, $4:16481-164993 2021.
Konstantin Schürholt, Boris Knyazev, Xavier Giró-i Nieto, and Damian Borth. Hyper-representtttons as generative models: Sampling unseen neural network weights. Advances in Neural Information Processing Systems, 35:27906-27920, 2022.
Zhiqi Shao, Dai Shi, Andi Han, Yi Guo, Qibin Zhao, and Junbin Gao. Unifying ever-smoothing and over-squashing in graph neural networks: A physics informed approach and beyond. arXiv:2309.022 02769, 2023.
Oleksandr Shchur, Maximilian Mumme, Aleksandar Bojchevski, and Stephan Günneeannn Pitfalls of graph neural network evaluation. 1eXivv///1.0558/ 2018.
Kai Shu, Deepak Mahudeswaran, Suhang Wang, Dongwon Lee, and Huan Liu. Fakenewsnet: A data repository with news content, social context, and spatiotemporal information for studying fake news on social media. Big data, 8(3):711-888 2020.
Jasper Snoek, Hugo Larochelle, and Ryan P Adams. Practical bayesian optimization of machine learning algorithms. Advances in Neural Information Processing Systems, 25, 2012.
Jasper Snoek, Oren Rippel, Kevin Swersky, Ryan Kiros, Nadathur Satish, Narayanan Sundaram, Mostofa Patwary, Mr Prabhat, and Ryan Adams. Scalable bayesian optimization using deep neural networks. In International Conference on Machine Learning, pp. 2171-2180, 2015.
Bedionita Soro, Bruno Andreis, Hayeon Lee, Song Chong, Frank Hutter, and Sung Ju Hwang. Diffusion-based neural network weights generation. arXiv:2402. 18/53, 2024.
Kenneth O Stanley, David B D' Ambrosio, and Jason Gauci. A hypercube- based encoding for evolving large-scale neural networks. Artificial Life, 15(2):185-212 2009.
Chuxiong Sun, Hongming Gu, and Jie Hu. Scalable and adaptive graph neural networks with self-label-enhamceed training. arXiv preprint arXiv:2104.0993 2021.
Joshua B Tenenbaum, Vin de Silva, and John Langford. A global geometric framework for nonlinear dimensionality reduction. Science, 290(5500):231--2353 2000.
Matthew Thorpe, Tan Minh Nguyen, Hedi Xia, Thomas Strohmer, Andrea Bertozzi, Stanley Osher, and Bao Wang. GRAND++: Graph neural diffusion with a source term. In International Conference on Learning Representations, 2022.
Jan Tönshoff, Martin Ritzert, Eran Rosenbluth, and Martin Grohe Where did the gap go? reassessing the long-range graph benchmark. Transactions on Machine Learning Research, 2024.
Thomas Unterthiner, Daniel Keysers, Sylvain Gelly, Olivier Bousquet, and Ilya Tolstikhin. Predicting neural network accuracy from weights. arXiv:2002. 1448, 2020.
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. In International Conference on Learning Representations, 2018.
13

---

Published as a conference paper at ICLR 2025
Kai Wang, Zhaopan Xu, Yukun Zhou, Zelin Zang, Trevor Darrell, Zhuang Liu, and Yang You. Neural network diffusion. arXiv:2402.13144, 2024.
Felix Wu, Amauri Souza, Tianyi Zhang, Christopher Fifty, Tao Yu, and Kilian Weinberger. Sim- plifying graph convolutional networks. In International Conference on Machine Learning, pp. 6861-6871, 2019.
Zonghan Wu, Shirui Pan, Fengwen Chen, Guodong Long, Chengqi Zhang, and S Yu Philip. A comprehensive survey on graph neural networks. IEEE Transactions on Neural Networks and Learning Systems, 32(1))4-24, 2020.
Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful graph neural networks? In Internation Conference on Learning Representations, 2019.
Rui Xue, Haoyu Han, MohamadAli Torkamani, Jian Pei, and Xiaorui Liu. Lazygnn: Large-scale graph neural networks via lazy propagation. In International Conference on Machine Learning, pp. 38926-38937, 2023.
Zhilin Yang. William Cohen, and Ruslan Salakhudinov. Revisiting semi-supervised learning with graph embeddings. In International Conference on Machine Learning, pp. 40-48. PMLR, 2016.
Zhitao Ying, Jiaxuan You, Christopher Morris, Xiang Ren, Will Hamilton, and Jure Leskovec. Hier- archical graph representation learning with differentiable pooling. Advances in neural information processing systems, 31, 2018.
Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks? Advances Neural Information Processing Systems, 27, 2014.
Jiaxuan You, Zhitao Ying, and Jure Leskovee. Design space for graph neural networks. Advances in Neural Information Processing Systems, 1309900111 2020.
Hanqing Zeng, Hongkuan Zhou, Ajitesh Srivastava, Rajgopal Kannan, and Viktor Prasanna. Graph- saint: Graph sampling based inductive learning method. In International Conference on Learning Representations, 2020.
Baoquan Zhang, Chuyao Luo, Demin Yu, Xutao Li, Huiwei Lin, Yunming Ye, and Bowen Zhang. Metadiff: Meta-lcarning with conditional diffusion for few-shot Icarning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 16687-16695, 2024.
Chris Zhang, Mengye Ren, and Raquel Urtasun. Graph hypernetworks for neural architecture search. In International Conference on Learning Representations, 2019.
Xin Zheng, Yi Wang, Yixin Liu, Ming Li, Miao Zhang, Di Jin, Philip S Yu, and Shirui Pan. Graph neural networks for graphs with heterophily: A survey. 220i:2202200082.. 2022.
Xuebin Zheng, Bingxin Zhou, Junbin Gao, Yu Guang Wang, Pietro Lió, Ming Li, and Guido Montúfar. How framelets enhance graph neural networks. International Conference on Machine Learning. 2021.
Andrey Zhmoginov, Mark Sandler, and Maksym Vladymyrov. Hypertransformer: Model generation for supervised and semi-supervised few-shot learning. In International Conference on Machine Learning, pp. 27075-27098, 2022.
Jie Zhou, Ganqu Cui, Shengding Hu, Zhengyan Zhang, Cheng Yang, Zhiyuan Liu, Lifeng Wang, Changcheng Li, and Maosong Sun. Graph neural networks: A review of methods and applications. Al Open, 1:57-81, 2020. ISSN 2666-6510.
Jiong Zhu, Yujun Yan, Lingxiao Zhao, Mark Heimann, Leman Akoglu, and Danai Koutra. Beyond homophily in graph neural networks: Current limitations and effective designs. Advances in Neural Information Processing Systems, 33:7793-7804, 2020.
Chunya Zou, Andi Han, Lequan Lin, Ming Li, and Junbin Gao. A simple yet effective framelet-based graph neural network for directed graphs. IEEE Transactions on Artificial Intelligence, 2023.
14

---

APPENDICES CONTENTS
RELATED WORKS
16
PSEUDO CODE OF GNN-DIFE
17
DETAILS OF GIN-DIFF
18
C.I GFE ARCHITECTU URES FOR VARIOUS TASKS
18
PAE AND G-LDM DENOISING NETWORK ARCHITECTURES
19
C33 GNN- DIFF SETTINGS
20
D BASELINE METHODS
20
DATASETS
22
E,1 DATASET SOURCES
22
DATASET STATISTICS
DATASET TRAIN/VALTTEST SPLITS
24
TARGET MODELS
24
TASKS AND RELEVANT DETAILS
25
G.I
BASIC NODE CLASSIFICATION
25
G.2 NODE CLASSIFICATION ON LARGE GRAPHS
25
G.3 NODE CLASSIFICATION
GRAPHS
25
G.4
LINK PREDICTION
26
HYPERPARAMETER TUNING SEARCH SPACE
27
H.1 FULL SEARCH SPACES BASIC NODE CLASSIFICATION
27
H.2 FULL SEARCU SPACES NODE CLASSIFICATION ON LARGE GRAPHS
27
H.3 FULL SEARCH SPACES NODE CLASSIFICATION LONG-RANGE GRAPHS
28
H.4
FULL SEARCH SPACES LINK PREDICTION
28
SUPPLEMENTARY EXPERIMENT RESULTS AND DISCUSSIONS
29
MORE RESULTS OF NODE CLASSIFICATION ON HOMOPHILIC GRAPHS
29
1.2
MORE RESULTS OF NODE CLASSIFICATION ON HETEROPHILIG GRAPHS
29
1.3
EXTENSION TO GRAPH-LEVEL, TASKS AND EXPERIMENT RESULTS
30
1.4
BAYESIAN OPTIMIZATION VS. GNN-DIEG
32
1.5
GRAPH FEATURE ENCODER (GFE) GNN-DIFF TIME ANALYSIS
ANALYSIS
33
16
35
1.7
COMBINE GNN- DIFF LARGE- SCALE TRAINING AL LLGGRRTHMS
35
15

---

Published as a conference paper at ICLR 2025
A
RELATED WORKS
Network Parameter Generation
Hypernetwork (Ha et al. 2017; Stanley et al. 2009) is early concept of deep learning technique for network generation and now serves as a general framework that encompasses many existing methods. A hypernetwork is a neural network that learns how predict the parameters of another neural network (target network). The input of hypernetwork may be as simple as parameter positioning embeddings (Stanley al. (2009). or more informative, such as encodings of data, labels, and tasks (Duutshh 2018
Ha et al. Ratzllt & Fuxin 2019 Zhang et al. 2019 Zhmoginov et al. 2022).
With the emergence of generative modelling, studies on distributions of network parameters have gained their popularity in relevant research. A pioneering work of (Unterthiner et al. (020) showed that simply knowing the statistics of parameter distributions enabled us to predict model accuracy without accessing the data. Enlightened by this finding, two following works (Schürholt et al. 2022, 202T) studied the reconstruction and sampling of network parameters from their population distribution. The similar idea was also investigated by (Peebles et al.) (022), who proposed that network parameters corresponding to a specific metric value of some specific data and tasks could be generated with a mansformer-diffusiin model trained tremendous amount of model checkpoints collected from past training on a group of data and tasks. This method, however, relies highly on the amount and quality of model checkpoints fed to the generative model. A more network- and data-specific approach, p-diff (Wang et al. 2024), alternatively collects checkpoints from the training process of the target network and uses an unconditional latent diffusion model to generate better- performing parameters. Another recent work, D2NWG (Soro et al., 2024), adopts dataset-conditions latent diffusion to generate parameters for target networks on unseen datasets. Diffusion-biis based network generation has also been widely applied to other applications, such as meta learning (Nava et al. 2022. /Zhang et al. 2024) and producing implicit neural-filed for 3D and 4D synthesis (Erkoq et al.
2023
GNN Training
Previous works on GNN training generally focus on two topics: GNN pre-training and GNN architecture search. GNN pre-training involves using node, edge, or graph level tasks to find appropriate initialization of GNN parameters before fine-tuning with ground truth labels (Hu et al. 2020b c Lu et al., 2021). This approach still inevitably relies on hyperparameter tuning to boost model performance. GNN architecture search aims to find appropriate GNN architectures given data and tasks (Gao et al. 2020 You et al. 2020). However, it was shown in (Shchur et al.) 2018) that under comprehensive hyperparameter tuning and proper training process, simple GNN architectures such as GCN may even outperform sophisticated ones.
Advanced Methods for Hyperparameter Tuning
There emerge many recent works on parameter-free optimization, such as (Defazio & Mishchenko 2023 Ivgi al. 2023] Mishchenko & Defazio 2024) that automatically set step size based on problem characteristics. However, most if not all methods require nontrivial adaptation of existing optimizers and show theoretical guarantees only for convex functions. In addition, the parameter-free optimization is only designed for tuning training-relate related hyperparameters such as the learning rate. This means it less suitable for GNN hyperparameter tuning. which usually involves model-related hyperparameters such as the teleport probability of APPNP. Bayesian optimization (Snoek et al. 2012,2015 on the other hand, a more advanced approach to tuning hyperparameters in a more informed way than grid and random search. Bayesian optimization usually adopts probabilistic surrogate model to approximate the loss function and then uses an acquisition function to sample hyperparameters for further search. Additionally, Hyperband (Li et al. 2018). method that is considered more efficient than Bayesian optimization, optimizes the search space by allocating resources (e.g., training time dataset size) to more promising configurations and pruning those that perform poorly. In our experiments, we consider the coarse-to-fine (C2F) search (Moshkelgoshal al. 2017 Payrosangari al.. 2020), a variant of random search and Hyperband that narrows the search space based on the result of coarse search.
16

---

Published as a conference paper at ICLR 2025
B PSEUDO CODE OF GNN-DIFF
Algorithm Algorithm of GNN-Diff Parameter Generation
Input:
Graph condition cg from Algorithm 2 the learned P-Decoder()) from Algorithm
the
learned denoising network Eş from Algorithm
Output: Generated parameters wo that can be returned to target GNN for prediction.
Run Algorithm 5 to sample latent parameter 2o
2: Reconstruct parameters with the learned PAE decoder: wo P-Decoder(Zo)
Algorithm 2 Training Algorithm of GFE (Basic Node Classification Example)
Input:
Graph signals X; graph structure A; graph label Y; graph training mask Mroin's number of training epochs ÉGFF.
Output: Graph condition Cg-
for 1, .... FGFE do
2: Compute G-Encoder:
Concat (A2XW, ,, W
3: Compute G-Decoder: Ý = Softmax(1)) ()))))
4: Take gradient step and update w (W1, W2, W3, W4, Ws}:
CrorssEEuropyy + Merains Ý 2 Mirain)
5: end for
6: Compute graph representation Concat (A XW,, AXW, 2, XW:) W.
7: Conduct mean pooling to obtain graph condition cg N Di= (1).
Algorithm 3 Training Algorithm of PAE
Input: Vectorized parameters collected based coarse search 9w (Wo); number of training epochs EPAE.
Output: Latent parameter representation 2n yx (zo); the learned decoder P-Decoder()).
for = 1, 2, .... FPAE do
2: Compute latent parameter representation Z = P-EEccdeerwwy
3: Compute PAE decoder for reconstruction Wo P-Decoder(Zo
4: Take gradient step and update PAE parameters with the loss function LPAE ||wu Woll
5: end for
6: Use PAE encoder produce latent parameter representation zo P-Eneoder(wo).
Algorithm Training Algorithm of ea in G-LDM
Input: Latent parameter representation Zo (z(2z) graph signals X; graph structure A; number of diffusion steps T'; noise schedule 3 = {3i, B2, ..., 3r ); number of training epochs EG-LDM-
Output: The learned denoising network cg.
Sample Za (z(z))
2: for i = 1,2, .... EG-LDM do
3: Uniform T', (0,
4: Take gradient step and update 0:
|| - (Vārzo + vI āge, 1, (, 1
V Voluzerng
5: end for
17

---

Published as a conference paper at ICLR 2025
Algorithm 5 Inference Algorithm of G-LDM
Input: Number diffusion steps T; noise schedule 3 = {31, $2,... 3r: diffusion sampling variance hyperparameter
{ ..., OT).
Output: Generated latent parameters Zo-
1: Randomly generate white noises 2T ~ (0,1)
2: for T 1, .... 1 do
3: e= 0 if = clse e ~ (0,1)
4: Compute and update
2
co te ore
VI -
-I-
5: end for
C DETAILS OF GNN-DIFF
C.1 GFE ARCHITECTURES FOR VARIOUS TASKS
We provide illustrations of the GFE architectures for various tasks in Figure In GFE encoders, we generally adopt graph convolutional layers to process graph data and structural information. The decoders, on the other hand, is single linear layer (MLP1), thus the graph information is entirely preserved in the output of the encoders, which will later be further processed and used as the graph condition in G-LDM. The details of GFE for each task are as follows. Please note that for simplicity, we show encoders without activation functions, but activation functions can be included in implementation.
Classitiation
Iode Clasattication on Large Graphs
Node Ciassticcttion on Long-range Graphs
Prediction
Cumeatraau)
Cancattuua)
A
Drea's
Grush
condition
condisee
Figure 8: GFE architectures for node classification on small, large, and long-range graphs, and link prediction. The graph condition is designed to incorporate both data and graph structural information and guided by specific tasks and the corresponding loss functions.
Basic node classification. Details can be found in Subsection
Node classification on large graphs. The input to the GFE is set of graph clusters, each with N; nodes for 1, 2..,, Details of how the graph clusters are obtained from the original large graph with N nodes discussed in G.2 The feature matrix and adjacency matrix of each cluster are denoted as X; and A, respectively. We adopt the same encoder architecture as for basic node classification, because the homophilic and heterophilic problem may also exist in large graph node classification. This may or may not be true for the datasets in our experiments, but we design in this way for future extension to other large graphs. Hence, for each cluster, the encoder is applied following Equation 2
7/4 G-Encoder(X,, A:) Concat
A.X, W4 N.D,
(4)
The decoder and loss function are the same the GFE for basic node classification as well. To form the graph condition, we shall first concatenate the outputs of the encoder in the node dimension as
Concat(m, 72. "/...,, mr) ) E Nx,,
which will be eventually turned into the graph condition via the node mean mean-pooling.
Node classification
long-range graphs. The input of GFE is set of batches, where each batch contains a set of small graphs. Suppose that we have batches and that small graph in batch has No.; nodes for b 1,2,.., ... B, and 1, ................ ..., Jb. where Jo J2, is the number of
18

---

Published as a conference paper at ICLR 2025
small graphs in each batch. Then, organize the small graphs in each batch into combined batch graph with N₁ = No,1 + No. + +... Ne.J, nodes. Let the feature matrix and adjacency matrix of
each batch graph formed by the small graphs in batch be X, and As, respectively. The encoder will be applied to each batch graph as
= G-Encoder(X,, As) = Beep-GCN(Xx, Ao) E R.X:
(6)
where Deep-GCN(, is GCN-based structure with GCN convolutional layers and residual connec- tions for the training purpose. We choose this deep GCN-based structure as the GFE encoder to better process long-ferm information among distant nodes, which is usually considered long-range graphs. The decoder is as simple as a single linear layer. Moreover, the loss function is the weighted cross entropy discussed in the appendix G.3 Similar large graphs, the graph condition constructed by mean-pooling the concatenation of encoder outputs:
important in
Concat(V) f2, Ma)) E N,,
where Nan is the total number of nodes of small graphs in all batches.
Link prediction. The GFE for link prediction is designed based on the implementation in (Kipf & Welling, 20T6). Details of the link prediction task be found in Appendix [G.4] The input of GFE is the feature matrix X and the adjacency matrix Auain with only training edges. The encoder is designed as a 2-layer GCN:
7 G-Encoder(X, A Asmin (Arain X i) W2 E RNND;
(8)
where W1 and W are learnable linear operators. The decoder is again a single linear layer, but is followed by the pairwise node multiplication and embedding aggregation for link prediction. The loss function is binary cross entropy computed based on the training edge labels. Lastly, the graph condition can be constructed by mean-pooing the graph nodes.
C.2 PAE AND G-LDM DENOISING NETWORK ARCHITECTURES
Illustrations of the PAE and the G-LDM denoising network éş provided in Figure and Figure 10 respectively. Taking the idea from Wann et al. 2024), both architectures have Convl d-based layers with LeakyReLU activation and instance normalization.
PAE. P-Encoder and P-Decoder are both composed of 4 basic blocks Figure 9 Since the input latent representation of vectorized parameters, the number of input channels of the first block in P-Encoder is 1. The number of output channels for the 4 blocks in P-Encoder is [6, 6, 6, 6]. Accordingly, the number of input channels of the first block in P-Decoder is also 6. The number of output channels of the blocks in P-Decoder is [512, 512, 8, 1].
mcoder-decoder structure with 8 basic blocks in Figure IO] The number of input channels and output channels, kernel size, and stride are the same as in (Wang et al. (024). In Figure 10, we also mark the position where the graph condition eg is involved. We include the graph condition in each encoder and decoder block by adding it to the input of the block directly.
G-LDM Eg. The denoising network éœ adopts
The Basic Block
Latent Paramoturs
Instancehiom ld
Reconstructed
Convid
Parameters
Lesky
Boole Blocks
Gesdlo Blooka
InstanoeNormtd
Convid
Indentily/ / Tunh
Figure 9: The PAE architecture. The PAE adopts an encoder-decoder structure with 4 encoder blocks and 4 decoder blocks. All blocks have the same structure shown in the basic block. The dimensionality of the input parameters is first reduced by the P-Encoder, and then recovered by the P-Decoder.
19

---

Published as a conference paper at ICLR 2025
The Busic Block
Indentity /LaakyRell
Noleed Latent Paramolers
Instanceloortd
Graph Condition
Convid
LeakyReLU
Shop Embbeddiig
Biosle Blocka
Bosle Booke
Instanceliornl
Convid
Enooder
Decoder
Indeniity / Tanh
Graph condition added imput
The eg architecture. Like PAE, Eo adopts an encoder-decoder structure with 4 encoder blocks and 4 decoder blocks. We mark the position where the graph condition is involved by taking sum with the input of each block.
Figure 10:
C.3 GNN-DIB SETTINGS
GFE is trained with the same graph data as target models. PAE is trained with vectorized parameter samples, while G-LDM learns from their latent representation. Batch size for PAE and G-LDM training is 50. All three modules are trained with the same AdamW optimizer (Loshehilov & Hutter 2019) with learning rate 0.001 and weight decay 0.002. The concatenation dimension Dg is set the same as the latent parameter dimension Dp. which decided by the Convld layer PAE. Dropout is applied with dropout rate 0.1. For G-LDM, we set the number of diffusion steps T 1000. We choose the linear beta schedule with 3n =0.0001 and Br =002
How to ensure high quality generation? We observe in experiments that GNN-Diff does not rely on hyperparameter tuning to generate top-performing GNNs. The only factor that may influence the generative outcomes is the dimension of the latent parameters, which can be easily adjusted by setting the kernel size and stride of Conv senvld layers in P-Encoder blocks. Additionally, the appropriate kernel size and stride can be easily decided based on the dimension of the original parameter samples (see T). Therefore, we highlight that GNN-Diff does not introduce additional hyperparameter tuning burdens, and all tuning efforts only associated with the tuning of target models in coarse search.
Table 6: Kernel size and stride of Convld layers in P-Encoder based on the input parameter dimension. The simple rule is: (1) the stride equals to the kernel size; (2) if the parameter dimension larger than 1000, we increase kernel size and stride by for every 5000 parameters.
Vectorized parameter dimension
Kemel size
Stride
< .99
2
2
100-
500-999 999
1000 1000-4999
5
5
5000 9999
10000 14999
7
7
15000 19999
20000 24999
D BASELINE METHODS
We involve four baseline search methods to compare with GNN-Diff, including grid search, random search, coarse search, and C2F search. As aforementioned in Subsection 5.1 random search and coarse search adopt 20% and 10% of the full search space in grid search. In terms of C2F search, first conduct the coarse search, and then fix the model-related hyperparameters and optimizer based on the coarse search result, and further tune other training-relate the hyperparameter configuration leading to the best validation accuracy is (optimizer: 'Adam', learning_rate rate: 0.01, diegy: 0.005, dropout: 0.3, silden_sizee 64), we will fix the optimizer
hyperparameters. For example,
20

---

Published as a conference paper at ICLR 2025
'Adam' and the hidden size 64, and tune the training related hyperparameter' in narrowed search space as flearning rate [0.005, 0.1, 0.05], weight decay [0.0005, 0.005, 0.05]. dropout [0.1, 0.3, 0.5]]. The number of configurations in the search spaces of different baseline methods presented in Table 7 and 10 More details of hyperparameter configurations can be found Appendix
Table 7: Number of hyperparameter configurations in the search spaces of baseline methods for basic node classification on homophilic and heterophilic graphs. The size of the C2F search space depends on the coarse search result.
Basic Node Classiciation
Grid Search
Random Search
Coarse Seurch
C2F Search
MLP
540
108
54
61 80
GCN
540
54
61 80
SAGE
540
108
54
61 80
APPNP
900
180
90
97 116
GAT
1620
162
169
ChebNet
1620
324
162
169 188
H2GCN
540
54
61 80
SGC
540
108
54
61 80
GPRONN
1440
288
144
151 170
MixHop
540
54
80
Table 8: Number of hyperparameter configurations in the search spaces of baseline methods for node classification large graphs. The size of the C2F search space depends the coarse search result.
Large Graphs
Grid Search
Random Search
Coarse Search
C2F Search
GCN
54
12
23
SAGE
54
12
13 23
APPNP
72
25
Table Number of hyperparameter configurations in the search spaces of baseline methods for node classification on long-range graphs. The size of the C2F search space depends on the coarse search result.
Long-range Graphs
Grid Scarch
Random Scarch
Coarse Search
C2F Scarch
ML.P
48
10
5
12 16
GCN
48
12
SAGE
48
10
5
12 16
APPNP
72
8
19
SGC
48
5
12
GPRGNN
72
16
8
15- 19
Mixllop
48
12-16
Table 10: Number of hyperparameter configurations in the search spaces of baseline methods for link prediction tasks. The size of the C2F search space depends the coarse search result.
Link Prediction
Grid Search
Random Search
Coarse Search
C2F Search
MLP
540
GCN
540
108
54
61 80
SAGE
540
108
54
80
APPNP
1620
324
162
169 188
ChebNel
1620
324
162
169 188
21

---

Published as a conference paper at ICLR 2025
E DATASETS
E.1 DATASET SOURCES
All datasets used in our experiments are publicly available and can be easily retrieved from the following sources.
Cora, Citeseer, Pubmed
Eorch_ Planetoid
Simputers,Phote
Amazon
CS
http:://.e.......etet.ttttccteer torch
Coauthor
Actor
Wisconsin
Squiirel
fatastts.Wikippeiaaietttoo
Flickr
ser.clggeeetrrlccccaaeeesssss.. c. Flickr
Reddit
Reddit
O0B-axXiv,0G--Prouucts
PascalVVO-SP2.OCO--S5
httpas//giithhbb.mmmm//iiovo
22

---

Published as a conference paper at ICLR 2025
E.2 DATASET STATISTICS
We provide statistics of 20 datasets used in our experiments in Table 11
12 13
and
15
Table 11:
Datasets in basic node classification on homophilic graphs. We provide statistics of the number nodes, edges, features, and node classes. "# means "the number of".
Dataset
#Nodes
#Egdes
#ecauress
#Node Classes
Cora
2,708
5,429
1,433
7
Citeseer
3,327
4,372
3,703
6
Pubmed
19,717
44,338
500
3
Computers
13,752
491,722
767
10
Phozo
7,487
126,530
8
CS
18,333
100,227
6805
15
Table 12:
Datasets in basic node classification on heterophilic graphs. We provide statistics of the number of nodes, edges, features, and node classes. "#" means "the number of".
Dataset
#Nodes
#Egdes
#IFeaures
#Node Classes
Actor
7,600
30,019
932
#1 sconsi n
251
515
1,703
5
Roman-Empize
22,662
32,927
300
Ame zon-Rat ing iggs
24,492
93,050
300
5
Minesweeper
10,000
39.402
7
Tal Taleksrs
11,758
519,000
2
Table 13:
Datasets in node classification on large graphs. We provide statistics of the number of nodes, edges, features, and node classes. "#" means "the number of"..
Dataset
#Nodes
#Egdes
#Fcatures
#Node Classes
Flickr
89,250
899,756
500
Reddit
232,965
114,615,892
602
41
OGB-azXiv
169,343
1,166,243
128
PiB-Producct.
2,449.029
61,859.140
100
47
Table Datasets node classification on long-range graphs. Long-range graph datasets contain multiple small graphs. We provide statistics of the number of graphs, all nodes, average nodes, edges, features, and node classes. "#" means "the number of".
Datasel
Graphs
WNodes (all)
#Nodes (avg.)
Mggde
#Features
#Node Classes
Pasca IVOC-SP
11,355
5,443.545
-479.4
30,777,444
14
21
COCO-SP
123,286
58,793,216
-476.9
332,091,902
14
Table 15: Datasets in link prediction. We provide statistics of the number of nodes, edges, features, and link classes. "#" means "the number of".
Dataset
#Nodes
#Egdes
#Features
#Link Classes
Cora
2,708
10,556
1,433
2
Citeseer
3,327
9,104
3,703
2
Chamel
2,277
36,101
2,325
2
Squi rrel
5,201
270,962
2,089
2
23

---

Published as a conference paper at ICLR 2025
E.3 DATASET TRAIN/VAL/TEST SPLITS
For Cora, Citeseer, and Pubmed in node classification tasks, we adopt the public fixed train/test/val splits in (Yang et al. 2016). For Computers, Photo, and CS, we generate the splits with 20 nodes and 30 nodes per class in the training and validation sets and the rest in the test set. For heterophilic datasets, we adopt their original splits from (Pei et al. 2020) and (Platonov) jet al., (023), and use the first split out of the 10 available splits for each dataset. This is because only one train/val/test split is considered in our experiments to easily compare the generated performance with the training performance over multiple runs. Large graph datasets are splited in the same way as in their original papers (Zeng et al.) 2020 Hamilton et al. 20199] Hu et al. (2020a). The long-range datasets, Pascal VOC- SP and COCO SP, both contain multiple graphs. Hence, we adopt the train, validation, and test loaders from the implementation in Dwivedi et al. (2022). Besides, in consideration of time, for COCO-SP, we only use 10% of graphs in each loader for our experiment, and we ensure that the reduced loaders contain all node classes. Furthermore, the link prediction datasets are splited based on edges. Please refer to Appendix G.4 for details.
F TARGET MODELS
In Table 16, we show details of the target models in our experiments. The original graph data is denoted as ( and the output of layer is denoted as ( In addition, (our) is the output of the target model, and we only show how X (oit) is computed when the last layer is different from the other layers. We use W(O) represent learnable linear operators in layer E, and represent the activation function. The adjacency matrix A can be normalized in implementation. We focus on the model-specific hyperparameters when designing the search spaces in our experiments. Please refer to the original papers for details of the other notations.
Table 16: Details of target models. We provide information of model layers, model-specific hyperpa- rameters, and implementation sources.
Model
Layer
Model-speciiic Hyperaarameter
MLP
x(f+ = of (X WiS)
GCN
( ((
SAGE
X = [x WX)
APPNP
X((+1) = (1 (I= O X(emn) = (X
Teloport probability or
GAT
X of (09) ((
Number of heads
ChebNet
x((+1) Ee w(
Chehyshev filter K
H2GCN
SAcH A if M. X CombieiX"" X
Hop set
SGC
X (A
Number hops
GPRONN
( 2 No. W
weights Nim
MixHop
X (1x) Combines (A
Hop set
Model
Implementation Source
MLP
GCN
SAGE
APPNP
GAT
ChebNet
H2GCN SGC
GPRGNN
MixHop httpa:/ http:////ehe..e.e....../////aaar
24

---

Published as a conference paper at ICLR 2025
G TASKS AND RELEVANT DETAILS
G.1
BASIC NODE CLASSIFICATION
Task description. The basic node classification is a semi-supervised task on small-scale homophilic and heterophilic graphs. During training, the GNN has access to the features of all nodes and the graph adjacency matrix, but the loss function computed only based on the training node labels. Likewise, validation and test are conducted with the validation and test node labels. Training, validation, and test labels are obtained with masks.
Target model architectures. We use 2-convolutional layer structure for most target models following the common practice. Similarly, for MLP, we use 2 linear layers, which often leads to good performance heterophilic graphs (Xu et al.) (019). For SGC and MixHop, which are designed to have only one graph convolutional layer, we adjust by setting 2 hops for SGC and (0, 1, 2) hops the
for Mip H2GCN also uses hop set of (0, 1, 2), but like most target models, number of layers to 2. The ReLU activation function is used all layers except for the last layer when applicable. Importantly, all target models have simple architectures without using advanced techniques like residual connections or layer normalization.
G.2 NODE CLASSIFICATION ON LARGE GRAPHS
Task description. The node classification on large graphs is the same as the basic node classification except that the graphs are very large, which may cause memory issues and slow computation. To solve this problem, we apply graph clustering as motivated by (Chiang et al. 2019). Graphs are clustered into multiple densely connected subgraphs by minimizing the number of edges cut between clusters. The number of clusters is set to 32 for Fl i ckr and a and 960 for Reddit and OC3-Producet loop through all clusters iteratively via cluster loader in each epoch. Validation and testing are implemented with the cluster loader without shuffling-
Each cluster has its own rain//vlidation/test masks. During training, we
Target model architectures. We use GCN, SAGE, and APPNP target models. GCN and SAGE adopt the 2-convolutional layer structure with the hidden size tuned in [64, 128, 256]. APPNP has 2 graph convolutional layers followed by linear layer to map the graph representation formed by graph convolutional layers to node classes. The ReLU activation function is used in all layers except for the last layer when applicable.
G.3 NODE CLASSIFICATION ON LONG-RANGE GRAPHS
Task description. The node classification on long-range graphs is different from the previous two tasks as there are multiple small graphs in long-range graph datasets. These small graphs are divided into train/validation/lestt sets. Therefore, the node classification long-range graphs is supervised task instead of a semi-supervised task. In addition, the node classes are highly imbalanced in the datasets used in our experiments. For example, in PascalVOC- SP, there are 21 classes with more than 70% of nodes in class O. Accordingly, we use the weighted cross entropy loss for training as in (Dwivedi al. 2022), , which applies higher weights to uncommon classes. In data preprocessing, we construct train/ muun/vllidation/een loaders with batch sizes 128, 500, and 500, respectively. The small graphs in each batch are connected into a large batch graph such that they can be processed by target models in parallel. We train the target model with 200 epochs with 50 batches in each epoch and then evaluate the model with all batches in the validation and test loaders.
Target model architectures. Since long-range graphs usually rely on very deep GNN architectures. we set the number of layers as a hyperparameter in the search spaces, For most target models, we adjust the number of layers in [8, 10] and the hidden size in [192, 256]. For SGC and MixHop, since they are designed to have only one graph convolutional layer, we tune the number of hops of SGC in [8, 10], and the hop set of MixHop in [[6. 8, 10), (8, 10)] Furthermore, considering the taget models are very deep, we adopt the GELU activation function and residual connections following the experiment settings in (Tönshoff al. 2024) as well layer normalization.
25

---

Published as a conference paper at ICLR 2025
G.4 LINK PREDICTION
Task description. The link prediction is binary classification task that predicts the existence of edges between pairs of nodes (Kipf & Welling 2016). If there exists an edge between a pair of nodes, then the edge is called a "positive link", which is corresponding to class 1. In contrast, if there is no edge between a pair of nodes, then it is treated as a "negative link", which is corresponding to class 0. The prediction requires two steps: (1) an encoder to construct node embeddings based on the positive links; (2) a decoder aggregate embeddings of target node pairs by taking inner products. An illustration of the prediction process is provided in Figure
Graph with positive and nogative links
Graph with positive links only
Encoder
embeddings
Node
Decoder
Graphh Conv.
Painiiee
puttppicctiin pepiicatiion
Emtoscoo Agps Seggnn
Sigmold
Graghh
esuili pullii
Prediction
node
Figure 11: Illustration of the link prediction process. This figure is adapted from a nice tuto- rial
www.toooooo.te.cceeeeccoooooooot wwa--maassisaaee masui/graph-neural neural oag.ap/eooottuaalaaaatttorrrooo main/ haapr/gnn_ygg_iiaee htt.:/immmeeeettttinnnonom .ip.tttttanniniii........
The train/validation/tesl split is conducted on the edge level. More specifically, we randomly select 5% edges as validation edges and 10% edges as test edges, and the rest will be training edges. The selected edges are the positive links, and the negative links will be randomly sampled from disconneted node pairs. The number of negative egdes is set be the same as the number of positive edges, which enables us use accuracy as the metric. We present how the training, validation, and testing edges are used Figure T2
Train
Validation
Test
Encoder
Traiiing edgen (negetivn)
vaticiaion [postive)
edges (negetive) Test edgou tire)
Test edgas (negative)
Figure 12: Edges used in the encoder and decoder during training, validation, and testing.
This figure adapted from a nice tutorial in graph- neura www.eeeleeeeerrrreeeeeee wwtwwwks/bblkk.
ccom/tomonoric -aariiimasu gnn ht/g//nnnngggggg/////// mp ementationn ipynb
Target model architectures. All target models used the encoder in the link prediction task. Target GNNs all adopt 2-convolutional layer structure, while the MLP is composed of two linear layers. For APPNP, we apply linear layer after the 2 graph convolutional layers to reduce the embedding dimension. The ReLU activation function is applied the encoder when applicable. The output dimension of the encoder is the same the hidden size of the target model, and is treated hyperparameter that requires tuning. The decoders are the same for all target models, taking inner products of target node pairs and then passing to the Sigmoid activation function for final prediction.
26

---

Published as a conference paper at ICLR 2025
H HYPERPARAMETER TUNING SEARCH SPACE
H.1 FULL SEARCH SPACES BASIC NODE CLASSIFICATION
Traaeed Hyperparameters
optimizer: ['SGD', 'Adam"]
rearnin_rrate: [0.005, 0.01, 0.05, 0.1, 0.5,
weight_decay: [0.0005, 0.005, 0.05]
dropout: 10.1, 0.3, 0.5, 0.7, 0.9]
scheduler: ['MultiStepLR'']
scheduler_millestone: [[100, 125, 150, 175]]
scheduler_gamma: [0.2]
Model-related Hyperparameters
MLP: hiddeh_ddiiiii size [16, 32, 64]
GCN: hidden_ [16, 32, 64]
SAGE: hidden_size [16, 32, 64]
APPNP: prbbirrbllyy [0.1, 0.3, 0.5, 0.7, 0.9]
GAT: hidden_size [16, 32, 64]; num_ num_heads [4, 8, 12]
ChebNet: hidden_size [16, 32, 64]; filter_size [1, 2]
H2GCN: hidden_sizz [16, 32, 64]
SGC: num_hops [1, 3]
GPRGNN: init ['Random', 'PPR',, GPR alpha [0.1, 0.2, 0.5, 0.9]
MixHop: hidden_sie size [16, 32, 64]; power_set [[0, 1, 2]]
H.2 FULL SEARCH SPACES NODE CLASSIFICATION ON LARGE GRAPHS
Traaeed Hyperparameters
optimizer: ['Adam']]
rearning_rate: [0.0005, 0.005, 0.05]
weight_decay: [0.0005, 0.005]
dropout: [0. 1, 0.3, 0.7]
scheduler: ['MultiStepLR')]
scheduler_milestonee mwwestone:: [[150, 170]]
stheduler_gammas [0.2]
Model-related Hyperparameters
GCN: hidden_size [64, 128, 256]
SAGE: hidden_size [64, 128, 256]
APPNP: telebort_probability 10.1, 0.3, 0.5, 0.9]
27

---

Published as a conference paper at ICLR 2025
H.3 FULL SEARCH SPACES NODE CLASSIFICATION ON LONG-RANGE GRAPHS
Training-related Hyperparameters
optimizer: ['AdamW"]
learning_rate: [0.0005, 0.001, 0.005]
weight_decayy [0.0005, 0.005]
dropout: [0.1, 0.3]
scheduler: None
Model-related Hyperparameters
MLP: hidden_size [192, 256], num_layers [8, 10]
GCN: hidden_size [192, 256], num_layers [8, 10]
SAGE: hidden_size (192, 256], num_layer [8, 10]
APPNP: hidden size [256], bum_layers [8, 10], teleport [0.1, 0.5, 0.9]
SGC: hidden_sizze [192, 256], num num_hops [8, 10]
GPRGNN: hidden size [256]. num nammaummaeaaaaaa layers [8, 10], init ['PPR'], GPR [0.1, 0.5, 0.9]
MixHop: hidden_ size [192, 256], power_set [[6, 8, 10), {8, 10H]
H.4 FULL SEARCH SPACES LINK PREDICTION
rraaned Hyperparameters
optimizer: ['SGD', 'Adam'J
rearning_rate: [0.005, 0.01, 0.05, 0.1, 0.5, weight_decay: [0.0005, 0.005, 0.05]
dropout: [0.1, 0.3, 0.5, 0.7, 0.9]
scheduler: ['MultiStepLR']
scheduler_milestonn [[100, 125, 150, 175]]
scheduler scheduler_gamma:
gamma: [0.2]
Model-related Hy perpaarraess
MLP: hidden_size [16, 32, 64]
GCN: hidden_ [16, 32, 64]
SAGE: hidden_size [16, 32, 64]
APPNP: hidden_size [16, 32, 64], teleport. [0.1, 0.5, 0.9]
ChebNet: hidden size [16, 32, 64]; filter filter_size size [1, 2, 3]
28

---

Published as a conference paper at ICLR 2025
1 SUPPLEMENTARY EXPERIMENT RESULTS AND DISCUSSIONS
1.1 MORE RESULTS OF NODE CLASSIHICATION ON HOMOPHILIC GRAPHS
Table 17: More results of basic node classification on heterophilic graphs. We report the average test accuracy (%) and the standard deviation of models selected by validation over 10 training or generation runs. The best results are marked in bold.
Dunsees
Pubmod (llomophlly)
Computera (lomnppilyy)
Midille
Cird
Randon
Comme
C2F
GIN-DDIT
Grid
Random
Charae
C2F
GNN-Dilf'
MLP
73.83
73,79
74.73
2.18
66.79
67.34
0.76
66.39
2,01
67.57 0.16
GCN
79.27 0.000
79.09- :: 0.47
78,70 1: 1.09
79.09
0.47
79.30
0.31
82.42 0.78
80.69
1.42
80:35
1.39
79.07 1: 0.442
82.39 0.29
SACIF
77.26 0.34
77.55
77:27
7
187
7869
3.06
1.89
APPNP
79.15 + 0:770
79:15 = 066
78.72
0.554
77.55
0.39
79.28 1: 0.20
82.77
1.55
82.35
1.58
83.30
= 0.51
K3.57 #: 1.62
83.93 = 1.58
GAT
78.43
78.35 a 0.64
78,08
78.46
0:03
82:70
0.55
82.43
0.3
82.06
0.58
52.94
0.96
83.16 0.44
Chebsees
78.29:00077
77.30 0.59
77.82 0.35
78.31
0.58
73.76
2.77
60.58
69.56
1.51
1.78 00.57
70.12 0.66
HIGCN
79.18 4: 0.33
79:46 = 0.12
78.99 4 0.42
79:01 = 0.21
79.76 0.47
77.29
1.36
75.21 4: 3,50
73.79 :: 4.34
75,02 à: 2.39
77.81 =: 0.92
18,50
77.46
78.33
0.19
80:91
1.56
81.88
0.91
51.79
GPRONN
79.56 0.56
55 + 0.42
79. 19 0:88
70:22- 0.83
79.69 0.08
82.42 0.50
62.79
1:40
82.58
0.58
60.58 + 0.58
82.80 0.30
MisHap
76.96
0:34
73.00
0.96
72.76 1.57
72.57
75.64
Dansees
Photo (llomophily)
CB (llomoophily)
Madele
Ciid
Randon
Coure
C2F
GNN-DDIT
Grid
Random
Clssre
C2F
GNN-Dill
MLP
50.51
61.45
0.91
86.90
1.28
57.59
GCN
90.83 2: 0.45
9081 = 0.37
90.96 1 0.47
91.10
1.57
91.41 : 0.01
91.34
0.39
91.04
0.33
91.25
0.34
91.17 2 0.14
91.35 = 0.02
SAGE
90.32 0.55
90.27 0.70
59.62
88.35
1.25
90.36
0.19
90.98
0.33
90,95
0.332
90.75
0.58
90.91
91.01 0.40
APPNP
50.43 1: 0.77
90:34 0.94
90.12 0.45
90:29-00
0.89
90.99
0.50
92.13
0.25
12.09:00.00
91.73:: 0.20
92.13:10.25
92.30 : 0.82
GAT
00.25 + 0.92
90.24 + 0.71
50.71 0.88
90
1.06
91.01
0.44
90.35
- 0:46
80.81 0.78
89:52++03 0.9
80.73 1.14
98.57 + 0.88
ChebNet
1,87
84.88
53.85
86.28
55.90
0:133
91.79
0.53
91.35: 01.
91.82
91,90 0.84
H2GEN
50.30++0.80
99 + 0.60
50.17 + 1.47
90.29
0.76
0.44 90.83
92.24
0.32
+0.11+0..13
91:27 + 041
90.07 + 0.34
92.24 + 0.84
SGC
90.27
90.47
0.11
90.99
0.23
91.02
GPRGNN
91.18 1: 0.33
90.28 =: 1.26
90.07 4: 1.20
91.24
0.59
91.36 1: 0.28
91.94
0,44
90.81
0.44
90.06 a: 0.53
90.75
1: 2.01
90.93 =: 0.14
MixIlop
$6.31 1:31
85.06 1.17
84,47
84.67
86.12
0.000
92.10
0.33
92,09 0.33
91.05
0.72
91,82
0.51
92.33
0.85
I.2 MORE RESULTS OF NODE CLASSIFICATION ON HETEROPHILIC GRAPHS
Table 18: More results of basic node classification on homophilic graphs. We report the average test accuracy (%) and the standard deviation of models selected by validation over 10 training or generation runs The best results are marked in bold.
Datiact
Ronan Empire (Hoteophilly)
Anazon Rattinge (Hotesophily)
Models
Random
Grid
Random
Coarse
GNN Difi
MLP
66.19
0.17
65.32 = 0136
65.27 0.29
65:27 = 0.29
66.11 0.08
40.20 0.52
38,52 :: 0.68
38.56 = 0.36
39,50 0.35
39.22 = 0.20
GCN
48.87
0.20
47 + 0.57
47.43
0:4
47:70=05
48.43
0.17
43.10
0.13
41.52 0.25
41.16
0.16
43.11 0:33
43.19 : 0.14
SAGE
78.12 0.22
77.32 0.20
76.91
0.32
7691 0:32
77.65
0.12
43.23
0.52
43.27
42.7%
0.57
43.03
43.32 0.06
APPNP
59.23 à 0.19
58.14 =: 021
57.71
à: 0.05
15.25:0.11
58.36
1: 0.34
38.79
0.37
39:03 0.13
3836
2: 0.63
38,86 0:6
39.26 = 0.16
56,86
55.53
1.26
56.10
55,40
0:04
44.11
44.39
0.37
44.42
44.77
ChebNet
50.05 0:27
79.09-
:: 0.40
78,03
0:2:1
8000
2005=022
0.27
$0.06
1 0.04
44.56
0.16
0.35
4396
0.28
43.93
46.06 : 0.18
H2GCN
63.63
2.330
02.96
0.6
63:32
0.07
43.24
0.27
43.09
42.65
0.00
42.76 0.29
42.85 0.15
SGC
0..10.1..11
39.02
1 0.12
38.28
1 0:05
39:01
0.12
38.47
1.166
39.96
0.53
39.20
39.56
0.18
39.37
0.17
39.60
0.12
GPRGNN
71.70++0.31
71663+004 0.46
60.83
+ 0.42
69
0.44
71.76
0.07
44.13
0:38
44.06+00.6
43.95
0.39
44.17
+ 0.52
44.24 + 0.17
MisHop
77.99 0:14
76.05 0311
74,10
0.13
79.01
74.04
0:13
43.83
0.32
41.25 0.36
41.64
0.25
42.29 0.07
Dansees
Mineswoaper (llttrphily)
Tolokers (llttrophily)
Madel
Gaid
Randon
C2F
GNN-DoT
Grid
Random
Cossse
C2F
GNN-Dill
MLP
$0,00
0.00
79.97 L 0.08
79.94 1 0.12
79:22
0.09
$0,00
0.00
78:22
0.04
78,18 0.03
78.16 1 0.01
18.17 1 0.02
78.24 1 0.85
GCN
50.35 0.10
80226 + 0.12
30.19 +
80.26
0.14
50.28
0.07
78.65
0:07
78.67
0.00
7863 0.10
78.67 + 0.05
7833 + 0.88
SAGE
55.63
85.56
+
0.30
85.78
0.32
78.25
0.20
78:34 0.12
73.47
75.51 0.86
APPNP
50.20 +: 0.07
80
0.13
79.96 0.08
80:01
- 0.13
$0.36
0.02
78.62
0.50
78.61
0.04
76:56 0 0.12
78.33 :: 0.04
78.33 0.03
51.96
81.34
0.37
0.06
70.50+0661
78:34 0.50
78.35 0,87
ChehhNet
86.82 2 0.29
86.64 = 0.38
54,66 4: 1.26
80.06
0.20
$6,39
1 0.27
79.43
0.50
78883
78.56 0.13
78.68 1: 0.14
79.66 = 0.19
HEGCN
$3.59 1.00
83.31 0.33
0:99
83.61
83.68
0.32
78.53
0.52
0..5
78.37 0.25
73.51 1 0.09
78.30 0.07
SGC
$1.51 1: 0.11
81
0.17
$1,46
0.20
81.55
0.22
$1.63
0.24
78.51
0.00
0..69:10.00
78.47 1 0.00
78.63 1: 0.04
78.57 1 0.03
GPRGNN
$3.92
1: 0.52
X3.92
:: 0.52
83.94
1 0.56
83.94
0.56
83.97
0
78.38
0,27
0.236
78.28
1 0.16
78.37 1: 0.22
78.28 2: 0.06
Misllop
$3.52
82.75
0.20
$3.46
0:21
78.7.0033
79.22 0.24
79,51
79.54
29

---

Published as a conference paper at ICLR 2025
1.3 EXTENSION GRAPH-LEVEL TASKS AND EXPERIMENT RESULTS
GNN-Diff for Graph Classification and Regression
Graph classification is a task where we try to predict the label of an entire graph from a collection of graphs. This is different from node classification, where predict labels for individual nodes within a graph with representations learned for each node. In graph classification, we need to create a global representation for the whole graph, which captures information from all the nodes in the graph, in order to make the final prediction. To achieve this, GNNs for graph classification usually construct node-level representations first, then apply graph pooling layer to integrate all node information (Liu et al., 2022). The graph pooling process can be simple as taking the sum or averuge of all node representations, or employing more advanced methods such differentiable pooling (Ying al.) 2018) and top-k pooling (Gao & Ji 2019). Graph regression is very similar to graph classification, with the main difference being the type of loss function and evaluation metrics.
Graph Classiilcation Graph Regression
X
Pooling
Aa
Graph
Figure 13:
GFE architecture for graph classification and graph regression. The graph condition is designed to incorporate both data and graph structural information for GNN parameter generation.
The extension of GNN-Diff to graph classification and regression involves the change of GFE (see Figure [3). The input of GFE is of graphs. We denote the feature matrix and adjacency matrix of each graph as X; and A; for = 1, 2,,,,., I. In the GFE encoder, each graph is processed by a graph convolutional layer, followed by mean pooling layer
7 = G-Ennoderr X,, A.) PONN, A.)) R
Note that the node dimension aggregated into by the pooling layer. In the following experiments, we apply a single GCN layer for the graph convolution. The GFE decoder is single linear layer. Moreoever, the loss function needs to adjusted based on the task. Normally, we may choose the cross entropy loss for graph classification, and the mean squared error loss for graph regression. Finally, to construct the graph condition, we take the mean of the global representations all input graphs as cg = E T ERP: Pr.
Experiments Results of Graph Classification
Table 19: Fake news datasets for graph classification experiments. Each dataset contain multiple graphs. We provide statistics of the number of graphs, all nodes, average nodes, edges, features, and node classes. "#" means "the number of".
Dataset
#Graphs
#Nodes (all)
#Nodes (avg.)
#Edges
#FFcatures
#Classes
Politifact
314
41,054
40,740
300
2
Gossipcop
5464
314,262
58
308,798
300
2
To evaluate the effectiveness of GNN-Diff on graph-level tasks, we conduct graph classification experiments for fake news detection using two datasets: Politifact and Gossipcop (Shu et al.) 2020; Dou et al.) 2021). Politiface contains political news, while Gossipcop focuses on entertainment
Both datasets consist of tree-structured graphs derived from Twitter, where the task is to classify each graph as either fake or real news. Node features are derived from embeddings of Twitter users' historical tweets. More details of the datasets can be found in Figure T9
30

---

Published as a conference paper at ICLR 2025
Table 20: Experiment results of graph classification. We report the average accuracy (%) and the standard deviation of models selected by validation over 10 training or generation runs. The best results are marked in bold.
Folitifact
Models
Grid
Random
Coarse
C2F
GNN-Diff
GCN
80.14
1,94
79.23 + 1.95
78.42 1.87
77.19
1,22
80.45 1: 1.12
SAGE
76.24 3.47
76.60 2.71
77.10 0.91
77.60
2.85
78.28 1.71
Gossipcop
Models
Grid
GNN DiIT
GCN
97.14
0.08
96.85 + 0.16
96.58 0.23
97.06
0.09
97.62 : 0.09
SAGE
96.73 0.24
96.72 1
96.59 i 0.14
96.84 0.14
96.73 1: 0.07
We use two target models, GCN and SAGE, and follow the data split and model implementation from the PyTorch_Geomet tric example The hyperparameter search space follows:
optimizer: ('SGD', 'Adam']
learning_rate: [0.005, 0.01, 0.05, 0.1, 0.5]
weight_decay. [0.0005, 0.005]
dropout: [0.0, 0.1, 0.3]
size [16, 32, 64]
ZAGE_hidden_siie [16, 32, 64].
The experimental results in Table 20 demonstrate that GNN-Diff consistently surpasses the baseline methods across both datasets Politifact and Gossipcop-in in the graph classification task for fake news detection. Specifically, for the Politifaee dataset, GNN-Diff achieves the highest accuracy for both GCN and SAGE. Similarly, the Gossi poop dataset, GNN-Diff maintains its superiority with GCN and SAGE, matching or exceeding the performance of other baselines while achieving the lowest variance. These results underscore GNN-Diff's ' robustness and effectiveness in addressing the graph classification challenges in the context of fake news detection, particularly in achieving high accuracy with consistent reliability.
httpp:://github.com http://hhhhhhhaggrgc.//////cc//h
upf
31

---

Published as a conference paper at ICLR 2025
1.4 BAYESIAN OPTIMIZATION VS. GNN-DIFF
In our main experiments, we compare GNN-Diff with some traditional search methods, including coarse search, random search and grid search, and a more advanced method, coarse-to-fine (C2F) search, to efficiently reduce the search space based on the coarse search results. Here we consider another advanced hyperparameter tuning method, Bayesian optimization, which is less commonly used in GNN-related works but a very useful tool in the general machine learning field (Snock et al., 2012, 2015). We implement the Bayesian search with Optuna (Akiba et al.] 2019). Optuna is a hyperparameter tuning framework that by default adopts Tree-structured Parzen Estimators (TPEs), probabilistic model-based optimization method that falls under the umbrella of Bayesian optimization. We compare the Bayesian search and GNN-Diff with basic node classification tasks on 6 homophilic and 6 heterophilic graphs (see Table 21). The Bayesian search space is almost the same as in Section except that we employ continuous range for the learning rate, weight decay, dropout and also the alphas in APPNP and GPRGNN to better suit the Bayesian hyperparameter tuning framework.
Table 21:
Comparison between Bayesian search and GNN-Diff. The average test accuracy (%) and the corresponding standard deviation are reported. The best results are marked in bold.
Basic Node Classffication- Homophille Graplis
Datssets
Cora
Citeseer
Pubned
Models
Bayosian
GNN-Diif
Royosian
GNN-TTif
Rayesian
GNN--TIIT
MLP
58.73 2 0.78
59.47 1: 0.43
57.32 1: 1.48
58.72 1 0.84
72.19 a: 1.98
74.73 1: 0.10
81.09
0.37
82.33
0.17
70.34
72.37
79.24
79.30 0.31
SAGR
80.43+0.34
80.60 - 0.15
70.40
0.46
70.45
+ 0.34
76,87
0.36
77.65
0.11
APPNP
78.99 1.06
82.51
0.29
68.60
1.15
71.44 1: 0.17
76.29
0.90
79.28 0.20
GAT
81,46
1.25
81.69
1 0.10
70.24
0.39
71.50
1 0.09
76.95
0.61
78.40 0,03
ChebNet
80.96 0.76
82.05 = 0.55
09:97
8: 0.58
71.68 :: 0.27
77.35 = 0.80
78.31 0.58
H2GON
81.37
68.12
71.78
0.25
79,07
79.76 0.47
SGL
0.24 81.94
82.10 =: 0.24
71.52
4: 2.71
72.10
:: 0.18
78,30 = 0.27
79.10 =: 0.19
GPRGNN
81.51
81.79
0.20
71.12
71,86
0.39
79.52
79.69
0.08
MisHop
78.42
1: 0.93
80.32 : 0.71
69
0.72
71.50
:: 0.49
76.71
0.25
78.03 0.21
Dutasets
Computers
Photo
C8
Models
Bayesian
GNN Diff
Bayesian
GNN-Dii
Buyesian
GNN
65.82
67.57 0.16
$1.69
0.02
87.63
0.28
87.63
0.01
GCN
82.24
0.91
82,39 : 0.29
90:47
0.69
91.41 : 0.01
90.50
0.12
91.35
0.02
SAGE
79.45
1.30
79.80
1 0.22
90.33
0.80
90.36 1: 0.19
90.95
0.31
91.01
0.40
APENP
83.01
83,93 :: 0.58
90.66
90.99
0.50
91.61
92.30
0.02
GAT
1.02 82.00
83.16 - 0.44
90.54
1.46
91.01 + 0.48
90.13 = 0.35
90.57 0.08
CheboNt
77.37
85.08 0.13
91.90
0.04
[12GCN
77.58
1.27
77.81 1 0.92
87.51
2.60
90.83
1 0.44
91.12
0.74
92.24 1 0.04
SGC
82.24
0.58
0.17
0.09
GPRENN
78.97
2.25
82,80
1 0.30
90.50
0.30
91.26
1 0.28
90.74
0.42
90.93
0.14
MixHop
73.14
75.64
1: 0.13
$7.77
86.12 0.60
92.19
92.23 0.05
Basic Node Classification Heterophilie Graphs
Duiasets
Actor
Wisconsin
Roman- Ronan-Bapi.re
Modell
Bayesian
GNN Diff
Doyesion
GNN Dif!
Bayesian
GNN Duff
MLP
37.85
0.63
37.89 =: 0.33
50.16 + 0.78
50,39 + 1.07
66.01 = 0.14
66.11 = 0.08
GCN
30.45
0.77
31.24 0.26
56.47
61.17
0.78
50.41
0.44
48.43 0.17
SAGE
36.00 0.42
36.11 1 0.99
75.49
3:95
76.47
1 2.32
77,42
1 0.37
77.65 1 0.12
APPNP
35.08 0.588
35.02
0.21
83.14
81.16
1.53
$7.33
58.36 0.34
GAT
30.30 1.20
29.88
1 0.23
52.75
1 2.39
52.94 1 0.05
55.29
1 1.16
55.40 0.04
ChebNee
37,06 0.80
37.49
0.16
$1.56
50.39
2,53
79.33
$0.06
H2GCN
33.63
34,16 : 0.21
78. 24
0.16
83.72 + 0.90
63.70
1.18
63.94 0.07
SGC
30.14 0.09
30.38
0.17
60.32
1.19
36.96
0.15
38,47
0.16
GPRGNN
33.39
2:94
36.84
1 0.41
80.31
0.84
81.53
1 0.96
70.92
0.25
71.76
0.07
Mixllop
37.32 1 0.57
38.15 11 0.38
$1.18 2.51
30.39 1. 0.07
0.15 76.68
74.94 0.13
Datssets
Ratingg
Mi Minesweagex
Tolokers
Madels
Bayesian
GNN-Doff
Boyesiin
GNN-DiI
Boyesian
GNN-IXff
40.41 0.59
$0.00
0.00
7H.24 0.05
GCN
42.91 0.40
43.19 :1 0.14
50.19 1: 0.10
90.28 1 0.07
78.54 -: 0.15
78.73 1 0.08
SAGE
43.28
43.32 = 0.06
85.
85.78
78.64
78.51 0.06
APENP
1..1
39.26 : 0.16
50.04
à: 0.02
90.26 1 0.02
75.42 -: 0.03
78.63 1: 0.03
44.39
44.77 = 0.04
$1.75
0.06
79,48
79,80 0.12
ChelNar
4.54
44.04 - 0.18
96.84
0.23
86,39 0.27
79.10 0.33
79.66 0.19
HEGGN
12.80
42.83 0.15
83.32
0.15 90.26
83.68
0.32
78.45
0.44
78.50 0.07
SGC
38.89 0.17
39.60 1 0.12
$1.63
0.24
78.28
0.02
78.57 0.03
GPRGNN
43.79 0.73
44.24 = 0.17
0.17 83.96
83.97
:: 0.03
78.26 = 0.11
78.28 0.06
MisHop
0.39
42.29
0J07
$4.28
$3.92
0.12
79.34 0.14
79.54 0.41
32

---

Published as a conference paper at ICLR 2025
I.5 ANALYSIS THE GRAPH FEATURE ENCODER (GFE)
Ablation Study of GFE Components
The ablation study of GFE components aims analyze the effectiveness of each component in Equation 2 We use 2 target GNNs, GCN and SAGE, and datasets, Cora and Actor. The results are shown in Table 14 "GCN2 GCNI MLP" refers the current GFE architecture, where the three components are combined via concatenation. The same implementation applies to other architectures as well. We also include "None" for comparison, which equivalent with p-diff, the unconditional baseline we have discussed in Table 5 Since the target GNNs only have 2 layers by our experiment design, we only consider up to 2-layer GCN (GCN2) in this analysis.
We observe that for different models on different datasets, specific component may contribute to promising results more than other components. For example, for GCN on Cora, the average accuracy tends to be higher when GCN2 is included in the GFE. Similarly, GCNI is shown to be more important for SAGE on Cora, while MLP may lead to better accuracy on the heterophilic graph, Actor, for both GCN and SAGE. In addition, the current GFE architecture with components generally produces the best average results among all other architectures in the ablation study. We suppose this is because the current architecture incorporates all factors that may lead to promising generation outcomes and enables the automatic selection of these components during the learning process.
Furthermore, a well-designed GFE architecture, such as "GCN2 & GCNI & MLP", enhances generation quality by improving both the average accuracy and the stability (as reflected in the standard deviation) compared to the unconditional generation model (" ("None").
GCN
Actor
Moan Accursiy
31.0
20.5
30.0
Mean
GO1 ccm H
GC12 HLP
MUP
OCA2 com
Gonz
MiP
Nore
Goa con NUP
GC12 MLP
GCH MRP
betz & oc1
GoH2
GCRI
HIBF
Nons
SAGE Cora
SAGE Actor
Wean Accuracy Noa
36.5
Accuracy
80.5
36.0
65.5
Mean Std Dey
Mean = Stdd Dav
25.0
Got2 GOH HLP
OCN2 MLP
GOH HLP
GoG con
odt2
coll
MLP
Got con HLP
cor MLP
oct2 Gon
ccRl
MiP
Nome
Go1
Figure 14: Results of GFE ablation study. The blue dots represent the average accuracy while the blue lines show the corresponding standard deviation. It can be observed that specific component may lead to comparably better results than other components.
33

---

Published as a conference paper at ICLR 2025
GFE with Various Graph Convolutions
In the previous discussion, we concentrated on the GFE architecture utilizing the GCN graph convolution. Here extend the analysis by exploring alternative graph convolutions the backbone of the GFE. Specifically, we investigate whether employing the graph convolution the target models within the GFE can yield improved prediction outcomes. For instance, in the of GAT, we compare the GFE described in Equation 2 with GFEs constructed using the GAT convolution.
We consider 3 target models, SAGE, GAT, and APPNP, and datasets, Cora and Actor. For each target model, except for the GFE adopted by GNN-Diff ("GCN2 & GCNI & MLP"), we also try GFE with 2 layers of the target model (e.g., GAT), and the concatenation with MLP (e.g.. GAT & MLP). In addition, we include MLP and no graph condition as baselines for comparison.
The results in Figure indicate that employing the target GNN convolution GFE can lead to moderately higher accuracy compared to GCN-based GFE models (e.g.. SAGE and GAT on Cora). However, this advantage is not consistently significant and may occasionally underperform (e.g.. GAT on Act or and APPNP on Cora). Therefore, we conclude that the current GFE architecture utilized by GNN-Diff serves as a reasonable default for tuning any GNN, Nevertheless, exploring alternative graph convolutions is advisable if computational resources and time permit.
Additional noteworthy findings include: (1) concatenating with an MLP proves be versatile and effective approach that can be integrated with many graph convolutions; and (2) consistent with observations from the ablation study, parameters generated using graph conditions exhibit significantly higher stability compared to those generated without any conditions. These results further validate the efficacy of the GFE in generating parameters for GNNs.
SAGE Cora
36.50
00:50
80:25
00.00
35.00
33.75
Meen
Meen Std Dev
G232 GCNI HEP
SAGE
HEP
0C32 GON1 MLP
SAGE HALP
MFP
Hore
GAT
Actar
12.5
Woan Accuracy (h)
30.0
82.0
01.5
01.0
29.6
80.5
Mean Stdl Dey
Mean # Stal Dev
GO12 GCN HLP
Gat MLP
16.F
Nome
Gat HLP
None
Goiz OCHS
APPNP Core
APPNP Actor
35.25
Rein Aecurety
36.00
78
75-
25.50
74
35.25
GON HDP
APPRI MUP
Nyopp
MLP
Nove
0002 GcM MLP
APpWP MlLp
Arme
18P
Hove
Figure 15: Results of GFEs with various graph convolutions, The blue dots represent the average accuracy while the blue lines show the corresponding standard deviation.
34

---

Published as a conference paper at ICLR 2025
1.6 GNN-DIFF TIME ANALYSIS
We provide the details of the time costs of GNN-Diff in Figure [6 In general, the coarse search consumes most of the time taken by the entire GNN-Diff process. This the inevitable cost that shall expect for GNN-Diff to generate high-performing GNNs, though the coarse search is much more efficient compared to the baseline search methods. The parameter collection time almost negligible for small graphs such as Cora and Actor. In contrast, it takes much longer to collect parameters for large-scale tasks, such as the node classification on Dascalvoc-S3 to the long training and validation time associated with large graphs. Similarly, the proportion of sampling time lower for small graphs and higher for large graphs, because of the longer validation and testing time with large graphs.
This due
12.705 nodes]
Actor (1 nodest
Coarce Porunder Colletion
owneciff varinng
ChikCort internnee
OCV
54.2
APPLP
OCV
SA32
APPNP
Hickr 169, 250 modes)
PescalVoC-gP (3,03,245 nodes)
50
Thene (rens)
Thmee
Figure 16: Time analysis of GNN-Diff. The time costs of GNN-Diff include the time for the coarse search, parameter collection, training, and inference.
1.7 COMBINE GNN-DIFF WITH LARGE-SCALE TRAINING ALGORITHMS
While our experiments on large graphs utilize clustering- based training for computational efficiency, future work could explore combining GNN-Diff with more advanced large-scale training algorithms.
One promising direction is integrating our method with the reversible connections proposed by Li et al. (202T), which enable the training of very deep wide GNNs achieve exceptional performance large graphs. This could involve incorporating reversible connections into the architectures of target models, allowing us to evaluate whether partial generation remains effective for significantly deeper or wider GNNs. Additionally, combining GNN-Diff with the Self-Label -aabell-nhhaned training described in (Sun al. 202T) could improve scalability and performance on large-scale datasets. Furthermore, lazy propagation in (Xue et al. (023) offers intriguing direction, as their efficient computation could further reduce the training costs our framework on large graphs.
Given that GNN-Diff is a general tuning framework and these strategies are generic training al- gorithms, their combination could lead a scalable, efficient, and model-agnostic approach for enhancing GNN performance on large-scale and complex graph datasets.
35


Reviews:
[{"review_id": "zhvS9bh94l", "reviewer": "Reviewer_8AwD", "summary": "The paper introduces GNN-Diff, a latent diffusion model designed to enhance GNN training while minimizing the need for hyperparameter tuning. The author claim that this approach not only improves GNN performance but also demonstrates significant stability across various tasks.", "questions": "1. Could you explain the reason of the claim that concatenating different receptive fields in message passing can handle both homophilic and heterophilic graphs?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1.\tThe overall motivation is clear: improving hyperparameter search strategies can save time and enhance performance. This is an area that has not been explored too much in GNN training so far.\n\n2.\tThe presentation is clear, with well-organized results and analysis provided for better interpretability.", "weaknesses": "1.\tThe reported performance is a significant concern, as many results are noticeably lower than expected. For instance, nearly all results in Table 2 fall substantially short of those reported in existing literature and on the OGBN leaderboard: for example, I believe that GCN on OGBN-Arxiv can achieve 71.6% (as opposed to the reported 69.2%), APPNP can reache 71.8% (versus the reported 55.05%), and GraphSAGE can easily achieve around 79% on OGBN-Products, compared to the reported 75.6%. This discrepancy raises questions about whether the authors adhered to standard settings, whether the reported accuracy is reliable, and, if not, how the experimental setup was configured. Clearer details on these aspects would help clarify the validity of the reported results.\n\n2.\tThe motivation for certain components is unclear. For instance, why do the authors choose to incorporate GAE in the training process?\n\n3.\tAn ablation study illustrating the impact of the proposed techniques would strengthen the findings.\n\n4.\tThe authors use classical GNNs for downstream tasks, but employing SOTA GNNs are necessary. For example, RevGAT[1], SAGN[2] and LazyGNN[3]. Many advanced architectures perform well without extensive parameter tuning, and using these models could better showcase the contributions of this work.\n\n5.\tFigure 7 reports only running time, without any indication of memory cost. Additionally, if three different models are trained in this work, I am wondering how the runtime remains lower than that of a random search? A comparison with coarse search would also be useful.\n\n6.\tCan the proposed techniques be applied to graph classification?\n\n[1] Training Graph Neural Networks with 1000 Layers\n\n[2] Scalable and Adaptive Graph Neural Networks with Self-Label-Enhanced training\n\n[3] LazyGNN: Large-Scale Graph Neural Networks via Lazy Propagation", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PVwekd1g4s", "reviewer": "Reviewer_TruH", "summary": "In summary， this paper proposes a strategy for tuning hyper-parameters of GNNs to maximize their potential. It is achieved by collecting a set of model parameters, encoding graphs, and then learning to generate parameters for GNNs. Extensive methods have been conducted in this paper to verify the designed method.", "questions": "The upper bound of the proposed method remains unclear. It is uncertain how a well fine-tuned GCN compares to lightly fine-tuned, more expressive models.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- This paper is well-organized.\n- Extensive experiments are conducted in this paper.", "weaknesses": "1. The technical contribution is limited given the related works.\nIt seems that GNN-diff is a combination of LDM and graph techniques. The sample collection method described in Section 4.1 is trivial, and the designed encoder is also widely used in existing methods. It would be better to highlight the technical contributions in comparison with the different methods mentioned in the related work section.\n\n2. The experiments seem somewhat unrigorous and fail to adequately highlight the efficiency and effectiveness of GNN-diff. \n- Baseline selection: While numerous target GNNs and datasets are utilized, the paper employs only a few comparable baselines. It appears that methods from \"Advanced Methods for Hyperparameter Tuning\" should be incorporated to validate the designed parameter tuning strategy more effectively. \n-  Evaluations of the effectiveness of Coarse search: In Tables 1-4, the \"Coarse\" variant struggles to outperform the \"Random\" and \"Grid\" baselines. With these results, justifying the effectiveness of the coarse search approach is challenging.\n\n3. The transferability of GNN-Diff is limited due to its hyper-parameter tuning being tailored for a specific dataset, relying on parameter collection and the GAE model.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tDDnm2s0B5", "reviewer": "Reviewer_gdji", "summary": "This article introduces an interesting hyperparameter search architecture that learns graph embedding information for specific graph tasks through a designed GNN architecture. This information is then integrated into conditional latent denoising diffusion probabilistic models, using a parameter encoding module to flexibly convert the input and output dimensions of the parameters.", "questions": "1. There is a spelling error at line 126 of the article, \"P-Eecoder()\".\n2. Why use such a GNN architecture? Would using classical graph embedding frameworks like GAT or GCN be effective?\n3. Clearly, graph data structures are diverse, and using just one type of GNN makes it difficult to effectively embed all types of graph data. When GNNs become ineffective, or when increasing the number of GNN layers causes the graph embeddings to become too smooth, what impact will that have on the method? Since this paper proposes a parameter tuning method, which can be considered a fundamental approach, we need to focus more on when it will fail or the lower limits of the method’s capabilities.", "limitations": "", "rating": 6, "confidence": 1, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The overall expression of the article is clear and easy to understand.\n2. The experiments are comprehensive, and the performance of the algorithm is impressive.\n3. It connects model parameter tuning with the data itself.", "weaknesses": "Simply using GNNs for embedding learning on graph-structured data and adding it to the G-LDM module does not analyze why this part of the graph embedding works for the method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xZziwmSK8W", "reviewer": "Reviewer_ti8E", "summary": "This paper proposes GNN-Diff, a graph-conditioned latent diffusion model designed to enhance Graph Neural Networks (GNNs) by generating effective parameters with minimal hyperparameter tuning. This is achieved by utilizing a coarse search strategy that identifies a subset of viable hyperparameters, which then guides the model’s parameter generation. GNN-Diff claims to reduce tuning time and computational cost by creating high-performance GNN parameters through diffusion in a latent space conditioned on graph data. The authors validate GNN-Diff through extensive experiments on various tasks including node classification and link prediction with numerous model backbones, showing improved efficiency and stability.", "questions": "Have the authors tried other forms of G-Encoders other than Eq. (2) and conducted ablation studies about it?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The idea of using the diffusion framework to help with hyperparameter tuning seems novel and reasonable.\n- The experiments are thorough, spanning a diverse set of tasks and backbone models, with results that are both convincing and reproducible.\n- The proposed method effectively enhances both speed and performance across most cases, showcasing its practical value.", "weaknesses": "- The heterophily datasets used in this paper have faced criticism for extensive duplicate nodes as noted in [1]. It would be beneficial for the authors to consider experimenting with the datasets proposed in [1] for a more robust evaluation of heterophilic cases.\n\n- Table 5 shows that, compared to p-Diff, the improvements with GNN-Diff are relatively marginal and often inconsistent, with many results falling within one standard deviation. This brings into question the effectiveness of the GNN-Encoder design (Eq. 2).\n\n- There is a lack of theoretical analysis. \n\n\n[1] A critical look at the evaluation of GNNs under heterophily: Are we really making progress?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)