Diffusing to the Top: Boost Graph Neural Networks with Minimal Hyperparameter Tuning

## Abstract

Abstract Graph Neural Networks (GNNs) are proficient in graph representation learning and achieve promising performance on versatile tasks such as node classification and link prediction.
Usually, a comprehensive hyperparameter tuning is essential for fully unlocking GNN’s top performance, especially for complicated tasks such as node classification on large graphs and long-range graphs. This is usually associated with high computational and time costs and careful design of appropriate search spaces.
This work introduces a graph-conditioned latent diffusion framework (GNN-Diff) to generate high-performing GNNs based on the model checkpoints of sub-optimal hyperparameters selected by a light-tuning coarse search. We validate our method through 166 experiments across four graph tasks: node classification on small, large, and long-range graphs, as well as link prediction. Our experiments involve 10 classic and state-of-the-art target models and 20 publicly available datasets. The results consistently demonstrate that GNN-Diff: (1) boosts the performance of GNNs with efficient hyperparameter tuning; and (2) presents high stability and generalizability on unseen data across multiple generation runs. The code is available at https://github.com/lequanlin/GNN-Diff .

## 1 Introduction

Graph neural networks (GNNs) are deep learning models that excel in capturing relationships within graph-structured data, making them highly effective for a variety of graph tasks such as node classification and link prediction ; . Although GNNs achieve good performance on simple graph tasks without sophisticated hyperparameter tuning , fully unlocking their potential to achieve top performance or handle complicated graph tasks still requires extensive hyperparameter tuning ; . This is often associated with high computational and time costs and careful design of appropriate search spaces. For example, in Figure 1, we show that the average test accuracy of GCN for node classification typically increases with the hyperparameter search space. Nevertheless, it may cost over $20\times$ of tuning time to gain an accuracy increase of less than $1\%$. This motivates us to develop a method that boosts the performance of GNNs while minimizing search space and tuning time. Furthermore, this method will be particularly beneficial for complex tasks such as node classification in large and long-range graphs ; , where hyperparameter tuning often plays a critical role.

In this paper, we propose a graph-conditioned latent diffusion framework (GNN-Diff) to generate high-performing GNN parameters with sub-optimal hyperparameters selected by a coarse search, which is a random search with a relevantly small search space (Figure 2).
Our method assumes that the performance of GNNs is primarily determined by the quality of the learned parameters. In addition, high-quality parameters exist in the underlying parameter population produced by a sub-optimal but sufficiently good configuration, which can be found by the coarse search. Based on our experiments, the coarse search space only needs to be $10\%$ of the grid search space for most graph tasks to find a promising configuration. The parameter generation is fulfilled by a latent denoising diffusion probabilistic model ; . While parameter generation have been extensively studied in previous research ; ; ; ; , the intrinsic relationship between data characteristics and network parameters remains underexplored. To fill this gap, we adopt a task-oriented graph autoencoder to integrate data and graph structural information as a condition for GNN parameter generation. Lastly, it is noteworthy that GNN-Diff does not require additional hyperparameter tuning for parameter generation, reducing tuning costs to only those incurred by the coarse search.

Results from 166 experiments across 4 graph tasks, 10 target models, and 20 datasets consistently show that GNN-Diff: (1) efficiently boosts GNN performance with minimal hyperparameter tuning; and (2) demonstrates high stability and generalizability over multiple generation runs on unseen data.

Figure: Figure 1: Effect of hyperparameter search space on GCN performance for node classification. The test accuracy is averaged over 10 search runs. We also show the results of our method, GNN-Diff.
Refer to caption: extracted/5909359/Figures/Intro_1_new.png

Figure: Figure 2: GNNs with parameters found by (a) grid search with the full search space, (b) random search with the sub-search space, (c) coarse search with the minimal search space, and (d) GNN-Diff generation based on sub-optimal hyperparameters selected by the coarse search. Top performance may be achieved by a large search space at a cost of time, or generated with GNN-Diff efficiently.
Refer to caption: extracted/5909359/Figures/Intro2.png

## 2 Preliminaries

### 2.1 Graph Neural Networks

Graph Neural Networks (GNNs) are deep-learning architectures designed for graph data. Typically, there are two types of GNNs: spatial GNNs that aggregate neighboring nodes with message passing ; ; and spectral GNNs developed from the graph spectral theory ; ; . The key component of GNNs is the convolutional layer, which usually consists of graph convolution, linear transformation, and non-linear activation function.

We denote an undirected graph with $N$ nodes as $\mathcal{G}\{\mathcal{V},\mathcal{E},\mathbf{A}\}$, where ${\mathcal{V}}$ and ${\mathcal{E}}$ are the node set and the edge set, respectively, and $\mathbf{A}\in{\mathbb{R}}^{N\times N}$ is the adjacency matrix containing information of relationships. $\mathbf{A}$ can be either weighted or unweighted, normalized or unnormalized. The graph signals are stored in a matrix ${\mathbf{X}}\in{\mathbb{R}}^{N\times D_{f}}$, where $D_{f}$ is the number of features.
Then, the convolutional layer in GCN can be written as $\sigma(\mathbf{A}{\mathbf{X}}{\mathbf{W}})$, involving graph convolution by node aggregation with the adjacency $\mathbf{A}$, a feature linear transformation operator ${\mathbf{W}}$, and the activation $\sigma$. Usually, learnable parameters of GNNs exist in the linear operator and sometimes in the parameterized graph convolution as well ; .

### 2.2 Latent Diffusion Models

Diffusion models generate a step-by-step denoising process that recovers data in the target distribution from random white noises. A typical diffusion model adopts a pair of forward-backward Markov chains, where the forward chain perturbs the observed samples eventually to white noises, and then the backward chain learns how to remove the noises and recover the original data. Let ${\mathbf{w}}_{0}\sim q_{{\mathbf{w}}}({\mathbf{w}}_{0})$ be the original data (e.g., vectorized network parameters) from the target distribution $q_{{\mathbf{w}}}({\mathbf{w}}_{0})$. Then, the forward-backward chains are formulated as follows.

Forward chain. For diffusion steps $t=0,1,...,T$, Gaussian noises ${\mathbf{\epsilon}}\sim{\mathcal{N}}(\textbf{0},{\mathbf{I}})$ are injected to ${\mathbf{w}}_{t}$ until $q_{{\mathbf{w}}}({\mathbf{w}}_{T})\coloneqq\int q_{{\mathbf{w}}}({\mathbf{w}}_
{T}|{\mathbf{w}}_{0})q_{{\mathbf{w}}}({\mathbf{w}}_{0})d{\mathbf{w}}_{0}
\approx{\mathcal{N}}({\mathbf{w}}_{T};{\mathbf{0}},{\mathbf{I}})$. By the Markov property, one may jump to any diffusion steps via ${\mathbf{w}}_{t}=\sqrt{\widetilde{\alpha}_{t}}{\mathbf{w}}_{0}+\sqrt{1-
\widetilde{\alpha}_{t}}{\mathbf{\epsilon}}$, where $\widetilde{\alpha}_{t}=\prod_{i=1}^{t}(1-\beta_{i})$ with $\beta=\{\beta_{1},\beta_{2},...,\beta_{T}\}$ is a pre-defined noise schedule.

Backward chain. The backward chain removes noises from ${\mathbf{w}}_{T}$ gradually via a backward transition kernel $q_{{\mathbf{w}}}({\mathbf{w}}_{t-1}|{\mathbf{w}}_{t})$, which is usually approximated by a neural network $p_{\mathbf{\theta}}({\mathbf{w}}_{t-1}|{\mathbf{w}}_{t})$ with learnable parameters ${\mathbf{\theta}}$. In this paper, we adopt the framework of denoising diffusion probabilistic models (DDPMs), which alternatively uses a denoising network $\epsilon_{\mathbf{\theta}}$ to predict the noise injected at each diffusion step with the loss function

$$ ${\mathcal{L}}_{\text{DDPM}}={\mathbb{E}}_{t,{\mathbf{w}}_{0}\sim q_{{\mathbf{w }}}({\mathbf{w}}_{0}),{\mathbf{\epsilon}}\sim{\mathcal{N}}({\mathbf{0}},{ \mathbf{I}})}\left\|{\mathbf{\epsilon}}-{\mathbf{\epsilon}}_{\mathbf{\theta}} \left(\sqrt{\widetilde{\alpha}_{t}}{\mathbf{w}}_{0}+\sqrt{1-\widetilde{\alpha} _{t}}{\mathbf{\epsilon}},t\right)\right\|^{2}.$ (1) $$

### 2.3 Latent Diffusion for Parameter Generation

Generating GNN parameters directly with diffusion models may lead to slow training and long inference time, especially when the model is heavily-parameterized. As a common solution to such problems, latent diffusion models (LDM) ; ; first learn a parameter autoencoder (PAE) to convert the target parameters ${\mathbf{w}}_{0}\in{\mathbb{R}}^{D_{w}}$ to a low-dimensional latent space ${\mathbf{z}}_{0}=\text{P-Eecoder}({\mathbf{w}}_{0})\in{\mathbb{R}}^{D_{p}}$ with $D_{p}\ll D_{w}$, then generate in the latent space before reconstructing the original target with a sufficiently precise decoder. The loss function of PAE is usually formulated as the mean squared error (MSE) between original parameters ${\mathbf{w}}_{0}$ and reconstructed parameters $\text{P-Decoder}({\mathbf{z}}_{0})$.

## 3 Related Works

This is a summary of related works, and we discuss more details in Appendix A. The related works cover three topics: network parameter generation, GNN training, and advanced methods for hyperparameter tuning. In network parameter generation, we start with hypernetwork, an early concept of predicting parameters of the target network ; , followed by a review on generative models for parameter generation ; ; ; ; . p-diff is the most relevant method to our approach, which collects checkpoints from the training process of the target network and adopts an unconditional latent diffusion model to generate high-performing parameters. Next, in GNN training, we discuss two major approaches to boost GNN performance, including pre-training ; ; and architecture search ; . However, both approaches inevitably rely on hyperparameter tuning to succeed. Lastly, we consider some advanced methods of hyperparameter tuning, such as Bayesian optimization ; and coarse-to-fine search ; .

## 4 Graph Neural Network Diffusion (GNN-Diff)

GNN-Diff is a graph-conditioned diffusion framework that generates high-performing parameters for a target GNN to match or even surpass the results of time-consuming hyperparameter tuning with a much more efficient process.
This is achieved via a pipeline of four steps: (1) input graph data, (2) parameter collection of the target GNN with a light-tuning coarse search, (3) training, and (4) inference and prediction. GNN-Diff has three trainable components: a parameter autoencoder (PAE), a graph autoencoder (GAE), and a graph-conditioned latent diffusion model (G-LDM). We provide the overview of the GNN-Diff pipeline in Figure 3. The input data and PAE have been previously discussed in Section 2, thus the rest of this section focuses on the remaining details in the GNN-Diff pipeline. The pseudo codes of training and inference algorithms are provided in Appendix B.

Figure: Figure 3: How GNN-Diff works for node classification. (1) Input graph data: input graph signals, adjacency matrix, and train/validation ground truth labels. (2) Parameter collection: we use a coarse search with a small search space to select an appropriate hyperparameter configuration for the target GNN, and then collect model checkpoints with this configuration. (3) Training: PAE and GAE are firstly trained to produce latent parameters and the graph condition, and then G-LDM learns how to recover latent parameters from white noises with the graph condition as a guidance. (4) Inference: after sampling latent parameters from G-LDM, GNN parameters are reconstructed with the PAE decoder and returned to the target GNN for prediction.
Refer to caption: extracted/5909359/Figures/Method_v1.png

### 4.1 Parameter Collection with Coarse Search

To produce high-performing GNN parameters, we need a set of good parameter samples such that the diffusion model can generate from their underlying population distribution. Intuitively, the quality of parameter samples relies on the hyperparameter selection. Hence, we apply a coarse search with a relevantly small search space to determine the suitable configuration. Two types of hyperparameters are considered: training-related such as learning rate and weight decay, and model-related such as hidden size and the teleport probability $\alpha$ in APPNP . We train the target model with multiple runs for each configuration. Then, we select the configuration with the best validation results on average. After the coarse search, model checkpoints trained with the selected configuration are saved as parameter samples. Parameter samples are reorganized and vectorized before they are further processed by the PAE.

### 4.2 Graph Autoencoder (GAE)

The GAE is designed to encode graph information in graph signals ${\mathbf{X}}$ and structure $\mathbf{A}$, which will be employed as a condition for GNN parameter generation. Our experiments involve node classification and link prediction tasks, thus the GAE encoder aims at producing a graph representation that works well on the specific task. Straightforwardly, we consider adopting a GNN-based architecture for the encoder to capture and process both data and graph information. This architecture is determined by the nature of each task. For example, multiple GNN layers may be well-suited for node classification on homophilic graphs, where connected nodes are normally from the same classes, but inappropriate for heterophilic graphs, where nodes with different labels are prone to be linked ; . Hence, enlightened by some previous works ; ; , we introduce a structure that is capable of handling node classification on both homophilic and heterophilic graphs. The GAE encoder (without activation) can be denoted as

$$ $\mathbf{\eta}=\text{G-Encoder}({\mathbf{X}},\mathbf{A})=\text{Concat}\left( \mathbf{A}^{2}{\mathbf{X}}{\mathbf{W}}_{1},\mathbf{A}{\mathbf{X}}{\mathbf{W}}_ {2},{\mathbf{X}}{\mathbf{W}}_{3}\right){\mathbf{W}}_{4}\in{\mathbb{R}}^{N \times D_{p}},$ (2) $$

which is composed of the concatenation of a two-layer GCN, a one-layer GCN, and an MLP, followed by a linear transformation ${\mathbf{W}}_{4}\in{\mathbb{R}}^{D_{\mathcal{G}}\times D_{p}}$, where $D_{\mathcal{G}}$ and $D_{p}$ are the dimensions of the concatenation and the latent parameter representation, respectively. Next, the GAE decoder for general node classification is a single linear layer $\text{G-Decoder}(\mathbf{\eta})=\mathbf{\eta}{\mathbf{W}}_{5}\in{\mathbb{R}}^{
D_{c}}$, with $D_{c}$ being the number of node classes. This ensures that graph information is fully encapsulated in the output of the encoder. Finally, the loss function is the cross entropy on node labels. We use this GAE for node classification on small homophilic and heterophilic graphs. The GAEs for node classification on large and long-range graphs and link prediction are discussed in Appendix C.1.

The graph condition is constructed via mean-pooling the nodes as

$$ ${\mathbf{c}}_{\mathcal{G}}=\frac{1}{N}\sum_{i=1}^{N}\mathbf{\eta}(i)\in{ \mathbb{R}}^{D_{p}},$ (3) $$

where $\mathbf{\eta}(i)$ is the i-th row of graph representation. The purpose of applying mean pooling on nodes is to construct an overall graph condition regardless of the graph size, which will later be combined with latent representation ${\mathbf{z}}_{0}\in{\mathbb{R}}^{D_{p}}$ as input of diffusion denoising network.

### 4.3 Graph-conditioned Latent DDPM (G-LDM)

G-LDM is developed from LDM with the graph condition as a consolidated guidance on latent GNN parameter generation. The forward pass of G-LDM is the same as traditional DDPM, while the backward pass uses a graph-conditioned backward kernel to recover data from white noises. Given the latent parameters ${\mathbf{z}}_{0}$ from the PAE encoder and the graph condition ${\mathbf{c}}_{\mathcal{G}}$ from the GAE encoder, the corresponding G-LDM loss function is written as

$$ ${\mathcal{L}}_{\text{G-LDM}}={\mathbb{E}}_{t,{\mathbf{z}}_{0}\sim q_{{\mathbf{ z}}}({\mathbf{z}}_{0}),{\mathbf{\epsilon}}\sim{\mathcal{N}}({\mathbf{0}},{ \mathbf{I}})}\left\|{\mathbf{\epsilon}}-{\mathbf{\epsilon}}_{\mathbf{\theta}} \left(\sqrt{\widetilde{\alpha}_{t}}{\mathbf{z}}_{0}+\sqrt{1-\widetilde{\alpha} _{t}}{\mathbf{\epsilon}},t,{\mathbf{c}}_{{\mathcal{G}}}\right)\right\|^{2}.$ (4) $$

For both PAE and denoising network ${\mathbf{\epsilon}}_{\mathbf{\theta}}$, we adopt the Conv1d-based architecture in . We include the graph condition by taking the sum of ${\mathbf{z}}_{t}$, ${\mathbf{c}}_{\mathcal{G}}$, and the time embedding as input to ${\mathbf{\epsilon}}_{\mathbf{\theta}}$. We observe in experiments that the most important factor in high-quality generation is to reduce parameter dimension sufficiently while ensuring reconstruction tightness. This can be easily done by setting the appropriate kernel size and stride for Conv1d layers in PAE based on the parameter size, which does not require hyperparameter tuning. We discuss more details about the PAE and ${\mathbf{\epsilon}}_{\mathbf{\theta}}$ architectures and how to ensure high-quality generation in Appendices C.2 and C.3.

### 4.4 Inference and Prediction

The last step is to obtain parameters and return them to the target GNN for prediction. G-LDM generates samples of latent parameters with the learned ${\mathbf{\epsilon}}_{\mathbf{\theta}}$. Then, the learned PAE decoder is applied to reconstruct parameters back to the original parameter space. These parameters are returned to the target GNN, thus we have a group of GNNs with generated parameters. Lastly, the model with the best validation performance will be used for final evaluation.

## 5 Experiments and Discussions

### 5.1 Experimental Setup

Tasks and metrics.
We evaluate our method with 4 graph tasks: (1) basic node classification on small homophilic and heterophilic graphs; (2) node classification on large graphs with a huge amount of nodes; (3) node classification on long-range graphs with very deep GNNs; (4) link existence prediction on undirected graphs. We use accuracy as the metric for all tasks except for the node classification on long-range graphs. Since node classes are highly imbalanced in long-range graphs, we employ macro F1-score as the metric. We will report 100$\times$ F1-scores for presentation purpose.

Baseline methods. We compare GNN-Diff with (1) grid search with full search space; (2) random search with 20% of full search space; (3) coarse search with 10% of full search space; (4) coarse-to-fine (C2F) approach that narrows down search space based on the result of coarse search to optimize hyperparameters.

Datasets. We conduct the experiments on 20 publicly available datasets. For basic node classification, we consider 6 homophilic graphs: Cora, Citeseer, and Pubmed , as well as Computers, Photo, and CS ; and 6 heterophilic graphs: Actor and Wisconsin , along with Roman-Empire, Amazon-Ratings, Minesweeper, and Tolokers . We also test on large graphs with 89,250 to 2,339,029 nodes, including Flickr , Reddit , and OGB-arXiv and OGB-Products . For long-range graphs, we use PascalVOC-SP and COCO-SP . Additionally, our link prediction experiments involve 4 datasets: Cora and Citeseer , as well as Chameleon and Squirrel .

Target models. 9 classic and state-of-the-art GNNs are selected as target models: GCN , SAGE , APPNP , GAT , ChebNet , H2GCN , SGC , GRPGNN , and MixHop . We also include multilayer perceptron (MLP) to represent baseline performance.

Experiment details.
All experiments are run on NVIDIA 4090 GPUs with 24GB memory.
GNN-Diff is trained with a consecutive training flow of its three components (200 epochs for GAE, 9000 epochs for PAE and 6000 epochs for G-LDM). We may generate either all or part of the target model’s parameters. The partial generation focuses on the last learnable layer, because earlier layers typically perform the role of feature extraction and representation learning, while the last layer is more task-specific and crucial for the final classification . For the sake of efficiency, GNN-Diff performs partial generation in all experiments.
To collect partial parameters, we fix the parameters in all layers except the last layer after 190 epochs, save the checkpoint with the best validation result, and further train it with a small learning rate $5e$-4 in another 10 epochs for 10 runs. The parameters collected in each run are different due to randomness caused by dropout, but they are assumed to be from the same population, as they are obtained by slightly altering the saved checkpoint. In evaluation, we generate 100 samples and test the sample with the best validation performance. For the baseline search methods, we first train the target model with each configuration for 200 epochs over 10 runs (3 runs for some large and long-range graphs in consideration of time), then select the configuration with the best average validation performance. To ensure a fair comparison with GNN-Diff, which fine-tunes the last layer for parameter collection, we also train the target model for 190 epochs and fine-tune the last layer for 10 more epochs using the best configuration from the baseline search. We then report the better test accuracy from either the full 200-epoch training or the fine-tuning scheme.

Reproducibility and comparability. The random seed was set as 42 to mitigate randomness in experiments. Our results may be different from some related works such as ; due to different data split and model architecture. For example, we only use the first split of Actor and Wisconsin because all the experiments are conducted based on one train/validation/test split. Also, the target GNNs mostly adopt simple architectures without tricks such as residual connection and layer or batch normalization.

Please refer to Appendices D, E, F, G, and H for more details and experiment settings.

### 5.2 Main Experiment Results

**Table 1: Results of basic node classification on homophilic and heterophilic graphs. We report the average test accuracy (%) and the standard deviation of models selected by validation over 10 training or generation runs. The best results are marked in bold.**
| Datasets | Cora (Homophily) | Citeseer (Homophily) |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Models | Grid | Random | Coarse | C2F | GNN-Diff | Grid | Random | Coarse | C2F | GNN-Diff |
| MLP | 59.41 $\pm$ 0.94 | 58.32 $\pm$ 1.21 | 58.28 $\pm$ 0.68 | 57.83 $\pm$ 1.58 | 59.47 $\pm$ 0.43 | 58.26 $\pm$ 1.04 | 57.63 $\pm$ 1.45 | 57.51 $\pm$ 1.30 | 57.63 $\pm$ 1.44 | 58.72 $\pm$ 0.84 |
| GCN | 82.04 $\pm$ 0.96 | 81.52 $\pm$ 0.71 | 81.89 $\pm$ 0.48 | 81.99 $\pm$ 0.95 | 82.33 $\pm$ 0.17 | 71.92 $\pm$ 1.10 | 72.04 $\pm$ 0.56 | 71.97 $\pm$ 0.67 | 72.13 $\pm$ 0.23 | 72.37 $\pm$ 0.29 |
| SAGE | 80.58 $\pm$ 1.04 | 80.49 $\pm$ 0.77 | 80.43 $\pm$ 0.78 | 80.43 $\pm$ 0.78 | 80.60 $\pm$ 0.15 | 70.56 $\pm$ 0.58 | 69.50 $\pm$ 0.61 | 68.81 $\pm$ 0.86 | 70.39 $\pm$ 0.91 | 70.45 $\pm$ 0.14 |
| APPNP | 81.91 $\pm$ 0.90 | 81.69 $\pm$ 0.62 | 81.47 $\pm$ 0.51 | 81.91 $\pm$ 0.90 | 82.51 $\pm$ 0.29 | 70.82 $\pm$ 1.40 | 69.87 $\pm$ 0.45 | 69.68 $\pm$ 0.63 | 69.60 $\pm$ 1.32 | 71.44 $\pm$ 0.17 |
| GAT | 81.10 $\pm$ 0.52 | 81.05 $\pm$ 0.82 | 80.13 $\pm$ 1.18 | 80.52 $\pm$ 1.29 | 81.69 $\pm$ 0.10 | 70.83 $\pm$ 0.58 | 70.81 $\pm$ 0.69 | 70.45 $\pm$ 1.13 | 69.82 $\pm$ 1.10 | 71.50 $\pm$ 0.09 |
| ChebNet | 81.83 $\pm$ 0.46 | 81.51 $\pm$ 0.85 | 81.30 $\pm$ 0.88 | 81.73 $\pm$ 1.09 | 82.05 $\pm$ 0.55 | 71.14 $\pm$ 0.13 | 71.24 $\pm$ 0.75 | 71.02 $\pm$ 1.12 | 71.14 $\pm$ 0.13 | 71.65 $\pm$ 0.27 |
| H2GCN | 82.05 $\pm$ 0.81 | 81.67 $\pm$ 0.71 | 81.63 $\pm$ 0.88 | 82.11 $\pm$ 0.72 | 82.17 $\pm$ 0.12 | 71.49 $\pm$ 0.89 | 71.30 $\pm$ 1.31 | 71.39 $\pm$ 0.19 | 71.12 $\pm$ 0.98 | 71.78 $\pm$ 0.25 |
| SGC | 81.89 $\pm$ 0.94 | 82.09 $\pm$ 0.55 | 81.60 $\pm$ 0.80 | 82.09 $\pm$ 0.55 | 82.10 $\pm$ 0.24 | 71.94 $\pm$ 0.56 | 71.82 $\pm$ 0.26 | 71.77 $\pm$ 0.23 | 71.70 $\pm$ 0.43 | 72.10 $\pm$ 0.18 |
| GPRGNN | 81.03 $\pm$ 0.65 | 81.34 $\pm$ 0.63 | 81.21 $\pm$ 0.64 | 81.41 $\pm$ 0.84 | 81.79 $\pm$ 0.20 | 70.93 $\pm$ 1.19 | 70.44 $\pm$ 0.71 | 70.05 $\pm$ 0.94 | 70.93 $\pm$ 1.19 | 71.86 $\pm$ 0.19 |
| MixHop | 80.08 $\pm$ 1.43 | 79.39 $\pm$ 0.77 | 78.81 $\pm$ 0.66 | 79.50 $\pm$ 0.73 | 80.32 $\pm$ 0.71 | 70.88 $\pm$ 0.94 | 70.54 $\pm$ 1.27 | 70.15 $\pm$ 0.68 | 70.81 $\pm$ 1.16 | 71.50 $\pm$ 0.49 |
| Datasets | Actor (Heterophily) | Wisconsin (Heterophily) |  |  |  |  |  |  |  |  |
| Models | Grid | Random | Coarse | C2F | GNN-Diff | Grid | Random | Coarse | C2F | GNN-Diff |
| MLP | 37.36 $\pm$ 0.71 | 37.81 $\pm$ 0.71 | 37.22 $\pm$ 0.77 | 37.91 $\pm$ 0.79 | 37.89 $\pm$ 0.33 | 78.63 $\pm$ 1.72 | 78.43 $\pm$ 2.26 | 78.43 $\pm$ 4.13 | 79.61 $\pm$ 3.09 | 80.39 $\pm$ 1.07 |
| GCN | 30.94 $\pm$ 0.46 | 30.43 $\pm$ 0.47 | 30.79 $\pm$ 0.46 | 30.79 $\pm$ 0.46 | 31.24 $\pm$ 0.26 | 59.80 $\pm$ 2.96 | 59.41 $\pm$ 2.62 | 59.02 $\pm$ 1.72 | 60.39 $\pm$ 3.40 | 61.17 $\pm$ 0.78 |
| SAGE | 35.54 $\pm$ 0.84 | 35.53 $\pm$ 0.84 | 35.35 $\pm$ 0.83 | 36.70 $\pm$ 0.81 | 36.11 $\pm$ 0.09 | 75.68 $\pm$ 4.26 | 74.90 $\pm$ 4.60 | 73.73 $\pm$ 4.15 | 73.73 $\pm$ 5.48 | 76.47 $\pm$ 2.32 |
| APPNP | 35.32 $\pm$ 0.53 | 35.09 $\pm$ 0.53 | 35.11 $\pm$ 0.75 | 35.53 $\pm$ 0.76 | 35.92 $\pm$ 0.21 | 80.98 $\pm$ 3.59 | 79.61 $\pm$ 3.60 | 79.60 $\pm$ 3.60 | 80.78 $\pm$ 3.89 | 81.16 $\pm$ 1.53 |
| GAT | 29.84 $\pm$ 0.61 | 29.79 $\pm$ 0.86 | 29.61 $\pm$ 0.73 | 29.87 $\pm$ 0.74 | 29.88 $\pm$ 0.23 | 53.92 $\pm$ 3.73 | 51.96 $\pm$ 4.26 | 52.35 $\pm$ 2.62 | 51.57 $\pm$ 4.72 | 52.94 $\pm$ 0.05 |
| ChebNet | 37.29 $\pm$ 0.75 | 36.48 $\pm$ 0.72 | 36.23 $\pm$ 0.53 | 37.39 $\pm$ 0.69 | 37.49 $\pm$ 0.16 | 80.00 $\pm$ 4.32 | 79.61 $\pm$ 2.95 | 79.61 $\pm$ 2.48 | 79.80 $\pm$ 4.03 | 80.59 $\pm$ 2.83 |
| H2GCN | 34.05 $\pm$ 0.76 | 34.05 $\pm$ 0.76 | 34.01 $\pm$ 0.59 | 33.63 $\pm$ 0.65 | 34.16 $\pm$ 0.21 | 83.14 $\pm$ 2.65 | 82.55 $\pm$ 3.39 | 80.98 $\pm$ 3.93 | 82.55 $\pm$ 3.39 | 83.72 $\pm$ 0.90 |
| SGC | 30.32 $\pm$ 0.39 | 30.32 $\pm$ 0.39 | 29.13 $\pm$ 0.32 | 28.87 $\pm$ 0.42 | 30.38 $\pm$ 0.17 | 57.25 $\pm$ 4.41 | 58.24 $\pm$ 3.34 | 57.84 $\pm$ 4.06 | 57.25 $\pm$ 4.41 | 60.32 $\pm$ 1.19 |
| GPRGNN | 36.32 $\pm$ 0.43 | 36.56 $\pm$ 1.21 | 36.26 $\pm$ 0.61 | 36.68 $\pm$ 0.57 | 36.84 $\pm$ 0.41 | 80.20 $\pm$ 3.63 | 79.41 $\pm$ 3.83 | 79.41 $\pm$ 3.49 | 80.78 $\pm$ 4.41 | 81.53 $\pm$ 0.96 |
| MixHop | 37.76 $\pm$ 1.00 | 37.57 $\pm$ 0.92 | 37.06 $\pm$ 0.59 | 37.60 $\pm$ 0.60 | 38.15 $\pm$ 0.38 | 77.45 $\pm$ 3.10 | 79.41 $\pm$ 2.31 | 79.61 $\pm$ 2.95 | 79.80 $\pm$ 3.07 | 80.39 $\pm$ 0.07 |

**Table 2: Results of node classification on large graphs. We report the average test accuracy (%) and the standard deviation of models selected by validation over 10 (Flickr and OGB-arXiv) or 3 Reddit and OGB-Products) training or generation runs. The best results are marked in bold.**
| Datasets | Flickr | Reddit |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Models | Grid | Random | Coarse | C2F | GNNDiff | Grid | Random | Coarse | C2F | GNNDiff |
| GCN | 52.54 $\pm$ 0.89 | 52.65 $\pm$ 0.99 | 52.64 $\pm$ 0.63 | 52.81 $\pm$ 0.76 | 52.84 $\pm$ 0.04 | 91.01 $\pm$ 0.09 | 91.07 $\pm$ 0.05 | 91.12 $\pm$ 0.07 | 91.12 $\pm$ 0.07 | 91.15 $\pm$ 0.04 |
| SAGE | 53.60 $\pm$ 0.39 | 53.53 $\pm$ 0.39 | 53.48 $\pm$ 0.47 | 53.63 $\pm$ 0.92 | 53.70 $\pm$ 0.02 | 93.26 $\pm$ 0.02 | 93.19 $\pm$ 0.06 | 93.28 $\pm$ 0.03 | 93.34 $\pm$ 0.06 | 93.45 $\pm$ 0.03 |
| APPNP | 52.37 $\pm$ 0.61 | 51.78 $\pm$ 0.64 | 51.92 $\pm$ 0.46 | 51.99 $\pm$ 0.21 | 52.23 $\pm$ 0.03 | 87.94 $\pm$ 0.07 | 86.44 $\pm$ 0.18 | 86.40 $\pm$ 0.12 | 87.93 $\pm$ 0.09 | 88.07 $\pm$ 0.05 |
| Datasets | OGB-arXiv | OGB-Products |  |  |  |  |  |  |  |  |
| Models | Grid | Random | Coarse | C2F | GNNDiff | Grid | Random | Coarse | C2F | GNNDiff |
| GCN | 69.22 $\pm$ 0.09 | 68.97 $\pm$ 0.05 | 68.53 $\pm$ 0.2 | 69.25 $\pm$ 0.15 | 69.38 $\pm$ 0.04 | 73.21 $\pm$ 0.19 | 73.29 $\pm$ 0.21 | 73.04 $\pm$ 0.23 | 73.94 $\pm$ 0.22 | 74.03 $\pm$ 0.16 |
| SAGE | 69.91 $\pm$ 0.38 | 69.85 $\pm$ 0.28 | 69.64 $\pm$ 0.17 | 70.03 $\pm$ 0.26 | 70.34 $\pm$ 0.07 | 75.68 $\pm$ 0.37 | 75.9 $\pm$ 0.33 | 75.6 $\pm$ 0.34 | 76.01 $\pm$ 0.42 | 76.52 $\pm$ 0.14 |
| APPNP | 55.05 $\pm$ 2.37 | 54.64 $\pm$ 4.05 | 54.58 $\pm$ 1.28 | 54.89 $\pm$ 1.12 | 55.09 $\pm$ 1.02 | 58.89 $\pm$ 0.21 | 56.09 $\pm$ 0.24 | 53.38 $\pm$ 0.36 | 54.22 $\pm$ 0.27 | 53.59 $\pm$ 0.18 |

**Table 3: Results of node classification on long-range graphs. We report the average test F1-score and the standard deviation of models selected by validation over 3 training or generation runs. We show $100\times$ the original values for presentation purpose. The best results are marked in bold.**
| Datasets | PascalVOC-SP | COCO-SP |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Models | Grid | Random | Coarse | C2F | GNN-Diff | Grid | Random | Coarse | C2F | GNN-Diff |
| MLP | 11.82 $\pm$ 0.16 | 11.79 $\pm$ 0.12 | 11.77 $\pm$ 0.14 | 11.89 $\pm$ 0.27 | 11.97 $\pm$ 0.04 | 3.15 $\pm$ 0.31 | 3.13 $\pm$ 0.16 | 3.05 $\pm$ 0.26 | 3.17 $\pm$ 0.42 | 3.20 $\pm$ 0.05 |
| GCN | 23.31 $\pm$ 0.14 | 23.20 $\pm$ 0.17 | 23.18 $\pm$ 0.11 | 23.33 $\pm$ 0.26 | 23.52 $\pm$ 0.08 | 7.94 $\pm$ 0.21 | 7.78 $\pm$ 0.26 | 7.92 $\pm$ 0.18 | 7.90 $\pm$ 0.31 | 8.07 $\pm$ 0.03 |
| SAGE | 27.98 $\pm$ 0.23 | 28.08 $\pm$ 0.32 | 27.42 $\pm$ 0.26 | 27.98 $\pm$ 0.23 | 28.24 $\pm$ 0.06 | 8.65 $\pm$ 0.61 | 9.19 $\pm$ 0.34 | 8.99 $\pm$ 0.64 | 8.65 $\pm$ 0.61 | 9.28 $\pm$ 0.08 |
| APPNP | 17.24 $\pm$ 0.12 | 15.82 $\pm$ 0.11 | 15.25 $\pm$ 0.14 | 17.24 $\pm$ 0.12 | 15.57 $\pm$ 0.05 | 4.28 $\pm$ 0.07 | 4.27 $\pm$ 0.08 | 4.06 $\pm$ 0.15 | 4.27 $\pm$ 0.08 | 4.43 $\pm$ 0.02 |
| SGC | 21.65 $\pm$ 0.23 | 21.55 $\pm$ 0.21 | 21.05 $\pm$ 0.26 | 21.32 $\pm$ 0.14 | 21.80 $\pm$ 0.12 | 5.56 $\pm$ 0.39 | 5.82 $\pm$ 0.29 | 5.53 $\pm$ 0.40 | 5.70 $\pm$ 0.33 | 5.84 $\pm$ 0.11 |
| GPRGNN | 15.56 $\pm$ 0.19 | 15.21 $\pm$ 0.10 | 15.03 $\pm$ 0.09 | 14.91 $\pm$ 0.12 | 15.26 $\pm$ 0.09 | 4.40 $\pm$ 0.07 | 4.33 $\pm$ 0.12 | 4.27 $\pm$ 0.08 | 4.39 $\pm$ 0.08 | 4.34 $\pm$ 0.07 |
| MixHop | 21.79 $\pm$ 0.18 | 22.37 $\pm$ 0.12 | 22.60 $\pm$ 0.16 | 22.60 $\pm$ 0.16 | 22.75 $\pm$ 0.03 | 7.06 $\pm$ 0.19 | 7.01 $\pm$ 0.15 | 6.55 $\pm$ 0.25 | 6.87 $\pm$ 0.21 | 6.74 $\pm$ 0.03 |

**Table 4: Results of link existence prediction. We report the average test accuracy (%) and the standard deviation of models selected by validation over 10 training or generation runs. The best results are marked in bold.**
| Datasets | Cora | Citeseer |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Models | Grid | Random | Coarse | C2F | GNN-Diff | Grid | Random | Coarse | C2F | GNN-Diff |
| MLP | 75.04 $\pm$ 1.12 | 74.61 $\pm$ 1.72 | 74.05 $\pm$ 1.25 | 74.79 $\pm$ 0.97 | 75.42 $\pm$ 0.19 | 78.26 $\pm$ 0.80 | 78.23 $\pm$ 1.23 | 76.79 $\pm$ 1.28 | 78.25 $\pm$ 1.79 | 78.87 $\pm$ 0.07 |
| GCN | 76.92 $\pm$ 0.83 | 76.73 $\pm$ 0.80 | 76.29 $\pm$ 0.88 | 76.47 $\pm$ 0.43 | 77.04 $\pm$ 0.12 | 77.90 $\pm$ 0.90 | 75.93 $\pm$ 0.72 | 76.90 $\pm$ 0.92 | 77.25 $\pm$ 0.92 | 77.94 $\pm$ 0.14 |
| SAGE | 75.09 $\pm$ 0.49 | 75.22 $\pm$ 1.14 | 74.97 $\pm$ 0.43 | 74.11 $\pm$ 2.24 | 76.33 $\pm$ 0.31 | 76.25 $\pm$ 1.15 | 76.37 $\pm$ 1.32 | 75.84 $\pm$ 1.91 | 76.43 $\pm$ 0.85 | 76.51 $\pm$ 0.08 |
| APPNP | 75.95 $\pm$ 0.93 | 75.80 $\pm$ 1.07 | 75.64 $\pm$ 0.91 | 75.95 $\pm$ 0.93 | 76.94 $\pm$ 0.15 | 75.98 $\pm$ 1.06 | 75.73 $\pm$ 0.81 | 75.68 $\pm$ 0.72 | 75.92 $\pm$ 0.85 | 76.15 $\pm$ 0.15 |
| ChebNet | 74.96 $\pm$ 0.79 | 74.53 $\pm$ 0.70 | 74.17 $\pm$ 0.48 | 74.31 $\pm$ 0.96 | 74.96 $\pm$ 0.15 | 78.31 $\pm$ 1.88 | 77.36 $\pm$ 2.77 | 76.76 $\pm$ 1.38 | 76.82 $\pm$ 0.82 | 78.35 $\pm$ 0.33 |
| Datasets | Chameleon | Squirrel |  |  |  |  |  |  |  |  |
| Models | Grid | Random | Coarse | C2F | GNN-Diff | Grid | Random | Coarse | C2F | GNN-Diff |
| MLP | 76.88 $\pm$ 0.39 | 76.56 $\pm$ 0.46 | 75.27 $\pm$ 0.66 | 76.61 $\pm$ 0.50 | 76.23 $\pm$ 0.10 | 73.24 $\pm$ 0.16 | 73.13 $\pm$ 0.07 | 73.13 $\pm$ 0.07 | 73.28 $\pm$ 0.22 | 73.32 $\pm$ 0.04 |
| GCN | 78.71 $\pm$ 0.54 | 78.50 $\pm$ 0.38 | 77.69 $\pm$ 0.43 | 77.91 $\pm$ 0.34 | 78.26 $\pm$ 0.02 | 74.72 $\pm$ 0.12 | 74.52 $\pm$ 0.45 | 74.61 $\pm$ 0.11 | 74.62 $\pm$ 0.17 | 74.73 $\pm$ 0.05 |
| SAGE | 82.06 $\pm$ 0.41 | 81.89 $\pm$ 0.35 | 81.88 $\pm$ 0.40 | 82.02 $\pm$ 0.51 | 82.08 $\pm$ 0.19 | 75.34 $\pm$ 0.32 | 74.49 $\pm$ 0.51 | 74.09 $\pm$ 0.26 | 74.37 $\pm$ 0.27 | 74.66 $\pm$ 0.02 |
| APPNP | 78.87 $\pm$ 0.75 | 77.67 $\pm$ 0.34 | 77.66 $\pm$ 0.29 | 77.74 $\pm$ 0.65 | 77.95 $\pm$ 0.04 | 73.61 $\pm$ 0.22 | 73.54 $\pm$ 0.25 | 73.51 $\pm$ 0.17 | 73.61 $\pm$ 0.22 | 73.76 $\pm$ 0.09 |
| ChebNet | 81.00 $\pm$ 0.54 | 77.52 $\pm$ 0.46 | 77.25 $\pm$ 0.44 | 77.25 $\pm$ 0.44 | 81.30 $\pm$ 0.18 | 75.48 $\pm$ 0.34 | 75.31 $\pm$ 0.32 | 73.56 $\pm$ 0.62 | 75.32 $\pm$ 0.43 | 75.54 $\pm$ 0.03 |

Results of basic node classification.
We show results of node classification on Cora, Citeseer, Actor, and Wisconsin in Table 1. More node classification results on homophilic and heterophilic graphs can be found in Appendix I. In general, models generated by GNN-Diff achieve the best average test accuracy for 36 out of 40 experiments in Table 1. For the rest of 4 experiments, GNN-Diff produces accuracy that still sufficiently matches the grid search performance. Both GNN-Diff and C2F show clear improvement on the coarse search results, while GNN-Diff outperforms C2F for most target models and datasets. Moreover, it is apparent that models generated by GNN-Diff exhibit more stable performance in terms of standard deviation.

Results of node classification on large graphs. The challenge of large graphs is often associated with the large number of nodes, which makes the computation of graph convolution considerably slow. In addition, GNNs often need a comparably larger hidden size (e.g., 256) to achieve good performance on large graphs , which increases the volume of parameters to learn. The experiment results in Table 2 show that GNN-Diff can produce high-performing GNNs on large graphs by generating only the last layer parameters efficiently.

Results of node classification on long-range graphs. In long-range graphs, nodes need to exchange information over long distances (i.e., information is passed through many edges to form useful graph representations), which usually relies on very deep GNNs with many graph convolutional layers . Partial generation is even more helpful in this case because a well-informed graph representation is formed just before the last layer, making it an important mapping to final classification. We observe in Table 3 that GNN-Diff generates top-performing GNNs for most long-range graph tasks as expected.

Results of link prediction. To validate GNN-Diff on various graph tasks, we conduct link existence prediction following the experiment design in . We use accuracy as the metric instead of AUC-ROC because we adopt equal-proportion sampling for positive and negative edges. The results are presented in Table 4. Models generated by GNN-Diff show outstanding performance in Cora, Citeseer, and Squirrel. On Chameleon, however, random search shows more promising results, though GNN-Diff generally better enhances the original performance from coarse search compared to C2F.

### 5.3 Visualizations

We use visualizations of node classification with GCN on Citeseer as examples to answer the following three questions. Please note that “input samples” refers to the parameter samples collected for GNN-Diff training.

Figure: Figure 4: (a) Scatter plot between validation and test accuracy for grid, random, and coarse search. The black circle indicates the final models selected by validation. (b) Visualization of generated and input sample parameters after dimension reduction with isomap. Each bubble represents one parameter sample, and the bubble size shows the corresponding test accuracy. (c) Kernel density estimation plot of test accuracy distributions corresponding to input samples, noised sample parameters, and GNN-Diff parameters.
Refer to caption: extracted/5909359/Figures/Visual_new.png

Q1. Why are some random or coarse search results even better than grid search?

We observe that some random or coarse search results may be even better than the comprehensive grid search. For example, with GCN node classification on Citeseer, results from coarse, random, and grid search are 71.97%, 72.04%, and 71.92%, respectively. This is different from our previous observation in Figure 1. In Figure 4 (a), we note that models that achieve higher validation accuracy may not perform better on the test set. We recognize this phenomenon as “overfitting on validation set” with overly extensive grid search. Random search with a smaller search space may solve the problem. But how do we know the appropriate search space size? In addition, more than 80% of experiment results show that grid search can find better generalized models than random search. Thus GNN-Diff, which learns from samples with sub-optimal validation performance, is considered as a better solution.

Q2. How does GNN-Diff generate better performance based on the input samples?

In short, GNN-Diff approximates the population of input samples and explores into better-performing regions. In Figure 4 (b), we visualize the input parameters (orange) and 100 parameters generated by GNN-Diff (red) via isometric mapping dimensionality reduction . Additionally, we plot each model point as bubbles with bubble size representing the corresponding test accuracy. We observe that parameters generated by GNN-Diff spread around the input sample distribution. Notably, parameters in the extended distribution may lead to better accuracy than the best-performing input sample (72.70% vs. 72.30%). This may verify our assumption that better-performing parameters exist in the population of sample parameters from a sub-optimal configuration, and can be found by GNN-Diff.

Q3. Can we find better-performing models by simply adding random noises to input samples?

There is a slight possibility, but GNN-Diff clearly shows more promising outcomes. To answer this question, we conduct an experiment in which we add random Gaussian noises to the input samples of GNN-Diff. We set the mean of noises as 0 and tune the standard deviation (std) in [1$e$-3, 5$e$-3, 1$e$-2, 2$e$-2]. All distributions are plotted based on 100 results. The model performance with lightly noised parameters (std = 1$e$-3) almost follows the sample distribution, while high-level noises (std = 2$e$-2) significantly diminish the model accuracy. We observe that when noise std = 5$e$-3, the corresponding accuracy distribution presents slightly better results than the original samples. Nevertheless, parameters from GNN-Diff consistently outperform the noised ones, indicating that GNN-Diff enhances model performance beyond random guessing.

### 5.4 Full Generation vs. Partial Generation

We found that GNN-Diff significantly enhances GNNs by generating only the last learnable layer of the target model. To confirm the effectiveness of this approach, we compared GNN-Diff with three different generation and sampling strategies: (a) partial generation as in the main experiments; (b) full generation (best checkpoint) with samples collected by training the best validation checkpoint from 190 epochs with another 10 epochs and a very small learning rate for 10 runs; (c) full generation (all checkpoints) with samples collected by saving all checkpoints over 200 epochs and discarding the first 10 that have not yet converged.
In Figure 5, we show the distribution of the GNN-Diff parameters and the input parameters and their test accuracy. The test accuracy distribution of GNN-Diff is based on the results of 30 generation runs of 100 samples. We only show the test accuracy corresponding to the best validation sample in each run. Overall, GNN-Diff effectively captures and expands the sample distribution across all three strategies. Moreover, the generated samples usually yield better test performance than the input samples. Partial generation and full generation (best checkpoint) produce similar results, while full generation (all checkpoints) shows noticeably inferior performance. This difference may be caused by the low-quality parameters in the input samples, as the test accuracy from the input samples collected with the all-checkpoint strategy exhibits a much wider range of lower values (see Figure 5 (c)).

Figure: Figure 5: GNN-Diff for GCN node classification on Cora with three generation and sampling strategies. Each row contains visualisations of (1) generated and input sample distributions of the first parameter in last layer bias; (2) generated and input sample distributions of the first parameter in last layer weights; (3) test accuracy of generated and input sample parameters.
Refer to caption: extracted/5909359/Figures/full_vs_part.png

### 5.5 Ablation Study on the Graph Condition

Previous works have discussed using datasets as conditions, but only for the purpose of tailoring their methods for different datasets ; ; . The role of specific data characteristics in parameter generation still remains underexplored. To fill this gap, GNN-Diff leverages the graph data and structural information as the generative condition. To assess the utility of such condition in GNN parameter generation, we use an ablation study to evaluate GNN parameter generation with and without the graph condition. Specifically, we compare GNN-Diff with p-diff , an unconditional diffusion model for parameter generation. GNN-Diff adopts nearly identical PAE and $\epsilon_{\mathbf{\theta}}$ as p-diff, minimizing the impact of model architecture differences. We apply the same training and hyperparameter settings for both methods to ensure that the only difference is the graph condition. The results are exhibited in Table 5. GNN-Diff outperforms p-diff in almost all experiments, and generally presents more stable generation outcomes with lower standard deviation. Therefore, we conclude that the graph condition is indeed helpful with GNN parameter generation by providing insightful guidance based on graph and data information.

**Table 5: Comparison between GNN parameter generation with and without the graph condition (GNN-Diff vs. p-diff). We consider node classification on 4 types of graphs, Cora (homophily), Actor (heterophily), Flickr (large), and PascalVOC-SP (long-range), and two target models, GCN and SAGE.**
| Models | GCN | SAGE |  |  |
| --- | --- | --- | --- | --- |
| Methods | GNN-Diff | p-diff | GNN-Diff | p-diff |
| Cora | 82.33 $\pm$ 0.17 | 81.96 $\pm$ 0.31 | 80.60 $\pm$ 0.05 | 80.19 $\pm$ 0.60 |
| Actor | 31.24 $\pm$ 0.26 | 30.96 $\pm$ 0.30 | 36.11 $\pm$ 0.09 | 36.25 $\pm$ 0.31 |
| Flickr | 52.84 $\pm$ 0.04 | 52.70 $\pm$ 0.11 | 53.70 $\pm$ 0.02 | 53.55 $\pm$ 0.23 |
| PascalVOC-SP | 23.52 $\pm$ 0.08 | 23.49 $\pm$ 0.09 | 28.24 $\pm$ 0.06 | 28.12 $\pm$ 0.02 |

### 5.6 Sensitivity Analyses

Sensitivity analysis on coarse search results. We would like to know whether a coarse search with 10% of full search space is able to find a sufficiently good configuration for GNN-Diff to generate high-performing parameters. So, we repeat the coarse search for GCN node classification on Cora and PascalVOC-SP without fixed random seed 7 times and obtain 7 configurations selected for parameter collection. Then, we train and generate with GNN-Diff based on these configurations. The results show that GNN-Diff consistently produces better test outcomes than grid search (see Figure 6 (a) and (b)), which confirms the feasibility of adopting the 10% coarse search space.

Sensitivity analysis on the number of samples generated by GNN-Diff. Since overfitting the validation data may lead to inferior results on unseen data, we wonder if generating too many samples from GNN-Diff will encounter this issue. Hence, with the same target model and datasets in the previous analysis, we set the number of samples generated by GNN-Diff to [10, 50, 100, 250, 500, 750, 1000]. In general, we observe that GNN-Diff’s performance on unseen data is not very sensitive to the number of samples, though generating 750 to 1000 samples indeed diminishes the test accuracy on Cora (see Figure 6 (c) and (d)). In consideration of sampling efficiency, we suggest that 100 samples are sufficient for GNN-Diff to produce good performance in a reasonable sampling time.

Figure: Figure 6: Sensitivity analysis. How average test results varies with coarse search results (left column) and number of samples generated (right column).
Refer to caption: extracted/5909359/Figures/sens.png

### 5.7 Time Efficiency

Finally, we compare the time costs of grid search, random search, C2F search, and GNN-Diff (see Figure 7). GNN-Diff’s time costs include the time for coarse search, parameter collection, training, and inference (see details in Appendix I.3). GNN-Diff shows a clear time advantage over grid and random search, though it may not be faster than C2F on small graphs such as Cora and Actor. This is acceptable, as GNN-Diff often yields better-performing models than C2F and is more efficient on larger graphs such as Flickr and PascalVOC-SP, which offers greater practical value, especially for complicated tasks such as node classification on large and long-range graphs.

Figure: Figure 7: Time costs of grid search, random search, C2F search, and GNN-Diff.
Refer to caption: extracted/5909359/Figures/Time_new.png

## 6 Conclusion

In this paper, we proposed a graph-conditioned latent diffusion framework, GNN-Diff, to generate top-performing GNN parameters by learning from checkpoints saved with a sub-optimal configuration selected by a light-tuning coarse search. We validate with 166 empirical experiments that our method is an efficient alternative to costly search methods and is able to generate better prediction outcomes than the comprehensive grid search on unseen data. Future works may involve adapting GNN-Diff to more graph tasks and GNN architectures.

## Appendix A Related Works

Network Parameter Generation

Hypernetwork [, ] is an early concept of deep learning technique for network generation and now serves as a general framework that encompasses many existing methods. A hypernetwork is a neural network that learns how to predict the parameters of another neural network (target network). The input of a hypernetwork may be as simple as parameter positioning embeddings [], or more informative, such as encodings of data, labels, and tasks [, , , , ].

With the emergence of generative modelling, studies on distributions of network parameters have gained their popularity in relevant research. A pioneering work of [] showed that simply knowing the statistics of parameter distributions enabled us to predict model accuracy without accessing the data. Enlightened by this finding, two following works [, ] studied the reconstruction and sampling of network parameters from their population distribution. The similar idea was also investigated by [], who proposed that network parameters corresponding to a specific metric value of some specific data and tasks could be generated with a transformer-diffusion model trained on a tremendous amount of model checkpoints collected from past training on a group of data and tasks. This method, however, relies highly on the amount and quality of model checkpoints fed to the generative model. A more network- and data-specific approach, p-diff [], alternatively collects checkpoints from the training process of the target network and uses an unconditional latent diffusion model to generate better-performing parameters. Another recent work, D2NWG [], adopts dataset-conditioned latent diffusion to generate parameters for target networks on unseen datasets. Diffusion-based network generation has also been widely applied to other applications, such as meta learning [, ] and producing implicit neural-filed for 3D and 4D synthesis [].

GNN and GNN Training
GNNs have become the fundamental tool for analyzing graph-structured data. As we have shown in the main part of this work, our target GNNs including both spatial [] and spectral GNNs [], with the first aggregates the feature information based on their local connectivities [, ], and later learns the coefficients for spectral filtering [, , , ]. In addition, some GNNs are with characteristics that aim to mitigate several GNN computation issues, such as over-smoothing [], over-squashing [] and heterophily adaption [].

In terms of previous works on GNN training generally focus on two topics: GNN pre-training and GNN architecture search. GNN pre-training involves using node, edge, or graph level tasks to find appropriate initialization of GNN parameters before fine-tuning with ground truth labels [, , ]. This approach still inevitably relies on hyperparameter tuning to boost model performance. GNN architecture search aims to find appropriate GNN architectures given data and tasks [, ]. However, it was shown in [] that under a comprehensive hyperparameter tuning and a proper training process, simple GNN architectures such as GCN may even outperform sophisticated ones.

Advanced Methods for Hyperparameter Tuning

There emerge many recent works on parameter-free optimization, such as [, , ] that automatically set step size based on problem characteristics. However, most if not all methods require nontrivial adaptation of existing optimizers and show theoretical guarantees only for convex functions. Bayesian optimization [, ], on the other hand, is a more advanced approach to tuning hyperparameters in a more informed way than grid and random search. Bayesian optimization usually adopts a probabilistic surrogate model to approximate the loss function and then uses an acquisition function to sample hyperparameters for further search. Additionally, Hyperband [], a method that is considered more efficient than Bayesian optimization, optimizes the search space by allocating resources (e.g., training time or dataset size) to more promising configurations and pruning those that perform poorly. In our experiments, we consider the coarse-to-fine (C2F) search [, ], a variant of ransom search and Hyperband that narrows the search space based on the result of a coarse search.

## Appendix B Pseudo Code of GNN-Diff

Figure: Algorithm 1 Algorithm of GNN-Diff Parameter Generation

Figure: Algorithm 2 Training Algorithm of GAE (Basic Node Classification Example)

Figure: Algorithm 3 Training Algorithm of PAE

Figure: Algorithm 4 Training Algorithm of ${\mathbf{\epsilon}}_{\mathbf{\theta}}$ in G-LDM

Figure: Algorithm 5 Inference Algorithm of G-LDM

## Appendix C Details of GNN-Diff

### C.1 GAE Architectures for Various Tasks

We provide illustrations of the GAE architectures for various tasks in Figure 8. In GAE encoders, we generally adopt graph convolutional layers to process graph data and structural information. The decoders, on the other hand, is a single linear layer (MLP1), thus the graph information is entirely preserved in the output of the encoders, which will later be further processed and used as the graph condition in G-LDM. The details of GAE for each task are as follows. Please note that for simplicity, we show encoders without activation functions, but activation functions can be included in implementation.

Figure: Figure 8: GAE architectures for node classification on small, large, and long-range graphs, and link prediction. The graph condition is designed to incorporate both data and graph structural information and guided by specific tasks and the corresponding loss functions.
Refer to caption: extracted/5909359/Figures/GAE.png

Basic node classification. Details can be found in Subsection 4.2.

Node classification on large graphs. The input to the GAE is a set of graph clusters, each with $N_{i}$ nodes for $i=1,2,...,I$. Details of how the graph clusters are obtained from the original large graph with $N$ nodes are discussed in G.2. The feature matrix and adjacency matrix of each cluster are denoted as ${\mathbf{X}}_{i}$ and $\mathbf{A}_{i}$, respectively. We adopt the same encoder architecture as for basic node classification, because the homophilic and heterophilic problem may also exist in large graph node classification. This may or may not be true for the datasets in our experiments, but we design in this way for future extension to other large graphs. Hence, for each cluster, the encoder is applied following Equation 2 as

$$ $\mathbf{\eta}_{i}=\text{G-Encoder}({\mathbf{X}}_{i},\mathbf{A}_{i})=\text{ Concat}\left(\mathbf{A}_{i}^{2}{\mathbf{X}}_{i}{\mathbf{W}}_{1},\mathbf{A}_{i} {\mathbf{X}}_{i}{\mathbf{W}}_{2},{\mathbf{X}}_{i}{\mathbf{W}}_{3}\right){ \mathbf{W}}_{4}\in{\mathbb{R}}^{N_{i}\times D_{p}}.$ (5) $$

The decoder and loss function are the same as the GAE for basic node classification as well. To form the graph condition, we shall first concatenate the outputs of the encoder in the node dimension as

$$ $\mathbf{\eta}=\text{Concat}(\mathbf{\eta}_{1},\mathbf{\eta}_{2},...,\mathbf{ \eta}_{I})\in{\mathbb{R}}^{N\times D_{p}},$ (6) $$

which will be eventually turned into the graph condition via the node mean-pooling in Equation 3.

Node classification on long-range graphs. The input of GAE is a set of batches, where each batch contains a set of small graphs. Suppose that we have $B$ batches and that a small graph in a batch has $N_{b,j}$ nodes for $b=1,2,...,B$, and $j=1,2,...,J_{b}$, where $J_{b}\in[J_{1},J_{2},...,J_{B}]$ is the number of small graphs in each batch. Then, we organize the small graphs in each batch into a combined batch graph with $N_{b}=N_{b,1}+N_{b,2}+\cdots+N_{b,J_{b}}$ nodes. Let the feature matrix and adjacency matrix of each batch graph formed by the small graphs in a batch be ${\mathbf{X}}_{b}$ and $\mathbf{A}_{b}$, respectively. The encoder will be applied to each batch graph as

$$ $\mathbf{\eta}_{b}=\text{G-Encoder}({\mathbf{X}}_{b},\mathbf{A}_{b})=\text{Deep -GCN}({\mathbf{X}}_{b},\mathbf{A}_{b})\in{\mathbb{R}}^{N_{b}\times D_{p}},$ (7) $$

where $\text{Deep-GCN}(\cdot)$ is a GCN-based structure with 7 GCN convolutional layers and residual connections for the training purpose. We choose this deep GCN-based structure as the GAE encoder to better process long-term information among distant nodes, which is usually considered as important in long-range graphs.
The decoder is as simple as a single linear layer. Moreover, the loss function is the weighted cross entropy discussed in the appendix G.3. Similar to large graphs, the graph condition is constructed by mean-pooling the concatenation of encoder outputs:

$$ $\mathbf{\eta}=\text{Concat}(\mathbf{\eta}_{1},\mathbf{\eta}_{2},...,\mathbf{ \eta}_{B})\in{\mathbb{R}}^{N_{\text{all}}\times D_{p}},$ (8) $$

where $N_{\text{all}}$ is the total number of nodes of small graphs in all batches.

Link prediction. The GAE for link prediction is designed based on the implementation in []. Details of the link prediction task can be found in Appendix G.4. The input of GAE is the feature matrix ${\mathbf{X}}$ and the adjacency matrix $\mathbf{A}_{\text{train}}$ with only training edges. The encoder is designed as a 2-layer GCN:

$$ $\mathbf{\eta}=\text{G-Encoder}({\mathbf{X}},\mathbf{A}_{\text{train}})=\mathbf {A}_{\text{train}}(\mathbf{A}_{\text{train}}{\mathbf{X}}{\mathbf{W}}_{1}){ \mathbf{W}}_{2}\in{\mathbb{R}}^{N\times D_{p}},$ (9) $$

where ${\mathbf{W}}_{1}$ and ${\mathbf{W}}_{2}$ are learnable linear operators. The decoder is again a single linear layer, but is followed by the pairwise node multiplication and embedding aggregation for link prediction. The loss function is binary cross entropy computed based on the training edge labels. Lastly, the graph condition can be constructed in the same way as in Equation 3.

### C.2 PAE and G-LDM Denoising Network Architectures

Illustrations of the PAE and the G-LDM denoising network ${\mathbf{\epsilon}}_{\mathbf{\theta}}$ are provided in Figure 9 and Figure 10, respectively. Taking the idea from [], both architectures have Conv1d-based layers with LeakyReLU activation and instance normalization.

PAE. P-Encoder and P-Decoder are both composed of 4 basic blocks in Figure 9. Since the input is latent representation of vectorized parameters, the number of input channels of the first block in P-Encoder is 1. The number of output channels for the 4 blocks in P-Encoder is [6, 6, 6, 6]. Accordingly, the number of input channels of the first block in P-Decoder is also $6$. The number of output channels of the 4 blocks in P-Decoder is [512, 512, 8, 1].

G-LDM ${\mathbf{\epsilon}}_{\mathbf{\theta}}$. The denoising network ${\mathbf{\epsilon}}_{\mathbf{\theta}}$ adopts an encoder-decoder structure with 8 basic blocks in Figure 10. The number of input channels and output channels, kernel size, and stride are the same as in []. In Figure 10, we also mark the position where the graph condition ${\mathbf{c}}_{\mathcal{G}}$ is involved. We include the graph condition in each encoder and decoder block by adding it to the input of the block directly.

Figure: Figure 9: The PAE architecture. The PAE adopts an encoder-decoder structure with 4 encoder blocks and 4 decoder blocks. All blocks have the same structure as shown in the basic block. The dimensionality of the input parameters is first reduced by the P-Encoder, and then recovered by the P-Decoder.
Refer to caption: extracted/5909359/Figures/PAE_new.png

Figure: Figure 10: The ${\mathbf{\epsilon}}_{\mathbf{\theta}}$ architecture. Like PAE, ${\mathbf{\epsilon}}_{\mathbf{\theta}}$ adopts an encoder-decoder structure with 4 encoder blocks and 4 decoder blocks. We mark the position where the graph condition is involved by taking sum with the input of each block.
Refer to caption: extracted/5909359/Figures/G-LDM_new.png

### C.3 GNN-Diff Settings

GAE is trained with the same graph data as target models. PAE is trained with vectorized parameter samples, while G-LDM learns from their latent representation. Batch size for PAE and G-LDM training is 50. All three modules are trained with the same AdamW optimizer [] with learning rate 0.001 and weight decay 0.002. The concatenation dimension $D_{\mathcal{G}}$ is set the same as the latent parameter dimension $D_{p}$, which is decided by the Conv1d layer in PAE. Dropout is applied with dropout rate $0.1$. For G-LDM, we set the number of diffusion steps $T=1000$. We choose the linear beta schedule with $\beta_{1}=$0.0001 and $\beta_{T}=$0.02.

How to ensure high quality generation?

We observe in experiments that GNN-Diff does not rely on hyperparameter tuning to generate top-performing GNNs. The only factor that may influence the generative outcomes is the dimension of the latent parameters, which can be easily adjusted by setting the kernel size and stride of Conv1d layers in P-Encoder blocks. Additionally, the appropriate kernel size and stride can be easily decided based on the dimension of the original parameter samples (see Table 6). Therefore, we highlight that GNN-Diff does not introduce additional hyperparameter tuning burdens, and all tuning efforts are only associated with the tuning of target models in coarse search.

**Table 6: Kernel size and stride of Conv1d layers in P-Encoder based on the input parameter dimension. The simple rule is: (1) the stride equals to the kernel size; (2) if the parameter dimension is larger than 1000, we increase kernel size and stride by 1 for every 5000 parameters.**
| Vectorized parameter dimension | Kernel size | Stride |
| --- | --- | --- |
| $\leq$ 99 | 2 | 2 |
| 100 - 499 | 3 | 3 |
| 500 - 999 | 4 | 4 |
| 1000 - 4999 | 5 | 5 |
| 5000 - 9999 | 6 | 6 |
| 10000 - 14999 | 7 | 7 |
| 15000 - 19999 | 8 | 8 |
| 20000 - 24999 | 9 | 9 |
| … | … | … |

## Appendix D Baseline Methods

We involve four baseline search methods to compare with GNN-Diff, including grid search, random search, coarse search, and C2F search. As aforementioned in Subsection 5.1, random search and coarse search adopt 20% and 10% of the full search space in grid search. In terms of C2F search, we first conduct the coarse search, and then fix the model-related hyperparameters and optimizer based on the coarse search result, and further tune other training-related hyperparameters. For example, if the hyperparameter configuration leading to the best validation accuracy is {optimizer: ‘Adam’, learning_rate: 0.01, weight_decay: 0.005, dropout: 0.3, hidden_size: 64}, we will fix the optimizer to ‘Adam’ and the hidden_size to 64, and tune the training related hyperparameters in a narrowed search space as {learning_rate [0.005, 0.1, 0.05], weight_decay [0.0005, 0.005, 0.05], dropout [0.1, 0.3, 0.5]}. The number of configurations in the search spaces of different baseline methods are presented in Table 7, 8, 9, and 10. More details of hyperparameter configurations can be found in Appendix H.

**Table 7: Number of hyperparameter configurations in the search spaces of baseline methods for basic node classification on homophilic and heterophilic graphs. The size of the C2F search space depends on the coarse search result.**
| Basic Node Classiciation | Grid Search | Random Search | Coarse Search | C2F Search |
| --- | --- | --- | --- | --- |
| MLP | 540 | 108 | 54 | 61 - 80 |
| GCN | 540 | 108 | 54 | 61 - 80 |
| SAGE | 540 | 108 | 54 | 61 - 80 |
| APPNP | 900 | 180 | 90 | 97 - 116 |
| GAT | 1620 | 324 | 162 | 169 - 188 |
| ChebNet | 1620 | 324 | 162 | 169 - 188 |
| H2GCN | 540 | 108 | 54 | 61 - 80 |
| SGC | 540 | 108 | 54 | 61 - 80 |
| GPRGNN | 1440 | 288 | 144 | 151 - 170 |
| MixHop | 540 | 108 | 54 | 61 - 80 |

**Table 8: Number of hyperparameter configurations in the search spaces of baseline methods for node classification on large graphs. The size of the C2F search space depends on the coarse search result.**
| Large Graphs | Grid Search | Random Search | Coarse Search | C2F Search |
| --- | --- | --- | --- | --- |
| GCN | 54 | 12 | 6 | 13 - 23 |
| SAGE | 54 | 12 | 6 | 13 - 23 |
| APPNP | 72 | 16 | 8 | 15 - 25 |

**Table 9: Number of hyperparameter configurations in the search spaces of baseline methods for node classification on long-range graphs. The size of the C2F search space depends on the coarse search result.**
| Long-range Graphs | Grid Search | Random Search | Coarse Search | C2F Search |
| --- | --- | --- | --- | --- |
| MLP | 48 | 10 | 5 | 12 - 16 |
| GCN | 48 | 10 | 5 | 12 - 16 |
| SAGE | 48 | 10 | 5 | 12 - 16 |
| APPNP | 72 | 16 | 8 | 15 - 19 |
| SGC | 48 | 10 | 5 | 12 - 16 |
| GPRGNN | 72 | 16 | 8 | 15 - 19 |
| MixHop | 48 | 10 | 5 | 12 - 16 |

**Table 10: Number of hyperparameter configurations in the search spaces of baseline methods for link prediction tasks. The size of the C2F search space depends on the coarse search result.**
| Link Prediction | Grid Search | Random Search | Coarse Search | C2F Search |
| --- | --- | --- | --- | --- |
| MLP | 540 | 108 | 54 | 61 - 80 |
| GCN | 540 | 108 | 54 | 61 - 80 |
| SAGE | 540 | 108 | 54 | 61 - 80 |
| APPNP | 1620 | 324 | 162 | 169 - 188 |
| ChebNet | 1620 | 324 | 162 | 169 - 188 |

## Appendix E Datasets

### E.1 Dataset Sources

All datasets used in our experiments are publicly available and can be easily retrieved from the following sources.

Cora, Citeseer, Pubmed

[https://pytorch-geometric.readthedocs.io/en/latest/generated/torch_geometric.datasets.Planetoid.html#torch_geometric.datasets.Planetoid](https://pytorch-geometric.readthedocs.io/en/latest/generated/torch_geometric.datasets.Planetoid.html#torch_geometric.datasets.Planetoid)

Computers, Photo

[https://pytorch-geometric.readthedocs.io/en/latest/generated/torch_geometric.datasets.Amazon.html#torch_geometric.datasets.Amazon](https://pytorch-geometric.readthedocs.io/en/latest/generated/torch_geometric.datasets.Amazon.html#torch_geometric.datasets.Amazon)

CS

[https://pytorch-geometric.readthedocs.io/en/latest/generated/torch_geometric.datasets.Coauthor.html#torch_geometric.datasets.Coauthor](https://pytorch-geometric.readthedocs.io/en/latest/generated/torch_geometric.datasets.Coauthor.html#torch_geometric.datasets.Coauthor)

Actor

[https://pytorch-geometric.readthedocs.io/en/latest/generated/torch_geometric.datasets.Actor.html#torch_geometric.datasets.Actor](https://pytorch-geometric.readthedocs.io/en/latest/generated/torch_geometric.datasets.Actor.html#torch_geometric.datasets.Actor)

Wisconsin

[https://pytorch-geometric.readthedocs.io/en/latest/generated/torch_geometric.datasets.WebKB.html#torch_geometric.datasets.WebKB](https://pytorch-geometric.readthedocs.io/en/latest/generated/torch_geometric.datasets.WebKB.html#torch_geometric.datasets.WebKB)

Roman-Empire, Amazon-Ratings, Minesweeper, Tolokers

[https://pytorch-geometric.readthedocs.io/en/latest/generated/torch_geometric.datasets.HeterophilousGraphDataset.html#torch_geometric.datasets.HeterophilousGraphDataset](https://pytorch-geometric.readthedocs.io/en/latest/generated/torch_geometric.datasets.HeterophilousGraphDataset.html#torch_geometric.datasets.HeterophilousGraphDataset)

Chameleon, Squirrel

[https://pytorch-geometric.readthedocs.io/en/latest/generated/torch_geometric.datasets.WikipediaNetwork.html#torch_geometric.datasets.WikipediaNetwork](https://pytorch-geometric.readthedocs.io/en/latest/generated/torch_geometric.datasets.WikipediaNetwork.html#torch_geometric.datasets.WikipediaNetwork)

Flickr

[https://pytorch-geometric.readthedocs.io/en/latest/generated/torch_geometric.datasets.Flickr.html#torch_geometric.datasets.Flickr](https://pytorch-geometric.readthedocs.io/en/latest/generated/torch_geometric.datasets.Flickr.html#torch_geometric.datasets.Flickr)

Reddit

[https://pytorch-geometric.readthedocs.io/en/latest/generated/torch_geometric.datasets.Reddit.html#torch_geometric.datasets.Reddit](https://pytorch-geometric.readthedocs.io/en/latest/generated/torch_geometric.datasets.Reddit.html#torch_geometric.datasets.Reddit)

OGB-arXiv, OGB-Products

[https://ogb.stanford.edu/docs/nodeprop/](https://ogb.stanford.edu/docs/nodeprop/)

PascalVOC-SP, COCO-SP

[https://github.com/vijaydwivedi75/lrgb](https://github.com/vijaydwivedi75/lrgb)

### E.2 Dataset Statistics

We provide statistics of 20 datasets used in our experiments in Table 11, 12, 13, 14, and 15.

**Table 11: Datasets in basic node classification on homophilic graphs. We provide statistics of the number of nodes, edges, features, and node classes. “#” means “the number of”.**
| Dataset | #Nodes | #Egdes | #Features | #Node Classes |
| --- | --- | --- | --- | --- |
| Cora | 2,708 | 5,429 | 1,433 | 7 |
| Citeseer | 3,327 | 4,372 | 3,703 | 6 |
| Pubmed | 19,717 | 44,338 | 500 | 3 |
| Computers | 13,752 | 491,722 | 767 | 10 |
| Photo | 7,487 | 126,530 | 745 | 8 |
| CS | 18,333 | 100,227 | 6805 | 15 |

**Table 12: Datasets in basic node classification on heterophilic graphs. We provide statistics of the number of nodes, edges, features, and node classes. “#” means “the number of”.**
| Dataset | #Nodes | #Egdes | #Features | #Node Classes |
| --- | --- | --- | --- | --- |
| Actor | 7,600 | 30,019 | 932 | 5 |
| Wisconsin | 251 | 515 | 1,703 | 5 |
| Roman-Empire | 22,662 | 32,927 | 300 | 18 |
| Amazon-Ratings | 24,492 | 93,050 | 300 | 5 |
| Minesweeper | 10,000 | 39,402 | 7 | 2 |
| Tolokers | 11,758 | 519,000 | 10 | 2 |

**Table 13: Datasets in node classification on large graphs. We provide statistics of the number of nodes, edges, features, and node classes. “#” means “the number of”.**
| Dataset | #Nodes | #Egdes | #Features | #Node Classes |
| --- | --- | --- | --- | --- |
| Flickr | 89,250 | 899,756 | 500 | 7 |
| Reddit | 232,965 | 114,615,892 | 602 | 41 |
| OGB-arXiv | 169,343 | 1,166,243 | 128 | 40 |
| OGB-Products | 2,449,029 | 61,859,140 | 100 | 47 |

**Table 14: Datasets in node classification on long-range graphs. Long-range graph datasets contain multiple small graphs. We provide statistics of the number of graphs, all nodes, average nodes, edges, features, and node classes. “#” means “the number of”.**
| Dataset | #Graphs | #Nodes (all) | #Nodes (avg.) | #Egdes | #Features | #Node Classes |
| --- | --- | --- | --- | --- | --- | --- |
| PascalVOC-SP | 11,355 | 5,443,545 | ~479.4 | 30,777,444 | 14 | 21 |
| COCO-SP | 123,286 | 58,793,216 | ~476.9 | 332,091,902 | 14 | 81 |

**Table 15: Datasets in link prediction. We provide statistics of the number of nodes, edges, features, and link classes. “#” means “the number of”.**
| Dataset | #Nodes | #Egdes | #Features | #Link Classes |
| --- | --- | --- | --- | --- |
| Cora | 2,708 | 10,556 | 1,433 | 2 |
| Citeseer | 3,327 | 9,104 | 3,703 | 2 |
| Chameleon | 2,277 | 36,101 | 2,325 | 2 |
| Squirrel | 5,201 | 270,962 | 2,089 | 2 |

### E.3 Dataset Train/Val/Test Splits

For Cora, Citeseer, and Pubmed in node classification tasks, we adopt the public fixed train/test/val splits in []. For Computers, Photo, and CS, we generate the splits with 20 nodes and 30 nodes per class in the training and validation sets and the rest in the test set. For heterophilic datasets, we adopt their original splits from [] and [], and use the first split out of the 10 available splits for each dataset. This is because only one train/val/test split is considered in our experiments to easily compare the generated performance with the training performance over multiple runs. Large graph datasets are splited in the same way as in their original papers [, , ]. The long-range datasets, PascalVOC-SP and COCO-SP, both contain multiple graphs. Hence, we adopt the train, validation, and test loaders from the implementation in []. Besides, in consideration of time, for COCO-SP, we only use 10% of graphs in each loader for our experiment, and we ensure that the reduced loaders contain all node classes. Furthermore, the link prediction datasets are splited based on edges. Please refer to Appendix G.4 for details.

## Appendix F Target Models

In Table 16, we show details of the target models in our experiments. The original graph data is denoted as ${\mathbf{X}}^{(0)}$, and the output of layer $\ell$ is denoted as ${\mathbf{X}}^{(\ell)}$. In addition, ${\mathbf{X}}^{(\text{out})}$ is the output of the target model, and we only show how ${\mathbf{X}}^{(\text{out})}$ is computed when the last layer is different from the other layers. We use ${\mathbf{W}}^{(\ell)}$ to represent learnable linear operators in layer $\ell$, and $\sigma$ to represent the activation function. The adjacency matrix $\mathbf{A}$ can be normalized in implementation. We focus on the model-specific hyperparameters when designing the search spaces in our experiments. Please refer to the original papers for details of the other notations.

**Table 16: Details of target models. We provide information of model layers, model-specific hyperparameters, and implementation sources.**
| Model | Layer | Model-specific Hyperparameter |
| --- | --- | --- |
| MLP | $\mathbf{X}^{(\ell+1)}=\sigma\big{(}\mathbf{X}^{(\ell)}\mathbf{W}^{(\ell)}\big{)}$ | - |
| GCN | $\mathbf{X}^{(\ell+1)}=\sigma\big{(}\mathbf{A}\mathbf{X}^{(\ell)}\mathbf{W}^{(<br>\ell)}\big{)}$ | - |
| SAGE | $\mathbf{X}^{(\ell+1)}=\sigma\big{(}\mathbf{X}^{(\ell)}\mathbf{W}_{1}^{(\ell)}+<br>\mathbf{A}\mathbf{X}^{(\ell)}\mathbf{W}_{2}^{(\ell)}\big{)}$ | - |
| APPNP | $\mathbf{X}^{(\ell+1)}=(1-\alpha)\mathbf{A}\mathbf{X}^{(\ell)}+\alpha\mathbf{X}<br>^{(0)}\quad\mathbf{X}^{(\mathrm{out})}=\sigma\big{(}\mathbf{X}^{(L)}\mathbf{W}<br>\big{)}$ | Teleport probability $\alpha$ |
| GAT | $\mathbf{X}^{(\ell+1)}=\sigma\big{(}\boldsymbol{\Theta}^{(\ell)}\odot\mathbf{A}<br>\mathbf{X}^{(\ell)}\mathbf{W}^{(\ell)}\big{)}$ | Number of heads |
| ChebNet | $\mathbf{X}^{(\ell+1)}=\sigma\big{(}\sum_{k=0}^{K}\mathcal{T}_{k}(\widetilde{<br>\mathbf{L}})\mathbf{X}^{(\ell)}\mathbf{W}^{(\ell)}\big{)}$ | Chebyshev filter size K |
| H2GCN | $\mathbf{X}_{i,h\in H}^{(\ell+1)}=\mathrm{AGGR}\{\mathbf{X}_{j}^{(\ell)}:j\in<br>\mathcal{N}_{h}(i)\}\quad\mathbf{X}^{(\text{out})}_{i}=\mathrm{Combine}(<br>\mathbf{X}_{i}^{(0)},\mathbf{X}_{i}^{(1)},\cdots\mathbf{X}_{i}^{(L)}){\mathbf{<br>W}}$ | Hop set $H$ |
| SGC | $\mathbf{X}^{(\text{out})}=\sigma\big{(}\mathbf{A}^{h}\mathbf{X}^{(0)}\mathbf{W<br>}\big{)}$ | Number of hops $h$ |
| GPRGNN | $\mathbf{X}^{(\ell+1)}=\sum_{m=0}^{M}\gamma_{m}\mathbf{A}\mathbf{X}^{(\ell)}<br>\mathbf{W}^{(\ell)}$ | GPR weights $\gamma_{m}$ |
| MixHop | $\mathbf{X}^{(\text{out})}=\mathrm{Combine}_{h\in H}({\mathbf{A}}^{h}\mathbf{X}<br>^{(0)}\mathbf{W})$ | Hop set $H$ |
| Model | Implementation Source |  |
| MLP | [https://pytorch.org/docs/stable/generated/torch.nn.Linear.html](https://pytorch.org/docs/stable/generated/torch.nn.Linear.html) |  |
| GCN | [https://pytorch-geometric.readthedocs.io/en/latest/modules/nn.html#convolutional-layers](https://pytorch-geometric.readthedocs.io/en/latest/modules/nn.html#convolutional-layers) |  |
| SAGE | [https://pytorch-geometric.readthedocs.io/en/latest/modules/nn.html#convolutional-layers](https://pytorch-geometric.readthedocs.io/en/latest/modules/nn.html#convolutional-layers) |  |
| APPNP | [https://pytorch-geometric.readthedocs.io/en/latest/modules/nn.html#convolutional-layers](https://pytorch-geometric.readthedocs.io/en/latest/modules/nn.html#convolutional-layers) |  |
| GAT | [https://pytorch-geometric.readthedocs.io/en/latest/modules/nn.html#convolutional-layers](https://pytorch-geometric.readthedocs.io/en/latest/modules/nn.html#convolutional-layers) |  |
| ChebNet | [https://pytorch-geometric.readthedocs.io/en/latest/modules/nn.html#convolutional-layers](https://pytorch-geometric.readthedocs.io/en/latest/modules/nn.html#convolutional-layers) |  |
| H2GCN | [https://github.com/GemsLab/H2GCN](https://github.com/GemsLab/H2GCN) |  |
| SGC | [https://pytorch-geometric.readthedocs.io/en/latest/modules/nn.html#convolutional-layers](https://pytorch-geometric.readthedocs.io/en/latest/modules/nn.html#convolutional-layers) |  |
| GPRGNN | [https://github.com/jianhao2016/GPRGNN](https://github.com/jianhao2016/GPRGNN) |  |
| MixHop | [https://pytorch-geometric.readthedocs.io/en/latest/modules/nn.html#convolutional-layers](https://pytorch-geometric.readthedocs.io/en/latest/modules/nn.html#convolutional-layers) |  |

## Appendix G Tasks and Relevant Details

### G.1 Basic Node Classification

Task description. The basic node classification is a semi-supervised task on small-scale homophilic and heterophilic graphs. During training, the GNN has access to the features of all nodes and the graph adjacency matrix, but the loss function is computed only based on the training node labels. Likewise, validation and test are conducted with the validation and test node labels. Training, validation, and test labels are obtained with masks.

Target model architectures. We use a 2-convolutional layer structure for most target models following the common practice. Similarly, for MLP, we use 2 linear layers, which often leads to good performance on heterophilic graphs []. For SGC and MixHop, which are designed to have only one graph convolutional layer, we adjust by setting 2 hops for SGC and {0, 1, 2} hops for MixHop. H2GCN also uses a hop set of {0, 1, 2}, but like most target models, we set the number of layers to 2. The ReLU activation function is used in all layers except for the last layer when applicable. Importantly, all target models have simple architectures without using advanced techniques like residual connections or layer normalization.

### G.2 Node Classification on Large Graphs

Task description. The node classification on large graphs is the same as the basic node classification except that the graphs are very large, which may cause memory issues and slow computation. To solve this problem, we apply graph clustering as motivated by []. Graphs are clustered into multiple densely connected subgraphs by minimizing the number of edges cut between clusters. The number of clusters is set to 32 for Flickr and OGB-arXiv and 960 for Reddit and OGB-Products. Each cluster has its own train/validation/test masks. During training, we loop through all clusters iteratively via a cluster loader in each epoch. Validation and testing are implemented with the cluster loader without shuffling.

Target model architectures. We use GCN, SAGE, and APPNP as target models. GCN and SAGE adopt the 2-convolutional layer structure with the hidden size tuned in [64, 128, 256]. APPNP has 2 graph convolutional layers followed by a linear layer to map the graph representation formed by graph convolutional layers to node classes. The ReLU activation function is used in all layers except for the last layer when applicable.

### G.3 Node Classification on Long-range Graphs

Task description.
The node classification on long-range graphs is different from the previous two tasks as there are multiple small graphs in long-range graph datasets. These small graphs are divided into train/validation/test sets. Therefore, the node classification on long-range graphs is a supervised task instead of a semi-supervised task. In addition, the node classes are highly imbalanced in the datasets used in our experiments. For example, in PascalVOC-SP, there are 21 classes with more than 70% of nodes in class 0. Accordingly, we use the weighted cross entropy loss for training as in [], which applies higher weights to uncommon classes. In data preprocessing, we construct train/validation/test loaders with batch sizes 128, 500, and 500, respectively. The small graphs in each batch are connected into a large batch graph such that they can be processed by target models in parallel. We train the target model with 200 epochs with 50 batches in each epoch and then evaluate the model with all batches in the validation and test loaders.

Target model architectures.
Since long-range graphs usually rely on very deep GNN architectures, we set the number of layers as a hyperparameter in the search spaces. For most target models, we adjust the number of layers in [8, 10] and the hidden size in [192, 256]. For SGC and MixHop, since they are designed to have only one graph convolutional layer, we tune the number of hops of SGC in [8, 10], and the hop set of MixHop in [{6, 8, 10}, {8, 10}]. Furthermore, considering the taget models are very deep, we adopt the GELU activation function and residual connections following the experiment settings in [] as well as layer normalization.

### G.4 Link Prediction

Task description.
The link prediction is a binary classification task that predicts the existence of edges between pairs of nodes []. If there exists an edge between a pair of nodes, then the edge is called a “positive link”, which is corresponding to class 1. In contrast, if there is no edge between a pair of nodes, then it is treated as a “negative link”, which is corresponding to class 0. The prediction requires two steps: (1) an encoder to construct node embeddings based on the positive links; (2) a decoder to aggregate embeddings of target node pairs by taking inner products. An illustration of the prediction process is provided in Figure 11.

Figure: Figure 11: Illustration of the link prediction process. This figure is adapted from a nice tutorial in [https://github.com/tomonori-masui/graph-neural-networks/blob/main/gnn_pyg_implementations.ipynb](https://github.com/tomonori-masui/graph-neural-networks/blob/main/gnn_pyg_implementations.ipynb).
Refer to caption: extracted/5909359/Figures/LP_illustration.png

The train/validation/test split is conducted on the edge level. More specifically, we randomly select 5% edges as validation edges and 10% edges as test edges, and the rest will be training edges. The selected edges are the positive links, and the negative links will be randomly sampled from disconneted node pairs. The number of negative egdes is set to be the same as the number of positive edges, which enables us to use accuracy as the metric. We present how the training, validation, and testing edges are used in Figure 12.

Figure: Figure 12: Edges used in the encoder and decoder during training, validation, and testing. This figure is adapted from a nice tutorial in [https://github.com/tomonori-masui/graph-neural-networks/blob/main/gnn_pyg_implementations.ipynb](https://github.com/tomonori-masui/graph-neural-networks/blob/main/gnn_pyg_implementations.ipynb).
Refer to caption: extracted/5909359/Figures/LP_edges.png

Target model architectures.
All target models are used as the encoder in the link prediction task. Target GNNs all adopt 2-convolutional layer structure, while the MLP is composed of two linear layers. For APPNP, we apply a linear layer after the 2 graph convolutional layers to reduce the embedding dimension. The ReLU activation function is applied in the encoder when applicable. The output dimension of the encoder is the same as the hidden size of the target model, and is treated as a hyperparameter that requires tuning. The decoders are the same for all target models, taking inner products of target node pairs and then passing to the Sigmoid activation function for final prediction.

## Appendix H Hyperparameter Tuning Search Space

### H.1 Full Search Spaces - Basic Node Classification

Training-related Hyperparameters

optimizer: [‘SGD’, ‘Adam’]

learning_rate: [0.005, 0.01, 0.05, 0.1, 0.5, 1]

weight_decay: [0.0005, 0.005, 0.05]

dropout: [0.1, 0.3, 0.5, 0.7, 0.9]

scheduler: [‘MultiStepLR’]

scheduler_milestone: [[100, 125, 150, 175]]

scheduler_gamma: [0.2]

Model-related Hyperparameters

MLP: hidden_size [16, 32, 64]

GCN: hidden_size [16, 32, 64]

SAGE: hidden_size [16, 32, 64]

APPNP: teleport_probability [0.1, 0.3, 0.5, 0.7, 0.9]

GAT: hidden_size [16, 32, 64]; num_heads [4, 8, 12]

ChebNet: hidden_size [16, 32, 64]; filter_size [1, 2, 3]

H2GCN: hidden_size [16, 32, 64]

SGC: num_hops [1, 2, 3]

GPRGNN: init [‘Random’, ‘PPR’], GPR_alpha [0.1, 0.2, 0.5, 0.9]

MixHop: hidden_size [16, 32, 64]; power_set [{0, 1, 2}]

### H.2 Full Search Spaces - Node Classification on Large Graphs

Training-related Hyperparameters

optimizer: [‘Adam’]

learning_rate: [0.0005, 0.005, 0.05]

weight_decay: [0.0005, 0.005]

dropout: [0.1, 0.3, 0.7]

scheduler: [‘MultiStepLR’]

scheduler_milestone: [[150, 170]]

scheduler_gamma: [0.2]

Model-related Hyperparameters

GCN: hidden_size [64, 128, 256]

SAGE: hidden_size [64, 128, 256]

APPNP: teleport_probability [0.1, 0.3, 0.5, 0.9]

### H.3 Full Search Spaces - Node Classification on Long-range Graphs

Training-related Hyperparameters

optimizer: [‘AdamW’]

learning_rate: [0.0005, 0.001, 0.005]

weight_decay: [0.0005, 0.005]

dropout: [0.1, 0.3]

scheduler: None

Model-related Hyperparameters

MLP: hidden_size [192, 256], num_layers [8, 10]

GCN: hidden_size [192, 256], num_layers [8, 10]

SAGE: hidden_size [192, 256], num_layers [8, 10]

APPNP: hidden_size [256], num_layers [8, 10], teleport_probability [0.1, 0.5, 0.9]

SGC: hidden_size [192, 256], num_hops [8, 10]

GPRGNN: hidden_size [256], num_layers [8, 10], init [‘PPR’], GPR_alpha [0.1, 0.5, 0.9]

MixHop: hidden_size [192, 256], power_set [{6, 8, 10}, {8, 10}]

### H.4 Full Search Spaces - Link Prediction

Training-related Hyperparameters

optimizer: [‘SGD’, ‘Adam’]

learning_rate: [0.005, 0.01, 0.05, 0.1, 0.5, 1]

weight_decay: [0.0005, 0.005, 0.05]

dropout: [0.1, 0.3, 0.5, 0.7, 0.9]

scheduler: [‘MultiStepLR’]

scheduler_milestone: [[100, 125, 150, 175]]

scheduler_gamma: [0.2]

Model-related Hyperparameters

MLP: hidden_size [16, 32, 64]

GCN: hidden_size [16, 32, 64]

SAGE: hidden_size [16, 32, 64]

APPNP: hidden_size [16, 32, 64], teleport_probability [0.1, 0.5, 0.9]

ChebNet: hidden_size [16, 32, 64]; filter_size [1, 2, 3]

## Appendix I Supplementary Experiment Results and Discussions

### I.1 More Results of Node Classification on Homophilic Graphs

**Table 17: More results of basic node classification on heterophilic graphs. We report the average test accuracy (%) and the standard deviation of models selected by validation over 10 training or generation runs. The best results are marked in bold.**
| Datasets | Pubmed (Homophily) | Computers (Homophily) |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Models | Grid | Random | Coarse | C2F | GNNDiff | Grid | Random | Coarse | C2F | GNNDiff |
| MLP | 73.83 $\pm$ 0.31 | 73.79 $\pm$ 0.45 | 74.07 $\pm$ 0.87 | 74.11 $\pm$ 0.58 | 74.73 $\pm$ 0.10 | 66.40 $\pm$ 2.18 | 66.79 $\pm$ 1.62 | 67.34 $\pm$ 0.76 | 66.39 $\pm$ 2.01 | 67.57 $\pm$ 0.16 |
| GCN | 79.27 $\pm$ 0.40 | 79.09 $\pm$ 0.47 | 78.70 $\pm$ 1.09 | 79.09 $\pm$ 0.47 | 79.30 $\pm$ 0.31 | 82.42 $\pm$ 0.78 | 80.69 $\pm$ 1.42 | 80.35 $\pm$ 1.39 | 79.07 $\pm$ 0.42 | 82.39 $\pm$ 0.29 |
| SAGE | 77.26 $\pm$ 0.34 | 77.55 $\pm$ 0.35 | 77.61 $\pm$ 0.41 | 77.27 $\pm$ 0.62 | 77.65 $\pm$ 0.11 | 80.19 $\pm$ 1.89 | 78.92 $\pm$ 1.87 | 78.69 $\pm$ 3.06 | 80.19 $\pm$ 1.89 | 79.80 $\pm$ 0.22 |
| APPNP | 79.15 $\pm$ 0.70 | 79.15 $\pm$ 0.61 | 78.72 $\pm$ 0.54 | 77.55 $\pm$ 0.89 | 79.28 $\pm$ 0.20 | 82.77 $\pm$ 1.55 | 82.35 $\pm$ 1.58 | 83.30 $\pm$ 0.81 | 83.37 $\pm$ 1.62 | 83.93 $\pm$ 0.58 |
| GAT | 78.45 $\pm$ 0.69 | 78.33 $\pm$ 0.64 | 78.08 $\pm$ 0.43 | 78.46 $\pm$ 0.68 | 78.40 $\pm$ 0.03 | 82.70 $\pm$ 0.85 | 82.43 $\pm$ 0.93 | 82.06 $\pm$ 0.58 | 82.94 $\pm$ 0.96 | 83.16 $\pm$ 0.44 |
| ChebNet | 78.29 $\pm$ 0.71 | 77.30 $\pm$ 0.59 | 77.82 $\pm$ 0.35 | 77.45 $\pm$ 0.70 | 78.31 $\pm$ 0.58 | 73.76 $\pm$ 2.77 | 69.58 $\pm$ 1.82 | 69.56 $\pm$ 1.81 | 69.57 $\pm$ 1.78 | 70.12 $\pm$ 0.66 |
| H2GCN | 79.18 $\pm$ 0.38 | 79.46 $\pm$ 0.12 | 78.99 $\pm$ 0.42 | 79.01 $\pm$ 0.21 | 79.76 $\pm$ 0.47 | 77.29 $\pm$ 1.36 | 75.21 $\pm$ 3.50 | 73.79 $\pm$ 4.34 | 75.02 $\pm$ 2.39 | 77.81 $\pm$ 0.92 |
| SGC | 78.50 $\pm$ 0.16 | 77.44 $\pm$ 0.77 | 78.32 $\pm$ 0.04 | 78.91 $\pm$ 0.57 | 79.10 $\pm$ 0.19 | 80.37 $\pm$ 1.65 | 80.91 $\pm$ 1.56 | 81.88 $\pm$ 0.91 | 81.79 $\pm$ 0.63 | 82.26 $\pm$ 0.58 |
| GPRGNN | 79.56 $\pm$ 0.56 | 79.35 $\pm$ 0.42 | 79.19 $\pm$ 0.83 | 79.22 $\pm$ 0.83 | 79.69 $\pm$ 0.08 | 82.42 $\pm$ 0.60 | 82.79 $\pm$ 1.40 | 82.58 $\pm$ 0.58 | 82.58 $\pm$ 0.58 | 82.80 $\pm$ 0.30 |
| MixHop | 76.96 $\pm$ 0.94 | 76.42 $\pm$ 0.34 | 76.91 $\pm$ 0.80 | 76.96 $\pm$ 0.69 | 78.03 $\pm$ 0.21 | 73.62 $\pm$ 1.44 | 73.00 $\pm$ 0.96 | 72.76 $\pm$ 1.57 | 72.57 $\pm$ 0.96 | 75.64 $\pm$ 0.13 |
| Datasets | Photo (Homophily) | CS (Homoophily) |  |  |  |  |  |  |  |  |
| Models | Grid | Random | Coarse | C2F | GNNDiff | Grid | Random | Coarse | C2F | GNNDiff |
| MLP | 80.61 $\pm$ 0.64 | 80.46 $\pm$ 0.69 | 80.45 $\pm$ 0.69 | 80.45 $\pm$ 0.69 | 81.69 $\pm$ 0.02 | 88.32 $\pm$ 0.60 | 87.45 $\pm$ 0.91 | 86.90 $\pm$ 1.28 | 87.59 $\pm$ 0.95 | 87.63 $\pm$ 0.01 |
| GCN | 90.83 $\pm$ 0.45 | 90.81 $\pm$ 0.37 | 90.96 $\pm$ 0.47 | 91.10 $\pm$ 1.57 | 91.41 $\pm$ 0.01 | 91.34 $\pm$ 0.19 | 91.04 $\pm$ 0.34 | 91.25 $\pm$ 0.14 | 91.17 $\pm$ 0.14 | 91.35 $\pm$ 0.02 |
| SAGE | 90.32 $\pm$ 0.55 | 90.27 $\pm$ 0.70 | 89.62 $\pm$ 0.91 | 88.35 $\pm$ 1.35 | 90.36 $\pm$ 0.19 | 90.98 $\pm$ 0.33 | 90.95 $\pm$ 0.32 | 90.75 $\pm$ 0.58 | 90.91 $\pm$ 0.17 | 91.01 $\pm$ 0.40 |
| APPNP | 90.43 $\pm$ 0.77 | 90.34 $\pm$ 0.94 | 90.12 $\pm$ 0.45 | 90.29 $\pm$ 0.89 | 90.99 $\pm$ 0.50 | 92.13 $\pm$ 0.25 | 92.09 $\pm$ 0.30 | 91.73 $\pm$ 0.20 | 92.13 $\pm$ 0.25 | 92.30 $\pm$ 0.02 |
| GAT | 90.25 $\pm$ 0.92 | 90.24 $\pm$ 0.71 | 89.71 $\pm$ 0.89 | 90.11 $\pm$ 1.06 | 91.01 $\pm$ 0.48 | 90.35 $\pm$ 0.46 | 89.81 $\pm$ 0.78 | 89.52 $\pm$ 0.9 | 89.73 $\pm$ 1.14 | 90.57 $\pm$ 0.08 |
| ChebNet | 87.73 $\pm$ 2.87 | 84.88 $\pm$ 2.13 | 83.85 $\pm$ 3.07 | 86.28 $\pm$ 1.81 | 85.98 $\pm$ 0.13 | 91.79 $\pm$ 0.53 | 91.76 $\pm$ 0.65 | 91.55 $\pm$ 0.39 | 91.82 $\pm$ 0.35 | 91.90 $\pm$ 0.04 |
| H2GCN | 90.30 $\pm$ 0.80 | 89.99 $\pm$ 0.60 | 89.17 $\pm$ 1.47 | 90.29 $\pm$ 0.76 | 90.83 $\pm$ 0.44 | 92.24 $\pm$ 0.32 | 92.11 $\pm$ 0.13 | 91.27 $\pm$ 0.41 | 92.07 $\pm$ 0.24 | 92.24 $\pm$ 0.04 |
| SGC | 90.27 $\pm$ 0.60 | 90.47 $\pm$ 0.52 | 90.46 $\pm$ 0.56 | 90.24 $\pm$ 1.20 | 90.65 $\pm$ 0.17 | 91.59 $\pm$ 0.12 | 90.99 $\pm$ 0.16 | 91.37 $\pm$ 0.23 | 91.02 $\pm$ 0.38 | 91.46 $\pm$ 0.09 |
| GPRGNN | 91.18 $\pm$ 0.38 | 90.28 $\pm$ 1.26 | 90.07 $\pm$ 1.20 | 91.24 $\pm$ 0.59 | 91.26 $\pm$ 0.28 | 90.94 $\pm$ 0.44 | 90.81 $\pm$ 0.44 | 90.66 $\pm$ 0.53 | 90.75 $\pm$ 2.01 | 90.93 $\pm$ 0.14 |
| MixHop | 86.31 $\pm$ 1.31 | 85.04 $\pm$ 1.17 | 84.47 $\pm$ 0.73 | 84.67 $\pm$ 1.39 | 86.12 $\pm$ 0.60 | 92.10 $\pm$ 0.33 | 92.09 $\pm$ 0.33 | 91.05 $\pm$ 0.72 | 91.82 $\pm$ 0.51 | 92.23 $\pm$ 0.05 |

### I.2 More Results of Node Classification on Heterophilic Graphs

**Table 18: More results of basic node classification on homophilic graphs. We report the average test accuracy (%) and the standard deviation of models selected by validation over 10 training or generation runs. The best results are marked in bold.**
| Datasets | Roman-Empire (Heterophily) | Amazon-Ratings (Heterophily) |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Models | Grid | Random | Coarse | C2F | GNNDiff | Grid | Random | Coarse | C2F | GNNDiff |
| MLP | 66.19 $\pm$ 0.17 | 65.32 $\pm$ 0.36 | 65.27 $\pm$ 0.29 | 65.27 $\pm$ 0.29 | 66.11 $\pm$ 0.08 | 40.20 $\pm$ 0.52 | 38.52 $\pm$ 0.68 | 38.56 $\pm$ 0.16 | 39.60 $\pm$ 0.35 | 39.82 $\pm$ 0.20 |
| GCN | 48.87 $\pm$ 0.20 | 47.44 $\pm$ 0.57 | 47.43 $\pm$ 0.42 | 47.70 $\pm$ 0.51 | 48.43 $\pm$ 0.17 | 43.10 $\pm$ 0.13 | 41.52 $\pm$ 0.28 | 41.16 $\pm$ 0.16 | 43.11 $\pm$ 0.33 | 43.19 $\pm$ 0.14 |
| SAGE | 78.12 $\pm$ 0.22 | 77.32 $\pm$ 0.29 | 76.91 $\pm$ 0.32 | 76.91 $\pm$ 0.32 | 77.65 $\pm$ 0.12 | 43.23 $\pm$ 0.52 | 43.27 $\pm$ 0.51 | 42.78 $\pm$ 0.57 | 43.03 $\pm$ 0.52 | 43.32 $\pm$ 0.06 |
| APPNP | 58.23 $\pm$ 0.19 | 58.14 $\pm$ 0.21 | 57.71 $\pm$ 0.06 | 58.23 $\pm$ 0.19 | 58.36 $\pm$ 0.34 | 38.79 $\pm$ 0.17 | 39.03 $\pm$ 0.13 | 38.86 $\pm$ 0.63 | 38.86 $\pm$ 0.63 | 39.26 $\pm$ 0.16 |
| GAT | 56.86 $\pm$ 0.58 | 55.53 $\pm$ 1.26 | 55.31 $\pm$ 0.73 | 56.10 $\pm$ 1.00 | 55.40 $\pm$ 0.04 | 44.74 $\pm$ 0.30 | 44.41 $\pm$ 0.46 | 44.39 $\pm$ 0.37 | 44.42 $\pm$ 0.35 | 44.77 $\pm$ 0.04 |
| ChebNet | 80.05 $\pm$ 0.27 | 79.09 $\pm$ 0.40 | 78.63 $\pm$ 0.21 | 80.05 $\pm$ 0.27 | 80.06 $\pm$ 0.04 | 44.56 $\pm$ 0.16 | 42.87 $\pm$ 0.35 | 43.96 $\pm$ 0.28 | 43.93 $\pm$ 0.29 | 44.04 $\pm$ 0.18 |
| H2GCN | 63.63 $\pm$ 2.30 | 62.96 $\pm$ 0.65 | 62.12 $\pm$ 1.19 | 63.32 $\pm$ 1.27 | 63.94 $\pm$ 0.07 | 43.24 $\pm$ 0.27 | 43.09 $\pm$ 0.26 | 42.65 $\pm$ 0.08 | 42.76 $\pm$ 0.29 | 42.83 $\pm$ 0.15 |
| SGC | 39.10 $\pm$ 0.17 | 39.02 $\pm$ 0.12 | 38.28 $\pm$ 0.05 | 39.01 $\pm$ 0.12 | 38.47 $\pm$ 0.16 | 39.96 $\pm$ 0.53 | 39.70 $\pm$ 0.27 | 39.56 $\pm$ 0.18 | 39.37 $\pm$ 0.17 | 39.60 $\pm$ 0.12 |
| GPRGNN | 71.70 $\pm$ 0.31 | 71.63 $\pm$ 0.46 | 69.82 $\pm$ 0.42 | 69.81 $\pm$ 0.44 | 71.76 $\pm$ 0.07 | 44.13 $\pm$ 0.38 | 44.06 $\pm$ 0.48 | 43.95 $\pm$ 0.39 | 44.17 $\pm$ 0.52 | 44.24 $\pm$ 0.17 |
| MixHop | 77.99 $\pm$ 0.14 | 76.95 $\pm$ 0.31 | 74.10 $\pm$ 0.13 | 79.01 $\pm$ 0.16 | 74.94 $\pm$ 0.13 | 43.03 $\pm$ 0.32 | 42.61 $\pm$ 0.37 | 41.25 $\pm$ 0.16 | 41.64 $\pm$ 0.25 | 42.29 $\pm$ 0.07 |
| Datasets | Minesweeper (Heterophily) | Tolokers (Heterophily) |  |  |  |  |  |  |  |  |
| Models | Grid | Random | Coarse | C2F | GNNDiff | Grid | Random | Coarse | C2F | GNNDiff |
| MLP | 80.00 $\pm$ 0.00 | 79.97 $\pm$ 0.08 | 79.94 $\pm$ 0.12 | 79.72 $\pm$ 0.09 | 80.00 $\pm$ 0.00 | 78.22 $\pm$ 0.04 | 78.18 $\pm$ 0.03 | 78.16 $\pm$ 0.01 | 78.17 $\pm$ 0.02 | 78.24 $\pm$ 0.05 |
| GCN | 80.26 $\pm$ 0.10 | 80.26 $\pm$ 0.12 | 80.19 $\pm$ 0.10 | 80.28 $\pm$ 0.14 | 80.28 $\pm$ 0.07 | 78.65 $\pm$ 0.07 | 78.67 $\pm$ 0.09 | 78.63 $\pm$ 0.10 | 78.67 $\pm$ 0.05 | 78.73 $\pm$ 0.08 |
| SAGE | 85.63 $\pm$ 0.20 | 85.56 $\pm$ 0.21 | 85.23 $\pm$ 0.27 | 85.63 $\pm$ 0.20 | 85.78 $\pm$ 0.32 | 78.25 $\pm$ 0.20 | 78.44 $\pm$ 0.12 | 78.34 $\pm$ 0.21 | 78.47 $\pm$ 0.11 | 78.51 $\pm$ 0.06 |
| APPNP | 80.20 $\pm$ 0.07 | 80.11 $\pm$ 0.13 | 79.96 $\pm$ 0.08 | 80.01 $\pm$ 0.13 | 80.26 $\pm$ 0.02 | 78.62 $\pm$ 0.80 | 78.61 $\pm$ 0.04 | 78.56 $\pm$ 0.12 | 78.63 $\pm$ 0.04 | 78.63 $\pm$ 0.03 |
| GAT | 81.98 $\pm$ 0.62 | 81.34 $\pm$ 0.53 | 81.18 $\pm$ 0.37 | 81.81 $\pm$ 0.73 | 82.31 $\pm$ 0.06 | 79.73 $\pm$ 0.50 | 79.50 $\pm$ 0.61 | 78.34 $\pm$ 0.59 | 78.35 $\pm$ 0.87 | 79.80 $\pm$ 0.12 |
| ChebNet | 86.82 $\pm$ 0.29 | 86.64 $\pm$ 0.38 | 84.66 $\pm$ 1.26 | 86.86 $\pm$ 0.20 | 86.89 $\pm$ 0.27 | 79.43 $\pm$ 0.60 | 78.83 $\pm$ 0.49 | 78.56 $\pm$ 0.13 | 78.68 $\pm$ 0.14 | 79.66 $\pm$ 0.19 |
| H2GCN | 83.59 $\pm$ 1.03 | 83.31 $\pm$ 0.33 | 82.67 $\pm$ 0.92 | 83.61 $\pm$ 0.49 | 83.68 $\pm$ 0.32 | 78.83 $\pm$ 0.52 | 78.56 $\pm$ 0.13 | 78.37 $\pm$ 0.25 | 78.51 $\pm$ 0.09 | 78.80 $\pm$ 0.07 |
| SGC | 81.51 $\pm$ 0.11 | 81.61 $\pm$ 0.17 | 81.46 $\pm$ 0.20 | 81.55 $\pm$ 0.22 | 81.63 $\pm$ 0.24 | 78.51 $\pm$ 0.02 | 78.49 $\pm$ 0.03 | 78.47 $\pm$ 0.09 | 78.63 $\pm$ 0.04 | 78.57 $\pm$ 0.03 |
| GPRGNN | 83.92 $\pm$ 0.52 | 83.92 $\pm$ 0.52 | 83.94 $\pm$ 0.56 | 83.94 $\pm$ 0.56 | 83.97 $\pm$ 0.03 | 78.38 $\pm$ 0.27 | 78.36 $\pm$ 0.21 | 78.28 $\pm$ 0.16 | 78.37 $\pm$ 0.22 | 78.28 $\pm$ 0.06 |
| MixHop | 83.52 $\pm$ 0.14 | 82.75 $\pm$ 0.20 | 83.46 $\pm$ 0.21 | 83.86 $\pm$ 0.14 | 83.92 $\pm$ 0.12 | 79.51 $\pm$ 0.15 | 78.7 $\pm$ 0.33 | 79.22 $\pm$ 0.24 | 79.51 $\pm$ 0.15 | 79.54 $\pm$ 0.41 |

### I.3 GNN-Diff Time Analysis

We provide the details of the time costs of GNN-Diff in Figure 13. In general, the coarse search consumes most of the time taken by the entire GNN-Diff process. This is the inevitable cost that one shall expect for GNN-Diff to generate high-performing GNNs, though the coarse search is much more efficient compared to the baseline search methods. The parameter collection time is almost negligible for small graphs such as Cora and Actor. In contrast, it takes much longer to collect parameters for large-scale tasks, such as the node classification on PascalVOC-SP. This is due to the long training and validation time associated with large graphs. Similarly, the proportion of sampling time is lower for small graphs and higher for large graphs, because of the longer validation and testing time with large graphs.

Figure: Figure 13: Time analysis of GNN-Diff. The time costs of GNN-Diff include the time for the coarse search, parameter collection, training, and inference.
Refer to caption: extracted/5909359/Figures/Time_analysis.png

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 3
presentation: 3
questions: 1. Could you explain the reason of the claim that concatenating different receptive fields in message passing can handle both homophilic and heterophilic graphs?
soundness: 2
strengths: 1.	The overall motivation is clear: improving hyperparameter search strategies can save time and enhance performance. This is an area that has not been explored too much in GNN training so far.

2.	The presentation is clear, with well-organized results and analysis provided for better interpretability.
summary: The paper introduces GNN-Diff, a latent diffusion model designed to enhance GNN training while minimizing the need for hyperparameter tuning. The author claim that this approach not only improves GNN performance but also demonstrates significant stability across various tasks.
weaknesses: 1.	The reported performance is a significant concern, as many results are noticeably lower than expected. For instance, nearly all results in Table 2 fall substantially short of those reported in existing literature and on the OGBN leaderboard: for example, I believe that GCN on OGBN-Arxiv can achieve 71.6% (as opposed to the reported 69.2%), APPNP can reache 71.8% (versus the reported 55.05%), and GraphSAGE can easily achieve around 79% on OGBN-Products, compared to the reported 75.6%. This discrepancy raises questions about whether the authors adhered to standard settings, whether the reported accuracy is reliable, and, if not, how the experimental setup was configured. Clearer details on these aspects would help clarify the validity of the reported results.

2.	The motivation for certain components is unclear. For instance, why do the authors choose to incorporate GAE in the training process?

3.	An ablation study illustrating the impact of the proposed techniques would strengthen the findings.

4.	The authors use classical GNNs for downstream tasks, but employing SOTA GNNs are necessary. For example, RevGAT[1], SAGN[2] and LazyGNN[3]. Many advanced architectures perform well without extensive parameter tuning, and using these models could better showcase the contributions of this work.

5.	Figure 7 reports only running time, without any indication of memory cost. Additionally, if three different models are trained in this work, I am wondering how the runtime remains lower than that of a random search? A comparison with coarse search would also be useful.

6.	Can the proposed techniques be applied to graph classification?

[1] Training Graph Neural Networks with 1000 Layers

[2] Scalable and Adaptive Graph Neural Networks with Self-Label-Enhanced training

[3] LazyGNN: Large-Scale Graph Neural Networks via Lazy Propagation

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: The upper bound of the proposed method remains unclear. It is uncertain how a well fine-tuned GCN compares to lightly fine-tuned, more expressive models.
soundness: 2
strengths: - This paper is well-organized.
- Extensive experiments are conducted in this paper.
summary: In summary， this paper proposes a strategy for tuning hyper-parameters of GNNs to maximize their potential. It is achieved by collecting a set of model parameters, encoding graphs, and then learning to generate parameters for GNNs. Extensive methods have been conducted in this paper to verify the designed method.
weaknesses: 1. The technical contribution is limited given the related works.
It seems that GNN-diff is a combination of LDM and graph techniques. The sample collection method described in Section 4.1 is trivial, and the designed encoder is also widely used in existing methods. It would be better to highlight the technical contributions in comparison with the different methods mentioned in the related work section.

2. The experiments seem somewhat unrigorous and fail to adequately highlight the efficiency and effectiveness of GNN-diff. 
- Baseline selection: While numerous target GNNs and datasets are utilized, the paper employs only a few comparable baselines. It appears that methods from "Advanced Methods for Hyperparameter Tuning" should be incorporated to validate the designed parameter tuning strategy more effectively. 
-  Evaluations of the effectiveness of Coarse search: In Tables 1-4, the "Coarse" variant struggles to outperform the "Random" and "Grid" baselines. With these results, justifying the effectiveness of the coarse search approach is challenging.

3. The transferability of GNN-Diff is limited due to its hyper-parameter tuning being tailored for a specific dataset, relying on parameter collection and the GAE model.

---

confidence: 1
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. There is a spelling error at line 126 of the article, "P-Eecoder()".
2. Why use such a GNN architecture? Would using classical graph embedding frameworks like GAT or GCN be effective?
3. Clearly, graph data structures are diverse, and using just one type of GNN makes it difficult to effectively embed all types of graph data. When GNNs become ineffective, or when increasing the number of GNN layers causes the graph embeddings to become too smooth, what impact will that have on the method? Since this paper proposes a parameter tuning method, which can be considered a fundamental approach, we need to focus more on when it will fail or the lower limits of the method’s capabilities.
soundness: 3
strengths: 1. The overall expression of the article is clear and easy to understand.
2. The experiments are comprehensive, and the performance of the algorithm is impressive.
3. It connects model parameter tuning with the data itself.
summary: This article introduces an interesting hyperparameter search architecture that learns graph embedding information for specific graph tasks through a designed GNN architecture. This information is then integrated into conditional latent denoising diffusion probabilistic models, using a parameter encoding module to flexibly convert the input and output dimensions of the parameters.
weaknesses: Simply using GNNs for embedding learning on graph-structured data and adding it to the G-LDM module does not analyze why this part of the graph embedding works for the method.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 4
questions: Have the authors tried other forms of G-Encoders other than Eq. (2) and conducted ablation studies about it?
soundness: 3
strengths: - The idea of using the diffusion framework to help with hyperparameter tuning seems novel and reasonable.
- The experiments are thorough, spanning a diverse set of tasks and backbone models, with results that are both convincing and reproducible.
- The proposed method effectively enhances both speed and performance across most cases, showcasing its practical value.
summary: This paper proposes GNN-Diff, a graph-conditioned latent diffusion model designed to enhance Graph Neural Networks (GNNs) by generating effective parameters with minimal hyperparameter tuning. This is achieved by utilizing a coarse search strategy that identifies a subset of viable hyperparameters, which then guides the model’s parameter generation. GNN-Diff claims to reduce tuning time and computational cost by creating high-performance GNN parameters through diffusion in a latent space conditioned on graph data. The authors validate GNN-Diff through extensive experiments on various tasks including node classification and link prediction with numerous model backbones, showing improved efficiency and stability.
weaknesses: - The heterophily datasets used in this paper have faced criticism for extensive duplicate nodes as noted in [1]. It would be beneficial for the authors to consider experimenting with the datasets proposed in [1] for a more robust evaluation of heterophilic cases.

- Table 5 shows that, compared to p-Diff, the improvements with GNN-Diff are relatively marginal and often inconsistent, with many results falling within one standard deviation. This brings into question the effectiveness of the GNN-Encoder design (Eq. 2).

- There is a lack of theoretical analysis. 


[1] A critical look at the evaluation of GNNs under heterophily: Are we really making progress?

Meta-review: The paper introduces a method for improving the performance of GNNs with minimal hyperparameter tuning.  It propose a graph-conditioned generative model to generate (better) hyperparameter configurations based on lightly-tuned suboptimal parameters. This approach to parameter tuning is novel as far as I know. While the combination of techniques is compelling, the lack of a fundamentally new contribution limits the overall impact. The experimental evaluation is decent. Although some strongly-relevant baselines were missing, the authors added one such baseline in the rebuttal. 

The biggest weakness of the paper is that the majority of the results are shown with "cluster" training, due to a lack of computational resources. Since, cluster training yields significantly worse performance than full training, the hyperparameters matter more and thus the proposed method looks better than it is. In response to the reviewers' request the authors included results with full training, where the gap to the baselines here is smaller. In my opinion this weakness is almost enough for a rejection, but I still recommend acceptance because the idea is interesting. However, I strongly advise the authors to include a prominent limitation section to highlight the issue.

Decision: Accept (Poster)