Unifying Generation and Prediction on Graphs with Latent Graph Diffusion
Authors: Cai Zhou
Xiyuan Wang
Muhan Zhang
Venue: NeurIPS 2024 poster (2024)
Keywords: latent graph generation
theory of diffusion model
general purpose generative models
Primary area: graph_neural_networks
TL;DR: We propose Latent Graph Diffusion, the first framework that enables solving graph learning tasks of all levels and all types (generation, regression and classification) with one diffusion model.

Abstract: In this paper, we propose the first framework that enables solving graph learning tasks of all levels (node, edge and graph) and all types (generation, regression and classification) using one formulation. We first formulate prediction tasks including regression and classification into a generic (conditional) generation framework, which enables diffusion models to perform deterministic tasks with provable guarantees. We then propose Latent Graph Diffusion (LGD), a generative model that can generate node, edge, and graph-level features of all categories simultaneously. We achieve this goal by embedding the graph structures and features into a latent space leveraging a powerful encoder and decoder, then training a diffusion model in the latent space. LGD is also capable of conditional generation through a specifically designed cross-attention mechanism. Leveraging LGD and the ``all tasks as generation'' formulation, our framework is capable of solving graph tasks of various levels and types. We verify the effectiveness of our framework with extensive experiments, where our models achieve state-of-the-art or highly competitive results across a wide range of generation and regression tasks.

Unifying Generation and Prediction on Graphs with Latent Graph Diffusion
Cai Zhou 1.2, Xiyuan Wang2 Muhan Zhang
Departmeet of Automation, Tsinghua University 2 Department of Electrical Engineering and Computer Science, Massachusetts Institute of Technology Institute for Artificial Intelligence, Peking University "School of Intelligence Science and Technology, Peking University caiz428@mit.edt edu, {wangxiyuan, muhan) ēpku. edu. cn
Abstract
this paper, we propose the first framework that enables solving graph learning tasks of all levels (node, edge and graph) and all types (generation, regression and classification) using one formulation. We first formulate prediction tasks including regression and classification into generic (conditional) generation framework, which enables diffusion models to perform deterministic tasks with provable guarantees. We then propose Latent Graph Diffusion (LGDD,, generative model that can generate node, edge, and graph-level features of all categories simultaneously. We achieve this goal by embedding the graph structures and features into a latent space leveraging powerful encoder and decoder, then training diffusion model in the latent space. LGD is also capable of conditional generation through specifically designed cross-attention mechanism. Leveraging LGD and the "all tasks as generation" formulation, our framework is capable of solving graph tasks of various levels and types. We verify the effectiveness of our framework with extensive experiments, where our models achieve state-of-the- of-the-art or highly competitive results across wide range of generation and regression tasks.
Introduction
Graph generation has become of great importance in recent years, with important applications in many fields, including drug design (Li al. 2018) and social network analysis (Grover et al., 2019]] However, compared with the huge success of generative models in natural language processing [Tou: vron al.] 2023) and computer vision [Rombach et al. 2021]. graph generation is faced with many difficulties due to the underlying non-Euclidean topological structures. Existing models fail to
generate structures and features simultaneously NNiu et al. 2020 Jo al. 2022], model arbitrary
attributes (Vignac et al. 2022).
Moreover, while general purpose foundation models built in NLP [Bommasani al., 2021 which can solve all types of tasks through the generative framework. there are no such general purpose models for graph data, since current graph generative models cannot address regression or classification tasks. People also have to train separate models for different levels (node, edge, and graph) of graph learning tasks. Therefore, it is beneficial and necessary build framework that solve (a) tasks of all types, including generation, classification and regression; (b) tasks all levels, including node, edge, and graph-level.
"Corresponding Author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Graph Space
Space
Zy
Conditioning
Denoising network Fo
x (T-1)
Q
X
2y-1
Zy
Masked Crahhs
Optonaal croas attention
Figure I: Illustration of the Latent Graph Diffusion framework, which capable performing both generation and prediction.
Our work. To overcome the difficulties, we propose Latent Graph Diffusion, the first graph generative framework that is capable of solving tasks of various types and levels. Our main contributions are summarized as follows.
We conceptually formulate regression and classification tasks as conditional generation, and theoretically show that latent diffusion models can address them with provable guarantees.
We present Latent Graph Diffusion (LGD), which applies score-based diffusion generative model in the latent space encoded through powerful pretrained graph encoder, thus overcoming the difficulties brought by discrete graph structures and various feature categories. Moreover, leveraging specially designed graph transformer, LGD is able to generate node, edge, and graph features simultancously in one shot with all feature types. We also design special cross-attention mechanism to enable controllable generation.
Combining the unified task formulation as generation and the powerful latent diffusion models, LGD is able perform both generation and prediction tasks through generative modeling. Experimentally. LGD achieves sate-of-theear or highly competitive results across various tasks including molecule (un)conditional generation as well as regression and classification on graphs. Our code available at
Related Work
Diffusion models. Recently, score-ber based generative models have demonstrated promising per- formance across a wide range of generation tasks. These models start by introducing gradually increasing noise the data, and then learn the reverse procedure which represents the gradient of the log-density function with respect to the data. This process allows them generate samples effectively. Two notable works in the realm of score-based generative models are score-matching with Langevin dynamics (SMLD) [Song and Ermon 2019) and the de- noising diffusion probabilistic model (DDPM) (Ho al. 2020 SMLD estimates the score function at multiple noise scales and generates samples using annealed Langevin dynamics to reduce the noise. In contrast, DDPM models the diffusion process
estimating the score function,
parameterized Markov chain and learns to reverse the forward diffusion process of adding noise. [Song et al. (200bb encapsulates SMLD and DDPM within the framework of stochastic differential equations (SDE), Song and Ermon (2200) and Song et al. (2020a) further improved the scalability and sampling efficiency of score score-based generative methods. More recently, Stable-Diffusion [Rombach et al.) 2021] applied diffusion models to latent space encoded by pretrained autoencoders, significantly improving computational efficiency and the quality of generated images. Our work has further applied latent space to graph tasks and
also improved the generation quality.

---

Diffusion models for graphs. In the context of graph data, Niu et al. [2020] were the first to generate permutation-invarita graphs using score-based methods. They achieved this by introducing Gaussian perturbations to continuous adjacency matrices. [Jo et al. [022] took this step further by proposing method to handle both node attributes and edges simultaneously using stochastic differential equations (SDEs). However, these diffusion models rely on continuous Gaussian noise and do not align well with discrete graph structures. To address this limitation, Haefeli et al.) [2022] designed a diffusion model tailored to unattributed graphs and observed that discrete diffusion is beneficial for graph generation. Addressing this issue even more effectively, DiGress (Vignac et al. 2022], one of the most advanced graph generative models, employs discrete diffusion process. This process progressively adds discrete noise to graphs by either adding or removing edges and altering node categories. Additionally, [immios et al. [2023] proposes method to scale DiGress by sampling a covering of subgraphs within a divide-and-conquee framework. To overcome the difficulties, instead of utilizing discrete noise, our work uses continuous noise in latent space, making model structure well aligned with general diffusion models and achieves better performance. Our generative framework supports both DDPM and SMLD. Given their equivalence in SDE formulations [Song] et al. 2020b). we focus on DDPM and its extensions in our main text for illustration. More details on SMLD-based methods and SDE formulations can be found in Appendix A
3 Regression and Classification as Conditional Generation
In this section, we use the notation x to denote the known data, and y to denote the labels to be predicted in the task. We use this unified notation not only for simplicity; actually, the formulation in this subsection is generic and not limited to graph data. Our framework is the first to show that diffusion models can provably solve regression and classification tasks.
From a high-level perspective, unconditional generation aims to model p(x). while conditional generation aims to model p(xly). For classification and regression tasks, traditional deterministic models give point estimates to minimize classification or regression loss. However, from a generative perspective, predicting point estimation is not the only solution we can also model the complete conditional distribution p(y xx). Since modeling (conditional) distribution is exactly what generative models are capable of (and good at), is possible solve classification and regression tasks with generative models.
Actually, the classification and regression tasks can be viewed as conditional generation tasks. Intuitively, we only need to exchange the positions of data x and y in the conditional diffusion model; then we obtain diffusion model that approximates p(y x). In this case, we use the data x as the condition and want to generate given the condition x. We add Gaussian noise to y in the diffusion process, and we use the denoising network reverse the process. In the parameterization where the denoising network co directly outputs the target instead of the noise analogous to Equation (8) i.e. ý co i t, x), then the diffusion objective is
C(Ee) == Exy. |ea f2x) - y11
It is straightforward to extend the above method to latent diffusion models, where the only difference that the diffusion model operates in the latent space of y. Since the formulation generic, the method is also applicable to graph data; see Section
Now we present our main theorem on the generalization bound of solving regression tasks with latent diffusion models, while more details on formal formulation and proof are given in Appendix B
Theorem 3.1. Suppose Assumption B. Assumption B.2 Assumption B.3 hold, and the step size
h : T/N satisfies h < L/L where L Z 1. Then the mean absolute error (MAE) of the conditional latent diffusion model in the regression task bounded by
MAF < (lw ( () - w T T v)ll + 41
(Lvdh, dh + [(m + m2)h)/ +
Cscore
convergence of forward procens
discretization error
score estimation error
eneoder error
where qa is the ground truth distribution of = w/ (1(x, v) E(x)).
3

---

Converting deterministic tasks such as regression and classification into generation has several potential advantages. First, generative models are capable of modeling the entire distribution, thus being able to provide information confidence interval or uncertainty by multiple predictions; in comparison, traditional deterministic models can only give point estimates. Secondly, generative models are often observed to have larger capacity and generalize well on large datasets [Kadkhodaie (ct al., 2024). Specifically for latent diffusion models, since the generative process is performed in the expressive latent space, the training of diffusion models could be much faster than training deterministic model from scratch, and we obtain a powerful representation of the data that can be used in other downstream tasks. Moreover, now that the deterministic tasks could be unified with generation, thus can be simultaneously addressed by one unified model, which is the precondition of training foundation model. We refer readers to Section and Appendix for detailed experimental observations.
4 Latent Graph Diffusion
As discussed in Section T and Section 2 existing graph diffusion models fail to generate both graph structures and graph features simultancously. In this section, we introduce our novel Latent Graph Diffusion (LGD), a powerful graph generation framework which is the first to enable simultaneous structure and feature generation in one shot. Moreover, LGD is capable of generating features of all levels (node-level, edge-level and graph-level) and all types (discrete or categorical, continuous or real-valued).
Notations. Given n E N. let [n]] denote {1, (....,..., For graph G with n nodes, we denote the node set as ve i [m], and represent the node feature of o2 æ; E Rdd In the context of node pairs (v,, v2), there can be either an edge Egy with its edge feature Eij E IRd. or no edge connecting is and Uz. To jointly model the structures and features (of node, edge, graph level), we treat the case where edge is observed between 2 and " as special edge type. We denote this Cij with augmented feature eiy- = E Rd., where e' is distinct from the features of existing edges, resulting in a total of n² such augmented edges. We represent all node features as X E Rexd, where X; = I;, and augmented edge features as A E Rnxxxdd where A., = Ciy (adjacency matrix with edge features). Graph-level attributes, denoted E 1 can be represented by a virtual node or obtained by pooling over node and edge attributes (see Section 4.2 for details). For brevity, temporarily omit graph-level attributes in the following text. Note that all attributes can be of arbitrary types, including discrete (categorical) and continuous (real-valued).
4.1 Overview
Simultaneous generation of structures and features the original graph space challenging due to discrete graph structures and diverse graph features at various levels and types. Therefore, we apply our graph diffusion models in the latent space H. rather than operating directly on the original graphs. Graphs are encoded in the latent space using powerful pretrained graph encoder & parameterized by é. In this latent space, all n nodes and n² "augmented edges" have continuous latent representations. The node and edge representations are denoted as H = (Z, W), where Z e Raxd and W e mxxx with d representing the hidden dimension.
H (Z, () AA,
To recover the graph structures and features from the latent space H. we employ a light- task-specific decoder Dg. which is designed to be lightweight and is parameterized by E. This decoder produces the final predicted properties of the graph for specific tasks, including node, edge, and graph-level attributes. It is important to note that we consider the absence of an edge as specific type of edge feature. Furthermore, we can derive graph-level attributes from the predicted node features X and edge features A, as explained in detail in Section 4.2
(X, ( (H)
To ensure the quality of generation, the latent space H must meet two key requirements: (a) it should be powerful enough to effectively encode graph information, and (b) it should be conducive to learning diffusion model [Rombach et al. 2021], thus should not be too scattered nor nigh-dimensional.

---

Consequently, both the architectures and training procedures of the encoder &g and the decoder Dg are of utmost importance. We will dive into the architecture and training details in Section 4.2
After pretraining the encoder and decoder, we get a powerful latent space that is suitable for training a diffusion model. We now train the generative model c parametrized by I operating on the latent space. As explained, the generative model could have arbitrary backbone. including SDE-based and ODE-based methods. For illustration,
still consider Denoising Diffusion Probabilistic Model (DDPM) (Ho et al.] 2020). while other generative methods can be analyzed similarly. In the forward diffusion process, we gradually add Gaussian noise to the latent representation H = (Z. W) until the latent codes converge to Gaussian. In the backward denoising process, we use a denoising model parameterized by Eg parametrized by 8 to reconstruct the latent representation of the imput graph. obtaining H. The network can be a general GNN or graph transformer, depending on tasks and conditions (see Appendix C.3) In the most general case, we use a graph transformer enhanced by expanded edges; see Section for architecture and training details.
In conclusion, we have successfully overcome the challenges posed by discrete graph structures and diverse attribute types by applying a diffusion model in the latent space. Our model the first to be capable of simultancously generating all node, edge, and graph-level attributes in single step. The overview of our latent graph diffusion is shown in Figure
4.2 Autoencoding
Architecture. In principle, the encoder can adopt various architectures, including Graph Neural Networks (GNNs) and graph transformers. However, generate structures and features at all levels effectively, the encoder must be able to represent node, edge, and graph-level features simultaneouslly As discussed in Section 4.1 to generate samples of high quality, the latent space H. should be both powerful and suitable for training a diffusion model. To fulfill these criteria, we employ specially designed augmented edge-enhanced graph transformer throughout this paper, as elaborated in Section 4.3 In cases where input graph structures and features are complete, we also allow the use of message-passingg bassed Graph Neural Networks (MPNNs) and positional encodings (PEs) during the pretraining stage. However, these techniques may not be applicable for the denoising network, as the structures of the noised graphs become corrupted and ill-defined.
Since the latent space is already powerful, the decoder can be relatively simple. The decoder is task-specific, and we pretrain one for each task along with the encoder. In generation tasks, the decoder is responsible for recovering the structure and features of the graph. In classification tasks or regression tasks, it serves as a classifier or regression layer, respectively. In all cases, we set the decoder for each task as a single linear layer.
Training, Following Rombach et al. [022 and Xu et al. 2023 we jointly train the encoder and decoder during the pretraining stage. This training process can be carried out in an end-to-end manner or similar to training autoencoders. The training objective encompasses reconstruction loss and regularization penalty.
= + Freg
(6)
The decoder reconstructs the input graph from the latent representation, denoted (X, A) D(H) D(E(X, A)). The specific form of the reconstruction loss Creeon depends on the type of data, such as cross-entropy or Mean Squared Error (MSE) loss. To force the encoder to learn meaningful representations, we consider the following reconstruction tasks: (i) reconstruct node features X from node representations Z: (ii) reconstruct edge features A from edge representations W; (ii) reconstruct edge features eij from representations of node and j; (iv) reconstruct node features T; from edge representations e,.; (v) reconstruct tuples (x;, from edge representations eij. When applicable, we opt to reconstruct node-level and edge-level positional encodings (PEs) according to the latent features. We can also employ masking techniques to encourage the encoder to learn meaningful representations. It is worth noting that downstream classification regression tasks can be seen as a special case of reconstructing the masked graph; for more details, refer to Section 5
To prevent latent spaces from having arbitrarily high variance, we apply regularization. This reg- ularization can take the form of a KL-ppnalty aimed at pushing the learned latent representations towards a standard normal distribution (referred to KL-reg), similar to the Variational Autoencoder
5

---

(VAE) Kingma and Welling 2013]. Alternatively, we can employ vector quantization technique (referred as VQ-reg), seen van den Oord et 2017 Yang al. 2023a
4.3 Diffusion model
Architecture. Note that during both the forward and backward diffusion processes, we either add noise the latent representations of nodes and edges attempt to denoise them. When need to generate the graph structures, the edge indexes are unknown, making the corrupted latent representations do not have clear correspondence with the originaledge information. Consequently, message-passing neural networks (MPNNs) and similar methods are not applicable, as their operations rely on clearly defined edge indexes.
To address the ambiguity of edges in the noised graph, we require operations that do not depend on edge indexes. Graph transformers prove be suitable in this context, they compute attention between every possible node pair and do not necessitate edge indexes. However, we can still incorporate representations of augmented edges into the graph transformers through special design. To facilitate the joint generation of node, edge, and graph-level attributes, we adopt the most general graph transformer with augmented edge enhancement. We design a novel self-attention mechanism that could update node, edge and graph representations as described in Equation (58) see Appendix C.T for details.
Training and sampling. Training the latent diffusion model follows a procedure similar to the one outlined in Appendix(A] with the primary distinction being that training is conducted in the latent space. During this stage, both the encoder and the decoder remain fixed. Denoting Ho = = A), in the forward process, we progressively introduce noise the latent representation, resulting in H..
LLGD(CO) = ( [Ico (H,,t) - c1122]
where we implement the function co (1) described in Equation (8
using time-conditional trans- former, as detailed in Equation (58). There is also an equivalent training objective in which the model co directly predicts Ho based on H, and f, rather than predicting 6t.
(LGD(E)) = Eg(x .A) | [ H,,t)
Holl
(8)
In the inference stage, we directly sample Gaussian noise in the latent space H. and sample H from the generative processes iteratively as described in Appendix [A] After we get the estimated denoised latent representation H. we use the decoder to recover the graph from the latent space, which finishes the generation process.
4.4 Conditional generation
Similarly to other generative models [Rombach al. 2021 our LGG also capable of controllable generation according to given conditions y by modeling conditional distributions p(h y). where h is the random variable representing the entire latent graph representation for simplicity. This can be implemented with conditional denoising networks eø (h, t, y). which take the conditions as additional inputs. Encoders and decoders can also be modified to take y as additional input to shift latent representations towards the data distribution aligned with the condition y.
General conditions. Formally, we preprocess y from various data types and even modalities (e.g. class labels or molecule properties) using domain specific encoder obtaining the latent embedding of the condition r(y) E mRd where m is the number of conditions and d- the hidden dimension. The condition could be embedded into the score network through various methods, including simple addition or concatenation operations [Xu et al., 2023) and cross-attention mechanisms (Rombach et al. 202T]. We provide details of the cross-attention mechanisms in Appendix C.I
The conditional LGD is learned via
LLGD == Eg(x A)..A [[loo (H,,H,,,,,T
c.
the denoising network is parameterized to predict the noise; eg can also predict Ho conditioning on H t, T(y) analogous to Equation

---

Masked graph. Consider the case where we have graph with part of its features known, and we want to predict the missing labels which can be either node, edge, or graph level. For example, in molecule property prediction task where the molecule structures and atom/bond types are known, we want to predict its property. This can be modeled as problem of predicting masked graph-level feature (molecule property) given the node and edge features. In this case, the condition is the latent representation of graph that is partially masked. We denote the features of a partially masked graph as (X AA, gd), where the superscript implies that the partially masked graph is the "condition", x = x; A = eij. _ are observed node, edge and graph-level features respectively. The missing labels we aim to predict are denoted as y. then the complete graph (X, A. g) can be recovered by (X",A,, ge) and y. Intuitively, this is similar to image inpainting, where hope reconstruct the full image based on a partially masked one. Therefore, we can model the common classification and regression tasks
conditional generation, where we want predict the label y (or equivalently, the full graph feature (X, A. g)) given the condition (X", A g.). see Section for formal formulations. In this case, as the condition is graph well (though it might be partially masked), we propose novel gross-attention for graphs as shown in Equation (60), see Appendix C.1 for more details.
5 Unified Task Formulation as Generation
Based on graph generative model on the latent space, we can address generation tasks of all levels (node, edge, and graph) using one LGD model. In this section, we detail how tasks of different types can be formulated as generation, thus becoming tractable using one generative model.
Latent Graph Diffusion framework, which is able to (1) solve tasks of all types, including generation, regression, and classification through the framework of graph generation; and (2) solve tasks of all levels, including node, edge and graph level. See Figure I for illustration.
Now summarize
First, since LGD is an internally generative model, it is able to perform unconditional and conditional generation as discussed in Section 4 For nongenerative tasks including regression and classification, we formulate them into conditional generation in Section X thus can also be solved by our LGD model. In particular for graph data, we want to predict the labels y which can be node, edge, or graph level and of arbitrary types, given the condition of a partially observed graph X A g" g To better align with the second target of full graph generation, we model this problem as predicting p(X, A. X g") where (X,A, g) = (X A.. y) is the full graph feature that combines the observed conditions and the labels to be predicted. In other words, the condition can be viewed as a partially masked graph whose masked part is the labels y. and our task becomes graph inpainting, ie. generate full graph features conditioning on partially masked features.
Now we have formulated all tasks as generative modeling. The feasibility of the second goal to predict all-level features is naturally guaranteed by the ability of our augmented-edge enhanced transformer architecture described in Section We leave the discussion of the possibility of solving tasks from different domains to Appendix D.2
Experiments
In this section, extensive experiments that cover tasks of different types (regression and classification) and levels (node, edge and graph) to verify the effectiveness of LGD. We first conduct experiments on traditional generation tasks to verify LGD's generation quality. We consider both unconditional generation and conditional generation tasks. We then show that utilizing our unified formulation, LGD can also perform well prediction tasks. To the best of our knowledge, are the first to address regression and classification tasks with a generative model efficiently. More experimental details and additional experimental results can be found in Appendix C
6.1 Generation task
For both unconditional and conditional generation, we use QM9 (Ramakrishnan et al. 2014]. one of the most widely adopted datasets in molecular machine learning research, which suitable for both generation and regression tasks. QM9 contains both graph and 3D structures together with several quantum properties for 130k small molecules, limited to 9 heavy atoms. We provide more generation results on larger dataset MOSES (Polykovskiy et al. 2020 in Appendix
7

---

Table 1: Unconditional generation results on QM9.
Model
Validity(%)1
Uniqueness($of
FCD4
NSPDK4
Novelty(%)1
MoFlow
91.36
98.65
4.47
0.017
94.72
GraphAF
74,43
88.64
5.27
0.020
86.59
GraphDF
93.88
98.58
10.93
0.064
98.54
GDSS
95.72
98.46
2.9
0.003
86.27
DiGress
99.01
96.34
0.25
0.0003
35.46
HGGT
99.22
95.65
0.40
0.0003
24.01
GruM
99.69
96.90
0.11
0.0002
24.15
LGD-small (ours)
98,46
97.53
0.32
0.0004
56.35
LGD-large (ours)
99.13
96.82
0.10
0.0002
38.56
Table Conditional generation results on QM9 (MAE L
Model
Qt
ÉHOMO
ÉLUMO
Åe
Gy
Xu et al. 2023
0.043
0.10
39
36
64
0,040
W (ours)
0.058
0.06
18
24
28
0.038
Random
1.616
9.01
645
1457
1470
6,857
N stom
1.053
3.86
426
813
866
1.971
EDM
1.111
2.76
356
584
655
1.101
GeoLDM
1.108
2.37
340
522
587
1.025
LGD (ours)
0.879
2.43
313
641
586
1.002
Unconditional generation. In the unconditional molecular modeling and generation task, we measure the model's ability to learn the distribution of molecular data and generate molecules that are both chemically valid and structurally diverse. Following the common setting [Xu al., 2003)). we generate 10k molecules in the evaluation stage, and report the validity and uniqueness, which are the percentages of valid (measured by RDKIT) and unique molecules among the generated molecules. We also report neighborhood subgraph pairwise distance kernel (NSPDK) and Frechet ChemNet Distance (FCD) metrics to measure the similarity between generated samples and the test set. We leave further discussions relaxed validity [Jo al. 2022 and novelty in Appendix C For bascline
models, we select classical and recent strong models including MoFlow (Zang and Wang 2020].
GraphAF [Shi al. 2020), GraphDF
[uo et al. 202I], GDSS [Jo et al. 2022 DiGress Vignac
ct al.
2022 HGGT Jang et al.
2024
GruM [Jo et al. 2024].
The results are shown in Table I LGD-large achieves the best results in FCD and NSPDK metrics,
and highly competitive in validity and uniqueness. This verifies the advantages of our latent diffusion model and the ability to generate nodes and edges simultaneously. It is worth mentioning different and potentially more difficult setting compared with some works like Geol.DM [Xu et al. 2023 In particular, LGD generates both atoms (nodes) and bonds (edges) simultancously in one shot, while GeoLDM only generates atoms (nodes), and then computes bonds (edges) with strong decoder based pair-wise atomic distances and atom types. We also do not use any 3D information, so it is indirect to compare with 3D baselines due to different training settings.
that our LGD works
Conditional generation. We still use the QM9 dataset for our conditional generation task, where we aim generate target molecules with the desired chemical properties. We follow the settings of [Xu et al. 2023] and split the training set into two parts, each containing 50k molecules. We train the latent diffusion model and separate property prediction networks (with architectures described in Equation (58)) on two halves, respectively. For evaluation, we first generate samples using the latent diffusion model given the conditioning property y, and use w to predict the property ý of generated molecules. We measure the Mean Absolute Error between y and ÿ, and experiment with six properties: Dipole moment μi, polarizability a, orbital energies cHоMO, é LUMO and their gap Ac, and heat capacity C...
8

---

We report EDM [Hoogeboom et al. 2022 and GeoLDM [Xu et al. 2023] as baseline models. We also incorporate (a) the MAE of the regression model w of ours and [Xu al. 2023], which can be viewed as lower bound of the generative models; (b) Random, which shuffle the labels and evaluate w, thus can be viewed as the upper bound of the MAE metric; (c) Natomss which predicts the properties based only on the number of atoms in the molecule.
As shown in Table 2 even if we do use 3D information and generate both atoms and bonds in one simultaneously, LGD achieves the best results in 4 out of 6 properties. The results verify the excellent controllable generation capacity of LGD.
6.2 Prediction with conditional generative models
We evaluate LGD extensively on prediction tasks, including regression and classification tasks of different levels. More results can be found in Appendix C
Regression. For regression task, we select ZINCI2K (Dwivedi et al., [2020], which is a subset of ZINC250K containing 12k molecules. The task is molecular property (constrained solubility) regression, measured by MAE, We use the official split of the dataset.
Table 3: Zinc12K results (MAE ().
Shown is the mean + std of 5 runs.
Method
Test MAE
Note that we are the first to use generative models to per- form regression tasks, therefore no comparable generative models can be selected as baselines. Therefore, we choose traditional regression models, including GIN [Xu et al.
GIN
0.163 + 0.004
PNA
0.188 + 0.004
GSN
0.115 1 0.012
DeepLRP
0.223 0.008
2018).
PNA [Corso et al. 2020]. DeepLRP [Chen et al.
OSAN
0.187 1 0.004
2020
OSAN [Qian et al. [2022], KP-GIN+ [Feng et
KP-GIN+
0.119 0.002
2022a
GNN-AK+ IZhao et al. 2021]. CIN [Bodnar et al.
GNN-AK+
0.080 + 0.001
2021
and GPS Raamásse et al. 2022 for comparison.
CIN
0.079 : 0.006
GPS
0.070 + 0.004
We train our LGD model and test inference with both DDPM and DDIM methods. While the generated predic- tions are not deterministic, we only predict once for each graph for a fair comparison with deterministic regression models. We will show in Appendix C that ensemble techniques can further improve the quality of prediction. As shown in Table 3 LDM (with DDPM) achieves the best results, even outperforming the powerful graph transformers GPS [Rampásek et al. 2022]. In comparison, the regression model with the same graph attention architecture as the score network in LGD can only achieve a worse 0.084 + 0.004 test MAE, which validates the advantage of latent diffusion model over traditional regression models. We also observe that inference with DDIM is much faster, but may lead to a performance drop, which aligns with previous observations [Song et al. 2020a Cao et al. 2023) Moreover, our LGD requires much less training steps compared with GraphGPSS see Appendix for more detailed discussions on experimental findings.
LGD-DDIM (ours)
0.081 0.006
LGD-DDPM (ours)
0.065 = 0.003
Classification. We choose node-level tasks for classification. Datasets include co-purchase graphs from Amazon (Photo) [Shchur et al. 2018), coauthor graphs from Coauthor (Physics) [Shchur et al. 2018), and the citation graph OGBN-Arxiv with over 169K nodes. The common 60%, 20%, 20% random split is adopted for Photo and Physics, and the official split based on publication dates of the papers is adopted for OGBG-Arxiv.
We choose both classical GNN models and state-of-the-at art graph transformers as basclines. GNNs include GCN [Kipf and Welling 2016), GAT [Velickovic et al.) 2017). GraphSAINT [Zeng et al.
2020
GRAND+ Feng et al. 2022bb. Graph transtormers include Graphormer (Ying et al. 2021]
SAN
Kreuzer et al. [ GraphGPS [Rampásek et al. 2022]. and the scalable Exphormer Shirzad
et al.
2023 and NAGphormer [Chen et al. 2023a]
As reported in Table 4 our LGD not only scales to these datasets while number of complex models like GraphGPS|Rand Rampásek et al. [2022) fail to do so, but also achieves the best results, even outperforming those state-of-the-ard graph transformers including Exphormer (Shirzad et al. 2023 and NAGphormer [Chen et al. 2023a]. Overall, our L.G reveals great advantages in both scalability and task performance.

---

Table 4: Node-level classification tasks (aceuracy 1) datasets from Amazon, Coauthor and OGBN- Arxiv. Reported are mean = std over 10 runs with different random seeds. Highlighted are best results.
Dataset
Photo
Physics
OGBN-Arxiv
GCN
92.70 = 0.20
96.18 + 0.07 96.17 + 0.08
GAT
93.87 + 0.11
GraphSAINT
91.72 + 0.13
96.43 + 0.05
GRAND+
94.75 + 0.12
96.47 + 0.04
Graphormer
92.74 0.13
OOM
OOM
SAN
94.86:001
0OM
OOM
GraphGPS
9
0OM
OOM 72.44 = 0.28
Exphormer
95.35 = 0.22
96.89 + 0.09
NAGphormer
95.49 = 0.11
97.34 0.03
LGD (ours)
96.94 + 0.14
98.55 + 0.12
73.17 + 0.22
7 Conclusions and Limitations
We propose Latent Graph Diffusion (L.GD), the first graph generative framework that is capable of solving tasks of all types (generation, regression and classification) and all levels (node, edge, and graph). We conceptually formulate regression and classification tasks conditional generation, and show that latent diffusion models can complete them with provable theoretical guarantces. We then encode the graph into powerful latent space and train latent diffusion model to generate graph representations with high qualities. Leveraging specially designed graph transformer architectures and cross cross-attention mechanisms, LGD can generate node, edge, and graph features simultaneously in both unconditional or conditional settings. We experimentally show that LGD is not only capable of completing these tasks of different types and levels, but also capable of achieving extremely competitive performance. We believe that our work is a solid step towards graph foundation models, providing fundamental architectures and theoretical guarantees. We hope it could inspire more extensive future research.
There are still some limitations of this paper that could be considered as future work. First, although the LGD is a unified framework, we train models separately for each task in our experiments. To build literal foundation model, we need to train single model that can handle different datasets, even from different domains. This requires expensive computation resources, more engineering techniques and extensive experiments. Second, it would be meaningful to verify the effectiveness of utilizing diffusion to perform deterministic prediction tasks in other domains, such as computer vision.
Acknowledgments and Disclosure of Funding
Muhan Zhang is supported by the National Natural Science Foundation of China (62276003).
References
Joe Benton, George Deligiannidis, and A. Doucet. Error bounds for flow matching methods. ArXiv, abs/2305.16666 2023.
Cristian Bodnar, Fabrizio Frasca, Nina Otter, Yu Guang Wang, Pietro Lio', Guido Montúfar, and Michael M. Bronstein. Weisfeiler and lehman go cellular: Cw networks. In Neural Information Processing Systems, 2021.
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportuni- ties and risks of foundation models. arXiv preprint arXiv:2108.072 2021.
Yu Cao, Jingrun Chen, Yixin Luo, and Xiang ZHOU. Exploring the optimal choice for generative processes in diffusion models: Ordinary vs stochastic differential equations. In seinth-
10

---

Conference on Neural Information Processing Systems, 2023. URL https::/oponrevvoo net/ Iorum/idd=1JJ0q62Yaaa
Jinsong Chen, Kaiyuan Gao, Gaichao Li, and Kun He. NAGphormer: A tokenized graph transformer for node classification in large graphs. In The Eleventh International Conference Learning Representations, 2023a. URL
Minshuo Chen, Kaixuan Huang, Tuo Zhao, and Mengdi Wang. Score approximation, estimation and distribution recovery of diffusion models on low-dimensional data. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 0472-4712. PMLR, 23-29 Jul 2023b. URL htttps://proceeddngg mlr. press/ v202// chen23o... html
Sitan Chen, Sinho Chewi, Jerry Li, Yuanzhi Li, Adil Salim, and Anru R Zhang. Sampling is as easy as learning the score: theory for diffusion models with minimal data assumptions. arXiv preprint arXiv:2209. 1215, 2022.
Zhengdao Chen, Lei Chen, Soledad Villar, and Joan Bruna. Can graph neural networks count .wwg/iv.orr//eoo 04025
substructures? CoRR, abs/2002.00025, 2020. URL
Gabriele Corso, Luca Cavalleri, D. Beaini, Pietro Lio', and Petar Velickovic. Principal neighbourhood aggregation for graph nets. ArXiv, abs/2004.05718, 2020.
Tingting Dan, Jiaqi Ding, Ziquan Wei, Shahar Kovalsky, Minjeong Kim, Won Hwa Kim, and Guorong Wu. Re-think and re-design graph neural networks spaces of continuous graph diffusion func- tionals. In A. Oh, T. Naumann, A. Globerson, K. Saenko. M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36. pages 59375-59387. Curran As- sociates, Inc., 2023. URL UtL//tttss:/pprrceeedaaa nwwr.ps....pp..re.. cc/ paper f fil.s/ https//ppprr/2/...////// Pwwpeeeeeereereeeeeeeeee
Teo Deveney, Jan Stanczuk, Lisa Maria Kreusser, Chris Budd, and Carola-Bibiane Schonlieb. Closing the ode-sde gap in score-based diffusion models through the fokker-planck equation. ArXiv, abs/2311.15996, 2023.
Vijay Prakash Dwivedi, Chaitanya K. Joshi, Thomas Laurent, Yoshua Bengio, and Xavier Bresson. Benchmarking graph neural networks. ArXiv, abs/2003.00982, 2020.
Vijay Prakash Dwivedi, Anh Tuan Luu, Thomas Laurent, Yoshua Bengio, and Xavier Bresson, Graph neural networks with learnable structural and positional representations. ArXiv, abs/2110.07875. 2021.
Jiarui Feng, Yixin Chen, Fuhai Li, Anindya Sarkar, and Muhan Zhang. How powerful are k-hop message passing graph neural networks. ArXiv, abs/2205. 133288 2022a.
Wenzheng Feng, Yuxiao Dong, Tinglin Huang, Ziqi Yin, Xu Cheng, Evgeny Kharlamov, and Jie Tang. Grand+: Scalable graph random neural networks. Proceedings of the ACM Web Conference 2022, pages 3248-3255 2022b.
Fabrizio Frasca, Beatrice Bevilaequa, Michael Bronstein, and Haggai Maron. Understanding and extending subgraph gnns by rethinking their symmetries. ArXiv, abs/2206. 1140, 2022.
Aditya Grover, Aaron Zweig, and Stefano Ermon. Graphite: Iterative generative modeling of graphs. In International conference on machine learning, pages 2434 2444. PMLR, 2019.
Kilian Konstantin Haefeli, Karolis Martinkus, Nathanael Perraudin, and Roger Wattenhofer. Diffusion models for graphs benefit from discrete state spaces. ArXiv, abs/2210.015444 2022.
Jonathan Ho, Ajay Jain, and Abbeel. Denoising diffusion probabilistic models. ArXiv, abs/2006.1 1239, 2020.
Emiel Hoogehoom, Victor Garcia Satorras, Clément Vignac, and Max Welling. Equivariant diffusion for molecule generation in 3d. ArXiv, abs/2203.110033, 2022.

---

Weihua Hu, Bowen Liu, Joseph Gomes, Marinka Zitnik. Percy Liang. Vijay S. Pande, and Jure Leskovec. Strategies for pre-training graph neural networks. arXiv: Learning, 2019.
Weihua Hu, Matthias Fey, Marinka Zitmik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning graphs. ArXiv, abs/2005.00687, 2020.
Qian Huang, Hongyu Ren, Peng Chen, Gregor Krvzmanc, Daniel Dajun Zeng, Percy Liang, and Jure Leskovec. Prodigy: Enabling in-context learning over graphs. ArXiv, abs/2305. 1203300..260000 16000, 2023.
Yinan Huang, Xingang Peng, Jianzhu Ma, and Muhan Zhang. Boosting the Cycle Counting Power of Graph Neural Networks with 12 (--GNNs. arXiv e-prints, art. arXiv:2210.13972 October 2022. doi:
018550/arXiv.220.133322
Md Shamim Hussain, Mohammed J. Zaki, and D. Subramanian. Global self-attention as replacement for graph convolution, Proceedings the and Data Mining, 2021.
the 28th ACM SIGKDD Conference on Knowledge Discovery
Yunhui Jang, Dongwoo Kim, and Sungsoo Ahn. Graph generation with Sk^2S-trees. In The Twelfth International Conference Learning Representations, 2024.
Jachyeong Jo, Seul Lee, and Sung Ju Hwang. Score-based generative modeling of graphs via the system of stochastic differential equations. In International Conference on Machine Learning, 2022.
Jachyeong Jo, Dongki Kim, and Sung Ju Hwang. Graph generation with diffusion mixture. Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp, editors, Proceedings of the 41 International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 12371-22400 PMLR, 21-27 Jul 2024.
Zahra Kadkhodaie, Florentin Guth, Eero P Simoncelli, and Stéphane Mallat. Generalization in diffu- sion models arises from geometry-adaptive harmonic representations, In The Twelfth International Conference on Learning Representations, 2024. URL ANvvVS2Yrr
Jinwoo Kim, Tien Dat Nguyen, Seonwoo Min, Sungjun Cho, Moontae Lee, Honglak Lee, and Seunghoon Hong. Pure transformers are powerful graph learners. ArXiv, abs/2207.002503 2022.
Diederík P. Kingma and Max Welling, Auto-encoding variational bayes. CoRR, abs/1312.61111 2013.
Thomas Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. 2016.
ArXiv,
Devin Kreuzer, Dominique Beaini, Will Hamilton, Vincent Létourneau, and Prudencio Tossou. Rethinking graph transformers with spectral attention. In M. Ranzato, A. Beygelz- imer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neu- ral Information Processing Systems, volume 34, pages 21618-21629. Curran Associates, Inc., 2021. URL htttp:://proceedinieen htwp:/occeediinnnneennneeeeeeeee mewrips. cc wwtpcppppppppeeeeecc. loorrffilee files/illeeeeee wtt.fs/s/aappee/////iiieeee (4<2c2260053000aaeeee7777711-111
pdf
Holden Lee, Jianfeng Lu, and Yixin Tan. Convergence of score-based generative modeling for general data distributions. ArXiv, abs/2209. 1t..22099.2211.... 12381, 2022.
Alexander C. Li, Mihir Prabhudesai, Shivam Duggal, Ellis L Brown, and Deepak Pathak. Your diffusion model is secretly zero-shot classifier. ArXiv, abs/2303. 2023a.
Puheng Li, Zhong Li, Huishuai Zhang, and Jiang Bian. On the generalization properties of diffusion models. Thirty-seventh Conference on Neural Information Processing Systems, 2023b. URL httpr//wwwww.eeeeeeoooom/ddddddd
Yibo Li, Liangren Zhang, and Zhenming Liu. Multi-objective de novo drug design with conditional graph generative model. Journal of cheminformatics, 10:1-24 2018.
12

---

Derek Lim, Joshua Robinson, Lingxiao Zhao, Tess E. Smidt, Suvrit Sra, Haggai Maron, and Stefanie Jegelka. Sign and basis invariant networks for spectral graph representation learning. ArXiv, abs/2202.1300 2022.
Stratis Limnios, Praveen Selvaraj, Mihai Cucuringu, Carsten Maple, Gesine Reinert, and Andrew Elliott. Sagess: Sampling graph denoising diffusion model for scalable graph generation. ArXiv, abs/2306. 16827, 2023.
Hao Liu, Jiarui Feng, Lecheng Kong, Ningyue Liang, Dacheng Tao, Yixin Chen, and Muhan Zhang. One for all: Towards training one graph model for all classification tasks. In The Twelfth International Conference on Learning Representations, 2024. URL httpt::/p//ppee httpps://opeerreiie net/ Forum?id=41T2ggcSv
Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning generate and transfer data with rectified flow. ArXiv, abs/2209.03003, 2022.
Sitao Luan, Chenqing Hua, Qincheng Lu, Jiaqi Zhu, Mingde Zhao, Shuyuan Zhang, Xiao-War Chang, and Doina Precup. Revisiting heterophily for graph neural networks. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho. editors, Advances in Neural Information Processing Systems, 2022. URL
Youzhi Luo, Keqiang Yan, and Shuiwang Ji. Graphdf: A discrete flow model for molecular graph generation. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 7192-7203, PMLR, 18-24 Jul 2021.
Liheng Ma, Chen Lin, Derek Lim, Adriana Romero-Soriano, Puneet Kumar Dokania, Mark Coates, Philip H. S. Torr, and Ser Nam Lim. Graph inductive biases in transformers without message passing. ArXiv, 1wt.22305.17889. 2023.
Haggai Maron, Heli HamuH Hadar Serviansky, and Yaron Lipman. Provably powerful graph networks. ArXiv, abs/1905. 11166, 2019.
Christopher Morris, Martin Ritzert, Matthias Fey, William L Hamilton, Jan Eric Lenssen, Gaurav Rattan, and Martin Grohe Weisfeiler and leman go neural: Higher- graph neural networks. In Proceedings of the AAAI conference on artificial intelligence, pages 4602-4609, 2019.
Christopher Morris, Gaurav Rattan, and Petra Mutzel. Weisfeiler and leman go sparse: Towards scalable higher-order graph embeddings. Advances in Neural Information Processing Systems, 33: 2020.
Chenhao Niu, Yang Song, Jiaming Song, Shengjia Zhao, Aditya Grover, and Stefano Ermon. Permu- tation invariant graph generation via score- based generative modeling. In International Conference Artificial Intelligence and Statistics, 2020.
Daniil Polykovskiy, Alexander Zhebrak, Benjamin Sanchez-Lengeling, Sergey Golovanov, Oktai Tatanov, Stanislav Belyaev, Rauf Kurbanov, Aleksey Artamonov, Vladimir Aladinskiy, Mark Veselov, Artur Kadurin, Simon Johansson, Hongming Chen, Sergey Nikolenko, Alan Aspuru- Guzik, and Alex Zhavoronkov. Molecular Sets (MOSES): A Benchmarking Platform for Molecular Generation Models. Frontiers in Pharmacology, 2020.
Chen Qian, Gaurav Rattan, Floris Geerts, Christopher Morris, and Mathias Niepert. Ordered subgraph aggregation networks. ArXiv, abs/2206.1 1 1168, 2022.
Raghunathan Ramakrishnan, Pavlo O. Dral, Matthias Rupp, and O. Anatole von Lilienfeld. Quantum chemistry structures and properties of 134 kilo molecules. Scientific Data, 1, 2014.
Ladislav Rampásek, Mikhail Galkin, Vijay Prakash Dwivedi, Anh Tuan Luu, Guy Wolf, and D. Beaini. Recipe for a general, powerful, scalable graph transformer. ArXiv, abs/2205. 2ttp//2255.1244,,,. 2022.
Robin Rombach, A. Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. 2022 IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10674-10685, 2021.
13

---

Oleksandr Shehur. Maximilian Mumme, Aleksandar Bojchevski, and Stephan Günnemann. Pitfalls of graph neural network evaluation. arXiv preprint
2018.
Chence Shi*, Minkai Xu*, Zhaocheng Zhu, Weinan Zhang, Ming Zhang, and Jian Tang. Graphaf: flow-based autoregressive model for molecular graph generation. In International Conference on Learning Representations, 2020.
Hamed Shirzad, Ameya Velingker, B. Venkatachalam, Danica J. Sutherland, and Ali Kemal Sinop. Exphormer: Sparse transformers for graphs. ArXiv, abs/2303.06147, 2023.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. ArXiv, fbss/201.005502. 2020a.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. ArXiv, abs/1907.05600) 2019.
Yang Song and Stefano Ermon. Improved techniques for training score-based generative models. ArXiv, abs/2006.0901 2020.
Yang Song, Jascha Narain Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. ArXiv, 0.s/2011.134455 2020b.
Hugo Touvron, Thibaut Lavril, Gautier Izacard. Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.1397,, 2023.
Aäron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete representation learning. ArXiv, 0ttps//711.009933 2017.
Petar Velickovic, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio', and Yoshua Bengio. Graph attention networks. ArXiv, abs/1710. 2017.
Clément Vignac, Igor Krawczuk, Antoine Siraudin, Bohan Wang, Volkan Cevher, and Pascal Frossard. Digress: Discrete denoising diffusson for graph generation. ArXiv, abs/2209. 2022.
Hongya Wang, Haoteng Yin, Muhan Zhang, and Pan Li. Equivariant and stable positional encoding for more powerful graph neural networks. ArXiv, abs/2203.000 00099, 2022.
Zhenqin Wu, Bharath Ramsundar, Evan N. Feinberg, Joseph Gomes, Caleb Geniesse, Aneesh S. Pappu, Karl Leswing, and Vijay S. Pande. Moleculenet: a benchmark for molecular machine learningt telectronic supplementary information (esi) available. see doi: 10.1039/c7sc02664a. S Chemical Science, 9:513 530, 2017.
Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful graph neural networks? ArXiv, abs/ 100.0080, 2018.
Minkai Xu, Alexander Powers, Ron O. Dror, Stefano Ermon, and Jure Leskovec. Geometric latent diffusion models for 3d molecule generation. In International Conference on Machine Learning, 2023.
Ling Yang, Ye Tian, Minkai Xu, Zhongyi Liu, Shenda Hong, Wei Qu, Wentao Zhang, Bin Cuiand Muhan Zhang, and Jure Leskovec. Vggraph: Graph vector quntti-quatiiztiion for bridging gnns and mlps. ArXiv, 2023a.
Run Yang, Yuling Yang, Fan Zhou, and Qiang Sun. Directional diffusion models for graph represen- tation learning. ArXiv, abs/2306. 13210, 2023b.
Zhilin Yang, William Cohen, and Ruslan Salakhudinov. Revisiting semi-supervised learning with graph embeddings. In Maria Florina Balcan and Kilian Q. Weinberger, editors, Proceedings of The 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pages 40-48, New York, New York, USA, 20-22 22 Jun 2016. PMLR
14

---

Chengxuan Ying. Tianle Cai, Shengjie Luo, Shuxin Zheng, Guolin Ke, Di He, Yanming Shen, and Tie- Yan Liu. Do transformers really perform bad for graph representation? Neural Information Processing Systems, 2021.
Hui Yuan, Kaixuan Huang, Chengzhuo Ni, Minshuo Chen, and Mengdi Wang. Reward-directed conditional diffusion: Provable distribution estimation and reward improvement. In Thirty-eeventh Conference Neural Information Processing Systems, 2023. URL
.ntt/
Forum?id-55tvvmnddtz
Chengxi Zang and Fei Wang. Moflow: An invertible flow model for generating molecular graphs. Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, 2020.
Hanqing Zeng, Hongkuan Zhou, Ajitesh Srivastava, Rajgopal Kannan, and Viktor Prasanna. Graph- saint: Graph sampling based inductive learning method. International Conference on Learning Representations, 2020. URL https:///ppeereiiee wwt.:nnuntttiiiieeemm -aeerummdd-Baaa
a BJe@pkHFWS
Bohang Zhang, Guhao Feng, Yiheng Du, Di He, and Liwei Wang. A complete expressiveness hierarchy for subgraph gnns via subgraph weisfeiler-lehman tests, arXiv preprint arXiv:2302.07090 2023a.
Bohang Zhang, Shengjie Luo, Liwei Wang, and Di He. Rethinking the expressive power of gnns via graph biconnectivity. ArXiv, abs/2301 100500,, 2023b.
Muhan Zhang and Pan Li. Nested graph neural networks. ArXiv, abs/21 10.1 2021.
Lingxiao Zhao, Wei Jin, Leman Akoglu, and Neil Shah. From stars to subgraphs: Uplifting any gnn with local structure awareness. ArXiv, abs/21 10.03753, 2021.
Cai Zhou, Xiyuan Wang, and Muhan Zhang. Facilitating graph neural networks with random walk on simplicial complexes. In Advances in Neural Information Processing Systems, volume 36, pages 16172-16206. Curran Associates, Inc., 2023a.
Cai Zhou, Xiyuan Wang, and Muhan Zhang. From relational pooling to subgraph GNNs: A universal framework for more expressive graph neural networks. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 42742-42768. PMLR, 2023b.
Cai Zhou, Rose Yu, and Yusu Wang. On the theoretical expressive power and the design space of higher-order graph transformers. In Proceedings of The 27th International Conference on Artificial Intelligence and Statisties, volume 238 of Proceedings of Machine Learning Research, pages 2179-2187, PMLR, 2024.
15

---

A Background on Diffusion Models
In this section we provide more background knowledge on diffusion models, including formal formulation of the families of DDPM (Ho et al. 2020] and SMLD [Song and Ermon 2019] methods,
well their unified formulation through the framework of SDE Song et 2020b ODE Song
et al. 2020a].
A.1
Denoising diffusion probabilistic models
Given samples from the ground truth data distribution ((xo). generative models aim to learn a model distribution po(xo) that approximates ((Xo) well and is easy to sample from. Denoising dffusion probabilistic models (DDPMs) (Ho et al. 2020) are family of latent variable models of the form
(x(x / Po (xorr)xx::
(10)
where
(x == pa (XT)
II Po (x--1)x:)
(11)
here X1, Xy are latent variables the same sample space (denoted as 2) as the data Xy-
A special property of diffusion models is that the approximate posterior q(x1,((xx) also called the forward process or diffusion process, is fixed Markov chain which gradually adds Gaussian noise to the data according to a variance schedule BI....,
((xiT Xo) = II
xx-1) 1=1
(12)
where
xxxx == N(x:: v B,x,-1,
(13)
The forward process variance 3 can either be held constant as hyperparameters. or learned by reparameterization (Ho al. 2020). One advantage of the above parameterization is that sampling X₃ at an arbitrary timestep has closed form,
((x)(xo)= (xxx Va,xo. a))
(14)
where o₁ 1 Bg and : IT as Utilizing the closed-form expression, express x; as a linear combination of Xy and noise variable ~ NNO,
X = == + VI aie
(15)
Note that ā, is decreasing sequence, and when to standard Gaussian for all Xo, thus we can naturally set pe px(x)) := (0,
ãT sufficiently close to 0, 9(x7 (xo) converges
In the generative process, the model parameterized by 0 are trained to fit the data distribution ((x)) by approximating the intractable reverse process ((x-1|/x,). This can be achieved by maximizing variational lower bound:
max (x [log pe (xx)] S max Ex Dlg po (Xo:T) log q(x1:T xx)]
(16)
The expressivity of the reverse process is ensured in part by the choice of conditionals in po(x;- (x.). all the conditionals are modeled as Gaussians with trainable mean functions and fixed variances as in [Ho al. 2020), then the objective in Equation (16) can be simplified to
C,(co) Σ X.E A
[|c- (1 (Vārx vI- vI-ayci) c
(17)
(t) X + X' indexed by t is where = - JT= is set of T' number of functions, each 6g
denoising function with trainable parameters ((1), À == (A...., Xr is positive coefficients, which set to all ones (Ho et al. 2020] and (Song and Ermon 2019 and we follow their settings throughout the paper.
16

---

In the inference stage for trained model, Xo is sampled by first sampling Xy from the prior pe (xT). followed by sampling x from the above generative processes iteratively. The length steps T in the forward process is hyperraaameete (typically 7 = 1000) in DDPMs. While large 7 enables the Gaussian conditionals distributions as good approximations, the sequential sampling becomes slow. DDIM [Song et al. 2020a proposes to sample from generalized generative process as follows,
Xt
(1) 0450 (xx)
(5))) + / V1- ã.-1 og (x,) + def
X = 04-1
(18)
Vān
04 01/00-1)/(1 04) for all the generative process becomes DDPM, while og = 0 results in DDIM, which is an implicit probabilistic model that samples with a fixed procedure, thus corresponding to ODE (Song et al. 2020aa]. In our work, both DDPM and DDIM
When o; /(1 (1-0)t1)(0 02/
considered in our generative process.
A.2 Score matching with Langevin dynamics
Although we mainly illustrate our framework using the family of DDPM and DDIM in the main text, the family score matching with Langevin dynamics (SMLD) [Song and Ermon 2019) methods are also applicable in our LGD.
Score matching methods estimates the score (gradient of -og-liielihood) of the data using a score network sg Fd + Rd by minimizing the objective
[100(()) - Vx log (x)
(19)
Given the equivalence of SMLD and DDPM explained in Appendix A.3
do not distinguish the two families of generative models in most cases of the paper. We also sometimes use the notation eg represent both the denoising network in DDPM and the score network in SMLD.
A.3 Unified formulation through the framework of SDE and ODE
Song et al. 2020b] formulates both SMLD and DDPM through the system SDE. The authors further propose ODE samplers which are deterministic compared with SDE, so does (Song et al. 2020a Liu et al. [2022] use ODE- based rectified flow to sample with high efficiency. Cao et al.
[2023] and Deveney al. 2023 analyze the connections and comparisons between ODE and SDE
based methods.
In particular, the diffusion process can be modeled as the solution to an SDE:
dx ((x,t)d g())dw
(20)
where is the standard Wiener process (a.k.a., Brownian motion), f(: Rd Rdd a vector- valued function called the drift coefficient of (t), and g(-) R - IR is a scalar function called the diffusion coefficient of x().. Note that the end time T the diffusion process described in the SDE has different meaning in the number of discrete diffusion steps in DDPM, and we denote the latter as E N in the SDE formulations.
To sample from the backward process, we consider the reverse-time SDE which is also a diffusion process,
dx = [f(x,t) t) g(t)
+ g(t)dw
(21)
where w is a standard Wiener process in which time reverse from T to 0. Again, here T is the end time of the diffusion process, which is different from the number of discrete diffusion steps in DDPM). dt is an infinitesimal negative timestep, which is discretized in implementation of DDPM, i.c. each time step is h ;= T/N where N is the number of discrete steps in DDPM.
A.4 Related work on theoretical analysis of diffusion models
The empirical success of diffusion models aroused researchers' interest in theoretical analysis of them recently. Chen et al. 2023b) and Yuan et al. 2023) provide some theoretical analysis on score approximation and reward-directed conditional generation, respectively. [i et al. (2023b) provide
generalization bound of diffusion models. Li al. 2203al) provide empirical evidence that diffusion
17

---

models can be zero-shot classifiers, Benton et al. [2023] analyzes the error bound of flow matching methods, and Lee et [0022) Chen al. 2022 analyzed the convergence of SDE-based score matching methods.
B Theoretical Analysis
In this section, we provide our novel theoretical analysis on the guarantees of solving regression or classification tasks with conditional latent diffusion model in Section Our theory generic and is not limited to graph data.
As described in the main text, while training the conditional latent diffusion for prediction tasks with labels, we aim to generate the latent embedding of labels y conditioning on the latent embedding of data æ. We first consider regression tasks in analysis, where the label is scaler y; classification tasks can be analyzed in a similar manner. We use a slightly different notation compared with the original DDPM formulation in (Ho al., 2020]; instead, we N denote the maximum number of discrete diffusion steps in a DDPM (which is denoted as 7 in (Ho et al. 2020),, and instead use T denote the end time in the SDE which controls the diffusion process.
According to the reasons Section 5 we choose to model the joint distribution of æ, y, thus use T(x, 1) IR to jointly encode the data and the label, and use E(x) IE to embed the data (condition) only. Here we suppose the output of each encoder is a d-dimensional vector. The diffusion model is trained to generate T y) conditioning decoder D,, which is linear layer parameterized by Rd First we have the assumptions on the reconstruction errors of the autoencodings.
E(x). We suppose two encoders share one
Assumption B.1. (First and second moment bound of autoencoding errors.)
e == E, ||| w F(x,8) 8) y/ll < oc
(22)
mf = E, 11 Txx y) y||²1 < 0c
(23)
6 1y11 {(x) y|ll
(24)
m = F2,lw E(()) y|| 0o
(25)
Typically, 62 is just the MAE of a traditional deterministic regression model; < é2 since takes as inputs and thus has an extremely small reconstruction error. Assumption B.1 generally holds and can be verified across all experiments.
To show why a generative model can perform better than the regression model, i.e. could have a smaller error than 62, we continue to make the following mild assumptions on the data distribution which also generally holds for practical data..
Assumption B.2. (Lipschitz score). For all t 2 0, the score In is L-Lipschitz.
Finally, the quality of diffusion obviously depends on the expressivity of score matching network (denoising network) cg
(t)
Assumption B.3. (Score estimation error). For all l: I,
Egga. (c
V In qael S 2 20o00
(26)
where h T/N the time step in the SDE.
This is the same assumption as in [Chen et al. 2022 In our main paper the denoising network is powerful graph transformer with architectures described Equation (58). which can well guarantee small score estimation error. However, detailed bounds of Escore is beyond the scope of this paper.
Finally, we consider simple version of embedding the condition, where the denoising network eæ still samples in the same way as unconditional generation, except that we directly add the embedding of condition to the final output of cg. In other words,
(2, (4) (x)()) + E(x)
(27)
where z is the latent being sampled.
18

---

Now we give our main theorem of the regression MAE of the latent diffusion. The theorem is proved through the SDE description of DDPM as Appendix A.3] where we use T to denote the end time of diffusion time in the SDE, N the number of discrete diffusion steps in DDPM implementation, and 7d = (0, Ta) as simplified notation of random variable from the standard Gaussian distribution.
Assumption [.2 Assumption B.3 hold, and the step size = T/N satisfies h 3 L where L 2 1. Then the mean absolute error (MAE) of the conditional latent diffusion model in the regression task bounded by
Theorem B.4. Suppose Assumption
MAE < Eg 111 (
f(x, v) w ((x,,))l +1L
(28)
= KL( exp( T)) (LVdh dh + L(m1 + m2)T +
Escore VT
convergence of forward procert process
diseretization ervor
score estimation crror
encoder error
(29)
where 4s is the ground Iruth distribution == w (T(ar,
(xx)..
Proof. Conditioning on E(x), the diffusion model eø aims to generate the refined representation T(., y). Denote the generated embedding as f(æ,, y), then
MAE:
E,1l| f(x, v) y|1l
(30)
Eulll(w f(xx,) u) w 7(x, v)) (w T(x, v) w)1l S E,lllw f(xx, y) - w T(x w)ll + E,ll|w T(xx v) - s/l]
(31)
(32)
EElllvo f(x, y) w r( v)ll +
(33)
We only need to prove the bound of Ey[lu ( w T v)lll Note that the generation is conditioned on (xx). Instead of cross attention, we consider a simple method to incorporate the condition, which use a residual connection that directly add the condition & {(x) the feature being denoised. Then the conditional generation is equivalent to the unconditional generation which
samples the distribution of (xx, y) (xx) from noise. It suffice to show the unconditional generation error of 7 7xx, y) E(x). Using the notation
21 == w T T(x y) -
(34)
22 = w T {xx)
(35)
where
|, é1
(36)
|= m
(37)
E,112211 62
(38)
11=2||11 = m²
(39)
We then have
w (T(z, 8) E(x)) 21 22
(40)
To complete our derivation, consider the following lemma proposed in (Chen al. 2022
Lemma B.S. (TV distance of DDPM) [Chen et al. 2022). Suppose that Assumption B. Assump-
generated has the ground truth ground truth distribution and the estimated distribution by the diffusion model is Pa- The second moment bond
tion B.2 Assumption B.3 hold. The data
M² = Eg III- [12] > co. Suppose that the step size := T/N satisfies 3 1/L where Z 1. Then the total variance (TV) distance satistfies
TV(ps. (KL((a/ll)))))pppp + (Lvdh dh + LMh) VT Escore VT
(41)
19

---

The proof of the lemma is in [Chen et al. 2022].
Back derivation, utilizing the Cauchy inequality, we have
y. |||
B |l æ x21l²]
(42)
s gg ||| 11 1 Eg [1|2|1 2E, 21
(43)
- mi m2
2
2 + 2 <
(44)
thus
TV (p,, (z) = By ||
=||
V (xI((==)))))xp(((1>1 (LVdh + L(mi m2)h) + Escore VT
(45)
where 2 ~~ Ps is the estimated embedding of z generated by the diffusion model. Recall the condition embedding is implemented via simply adding the condition to the generated é, i... w f(x, y)
=
w/ (x). we have
Eylll f(x, y) - w w (xr,y)))
(46)
=E, 2 - (w' *T(x,)) v) w {()))1I
(47)
-E,, [l|e - =|11
(48)
=/ (L((s=lll exp(-T)) + (Lvdh dh L(m1 + m2)h)/T + Escore VT
(49)
where we omit the constant factor in Equation 45 Finally we complete our proof,
MAE s Ey[l|w (elll[T((xz))) f(x,)) w. T(z, v)ll +1
(50)
= (KL( + (Lvdh + Lmm + m2) VT
Cscore VT
convergence of forward process
discretization error
SCOre estimation error
encoder error
(51)
We now give more interpretations Theorem B.4 The first term v (L 1 (1d)
exp(-I') is the сттог of convergence of the forward process, which can be exponetially small when we have enough SDE diffiusion time T, or equivalently, ā is efficiently small make @r close enough to the standard Gaussian 7d := (0, Ia).
The second term (Lvdh + [(m + m2))h) is the discretization error, since when implementing the DDPM we have use discrete N steps to simulate the ODE. interesting to see that this term is affected by both the latent dimension d and the second moment of the variation encoding errors my and m2 (i.e. the reconstruction MSE loss of labels and conditions). The results implies that lower dimension of the latent space H. is more suitable for lcarning diffusion model. However, note that the reconstruction MSE loss m1, m2 may also be affected by d: when the rank of data is high, low dimension latent space may not represent the data well, and the projection made by the decoder Rdd is likely to cause information loss, thus increasing my and m2. Therefore, the latent dimension d should be carefully chosen, which should not be neither too low (poor representation power and large reconstruction error) nor too high (large diffusion error). This verifies the intuition discussed in Section 4
The third term is the score estimation error, which is related the expressivity of denoising (score) networks. Detailed theoretical analysis of various architectures is beyond the scope of the paper, and we refer readers more relevant papers on expressivity or practical performance of potential architectures, including MPNN Xu et al. 2018). high-order and subgraph GNNs [Moriis et al., 2019 2020 Feng et al. 2002a Zhou et al. 2023b Zhang al. 2023a], and graph transtormers [Kim et al. 2022 Zhou et al. 2024|. For prediction tasks where we have ground truth graph structures as inputs, we can also use positional encodings and structural encodings to enhance the theoretical and empirical expressivity of score networks [Zhang et al.) 2023b Dwivedi et al. 2021 Wang et al. 2022 Zhou al.] 2023a Lim al.] 2022). remarkable that our architecture reveals strong performance in experiments. Empirically, the graph transformer in Equation (58) we adopted is powerful in function
20

---

approximation. In the language of distinguishing non-isomorphic graphs, our transformer is at least as powerful 1-order Peisfeiler-Lehmane test [uu et al. 2018 Zhou al. 2024].
The last term is the error brought by autoencoding. However, as is the reconstruction MAE which aims to encode the labels, 61 is typically much smaller than the MAE of regression models c2- Our experimental observations support the above assumption. Therefore, the conditional generation model outperforms the traditional regression model long
V exxlsll))))pp(( + (Lv/hh L(m m2)h)/ Sscore VT +
(52)
This can be achieved by searching hyperparameters the diffusion model, including N, h. Intuitively, a sufficiently small (or equivalently, sufficiently large number of diffusion steps N) would lead a small error. Following [Chen et al. 2022), suppose (L(g=lh")) exp(-T) poly(d),
my + m2 & d, then by choosing T = In(KT(g=ll>> (//l/))//)))))))//)))))))))))( and h x Lza have
2
and hiding logarithmic factors,
MAE < O(E + Escore + E1)
(53)
where is the error scale we desire. In this case, need N = O 1.d steps. If éscore < (e) and e S O(e), then MAE S.
(-)) number of discrete diffusion
Finally, we have the following statement which is straightforward to prove. Corollary B.6. When the condition is embedded as in Equation diffusion model whose MAE not larger than the auloencoder.
27 there exists at least one latent
Proof. The extreme case that the denoising network always outputs 6g (2) = 0 for all 4, then according to Equation (27). the generated representation
fax y) = (ax)
(54)
Then the MAE would be simply the MAE of the autoencoder:
MAE = E2I T f(x y) - =lll
(55)
= Eyll [(x)) y|l
(56)
62
(57)
This corollary reveals that there always exists latent diffusion models that performs least well as the autoencoder (which could also be viewed as a traditional regression model). Combining with Theorem B.4, the diffusion models could outperform the traditional regression models.
C Experimental Details and Additional Results
C.1 Architecture design and implementation details
Graph self-attention with augmented edges. As explained in Section 4.2 and Section 4
need a single model to jointly represent and update node, edge and graph features. For denoising network, it should able compute scores of all n² possible edges as want generate both the structure and the features of the graph. We therefore design a special graph transformer with augmented edges, which shares a philosophy similar to GRIT Ma et al. 2023 and EGT [Hussain et al. 2021]. Formally,
(+ σ (p(x(Qa)) Ka,) O Ewel,) + Egely) Rd
1j
Ø; = Softmax Wxe (+1 E R (+1 V + E, e E Rd
(58)
[+1 x
JE(n
where / represents the number of attention layers, which has been omitted for simplicity in the
weight matrices. The matrices Q IR xd K E Rdd xd V E pdd xd En dd''d E, pd
21

---

WA E Rixd", E. E Rd are all learnable weight matrices in the attention mechanism. The attention kernel K can be an clement-wiss addition, as seen in [Ma et al. 2023), or the commonly used Hadamard product. The symbol @ denotes the Hadamard product felement-wise multiplication). The optional non-linear activation function a can applied, such as ReL.U, or the identity mapping can be used if no non-linear activation is desired. The function p(æ:) is optional and defined as / //ReU(xx)> / (ReLU(-<)), which aims to stabilize the training by preventing large magnitudes in the gradients.
To reduce computational complexity, we provide a simplified version where W is set E RR"" which suffices sum the latent dimension. In this case, the attention mechanism reduces to the standard inner product if the kernel A is chosen to be the Hadamard product. Additionally, in simplified attention, E, can also be set to an identity matrix.
In the edge-enhanced transformer described Equation (58), we explicitly maintain representations for all nodes and augmented edges. These representations are denoted as X E Rnxd and A E Raxxxd where X; x; and A = Cij. with d being the hidden dimension in the corresponding layer. The attention mechanism can be naturally extended to multi-head case, similar to standard attention. Additionally, residual connections and feedforward layers can be incorporated for both nodes and edges. When representing graph-level attributes, we have two options: (a) utilizing the virtual node trick, where a virtual node used to represent the graph-level feature; or (b) obtaining the graph-level feature by pooling the node/edge features. It is straightforward to verify that Equation (58) exhibits permutation equivariance for both nodes and edges, and permutation invariance for graph-level attributes.
Cross attention mechanism for general conditions. In conditional generation, we need to encode the conditions into the denoising networks. Cross attention is a widely adopted method [Rombach] al. 202T]. We propose a cross-attention mechanism specifically for graphs, which is implemented by
x; a softmax (Q (KAT()))
VAT(y)
Vi Q Qqeet (KaT V.r(y)
(59)
e softmax(
Vd
where Qn, Q d'd' Kn. Vn, K., Ve E Rd'xd, learnable weight matrices. Residual con- nections and feedforward layers are also allowed. The cross-attention mechanism is able to process multiple conditions with different data structures, and becomes an addition in the special case where there is only one condition.
Cross attention mechanism for graphs. As stated in Section 4.4 conditions are a graph with part of its features masked. We now propose a novel cross attention mechanism compute scores of the graph be generated conditioning on the masked graph.
there are where the
We assume the latent dimension of the conditioning graph and that of the graph to be generated are identical (otherwise could use linear layers project the features of the conditioning graph), and use characters without superscript e to denote the graph being generated from noise (the superscript ( is still the number of layer), then
(((((2)) Kxf) @ Eu(ely + e(,)) + Ebe,, + Geg") e R
ai SoftmaxJEm/ W (Weeth) 1+1 ER
(60)
E ond (Va + E Who + Gig e c. jEin]
where learnable weight matrices Q, K, V, E WA E Eiid", E. E d'd''', attention kernel K, activation a and gggned-squar-rooos operation p are identical to those in Equation (58):
the new notations G., G., Wn, E Rdd xd are also learnable weight matrices. The cross-attention Equation (60) and the self-attention Equation (58) have two major differences: (a) cross-attention uses the features of the conditioning masked graph as the keys and values of the attention; (b) to make the model aware of the node/edge correspondence between the conditioning graph and the main
22

---

Table 5: Overview of the datasets used in the paper.
Avg. # nodes
Avg. edges
Prediction level
Prediction
Dataset
#Graphs
Mctric
task
QM9
130,000
18.0
37.3
graph
regression
Mean Abs. Error
ZINC
12,000
23.2
24.9
graph
regression
Mean Abs. Error
ogbg-molhiv
41,127
25.5
27.5
graph
binary classif.
AUROC
Cora
2.708
10.556
edge
binary classif.
accuracy
PubMed
19,717
88,648
edge
binary classif.
accuracy
Cora
2,708
10,556
node
7-way classif.
accuracy
PubMed
19,717
88,648
node
3-way classif.
accuracy
Physics
34,493
495,924
node
5-way classif.
accuracy
Photo
7,650
238,162
node
8- 8-way classif.
accuracy
OGBN-Arxiv
169,343
1,166,243
node
40-way classif.
accuracy
graph to be generated, cross-attention has node-wise and edge-wise addition of the conditioning node/edge features æt, eij the node/edge features wt, ely of the main graph.
C.2 Additional experiments
In this subsection, we provide further experimental results in additional to the ones the main text. The datasets of these additional experiments include QM9 [Ramakrishnan et al. [2014). small
molecular graphs from gebg-molhiv [Hu et al., 2020], and two citation networks from Planetoid (Cora and PubMed) (Yang al. 2016]. The statistics of these datasets are summarized in Table 5
We summarize the results follows.
SOTA performance. Our LGD achieves state-of-the-art performance on almost all of these datasets, which verifies that diffusion models can outperform traditional regression and classification models. Our advantages over traditional models are generally more significant when the datasets are large, since diffusion models are capable of modeling complex distributions and empirically perform well with sufficient training data. This implies the strong performance, scalability and generalization of LGD.
Strong scalability. Our extensive experiments show that LGD could be extremely efficient with proper design, which can completely scale to extremely large graphs with over 169K nodes such as OGBN-Arxiv. Moreover, LGD efficient in both memory and time consumption: all the experiments are carried out a single RTX 3090 GPU, and both training and inference procedures are fast (e.g. 0.2s/epoch on Cora).
Flexibility and universality. These experiments with tasks of different levels of types again verifies the flexiblity and universality between scalability and performance for all tasks.
framework. We can always strike good balance
Large-scale generation. The generation results on large-scale molecule dataset MOSES [Polykovskiy et al. 2020 reported in Table 6 show that LGD has superior perfor- mance compared with DiGress in terms of validity and novelty metrics. Apart from DiGress and ConGress, we are the only model to scale to this large dataset, bridging the gap between diffusion-based one-shot graph generation model and other traditional methods like SMILES-based. fragment-baas baaee,,t-baadd,,.... and autoregressive models.
Graph-level regression. We still choose the QM9 dataset for the graph-level regression tasks, Base- line models include MPNN, DTNN [Wu al. 2017). DeepLRP [Chen et al. 2020), PPGN [Maron et al. 2019]. Nested GNN [Zhang and Li 2021 and 4-IDMPNN [Zhou et al. 2023b). We consider the same six properties as the conditional generation task. The results are reported in Table LGD achieves SOTA performance in 5 out of 6 properties, and the advantages are significant. This is an inspiring evidence that generative models could outperform traditional models under our framework.
23

---

Table 6: Large-scale generation on MOSES
Polykovskiy et al.
2020 dataset.
Model
Class
Validity
Uniqueness
Novelty
FCD
VAE
SMILES
97.7
99.8
69.5
0.57
JT-VAE
Fragment
100
100
99.9
1.00
GraphINVENT
Autoreg.
96.4
99.8
1.22
ConGress
One-shot
83.4
99.9
96.4
1.48
DiGress
One-shot
85.7
100
95.0
1.19
LGD (ours)
One-shot
97.4
100
95.9
1.42
Table 7: QM9 regression results (MAE 4). Highlighted are first, second best results.
Target
MPNN
DTNN
DeepLRP
PPGN
Nested GNN
4-IDMPNN
LGD (ours)
μ
0.358
0.244
0,364
0.231
0.433
0.398
0.088
a
0.89
0.95
0.298
0.382
0.265
0.226
0.130
{HOMO
0.1472
0.1056
0.0691
0.0751
0.0759
0.0716
0.0363
(LUMO
0.1695
0.1393
0.0754
0.0781
0.0751
0.0778
0.0276
Ac 0.1796
0.3048
0.0961
0.1105
0.1061
0.1083
0.0395
Gy
0.42
0.27
0.129
0.184
0.0811
0.0890
0.0817
Graph-level classification. For graph-level classification task, we choose the ogbg-molhiv [Hu et al. 22200 dataset which contains 41k molecules. The task graph binary classification to predict whether a molecule inhibits HIV
Table 8: Ogbg-molhiv results (AUC 1).
Shown is the mean + std of runs.
Method
Test AUC
virus replication or not, and the metric AUC. We use
the official split of the dataset. PNA [Corso et al. 2020
PNA
79.05 + 1.32
DeepLRP [Chnn et al. 22020, NGNN [Zhang and Li 2021
DeepLRP
77.19 + 1.40
KP-GIN [Feng et al.
2022a I"-GNN (Huang al. 2022
NGNN
78.34 1.86
CIN Bodnar et al. 2021] and SUN(EGO) Frasea et al.
KP-GIN+
78.40 + 0.87
2022 are selected as baselines. As shown in Table S LGD
GNN
78.68 + 0.93
cxhibits comparable performance compared with state-of- the-art models. A possible reason is that ogbg-molhiv adopts scaffold split which is not random split method, leading to imbalance samples in hain/validaaionntestt sets.
CIN
80.94 = 0.57
SUN(EGO)
80.03 + 0.55
GPS
78.80 1.01
LGD-DDPM (ours)
78.49 0.96
As generative models directly model the distribution, a distribution shift may negatively influence the performance.
Node and edge classification. The results of node and edge classification on ciatation networks Cora and PubMed reported in Table The common 60%, 20%, 20% random split is adopted. LGD significantly outperforms all the baselines, including OFA [Liu et al. 2024 which uses LLMs as augmented features.
C.3 Experimental settings
We now provide more details of the experiments in Section including experimental settings and implementation details.
C.3.1 General settings
Diffusion process. In all experiments, we use a diffusion process with T 1000 diffusion steps, parameterized by a linear schedule (4, and thus a decaying ä, For inference, we consider both (a) DDPM; (b) DDIM with 200 steps and a, = 0. Instead of predicting the noises, we use the parameterization where denoising (score) networks e set to directly predict the original data Xg- We set À 1 in Equation (18) most literature for simplicity.

---

Table 9: Node-level and edge-level classification tasks (accuracy 1) on two datasets from Planetoid. Reported are mean = std over 10 runs with different random seeds. Highlighted are best results.
Dataset
Cora Link
PubMed Link
Cora Node
PubMed Node
Task type
GCN
90.40 = 0.20
91.10 + 0.50
87.78 0.96
88.9 - 0.32
GAT
93.70 = 0.10
91.20 + 0.10
83.00 + 0.70
83.28 + 0.12
OFA [Liu al. 2024)
94.53 = 0.51
98.59 + 0.10
74.76 1.22
78.25 = 0.71
ACM-GCN Luan et al. 2022
89.13 + 1.77
90.66 + 0.47
ACM-GCN+
89.75 + 1.16
90.46 + 0.69
ACw-Slowball-3
89.59 + 1.58
91.44 0.59
LGD (ours)
96.48 + 0.17 99.03 + 0.08
93.91 + 0.55
92.88 + 0.29
Model architectures. In our main paper, we introduce the formulation where we generate the full A E R=xxxdd so that we have a unified framework that can tackle tasks of all levels. However, this does not necessarily mean that always need to generate A-instead, can generate only the features that we desire. Correspondingly, although we use the specially designed graph transformers and
mechanism for denoising networks, other architectures (c.g. general GNNs) are also applicable, as we have already mentioned in the main text (see Section 4.1 For example, can generate
in node classification tasks where only need to predict the labels of nodes, only the node features, where a simple MPNN (e.g., GCN) would be enough for the denoising network. We implement this setting in some of our experiments, showing that our LGD can scale to extremely large graphs with over 169K nodes (e.g.. OGBN-Arxiv). The actual training and inference procedures are also very efficient, for example each epoch takes only ~0.2s in Cora dataset.
For all graph-level tasks, we use the encoder &g with the augmented edge enhanced graph transformer as described in Equation (58). For all node-level tasks, we use MPNN model (e.g. GINE (Hu ct al. 2019), GCN [Kipf and Welling 2016] or GAT [Velickovic et al., 2017)) as the encoder. For complete,
can also add positional encoding and MPNN modules to the encoder. For all tasks, each task-specific decoder De is single linear layer for reconstruction prediction. Due to the discussion in Appendix [B] we choose the dimension of the latent space from [4, 8, 16, 32). We pretrain the autoencoders separately for each tasks. Pretraining across datasets (especially across domains) is more challenging setting, but may also benefit the downstream tasks, which would be an interesting future direction.
regression and classification tasks, as the input graph structures
For denoising (score) network eg, we consider two cases: (i) we want to generate structures and features simultancouslyy; (ii) we only want to generate3 features. In the first case, due to the reason explained in Section 4 MPNNs and PEs are ill-defined in the corrupted latent space H, and are hence not applicable. Therefore, we use pure graph transformer architectures for cg. We classify all generation tasks and graph-level prediction tasks into this case. For unconditional generation, each layer computes self-attention as in Equation (58). For conditional generation, each module consists of self-attention layer, a cross-attention layer, and another self-attention layer. For QM9 conditional generation tasks where the conditions are molecular properties, we use an MLP as the conditioning encoder т and the cross-attention in Equation (59) For regression ogbg-molhiv, since the conditions are masked graphs, we use the specially designed cross-attention in Equation (60p. In these cases, the conditioning encoder 7 shared with the graph encoder Eg. To enable good representation of both masked graphs and the joint representation of graphs with labels, we randomly mask the labels in the pretraining stage, so that both ( A) and (x, A. y) have small reconstruction errors- though the latter is typically much smaller than the former one. We do not add virtual nodes in QM9 datasets, and the graph feature is obtained by (sum or mean) pooling the nodes (and optionally, edges). For Zinc and egbg-molhiv we add virtual node where we embed the graph-level label via the encoder to obtain target refined full graph representations, and we use an embedding layer for class labels and an MLP for regression labels. All networks have ReL.U activations, and we use layer norm instead of batch norm for all transformers for training stability. However, in case (ii), as the structures are clearly defined and known, we can still utilize the MPNN
Zine and classification on
25

---

Table 10: Ablation study on latent dimension on Zinc (MAE 4).
Shown is the mean + std of 3 runs.
Latent dimension
Test MAE
0.081 0.006
8
0.080 + 0.006
16
0.084 + 0.007
family denoising models. We classify all node-level tasks into this case so that we could use MPNNS with (n) complexity to avoid the extensive computation of attention mechanisms.
C.3.2 Task specific settings
Unconditional generation on QM9, Here we provide other hyperparemeters for the experiments of QM9 unconditional generation. We denoising network has 256 hidden channels and self-attention layers. We train the model for 1000 epochs with batch size of 256. We adopt the cosine learning rate schedule with 50 warmup epochs, and the initial learning rate is le - 4.
encoder with 96 hidden features and layers. The
Conditional generation on QM9. We train a separate model for each target. The encoders have 3 layers and 96 hidden channels. The denoising networks have cross- cross-attention modules (8 self- attention layers and 4 cross-attention layers in total) and 128 hidden channels. We train for 1000 epochs with a batch size of 256, and the cosine learning rate schedule with initial learning rate le le-4. 4.
Regression Zine. The encoder has 64 hidden channels and 10 layers, augmented with GINE (Hu et al. 2019] and RRWP positional encodings [Ma et al., 2023]. The score network has 64 hidden channels and 4 cross-attention modules (8 self-attention layers and 4 cross-attention layers in total). We train the diffusion model for 2000 epochs with a batch size of 256, and a cosine learning rate with 50 warm-up epochs and initial value 1e- 4. We also conduct ablation studies on Zinc, see Appendix C.4.
Classification on ogbg-molhiv. Same as Zinc, the encoder has 64 hidden channels and 10 layers, augmented with GINE [Hu al., [2019] and RRWP positional encodings [Ma et al.] [2023]. The score network also has 64 hidden channels and 4 cross-attention modules. We train the diffusion model for 500 epochs with a batch size of 256, and a cosine learning rate with 20 warm-up epochs and initial value le 4. We observe overfitting phenomenon in the early stage, which may be caused by the imbalanced distribution between training set and test set, see Appendix C.5 for discussions.
Classification on node-level datasets. We use the combination of GCN Kipp and Welling 2016 and GAT [Velickovic et al. 2017] as the encoder, with 7 layers and 160 hidden dimensions. We use LapPE and RWSE. The score network has 192 hidden channels and cross-attention modules. We train the diffusion model for 500 epochs with batch size of 1, and cosine learning rate with 50 warm-up epochs and initial value le-44 4.
C.4 Ablation studies
We conduct ablation studies on Zinc dataset to investigate (a) the impacts of the dimension of the latent embedding; (b) whether ensemble technique helps improve the prediction quality as the generation is not deterministic.
Dimension of the latent embedding. We use encoders with output dimension (i.e. the dimension of the latent embedding for LGD) 4, 8, 16 respectively, and inference with the fast DDIM for 3 runs with different random seeds in each setting.
According to the results in Table 10 LGD-DDIM reveals similar performance with these latent dimensions, while
leads slight performance drop. This coordinates with our theoretical analysis in Appendix B large dimensions of latent space may cause large diffusion errors. It would
26

---

Table 11: Ablation study of ensembling on Zinc (MAB 4). Shown is the mean + std of 3 runs.
of ensembled predictions
Test MAE
0.081 + 0.006
5
0.081 + 0.006
9 0.079 = 0.006
be interesting to the performances in a much smaller (c.g. d = 1) or much larger (e.g. d = 64) latent space.
Ensemble of multiple generations. One possible issue with solving regression and classification tasks with a generative model that the output is not deterministic. To investigate whether the randomness improve or hurt the performance, test with DDIM using same trained model which ensembles over 1, 5, predictions respectively. We use the median the multiple predictions in each case.
As shown Table [1] ensembling over multiple predictions has almost no impacts on the performance. This is somewhat surprising at the first glance. It shows that the diffusion model can often output predictions around the mode with small variances, thus modeling the distributions well. As mentioned in the main text, the SDE-based DDPM also outperforms ODE-based DDIM, aligns with previous empirical and theoretical studies that the randomness diffusion models could lead to better generation quality (Cao et al. 2023).
C.5 Experimental findings and discussions
Evaluation metries in unconditional molecule generation. We now further discuss the evaluation metrics of unconditional molecule generation with QM9 dataset. In the main text we report the widely adopted validity and uniqueness metrics as most literature do. However, it is notable that the validity reported in Table (and also in most papers) is computed by building molecule with RdKit and calculating the propertion of valid SMILES string out of it. As explained in Jo et al. (2022). QM9 contains some charged molecules which would be considered as invalid by this method. They thus compute more relaxed validity which allows for partial charges. As a reference, GDSS-seq and GDSS Jo et al. 2022) achieve 94.5% and 95.7% relaxed validity, respectively; in comparson, our LGD achieves 95.8% relaxed validity, which is also better. We do not report the novelty result for the same reasons explained in (Vignac et al. 2022]. This is because QM9 is an exhaustive enumeration of the small molecules that satisfy certain chemical constrains, thus generating novelty molecules outside the dataset may implies the model has not correctly learned the data distribution.
Experimental findings. In our experiments, we observe many interesting phenomenon, which are listed below. We hope these findings could give some insights into further studies.
Regularization of the latent space. We experimentally find that exerting too strong regular- ization the latent space, e.g. large penalty of KI. divergence with the standard Gaussian, would lead to performance drop of the diffusion model. This is understandable as the latent space should not only be simple, but also be rich in information. Strong regularizations will cause the model to output a latent space that is too compact, thus hurt the expressiveness. We find in our experiments that a layer normalization of the output latent representations tends to result in better generation qualities, while KL or VQ regularization are not necessary.
Rapid convergence. We find that our latent diffusion model tends to converge rapidly in the early training stage. Compared with non-generative models such as GraphGPS [Rampásek) et al. 2022 LGD typically needs only 1 ~ number of epochs similar training loss, which makes the training of L.D much faster. We attribute this the advantages of the powerful latent space, as the diffusion model only needs to "refine" the representation top of the condition that is already close to ground truth.
them to achieve
Generalization. Interestingly, we find that while LGD usually have good predictions (in both train and test set) in the early stage of training, the test loss may increase the final stages, forming a "U" curve in test loss. This is obviously sign of overfitting due to the
27

---

strong capacity of diffusion models to model complex distributions. If the distribution shift between train and test set becomes larger (e.g. ogbg-molhiv compared with Zinc), the overfitting phenomenon gets more obvious. This implies that L.GD is extremely good at capturing complex distributions, and thus has the potential scale well larger datasets that fits with the model capacity.
D Discussions
D.1 More discussions on graph generative models
Among graph generative models, although some auto-regressive models show better performance, they are usually computationally expensive. Moreover, one worse property of auto-regressive graph generative models cannot maintain the internal permutation invariance of graph data. On the other hand, some one-shot graph generative models are permutation invariant, but those based on VAEs, normalizing flows and GANs tend to under-perform autoregressive models. Currently, graph diffusion generative models are the most effective methods, but they have to overcome the difficulty brought by discreteness of graph structures and possibly attributes. Consequently, [Niu et al. 2020) and (J al., 2022] can only generate the existence of edges via comparing with a threshold on the continuous adjacency matrix A E R.*x. Besides, [Niu et al. 2020 cannot deal with node or edge features, while [Jo al. 2022] can only handle node attributes. DiGress [Vignac et al. [022] operates on discrete space, which may have positive impacts on performance in some cases where discrete structures are important (e.g. molecule generation or synthetic graph structure data). However, DiGress is only suitable for discrete and categorical data, while they fail to handle continuous features. All these score-based or diffusion models are unable to provide all types of node and edge (and even graph) features, including categorical and continuous ones. In comparison, our model is able to handle features of all types including continuous ones. Moreover, our continuous formulation is empirically more suitable for well established classical diffusion models compared with discrete space. Our method also differs from Xu et al. [2023], where their model can only generate nodes and requires additional computations to predict edges based the generated nodes. Overall, our LGD is the first to enable generating node, edge and graph-level features of all categories in one shot.
Some concurrent works provide some insights and implications graph generation. Yang et al. [2023b) incorporates data-dependent, anisotropic and directional noises in the forward diffusion process to alleviate transforming anisotropic signals to noise too quickly. The denoising network directly predicts the input node features from perturbed graphs, and Yang et al. 2023bb also propose to use diffusion models for unsupervised graph representation learning by extracting the feature maps obtained by the decoder. Dan et al. [0033 estalishes a connection between discrete GNN models and continuous graph diffusion functionals through Euler-Lagrange equation. On top of this, Dan et al.) (023) propose two new GNN architectures: (1) use total variation and a new selective mechanism to reduce oversmoothing: (2) train GAN predict the spreading flows in the graph through neural transport equation, which is used to alleviate the potential vanishing flows.
D.2 Building the pretrain, finetune and in-context learning framework upon LGD
A complete training framework. We now describe the overall architecture and training procedure of the general purposed graph generative model that can be built upon LGD, and discuss the potential of LGD as the framework for future graph foundation models. As described before, the entire model consists of a powerful graph encoder & and latent diffusion model 0, which can be shared across different tasks and need to be pretrained although we currently train them separately for each task. As for different downstream tasks, we can adopt finctuning or prompt tuning strategies. In both cases, we need to train lightweight task-specific graph decoder D. We need to update the parameters of and 0 while finetuning. In prompt tuning, however, we keep these pretrained parameters fixed and only train a learnable prompt vector concatenated with input features of prompt graphs, which enables in-context learning ability for pretrained graph models. Notably, some prior work like PRODIGY (Huang al., 2023] conduct in-context learning by measuring the "similarity" between test and context examples, which is different from the conditional generation and prompting as in NLP and LGD.
promising future direction to pretrain the graph encoder and the generative model across a wide range of tasks. These two parts can be trained simultancous sequential, as well as through
28

---

different paradigm. Here. & includes the initial tokenization layer as well as a powerful graph learner backbone (e.g. GNN or graph transformer) that encodes input graph into semantic meaningful latent space for all downstream tasks. Pretrain of & consists of both unsupervised learning and supervised learning trained on downstream tasks with labels. The unsupervised loss includes both contrastive loss Leontrast and reconstruction loss of self-masking Frecon. The supervised learning is jointly trained with a downstream classifier or regression layer (that will not be used later), according to classification entropy loss Lee or regression MSE/MAE loss Cregrese.
We train the diffusion model in a supervised manner. Extra tokens are applied to represent masked values or attributes that need to be predicted. In traditional generation tasks, the training procedure is similar stable-diffusion, where we compute loss between predicted denoised graph and the input graph in the latent space encoded by 3. As for classification and regression tasks, we only input part of graph features, and expect the model to reconstruct the input features while predicting those missing features. For example, in a link prediction task, the input graph contains all node features and only part of edges. Then the model need to reconstruct the known input features and simultaneously predict the rest links. Therefore, the loss is the summation of reconstruction loss (in the latent space) and the corresponding classification or regression loss.
For tasks of different levels and types, note that the reconstructed graph representation in the latent space contains all node and edge features that are powerful enough for any downstream tasks. Thus only a (lightweight) task specific decoder D is needed, like classifier for classification task and a regression layer for regression task. The node and edge level attributes can be directly obtain based on corresponding latent representation, while graph attributes is available by pooling operations, or by adding a virtual node with trainable graph token for every task. To train a task specific decoder, we can either finetune along with the entire pretrained model on the target dataset (whole dataset and few shot are both possible), or only train them with prompt vectors while fixing the parameters of pretrained graph encoder E and generative model 0.
Domain adaptation. We now present the potential of our framework in domain adaptation, while leaving the experiments for future work. To deal with graphs from various domains, we create graph dictionary for all common node and edge types in graph datasets, such as atom name in molecular graphs. The vectorization is via the first embedding layer of the graph encoder network &, which is analogous to text tokenization in LLM. For those entities rich in semantic information, we can also make use of pretrained foundation models from other domains modals through cross attention mechanism in the latent diffusion model.
After finishing pretraining the powerful graph encoder E and the latent generative model 6, we need to adapt the entire model to target tasks. Previously, a widely adopted paradigm in graph learning field is the gvetrain-finetum. framework. During the finetune procedure, we jointly train the task specific decoder D, and finetune E and 9 on the target task. However, finetuning has at least the following drawbacks: (I) significantly computation cost compared with prompting, since we need to compute gradients of all parameters; (2) may lead to catastrophic forgetting of the pretrained knowledge, thus performing worse on other tasks and possibly requiring one independent copy of model parameters for each task.
Prompting is another strategy to adapt the pretrained model to downstream tasks, which designs or learns prompt vector as input while keeping the pretrained E and fixed. Compared with finetuning. prompting requires much less computation resources since we only need to Iearn an input vector. Here we introduce graph prompt mechanism which is the first to allow for real in-context learning. In the target task, we provide prompt example graphs with labels, whose input node and edge features are concatenated with a learnable task-specific prompt vector. The prompt vectors are learned through this few shot in-context learning, which adapt the model better to the task without retraining or finetuning. Besides, keep the hidden dimension aligned. we also concatenate a prompt vector in the pretrain and finetune stages, which can be serve as initialization of downstream prompt vectors in similar tasks.
D.3 Complexity of LGD and comparison with traditional models
As is shown in Appendix C Cempirically, L.D has good scalability with proper settings. We now provide thorough discussion the computation complexity and scalability of LGD.
29

---

The complexity of diffusion models mainly depends on: (i) the internal complexity of the denoising networks; (ii) the complexity brought by the diffusion process. For the complexity of the denoising networks, our L.GD framework is compatible with flexible network architectures, thus could always strike a good balance between complexity and expressivity. We consider both the generation tasks and prediction tasks. In the generation setting, we have n² complexity as we utilize our proposed graph transformers, which is generally unavoidable while generating from scratch. It is remarkable that previous models also have large complexity. Actually, GDSS [Jo et al. [2022] also models the
entire n X n adjacency matrix; GeoLDM [Xu et al. 2023] and EDM Hoogeboom et al. [022] both use EGNN, resulting to n² computation complexity a graph transformer with n² complexity. However, in the prediction settings (e.g. graph-level and node-level classification and regression tasks), we could leverage arbitrary GNN models when the graph structures are clearly defined - we only need to generate features. The complexity of LGD thus reduces to (n) if use MPNN models, which enables LGD to scale extremely large graphs just as traditional graph models do.
well: Digress |Vigncc et | [0222] also uses
For (ii) the complexity of the diffusion process, since in the training stage we always sample only one diffusion step, the complexity of diffusion models do not differ from traditional models (need only one forward pass of the denoising networks). In the inference stage, while the diffusion model needs to iteratively denoise, the inference mode is internally more efficient than training mode. We have also shown that LGD is capable of incorporating efficient inference methods including DDIM, which requires much less inference steps compared with T. Morcover, for easier tasks we can set T a relatively small value, which also helps to improve the overall efficiency.
Superior training efficiency of LGD reported Appendix C is also remarkable. Actually as we discussed in Appendix C.5] we empirically observe that the diffusion models have better training efficiency compared with traditional regression or classification models. This may due to the fact that compared with traditional models, diffusion models can learn the distribution in a very efficient way by through iterative sampling, which decompose the complex distribution transformation into easier steps. LGD also benefit from the powerful encoder, which uses the latent embedding containing label information as the training target of diffusion models. In detail, the diffusion models tries to recover E(x 3) conditioning on Ezx) When the encoder already have good representations (i.e. (xx is good estimator of y), the training of diffusion model can be extremely efficient. As our theory in Appendix B reveals, the latent diffusion models can outperform the encoders (which can be regarded as traditional regression or classification models) with some mild assumptions hold. Besides the rapid convergence in early training stage, we also observe some overfitting phenomenon (especially on smaller datasets), which indicate LGD's 's power in learning distributions and its potential to perform better when scale to larger datasets it would be an interesting future direction observe whether there would be emergence ability of diffusion-
generative models like LGD.
Moreover, we can always sample subgraphs for the large graphs that even the full graph trans- formers would be applicable. In summary, our framework is general and extremely flcxible. We can always balance between the complexity and effectiveness according to different task settings.
D.4 Discussion on solving prediction tasks with diffusion models
Although Li et al. [023a] also mention the idea of doing classification with diffusion models, their formulation is complex and computational costly as they have to enumerate all classes. In our framework, the diffusion model directly learns the conditional distribution p(y(z), while the diffusion models in [Li et al., 2023a] still Icarns p(r g) and they rely Bayes' thcorem to inference p()).)) Therefore, our model is more efficient since we only need one forward pass of diffusion models, while [Li ct al. 2023a needs N passes where N is the number of classes (which could be very large). Moreover, their setting is only suitable for classification, as they cannot apply Bayes' theorem to formulation
continuous prediction targets, thus failing to handle regression tasks. In comparison, is much simpler and more efficient, and is applicable to both regression and classification tasks. We also derive error bounds for the conditional latent diffusion models in these tasks, see Appendix B
is also interesting to explore the possibility of applying diffusion models to regression and classi- fication tasks in other fields, such as computer vision. It remains to see whether diffusion models could outperform traditional non- non-generative models in all task types, both inside and outside the graph learning field.
30

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and introduction clear state the claims made including contri-
butions, assumptions and limitations. The claims match the theoretical and experimental results.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: We include the Limitation section together with Conclusion section (due to limited space). Our empirical results are extensive and explained the detailed settings and computational efficiency in the paper. We do have problems of privacy and fairness.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged
the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
31

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: We clearly numbered and cross-referenced all theorems, formulas and proofs. The assumptions are learly stated and the proofs appear the supplmental material. Theo- rems and Lemmas that the proof relies upon are properly referenced.
Guidelines:
The answer NA means that the paper does not include theoretical results,
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We fully described the novel architectures, with detailed instructions for how to replicate the results. We properly describe the new algorithms and the new architectures.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to model. In general. releasing code and data often one good way to accomplish this, but reproducibilityy can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open- open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibiliity In the case of closed-source models, it may be that access to the model is limited in
32

---

some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: All the data are open sourced. We already provide sufficient experimental details in the appendix. We have released the code and the link is available in the paper.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://niis..ce)
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
7 //nips. /tipss.c/ppb.i.. httpccppccccccceeeeeeeeeeeeeeeee wwtpc//guiiis/e/eee SaweeeeeeoeSSeeeeeiieiii.e.e
for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification:
All the experimental setting is presented in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer:
[Yes]
Justification:
We report standard deviation of all results when possible.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
33

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We explaine in the appendix that all experiments could be carried out on one single RTX3090 GPU.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification:
We follow the NeurIPS Code of Ethics and preserve anonymity.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. if there special consid- eration due to laws or regulations in their jurisdiction..
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This is fundamental research paper and there's no social impact.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
34

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
Our model is for graph generation, which could hardly be misused.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We cite all the original papers that produced the code packages and datasets.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URI.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
35

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssi
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
36

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
37


Reviews:
[{"review_id": "KkWpRVzxBC", "reviewer": "Reviewer_suUP", "summary": "This paper presents Latent Graph Diffusion (LGD), a graph generation framework adept at handling a variety of tasks including generation, regression, and classification at different objects—node, edge, and graph levels. LGD approaches regression and classification tasks as conditional generation challenges and employs a latent diffusion model within the latent space to concurrently generate features for nodes, edges, and graphs.", "questions": "1.\tIt would be valuable to compare the computational efficiency of the evaluated methods during training and sampling to provide a comprehensive analysis.", "limitations": "N/A.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper introduces the Latent Graph Diffusion as a novel approach to address the challenges of using a unified framework to achieve classification, regression and generation tasks on graph.\n\n2. This paper clearly shows its idea.", "weaknesses": "1. While GDSS and DiGress are designed to handle various types of graph data beyond molecular diffusion, it is unclear if LGD has conducted similar graph generation experiments. The article lacks relevant general graph generation experiments and evaluations using metrics like MMD (Maximum Mean Discrepancy) to assess the effectiveness of graph generation.\n\n2. The authors have not included some successful practices of diffusion models. Recent studies [1,2] have demonstrated better performance than the proposed method on the QM9 dataset.\n\n3. The QM9 dataset used for graph generation is relatively small, and it would be beneficial for the authors to validate their proposed model on larger benchmark datasets, such as Moses [3], which also provide a wider range of evaluation metrics.\n\n4. The description of the model's architecture details is unclear and could benefit from further clarification.\n\n5. The submission does not include an open-source implementation of the proposed model.\n\n[1] DiGress: Discrete denoising diffusion for graph generation. ICLR2022.\n[2] Graph generation with diffusion mixture., ICML 2024. \n[3] Molecular sets (moses): a benchmarking platform for molecular generation models. 2020.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the useful comments and suggestions. As presented in the global response, we have conducted extensive new experiments to improve the paper. Here we address your concerns as follows.\n\n1. General graph generation (weakness 1). As explained in our paper, LGD is capable of generating arbitrary graphs through the powerful continuous latent space, hence is able to handle various types of graph data including generic and synthetic graphs. We conduct generic graph generation experiments and report the results in the global response. We use SBM, Planar and Community-20 datasets for evaluation. LGD performs better than most traditional graph generation methods on these datasets (although it is not always the best). However, other methods often have hard set rules or structural information in the diffusion process, while LGD performs a pure diffusion process in the Euclidean latent space without input structural information, requiring the encoder to be powerful enough to encode structural information. While MPNN and graph transformers are known to have limited expressivity, they may fail to encode enough structural information like degree, clusters, orbits and eigenvalues. Moreover, the autoencoder is trained through reconstruction loss, which has limited supervision on high-order structures. Therefore, if expressive higher-order GNNs or proper structural encodings are used, or more pretraining tasks related to complex structures are adopt, we will be able to obtain a more powerful encoder and thus better generation qualities, which we leave as future work. Please note that LGD has superior performance in real-world tasks where node/edge features are often more important than high-order structures, making it easier to obtain more powerful and suitable encoders for diffusion.\n\n2. Baselines and metrics (weakness 2). We rerun our experiments on QM9 and compare our results with state-of-the-art methods including DiGress [1], HGGT [2] and GruM [3]. We also include the FCD and NSPDK metrics to better evaluate the quality of generation. The results are shown in Table 1 of the pdf in the global response. In our new experiments, we train a more powerful encoder/decoder by incorporating positional encodings and more complex reconstruction tasks to obtain better latent graph representations, and carefully choose the hyper-parameters of the diffusion model. It is remarkable that our new results now are highly competitive or superior than the baselines in nearly all metrics. Especially, it achieves a better FCD score (in favor of generating graphs similar to the training data) than the three strong recent baselines [1,2,3] while sacrificing the least novelty.\n\n3. Larger datasets (weakness 3). We are conducting new experiments on the larger benchmark dataset MOSES. The results are not yet ready due to the limited time. We will update the results as soon as possible, using comments and the link in the global response. We will include the new results in our revised paper. It is also notable that most recent studies including HGGT [2] and GruM [3] also do not include these datasets at scale.\n\n4. Model architecture details (weakness 4). Actually, we have included our model architecture details in the appendix of our submission. Please refer to Appendix D.1 for our architecture design and implementation details, and Appendix D.3.1 for the detailed architectures utilized in each experiment. \n\n5. Open-source implementation (weakness 5). We provide the code through the anonymous link in the global response. We will definitely include the open-source code in the final version of our paper.\n\n6. Computation efficiency (question 1). We have actually included comprehensive analysis on computation efficiency in Appendix E.2 of our submission. In summary, the complexity of LGD depends on the specific architecture of the autoencoder and the denoising network, which is generally $O(n^2)$ for generation tasks and $O(n)$ for regression/classification tasks. The training stage of the diffusion model is fast, and the inference stage can also be accelerated through various methods (such as DDIM, which we already implemented). As a reference, training LGD requires about $0.2$s/epoch on Cora and about $16$s/epoch on ZINC.\n\n[1] C. Vignac, I. Krawczuk, A. Siraudin, B. Wang, V. Cevher, and P. Frossard. DiGress: Discrete denoising diffusion for graph generation. In International Conference on Learning Representations, 2022.\n\n[2] Y. Jang, D. Kim, and S. Ahn. Graph generation with K2-trees. In The Twelfth International Conference on Learning Representations, 2023.\n\n[3] Jaehyeong Jo, Dongki Kim, and Sung Ju Hwang. Graph generation with diffusion mixture., ICML 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Bz2ZijWQDf", "reviewer": "Reviewer_XBTL", "summary": "This paper proposes a model framework, LGD, which addresses multiple types of graph tasks and can simultaneously handle both generative and predictive tasks. Specifically, the LGD framework employs a method similar to stable diffusion, using a pretrained graph encoder to obtain latent representations and then training a diffusion model in the latent space. To use the diffusion model for predictive tasks, the authors incorporate the label in regression and classification tasks as a partially masked feature, using the rest of the training set as a condition to predict the masked label. Finally, the authors validate their approach through experiments for prediction, classification, and regression tasks.", "questions": "- Did the authors use a shared pretrained diffusion model for different tasks? If so, what data was it trained on?\n\n- What do the authors think about the feature misalignment problem in graph pretraining?", "limitations": "Please refer to the Weakness and Questions above.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The idea of using a conditional diffusion model for predictive tasks is novel. Unlike traditional representation learning approaches, LGD can directly generate labels using the diffusion model, providing a new perspective for the community.\n\n- The experiments cover a comprehensive range of tasks, datasets, and baselines. The model shows promising results on most datasets.\n\n- The authors provide theoretical support for using latent diffusion models in regression tasks and discuss the advantages of converting deterministic tasks into generation tasks.", "weaknesses": "- The model functionality described in the introduction is somewhat misleading. In lines 28-29, the authors claim to construct a graph model capable of solving all task types. However, the implementation does not use a single foundation model to accomplish all tasks. Since the proposed framework still cannot solve the feature misalignment problem, specific tasks require training specific encoders and decoders. This makes the introduction seem like an overclaim, as LGD is just a framework that can handle multiple tasks but includes many components that need task-specific training. From a training cost perspective, this is similar with using separate models for different tasks (and potentially more expensive due to the need for GT and encoder-decoder). It is noteworthy that the authors mention this drawback in the limitations section.\n\n- The experiments lack details on training and potential unfairness.\n\n(1) In Appendix C, the authors describe that the encoder $\\mathcal{E}$ was pretrained with both unsupervised and supervised methods. However, the main text does not specify which datasets were used for the encoder's training and whether they are the same as those used for training the diffusion model or if additional datasets were used. For instance, in the Conditional generation task, if the encoder was supervised-trained on the QM9 dataset, the results in Table 2 would lack fairness. This is because other baselines do not have access to supervised training information when providing conditions, giving the encoder a clear advantage. The authors should clarify the training data details for the encoder.\n\n(2) In line 762, the authors state that the latent diffusion model is shared across different tasks, but the paper does not provide details on what data this shared diffusion model was trained on. Was it trained on the entire dataset across all tasks? This also lacks detailed description.\n\n- Some of the authors' statements are overclaimed or inaccurate:\n\n(1) In the analysis of the unconditional generation task (QM9), the authors claim that generation based on 3D information is easier, highlighting their method's effectiveness. This premise is incorrect. First, RNN-based models [1], which generate based on SMILES, can achieve a validity of 99% and high uniqueness without 3D information. 3D generation actually targets a different need—conformation generation—related to molecular dynamics and has broader research and practical value, hence the use of 3D information. It is not that generation based on 3D information is easier. Moreover, the three metrics the authors compare are not the best indicators of molecular generation quality. I suggest the authors use more effective metrics such as QED and TPSA. Previous works, especially those focusing on molecular generation tasks, do not limit evaluations to these two metrics due to their limited persuasiveness.\n\n(2) In line 767 of the appendix, the authors claim their work is the first to achieve in-context learning on graph pretrained models. In fact, there are several prior works on graph prompts, such as PRODIGY [2].\n\n[1] Molecular de-novo design through deep reinforcement learning. Journal of Cheminformatics.\n[2] PRODIGY: Enabling In-context Learning Over Graphs. NeurIPS 2023.\n\n- The discussion in lines 111-112 about the capabilities of generative models lacks citations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewers for the constructive comments and suggestions. We address your concerns as follows.\n\n1. Clarification on the introduction section (weakness 1). The introduction section mainly states the motivation to build a graph foundation model, but we do not claim that we have already done so (even in lines 28-29). We will surely rewrite the paper to avoid misunderstanding. Actually, LGD is indeed a unified framework to handle multiple tasks. Although we do not train a single model currently (as mentioned in our paper), we can indeed build a single graph foundation model under the LGD framework. For example, we can use OFA [1] as the autoencoder, which is a single model that embeds graphs from various domains into a unified latent space through a powerful LLM-GNN mixture architecture. We can then train a diffusion model in this latent space to tackle tasks of all levels and all types, utilizing the unified formulation in our paper. We believe the unified formulation of our paper is a significant step and a necessary stage towards graph foundation models.\n\n2. Clarification on experiments (weakness 2). Appendix C actually describes the potential to build a complete pretrain, finetune and in-context learning framework using LGD. The contents are mainly discussions on future work about LGD, where we have not fully implemented them. We will move Appendix C to the discussion and future work section in the revised version of our paper and make it clearer.\n\n* For your question 2.(1) regarding the conditional generation task on QM9, the encoder is pretrained through the reconstruction loss of nodes and edges as described in the main context, without targets as supervision signals. The diffusion model still generates the molecules, and we follow the prior practice to use another pretrained model to evaluate the properties of all generated molecules, so our method does not introduce any unfairness. \n\nIn addition, we would like to emphasize that even the encoders are trained in a supervised manner, it still does not introduce unfairness. Denote the conditions (properties) as $y$ and the molecules as $x$, there is a ground truth distribution $p(x|y)$. In training we sample $x\\sim p(x|y)$ and denote it as a probabilistic mapping $f(y)$. The traditional conditional generation methods model $\\hat x:=\\theta(y)$ with a supervision signal $f(y)$. In comparison, our latent diffusion model learns the distribution of latent representation $\\mathcal E(\\hat x, \\hat y):=\\theta(y)$ with a supervision signal $\\mathcal E(x, y)=\\mathcal E(f(y), y)$. We can observe that both schemes include a supervision signal containing $y$ for generative models, except that we use an encoder to better represent the desired property. This is actually an advantage: utilizing representation learning, latent diffusion can better learns conditional distribution without introducing unfairness.\n\n* For your questions 2.(2), we train a separate encoder/decoder and diffusion model for every dataset for now. However, encoders and diffusion models trained across multiple datasets may have better performance and generalization, which we leave as future work. \n\n3. Clarification on some statements. We thank the reviewer for the advice. There are indeed some statements that can be refined - we will correct them in the revision.\n\n(1) It is true that 3D generation is not necessarily easier. Our recent experiments also find that 3D generation can sometimes be harder due to the difference in data distribution and internal network architectures. We will correct related discussions in the paper. We also acknowledge the practical value of 3D generation and leave it as future work. Regarding the evaluation metrics, we additionally introduce the persuasive FCD and NSPDK metrics in our new experiments for better evaluation, and LGD achieves highly competitive performance. We do not include QED and TPSA for the QM9 dataset, since drug-likeliness is not suitable for the small molecules in QM9.\n\n(2) We are aware of PRODIGY and did not mention it since it works in a slightly different manner compared with the in-context learning in NLP: PRODIGY works by measuring the \"similarity\" between test and context examples, which is different from the conditional generation as in NLP. However, it is true that PRODIGY is also one sort of ICL, and we will make sure to include this citation and discussion in the revised version.\n\n4. We will add the discussion concerning generalization ability of generative models, including LLM (Llama-3 and GPT-4 etc.) and diffusion models [2-3].\n\n5. Answers to questions. (1) We train diffusion models separately as explained in 2. (2) Feature misalignment is indeed one of the obstacles to build a graph foundation model. LGD provides the potential simultaneously perform generation and regression/classification as long as graphs with different features are mapped to a unified latent space. Actually, OFA [1] is a single model that embeds graphs from various domains into a unified latent space through a powerful LLM-GNN mixture architecture. Combining LGD and OFA, we can then train a generative model in the latent space to overcome the difficulty of feature misalignment and task unification problems, which is left for future work. Overall, we believe that LGD is a significant contribution to the community and could inspire future work on graph foundation models.\n \n[1] Liu, H., Feng, J., Kong, L., Liang, N., Tao, D., Chen, Y., and Zhang, M. One for all: Towards training\none graph model for all classification tasks. ICLR, 2024.\n\n[2] Zahra Kadkhodaie, Florentin Guth, Eero P Simoncelli, and Stephane Mallat. Generalization in diffusion models arises from geometry-adaptive harmonic representations. ICLR, 2024.\n\n[3] Puheng Li, Zhong Li, Huishuai Zhang, and Jiang Bian. On the generalization properties of diffusion models. NeurIPS, 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "p6xgvWOFou", "reviewer": "Reviewer_a5WK", "summary": "The paper proposes Latent Graph Diffusion to generate node, edge, and graph-level features to meet the need of different tasks under the unified framework. Extensive experiments show the competitive performance across various graph-based tasks.", "questions": "1. How does the LGD framework scale with very large graphs? \n2. In the task of node classification from Planetoid, is the LGD model capable of generating a new graph almost the same as Cora or PubMed?\n3. The graph will be too dense when considering no link as a special type of edge? How to deal with this problem", "limitations": "1. The lack of detailed analysis of time and space complexity makes the performance of the model in practical applications less transparent.\n2. The sensitivity analysis of parameters in the conditional generation task is insufficient, especially since the influence of important hyperparameters is not studied deeply.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. It is good to reformulate regression and classification problems with diffusion generation task.\n2. The paper is technically sound, and the introduction of diffusion in latent space is well-motivated.\n3. The unified framework has theoretical support.", "weaknesses": "1. Readers have to jump to Appendix over and over again to have a general understanding of your proposed model. \n2. Despite the unified formulation as generation, other parts of the paper lack novelty. For example, there is a lack of new breakthroughs in specific model design and diffusion models. The underlying diffusion and denoising models largely build upon existing methods.\n3. No code or demo available.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the comments and acknowledgement. \n\n1. Writing (weakness 1). We will refine the writing of the paper and provide as much important information as possible in the main text.\n\n2. Novelty (weakness 2). We believe that our unified formulation is novel and beneficial. Regarding model design, we design novel self-attention and cross-attention mechanisms for graphs with augmented edges, as shown in Appendix D.1. For the diffusion model, we explained in our main text that we use the latent diffusion to tackle the problems brought by the discrete structure, feature misalignment and task variety. It is novel to use a latent diffusion model to perform graph learning tasks of all levels and all types. Furthermore, while architecture and diffusion model design are not our main focus, our LGD framework is extremely flexible, being able to incorporate various model designs and diffusion processes (e.g. DDPM, DDIM, Rectified flow etc.).\n\n3. Source code (weakness 3). We provide the code through the anonymous link in the global response. We will definitely release all the code and checkpoints in the camera-ready version.\n\n4. Scalability of LGD (question 1 and 3). We have explained the scalabiltiy of LGD in multiple places in appendix, such as architecture design in Appendix D.3, theoretical complexity in Appendix E.2, and experimental verifications in Appendix D.2 (line 887-891). We do not have to always model the dense graph where no link is a special type of edge---actually when we do not need to generate structures, we can utilize the sparse graph and only consider the real edges. We can thus use efficient MPNN based models to scale LGD to extremely large graphs such as OGBN-Arxiv, see Table 3 in the paper.\n\n5. Node classification (question 2). In classification and regression tasks, generating the entire graphs is not our main focus---LGD mainly aims to generate the class labels of nodes. However, since LGD actually refines the representation in the latent space, it is still able generate the whole graph. LGD outputs the refined representation, which contains the information of both input (graph structures and known features) and class labels to be predicted, thus it could generate new graphs similar to the training sets.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "apvyAN3Got", "reviewer": "Reviewer_PVEb", "summary": "This paper proposes a Latent Graph Diffusion model that unifies multiple tasks such as classification, regression, and generation in a generative task. Additionally, this model achieves feature generation in multiple categories, including node-level, edge-level, and graph-level.", "questions": "1.\tWhether the Encoder and Decoder are permutation equivariant?\n2.\tWill the authors release the source code and checkpoints?", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. This paper attempts to address a crucial task in the field of graphs, which is to propose a model capable of handling various tasks such as classification, regression, and generation at different levels, including node-level, edge-level, and graph-level.\n\n2. The writing in this paper is relatively clear.", "weaknesses": "1. The evaluations conducted by the authors are insufficient. For example, in the generation task, they avoid comparing their proposed method with classical diffusion-based graph generation models such as DiGress [1], HGGT [2], and DruM [3], which perform better in terms of metrics like Validity on QM9. Additionally, they avoid comparisons on larger datasets like MOSES [4] or GuacaMol [5]. Furthermore, they do not compare with commonly used metrics for graph generation tasks, such as FCD and NSPDK.\n\n2. Regarding the approach of treating classification and regression tasks as generation tasks, the authors should provide a simple baseline (which can be seen as an ablation study) to demonstrate that they have not unnecessarily complicated simple and intuitive tasks. For example, they could use condition features as input and train a baseline prediction model with the same architecture as the denoising model, using the corresponding masked global features as supervised signals. This would help assess the benefits brought by the generation model.\n\n3. The authors aim to develop a model that can address graph learning tasks at all levels (node, edge, and graph) and types (generation, regression, and classification), which is hard to achieve. The authors are suggested to provide a clear table to include detailed information about the source and size of the training dataset, as well as the number of model architecture and parameters for reference.\n \n4. The explanation of how the decoder of generation tasks reconstructs the structural information, i.e., A, of the graph is not clear.\n\n5. Figure 1, mentioned in the Section 5, is displayed in the Appendix.\n\n[1] C. Vignac, I. Krawczuk, A. Siraudin, B. Wang, V. Cevher, and P. Frossard. DiGress: Discrete denoising diffusion for graph generation. In International Conference on Learning Representations, 2022. \n\n[2] Y. Jang, D. Kim, and S. Ahn. Graph generation with K2-trees. In The Twelfth International Conference on Learning Representations, 2023. \n\n[3] Jaehyeong Jo, Dongki Kim, and Sung Ju Hwang. Graph generation with diffusion mixture., ICML 2024. \n\n[4] D. Polykovskiy, A. Zhebrak, B. Sanchez-Lengeling, S. Golovanov, O. Tatanov, S. Belyaev, R. Kurbanov, A. Artamonov, V. Aladinskiy, M. Veselov, et al. Molecular sets (moses): a benchmarking platform for molecular generation models. Frontiers in pharmacology, 11:565644, 2020. \n\n[5] N. Brown, M. Fiscato, M. H. Segler, and A. C. Vaucher. Guacamol: benchmarking models for de novo molecular design. Journal of chemical information and modeling, 59(3):1096–1108, 2019.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the useful comments. We now address your concerns as follows.\n\n1. Experiments (weakness 1). \n\n* Originally, LGD is designed to be a single framework that can do both generation and regression/classification, where we put our experimental focus on demonstrating that latent diffusion models can achieve state-of-the-art regression/classification results. Now, we have rerun our generation experiments on QM9 and (i) compare our results with state-of-the-art diffusion-based graph generation methods including DiGress [1], HGGT [2] and GruM [3]; (ii) use metrics including FCD and NSPDK to better evaluate the quality of generation. In our new experiments, we train a more powerful encoder/decoder by incorporating positional encodings and more complex reconstruction tasks to obtain better latent graph representations, and carefully choose the hyper-parameters of the diffusion model. The results are shown in Table 1 of the pdf in the global response. It is remarkable that our new results are highly competitive or superior than the baselines in nearly all metrics, which reveals the strong generation ability of LGD. Especially, LGD achieves a better FCD score (in favor of generating graphs similar to the training data) than the three strong recent baselines [1,2,3] while sacrificing the least novelty.\n\n* We are conducting new experiments on the larger dataset MOSES. Since the time is limited, the results are not yet ready. We will update the results as soon as possible, using comments and the link in the global response. We promise to include the new results in our camera-ready paper. It is also notable that most recent studies including HGGT [2] and GruM [3] also do not include these datasets at scale.\n\n2. Clarifications on treating classification and regression tasks as generation tasks (weakness 2). Actually, in Appendix B of our paper, we already provided solid theoretical analysis and clarifications on why diffusion models can perform better than traditional deterministic models in classification and regression tasks. In detail, Theorem B.4. provides error bounds of diffusion models on regression tasks, equation (52) explicitly states the condition where diffusion models outperform regression models, and Corollary B.6. provides guarantees that there exists at least one latent diffusion model whose MAE is not larger than the autoencoder. Combining all these pieces, we can use arbitrary existing prediction models as the autoencoder, and the diffusion model trained on top this autoencoder is able to perform better than this prediction model. To empirically verify this, we utilize the ZINC dataset and train a regression model with our proposed graph attention architecture (the same as the denoising model) using the masked global features (regression target) as the supervising signal. This prediction model has $0.084\\pm 0.004$ test MAE on ZINC, while the diffusion model trained upon the autoencoder with the same architecture achieves $0.065\\pm0.003$ MAE. This results verifies that we are not complicating simple tasks---instead, diffusion models can provably outperform traditional prediction models.\n\n3. Clarification on training (weakness 3). First of all, LGD is a unified framework (and not necessarily a single model) for all levels of graph learning tasks. As explained in both the limitation section and the experiment section of our paper, we currently train our models separately for each task. However, we are indeed able to train a single model for all tasks using LGD. For example, we can use OFA [4] as the autoencoder, which is a single model that embeds graphs from various domains into a unified latent space through a powerful LLM-GNN mixture architecture. We can then train a diffusion model in this latent space to tackle tasks of all levels and all types, utilizing the unified formulation in our paper. We believe the unified formulation of our paper is a significant contribution and a necessary stage towards graph foundation models. As for the model architectures, details are provided in Appendix D.1 and D.3.1. As a reference, the total number of parameters of LGD (including both the autoencoders and the diffusion model) is 1,825,481 for the regression task on ZINC, and 6,605,545 for the generation task on QM9.\n\n4. Decoding graph structure (weakness 4). As clearly explained in our main text, the latent space contains both the structural information and edge features, since the encoder treats absence of an edge as a special feature. We have a latent representation $W\\in \\mathbb R^{n\\times n\\times d}$ for adjacency matrix $A$ (line 145), which is exactly the diffusion target. After obtaining the generated latent edge-augmented representations $\\hat W\\in \\mathbb R^{n\\times n\\times d}$, we directly apply the pretrained decoder (which is actually a linear layer) to reconstruct both the graph structure ($A$) and edge features. In summary, LGD is a one-shot generation method that generates graph structures and features simultaneously.\n\n5. Figure (weakness 5). Figure 1 is presented in the Appendix due to limited space. We will move the figure to the main context in the camera ready version.\n\n6. Equivariance (question 1). The encoder and decoder are indeed permutation equivariant. We can either use existing models such as MPNN and graph Transformers, or use our specially designed graph attention mechanisms as encoder/decoder. All of the above models are permutation equivariant. Actually, we also use MPNN and graph transformers for the denoising network, making our diffusion model also permutation equivariant (in terms of each denoising step).\n\n7. Source code (question 2). We provide our code through the anonymous link in the global response. We will definitely release all the source code and checkpoints in the final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely thank the reviewers for their insightful comments and suggestions. To address the concerns, we conduct new experiments as follows.

* For QM9 generation task, we (i) add more persuasive evaluation metrics including FCD and NSPDK to better evaluate the generation quality; (ii) take recent state-of-the-art baselines into account; and (iii) rerun our experiments with better configurations. In detail, we pretrain a more powerful encoder which takes nove-level and edge-level positional encodings (PE) as inputs, which is trained through more complex reconstruction tasks (decoding bond type with two connected atoms and decoding atom type with connected bonds). We then train a larger diffusion model with carefully searched hyper-parameters. Our new model achieves highly competitive and even state-of-the-art performance in all metrics except novelty. The convincing results validate the strong generation capability of LGD. Detailed results are shown in the pdf.

* We conduct generic graph generation experiments and report the results in the attached PDF. We use SBM, Planar and Community-20 datasets for evaluation. LGD performs better than most traditional graph generation methods on these datasets (although it is not always the best). However, other methods often have hard-set rules or additional structural information in the generation process, while LGD performs a pure diffusion process in the Euclidean latent space without explicitly input structures or hard rules, which requires the encoder to be very powerful to distinguish subtle structural differences. Since MPNN and graph transformers are known to have limited expressivity, they may fail to encode some structural information like clusters, orbits and eigenvalues, resulting in lower generation quality in pure-structure tasks. Please note that LGD has superior performance in real-world tasks where node/edge features are often as important as structures, making it easier to obtain more powerful and suitable encoders for diffusion. Overall, LGD is designed to handle various tasks on real-world graphs as shown by the other parts of our experiments.

* We are conducting experiments on larger datasets including MOSES. Due to the limited time, the final results are not yet ready. We will update the results as soon as the experiments are finished, through both official comments and the anonymous link below: https://anonymous.4open.science/r/NeurIPS2024Rebuttal-0290/.

* We also provide the code through the same anonymous [link](https://anonymous.4open.science/r/NeurIPS2024Rebuttal-0290/) above. We will release our source code in the final version.

Meta-review: This paper presents a latent graph diffusion model which not only serves as a generative model, but it can also deal with classification and regression tasks at the node-, edge-, and graph-level.

While reviewers initially had some concerns and questions, most of them were addressed during the rebuttal phase, and all the reviewers are positive about this paper. The reviewers agree that the paper is generally well-written and that the idea of using a unified framework to solve classification, regression and generation tasks on graphs is novel. Indeed, while many of the ideas in this paper are not new (latent diffusion models have already been explored in the context of graphs [1]), unifying generation and prediction is an interesting direction. I would thus recommend the authors put more focus on this aspect. For example, the proposed method could be further evaluated in classification and regression tasks.

Overall, the consensus is that this is a useful contribution and should be accepted. I strongly advise the authors to include into the final version of the manuscript additional results and other revisions suggested by the reviewers and promised in the rebuttal.

[1] Xu, M., Powers, A. S., Dror, R. O., Ermon, S., & Leskovec, J. Geometric Latent Diffusion Models for 3D Molecule Generation. In ICML'23.
Decision: Accept (poster)