Bias Mitigation in Graph Diffusion Models
Authors: Meng Yu
Kun Zhan
Venue: ICLR 2025 Poster (2025)
Keywords: Diffusion models
Graph learning
Bias analysis
Primary area: generative models

Abstract: Most existing graph diffusion models have significant bias problems. We observe that the forward diffusion’s maximum perturbation distribution in most models deviates from the standard Gaussian distribution, while reverse sampling consistently starts from a standard Gaussian distribution, which results in a reverse-starting bias. Together with the inherent exposure bias of diffusion models, this results in degraded generation quality. This paper proposes a comprehensive approach to mitigate both biases. To mitigate reverse-starting bias, we employ a newly designed Langevin sampling algorithm to align with the forward maximum perturbation distribution, establishing a new reverse-starting point. To address the exposure bias, we introduce a score correction mechanism based on a newly defined score difference. Our approach, which requires no network modifications, is validated across multiple models, datasets, and tasks, achieving state-of-the-art results.

Published as a conference paper at ICLR 2025
BIAS MITIGATION IN GRAPH DIFFUSION MODELS
Meng Yu, Kun Zhan"
School of Information Science & Engineering, Lanzhou University
ABSTRACT
Most existing graph diffusion models have significant bias problems. We observe that the forward diffusion's maximum perturbation distribution in most models de- viates from the standard Gaussian distribution, while reverse sampling consistently starts from a standard Gaussian distribution, which results in a reverse-starting bias. Together with the inherent exposure bias of diffusion models, this results in degraded generation quality. This paper proposes a comprehensive approach mitigate both biases. To mitigate reverse-starting bias, we employ newly designed Langevin sampling algorithm to align with the forward maximum perturbation distribution, establishing new reverse-starting point. To address the exposure bias, we introduce a score correction mechanism based on a newly defined score difference. Our approach, which requires network modifications, is validated across multiple models, datasets, and tasks, achieving
results.
INTRODUCTION
In recent years, graph diffusion models have made significant progress. GDSS (Jo et al. 2022)
introduced the score-based diffusion model to the graph generative task, demonstrating remarkable results and outperforming existing baselines. This was followed by the development of more advanced
graph diffusion models such as MOOD (Lee et al. 2023), GSDM (Luo et al. (024), and HGDM (Wen et al. 2024). Due to the constraints imposed by the scale of graph data and the learning capacity of the networks, these models truncate the forward diffusion process improve performance, preventing from fully reaching the standard Gaussian distribution. However, in reverse sampling, they typically start from standard Gaussian distribution without employing any specific strategy. We identify this mismatch as a critical bias issue. In addition, graph diffusion models also suffer from exposure bias, and we are working on addressing both of these challenges.
Diffusion models (Ho et al. 2020; Song et al., 2021) consist of a forward noising and reverse denoising process. In the forward process, the data is gradually corrupted by noise over multiple steps. This process can be divided into four stages with the reduction of the signal-to-noise ratio: (1) the data distribution, (2) the low-noise stage, (3) the high-noise stage, and (4) the standard Gaussian.
Reverse-Starting Bias. Ideally, the forward process gradually perturbs the data to the standard Gaussian, while the reverse process starts from the standard Gaussian and gradually recovers clean data. However, in graph learning, due to limitations in data scale and the network's learning ability, it is difficult to accurately predict scores from the high-noise state or the standard Gaussian. This constrains the forward perturbation to follow conservative strategy, where the maximum perturbation distribution deviates significantly from the standard Gaussian (Jo et al. [002]] [uo et al. 2024 WWnn al. 2024). Yet, the reverse-starting point remains the standard Gaussian in sampling, resulting in severe reverse-starting bias as shown in Fig. T which significantly affects the generation quality.
Exposure Bias. During the forward process, the model generates noisy sample æ; based on clean sample by adding noise. During the reverse process, the model starts from the standard Gaussian and iteratively denoises to obtain the predicted sample & using the score network. Due to the prediction error of the score network, this leads to exposure bias: mismatch between æ; in the forward diffusing and á; in sampling. This bias accumulates and propagates as sampling progresses, ultimately affecting the quality of the generated samples. Naturally, the most direct approach to address exposure bias is to reduce the prediction errors of the score network.
"Corresponding Author, Email: kzhan @ iru.edu.cn

---

Published as a conference paper at ICLR 2025
Model
GSDM
HGDM
300
300
Dotorer
Corm
Lnz
QMY
ZENC25K
Training Sampling
Triniing Sumpling-S++t(uum
Type
Edge
Eigen
Edge
Node
Score nomE
200
SDE
VPSDE
VPSDE
VESDE
VPSDB
0.1
0.1
0.1
100
100
1.0
0.7596
0.4231
%
430
1000
%
600
1000
Lime
time
(a) Reverse-starting bias
(b) Baseline model-GDSS
(c) Our approach GDSS-S++
Figure 1: (a) According to the diffusion equation, write the perturbation distribution as ((2;) (ato) = N (arelu,,2 luy,0; off).The The maximum perturbation distribution the forward diffusing is V por, of I). but the reverse-starting point always follows N (xxr 0. I), leading to signifi- cant inconsistencies between forward diffusing and reverse sampling. In particular, we observe that 2r of different baselines are in the low-noise state since their signal-to-noise ratios are always greater than one. More details provided in Appendix A (b) and (c) Expectation of [| (-) during sampling (without the corrector) and training on Community
Due to the reverse-startng bias, there a significant difference between forward diffusing and reverse sampling in the early stages of sampling in (b). However, after applying our approach, not only is the reverse-starting bias mitigated, but the exposure bias during the sampling process also significantly reduced.
Rather than exploring the two biases independently, this paper aims to analyze and mitigate these two biases in graph diffusion models from comprehensive perspective:
Q1: Is it possible to mitigate exposure bias while addressing reverse-starting bias? It originates from a key finding: when æ; in the high-noise stage, the model is highly sensitive to the prediction error of the score network, which means the prediction етгоr at this stage can significantly affect the generative quality. Conversely, when 2, in the low-noise state, the model is quite resistant to the prediction error. Coincidentally, the forward maximum perturbation distribution of many models are in the low-noise state as shown in Fig. [a.. Thus, for a given score network sg,x()). we use Langevin sampling (Song & Ermon 2019) with sa,r(-) )-) to estimate the forward maximum perturbation distribution y ( Fo) mitigates the reverse-starting bias, meanwhile, it pushes the reverse-starting point towards the low-noise state, utilizing the model's resistance to prediction error to avoid exposure bias.
However, the prediction error of the score network severely affects the stable distribution of Langevin sampling, forcing us to improve the prediction accuracy of the network, which also beneficial for mitigating exposure bias in the sampling process. In particular, we also focus the cost of achieving:
Q2: How to correct scores without modifying the network or introducing other components?
The issue arises from key situation: current graph diffusion models design different networks based on various domains, such as spatial, spectral, and hyperbolic domains. Our goal is to seamlessly integrate our correction method into these models without modifying the network architecture or model parameters. Additionally, we do not introduce any extra learner, such as generator (Zheng 2023) or a discriminator (Kim et al. 2023). Instead, we aim fully leverage the existing diffusion model to address its inherent bias issues. First, we use the pretrained score network to generate batch of samples. Second, we train a pseudo score network using these generated samples. Third, we use the score difference between the two networks to correct the score.
In summary, the main contributions this paper are as follows: (1) To the best of our knowledge, we are the first to systematically address bias issues in graph diffusion models, effectively employing Langevin sampling to mitigate reverse-starting bias while significantly mitigating exposure bias in graph sampling. (2) We propose score correction mechanism based the score difference, and provide a logical analysis that the corrected scores are closer to the true scores, further mitigating sweers--tarring bias and exposure bias. (3) Our approach does not require modifying the network or introducing new components. It has been validated on different graph diffusion models, different datasets, and different tasks, achieving state-of-the-art metrics.
2

---

Published as a conference paper at ICLR 2025
2 RELATED WORK
Diffusion models were first introduced by Sohl-Dickstein et al. (2015) and later improved by Ho et al. (2020). Notably, Song al. (202I) proposed unified framework for diffusion models based on stochastic differential equations, greatly advancing their development. GDSS (Jo et al.] 2022) was the first to introduce diffusion models to both nodes and edges of graphs. GSDM (Luo et al. 2024) extended GDSS by introducing the diffusion process of adjacency matrices into the spectral domain. HGDM (Wen et al.) 2024) introduced node diffusion into hyperbolic space based degree distribution characteristics. MOOD (Lee et al. 2023) propose the molecular out-Of-distribet diffusion to generate novel molecules with specific properties. Huang et al. (2023) proposed conditional diffusion model based on discrete graph structures. Furthermore, Xu et al. (2022)
proposed a graph diffusion model for predicting molecular conformations.
The signal leakage in image diffusion models is somewhat similar to the wwerse-starting bias we found in graph diffusion models. was first identified Lin et al. (2024), who modified the diffusion noise schedule to ensure that the final time step in the forward process achieves a zero signal-to-noise Ratio. Then, Everrert et al. (2024) estimated the actual maximum perturbation distribution in the forward process as the reverse-starting point for sampling. The exposure bias of diffusion models was initially discovered in ADM-IP (Ning et al. 2023). which proposed re-perturbing the perturbation distribution to simulate exposure bias during sampling. EB-DDPM (Li & van der Schaar 2023) estimated the upper bound of cumulative errors and incorporated as a regularization term to retrain the model. MDSS (Ren et al. (024) introduced multi-step timed sampling strategy mitigate exposure bias. Notably, ADM-IP, EB-DDPM, and MDSS all require retraining the model. In contrast, ADM-ES (Ning et al. 2024 proposed noise scaling mechanism to mitigate exposure bias without retraining, while DS- (Li et al. 2024) only requires finding the optimal time points during reverse sampling to match the forward process as closely as possible.
Our approach aims to effectively mitigate both reverse-starting bias and exposure bias without altering the network architecture or incorporating additional components.
MOTIVATION
3.1 GRAPH DIFFUSION MODELS
Firstly, we define graph with V nodes as G = (X, A), where X E RVP represents node features, with F indicating that each node has F features; A E VVV represents the weighted adjacency matrix. Then, we formally represent the graph diffusion process as the trajectory of the random variable G over time [0, T). The forward diffusion process is given by:
dG, dd. + 9: (G.)dw, Go Pdata:
(1)
where (..) is the linear drift coefficient, ge (G.) is the diffusion coefficient, is the standard
Wiener process, and Go is a graph from the true data distribution Pdata- The stochastic differential equation (SDE), Eq. (I). describes forward diffusion process. Specifically, we replace G in Eq. (T)
with node X or edge A, represeninng the forward diffusion process of node X or edge A, respectively.
Following GDSS (Jo et al. 2022). we separate X and A in the reverse diffusion:
dx,
(X,, o.. V x, log pe (X,, A.)] dř 91,1011,
dA.- = (A,) 93,5 V A. log Pr (x,, Ar)] dē + 920
where fie and fz. satisfy f,(X, A) = (X), f2, (A)). representing the drift coefficients of the reverse process for X and A respectively. I1.e and l2.t are the corresponding scalar diffusion coeffi- cients, w and Wy are reverse-time Wiener processes in reverse diffusion, and V log (X, A.) and V A, log p [x,, A.) represent the partial scores of the node and the edge, respectively. It's worth noting that two SDEs in Eq. (2) corresponds to the diffusion process of X and A, respectively. We will choose different types of SDEs for X and A based on actual conditions. For example, for VPSDE Maariance-Presevvinn SDE) (Song et al. 2021), (X, = B, (2,) - A,
G1.t 92,t VB., Br Pmin + t(3max B.in) ). where Smax and P.in are hyperparameters.
Then, we use sp. (G.) and (G.) to estimate the partial scores Vx. logp (X,, A.) and V A, logp (X,, A.), respectively. Based on the idea of reverse denoising score matching, we derive
3

---

Published as a conference paper at ICLR 2025
pobling dieciio
aotiing
50-
50
Spore soom
30
10
1000
1000
step
(a) 80,e())
(b) (-.e
(c) FCD metric
Figure 2: (a) and (b) The E norm of the predictions from the two score networks different time steps. (c) The response of the predictions of the two score networks to perturbations at different time steps in sampling.
8 R V Por X, Xo) and 8G. 2 V VA, Por (A, Ao), and the loss is:
wn (1) E. 8 (.)) - V Pog X, X 112)
minE, 0
(3)
mink3 { (t) Ec, s0. V log Po: A A.
where wi (1) and w2(() are positive weight functions, / is uniformly sampled in the range of [0, 0]. For nodes, have Xo po(X), X, por (XxXX))), and similarly for edges, have Ao po (A), A (oc(A,)AAn)) Since fi, and f2, are affine, the transition kernels por (X, (o) and (o,(AA))) always Gaussian distributions, and closed-form means and variances are obtained based on standard techniques. For example, the node transition kernel in VPSDE (Song et al. 20211 is:
Por X X =W
-V X,le (Pmaxx (mmin) - 110mm Xo, - Ie 2 mox--mx-- Pmin) 11,3mm
-
For simplicity, the subsequent derivations focus X in VPSDE, the derivations for both X and A in VPSDE and VESDE (Variance Exploding SDE) are similar to those for X in VPSDE.
3.2 WHY BASELINE MODELS ARE TRUNCATED?
In this section, we use GDSS as the basic model and QM9 (Ramakrishnan al. 2014) as the dataset to demonstrate the reverse-starting bias and effectively validate our motivations. Specifically, we have two score networks: the first a pretrained network 8.,((.) whose forward maximum perturbation far from reaching the standard Gaussian; the second is network (- whose forward maximum perturbation distribution constrained to follow the standard Gaussian.
Figs. a) and b) show the f2-norm distribution of the predictions of two score networks at different timesteps. Taking Fig. [2] (a) as an example, we obtain perturbed samples through forward noising predict the
and calculate the corresponding f2-norm value. We present the details of Fig. 2 in Appendix B At time O, the score &2 norm the ground-truth Xo spans (0, 2500) approximately, demonstrating the diversity of the true data and its scores. As the noise intensity increases, the range of the score &g norm narrows, eventually stabilizing within (0, 10). The evolution of the score & norm of perturbed samples different time indicates that as the distribution approaches the standard Gaussian distribution, the model becomes highly sensitive to score changes. The tightened score l norm implies that the slight perturbation in score during the early sampling stage can significantly affect the generation quality. For Fig. b)) the evolution pattern of (..;()) consistent with *g,((), but since the maximum perturbation distribution of sg,(()) is far from reaching the standard Gaussian, its score & norm range is wider, indicating higher tolerance for score deviations.
each step, then use $g,(())
Fig. C(c illustrates the response of the predictions of the two score networks to perturbations at different time steps in sampling. Each point in Fig. 2 corresponds to perturbation experi- ment. The z-axis represents the addition of standard Gaussian noise perturbation the predicted score at the current time, while the y-axis represents the final generation metric resulting from that perturbation experiment (details in Appendix B). For $g. time 0, start from standard
4

---

Published as a conference paper at ICLR 2025
Gaussian and perturb the predicted score at the current step. No perturbation is applied during the subsequent sampling steps, leading to poor generation metric. As the time step of the per- turbation increases, the negative impact generation quality diminishes and eventually stabilizes after 200 steps. This evolution highlights how diffusion models are highly sensitive to accurate scores during the early sampling stages, where deviations in the score significantly degrade the generated quality, For se (-) instead of starting from standard Gaussian, we use samples from the forward maximum perturbation distribution as the initial point eliminate the reverse-starting bias.
Fig. c shows that diffusion models are highly sensitive to score deviations in high-noise states, while low-noise states, their resistance to score deviations significantly increases. Notably, this provides us with two potential approaches for ad- dressing the reverse-starting bias: 80,( (-) suggests retraining the model and constraining the forward maximum perturbation distribution to follow a standard Gaussian, while s..e(-) suggests explor- ing a reverse-starting distribution aligned with the forward maximum perturbation distribution dur- ing sampling. We observe that the latter not only mitigates the reverse-starting bias but also offers greater tolerance to subsequent deviations.
Optimall dircetion
X
Predicted score
Pxcudosaoe
Conrected score
Conrection scope
Adjusting sagle
Adjusting magnaiudee
soc(.) mx
(-2())
x.
(-p(()
4
METHODOLOGY
Figure 3: Score correction based on the score dif-
ference. At the reverse-sampling time t, the opti- mal score always points X s0,(()) points to Y: Xo with some deviation (partially containing
4.1
LANGEVIN SAMPLING
Langevin sampling is a key component of SDE- based diffusion models. Given sufficiently small step sizes and a large number of steps, Langevin sampling utilize the score function to obtain samples from a probability distribution. Impor- tantly, the prior distribution of Langevin sampling can be consistent with that of the diffusion model, typically standard Gaussian distribution. More- over, we already have a pretrained score network (.)r(-) V Xx log (X, Xo). This score guides Langevin sampling (Song & Ermon 2019) to sample from the distribution p( (x)) Rd (xx))
Xo), while sy.e(+) points to 7o with larger deviation (containing little Xo). The difference between predicted and pseudo scores guides the predicted score towards the optimal score. We use control the angle and to adjust the magnitude of the corrected score. The final corrected score flexibly approaches the optimal score within the dashed box.
X X
1/T so (o( T)+ VATET
(5)
where the subscript T represents the reverse-starting time. In the reverse-startie starring alignment stage, we only use the score (-)r(-) at time T. T/ represents the step size at time T, and ey is standard Gaussian noise. After iterating Eq. ( M times, we assume the distribution ((x)) has transitioned to stable state. We refer to this stage as the reverse-starting alignment stage.
4.2 S++ BIAS CORRECTION METHOD
Xr almost follows the distribution ((Xr(Xo) However, the pretrained score s..r(-) struggles to accurately learn the true score Vx, log (XT (o)) X where ā₁ 1 3, Vt [0,T]. We have to consider the exposure bias. Without loss of generality, consider the predicted score at any time with
In theory, after Langevin sampling based on Eq. 5
I-ar
X,
V
sx,((X,)
(6)
āi
We can rewrite Eq. (6) as
X
X, + (1 C.) (.)s0,((8 (Xi)) = XolX., 0).
(7)
V 02
5

---

Published as a conference paper at ICLR 2025
Straightforwardly, it is challenging to predict Xo from X, analytically. Following Zhang al. (002)) model the estimate of Xo by
Xo 7 Xo + òtE,
(8)
where "t. and óz are constants that reflect the similarity of Xo to Xo and 6a. respectively, with Ea ~ (0, Consequently. for all j. such that S < k S N, holds that %/ > % 2 and 0 & 6j < ōд. We then use the model S0,1 (X,, to generate a dataset X and train a pseudo score network (.),(()) using X Since Xo always has bias compared to Xo. the pseudo score 8 (.)()), trained the pseudo data Xú, naturally learns the bias. Similarly, estimate X vb) as X = X2 + 8, where is straightforward that 7/2 < 7t- Then, we define a score difference between the two scores at the same time point for the same sample:
80. (x) Supi (x) Vol (70 W Xo + (ò.E. - 8,E6)
1
We find that the score difference contains information about X, as shown in Fig. 3 We aim to utilize this information. Inspired by classifier-free guidance (Ho & Salimans 2021 and extrapolation operations (Zhang et al. (023), we define a new score adjustment by
=0 x.) e) + ( sx,((
X. aou (TH A(7a 71)) Xo + d.Ea + A(Orca K76))
(10)
where À 2 0 represents the step size for correcting the score using the score difference, Eq. (9) When O, no correction is applied. Conceptually, the correction operation pulls the biased direction towards the unbiased direction. Although there is some noise in this correction direction, choosing appropriate parameters À improves the score accuracy. Then, we divide the se, by scalar to adjust the score magnitude, further driving se,e (X,, closer the true score:
$0.0
80,8
(1I)
In particular, emphasize that the S++ score correction at time T' is far more critical than at other times. In some cases, the correction at time T is so effective that no further correction is needed in subsequent stages, especially in truncated model. Therefore, correction parameter at time T from those at other times during the actual score correction process. Specifically, by using Langevin sampling (Song & Ermon] 2019) align the distribution with the forward maximum perturbation distribution, which is in a low-noise state and retains some data information from Xo, we can shorten the sampling chain and significantly reduce sampling time. Experimental validation is provided in $5.3
recommend decoupling the
The score correction method described here can also be applied the reverse-starting alignment stage, reverse-starting starting bias is corrected, it naturally helps mitigate the exposure bias in subsequent sampling stages. Additionally, our score correction method is applicable wherever scores are utilized, ensuring its effectiveness across different stages of the reverse sampling.
as Langevin sampling also relies on scores. Once the reverse-
We emphasize that utilizing Langevin sampling to obtain aligned samples and using the difference signal correct scores are indispensable components for addressing the reverse-starting bias and the exposure bias. The effect is shown in Fig. Tc). Additionally, we conduct extensive ablation experiments in $5.4 demonstrate this point. We provide a detailed geometric illustration in Fig and provide detailed derivations and proofs of the equations from $4 in Appendix C
EXPERIMENT
In this section, we select three generic graph datasets and two molecular datasets to evaluate the performance of our approach. order to demonstrate the broad applicability of this method in addressing the reverse-starting bias and mitigating exposure bias, we tested it variety of

---

Published as a conference paper at ICLR 2025
Table 1: Generation results on the generic graph datasets (Lower is better). The results of the Enzymes dataset of GDSS are reproduced by ourselves, the results of other baselines are all from published papers, and we give detailed settings and instructions in Appendix
Dataset
Community-son
Enzymes
Grid
Info.
Synthetic, 12 [V]
Real, 10 S [V] 125
Synthetic, 100 s [V] 100
Method
Deg.l
Clus.
Orbit..
Avg-4
Deg.4
Clus.
Orbit]
Avg.
Deg.4
Clus.
Orbit.
Avg-L
GDSS-OC
0.050
0.132
0.011
0.064
0.052
0.627
0.249
0.309
0.270
0.009
0.034
0,070
DISS-OC-S++
0.021
0.061
0.005
0.029
0.067
0.099
0.007
0.058
0.105
0.004
0.061
0,066
GDSS-WC
0.045
0.088
0.007
0.045
0.044
0.069
0.002
0,038
0.111
0.005
0.070
0,070
GDSS-WC-S++
0,019
0,062
0.004
0.028
0.031
0.050
0,003
0.028
0.105
0.004
0,061
0.057
HGDM-OC
0.065
0.119
0,1124
0.069
0.125
0.625
0.371
0,374
0.181
0.019
0.112
0,104
HGDM-OC-S++
0.021
0.034
0.005
0.020
0.080
0.500
0.225
0.268
0.023
0.034
0.004
0.020
HGDM-WC
0.017
0.050
0.005
0.024
0.045
0.049
0.003
0.035
0.137
0.004
0.048
0.069
HGDM-WC-S++
0.021
0.024
0.004
0.016
0.040
0.041
0.005
0.029
0.123
0.003
0.047
0.058
GSDM-OC
0.142
0.230
0.043
0.138
0.930
0.867
0,168
0.655
1.996
0.0
1.013
1.003
GSDM-OC-S++
0.011
0.016
0.001
0.009
0.012
0.087
0.011
0.037
1.2c-4
1.2c-4
0.066
GSDM-WC
0.011
0.016
0.001
0.009
0.013
0.088
0.013
0.038
0.002
0.0
0.0
7.2c-5
GSDM- -WC+
0.011
0.016
0.001
0.009
0.011
0.086
0.010
0.036
5.0e-5
0.0
1.1e-5
0.066
mainstream graph diffusion models, namely GDSS (Jo et al. 2022). GSDM (Luo et al.) 2024).
HGDM (Wen al., (024), and MOOD (Lee et al., (023). Our improved model is prefixed with the basic diffusion model and denoted by S++ at its suffix. At the same time, we perform extensive downstream task testing and ablation study to further illustrate the effectiveness and necessity of S++.
5.1 GENERIC GRAPH GENERATION
Experimental Setup We select three generic graph datasets to test our approach: (1) Community-
small: 100 artificially generated graphs with community structure: (2) Enzymes: 600 protein maps representing the enzyme structure in BRENDA (Schomburg et al. 2004); (3) Grid: 100 standard 2D
grid diagrams. To evaluate the quality of the generated graphs, we tollow GDSS (Jo et al. (022) and we use Maximum Mean Difference (MMD) (Gretton et al. 2012 You et al. 2018) to compare the statistical distribution of the graphs between the same number of generated plots and the test plots, including the distribution of measured degrees, clustering coefficients, and the number of orbits of the node track.
Results Table shooss that S++ outperforms all of baselines. For the uncorrected sampling method, the performance indicators of the baseline model are particularly poor due the existence of the reverse-starting bias and score exposure bias, while S++ can significantly improve the performance of all baseline models and reach or even exceed the level of the baseline model with correctors. Because the method without correctors can significantly reduce computational complexity, we believe that S++ can really release the ability of the graph diffusion model, which is enlightening for large-scale datasets. For the sampling method with aligners, S++ is still significantly better than all baseline models. At the same time, we also give experimental comparisons of other advanced models in Appendix F and results show that S++ achieves SOTA indicators of the corresponding tasks.
5.2 MOLECULAR GRAPH GENERATION
Experimental Setup We select two widely recognized molecular datasets to evaluate our approach:
QM9 (Ramakrishnan et al.) 2014) and ZINC250k (Irwin et al. 2012). We generate 10,000 molecules and select the following metrics: Frechet ChemNet Distance (FCD) (Preuer et al. 2018). Neigh- borhood subgraph pairwise distance kernel (NSPDK) MMD (Costa Grave 20TO). validity w/o correction, and the generation time. (1) FCD uses the activation of the penultimate layer of ChemNet to calculate the distance between the benchmark molecular dataset and the generated dataset to characterize the similarity between them, and the lower the FCD, the higher the similarity between distributions. (2) NSPDK MMD considers the characteristics of nodes and edges at the same time, and calculates the MMD between the benchmark molecular dataset and the generated dataset; (3) Sampling time is used to evaluate the speed in generating large-scale molecular datasets, and we only count the time spent on sampling, regardless of the time spent on preprocessing and evaluation.
7

---

Published as a conference paper at ICLR 2025
Table Comparison of different methods on QM9 and ZINC250K datasets.
QM9
7INC250K
Method
Sampling
NSPDK MMD | FCD 4
Sampling time
NSPDK MMD
FCD 4
GDSS-OC
0.73c2
0.016
4.584
0.73c3
0.047
20.53
GDSS-OC-S+
5.10
0.001
1.661
0.70e3
0,050
16.79
GDSS-WC
1.61e2
0.004
2,550
1.41e3
0,019
14.66
GDSS WC---C--++
9.25
0.001
1.661
0.98e3
0.012
12.70
HGDM-OC
0,62e2
0.005
3.164
0.76e3
0.033
21.38
HGDM-OC-S++
0.62e2
0.003
2.512
0.77e3
0.034
20,79
HGDM-WC
1.16e2
0.002
2.147
1.52c3
0,016
17.69
HGDM-WC-S++
0.98e2
0.001
2.001
1.17c3
0.016
16.24
Results Table Tholls that both in terms of sampling time and generation quality, S++ is significantly better than the baseline models. For the sampling method without correctors, due the existence of the eeverse-ttrring bias and score exposure bias, the quality of generation from the baseline model is particularly poor, while S++ can significantly improve the performance of all baselines and approximate the sampling methods with correctors of the baseline model. For the sampling method with aligners, S++ is still significantly better than all baseline models and greatly reduces the sampling time. At the same time, we provide more comparative experimental results in Appendix and provide parameter sensitivity experiments in Appendix
5.3 DIVERSITY GENERATION
Characteristic molecule generation To evaluate the performance of S++ in generating novel, drug-like, and synthesizable molecules, we follow (Lee et al. (023)) and assess S++ in the five docking score (DS) optimization tasks under the quantitative estimate of synthetic accessibility (SA), drug-likeness (QED) and novelty constraints. The property is defined by
(G) = DS QED(G) x SA(G) E [0,
(12)
where DS refers to the normalized docking score, SA denotes the normalized synthetic accessibility, and QED represents drug-likeness. We use MOOD- S++ to generate 3000 molecules and evaluate performance using the following metrics. Novel hit ratio (%) is the fraction of unique hit molecules whose maximum Tanimoto similarity with the training molecules is less than 0.4. In particular, hit molecules are defined as the molecules that satisfy the following conditions: DS < (the median DS Novel top 5% docking score refers the
of the known active molecules), QED > 0.5, and SA average DS of the top 5% unique molecules that satisfy the constraints QED > 0.5 and SA < 5 and their maximum similarity with the training molecules is below 0.4. To avoid bias target selection, we utilize five protein targets: parpl, fa7, 5htlb, braf, and jak2.
Results Tables 3 and 4 show that MOOD-S++ is significantly better than baseline in all target proteins. This indicates that S++ still has advantages in the discovery of drug-like, synthesizable, and novel molecular tasks with high binding affinity, and can be seen that the reverse-startng bias and exposure bias pose significant threat various generation tasks.
Accelerate generation demonstrate that S++ can generate good samples faster by using fewer steps of reverse diffusion, we choose GDSS-OC the benchmark, and QM9 and Comm are selected to test the performance of our approach and benchmark models at different sampling total time steps.
Results Table 5 shows that S++ is significantly better than the baseline model at different sampling total time steps N. S++ is not only able to generate samples with fewer reverse-diffusion steps but also achieves consistent improvements across generation metrics, especially the QM9 dataset, where S++ remained close to optimal performance even with a significant reduction in the sampling time step (N 100), while the performance of the benchmark model decreased significantly.
In conclusion, S++ shows higher efficiency, better quality, and stronger robustness in graph generative tasks, which provides powerful improvement scheme for the application of the diffusion model.
8

---

Published as a conference paper at ICLR 2025
Table Novel hit ratio (%) results (1).
Target protein
Method
parp1
fa7
5hilb
braf
jak2
MOOD
7.017 (# 0.428)
0.733 (# 0.141)
18.673 (# 0.423)
5.240 (# 0.285)
9.200 (#: 0.524)
MOOD-S+
8.286 (# 0.214)
0.900 (# 0.068)
20.354 (# 0.672)
5.653 (# 0.073)
9.167 (# 0.067)
Table 4: Novel top 5% docking score (kcal/mol) results (4).
Target protein
Method
parp1
fa7
Shilb
braf
jak2
MOOD
10..65 (+ 0.113)
-8. 160 (+ 0.071)
-11.145 (# 0.042)
-11.063 (+ 0.034)
-00.147 (# 0.060)
MOOD-S++
10.961 (# 0.027)
88.82 (# 0.028)
-11.231 (# 0.036)
-11.143 (# 0.025)
-00.163 (# 0.015)
Table Comparison of different methods under different total sampling time steps.
QM9
Community-sman
N
Method
Val. w/o corr. t
NSPDK MMD
FCD
Deg.l
Clus. +
Orbit 4
Avg. +
GDSS-OC
73.5
0.015
4.584
0.050
0.132
0.011
0.064
1000
GDSS-OCS++
94.0
0.001
1.671
0.021
0.061
0.005
0.029
GDSS-OC
46.2
0.045
7.960
0.136
0.456
0.151
0.248
500
GDSS-OC-S++ OC-S++
93.9
0.001
1.665
0.029
0.142
0.008
0.060
GDSS OC
37.8
0.069
9.951
0.092
0.666
0,394
0.384
100
GDSS-OC-S++
93.9
0.001
1.663
0.061
0.414
0.140
0.205
Table 6: Ablation experiments on the QM9 dataset.
QM9
Method
Val. w/o corr.
NSPDK MMD
FCD L
GDSS-OC
73.5
0.0157
4.58
GDSS-w/o correction in sampling
94.8
0.0037
2.65
GDSS-w/o eeerreestarting alignment
89.8
0.0031
2.01
GDSS-OC-S++
94.0
0.0014
1.67
5.4 ABLATION STUDY
Table 6 demonstrates that GDSS without reverse-starting alignment and score correction indepen- dently improves performance to varying degrees. Among these, the case without reverse-starting alignment is more general, it does not require 7/r and T. Even in their absence, our approach still achieves impressive results, highlighting its robustness and ficxibility. Additionally, we provide a comparative analysis of the two biases in the image and graph domains in Appendix aHd present comparative experiments of S++ with existing methods on images in Appendix
CONCLUSION
In this paper, we use Langevin sampling to obtain samples aligned with the forward maximum perturbation distribution, which mitigates the reverse-starting bias and greatly mitigates the exposure bias of the score network, and we propose score correction mechanism based on score difference to further promote the stable-state distribution Langevin sampling to the true forward maximum perturbation distribution, and further mitigate the exposure bias in sampling. Since the score is also used Langevin sampling, the proposed score correction is also applied to Langevin sampling as well. Our approach does not require network modifications the introduction of new learners and can be naturally integrated into existing graph diffusion models to achieve SOTA metrics on multiple datasets and multiple tasks.

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work was supported by the National Natural Science Foundation of China under Grant No. 62176108.
REFERENCES
Fabrizio Costa and Kurt De Grave. Fast neighborhood subgraph pairwise distance kernel. In ICML, pp. 255-262. 2010.
Martin Nicolas Everaert, Athanasios Fitsios, Marco Bocchio, Sami Arpa, Sabine Süsstrunk, and Radhakrishna Achanta. Exploiting the signal-leak bias diffusion models. In WACV, pp. 4025- 4034, 2024.
Arthur Gretton, Karsten M. Borgwardt, Malte J. Rasch, Bernhard Schölkopf, and Alexander Smola. A kernel two-sample test. Journal of Machine Learning Research, 13(25):723-773 2012.
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In NeurlPS 202/ Workshop Deep Generative Models and Downstream Applications, 2021.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In NeurlPS, pp. 68500 2020.
Han Huang. Leilei Sun, Bowen Du, and Weifeng Lv. Conditional diffusion based on discrete graph structures for molecular graph generation. In AAAI, volume 23, pp. 4302-4311, 2023.
John Irwin, Teague Sterling, Michael M Mysinger, Erin S Bolstad, and Ryan Coleman. ZINC: free tool to discover chemistry for biology. Journal of Chemical Information and Modeling, pp. 1757-1768, 2012.
Jaehyeong Jo, Seul Lee, and Sung Ju Hwang. Score-based generative modeling of graphs via the system of stochastic differential equations. In ICML, pp. 10362-10383, 2022.
Dongjun Kim, Yeongmin Kim, Se Jung Kwon, Wanmo Kang, and II-Chul Moon. Refining generative process with discriminator guidance in score-based diffusion models. In ICML, pp. 16567-16598, 2023.
Seul Lee, Jaehyeong Jo, and Sung Ju Hwang. Exploring chemical space with score-based out-of- distribution generation. In ICML, pp. 18872-18892, 2023.
Mingxiao Li, Tingyu Qu, Ruicong Yao, Wei Sun, and Marie-Francina Moens. Alleviating exposure bias in diffusion models through sampling with shifted time steps. In ICLR, 2024.
Yangming Li and Mihaela van der Schaar. On emor propagation of diffusion models. In ICLR, 2023.
Yujia Li, Oriol Vinyals, Chris Dyer, Razvan Pascanu, and Peter Battaglia. Learning deep generative models of graphs. arXiv preprini arXiv: 03324, 2018.
Shanchuan Lin, Bingchen Liu, Jiashi Li, and Xiao Yang. Common diffusion noise schedules and sample steps are flawed. In WACV, 2024. 2024.
Meng Liu, Keqiang Yan, Bora Oztekin, and Shuiwang Ji. GraphEBM: Molecular graph generation with energy-based models. Energy Based Models Workshop-ICLR 2021, 2021.
Tianze Luo, Zhanfeng Mo. and Sinno Jialin Pan. Fast graph generation via spectral diffusion. IEEE Transactions Pattern Analysis and Machine Intelligence, 4645)33496,3588 2024.
Mang Ning, Enver Sangineto, Angelo Porrello. Simone Calderara, and Rita Cucchiara. Input perturbation reduces exposure bias in diffusion models. In ICMI, volume 40, pp. 26245-26265. 2023.
Mang Ning, Mingxiao Li, Jianlin Su, Albert Ali Salah, and Itr Onal Ertugrul. Elucidating the exposure bias in diffusion models. In ICLR, 2024.
10

---

Published as a conference paper at ICLR 2025
Chenhao Niu, Yang Song, Jiaming Song, Shengjia Zhao, Aditya Grover, and Stefano Ermon. Permu- tation invariant graph generation via score-based generative modeling. In A/STATS, pp. 4474 4484, 2020.
Kristina Preuer, Philipp Renz, Thomas Unterthiner, Sepp Hochreiter, and Gunter Klambauer. Fréchet chemnet distance: metric for generative models for molecules in drug discovery. Journal of Chemical Information and Modeling, pp. 1736-1741, 2018.
Raghunathan Ramakrishnan, Pavlo O Dral, Matthias Rupp. and Anatole Von Lilienfeld. Quantum chemistry structures and properties of 134 kilo molecules. Scientific Data, 2014.
Zhiyao Ren, Yibing Zhan, Liang Ding, Gaoang Wang, Chaoyue Wang, Zhongyi Fan, and Dacheng Tao. Multi-step denoising scheduled sampling: Towards alleviating exposure bias for diffusion models. In AAAI, pp. 4667-467, 2024.
Ida Schomburg, Antje Chang, Christian Ebeling, Marion Gremse, Christian Heldt, Gregor Huhn, and Dietmar Schomburg. BRENDA, the enzyme database: updates and major new developments. 2004.
Nucleic Acids Research,
Chence Shi, Minkai Xu, Zhaocheng Zhu, Weinan Zhang, Ming Zhang, and Jian Tang. GraphAF: flow-based autoregressive model for molecular graph generation. In ICLR, 2020.
Martin Simonovsky and Nikos Komodakis. Graphvae: Towards generation of small graphs using variational autoencoders. In ICANN, pp. 412-422, 2018.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In ICML, pp. 2256-2265. 2015.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. In NeurIPS, volume 32, pp. 11918-11930, 2019.
Yang Song, Jascha Sohl-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In ICLR, 2021.
Lingfeng Wen, Xuan Tang, Mingjic Ouyang, Xiangxiang Shen, Jian Yang. Daxin Zhu, Mingsong Chen, and Xian Wei. Hyperbolic graph diffusion model. In AAAL, pp. 15823-15831, 2024.
Minkai Xu, Lantao Yu, Yang Song, Chence Shi, Stefano Ermon, and Jian Tang. GeoDiff: a geometric diffusion model for molecular conformation generation. In ICLR, 2022.
Jiaxuan You, Rex Ying, Xiang Ren, William Hamilton, and Jure Leskovee. GraphRNN: Generating realistic graphs with deep auto-regressive models. In ICML, volume 35, pp. 3708-5712 2018.
Guoqiang Zhang, Kenta Niwa, and W Bastiaan Kleijn. Lookahead diffusion probabilistic models for refining mean estimation. In CVPR, pp. 1421-1429, 2023.
Huangjie Zheng, Pengeheng He, Weizhu Chen, and Mingyuan Zhou. Truncated diffusion probabilistic models and diffusion based adversarial auto-encoders. In ICLR, 2023.

---

Published as a conference paper at ICLR 2025
A DEVERSE-STARTING BIAS
In this section, we provide a detailed discussion of the reverse-starting bias. It is worth noting that these models are based on SDE (Song et al. 2021). For VPSDE, obtains perturbed samples through Eq. (4) which corresponds to Eq. (33) in (Song et al.
2021 At = T. Eq. (4) gives the maximum perturbation distribution. which is N (X I). Similarly, for VESDE, the diffusion
model obtains perturbed samples through Por (X, = N (Xx (X. of1), where miin = 01 S
at S OT = 0max- When this expression is extended continuously, we have
Puc Xx) N X X o
(xxxx
mmax
2
I)
.t E [0.1]
(13)
min
Eq. (13) corresponds to Eq. (31) of (Song et al. 2021 At = T Eq. 13 achieves the maximum perturbation distribution, which is N (Xx Xn, , 0 maxx) Ij. In particular, we need to make sure that max is large enough that ((x Xo, o max I) R ((x 0 mAx I).
However, in practice, lots of diffusion models (Jo et al. 2022 Luo al. 2024
Wen et al. 2024
adopted a rather conservative strategy when training the network. For VPSDE, results in the maximum forward perturbation distribution being A ((Xrluua (lur20, o I). which is far from reaching N (Xr 0. I). For VESDE, due to max not being large enough, the maximum forward perturbation distribution ((T Xx I), which cannot be approximated by W(Xr|0, max However, baselines always start reverse sampling from the standard Gaussian distribution, which leads to significant reverse-starting bias. A detailed comparison of parameters shown in Tables
X. and
B FIGURE DETAILS
In this section, we present the detailed procedures plot Fig. 2
Let sg,(()) represent the pretrained
GDSS score network. Due to the conservative strategy of GDSS, with 8min 0.1 and 8max 1,
Table 7: The actual parameters of the forward perturbation of GDSS.
Model
GDSS
Dataset
Community-small
Enzymes
Grid
QM9
ZINC250K
Type
Node
Edge
Node
Edge
Node
Edge
Node
Edge
Node
Edge
SDE
VPSDE
VPSDE
VPSDE
VESDE
VPSDE
VPSDE
VESDE
VESDE
VPSDE
VESDE
Sain miin
0.1
0.1
0.1
0.1
0.2
0.1
0.1
0.2
Pmax (Faax
1.0
1.0
1.0
1.0
1.0
0.8
1.0
1.0
1.0
1.0
uy
0.7596
0.7596
0.7596
1.0
0.7596
0.7788
1.0
1.0
0.7596
1.0
0.4231
0.4231
0.4231
1.0
0.4231
0.3935
1.0
1.0
0.4231
1.0
Table 8: The actual parameters of the forward perturbation of HGDM.
Model
HGDM
Dataset
Community-small
Enzymes
Grid
QM9
ZINC250k
Type
Node
Edge
Node
Edge
Node
Edge
Node
Edge
Node
Edge
SDB
VPSDE
VPSDE
VPSDE
VESDE
VPSDE
VESDE
VPSDE
VESDE
VPSDE
VESDE
Amin Omin
0.1
0.1
0.1
0.2
0.1
0.2
0.1
0.1
0.1
0.2
Bmax [Tmox
1.0
1.0
1.0
1.0
7.0
0.8
2.0
1.0
1.0
1.0
WT
0.7596
0.7596
0.7596
1.0
0.1695
1.0
0.5916
1.0
0.7596
1.0
o7
0.4231
0.4231
0.4231
1.0
0.9713
0.64
0.6501
1.0
0.4231
1.0
Table 9: The actual parameters of the forward perturbation of GSDM and MOOD.
Model
GSDM
MOOD
Dataset
Community-small
Enzymes
Grid
QM9
ZINC250k
Type
Node
Eigen
Node
Eigen
Node
Eigen
Node
Edge
Node
Edge
SDE
VPSDE
VPSDE
VPSDE
VPSDE
VPSDE
VPSDE
VPSDE
VESDE
VPSDE
VESDE
Bmin O'iin
0.1
0.1
0.1
0.1
0.1
0.2
0.1
0.1
0.2
Paaxx (Fanx
1.0
1.0
1.0
1.0
1.0
0.8
1.0
1.0
1.0
1.0
Mr
0.7596
0.7596
0.7596
0.7596
0.7596
0.7788
1.0
1.0
0.7596
1.0
0.4231
0.4231
0.4231
0.4231
0.4231
0.3935
1.0
1.0
0.4231
1.0
12

---

Published as a conference paper at ICLR 2025
PAIE
Villdrty weto coemertioon
06
J0HDX MMD
samplit draction
0.04
deg-dracionn
0.23
Kaiplig dieciet
0.02
0.0
Tincsoop
(a) FCD
(b) NSPDK MMD
(c) Validity w/o correction
Figure 4: Generation metric responses to perturbations different time steps for two score networks.
the maximum perturbation distribution is / ( (Xx](0.759XX, 0.4231I) at / T. On the other hand, (.)(() is defined with the constraints of Amin 0.1 and Smax 20. At t T, the maximum perturbation distribution is (xxr 00,1) Then, (.,+()) (.) and (.,+(-) not only represent two different networks but also indicate that their maximum perturbation distributions in the forward diffusing completely different.
To plot Figs. 2(a) and 2(b), freeze the converged s.,(()) and s then replace X, in Eq. with A, to obtain perturbation samples of 1024 edges at different time steps. We then compute [| and [|)xp,()))]2 and plot them in Fig.
To plot Fig. Cc), we introduce perturbations to (.)+(-) at different time stepsin sampling. We employ sampling method without a corrector and perturb the score at the selected time step (y-axis) using Gaussian noise:
(. s0,e(-) Zt
(14)
where (z, 1). For the other time steps, we do not introduce any perturbations, allowing the diffusion model to perform sampling and record the generation metrics. conduct the perturbation experiment (-) using the same method, and ultimately compare the results of the two perturba- tion experiments based on the time steps to evaluate how different score networks the diffusion model resist bias various time steps. We present detailed comparison of the generation metrics from the perturbation experiments, as shown in Fig.
C DERIVATIONS FOR $4.2
For a diffusion model, let the true data Xo, and the pretrained score network be s.,((-). Based on the set noise addition method, have
X, Xo
Vx, log g( X.)
(15)
1 ō2
However, se,((-) often deviates from the optimal logarithmic gradient of (X) In the reverse- sampling process, assuming the current time has data state X,, the predicted score
$0 sx,( = X. Va, Xo
(6)
1- āi
We model Xo by
Xo No + 8.Ea
(8)
Eq. (6)
becomes
X, vorl Xo + d,Ea)
80. X1)
(16)
de
Then, train new score network 8-,(-) based on the generated data X' from the score network. Following the above derivation, we can write the predicted score at the current time t as
X,
VorXi
S (Xi)
(17)
13

---

Published as a conference paper at ICLR 2025
Due to the bias of $0 (-). the generated pseudo data X always deviates from Xu, and considering the prediction error the network, we can easily obtain
X - 7 6,Eb
(18)
where 7/2 Tt. 8/ > 6,, meaning that the same time I, (.,)))) trained on generated data has larger bias in predicting the target distribution than (0,( trained on the true data. By substituting Eq. (18) into Eq. (77) we have
/0 + 8,66)
S X.)
(19)
The score difference defined as the difference between Eq. (16) and Eq. 19
X, vāi(ūxo + d(ca)
X;
(1
X +oo+o
8766)
(X,) (x,)
04z
02
(++ X0 + (d,E. 81E6)
We incorporate this score difference as correction term into the predicted score and introduce hyperparameter to fine-tune the adjustment, improving the alignment of the predicted score with the true score. It is given by
s0. = s0 (X, ((50, (X,) s4, (x,))
X. V 7: Xo + die, + +A71 No + A(dre, - 8/E6))
1 5
X, Vas ( M(x 11) Xo d,E, + A(OsE, - -(E,)
(10)
ā
Since 7 < Tt, this score difference helps the predicted score incorporate more information from the true data Xg. By appropriately setting a hyperparameter X, we can consistently use the information from Xo to guide the score correction process. Finally, introduce adjustment magnitude coefficient to further refine the score correction based on the score difference, which improves the alignment of the predicted score with the true score.
$0.0 (X,)
80.0 (X.)
(11)
We provide logical analysis showing that the score difference helps to correct the score.
D DETAILS FOR EXPERIMENT
We provide detailed parameters for experiments described in $ shown in Tables TO and Specifically, we distinguish between the relevant parameters for sampling methods with correctors and those without correctors. As shown Tables [0] both Langevin sampling and the S++ correction are implemented in the reverse-starting alignment stage. After this stage, the subsequent sampling stages require little to no further correction. This suggests that in a truncated diffusion model, once correction is applied at a certain point, the later stages can achieve good sampling performance without the need for additional corrections.
E SAMPLING ALGORITHM
In this section, we present the sampling algorithm procedure for S++, as shown Algorithm 3r 8min + t(3max Bmin) for E [0, 1]. Additionally, our approach can be naturally integrated into the reverse sampling of various diffusion models without a corrector, greatly improving the generation quality of sampling methods. For methods with corrector, the correction time interval by introducing a truncation time. Specifically, we apply the corrector when the time exceeds further reducing the computational cost.
significantly reduce
14

---

Published as a conference paper at ICLR 2025
Algorithm 1 The S++ sampling algorithm.
Input: An inference model se,(()), pretrained pseudo model sw,(()), and the cut-off time tc. Initialize: XT ~ WN( I) where 7 N
for j = 1 to M do
s0.r (e.r((x))+t (Xx)) ma. (.
80.7 (x)) t-
6 (0, I);
XT XT
nr S0.T (XT) VITE:
end for
for N 0 do
== N
$0.1 (X,)
42
OO.
X, + (2 VI 3r X1 (X,) + VBEE;
if S l then
if corrector then
~~ N0,
X, f- Xx 2 ( TiE;
end if end if end for return X
Table Experimental parameters for sampling methods without a corrector (OC).
Model
Hyper.
Comm
Enzymes
Grid
QM9
ZINC250K
M
400
420
350
X
0.2
0.0008
0.06
1.19
GDSS-OC-S++
W1
0.998
1.0
1.0
1.09
1.0
22
0
W2
1.0
1.0
1.0
1.0
1.0
M
280
310
280
240
220
0.02
0.0
0.02
0.1
0.025
HGDM-OC-S++
1.0
1.0
1.0
1.0
1.07
0.36
0.0
0.0
0.36
0,0
1.0
1.0
1.0
0.78
1.0
M
200
400
400
0.0
0.0
GSDM-OC-S++
W1
1.0
1.0
1.0
0.0
0.0
0.0
W2
1.0
1.0
1.0
F ADDITIONAL EXPERIMENTS
To demonstrate the superiority of S++, we select generative models other than diffusion models as baseline models for comparison. GraphVAE (Simonovsky & Komodakis 2018) is a graph generative model based on variational autoencoders; DecpGMG (Li et 2018) is a deep generative model that generates graphs in sequential, node-by-node manner; GraphAF (hhi al.) 2020) is an autoregressive flow-based model. GraphRNN (You et al. 2018) is an autoregressive model using recurrent neural networks to generate graphs; EDP-GNN (Niu et al. 2020) a score-based generative model using energy-based dynamics. GraphEBM (Liu al. 2021) is energy-based generative model that generates molecules by minimizing energy through Langevin dynamics, which categorized
one-shot generative method. We provide detailed comparative experiments in Tables [2 and [3] and the results show that our approach significantly outperforms the baseline models and other generative models.
15

---

Published as a conference paper at ICLR 2025
Table 11: Experimental parameters for sampling methods with corrector (WC).
Model
Hyper.
Comm.
Enzymes
Grid
QM9
ZINC250k
M
400
420
350
400
400
A1
0.2
0.0008
0.06
1.19
2.5
GDSS-WC-S++
W1
0.998
1.0
1.0
1.09
1.0
tc
0.2
0.45
0.055
0.1
0.4
X2
0
0
0
0
0
W2
1.0
1.0
1.0
1.0
1.0
M
280
200
360
240
220
X1
0.02
0.0
0.18
0.1
0,25
HGDM-WC-S++
W1
1.0
1.0
1.0
1.0
1.07
t
0.2
0.5
0.1
0.65
0.6
A2
0,36
0.0
0.0
0.0
0.0
W2
1.0
1.0
1.0
1.44
0.87
M
200
400
400
X1
0.0
0.0
0.0
GSDM-WC-S++
W1
1.0
1.0
1.0
fo 0.05
0.70
0.45
22
0.0
0.0
0.0
W2
1.0
1.0
1.0
Table 12: Additional experiments on generic graph datasets.
Datasel
Community
Enzymes
Grid
Info.
Synthetic, 12 S VV < 20
Real, 10 S [V|
S 125
Synthetic. 100 s [V] s 400
Method
Deg.
Clus.
Orbit
Avg.
Deg.
Clus.
Orbit
Avg.
Deg.
Clus.
Orbit
Avg.
DecpGMG
0.220
0.950
0.400
0.523
GraphRNN
0.080
0.120
0.040
0.080
0.017
0.062
0.046
0.042
0.064
0.043
0.021
0.043
GraphAF
0.18
0.20
0.133
1.669
1.283
0,266
1.073
GraphDF
0.06
0.12
0.070
1.503
0.202
0,922
Gre
0.350
0,980
0.540
0.623
1.369
0.629
0.191
0.730
1.619
0.0
0.919
0.846
EDP-GNN
0.053
0.144
0.026
0.074
0.023
0.268
0.082
0.124
0.455
0.238
0.328
0.340
GDSS-OC
0.050
0.132
0.011
0.064
0.052
0.627
0.249
0.309
0.270
0.009
0.034
0.070
GDSS-OC-S++
0.021
0.061
0.005
0.029
0.067
0.099
0.007
0.058
0.105
0.004
0.061
0.066
GDSS-WC
0.045
0.088
0.007
0.045
0.044
0.069
0.002
0.038
0.111
0.005
0.070
0.070
CDSS-WC-S++
0.019
0.062
0.004
0.028
0.031
0.050
0.003
0.028
0.105
0.004
0.061
0.057
HGDM-OC
0.065
0.119
0.024
0.069
0.125
0.625
0.371
0.374
0.181
0.019
0.112
HGDM-OC-S++
0.021
0.034
0.005
0.020
0.080
0.500
0.225
0.268
0.023
0.034
0.004
0.020
HGDM- WC
0.017
0.050
0.005
0.024
0.045
0.049
0.003
0.035
0.137
0.004
0.048
0.069
HGDM- WC-S++
0.021
0.024
0.004
0.016
0.040
0.041
0.005
0.029
0.123
0.003
0.047
0.058
GSDM-OC
0.142
0.230
0.043
0.138
0.930
0.867
0.168
0.655
1. 1,996
0.0
1.013
1.003
GSDM-OC-S++
0.011
0,016
0.009
0.012
0,011
0.037
1.2e-4
0.0
1.2e-4
GSDM-WC
0.011
0.016
0.001
0.009
0.013
0.088
0.013
0.038
0.002
7.2c-5
GSDM-WC-S++
0.011
0.016
0.001
0.009
0.011
0.086
0.010
0.036
5.0e-5
0.0
1.1e-5
0.066
Table 13: Additional experiments QM9 and ZINC250K datasets.
QM9
ZINC250K
Method
Val. w/o curr. (%))
NSPDK MMD |
FCD [
Val. w/u corr. (%0
NSPDK MMD
FCD
GraphAF
67.00
0.020
5,268
68.00
0.044
16.289
GraphDF
82.67
10.816
89.03
0.176
34,202
MoFlow
91.36
0.017
4.467
63.11
0.046
20.931
EDP-GNN
47.52
0.005
2,680
82.97
0.049
16.737
GraphEBM
8.22
0.030
6.143
5.29
0.212
35.471
GDSS-OC
73.49
0.015
4.584
41.84
0.047
20.53
GDSS-OC-SA
93.74
0.001
1.661
59.50
0.050
16.79
GD
94.91
0.004
2,550
95.83
0.019
14.66
GDSS-WC-SH
93.79
0.001
1.661
93.15
0.012
12.70
HGDM-OC
92.22
0.005
3.164
66.47
0.033
21.38
CGDM-OC-S++
94,95
0.003
2,512
67.12
20.79
HGDM-WC
98.02
0.002
2.147
93.26
0.016
17.69
HGDM-WC-S++
97.03
0.001
2,001
91.03
0.016
16.24
16

---

Published as a conference paper at ICLR 2025
Table 14: FCD(L) on GDSS-OC baseline and QM9 under different parameters À (A 0 repre- sents the baseline).
Table 15: Degree(J) on GDSS-OC baseline and Community under different parameters À 0 represents the baseline).
0
1.17
1.18
1.19
1.20
1.21
0.18 0.19 0.20
0.21 0.22
FCD 4.583
1.768
1.756
1.754 1.761
1.762
Degree 0.05 0.023 0.024 0.022 0.024 0.026
G INSENSITIVITY OF X
We emphasize that S++ is largely insensitive to the choice of X. as performance improvements are consistently observed across wide range of values, as shown in Tables [4 and 15
H TWO BIASES IN IMAGE AND GRAPH
Given the existing research on reverse-starting bias and exposure bias in the image domain, this section highlights the key differences between these biases in image and graph data, focusing on three aspects: data scale, data structure, and network performance.
1-a Reverse-starting bias images. DPM-Fixes (Lin et al. 2024) was the first to identify that traditional noise scheduling strategies fail to ensure the maximum forward perturbation distribution aligns with a standard Gaussian distribution:
TT 0.068265xz + 0.997667ET
(20)
This shows a slight deviation from the standard Gaussian starting point used in reverse sampling. To address this, DPM-Fixes constrains the forward ør follow the standard Gaussian by adjusting the noise scheduling scale. This approach benefits from large image datasets and robust network performance, which enable accurate noise (or score) predictions even in high-noise states. On the other hand, DPM-Leak (Everaert et al.) 2024) estimates the maximum forward perturbation distribution during diffusion based on pixel modeling, using it as the new reverse-starting point for sampling. This strategy capitalizes on the image data structure and scale, which allows pixels to be modeled independently and follow a Gaussian distribution.
1-b Reverse-starting bias in graphs. Unlike image data, where the models can rely on more manageable data scales and network capacities, graph models are hindered by the challenges of accurately predicting noise (or score) from high-noise states. As result, baseline models tend adopt a conservative strategy during training, leading to the maximum forward perturbation distribution significantly deviating from standard Gaussian. While this approach avoids the instability of high-noise states, it introduces considerable reverse-starting bias. Consequently, strategies like DPM-Fixes, which enforce dr follow standard Gaussian distribution, are not suitable. Given the interdependencies of nodes and edges in graph data and its inherent sparsity, we cannot assume that nodes or edges follow Gaussian distribution estimate the maximum forward perturbation. This makes approaches like DPM-Leak equally inappropriate.
2-a Exposure bias in images. In images, exposure bias refers the mismatch between forward process x, and reverse process &,, with differences accumulating throughout sampling, ultimately affecting generation quality. Many current image exposure bias works assume no reverse-starting bias exists, focusing on sampling process bias, as reverse-starting bias in images is indeed quite minimal.
2-b Exposure bias graphs. Unlike the minor signal leakage seen in image diffusion models, graph diffusion models suffer from significant reverse-starting bias, leading to substantial exposure bias after the initial sampling step. In other words, the exposure bias in graphs not solely a result of network prediction errors the accumulation of sampling iterations, but is heavily influenced by the reverse-starting bias. Therefore, effectively addressing graph exposure bias requires first mitigating the reverse-starting bias.
In conclusion, we emphasize that evversesttartin bias is a particularly acute and unique issue in graph diffusion models. While exposure bias also affects graphs, solutions developed for image-based based models cannot be directly applied. This work is the first to focus specifically on reverse-starting bias in graph diffusion models, proposing simple yet effective solution.
17

---

Published as a conference paper at ICLR 2025
Table 16: FCD(L) on QM9 without corrector.
Table 17: FCD(U) on QM9 with a corrector.
1.1
0.9
1.0
1.1
1.2
1.3
GDSS-OC-ES 3.57 3,417 3.768
4.584
5.517
GDSS-WC-ES 2.809 2.552
2.319
2.301
2.542
GDSS-WC-S++ 2.94 2.814 3.198 4.187 5.201
GDSS-WC-S++ 2.321 2.034 1.858
1.852
2.152
S++ AND EXISTING SOLUTIONS IN IMAGES
Although there are many solutions mitigate the exposure bias in current images, these solutions cannot replace S++. In this section, we choose to compare S++ in detail with similar works (Li et al., 2024
Ning et al. 2024), as they are plug-and-play solutions that do not introduce new components.
TS-DPM (Li et al. 2024) proposes searching optimal time during sampling. TS-DPM relies on two fundamental assumptions: (a) image pixels are independent and follow Gaussian distribution,
Eq. (13) in Appendix J of ct al. (2024); (b) sample pixel variance approximates population variance, Eq. (20) in Appendix oi et al. (2024). These assumptions are based on large image datasets and large number of pixels. However, nodes and edges in graphs highly sparse. Specifically, many graph datasets are small (Community-smn -mall, Enzymes, and Grid have fewer than 1000 samples). Both assumptions do not hold for graph data, making TS-DPM inapplicable to graph diffusion models.
ADM-ES(Ning et al. 2024) proposes reducing sa,{()). originally noise cg,{(-). during sampling to mitigate exposure bias However, this approach does involve the angle of s0,()))- Our approach addresses this limitation:
80. (x)) (X,, + (Xso, (X,)
(21)
w
If 0, it equivalent to ADM-ES. other words, ADM-ES a special case of S++, while the score difference (x, S (X:) provides the direction information.
For fair comparison with ADM-ES, we introduce A(s0, (X,) - sw.e( for each magnitude factor with Ả uniformly set to 0.5, to examine whether sg. (X,) s brings improvements over
ADM-ES. Tables 16 and 17 demonstrate that the angle information in S++ leads to significant gains.
18


Reviews:
[{"review_id": "s08q4fB2Ml", "reviewer": "Reviewer_wgRC", "summary": "In view that the existing graph diffusion models can not reach the standard Gaussian distribution if following their defined transition distribution, this paper proposes a Langevin sampling algorithm to align with the forward maximum perturbation distribution. Extensive experiments have verified the effectiveness of the proposed method in generating better graphs.", "questions": "(1) In line 87, the author mentions that the proposed bias correction method can be integrated into existing methods (e.g., spatial, spectral, and hyperbolic domains). I wonder if it also includes discrete graph diffusion, such as DiGress, since that one proves to be more effective in generating discrete graph structure.\n\n(2) In line 205, how do we get the score function $s_{\\bar{\\theta}, t}(\\cdot)$? Is it $s_{\\phi, t}(\\cdot)$ based on the Appendix B?\n\n(3) I do not fully understand the insights drawn between line 232 and line 236. Why do these experiments provide these two directions for addressing the reverse sampling bias?\n\n(4) How is the continuous diffusion model equipped with the proposed strategy here compared with the discrete diffusion model such as GDSS?\n\n(5) In Figure 2(c), why the perturbation at the very early stage does not lead to any tweaking impact in $s_{\\theta, t}(\\cdot)$?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "(1) The observed misalignment between the maximum perturbation distribution and the actual starting distribution in the generation phase is very important and novel.\n\n(2) The proposed technique for correcting the misalignment bias is technically sound. Furthermore, Figure 1 has demonstrated its effectiveness.\n\n(3) Extensive experiments have been conducted to verify the effectiveness of the proposed method.", "weaknesses": "(1) Some of the motivations are not so clear based on the experiments. See question in (3).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zlri9JJ9JU", "reviewer": "Reviewer_fe9F", "summary": "This paper improves GDSS  from a unified perspective and solves a significant problem of exposure bias during graph sampling.\nTheir approach requires no network modifications, which is validated across multiple models, datasets, and tasks compared with SOTA methods.", "questions": "1. Can the proposed model generate attributes?\n\n2. The authors repeatedly emphasize that \"Their approach requires no network modification.\" My confusion is that the noise or bias will not be the same for different networks. Suppose the proposed approach is robust to the network and does not require modifications in different scenarios. How can it ensure that the new reverse starting point is robust and correct across different noises in different networks?\n\n3. The authors claim that they can achieve good performance across multiple tasks and have strong reusability. However, since the code is not open-sourced, I am unable to evaluate it.\n\n4. The abstract is very intriguing, but the introduction is somewhat difficult to read. For example, some symbol definitions and descriptions are not explained, and the authors assume that this knowledge is already known to readers. Additionally, I did not fully understand how the authors addressed the existing problems in Q1 and Q2, as this was not clearly explained.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed method effectively mitigates reverse starting bias by employing a newly designed Langevin sampling algorithm.  \n\n2. It introduces a fraction correction mechanism based on a newly defined score difference to address exposure bias.  \n\n3. The approach requires no network modifications and demonstrates state-of-the-art performance across multiple models, datasets, and tasks.", "weaknesses": "For me, I understand the problems the author wants to solve, and they are indeed meaningful (if solved). But I think the biggest problem is that the paper's presentation is not satisfactory, so I cannot give it a higher score. Perhaps, in the rebuttal process, reasonable explanations can alleviate my bias.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JTgivsovdu", "reviewer": "Reviewer_bDyB", "summary": "The authors focused on two problems in the Diffusion models for graph data. The first problem is the reverse starting point bias, in which the maximally perturbed data distribution is far from the standard normal distribution from which the reverse process starts during the inference. The second problem is the exposure bias, in which  diffusion model generates data with errors that accumulate across every step of the reverse process, leading to a large gap between the generated data and the true data distribution. The paper proposed solutions to the two problems respectively, and the empirical results are provided to verify the effectiveness of the methods.", "questions": "1. What is the key features and novelty of the proposed exposure bias correction in light of the existing methods?\n2. How was the $\\lambda$ selected in the experiment? Why do they differ across different experiments? How sensitive the results are to the choice of $\\lambda$ values?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "The paper is well motivated and explained the intuitive and mathematical explanations of the proposed solutions to the problems. In particular, the solution based on the Langevin sampling to the first problem sounds well and is novel.", "weaknesses": "**Main arguments**:\n\n- *A1*: Two solutions are proposed in the paper, which are not synergetic and not specific to graph diffusion models (e.g., they can be applied to other types of data). While this is a another strength, given its generic nature, the paper needs to position itself in the existing literature, in particular in context of the second problem (exposure bias). For example, some solutions to the exposure bias exists [1,2] for image generation, and they can be replaced with the proposed solution to the exposure bias. How effective is the proposed solution compared to the existing ones is not clear in the paper, questioning the significance of the solution.\n  1. Li, Mingxiao, et al. \"Alleviating exposure bias in diffusion models through sampling with shifted time steps.\" arXiv preprint arXiv:2305.15583 (2023).\n  2. Ning, Mang, et al. \"Elucidating the exposure bias in diffusion models.\" arXiv preprint arXiv:2308.15321 (2023).\n\n- *A2*: The solution to the exposure bias has a key parameter, $\\lambda$, that controls the degree of correction during inference. In the experiment, the authors used specific value of $\\lambda$. However, it is not clear how did they choose and whether the method is sensitive to the choice of $\\lambda$. If  $\\lambda$ is a highly sensitive parameter, the practical value of the method is limited.\n\nGiven the above arguments, I believe that the paper is not strong enough to be accepted at ICLR 2025 at its current form.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5I5hK7P1NB", "reviewer": "Reviewer_S73B", "summary": "To solve the exposure bias and the reverse starting bias at the same time, the authors propose to adopt Langevin sampling to obtain samples aligned with the forward maximum perturbation distribution. A fraction correction mechanism is presented. It is based on score difference to solve the exposure bias of the fraction network. The approach is free of network modification and introduction of new components. Empirical experiments demonstrate that the proposed method can achieve state-of-the-art performance on multiple datasets and multiple tasks.", "questions": "Can you elaborate more about how graph diffusion modeling is different from the general diffusion modeling regarding the two bias?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Contribution: This paper aims at mitigating the reverse starting bias and the exposure bias at the same time, which is obviously an important topic that yet to be solved.\n\nSolid and sound method: The proposed fraction correction mechanism is elaborated clearly. Multiple medias are employed to help with understanding the methods.\n\nAmple empirical analysis: The proposed method is compared with multiple modern baselines on several datasets. The experiment results are presented clearly. Ablation tests are provided to demonstrate the importance of S++.", "weaknesses": "Novelty: Langevin sampling has long been recognized as one of the best approach to learn the distribution in diffusion models. This paper adapts it to the graph diffusion modeling task.\n\nExperiment: The environment of the experiment is not specified.\n\nAcceleration measurement: S++ is claimed to be able to generate good samples faster by using fewer steps of reverse diffusion. Experiment results showed that it could achieve satisfying samples in fewer steps. However, whether it costs more time in each step due to the extra computational burden is not discussed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)