Proper Hölder-Kullback Dirichlet Diffusion: A Framework for High Dimensional Generative Modeling
Authors: Wanpeng Zhang
Yuhao Fang
Xihang Qiu
Jiarong Cheng
Jialong Hong
Bin Zhai
Qing Zhou
Yao Lu
Ye Zhang
Chun Li
Venue: NeurIPS 2025 poster ()
Keywords: Dirichlet Distribution; Proper Hölder–Kullback divergence; Non-Gaussian diffusion processes
Primary area: probabilistic_methods

Abstract: Diffusion-based generative models have long depended on Gaussian priors, with little exploration of alternative distributions. We introduce a Proper Hölder-Kullback Dirichlet framework that uses time-varying multiplicative transformations to define both forward and reverse diffusion processes. Moving beyond conventional reweighted evidence lower bounds (ELBO) or Kullback–Leibler upper bounds (KLUB), we propose two novel divergence measures: the Proper Hölder Divergence (PHD) and the Proper Hölder–Kullback (PHK) divergence, the latter designed to restore symmetry missing in existing formulations. When optimizing our Dirichlet diffusion model with PHK, we achieve a Fréchet Inception Distance (FID) of 2.78 on unconditional CIFAR-10. Comprehensive experiments on natural-image datasets validate the generative strengths of model and confirm PHK’s effectiveness in model training. These contributions expand the diffusion-model family with principled non-Gaussian processes and effective optimization tools, offering new avenues for versatile, high-fidelity generative modeling.

Proper Hölder-Kullback Dirichlet Diffusion: A Framework for High Dimensional Generative Modeling
Wanpeng Zhang', Yuhao Fang', Xihang Qiu', Jiarong Cheng', Jialong Hong', Bin Zhai', Qing Zhou?.3, Yao Lu?, Ye Zhang' Chun Li
Beijing Institute of Technology, Beijing, 100081, China
Shenzhen MSU-BIT University, Shenzhen, 518172, China
Lomonosov Moscow State University, Moscow, 119991, Russia
lichu20020@smmu.edu https:///gittubb.com//uooooo
Abstract
Diffusion-basse generative models have long depended on Gaussian priors, with lit- de exploration alternative distributions. We introduce Proper Dirichlet framework that uses time-varying multiplicative transformations to de- fine both forward and reverse diffusion processes. Moving beyond conventional reweighted evidence lower bounds (ELBO) or Kullback-Leible upper bounds (KLUB), we propose two novel divergence measures: the Proper Hölder Divergence (PHD) and the Proper restore symmetry missing in existing formulations. When optimizing our Dirichlet diffusion model with PHK, we achieve Fréchet Inception Distance (FID) of 2.78 on unconditional CIFAR-10. Comprehensive experiments on natural-image datasets validate the generative strengths of model and confirm PHK's ' effectiveness in model training. These contributions expand the diffusion-mode family with principled non-Gaussian processes and effective optimization tools, offering new avenues for versatile, high-fidelity generative modeling.
(PHK) divergence, the latter designed to
Introduction
Diffusion-basee generative models are revolutionizing deep learning by simulating gradual noise addition and removal processes to transform simple priors into complex data distributions. From high-fidelity image synthesis and editing [15 to video generation [6 7 and protein design [8-I0] As such, there great interest in applying diffusion models and improving them further terms of distribution quality, training cost, and image performance.
Recent efforts have investigated non-Gaussian diffusion processes, extending to categorical dis- tribution [1], Poisson distribution [15], and Beta distribution [16]. However, Beta diffusion incurs limitation that it applies one-dimensional Beta tr ansitions independently along each coor- dinate, preventing it from capturing interdependencies among components in high-dimensional simplex. Traditional divergence measures such as the Bellbackk-eibblle (KL. divergence suffer from asymmetry and computational complexity.
To address these limitations, in this work, we proposed Proper Külder-Kullback Dirichlet Diffusion, which is a fundamentally novel framework and can more faithfully captures the sinplex-structured target space.
Cooresponning author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Its main contributions are summarized as follows:
1. K K-dimensional Dirichlet diffussin on simplex is established. Beta diffusion is general- ized to K K-dimensional Dirichlet process that preserves the simplex structure at every noising step.
2. Strictly proper Hölder training objectives are proposed. Closed-form losses based on the Proper Hölder Divergence (PHD) and Proper Hölder- (PHK) are obtained, and analytic gradients under the Dirichlet exponential- fammily oog-ooraalizer are guaranteed.
3. Variance-reduced gradient estimates are proven. It is shown that PHD yields lower- variance gradients compared to pseudo-divergence objectives, improving training stability and efffcciecy.
State-of-the-an non-Gaussian performance is demonstrated unconditional CIFAR-I0: FID of 2.78 is achieved, outperforming all prior non-Gaussian diffusion methods and most Gaussian baselines. while PHK concomitantly reduces approximately a 9 % training cost compared with KLUB. Additional uncurated samples trained on CIFAR-10, 10, AFHQ, FFHQ and CVUSA dataset exhibit both high fidelity and diversity. By breaking free from Gaussian noise and KL-centric optimization. Proper Kullbckk Dirichlet Diffusion opens up a richer design space for generative modeling, and offering both rigorous theoretical guarantees and clear empirical gains.
2 Background
Diffusion- deep generative models have shown remarkable potential for development. Both Langevin dynamics and variational inference interpretations of diffusion models fundamentally depend on the properties of Gaussian noise employed in training and sampling pipelines [7]. Initially inspired by diffusion models utilizing Gaussian noise, various Gaussian diffusion models have emerged [8] 2 19 3. These models use Gaussian Markov chains to progressively diffuse images into Gaussian noise during training. Conversely, the learned reverse diffusion process, defined by reversing the Gaussian Markov chain, iteratively refines the noisy inputs to generate clean, realistic images.
Gaussian diffusion models can be further categorized from perspectives such as score matching 20
21
18
19 and stochastic differential equations (22). These models have demonstrated notable
successes across diverse tasks, including image generation and editing [4] 5 23 2D-to-3D
conversion [27 28
calibrated Bayesian inference (29), audio synthesis 30 32 conditional video
generation 6 7 reinforcement learning 35) few-shot learning (36] protein design 10 and
cross-modal visual generation and editing 37 38
Typically, diffusion-based generative models are built following standard procedure 1 involving three core steps: (1) defining a forward diffusion process that progressively degrades data by introducing noise and reducing the signal-to-noiss ratio (SNR) from time 0 to 1; (2) specifying
reverse diffusion process that denoises the data in reverse time, from to O; and (3) discretizing the time interval [0.1] into finite steps, with the forward and reverse processes serving as a fixed inference network and learnable generator, respectively. Variational autoencoder inference [40 41 subsequently applied to optimize generator parameters by minimizing a weighted negative evidence lower bound (ELBO), comprising Kullback-Leibler (KI.) divergence losses at each discretized reverse step.
In this work, we explore various effective distributions and divergences to investigate the multiple possibilities of the model. Our focus has progressively shifted towards diffusion models based on the Dirichlet distribution and Hölder divergence. Unlike traditional Gaussian diffusion, our approach is based on models employing non-tran -radiiaaiiional distributions and the deliberate incorporation of specifically tailored loss functions. Specifically, the adoption of Dirichlet distribution and Hölder divergence, which eschews the reliance on Gaussian distributions during both training and testing phases, enables us to explore alternative distributions and divergences, thereby unlocking new pathways for generative modeling. This innovative approach has the potential to engender models with distinct theoretical properties and to stimulate the emergence of innovative research trajectories.

---

Method
3.1 From Beta distribution to Dirichlet distribution
The Beta distribution is a continuous probability distribution defined on the interval [0, 1] commonly used model probabilities proportions. Its probability density function is given by 442
a) xa (1 x 2)8-1
(1 (1-x)*- 8)
x 1E
(1)
Jo 4 (1 u a du
B(a, 3)
where B(xx, B) is the Beta function, and and 3 are shape parameters. Beta function can be expressed terms of the Gamma function: B(a, 3) [(a) r(++3)
For the Beta distribution, the mean and variance are:
a3
B[X] a a+B*
Var(X) (a+ ( +P(a+++)]]
In Beta diffusion [16]. for scalar data To E [0. 1]. the forward diffusion process is defined as: 2, =r Beta(7aa.2o, 77 o,zo)), where a, is time-dependent parameter, and is scaling parameter. When T, zт approaches uniform noise. During training, the parameters of the reverse process are obtained by minimizing the Kullback-Leibler (KL) divergence to obtaín parameterized model So (z l):
f = arg min Ez,,.o [KL(1(2, (2, 2o) ((2, [2, to = fo(z,, t)))
(3)
The Dirichlet distribution is a nigh-dimensional generalization of the Beta distribution, used model distributions on the simplex. Its probability density function is given by [43]:
T (E 08
K
a();a)
II o
x E SK, 2-11
1I I(a;)
(=1
where SK denotes the K-dimensional simplex, and a; is the shape parameter for the i-th dimension. For the Dirichlet distribution, the mean and variance are 43
(E aj -as)
E [X
Var( =
(5)
K 2j=1
aj
(Σj= a.)
k E/ ag +
is worth noting that the Dirichlet function can also be expressed in terms of the Gamma function. Compared to the Beta function, it extends to dimensions in the same form:
I I(ax)
Dir(a)
(6)
k=
Based on the above formulas and derivations, the one- mie-dimesioonl diffusion in Beta diffusion can be extended to K-dimensioonal diffusion to reasonably implement the diffusion process on high- dimensional simplices:
24 300 Dir(nor,11001, 700,,, T0r Ko,,
(7)
where a, are time-dependent parameter functions. By scheduling Ctti 2g 2o can be nearly noise-free, while approaches uniform distribution. The expected value is:
E (2.,110))
(8)
Σ, ar,
Dirichlet distribution enables direct handling of high- -ith-mmiiiiinll data without decomposing the high- dimensional problem into multiple one-dimensional problems. It offers significant advantages in mod- eling data distributions high-dimensional simplices and naturally extends from low-dimensional to high-dimensional diffusion processes. This extension makes the Dirichlet distribution more flexible and efficient in dealing with digeenimenii data.

---

3.2 Training via Proper Hölder-Kullback Divergence
For the diffusion process of the Dirichlet distribution on high-dimensional simplices, to ensure that the objective function has closed-form solution under the Dirichlet distribution, this study considers introducing an objective function based on the Hölder divergence.
Two objective functions that meet the criteria are available for further experimental validation: the Hölder Statistical Pseudo-DDivergence (HPD) and the Proper Hölder Divergence (PHD) [4].
The objective function based on HPD expressed as:
Dd (p 4) | F(a0,) + + F(30g) F(O, + 0g), B
(9)
For the Dirichlet distribution, F(0) has a closed-form expression, where 0, = 04 1. If == g(z, Zts ro) Dir(ap), q = (t, 20) (i,)
(10)
let pp.a Op,i objective is:
1, 0g Oy 1, then D (p q) can be computed analytically. The training
C E., No DD (g(2012)
&o = fo(2,.))))
(11)
The objective function based on PHD expressed as:
DH (p | F()0))))) + F(10g) F( 2220p +
3 3
(12)
Similarly, closed-fore solution can be obtained. The training objective is:
CPHD = Es,, D a.7 (q(2a 22, zo) (l4 22, Zo = fe(a, t)))]
(13)
Theory and experiments indicate that PHD yiclds superior performance, advantage associated with the strict propriety of PHD. A detailed explanation provided in Appendix A
Furthermore, recognizing the unique advantages of PHD, combined with the mature foundation of KL theory the diffusion model domain, KLUB [16], this study introduces the Proper Kullbak Divergence (PHK). which also has a closed-form solution for the Dirichlet distribution. Given 5, e [0, 1] as weight coefficients, the objective function based on PHK expressed as:
cP CPHK = SCF + (1 8 + +) CKLUB
(14)
We summarize the training and sampling algorithms in Algorithms and respectively.
Algorithm Training
Require: Dataset: D (Xm)
m=1 Vn n=1 0.6, data scaling parameter Stale 0.39, gen- 0.95, and a sigmoid schedule defined by a [0, where co 10 and c 13
Mini-batch size B, concentration parameter
10000, data shifting parameter Smi erator for time reversal coefficient
1/(1 (11+6 (0- ggiven
1: repeat
2: Draw mini-batch Xo 10 pe from D for = to B do
3:
can be run parallel
4:
Unif(1e 1)
5:
8; TT;
6:
Compute as; and ate
z = (i) Scale + Shit
(i)
7:
8:
2t Dir(naa,
(5)
A ar, ((,)
( fo(2,, f.) Scate Shan
9:
10: compute the loss L:
end for
12: Perform SGD with v, DB C;
13:
until converge

---

Algorithm 2 Sampling
Require: Number of function evaluations (NFE) J 200, generator fo, and timesteps {6)}=0'
to (1 1e-1e- 5) 5)
-))(J for j ....... and to 0
1: if NFE > 350 then
2:
Oit; 1/(1 e ---(()))))))
3: else
4:
5: end if
6: Initialize 80 = R[mo] Scale + Smn
7: 2t, ~ $o: ... k- (1 Co)
8: for j J to do
9: Tu fo(2,, a,,) * Seate SSin
10: P(t,-1t(,) Dir(n((a,,, at, J:o. R- at, _0)
11: 21,-1 = 21, + +(1 21 JPL,, 1005--115 (x(
12: end for
13: return (án Sain / Seate or (2t0/ Smin// Scale
3.3 Forward and Reverse Diffusion
Figure 1: Demonstration of the forward diffusion process on five samples. the first column displays the original iamge, and the subsequent 21 columns illustrate progressively noising and masking images at time steps / 0, 0.05.
This study conducts analysis of the forward and reverse diffusion processes by incorporating process images. First, the Dirichlet forward diffusion process is visualized by displaying the real image & and its noise-corrupted version during the forward diffusion process. Specifically, the images with noise and masking at times t = 0, 0.05,.0.1. 0.1, are represented as:
2, max
min
Ft
Shin . .1) 0)
Dir(naa5o, K a.000)) (15)
Scate
As can be clearly seen from Figure during the forward diffusion process, the image becomes increasingly noisy and sparse with the time, and is ultimately almost estroyed. It evident that the forward process of Dirichlet diffusion involves simultaneous noising and masking of pixels. This distinct from traditional Gaussian diffusion, as the forward diffusion process in Gaussian diffusion gradually applies additive random noise and eventually concludes with Gaussian random noise.
Similarly, the reverse diffusion process of traditional Gaussian diffusion considered a denoising process, whereas the reverse diffusion process of Dirichlet diffusion involves simultaneous denoising and demasking of the data. The images representing denoising and demasking are expressed as:
4-1 = max
min
-t-1
Shan ).1)))))
(16)
Seate
aa-1
5

---

Figure 2: Demonstration of the reverse diffusion process on five examples. The first column displays the initial noisy, and the subsequent 21 columns illustrate the progressively denoising and demasking at time steps & 0.9...... O..
where 2 = 24 (1 2t Pit,,-,,t- -);--) is iteratively computed according to Algorithm 2 High- resolution images require more computational resources. Final generated image is represented as:
to = +t) E(ro za5]:
(17)
The generated image data will more accurately approximate the real iamge data when @ increasingly approaches its theoretical optimal value f", such that so- (2),, = E[ro sty).
As shown in Figure 2 starting from random noise
1or F0,.......,.,,,,...... mo,KK), ig = E(
(18)
most of the pixel values will entirely black. Dirichlet diffusion gradually denoises and restores the image to clean state through multiplicative transformations, demonstrated in Algorithm 2
Figure 3: Comparison of generated samples from the Beta Diffusion model and Dirichlet Diffusion model trained unconditional CIFAR-10 image dataset. (L) Beta Diffusion and (R) Dirichlet Diffusion. This figure presents side-by-side examples of images synthesized on the CIFAR-10 image dataset by diffusion frameworks.
4 Experiments
4.1
Comparison Study
This study offers comprehensive numerical comparison with diffusion models based on traditional Gaussian distributions and those based non-Gaussian quasi- GausiGa distributions that have

---

T encompasses wide range of diffusion models. The results demonstrate that the new diffusion model proposed in this study achieved Fréchet Inception Distance (FID) 45 score of 2.78 on the unconditional CIFAR-10 10 dataset [46], surpassing all mon-Gaussian-bant baase diffusion models, including the cold diffusion model based on deterministic diffusion [77]. the Inverse Heat Dispersion based on deterministic diffusion [4]] the D3PM diffusion model based on categorical distribution [12]. and the JUMP diffusion model based on Poisson distribution [1]] Compared with the family of Gaussian diffusion models, the new diffusion model this study outperformed the majority of Gaussian-like models, including NCSNv2 19.. DDPM 22. DiffuEBM 48). VDM [3,, Improved DDPM [49, TDPM+ [50], Soft Diffusion [51. Blurring Diffusion [52]. etc. thus serves as highly competitive alternative when employing diffusion models. As newly proposed diffusion model, constrained by limitations in computational resources and training time, this study has not yet been able to conduct a detailed investigation and global search of all model parameters, nor has it introduced targeted enhancement modules. As novel perspective on diffusion models, this study holds broad potential for future research. hyperparameter settings and training details of the experiments provided in Appendix Faand
been publicly released in recent years at NeurIPS, ICLR, ICML, etc. Table
Table FID scores various generative models trained on the CIFAR-10 dataset. Lower FID indicates better image fidelity and diversity.
Distribution
Model
Year
FID (L)
DDPM 2
NeurlPS'20
3.17
NCSNv2 19
NeurlPS'20
10.87
DiffuEBM
ICML'21
9.58
VDM 3
NeurIPS'21
4.00
Improved DDPM 49
ICML'21
2.94
Gaussian
TDPM+ (50)
ICLR'23
2.83
Soft Diffusion
51
TMLR'23
3.86
Blurring Diffusion
15
ICLR'23
3.17
Blackout Diffusion [53
ICML'23
4.58
GET [54]
NeurlPS 24
5,49
Diff-Instruct (55
24
4.53
UniPC with optimized step [56
CVPR'24
3.13
RDUOT 57
ECCV'24
2.95
80.08 (deblurring)
Cold Diffusion
NeurlPS'23
Deterministic
8.92 (inpainting)
Inverse Heat Dispersion 47
ICLR'23
18.96
Categorical
D3PM 12
NeurIPS'21
7.34
Beta
Beta Diffusion [6
NeurIPS'23
3.06
Poisson
JUMP [5]
ICML'23
4,80
Dirichlet
Ours
2.78
4.2
Ablation Study
In this study, we meticulously designed and conducted series of ablation experiments to systemati- cally validate the effectiveness the proposed PHD and PHK based on Hölder divergence. Through these experiments, we investigated the performance and potential advantages of these two methods.
As shown in Table the experimental results clearly reveal the superiority of PHD and PHK over traditional methods. Specifically, compared with the widely used -ELBO, PHD achieves better generation performance and demonstrates unique capabilities in image processing. Moreover, PHK. as hybrid optimization method, can effectively integrate the advantages of different divergences. processes image data more comprehensively than other divergences, thereby achieving optimal performance. This result further validates the effectiveness and superiority of our proposed methods, indicating that PHD and PHK hold broad research prospects and application potential in the field of image generation.

---

Table 2: Ablation study on CIFAR-IO varying number function evaluations (NFE) and mini-batch size B,
identifying the optimal NFE-B configuration that maximizes performance per computation and guides efficient resource allocation.
Loss B
ELBO
-ELBO
KLUB
KLUB
PHD
PHD
PHK
PHK
512
288
512
288
512
288
512
288
20
15.81
16.02
16.62
16.35
16.84
16.95
15.92
16.53
50
6.53
6.65
6.16
6.20
6.48
6.93
5.87
6.15
200
4.49
4.68
3.46
3.55
3.85
4.43
3.20
3.29
500
4.33
4.42
3.39
3.46
3.72
4.23
2.92
2.99
1000
4.28
4.42
3.32
3,40
3.52
4.19
2.84
2.91
2000
4.31
4.39
3.25
3.21
3.51
4.21
2.82
2.81
We also include Figure to visually compare generated images under Beta Diffusion and Dirichlet Diffusion on CIFAR-IO, Figure visually compare generated images under Dirichlet Diffusion on FFHQ [58] and AFHQ [59]. Owing the considerable time required for each training iteration and the computationally demanding nature FID assessment, we have not yet explored the optimization of these hyperparameter combinations, given the constraints of our current computational resources. Consequently, although the results presented in this paper illustrate that Dirichlet diffusion can achieve competitive performance in image generation, they do not fully capture the potential capabilities of Dirichlet diffusion. Further enhancements to these results may be attainable through optimized hyperparameter configurations network architectures specifically designed for Dirichlet diffusion. We defer these investigations to our future work.
(a) FFHQ
(b) AFHQ
Figure 4: Generated images from the Dirichlet Diffusion model on the FFHQ and AFHQ datasets. (a) FFHQ samples.(b) AFHQ samples. These results demonstrate that Dirichlet Diffusson produces high-fidelity, diverse images, capturing both fine-grained human facial details and the varied textures and structures of animals.
5 Related Work and Future Directions
Various diffusion processes, including Gaussian diffusion, Beta diffusion, Poisson diffusion, and Dirichlet diffusion, employ specific distributions in both forward and reverse sampling. Gaussian- based diffusion models initiate their reverse process from the standard normal distribution N ((, 1), whereas Poisson, Beta, and Dirichlet diffusion all start from The reverse sampling Dirichlet diffusion is monotonically non-decreasing process, similar to Poisson diffusion. However, while Poisson diffusion involves discrete jumps in count values, Dirichlet diffusion employs continuous jumps in probability values, consistent with Beta diffusion.

---

In recent years, several works have actively explored diffusion models based on non traditionnl Gaussian distributions. Bansal et al. [77] proposed cold diffusion, which constructs models based on arbitrary image transformations rather than Gaussian noise. Rissanen et al. 47 introduced Inverse Heat Dispersion, which generates images by stochastically reversing the heat equation. Austin et al. [12] proposed the D3PM diffusion model based on the Categorical distribution. Chen et al. [T5] proposed the JUMP diffusion model based on the Poisson distribution. In comparison, Dirichlet diffusion, which innovates in terms of noise distribution and training loss, has achieved new state-of-the-art FID for non-Gaussian models on the unconditional CIFAR-IO dataset.
There have also been several works in the quasi- Gaussian diffusion domain in recent years. Hooge- boom and Salimans [60] proposed blurring diffusion, which shows that blurring can equivalently defined using a Gaussian diffusion process with anisotropic noise and proposed incorporating blurring into Gaussian diffusion. Daras al. [51] proposed soft diffusion, which employs linear erosion processes such as Gaussian blurring and masking. Rissanen et al. 47 proposed a diffusion process based on inverse heat diffusion, which reverses the heat equation using the inductive bias from Gaussian diffusion models. These diffusion processes share similarities with Gaussian diffusion in terms of loss definition and the use of Gaussian-based reverse diffusion for generation. contrast, Dirichlet diffusion differs from them in terms of noise distribution and training loss and outperforms most quasi- Gaussian models on the unconditional CIFAR- 10 dataset.
For a long time, the standard diffusion models have involved adding noise and reversing the degrada- tion of image processing. Bansal et al. [77 observed that the generative behavior of diffusion models does not strongly depend on the choice of image degradation. Hoogeboom et al. (52) showed that blurring can equivalently be defined through a Gaussian diffusion process with anisotropic noise, serving as an alternative to isotropic Gaussian diffusion. This has sparked the potential consideration of Dirrchlet+Blurring By establishing this connection, it is hoped to bridge the gap between reverse heat diffusion and denoising diffusion, leading to a generalized diffusion model that can reverse any process and thereby effectively eliminate the dependence on single type of noise.
The improvement of neural image compression and reconstruction is also direction worth discussing. The general consensus is that replacing the decoder with a conditional diffusion model can enhance the perceptual quality of neural image compression and reconstruction. However, their lack of inductive bias for image data limits their ability to achieve oott--ff-hhe-rt perceptual performance. Khoshkhahtinat et al. [6] proposed that employing an anisotropic diffusion model on the decoder side can disentangle frequency content, thereby facilitating the generation of high-quality images. Dirichlet diffusion, which has room for approximate improvement on the decoder side, provides the capability for seamless integration within the operational space of neural networks, thus further optimizing image generation quality.
6 Limitations and Conclusions
Existing diffusion models predominantly rely on traditional Gaussian distributions for both the forward and reverse processes. In this work, we demonstrate that it is possible to integrate the non-traditional Dirichlet distribution and the Hölder-based Divergence within the framework of diffusion models. Our experimental results confirm its superior performance image generation tasks. with highly competitive quantitative metrics. as well as its unique qualities when applied to generative modeling of high-dimensional simplex data. Additionally, these results highlight the effectiveness of PHD and PHK in optimizing Dirichlet diffusion. This framework offers a broader and more in-depth space for the application of more diverse diffusion models that transcend the Gaussian noise paradigm, including but not limited to image generation.
Despite its effectiveness, the method has some limitations: parameters; the existing experiments have not covered all possible combinations of hyperparameters, necessitating broader range of experiments and validation. The training cost high; processing 200 million CIFAR-10 images using four Nvidia L40S GPUs takes approximately 64 hours with batch size B 512, and a more substantial computational resource required given the multitude of parameter quantities and extensive tuning scenarios.
is sensitive to the selection of hyper-

---

7 Acknowledgments
This research was supported by Guangdong Basic and Applied Basic Research Foundation (No. 2024A151501 1774), the National Key Research and Development Program of China (No. 2022YFC311000), the National Natural Science Foundation of China (No. 12171036), Shenzhen Sci-Tech Fund (Grant No. RCJC20211211000000000
References
[1] Jascha Sohl-Dickstein, Eric A. Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In Proceedings of the 32nd International Conference on Machine Leurning, ICML 2015, Lille, France, 6-11 July 2015, pages 2256-2265, 2015.
[2] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic mod- els. In Advances in Neural Information Processing Systems 33: Annual Confer- ence on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12 2020, virtual, 2020. URL 4c5bcfec8588 www.o///f1a11011110101
weg.ssseuriip... Abstract.hcae html
[3] Diederik P. Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. Advances in Neural Information Processing Systems 34: Anmual Conference on Neural Information Processing Systems 2021, NeurlPS 2021, December 6-14, 2021, virtual, 34:2169--21707, 2021. URL httpsttt/////rreeddinng htwpo/eeeddiinnnnnnnuuiss..eeeee wwtpuriiss.cpppppppe..eoo ffless/aperr paper/ pdf
[4] Prafulla Dhariwal and Alexander Quinn Nichol. Diffusion Models Beat GANs on Image Synthesis. In Advances in Neural Information Processing Systems 34; Anmual Conference on Neural Information Processing Systems 2021, NeuriPS 2021, December 6- 6-14, 202 virtual, pages 8780 8794, 2021. URL https: htttp:://proceedingg neurips. cttpsccccpppp////pppppppppp
happs/appprrpr/e/ / 2021 h0/h/hhhhh///
(942232eeee2eebaara00/aa...d. wts.ccaaaa-btttattttt html
[5] Jonathan Ho, Chitwan Saharia, William Chan, David Fleet, Mohammad Norouzi, and Tim Salimans. Cascaded Diffusion Models for High Fidelity Image Generation. Journal Machine Learning Research, 23:47-1, 2022.
[6] Qi Tang, Yao Zhao, Meiqin Liu, and Chao Yao. Seeclear: Semantic distilla- tion enhances pixel condensation for video super-resolution. In Advances Neu- ral Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 wwpp.cc//ppp. wwtpppprrfilleaaaeeeeeeeeepeeeee www.ffeleepppe//eeeeee.. pappppppp////aeee/p Abstract- Conf
15, 2024, 2024. URL oagara/aaeeeaattaattaaaooooreeee html 15556228800dd33403244666963
[7] Giannis Daras, Weili Nie, Karsten Kreis, Alex Dimakis, Morteza Mardani, Nikola B. Kovachki, and Arash Vahdat. Warped diffusion: Solving video inverse prob- lems with image diffusion models. In Advances in Neural Information Process- ing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurlPS 2024, Vancouver, BC, Canada, December 10 15, 2024, volume 37, pages 101116-10114,, 2024. UtL/http:/ppapet mips. ct/pspsrrrppppp___cpee_ phpape_ffles__ http:/aapp/222////ppp 07384440338876652499bbbbbbb Aww.eeo--bbsttaacc Confott-ooooeeeeee html
[8] Chence Shi, Shitong Luo, Minkai Xu, and Jian Tang. Learning gradient fields for molecular conformation generation. In Proceedings of the 38th International Conference on Machine Learning, ICML. 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Ma- chine Learning Research, pages 9558-9568. PMLR, 2021. URI. htl.:http::/prrceediii .mml press/v139/shr shi21b........ sttiiibb.haat html
[9] Shitong Luo, Yufeng Su, Xingang Peng, Sheng Wang, Jian Peng, and Jianzhu Ma. Antigen- Specific Antibody Design and Optimization with Diffusion-Based Generative Models for Protein Structures. In Advances Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November
10

---

28 December 9, 2022, 2022. URL.ttpp://paprrrs nips ptpppccccpppr...lleeeeeeeeeeeeep /022/ htap://2...caaaaa....oooooo6666. -btt:attt7t-tttttttttttttooooo Cebstract-Cooferencce Conference.eccoeoo htmll
2022/
[10] Bowen Jing, Gabriele Corso, Jeffrey Chang, Regina Barzilay, and Tommi S. Jaakkola. Torsional Diffusion for Molecular Conformer Generation. In Advances in Neural In- formation Processing Systems 35: Anmual Conference on Neural Information Process- ing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 Decem- ber 9, 2022, 2022. URL http: http::/papprs...ipp cttpscccpppppeeeee. 99454562308bbbbccc/eee e336687777 eea9e44f www.466ffAbstrrat---ooeoe----- CenttractCConferee .tml
[11] Emiel Hoogeboom, Didrik Nielsen, Priyank Jaini, Patrick Forré, and Max Welling. Argmax flows and multinomial diffusion: Learning categorical distributions. In Ad- vances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual, pages 12454-12465, 2021. URL proceeddigg. www.diiinnnnnn.iss.....mm. cc/ wwtpccpppppp////eeeeeee 202//hash/ 1779d55aaddff1119adaaaaaaa -bbtrsoot2447--ttttattt.t..... html
[12] Jacob Austin, Daniel D. Johnson, Jonathan Ho, Daniel Tarlow, and Rianne van den Berg. Structured denoising diffusion models in discrete state-spaces. In Advances Neural Information Processing Systems 34: Annual Conference on Neural Informa- tion Processing Systems 2021, NeurlPS 2021, December 6-14, 2021, virtual, pages
17981-17993, 2021. URL https::/proceeddii wwwpeeddiinnnn.uuup..... nwurips..nccmmm cc/ pappp//222///ashh 200p 958c530554f (wtee//edc888ebbbbeee
[13] Shuyang Gu, Dong Chen, Jianmin Bao, Fang Wen, Bo Zhang, Dongdong Chen, Lu Yuan, and Baining Guo. Vector quantized diffusion model for text-to-image synthesis. In IEEF/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pages 10686-10696. IEEE, 2022. doi: 10. 1AL11/CCV222888.2220000000 URL httpp:///doo orgo 1119/
[14] Minghui Hu, Yujie Wang, Tat-Jen Cham, Jianfei Yang, and Ponnuthurai N. Suganthan. Global context with discrete diffusion in vector quantised modelling for image generation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pages 11492-11500 IEEE, 2022. doi: 10.109//CPR5228..22200000 URL https: /oi. /ttp::/::///ooo/////////////// 110.11//511000//022888888 2022. 01121
[15] Tianqi Chen and Mingyuan Zhou. Learning to Jump: Thinning and Thickening Latent Counts for Generative Modeling. International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawai, USA, volume 202 of Proceedings of Machine Learning Research, pages 5367-5382. PMLR, 2023.
[16] Mingyuan Zhou, Tianqi Chen, Zhendong Wang, and Huangjie Zheng. Beta Diffusion. In Advances in Neural Information Processing Systems 36: Annual Conference Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 16. 2023, 2023. URL http::/pppperrs.etttm nips 4ffe1155.ee22f/6333//6eeeccccec2 Abw.:/essscccta-aaaaaaaaaae Corf--oooffeeeeeee htmmen
sopper_fllssppaee http:/paap//22///eeeee 2223//aah/
[17] Arpit Bansal, Eitan Borgnia, Hong-Min Chu, Jie Li, Hamid Kazemi, Furong Huang, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Cold Diffusion: Inverting Arbitrary Image Transforms Without Noise. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, New Orleans, LA, USA, December 10 16, 2023, 2023. URL http:://///p nips cc/ paper files www.eeesspper/eaeeeaaaaaaaaaeeee http:/appe/22///aa// 80f 00f65517680077111188222244444443 Abeetld----bbaaacc wwwncereweec.ccooooooooocoooooee html
[18] Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. In Advances Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NewrIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 11895-11907 2019. URU/tttsss//pprceeiiiie nwwrinuurp.... cc/paper.........epcee htw.://9....Atttt.......tt

---

[19] Yang Song and Stefano Ermon. Improved techniques for training score-based genera- tive models. In Advances in Neural Information Processing Systems 33: Annual Con-
ference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-
12. 2020, virtall, 2020. URL httpps://proceeedinnnee http://prroreddddnnn..errppppppp 92c3b91631 9 .htm.
[20] Aapo Hyvärinen. Estimation of Non-Normalized Statistical Models by Score Matching. Journal of Machine Learning Research, 6(24):695-770) 2005. htRp:htp:///pt http://mwwoompppprraassea
hyvarinen05a. html
[21] Pascal Vincent. A connection between score matching and denoising autoencoders. Neural Computation, 23(7):1661-1644 2011.
[22] Yang Song, Jascha Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In 9th In- terattonnl Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May www.:nee/eeeeee hett/forumidd==xIlle GIRHHIS
7, 2021. OpenReview.net, 2021. URL.htttss://opeeeeee
[23] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hicrarchical text-conditional image generation with clip latents. arXiv preprint wrXiv:2204.06125 2022.
[24] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. In IEEE/CVE Conference on Computer Vision and Puttern Recognition, CVPR 2022, New Orleans, LA, USA, June 18- 24, 2022, pages 10674-10685 IERE, 2022. doi: R21110/CCVR52288.00222222 https://doi. http:://doo..og/////1 100.111////VP288B..OO
URL
[25] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L. Den- ton, Seyed Kamyar Seyed Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, Jonathan Ho, David Fleet, and Mohammad Norouzi. Photorealistic text-to-image diffusion models with deep language understanding. In Advances in Neu- ral Information Processing Systems 35: Annual Conference on Neurul Information Pro- cessing Systems 2022, NeurlPS 2022, New Orleans, LA, USA, November 28 Decem- Saaper/202//has/
ber 9, 2022, 2022. URL http::/ nips. cc/ paperrappr.... httpppaapepeeaeaaallllllllaapeee wwwpeeewweeeeeeeeeeeeeeeeeeeeeee (7795aaadaebb7223faaccceeef wwwpareroooooAAAoAAAaaeee Awstract- Conf-Coooeeeeeeee erence html
[26] Zhendong Wang, Yifan Jiang, Yadong Lu, Yelong Shen, Pengcheng He, Weizhu Chen, Zhangyang (Atlas) Wang, and Mingyuan Zhou. In-Context Learning Unlocked for Diffu- sion Models. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10 - 16, 2023, 2023. URL http::/ppppes nips cc/ wwtpcpcppppppepecccc htpp:/aappesssssss////aalaesssse files/ wwwps/iaateeeeeeeeeeaa 1D77080008881110043877777900-0 wttp://eooooabatttt------oooooee
www.bstrract-oooeeeeneeee wwwferenceeeccoccoocooooo html
[27] Ben Poole, Ajay Jain, Jonathan T. Barron, and Ben Mildenhall. DreamFusion: Text-to-3D using 2D Diffusion. In The Eleventh International Conference on Learning Representatiinss ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL /ttp:
hot//forummiddFFJlyyyScc.
[28] Mohammadreza Armandpour, Huangjie Zheng, Ali Sadeghian, Amir Sadeghian, and Mingyuan Zhou. Re-imagine the negative prompt algorithm: Transform 2D diffusion into 3D, alleviate Janus problem and beyond. arXiv preprint arXiv:2304. 2444063,0 2023. https:::://pp rithub.io/ (the first three authors contributed equally).
[29] Daniela de Albuquerque and John M. Pearson. Inflationary Flows: Calibrated Advances in Neural Infor-
Bayesian Inference with Diffusion-Based Models. mation Processing Systems 38: Annual Conference on Neural Information Pro- cessing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10
15, 2024, 2024. URL Inttp://papers: http:://pppprr..i...... hwtpcccccppppeeeeeeeeeeeeeeeeeee
wtw.:errrtsebbbaaaaaatatt------e Abstract-Conf wttps/ffleeppppap/eeaaee
html
[30] Nanxin Chen, Yu Zhang, Heiga Zen, Ron J. Weiss, Mohammad Norouzi, and William Chan. WaveGrad: Estimating Gradients for Waveform Generation. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net,
2021. URL httpttttt//pprreeeeeeee http://ppeenreeeeeeeeeeeorrrrrra httt:/oorumii.--eelllijiicBBo..e
12

---

[31] Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. DiffWave: A Versatile Diffusion Model for Audio Synthesis. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021.
[32] Donghhao Yang, Jianwei Yu, Helin Wang, Wen Wang, Chao Weng, Yuexian Zou, and Dong Yu. Diffsound: Discrete Diffusion Model for Text-to-Sound Generation. IEEE ACM Trans. Audio Speech Lang. Process., 31:1720-1733, 2023. URI. https://doii org/ 10. 1109/TASLT
2023. 3268730
[33] Michael Janner, Yilun Du, Joshua B. Tenenbaum, and Sergey Levine. Planning with diffusion for flexible behavior synthesis. In International Conference on Machine Learning, ICMI. 2022, 17-23 July 2022, Baltimore, Maryland, USA. volume 162 of Proceedings of Machine Learning Research, pages 9902-9915. PMLR, 2022.
[34] Zhendong Wang. Jonathan J. Hunt, and Mingyuan Zhou. Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learning. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023.
[35] Tim Pearce, Tabish Rashid, Anssi Kanervisto, David Bignell, Mingfei Sun, Raluca Georgesou, Sergio Valcarcel Macua, Shan Zheng Tan, Ida Momennejad, Katja Hofmann, and Sam Devlin. Imitating Human Behaviour with Diffusion Models. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023.
[36] Baoquan Zhang, Chuyao Luo, Demin Yu, Xutao Li, Huiwei Lin, Yunming Ye, and Bowen Zhang. Metadiff: Meta-learning with conditional diffusion for few-shot learning. In Thirty- Fighth AAAI Conference on Artificial Intelligence, AAAI 2024, Thirty-Sixth Conference on Innovative Applications of Artificial Intelligence, IAAI 2024, Fourteenth Symposium Ed- ucational Advances in Artificial Intelligence, EAAI 2014, February 20-27, 2024, Vancouver, Canada, pages 16687-16695. AAAI Press, 2024. doi: 1609/ aaai. v38115. 29608
2,2688 URL
https: /sttp::/doi..rgge
[37] Ling Yang. Zhilong Zhang, Zhaochen Yu, Jingwei Liu, Minkai Xu, Stefano Ermon, and Bin Cui. Cross-Modal Contextualized Diffusion Models for Text- Guided Visual Generation and Editing. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024, 2024. URL https://opeerrvvi.et
[38] Ye Tian, Ling Yang. Haotian Yang, Yuan Gao, Yufan Deng, Xintao Wang, Zhaochen Yu, Xin Tao, Pengfei Wan, Di Zhang, and Bin Cui. VideoTetris: Towards Compositional Text-to- Text-to-Video Generation. In Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurlPS 2024, Vancouver, BC, Canada, December 10 - 15, 2024, 2024. URL http::/pappers.nnte nips. httpp//ecccplllpeeeeaaeeaeaa files/paper 10244/aahh344/ http:/345520bbbbb6114461661 wtw.://faaa.cca0eeeeaaeattttttta Conferrrcct---oooeeceeceeeeeee html
[39] Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the de- sign space of diffusion-based generative models. In Advances in Neural Infor- mation Processing Systems 35: Anmual Conference on Neural Information Process- ing Systems 2022. NeurIPS 2022, New Orleans, LA. USA, November 28 Decem- ber 9, 2022, 2022. URL http://papers. nips. cc/ htcccpaper_fiiie htteeeeffileepppppe/eaaae Conf html
[40] Diederik Kingma and Max Welling. Auto-Encoding Variational Bayes. In 2nd International Conference on Learning Representations, ICI. 2014, Banff, AB, Canada, April 14- 16, 2014, Conference Track Proceedings, 2014.
[41] Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic Backpropagation and Approximate Inference in Deep Generative Models. In Proceedings of the 31th International Conference on Machine Learning, volume 32 of JMLR Workshop and Conference Proceedings, pages 1278-1286. 2014.
[42] Karl Pearson. Contributions to the mathematical theory of evolution. Philosophical Transactions of the Royal Society of London. A, 185:71-110, 1894.
13

---

[43] Kai Wang Ng, Guo-Liang Tian, and Man-Lai Tang. Dirichlet and related distributions: Theory, methods and applications. John Wiley & Sons, United States, 2011.
[44] Frank Nielsen, Ke Sun, and Stéphane Marchand-Maillet. On hölder projective divergences. Entropy, 19(3):22,, 2017.
[45] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. In Advances in Newral Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 6626 6637. hwt.ggs..urii.... httpinnn...iiii..coceeeeeeeeeeee httprips.ccppppp/eeeee cc/ppppee/p11/ Abstract html
Curran Associates Inc., 2017. URL https: /ttp:::://prooee ppt.tteed///.oooednnggg...ss
[46] A. Krizhevsky and G. Hinton. Learning multiple layers of features from tiny images. Technical Report TR-2 University of Toronto, Toronto, Canada, 2009. URL https://wwwt cs
toronto. wrri// cifar html
[47] Severi Rissanen, Markus Heinonen, and Amo Solin. Generative Modelling with Inverse Heat Dissipation. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023.
[48] Ruiqi Gao, Yang Song, Ben Poole, Ying Nian Wu, and Diederik P. Kingma. Learning Energy- Based Models by Diffusion Recovery Likelihood. In International Conference on Learning Representations, Virtual Event, Austria, May 3-7, 2021, 2021.
[49] Alexander Quinn Nichol and Prafulla Dhariwal. Improved Denoising Diffusion Probabilistic Models. In Proceedings of the 38th International Conference Machine Learning, 1021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 8162-8171. PMLR, 2021.
[50] Huangjie Zheng, Pengcheng He, Weizhu Chen, and Mingyuan Zhou Truncated Diffusion Probabilistic Models and Diffusion- Adversarial auto-Encoders. In The Eleventh Interna- tional Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL httpttttt////peeeeeett
net/forum? H-HDxxaKK5K56
[51] Giannis Daras, Mauricio Delbracio, Hossein Talebi, Alex Dimakis, and Peyman Milanfar. Soft Diffusion: Score Matching with General Corruptions. Transactions on Muchine Learning Research, 2023. ISSN 2835-8856.
[52] Emiel Hoogeboom and Tim Salimans. Blurring diffusion models. In The Eleventh Interna- tional Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023.
[53] Javier E. Santos, Zachary R. Fox, Nicholas Lubbers, and Yen Ting Lin. Blackout diffusion: Generative diffusion models in discrete-state spaces. In International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202, pages 9034-9059. PMLR, 2023, URL /ttp:ttt////occeddiiii
mlr /eres//v00// /atp:///o/2/saaso..eaeo santos23a. html
[54] Zhengyang Geng, Ashwini Pokle, and J Zico Kolter. One-step diffusion distillation via deep cquilibrium models. Advances in Neural Information Processing Systems 38: Annual Confer- ence on Neural Information Processing Systems 2024, NewrIPS 2024, Vancouver, BC, Canada, December 10 15, 2024, 36:41914 41931, 2024.
[55] Weijian Luo, Tianyang Hu, Shifeng Zhang. Jiacheng Sun, Zhenguo Li, and Zhihua Zhang. Diff- instruct: A universal approach for transferring knowledge from pre-trained diffusion models. In Advances in Neural Information Processing Systems 38: Annual Conference Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10- 15, 2024, 2024.
[56] Shuchen Xue, Zhaoqiang Liu, Fei Chen, Shifeng Zhang, Tianyang Hu, Enze Xie, and Zhenguo Li. Accelerating diffusion sampling with optimized time steps. IEEE/CVF Conference on Computer Vision und Pattern Recognition, CVPR 2024, Seattle, WA, USA, June 16-22, 2024, pages 8292-8301. IEFE, 2024. doi: 101109CCCPP55233.00....00 URL https
//doi org/ /00 1 1109/ /B/////VRR2233.3 2024.
14

---

[57] Quan Dao, Binh Ta, Tung Pham, and Anh Tran. A high-quality robust diffusion framework for corrupted dataset. In Computer Vision ECCV 2024 18th European Conference, Milan, Italy, September 29- October 4, 2024, Proceedings, Part LXXXIV, volume 15142, pages 107-123. Springer, 2024. doi: 00.1007/978--3-01-722999 000-3-031-79007-20
URL httpp:://doi wttp//i...r/1....000...0
[58] Tero Karras, Samuli Laine, and Timo Aila. A Style-Based Generator Architecture for Generative Adversarial Networks. In Conference Computer Vision and Pattern Recognition, pages 4401-4410, 2019. doi: 10. 201.9/.....1190455550
[59] Yunjey Choi, Youngjung Uh, Jaejun Yoo, and Jung-Woo Ha. StarGAN v2: Diverse Image Synthesis for Multiple Domains. In Conference on Computer Vision and Pattern Recognition, pages -185-8194, 2020. doi: 1119//CVR22200000000002..
[60] Emiel Hoogeboom and Tim Salimans. Blurring Diffusion Models. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023, 2023.
[61] Atefeh Khoshkhahtinat, Ali Zafari, Piyush M. Mehta, and Nasser M. Nasrabadi. Laplacian- guided entropy model in neural codec with blur-dissipated synthesis. 2024 IEEE/CVF Con- ference on Computer Vision and Pattern Recognition (CVPR), pages 3045-3054, 2024. URL https://api. http:////pp.ssaaniiiisseeeee www.ntiiscchitttoooeco..o .org///roooo/oooooooooo CorpusID::2338B000
[62] Paul Holland and Samuel Leinhardt. An exponential family of probability distributions for directed graphs. Journal of the American Statistical Association, 76(373):33 1981.
[63] Charles Elkan. Clustering documents with an exponential-famde approximation of the dirichlet compound multinomial distribution. In William W. Cohen and Andrew W. Moore, editors, Machine Learning, Proceedings of the Twenty- Third International Conference (ICML 2006), Pittsburgh, Pennsylvania, USA, June 25-29, 2006, volume 148 of ACM International Conference Proceeding Series, pages 289-296 ACM, 2006. doi: 10.1 145/1 143844. 143881. 7/doi. or1/ 01445/1111444.... 1143881
URL https
[64] David Blackwell and Lester E. Dubins. A converse to the dominated convergence theorem. Illinois Journal of Mathematics, 7:508-514, 1963. httpss//p/p semanticscholar
org/CorpusID 122751594
[65] Brian DO Anderson. Reverse-time diffusion equation models. Stochastic Processes and their Applications, 12(3):313-326, 1982.
[66] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2009), 20-25 June 2009, Miami, Florida, USA, pages 248-255. IEEE Computer Society, 2009. doi: 10.1109/CVPR.20000 200..P....09.5066660 5206848
URL https:
7/doi. orooi..rrg11... 1109//.....// 2009. 5206848
[67] Krishna Regmi and Ali Borji. Cross-view image synthesis using conditional gans. In Proceed- ings of the IEEE conference on Computer Vision and Pattern Recognition, pages 3501-3510, 2018.
[68] Hao Tang, Dan Xu, Nicu Sebe, Yanzhi Wang, et al. Multi-channel attention selection gan with cascaded semantic guidance for cross-view view image translation. In Proceedings of the IEEF/CVF conference computer vision and pattern recognition, pages 2417-2426, 2019.
[69] Taesung Park. Alexei A Efros, Richard Zhang, and Jun-Yan Zhu. Contrastive learning for unpaired image-to-image translation. In Computer Vision-ECCV 2020: 16th European Confer- ence, Glasgow, UK, August 23-28, 2020, Proceedings, Part IX 16, pages 319-345. Springer, 2020.
[70] Aysim Toker, Qunjie Zhou, Maxim Maximov, and Laura Leal-Taixé. Coming down to earth: Satellite-to-stres view synthesis for geo-localization. In Proceedings of the IEEF/CVE Confer- ence Computer Vision and Pattern Recognition, pages 6488-6497, 2021.
15

---

[71] Divya Kothandaraman, Tianyi Zhou, Ming Lin, and Dinesh Manocha. Aerial diffusion: Text guided ground-to -oarral--rrrial view synthesis from single imgge using diffusion models. In SIG- GRAPH Asia 2023 Technical Communications, pages 1-4. 2023.
[72] Tim Brooks, Aleksander Holynski, and Alexei A Efros. Instructpix2pix: Learning to follow image editing instructions. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18392-18402, 2023.
[73] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 3836-3847, October 2023.
[74] Gaurav Parmar, Taesung Park, Srinivasa Narasimhan, and Jun- Yan Zhu. One-step image translation with text-to-image models. arXiv preprint arXiv:2403. 120066 2024.
[75] Ahmad Arrabi, Xiaohan Zhang, Waqas Sultan, Chen Chen, and Safwan Wshah. Cross-view meets diffusion: Aerial image synthesis with geometry and text guidance. arXiv preprint arXiv:2408.04222 2024.
[76] Junyan Ye, Jun Hc, Weijia Li, Zhutao Lv, Jinhua Yu, Haote Yang, and Conghui He. Skydiffusion: Street-to-satellite image synthesis with diffusion models and bev paradigm. arXiv preprint 2000:2408.01812. 2024.
16

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction articulate the contributions and scope of this study, including the methods, experimental results, and impacts.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We address the limitations of this research and the directions for future work in Section 5 [
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiitiinn asymptotic approximations only holding locally). ). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
17

---

Answer: [Yes]
Justification:
Every theoretical result has an associated proof in the appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The paper fully describes the experimental information, including the algorithm steps, hyperparameter settings, training details, and so on, in the main text and the appendix
F Hand G
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
18

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: The datasets used in the experiments publicly available. Code will be released with instructions.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nnis.cc
aublic/guudees/Coeeeuuiii.s cosssiiinolyyy for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ wttps/cccppclcccuui.eeeeeeccc
Sttelee//ooSSSSeissSioeaa www.ssssiicggoooooo for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The paper presents all the training and testing details. The information is introduced both in the main text and appendix G
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Consistent with previous work, error bars are not reported. The related work is conducted in accordance with precedent.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoo bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
19

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: For all experiments, the paper supplies details regarding the computational infrastructure employed.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics Ethics/https: httpp:///eerrrsssccppppll//hceie
Answer: [Yes]
Justification:
The research conducted in this paper adheres the Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. if there special consid- eration due to laws or regulations in their jurisdiction..
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
The broader impacts discussed in appendix H
Guidelines:
The answer NA means that there is no societal impact of the work performed.
20

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g.. gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
The image generation model presented this paper is equipped with essential protective measures, mandating that users comply with the usage guidelines for generative models to ensure controlled application of the model. The datasets used in the experiments are publicly available.
Justification:
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We cite both the paper and the license of each model and dataset
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
21

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperssi thood cww datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Assets are documented with README files and the documentation is provided alongside the assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question:
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
22

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does not involve LLMS as any important, original, non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy (https://neurips: hett::///eerrrps.ccc//ooeeereere cc///onfereeeee//222///.... for what should or should not be described.
23

---

A Theoretical Analysis
Definition (Proper vs. Improper Divergence 44 Let P denote the of all probability densities
on a measurable space (S2, v). A nonnegative functional D : P x P + R20 is strictly proper if
D(p ④) 0 =
p(z) ((x) for v-a.e. E Ω
(19)
there exist distinct densities p F 9 such that D(p :) O, then D is improper.
Definition 2 (Hölder Statistical Pseudo-Divergence (HPD) [44)). Let (Ω, F, be a measurable space with Lebesgue measure μ.. Fix conjugate exponents a, B satisfying 1/a + 1/3 For densities p E L (2, (1) and LL LL2, p)), each absolutely continuous w.r.t. μs, the HPD is
Ja p(x) q(x) du(x))
D.d (p: :p::0 q) = -log
(20)
(()())) du(x)) 1/a (Sn (()) du(x))) 1/8"
Definition 3 (Proper Hölder Divergence (PHD) [44). Using the same exponents a, 8 > and a parameter of 0, the proper Hölder divergence between densities p. q E [2, (1) is
Paz(z)/a g())1/ du(x)
D (p: :))p)=)0)))))))) D. (p 9218) log
(21)
du(xx) 1/a (In ((x) du(x)) 1/8"
Definition 4 (Exponential Family of Distribution 62 ). The probability density function of the Dirichlet distribution expressed as follows: p (a: exp T(x) F(0) + Bz)),, where is the natural parameter, T(x) is the sufficient statistic, F(0) is the log-normalizer, and (xx) is the base measure.
Definition 5 (Dirichlet Distribution (3)). The Dirichlet distribution of order K (where K 2 2) with parameters as > 0, 00:=1,25 1,2,3... is defined by probability density function with respect to Lebesgue measure the Euclidean space RK as follows:
Σ
O
K
f(r1...., ......,KKKa IK|aI, ak)
II
(22)
K
I(a))
1=1
where E Sx, and Sx is the standard K- dimentional simplex, namely.
SK
(2), 22 ..., kK)
Σ x4 =, O..05. 21. FK S i=1
(23)
and I(.) the gamma function, defined as: I(s) So x--1e-" dx, s> 0.
Definition 6 (The Exponential form of the Dirichlet Distribution 63 Exponential formulation of the Dirichlet distribution probability density function can be rewrite as
exp E (a; log ΣK log ((s)) - log ΣK 03) } Allowing obtain the
K
((ai)) v Σ
05)
==1
canonical form terms: =
a T(p) II(()), F(n)
v(ax) - /b Σ 04))
K K (a.) In Σ a.), B(x) = - -I(x),, and vp is the digamma function, defined as: vp(x)=
Σ
[())
Assumption 1. The model density is Dirichlet
go(2) Dir(2 (()),
ax(0)) > 0, an(() Σ (o(
(24)
supl [) < co.
(25)
0,
24

---

Assumption 2 (Integrability for PHD). For > O.
(a) 1 < S 1, no extra condition is needed (because p" E L' automatically for probability densities p.q).
(b) If y > 1, assume for every model instance that each Dirichlet concentration satisfies: (,) > 1 Vi. This guarantees p².q"' E L'"
Theorem 1 (HPD is improper, PHD is strictly proper). Let a, 3 > 0 with a a + and 0.
Then:
(i) HPD is improper. There exist densities # with D (p q) = 0.
(ii) PHD is strictly proper for a > 1. For densities p.q.
D (p ②) = p (v-a.e.).
(26)
(iii) Boundary 7 1. Statement (ii) still holds without any extra condition.
(iv) Right-limit a + 1*. Define (p q) == lim D (p q) (the limit exists). If
DH 1+,5 (P q) = 0. then p 9 a.e. Thus strict propriety extends to the a - 1 limit.
Proof. We write the two divergences in their "normalised Hölder form":
f |pqde
I Drtagii du
-log (f q2d /00 D (p:(p:9) D. (p: :) -log I ('a
{ D'd)) d' (
(27)
(i) HPD is improper. By Hölder's inequality, f pq s Illlaa1991e, so ( q) 2 0. Equality in Hölder holds iff pd =cq² a.e. for some c >>>>>> 0. One can pick p + q satisfying that proportionality (e.g. on (0, 1]. take p(x) 2r and g(x) 3x for (a, B) (3/2,3)): then D (p q) although p A Hence HPD is improper.
(ii) PHD is strictly proper when a > 1. Apply Hölder with f = pi/aa and g = 9 18:
f10 [9 S ( (e) 1/ (o) 1/
1/8 gr) (f p²) (r)y/()) ///l/()))
()))) 17)400
(28)
Thus DH (p: 2 0. Assume D (p : D4
a.7
= 0. Then we are in the equality case of Hölder, so
(p'/a) a = p² = =cq" = =c 00 a.c.
(29)
for some 0. Hence pd eq a.e., ie. p c cltig a.e. Since and 9 are densities, fp- = c/7 f = c//7, so c = 1, giving p = q a.e. Conversely, = the ratio is 1, hence DH (p: ②) 0.
(iii) The boundary 7 1. Put 1 in the PHD definition. The same equality- case argument applies verbatim (no extra integrability needed), so strict propriety still holds.
(iv) Right-limit a + 1. For a > define
prlag dv
((a) - log
101(p' D (p::)) q).
(30)
( (T) (pr)//// {
The maps + p/aa
97/8 are pointwise continuous for 1, and all integrands are dominated by integrable functions thanks to Assumption Therefore, each integral appearing in ((a) is continuous in a, and so is T.
I # q. the argument in (ii) shows that for every > 1, P(a) > (because equality in Hölder would force y = q). By continuity (1, there exists > 0 such that +(ax) 2 for all sufficiently close to 1. Consequently, lim (aa) 2 < > 0. Hence DH (p: (p * when # q. If p q, clearly (aa) = 0 for all whence the limit is 0. Thus the limit divergence Dit,, remains strictly proper.
D
25

---

This distinction explains why, in Dirichlet diffusion training, PHD supplies a reliable approximation signal, while HPD may induce "false convergence" phenomenon.
Lemma 1. Let Z ~ Dir(a) with on = =E 04 For any model parameter vector 0,
K
go(2) = Σ
log Z, - -b(s;)))) + vb(o)) Oga;.
(31)
Proof. The Dirichlet density on the (K |)-impleex SK is
K
I1 M(ax) T(as)
go(2)
11
2
-a-1 B(a)
(32)
B(a)
i=1
Hence
K +
log (000o=)==oo0
= - B(x))
(a 1) log zi.
(33)
(=1
Because 0, log [x)) (2) (the digamma function),
Og log B(a)) Σ ((as) Ooa; (b(o)) asa0:
0gar Σ
Daag
(34)
Differentiate 33 and group like terms:
(0(2) Vo log g0())
(35)
Σ (aas) doos ((o)) Σ Spay Σ (log 2;) 00as.
Insert the duplicated sum in the second term and combine:
K
go(2)
Σ
log 2, ((as) + ((ao) Ogas,
(36)
Finally evaluate at the random variable Z to obtain the stated identity.
Lemma 2. Let Z Dir(a) ()) the (K-1)- Sk. with (a1...., ak) and om Σ Og Fix an index E {1,..., K} and any X > -1. Define μ == v((,)) - vb(an), where ij is the digamma function. Then:
I(A) = { Z (log Zi - 1)² Dir(z dz
5x
(37)
I((a)) I'a: + +1)
Va + - s'(ao + A) [ab(a + A) vla0 + 1) -
[1]3]
[(a; + 2)
Moreover, using (2) S 1/2: for x > O and the mean-value bound (v(()) <b(u)] < 1u
wl/ minfu, v), we obtain the (loose but handy) upper bound:
I((a)) I'(a; + +X) I(a.)
2
I(A)
(38)
(a; A) + +
Proof. Step 1: Reduce to a one-dimensional Beta integral. The joint pdf of Z
T(ao) Dir(z a) 1 T(a;) j=1
K
11
2 Sk.
(39)
Because only involve Z,, integrate out =-; the marginal:
T(oo) to 100 - a000 a- t (0,
fz. [(a,)[((a - aa;)
(40)
26

---

i.c. Z₃ ~ (as ao - a.). Therefore
T(a))
L.
((A) = ((a;))(ao a.)
(1- (1-)ao - -11² dt.
(41)
Step 2: Notation and Beta derivatives. Set a := 0; + X, = 00 as B(a,b) a(a = So 10-
6-61 [[++6]
[ba)((b) Then
I(ao) I(A) [(a
/ (0-1(1 -
-1) dt.
(42)
t - 11)² dt. We need Ja, b; 1)) in closed form. Recall Define J(a, by pi) := f/ 10 - (logt the identities:
0 B(a, b) = B(a, b) [b(()) - s(a + bb],
(43)
da
0²
da B(a,b) b) = B(a, b) (a 4s + b) + [v((a) - 1i + b)]3]
(44)
Step 3: Express J via B and its derivatives. Expand the square: (log L - 11)² = (log 1 2/ loglog
2 Hence
Ja, b:), = (/' 10-1(1 - (log dt - 2/ (- (1-1(11-)6-1
- 111
log t dt + ((, b). (45)
S (1--(111)0-11 1)0-~)oog logt dt = da
f(0-1(1 ta-1(1 - (1-1 (1-1)-1-111000 t)2 dt Ø²
[
b),
B(a, b). (46)
-
da
Thus
8²
-2 á b(a, B(.
ba,b; da
B(a,b)
(47)
Da
Substitute the derivative formulas:
bx bia, (8 1/ (a)
(b(a) via b))²]
- -2pB(a, +
= (10) V'aa
(+(a) - ssa + b) - -11²}
Step 4: Plug back and simplity. Therefore
I'(ao) I(A) [(a:)
I'(ao)
b: T(a:)T(1)
-/0 111
(48)
[(a) T(a + and a a: +A a+b == 0m + X, obtain
Since B(a,b)
[ba B(a, b) = r(ao) (())(()) I(a + A)
r(ao)
(49)
[(a,) +(a+ I(a,)] X)
I(A) ((a A) (ai+ v' A) [a(a= + A) vaan A) -)) [=]=]] w1²]. (50)
[(a + A)
Step 5: A convenient upper bound. Use v (xx < 1/4 for x > 0 and, by the mean-value theorem, [u-o]
(u, 0). Here a; X, a; au X, v Crg so the squared bracket term is O X2 ///0(( + 1)1) + d /l(00 + 2)) Absorbing absolute constants, we
(b( (b)) min(u.0)
obtain the loose bound
T(X) < I(aa) +X)
22
(51)
[(a + X) (as + A) + X)
27

---

Lemma 3 (Gradient representations for HPD and PHD). Assume Assumptions normalising constants
Define the
/ Jsk p(z) (o(=) dz, / (z)dd
1 Ssc g0(2)² dz,
(52)
Jsx
p(2)7 dz, Ts f 90(2)² d2.
T SsK
p(2)7/0 go (2)2111 dz, T2
(53)
5x
Ss
Let model score is and introduce the probability weights
g)(2) Vo log
(54)
(2) p(2) (0())
(x) 20(2)
C
(55)
(x) p(2)/// (=)) Ti
((2) go(2)7
T'
Then
Vo voDa(p= : g0))
B go + tn go),
(56)
Vo 0. (p: qa)
7 3
(-Eascnees [80] + Eaace
(57)
Throughout, the interchange of Va and Jsx is justified by the dominated converggeeeeeeeeeeeeeeeee theorem (64] because Assumption [ places 0 in compact set and the Dirichlet density is C in 0. (i) HPD. Write
Proof.
1 B logc.
(= : go) = log A + log B +
(58)
Only A and C depend on 0. Using
Ozge(z) = =qo(z) (o0o)))))))))=)))))))) = (2) ge(z),
(59)
we obtain
VOA = / p pogq = A 1E,ass/oo
V.C = = 1. (go]-
(60)
Hence
1 VeA+
1 VoC = -Ema: [go] + 12 [B0],
VoD = A
+ 3 3
(61)
which is Equ. (56)
(ii) PHD. Similarly,
1 -
1 log T2 + 3 log T3.
DH, (p: : qo) log Ti
(62)
a
Only Ti and T3 involve 0. Chain differentiation gives
00g7
1/8
7 //8
%
0ogg = = 10.80 go.
(63)
B
hence
V = 2 Tı ggg [Bo)
V = 7 TT's Eggcn [Bo]
(64)
3
Substituting into the derivative of D
VODH
1 Voli
1
1
Vols
(-B [go] + (ege)/g)))
7 (-
(65)
0. Ti
3 T3
which is exactly Equ. (57
Theorem 2 (Variance domination of PHD). Adopt Assumptions 2 and suppose in addition that the Hölder scale satisfies y 2 1. Then
2
Var V VOLH
Ta (p: go)
<
Var
Var 2 VoD (p: go)
(66)
a
28

---

Proof. Step Gradient decompositions. Equs. (56) (57 give
Gn = VoD qo) = Ei [go] ( [go],
(67)
Gp VoD (p q0) 3 Bggus [oo] Bagcer lool)
(68)
Step 2 Two-variance trick. For any random vectors X, Y one has Var(X- - 2(Var X Y). Hence
Var(Gu) S 2( , [gol + (eo
(69)
Var(Gp) < 2 (Var [ao] Var [gal).
(70)
Step 3 Bounding the four variances. Lemma 2 with À B(a; 1) yields
K 2²
Var [go] s Σ ( (1 8)o oo
(71)
(1 + 88)mmin
with Omin
min;,0 mi)))
The same calculation (setting
B = 1) gives Var [gol 1
K K22 (2ammi))
Thus
Var(GH) 4K c²
(72)
Qmin
For the PHD weights choose À = 7(00 1). Because 7 2 1, Assumption Znsuress X > -1. Lemma
2
gives
K k²²
K K²
Var [90] $ (1 7)
Vargm [90l 1
(73)
(1 + 7)
Hence
4K c2q2
Var(Gr) S
(74)
82 (1 7)
Step 4 Ratio comparison. We can obtain:
Var(Gp)
7.2
72
<
I
(75)
Var Var(CH)
32 + +?)
a3
because (i) Omnin S. a: (ii) 8 2 1, and (iii) + 2 3 when 7 2 1. Multiplying through by Var(Gu) establishes the theorem.
Lemma 4 (HPD and PHD for Conic or Affine Exponential Family [44))
For distributions pla; 0p) and q(:; 0g) in the same exponential family with a conic or affine natural parameter space, the Hölder statistical pseudo-divergence and its generalization admit closed-form expressions:
D (p: :) 1 =(
1
F(30.) -F(e, 0g))
3
(76)
D (p: a)= 1 F(70,) F(r0,) -F(2
1
2 Op + 3 0o)
Here F(0) is the log-normalizer (cumulant generating function) of the exponential family.
Lemma 5 (Symmetric HPD and PHD for Conic Affine Exponential Family [44]). Under the same assumptions, the symmetric versions of these divergences also have flomed-form:
SH (p: = 1 + + F(a0,)) + F(B0.) - -F(Op + 0g).
2 F(o + F(80,)
(77)
(p: -
[ (000,)FF(r7,)) F(2 0 + 0,) F(3 3 6, + [.,)].
29

---

Set A = - 0r = E/= bi, a - + 1, a, > 0,y > 0. Via the Lemmas 4 S
We can
obtain:
(Di(((()): Dir(b)) 1
log + +a(A-A K)) Σ LoggIK+a(aa
log I(1 + a(aa 11)]
-=1
log [(K K +B(BB K))
Σ
log T(1 B(b. - 11)]
(78)
(gg((A- B [-K)+ +
Σ
log D(a: + b, 1).
i=1
D 0.7 (Dir(a): Dir(b)) _ log [(K' + 7(A ((A)))) -
Σ log 1(1 + 7(a - 1))
a1
k
log N(K + 7BB - K)) - Σ
log T(1 + +7(4 ++)bb-1))] 1))
+
3
(79)
- log I(K + 7A+ +B -K))
+
Σ
log I 1 + [a: 1) + [b, -01))))) 11)
==1
(Dir((a): Dir(b)) = _ 2 I(K + a K)) + log a(B K)))
+ [log I(K + 3(A K)) + logI + 3(B K)))
K
Σ llg I(1 + a(a - + tog + 1))
(80)
+ log T(1 + 3(a; - 1)) + log I[1 + B(b, 1)))
K
BogT(A+ + B - K) +
Σ
log I(a; + b; 10-11 1).
==1
1
SH (Dir(a) Dir(b))
log I(K + 7A1 K)) + log + + K))
2
K
log [11 + 7 1)) + log + 7(b:
7 - -1))}
(81)
Σ
I(K + + + B B-
2/2
m
K
Σ
log 1(1 + [a; - 1) + 20; 1
where ā and 3 a.
Assumption 3 (Smooth Dirichlet model). There exists O such that supo. [0)]) a
Assumption 4 (Admissible 7). If > 1, then a;(0) > 1 - 1/2 for all i and all 0. (No extra condition is needed when 0 < Y 1.)
Theorem 3 (Uniform variance domination of PHD over HPD). Let p be the target density and ga(z) Dir(z (0) the model, with (0) > 0 and ag (0) Σ ا= au(0), satisfying Assumptions 3 Let
30

---

1/a + 1/8= 1 with a, 3 > 0, and ry 2 1. Denote g0(2) = V log (0)). Then, for every 8,
22 ag ao(0)
Varp VoD (p: qo)l S
aB or () + Var, VoD (p qo)l
(82)
Moreover. ou+1 ag
1 and decreases monotonically as 7 + oc, implying a strict improvement for
any 7 1.
Lemma (Dirichlet score), I ~ Dir(a) with og = Eim as then
go (
Σ
log Zi - v(ax)) + vlao) Ogai
where &a is the digamma function. Lemma (Gradient forms of HPD and PHD). Let
A =- p B
= Dr, pppcc= o/: Ti =f 1/
1 7/8
40
T2 / p², T3 - 9/- =
(83)
Define weights (1) pçe A
w(2) 2g
((l) 1100g//0
11 w112)
2
Then:
C
VoD (p: qo) = (v [90] + tn l9ol. Vo a.y (p: :0))====))))))))) = 7 Egw [gol + Eggon lool) (84)
B
Lemma 8 (Second-moment bound under powered Dirichlet). Lei Z ~ Dir(a)), a; > 0, and X > -I Let pk v0(a;) v)(())). Then
I z (log Z, -1:) Dir(z a)) de [(oo)T(a; + A) 5xx
(85)
[(a:)[(aa +
where A((A) = wlan+ + 2) - 400000 + X) - 14 and v/ is the trigamma function. Moreover
2 (a, A)(au A)
(log Zi - m1) Dir(z | a.) d2 S
(86)
9x
Proof. We split the proof into four steps. Throughout, Z Dir(a), ao = ΣK as and go(2) Va log (2)).
Step Notation and two gradient decompositions. From Lemma define
Gи = VeD (p: ge) = t. [ge] +E, [ge]. Gp == VoD" (p: go) = B
Ega [ [go]]
(87)
Step 1: A variance upper bound for differences. For any random vectors X, Y, vY.
- -Y)
2 (VarXX+ VarY). Applying this Gи and Gp gives:
Var(GH) < 2(V arwen [go] + Varwen
(88)
Var(Gp) s 2 ( (Varga [oo] + Var Var (90))
2
(89)
Thus suffices to bound the four weighted variances of go-
Step 2: Bounding Var[go] under the four weights. Lemma yields
K
(0(2)= (z(2) ooan, (z(2) = log 2; - vb(s)) + v(oo).
(90)
1=1
Let a (0ga1,.... 0gak) T and u(2) = (u), ..., uk T2)) Then
go (2))) u(z) a. V yarrlga) a Cor(uja
(91)
Using Amnas (Cov (u]) < tr(CCvvul) Σ, Var[u.]. we get
K
Var[ga] S ||a|| Σ Var[ui] S k²² max Var[u:].
(92)
31

---

where Assumption 3 gives ||al I Ko?
Under each weight, the marginal of Ze is (powered) Dirichlet/Beta, so Lemma 8 applies with:
0
for w((),
B(a: - 1)
for w(())
A
r(a; 1)
for w((3)
7 (aa 1)
for w(1)
Its inequality form gives:
Var.. [us] 5
(93)
0,300
2
Varme)/uu 5
(94)
(1 + M) ap
2
Vargeeluus) S
(95)
(1 + ++, Coo
2
Vargw/lu] S
(96)
(1 + 7a; 00
Hence
Kr²
Varnets 5 Omin Okg
(97)
2Kc²
Varwcolgol S (1 + B) 3)
(98)
2K
Vangaulga [g0] S (1 + 70
(99)
2KT
Varmo [ga] S
(100)
(1 + am
Plugging into 88
Var(GH) < 4 2
(101)
Omin
8Ko222
Var(Gp) S BP + 7m
(102)
Step 3: Comparing bounds and simplifying constants. Divide (002) by (0O1:
Var(Gr) Var(GH)
8K722²
4Kr²
2n²
(103)
+1 ag amin
5² +
We now relate this to the desired factor. Because 1/a
1/3 = 1, we have 3 2 1, a S 1, hence
a < B. Moreover, for any ag > and 7 2 1.
2 (L 77)
1 a0 +1
(104)
ag3 0g +
(A short algebraic verification is given in Appendix
A.2.) Therefore,
Var
72 og +1
(105)
Var(Gn)
0,3 Gg +
and multiplying both sides by Var(Gn) yields 82 Step 4: Tightness and monotonicity. When
y yo. both gradients vanish in expectation, and the ratio approaches the constant factor (tight up to
trigamma bounds). The factor (oo + 1)/(dg + +7) < 1 is decreasing in 7 and tends to 0 as 7 xc.
giving strict improvement for any 7 1.
32

---

B SDE Formulation
Data sample: To (z) Time: [] (or [0, T] in other conventions). State: æt E R.
Jezo: standard Wiener process. Score function: V log Pt (z), the gradient of the log-density of
Ft. Forward Diffusion SDE The forward (noising) process described by an Itō SDE 22
dx; [ l) + s(() dW.,
To
Pdata-
(106)
The corresponding density pc(z) evolves according to the Bookker-Plancee equation
(xp)(x) - V (f(z, + =0(1)² (x))(x).
(107)
Reverse- SDE (Generation Process) By Anderson's time-reversal theorem (1982) 65 the time-reversed process t 1 - is also SDE:
dæ = fz t) v( log (z)) dt + g(t) dW,,
(108)
where W, is another Wiener process (independent of W,), and dt < 0 when integrating backward in time. In practice, we reparameterize time to integrate forward (e.g., T = t) and approximate the score with neural network s0 s V. Pz()..
Forward Simulation (Training-Tinn Noise Sampling) Using Euler Mardurmma
At+At = Zt + fa t) At + g(t) e~ N(0,).
(109)
Reverse- Tiverse-Time Generation Discretizing gives
It At = 2z + [[(rr, () g(1)² so(21, 0) AL+ (01l e~ (110)
SDE for Proper Hölder- Dirichlet Diffusion Let Tg (ama(x) and 2, E AK- {a RK 1 1} for 1]. We denote by {W1):20 a K-dimensional Wiener process. For a time-varying Dirichlet schedule (xt) = (a) (t)) with att) > 0 and an(() = ΣK az(t), define the simplex projection covariance
E(x) Diag(æ) xx. RR
(111)
Let (x) = V log (z)) be the true score of the forward marginal (x)), approximated in practice by a neural network (x, trained with the PHK objective.
Forward (Noise-Injection) SDE We adopt a Tright-Fisher/Drrihhette
diffusion to ensure z
remains on AK-1. For positive noise schedule 3(t) 0:
dr: 2 [()))
Nz dt + VB() ((:) au(t)
a(t)
(112)
To ~ Pdata-
The drift pulls v, toward the instantaneous Dirichlet mean a<()/oo(!), while the diffusion term 1/2 keeps trajectories inside the simplex.
Reverse (Generation) SDE By Anderson's 's time-reversal theorem, the reverse-time process / satisfies
(t) dx, = 2 (t xx) ) - ((t))) dt + B(t I(x,)
(t)
(113)
where W, is another Wiener process (independent of the forward one) and di < 0 when integrating backward. In practice we reparameterize time by 7 1 to integrate forward in
Discrete- Riscrete-Time Euler-Maruyama Schemes For small step At:
Forward simulation (training-time noising):
(t)
Z1+ = It + 2 3()
2t At B(tA)t
2() 122,,
e0,(1,
(114)
(1)
33

---

Reverse sampling (generation):
1 (t) 2 8())
x1 - - (()))(())))000 1) At + v 3())AA)) 1(,,) 6. (115)
Ji-A: Jz
+
(t)
Higher-order SDE/ODE solvers (e.g., Heun, DPM-Solver, UniPC) can be used place of Euler steps.
Score Learning with Proper Hölder-Kullback (PHK) Objective PHK combines Proper Hölder Divergence (PHD) [44] and KL-based terms to stabilize and symmetrize training:
PLIK Dd, (gt qo) + (1 - 8 + + Ku (l L0
6,e [0, 1],
(116)
where oi denotes the "teacher" (true or target) conditional and qe is the Dirichlet model induced by the network. Minimizing (16) yields sæ(x, t) st (z), which then plugged into (113).
C HPD Impropriety Counter Example
The zero condition for the Hölder projective divergence (HPD) is
p(x) (x))
D (p: = =
a
(117)
p g
which is equivalent to
p(x) =c(x)) for some 0.
(118)
Corrected counter-example (discrete case). Let p = (pI..... .p. be any non-uniform probability vector and choose (thus 3 a/(a 1)). Define
pi
4:
1,..,, K.
(119)
2==1 Pj
Then * (unless p is uniform), yet
p. p.
(120)
Pi (a-1)3
of¹
(121)
Σ Σ Dj
10-11
j=1 Pj 1-1))3
Hence
Σ p 1
K
3
pd = c with c
(122)
j=1
and by the equality condition of Hölder's inequality the numerator and denominator of HPD are identical, yielding DH (p q) 0 although p L q. This establishes the impropriety (failure of identity of indiscernibles) of HPD stated in [44]
Numerical illustration. For example, let 1.5 and p (0.7,0.3) Then
x (p) 2--1 P2 = (0.7 0.3%.5).
and D (p: q) evaluates to 0 up to machine precision.
34

---

D Aditional Comparison
As shown in Table 3 PHK-Dir represents 12 % improvement DDPM's 3.17 2 When B=128 and NFE=1000, As shown in Table 4 PHK-Dir represents 18.98 FID on the CVUSA 66 dataset.
Table 3: FID scores for DDPM and PHK-optimized Dirichlet Diffusion on the CIFAR-I0 image dataset with varying NFEs under several different mini-batch size B.
Model
B
NFE=20
50
200
500
1000
2000
PHK-Dir 512
15.92
5.87
3.20
2.92
2.84
2.82
PHK-Dir
288
16.53
6.15
3.29
2.99
2.91
2.81
PHK-Di
128
16.85
6.14
3.25
2.83
2.79
2.78
DDPM
128
3.17
Table 4: FID and LPIPS scores for different models on the CVUSA (66 dataset.
Model
FID (L)
LPIPS (L)
X-Seq 66
161.16
0.706
SeIGAN 68
116.57
0.742
CUT [69
72.83
0.687
CDTE 70
122.84
0.694
Aerial Diff
136. 18
0.855
Instr-p2p 72
38.01
0.697
ControlNet 73
32.45
0.650
121-Turbo 774
77.95
0.685
GPG2A [75
58,80
0,691
BEV [76
29.18
0.635
PHK-Dir(20% data)
21.80
0.599
E
Compositional Data
As shown in Figure 5 Dirichlet ELBO and Dirichlet PHK achieve better performance in the positional data. Dirichlet ELBO is better than Gauss ELBO. Dirichlet PHK is better than Dirichlet ELBO.
Geuss ELBO ELBO
Dirichett ELBO
Dir chiet, PHK
Empinial Enginicl/requene Frequency
Empiitaall Eunghticl Prequencyy
ampitial Mequucncy
0.3
0.3
0.3
0.2
0.0
0.0
00
0.7
0.8
Figure 5: Comparison of true probability mass function (PMF) and empirical PMFs of three meth- ods-Gauss ELBO. Dirichlet ELBO, and Dirichlet PHK computed Each true PMF is depicted with red square. Each empirical PMF is computed based on 100k data generated by the model trained after 400k iterations with 200 Numbers of Function Evaluations (NFE).
F Hyperparameter Settings
In the experiments, the neural network function (So in equations) is implemented as UNet architec- ture, as typical modern diffusion models 2 we utilize the parameterization of Beta Diffusion
[16] as the code base.
35

---

We demonstrate the intuition behind the setting of model parameters, including Shit, Seale, C1. a, and B. Given the limitations of computational resources, the meticulous tuning these model parameters is left for future work.
For the diffusion concentration parameter v/ value of 7) provides greater discriminative power among different pixel values but requires more discretization steps during sampling, leading to slower training speeds.
moderately high value of 1000. A larger
We set the linear scaling and shifting parameters as Scale = 0.39 and Snn 0.60, and sigmoid- based schedule parameters cu 10 and C1 13.. These settings are based on the work of Kingma et al. and Zhou et al. [66
We set the shape parameters of the Proper Hölder Divergence as 2 and - 2. This choice is motivated by an intuitive understanding of the Hölder Divergence, where a smaller a would lead to the loss of symmetry.
As shown in Table 7 the training-tin time cost of KLUB substantially more expensive. When other conditions identical and B 512, KLUB incurs approximately 15 % additional training- training-time time cost compared with PHD.
Table 5: FID scores for negative ELBO, KLUB, PHD and PHK-optimized Dirichlet Diffusion on the CIFAR-10 image dataset with varying NFEs under several different combinations of concentration parameter y and mini batch size B.
Loss
7) 10-4
B
NFE
50
200
500
1000
2000
ELBO
512
15.81
6,53
4.49
4.33
4.28
4.31
ELBO
288
16.02
6.65
4.68
4.42
4.42
4.39
KLUB
512
16.62
6.16
3.46
3.39
3.32
3.25
KLUB
288
16.35
6.23
3.58
3,46
3,40
3.21
PHD
512
16.84
6.48
3.85
3.72
3.52
3.51
PHD
288
16.95
6.93
4.43
4.23
4.19
4.21
PHK
512
15.92
5.87
3.20
2.92
2.84
2.82
PHK
512
17.08
6.60
3.55
3.09
3.14
PHK
512
20.67
9,67
4.54
4.58
PHK
16.53
6.15
3.29
2.99
2.91
PHK
288
6.71
3.61
3.09
3.03
3.02
PHK
0.1
288
20.33
9.56
5.71
4.77
4.75
4.72
PHK
128
16.85
6.14
3.25
2.83
2.79
2.78
PHK
0.5
128
16.32
6.40
3.57
3.23
3.09
3.06
PHK
0.1
128
20.22
9.73
6.00
5.00
4.93
5.00
Table 6: Frechet Inception Distance (FID) scores for PHK -optimpteeee Dirichlet Diffusion on the unconditional CIFAR- 10 dataset with varying Numbers of Function Evaluations (NFE) under several different combinations of the key loss weight coefficients 5 and with B 288.
NFE=20
50
200
500
1000
2000
6-0.5
0=1
16.62
6.38
3.78
3.56
3.42
3.35
8=0.5
-=
16.14
6.17
3.66
3.43
3.34
3.29
8=0.5
<=0.1
16.71
6.16
3.25
2.92
2.82
2.84
8=0.5
c=0.15
16.74
6.06
3.20
3.00
2.86
2.78
8=0.5
<=0.2
16.53
6.15
3.29
2.99
2.91
2.81
8=0.5
c=0.25
16.58
6.25
3.47
3.12
2.98
2.93
8=0.5
e=0.5
16.07
6.23
3.61
3.39
3.35
3.31
G Training Details
In the course of our model training, we employed the Adam optimization algorithm with specific
parameter settings: the learning rate 1r = 5 x 10-4 the exponential decay rate for the first moment estimates 3 0.9, the exponential decay rate for the second moment estimates B2 = 0.999, and the numerical stability parameter
10
36

---

For the purpose of training the model, we utilized 200 million images. Specifically, we leveraged four Nvidia L40s GPUs for computation. and with a batch size = 512, processing 1000 images of size 32 x 32 x 3 required approximately 1.16 seconds. The processing of 200 million CIFAR-10 images took approximately 64 hours.
During the training process, we saved a checkpoint every 25,000 steps and selected the model with the best performance based the Fréchet Inception Distance (FID) scores obtained from these checkpoints. We assessed the FID scores using 50,000 samples, practice consistent with previous research endeavors.
Table 7: The training time cost of the KLUB, PHD, and PHK-optimized Dirichlet Diffusion models on unconditional CIFAR- dataset across two mini-batch sizes B with four Nvidia L40s GPUs.
B
KLUB
PHD
PIIK
512
70h
61h
64h
288
73h
64h
68h
Table 8: The training time cost of the PHK-optimized Dirichlet Diffusion on the CIFAR10-32x33 AFHQ-64x66 FFHQ-64 x 64 datasets across two mini-batch sizes with four Nvidia L40s GPUs.
B
CIFARI0-32x32
AFHQ-64x66
FTHQ-64x64
512
64 hours
10 days
11 days
288
68 hours
11 days
13 days
Table 9: The Sampling time cost of the PHK-optimized Dirichlet Diffusion on the unconditional CIFARIO 10 at varying Numbers of Function Evaluations (NFE) with two or four Nvidia L40s GPUs.
Nvidia L40s GPUs
NFE = 20
50
200
500
1000
2000
2
11m
21m
Ih 13m
3h 2m
6h 4m
12h 7m
4
7m
10m
37m
1h 33m
3h 5m
6h 9m
H
Broader Impacts
This study's 's effective diffusion model is capable of generating images required for practical appli- cations and further developing based on this foundational model, which will have a positive social impact related fields of work. Technologies such as virtual try-on and video generation, formed based on diffusion models, provide more convenient development space for people's 's social lives.
This research may also raise concerns about the potential negative social impacts when training on maliciously curated image datasets, which is shared challenge in the field of generative models. is crucial to consider how to use these models responsibly to improve society, prepare comprehensive contingency plans, and mitigate any potential adverse consequences.
I Uncurated Samples
9 show more uncurated samples from our trained models on CIFAR-10, FFHQ, AFHQ and CV A.VUA..TrOUU these extensive results, the model's s capacity to generate diverse and high-fidelit images without the need for manual selection management can be clearly elucidated. This feature not only underscores the model's efficacy image generation tasks but also further corroborates its generalization and adaptability complex datasets.
Figures 6
37

---

2
Figure 6: Additional uncurated samples generated by model trained CIFAR-10 10 dataset.
38

---

Figure 7: Additional uncurated samples generated by our model trained on FFHQ dataset.
39

---

Figure 8: Additional uncurated samples generated by model trained on AFHQ dataset.
40

---

Figure 9: Additional uncurated samples generated by our model trained on CVUSA dataset.
41


Reviews:
[{"review_id": "UXNd8mtRdQ", "reviewer": "Reviewer_boWF", "summary": "This paper proposes a Dirichlet-based diffusion model that replaces Gaussian noise with the Dirichlet distribution, which better suits probability simplex data. To support this, the authors introduce two closed-form divergences—Proper Hölder Divergence (PHD) and Proper Hölder–Kullback Divergence (PHK), significantly improving optimization efficiency and model flexibility. The method achieves strong results on CIFAR-10, reaching FID = 2.81, outperforming existing non-Gaussian diffusion models.", "questions": "1. What does the function F(θ) in Equations (9) and (12) represent? Please clarify its definition and theoretical background in the paper.\n2. Why do the intermediate results in the forward diffusion process (Figure 1) closely resemble the original image, while those in the reverse process (Figure 2) appear largely noisy or masked? Does this imply an asymmetry between training and inference?\n3. How were the hyperparameters Shift, Scale, and π selected? What effect do they have on performance? Has any ablation or sensitivity analysis been conducted?\n4. Has the model been evaluated on class-conditional, text-to-image, or high-resolution datasets ? If not, what are the key challenges?\n5. While the paper claims that the Dirichlet distribution is better suited for high-dimensional data, it lacks a deeper theoretical or empirical analysis comparing it to Gaussian, Beta, and Poisson distributions under high-dimensional image modeling settings. Please provide additional discussion or analysis to substantiate this claim.", "limitations": "1. It appears sensitive to hyperparameter choices, such as Dirichlet concentration, data shift/scale factors, and PHK loss weights, but only a limited range has been explored, leaving its robustness under diverse settings unclear.\n2. The training cost is notably high: processing 200 million CIFAR-10 images takes around 64 hours on four Nvidia L40S GPUs, which may hinder scalability to larger datasets or more complex tasks.", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths: \n1. The Dirichlet-based diffusion offers significant advantages in modeling data distributions on high-dimensional simplices and naturally extends from low-dimensional to high-dimensional diffusion processes. This extension makes the Dirichlet distribution more flexible and efficient in dealing with high-dimensional data. \n2. The proposed PHD and PHK divergences are symmetric, differentiable, and analytically tractable, offering stable and efficient optimization during training. \n3. The method achieves superior empirical performance, outperforming existing approaches in unconditional image generation based on the FID metric.\n\nWeaknesses： \n1. The derivations of Equations (4), (5), and (6) are not provided in detail in the paper. The function F(θ) used in Equations (9) and (12), which may corresponds to the log-normalization constant of the Dirichlet distribution, is not clearly defined in the main text. \n2. In addition, the asymmetry between the forward and reverse processes is not sufficiently explained—Figure 1 shows that the intermediate states in the forward process preserve much of the original structure, whereas the reverse process in Figure 2 appears visually less structured in intermediate steps compared to the forward process, which may indicate an inconsistency between training and inference. \n3. The choice of key hyperparameters such as Shift = 0.6, Scale = 0.39, and π = 0.95 is not explained, nor is their impact analyzed. \n4. The experimental scope is also limited—only unconditional generation on CIFAR-10 is evaluated, with no validation on higher-resolution datasets, conditional generation, or other tasks such as image editing and restoration. Furthermore, the evaluation relies solely on the FID metric, lacking additional measures such as IS to provide a more comprehensive assessment of generation quality.", "was_revised": false, "final_justification": "", "rebuttal": "Thank you profoundly for the constructive comments. Below, we provide point-by-point responses to specific comment.\n\nQuestion1:\n\nThe log-normalizer function F ($\\theta$) in Equations (9) and (12) is a strictly convex function also called cumulant generating function. For reference, please see Ref[42]: *Frank Nielsen,et al. On hölder projective divergences.456 Entropy.* We will clarify it more prominently in the revised version.\n$$\nF(\\theta)=\\sum_{i=1}^K\\log\\Gamma(\\theta_i)-\\log\\Gamma (\\sum_{i=1}^K\\theta_i)\n$$\nWhere $\\Gamma$ is the Gamma function, $K$ is the number of parameters. $\\theta_i$ is the parameter of Dirichlet distribution, typically represented as $\\alpha_i - 1$, where $\\alpha_i$ are the original parameters of Dirichlet distribution.\n\nFurthermore, we will label the sources of Equations(4-6), which are respectively the PDF, expectation and variance of Dirichlet distribution, and the form of Dirichlet function expressed using the Gamma function. For reference, please see Ref[59]:*Kai Wang Ng,et al.Dirichlet and Related Distributions: Theory, Methods, and Applications.*\n\nQuestion2:\n\nThe differences in appearance of intermediate results are due to the implementation methods of the diffusion model during Forward and Reverse Diffusion phases. During the Forward phase, the model is required to add noise to transform the original image into a completely noisy state. This process is deterministic and part of the model training, where the model learns to add noise in a structured manner so that the process can be reversed later. The intermediate results are indistinguishable from the original image to the naked eye because the noise added at the beginning of the learning process is relatively small.\n\nIn Reverse phase, the model needs to remove the noise from the completely noisy state and restore the original image. This is an inferential process and also an inverse process. The intermediate results in Reverse Diffusion appear noisy because the model is actively trying to remove the noise. At each step, the model's prediction of the noise to be removed may not be perfect, leading to intermediate images that remain noisy or partially masked. This is a natural part of the process as the model gradually refines its predictions. For reference, please see Ref[4]: *Jonathan Ho,et al.Denoising diffusion probabilistic models.NeurIPS 2020.*\n\nThis indeed implies a certain asymmetry between training and inference. Such asymmetry can also be observed in works such as VDM, Beta Diffusion, Soft Diffusion, etc. For references, please see Ref[6,47,49,...].This does not imply a problem or error but rather highlights the challenge of accurately predicting and removing noise in the reverse process. The effectiveness of the model in the reverse process is crucial for applications such as image generation and restoration, where the goal is to recover or generate high-quality images from noisy or incomplete data.\n\nQuestion3:\n\nIn Appendix B 'Hyperparameter Settings', we elucidate the rationale: 'These settings are predicated on the research conducted by Kingma et al.[6] and Zhou et al.[41]'. The appendix G of Ref [41] provides a comprehensive exposition on the selection and analysis of the hyperparameters in question.*Mingyuan Zhou,et al.Beta Diffusion.NeurIPS 2023*\n\nQuestion4:\n\nAs detailed in Supplement 3, the model achieves an FID score of 2.43 on the class-conditional CIFAR-10 dataset.\n\nThe model has not been evaluated on text-to-image or higher-resolution datasets. Most related works in the field—including Learning to Jump, Beta Diffusion, Blurring Diffusion, D3PM, DiffuEBM, and others—conduct their evaluations and comparisons on unconditional CIFAR-10.\n\nPrimary challenge: only 4 NVIDIA L40 GPUs are available for most time in this work. As detailed in Supplement 4, one complete CIFAR-10 training run requires roughly 68 hours, whereas AFHQ-64×64 would demand about 11 days, and FFHQ-256×256 exceeds the GPUs’ memory capacity. As detailed in Supplement 5, sampling is also time-consuming.Under these resource constraints and the need for extensive hyper-parameter tuning, the available time only permitted attempts and experiments on CIFAR-10.\n\nQuestion5:\n\nWe wish to emphasize that Dirichlet distribution is more suitable for high dimensional simplex data.For reference, Avdeyev et al. highlighted this advantage and introduced a Dirichlet Score Model at ICML 2023. *Avdeyev,et al.Dirichlet diffusion score model for biological sequence generation.ICML,2023.*\n\nAs detailed in Supplement 6, we have supplemented experiments on discrete compositional data, which is naturally High Dimensional simplex-structured. It shows that, Dirichlet ELBO is better than Gauss ELBO. Dirichlet PHK is better than Dirichlet ELBO. Moreover, §4.1 'Comparison study' also covers comparisons with these other distributions, including Gaussian, deterministic, categorical, beta, and Poisson.\n\nSupplement1:\n\nThe Frechet Inception Distance (FID) scores for -ELBO, KLUB, PHD, and PHK-optimized Dirichlet Diffusion on unconditional CIFAR-10 with varying Numbers of Function Evaluations (NFE) under several different combinations of concentration parameter $\\eta$ and mini-batch size $B$:\n\n\\begin{array}{ccc|cccc}\n\\hline\nLoss&\\eta \\times 10^{-4}&B&NFE = 20&50&200&500&1000&2000 \\\\\\\\\n\\hline\n-ELBO&1&512&15.81&6.53&4.49&4.33&4.28&4.31\\\\\\\\\n-ELBO&1&288&16.02&6.65&4.68&4.42&4.42&4.39\\\\\\\\\nKLUB&1&512&16.62&6.16&3.46&3.39&3.32&3.25\\\\\\\\\nKLUB&1&288&16.35&6.23&3.58&3.46&3.40&3.21\\\\\\\\\nPHD&1&512&16.84&6.48&3.85&3.72&3.52&3.51\\\\\\\\\nPHD&1&288&16.95&6.93&4.43&4.23&4.19&4.21\\\\\\\\\nPHK&1&512&15.92&5.87&3.20&2.92&2.84&2.82\\\\\\\\\nPHK&0.5&512&17.08&6.60&3.55&3.09&3.10&3.14\\\\\\\\\nPHK&0.1&512&20.67&9.67&5.66&4.54&4.58&4.53\\\\\\\\\nPHK&1&288&16.53&6.15&3.29&2.99&2.91&2.81\\\\\\\\\nPHK&0.5&288&16.94&6.71&3.61&3.09&3.03&3.02\\\\\\\\\nPHK&0.1&288&20.33&9.56&5.71&4.77&4.75&4.72\\\\\\\\\nPHK&1&128&16.85&6.14&3.25&2.83&2.79&2.78\\\\\\\\\nPHK&0.5&128&16.32&6.40&3.57&3.23&3.09&3.06\\\\\\\\\nPHK&0.1&128&20.22&9.73&6.00&5.00&4.93&5.00\\\\\\\\\n\\hline\n\\end{array}\n\nSupplement2:\n\nThe training-time cost of KLUB is substantially more expensive. When all other conditions are identical and $B=512$, KLUB incurs approximately a 15 \\% additional training-time cost compared with PHD. The training time cost of KLUB, PHD, and PHK-optimized Dirichlet Diffusion on unconditional CIFAR-10 across two mini-batch sizes $B$ with 4 Nvidia L40s GPUs:\n\\begin{array}{c|cc}\n\\hline\nB&KLUB&PHD&PHK\\\\\\\\\n\\hline\n512&70h&61h&64h\\\\\\\\\n288&73h&64h&68h\\\\\\\\\n\\hline\n\\end{array}\n\nSupplement3:\n\nThe FID scores for PHK-optimized Dirichlet Diffusion on class-conditional CIFAR-10 with varying NFE:\n\\begin{array}{ccc|cccccccc}\n\\hline\nLoss&cond/uncond&B&NFE=20&50&200&500&1000&2000\\\\\\\\\n\\hline\nPHK&cond&512&13.21&5.45&2.92&2.67&2.52&2.43\\\\\\\\\n\\hline\n\\end{array}\n\nSupplement4:\n\nThe training time cost of PHK-optimized Dirichlet Diffusion models on unconditional CIFAR10-32x32, AFHQ-64x64, FFHQ-64x64 image datasets across two mini-batch sizes $B$ with 4 Nvidia L40s GPUs:\n\\begin{array}{c|cc}\n\\hline\nB&CIFAR10&AFHQ&FFHQ\\\\\\\\\n\\hline\n512&64 hours&10 days&11 days\\\\\\\\\n288&68 hours&11 days&13 days\\\\\\\\\n\\hline\n\\end{array}\n\nSupplement5:\n\nThe Sampling time cost of PHK-optimized Dirichlet Diffusion on unconditional CIFAR10 at varying NFE with two or 4 Nvidia L40s GPUs:\n\\begin{array}{c|cccccc}\n\\hline\nNvidia L40s GPUs&NFE=20&50&200&500&1000&2000\\\\\\\\\n\\hline\n2&11m&21m&1h 13m&3h 2m&6h 4m&12h 7m\\\\\\\\\n4&7m&10m&37m&1h 33m&3h 5m&6h 9m\\\\\\\\\n\\hline\n\\end{array}\n\nSupplement6:\n\nIn the revised version , we have supplemented experiments on the discrete compositional data, which is naturally High Dimensional simplex-structured, and provide Figure: 'Comparison of true probability mass function (PMF) and empirical PMFs of three methods—Gauss ELBO, Dirichlet ELBO, and Dirichlet PHK-computed. Each empirical PMF is depicted with solid dots.' Each empirical PMF is computed based on 100k data generated by the model trained after 400k iterations with 200 Numbers of Function Evaluations (NFE).\n\nBecause Rebuttal does not permit links or images, we can only describe the results here: In Gauss ELBO, the average discrepancy between empirical PMF and true PMF is approximately 11.02\\%. In Dirichlet ELBO, the average discrepancy between empirical PMF and true PMF is approximately 9.81\\%. In Dirichlet PHK, the average discrepancy between empirical PMF and true PMF is approximately 2.17\\%. The lower discrepancy between empirical PMF and true PMF signifies superior generation quality.\n\nIt shows that, Dirichlet ELBO and Dirichlet PHK achieve better performance in the task where data is naturally High Dimensional simplex-structured. Dirichlet ELBO is better than Gauss ELBO. Dirichlet PHK is better than Dirichlet ELBO.The figure that includes both empirical PMF and true PMF would more visually demonstrate this result; however, the rebuttal page does not permit links or images.\n\nSupplement7:\n\nThe FID scores for PHK-optimized Dirichlet Diffusion on unconditional CIFAR10 with varying NFE under different combinations of the key loss weight coefficients $\\delta$ and $\\epsilon$ with $B=288$:\n\\begin{array}{cc|cccccc}\n\\hline\n\\delta&\\epsilon&NFE=20&50&200&500&1000&2000\\\\\\\\\n\\hline\n0.5&-0.1&16.62&6.38&3.78&3.56&3.42&3.35\\\\\\\\\n0.5&0&16.14&6.17&3.66&3.43&3.34&3.29\\\\\\\\\n0.5&0.1&16.71&6.16&3.25&2.92&2.82&2.84\\\\\\\\\n0.5&0.15&16.74&6.06&3.20&3.00&2.86&2.78\\\\\\\\\n0.5&0.2&16.53&6.15&3.29&2.99&2.91&2.81\\\\\\\\\n0.5&0.25&16.58&6.25&3.47&3.12&2.98&2.93\\\\\\\\\n0.5&0.5&16.07&6.23&3.61&3.39&3.35&3.31\\\\\\\\\n\\hline\n\\end{array}\n\nWe sincerely appreciate the time and efforts of all reviewers and ACs, and believe that the additional experiments, analysis, and explanations significantly improve the quality of our mission. We hope that this provides sufficient reasons to raise the score."}, {"review_id": "ygTrZFHrAA", "reviewer": "Reviewer_W73X", "summary": "The paper suggests using the Dirichlet distribution on simplices as a replacement for the standard Gaussian/Poisson/categorical/Beta noises in high-dimensional diffusion models. Inference minimizes a proper variant of the Holder-divergence (weighted with the standard KL divergence), which is claimed to have a natural form in this setting. The sample quality is demonstrated on a  real-world dataset.", "questions": "[Q1] Motivation: You write in the introduction that limitations to common distribution significantly restrict their effectiveness in modeling specific data types prevalent in practical scenarios. Could you give a few examples for such data types and explain why there are not well modeled by standard distributions?\n\n[Q2] Introduction: You write that the computational complexity of the KL divergence increases with the dimension? Why is this different from the standard L2 loss or the proposed Holder divergence? It seems that both have a similar dependence on the dimension. \n\n[Q3] Background: It is common in many learning methods that the Gaussian case can be analytically analyzed, whereas the method itself is more broadly applicable. In what sense your method “allows us to challenge the extant theoretical boundaries”. Is there a specific sense in which the proposed method goes beyond theory, beyond the common situation?\n\n[Q4] Line 95: Do you mean in terms of the Gamma function rather than distribution?\n\n[Q5] In (3), how the conditional distribution $q$ is defined?\n\n[Q6] Lines 107-109: The discussion on “distributed training” is out of context. Also, how equation (6) is a logical continuation of this sentence? This is completely out of place. \n\n[Q7] In line 113 you write that \"By scheduling .... $x_0$ can be nearly noise-free, while $z_T$ approaches a uniform distribution\". Can you support/reference this claim?\n\n[Q8] I could not realize the difference between $x_0$ and $z_0$, are they identical? since, if I get it right, $z_t$ should lie on the simplex, while $x_0$ is a sample (e.g. a natural image). Please clarify this point.\n\n[Q9] In (14), how the KLUB loss is defined? \n\n[Q10] Figure 1 promotes natural images as the application for Dirichlet high dimensional modeling. Do you have any evidence that natural figures are well modeled by Dirichlet distribution?\n\n[Q11] You write that the forward diffusion Dirichlet process involves masking, compared to additive noise in Gaussian diffusion. Could you mathematically explain why this is the case for the Dirichlet distribution? \n\n[Q12] Figures 1 and 2, are both difficult to inspect (as each contain 9 examples), and fill up a lot of space, which could be used to improve the exposition. \n\n[Q13] The sentence “High resolution images require more computational resources.” in line 153 is both trivial and out of context. \n\n[Q14] Figure 3: It is impossible to obtain any meaningful subjective assessment of the quality of the images from this figure. Consider finding a better way to display these images, which will make the improvement proposed by your method evident. Similar things are said regarding Figure 4. \n\n[Q15] Many of the terms and definitions only appear in the appendix to save space. First, as said, the figures in the paper take up a lot of unnecessary space, which could be utilized, e.g., for proper definitions in the body of the paper. Page 3 also contains various unused definitions (Beta distribution, e.g., which is immediately generalized to a Dirichlet). Second, it would be good to refer the reader to the appendix for those essential terms. \n\n[Q16] Typos: Line 76: Spaces before citations. Line 80: capital letters in “In this work, We explore”", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n\n\n[S1] The suggested diffusion model is interesting and novel.  \n\n[S2] The method be applicable to datasets for which other modeling assumption fail. \n\n**Weaknesses**\n\n[W1] The work is purely empirical, with no theoretical contribution. While this is acceptable, as such, the empirical evidence for the proposed method supremacy in high-dim generation should be much better. First, experiments on multiple additional datasets, from *various* domains should have been made, with FID score computed. Otherwise, the single data-set example appear as cherry-picking. Second, it is difficult to make (see also below in questions). \n\n[W2] Methodology: It is claimed that the Dirichlet modeling offers a significant advantage for some types of data. However, no concrete example for such dataset is given, and the only experiment is on standard images. I don't think that there is an evidence that a Dirichlet distribution is a good model for natural images. \n\n[W3] Methodology: It is well known that in moderate dimensions, and even more so in high dimensions, the multinomial distribution of a random vector is close to an *independent* Poisson distribution over its components. Accordingly, the Dirichlet prior could be replaced by a Poisson prior for each of the components. Given this observation, the viability of the proposed method needs stronger justification.", "was_revised": true, "final_justification": "I accept the good results on the additional experiments and therefore raised my score. The final score is justified based on the original submission (to be fair with other authors), and my general impression of the novelty, significance and competitiveness of the paper.", "rebuttal": "Thank you profoundly for the constructive comments. Below, we provide point-by-point responses to specific comment.\n\nQ1:\n\nSuch data includes categorical image, recommender rating, and portfolio weights, which can satisfy the simplex constraint. For instance, Avdeyev et al. highlighted this advantage and introduced Dirichlet Score Model at ICML 2023.*Avdeyev, et al.Dirichlet diffusion score model for biological sequence generation.ICML 2023.*\n\nWe wish to emphasize that Dirichlet is more suitable for such data. As detailed in Supplement 2, we have supplemented experiments on compositional data, which is naturally simplex-structured. It shows that, Dirichlet ELBO is better than Gauss ELBO. Dirichlet PHK is better than Dirichlet ELBO. \n\nQ2:\n\nIn the paper, we will replace line 33-34 with 'Kullback–Leibler divergence exhibits drawbacks such as asymmetry and computational complexity.', and remove the sentence 'These problems become more pronounced when handling high-dimensional and intricate distributions.'\n\nThe training-time cost of KLUB is substantially more expensive. A training cost comparison is provided in Supplement 3. As shown in Supplement 3, when all conditions are identical and B=512, KLUB incurs approximately 15 \\% additional training time cost compared with PHD.\n\nQ3:\n\nWe will replace line 86-87 with 'enables us to explore alternative distributions and divergences, thereby unlocking new pathways for generative modeling.'. We wish to emphasize the novelty of this work. As noted, compared with the Gaussian, the distributional theory and divergence methodology of this work are innovative and competitive. \n\nQ4:\n\nTypo acknowledged. The wording has been updated to 'Gamma function'.\n\nQ5:\n\n$$\nq(z_s|z_t,x_0)=\\frac{1}{1-z_t}Dir(\\frac{z_s-z_t}{1-z_t};\\eta(\\alpha_s-\\alpha_t)x_0,...,\\eta\\frac1{K-1}(1-\\alpha_sx_0)).\n$$\nWe will updated it in the paper.\n\nQ6:\n\nWe will remove this statement and provide more appropriate logic before Equation(6). We will replace line 107-109 with following statement:\n\nIt is worth noting that Dirichlet function can also be expressed in terms of Gamma function. Compared to Beta function, it extends to K dimensions in similar form:\n$$\n\\rm{Dir}(\\alpha)=\\frac{\\prod_{k=1}^K\\Gamma(\\alpha_k)}{\\Gamma(\\sum_{k=1}^K\\alpha_k)}.\n$$\nQ7:\n\nThe statement combines two well‑established facts, which we now reference separately: By choosing a monotonically decreasing schedule $\\alpha_t$ with $\\alpha_0\\approx 1$ (so forward variance at $t=0$ is negligible and $x_0$ remains essentially noise-free) and $\\alpha_T\\to0$, our forward kernel $q(z_t|x_0)$ converges to $Dir(1)$,i.e. a uniform distribution on the simplex. \n\nThis start/end behavior is standard in diffusion models.*Jascha,et al.Deep Unsupervised Learning using Nonequilibrium Thermodynamics.ICML 2015.Ho J, et al.Denoising diffusion probabilistic models.NeurIPS 2020. Kingma D, et al.Variational diffusion models.NeurIPS 2021.Tero Karras, et al.Elucidating the Design Space of Diffusion-Based Generative Models.NeurIPS 2022.*It is also explicitly adopted in non-Gaussian variants *Zhou M, et al.Beta diffusion.NeurIPS 2023.Austin J, et al.Structured denoising diffusion models in discrete state-spaces.NeurIPS 2021.Hannes Stärk, et al.Dirichlet Flow Matching with Applications to DNA Sequence Design.ICML 2024*\n\nThe uniformity of $Dir(1)$ is a classical result *Aitchison, et al. The Statistical Analysis of Compositional Data. Chapman and Hall, 1986.*\n\nQ8:\n\nNo, $x_0$ and $z_0$ are not exactly identical. $x_0$ denotes the original image, and $z_0$ is the diffusion initial state after preprocessing, which lies on the $K$-simplex. We will clarify this more clearly in the paper.\n\nQ9:\n\n$$ L_\\theta^{KLUB}=\\mathbb{E} _{z_t,z_0}[KL(q(z_s|z_t,x_0)||q(z_s|z_t,\\hat x_0))].$$\n*Mingyuan Zhou,et al.Beta Diffusion.NeurIPS 2023.*\n\nQ10:\n\nQualitative evidence: Please refer to Figure 3. Comparing sub-images in the first row, third and fourth columns of Figures 3(a) and 3(b)—corresponding to “car” and “deer” classes, respectively—our model produces more photorealistic natural images.  \n\nQuantitative evidence: Please refer to Table 1, where a lower FID indicates superior generation quality; our model achieves the better FID score of 2.81.\n\nQ11:\n\nYes, we can mathematically explain that Dirichlet forword process involves masking.\n\nIt is also reflected in related article, such as *Avdeyev,et al.Dirichlet diffusion score model for biological sequence generation.ICML 2023.*\n\nGaussian Forward Kernel:\n$$x_t=\\sqrt{\\alpha_t}x_0+\\sqrt{1- \\alpha_t}\\varepsilon,\\varepsilon\\sim\\mathcal N(0,I),$$\nThe pixel values independently add Gaussian noise with variance $1-{\\alpha}_t$, which is a form of element-wise addition and represents absolute perturbation.\n\nDirichlet Forward Kernel:\n$$z_t=\\frac{z_0\\odot\\alpha_t}{\\|z_0\\odot\\alpha_t\\|_1}.$$\nWhen $\\alpha_t\\rightarrow1$, it means no perturbation; when $\\alpha_t$ approach 0,the pixel values are gradually masked. The changes originate from element-wise multiplication, which is relative perturbation.\n\n|Property|Dirichlet|Gaussian|\n|-|-|-|\n|Element|Element-wise Multiplication|Element-wise Addition|\n|Perturbation|Relative Perturbation|Absolute Perturbation|\n\nQ12:\n\nFollowing this comment, we will reduce the number of examples in Figures 1 and 2 from 9 to 4. This change allows us to allocate more space in main text for terms and definitions, and to incorporate other suggestions to enhance the overall presentation of our work.\n\nQ13:\n\nFollowing this comment, we will eliminate the sentence 'High resolution images require more computational resources' from line 153.\n\nQ14:\n\nPlease refer to Figure 3. Comparing sub-images in the first row, third and fourth columns of Figures 3(a) and 3(b)—corresponding to the “car” and “deer” classes, respectively—our model produces more photorealistic natural images.\n\nIn revised version, we have added some one-to-one comparative displays of these clearly distinguishable sub-images like what have been mentioned above, so as to more clearly highlight the superior generation quality of our model.\n\nQ15:\n\nFollowing this comment, we have adjusted space in main text for clearer presentation. The relevant terms and definitions, including those for Beta and Dirichlet, have been moved to main text §3. The unused definitions have been reduced or adjusted. Other parts involving appendix have been updated to include more pointers directing readers to appendix.\n\nQ16:\n\nTypo acknowledged. spaces before citation and capital letter typos have been corrected in revised version.\n\nSupplement1:\n\nThe FID for -ELBO, KLUB, PHD, and PHK-optimized Dirichlet Diffusion on unconditional CIFAR-10 with varying NFE under different combinations of concentration parameter $\\eta$ and mini-batch size $B$ :\n\\begin{array}{ccc|ccc}\n\\hline\nLoss&\\eta\\times 10^{-4}&B&NFE=20&50&200&500&1000&2000 \\\\\\\\\n\\hline\n-ELBO&1&512&15.81&6.53&4.49&4.33&4.28&4.31\\\\\\\\\n-ELBO&1&288&16.02&6.65&4.68&4.42&4.42&4.39\\\\\\\\\nKLUB&1&512&16.62&6.16&3.46&3.39&3.32&3.25\\\\\\\\\nKLUB&1&288&16.35&6.23&3.58&3.46&3.40&3.21\\\\\\\\\nPHD&1&512&16.84&6.48&3.85&3.72&3.52&3.51\\\\\\\\\nPHD&1&288&16.95&6.93&4.43&4.23&4.19&4.21\\\\\\\\\nPHK&1&512&15.92&5.87&3.20&2.92&2.84&2.82\\\\\\\\\nPHK&0.5&512&17.08&6.60&3.55&3.09&3.10&3.14\\\\\\\\\nPHK&0.1&512&20.67&9.67&5.66&4.54&4.58&4.53\\\\\\\\\nPHK&1&288&16.53&6.15&3.29&2.99&2.91&2.81\\\\\\\\\nPHK&0.5&288&16.94&6.71&3.61&3.09&3.03&3.02\\\\\\\\\nPHK&0.1&288&20.33&9.56&5.71&4.77&4.75&4.72\\\\\\\\\nPHK&1&128&16.85&6.14&3.25&2.83&2.79&2.78\\\\\\\\\nPHK&0.5&128&16.32&6.40&3.57&3.23&3.09&3.06\\\\\\\\\nPHK&0.1&128&20.22&9.73&6.00&5.00&4.93&5.00\\\\\\\\\n\\hline\n\\end{array}\n\nSupplement2:\n\nIn the revised version , we have supplemented experiments on the discrete compositional data, which is naturally High Dimensional simplex-structured, and provide Figure: 'Comparison of true probability mass function (PMF) and empirical PMFs of three methods—Gauss ELBO, Dirichlet ELBO, and Dirichlet PHK-computed. Each empirical PMF is depicted with solid dots.' Each empirical PMF is computed based on 100k data generated by the model trained after 400k iterations with 200 Numbers of Function Evaluations (NFE).\n\nBecause Rebuttal does not permit links or images, we can only describe the results here: In Gauss ELBO, the average discrepancy between empirical PMF and true PMF is approximately 11.02\\%. In Dirichlet ELBO, the average discrepancy between empirical PMF and true PMF is approximately 9.81\\%. In Dirichlet PHK, the average discrepancy between empirical PMF and true PMF is approximately 2.17\\%. The lower discrepancy between empirical PMF and true PMF signifies superior generation quality.\n\nIt shows that, Dirichlet ELBO and Dirichlet PHK achieve better performance in the task where data is naturally High Dimensional simplex-structured. Dirichlet ELBO is better than Gauss ELBO. Dirichlet PHK is better than Dirichlet ELBO.The figure that includes both empirical PMF and true PMF would more visually demonstrate this result; however, the rebuttal page does not permit links or images.\n\nSupplement3:\n\nThe training-time cost of KLUB is more expensive. KLUB incurs 15 \\% additional training-time cost compared with PHD.The training time cost of KLUB, PHD, and PHK-optimized Dirichlet Diffusion on unconditional CIFAR-10 with 4 Nvidia L40s GPUs:\n\\begin{array}{c|cc}\n\\hline\nB&KLUB&PHD&PHK\\\\\\\\\n\\hline\n512&70h&61h&64h\\\\\\\\\n288&73h&64h&68h\\\\\\\\\n\\hline\n\\end{array}\n\nSupplement4:\n\nThe training time cost of PHK-optimized Dirichlet Diffusion on unconditional CIFAR10-32x32, AFHQ-64x64, FFHQ-64x64 datasets with 4 Nvidia L40s GPUs:\n\\begin{array}{c|cc}\n\\hline\nB&CIFAR10&AFHQ&FFHQ\\\\\\\\\n\\hline\n512&64 hours&10 days&11 days\\\\\\\\\n288&68 hours&11 days&13 days\\\\\\\\\n\\hline\n\\end{array}\n\nWe sincerely appreciate the time and efforts of all reviewers and ACs, and believe that the additional experiments, analysis, and explanations significantly improve the quality of our mission. We hope that this provides sufficient reasons to raise the score."}, {"review_id": "oQ0OcEa8zx", "reviewer": "Reviewer_YQgi", "summary": "This paper introduces a novel non-Gaussian-based diffusion model framework, which is built upon Dirichlet distribution. It is theoretically well-suited for high-dimensional data on a probability simplex. To train this model, the authors introduce two novel divergence measures derived from Hölder divergence. They validate the effectiveness of the proposed framework by achieving state-of-the-art results on natural image datasets.", "questions": "1.  The paper's main motivation is the suitability of the Dirichlet distribution for data on a high-dimensional simplex. While image data can be mapped to a simplex, this is not its natural domain. Could the authors comment on or show results for a task where the data is naturally simplex-structured (e.g., compositional data, topic modeling proportions)? This could more fundamentally demonstrate the advantages of the proposed framework.\n2.  Beyond the FID score, are there any qualitative advantages to using Dirichlet diffusion? For example, does it handle certain types of images, textures, or color distributions better than Gaussian models? A qualitative comparison could help clarify the unique benefits of this approach.\n3.  The authors mention that the model is sensitive to hyperparameters. Could they provide more insight into the tuning process? For instance, how critical are the $S_{hift}$ and $S_{cale}$ parameters, and how much effort was required to find the optimal values for the PHK loss weights ($\\delta$ and $\\epsilon$)?\n4.  Is it possible to distill or accelerate the proposed framework like Gussian diffusion models?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 4, "strengths_and_weaknesses": "**Strengths:**\n1. The paper's primary strength is its originality. It makes a fundamental contribution by proposing both a new distributional family (Dirichlet) for the diffusion process and a new, principled optimization objective (PHK divergence). This work successfully expands the theoretical foundations of diffusion models beyond the dominant Gaussian paradigm.\n\n2. The empirical results are promising. Achieving an FID of 2.81 on unconditional CIFAR-10 is not easy especially using a non-Gaussian-based diffusion model. The comprehensive comparison in Table 1 and the ablation study in Table 2 also provide strong evidence for the method's effectiveness.\n\n3. The approach is well-grounded in theory. The motivation for using the Dirichlet distribution for simplex data is sound, and the derivation of the PHK divergence to obtain a symmetric, closed-form loss for this specific distribution is elegant and well-justified.\n\n**Weaknesses:**\n1. The paper is mathematically dense. While technically sound, the presentation could be improved by providing more intuition behind the core concepts. For instance, a clearer, high-level explanation of why the Hölder divergence is a better fit for the Dirichlet process than the standard KL divergence would make the work more accessible to a broader audience.\n2. The practical benefit of this added complexity is not fully explored. While the FID score is excellent, the paper does not discuss the qualitative differences between samples generated by Dirichlet diffusion versus Gaussian diffusion. Does this framework offer unique advantages, such as better modeling of textures, colors, or compositions, that would justify its adoption over simpler, more established Gaussian models? Strong results on other datasets (e.g., texts or DNA sequences) where Gaussian diffusion models underperform will help clarify it.\n3. The authors acknowledge that the method is sensitive to hyperparameter tuning and that the hyperparameter space has not been fully explored. This raises questions about the practical difficulty of applying this method. A more detailed discussion of the tuning process or sensitivity analysis would strengthen the paper's practical contribution.", "was_revised": true, "final_justification": "The authors addressed most of my concerns in their rebuttal. After reviewing their discussion with other reviewers, I remain positive about this paper and thus keep my original rating.", "rebuttal": "Thank you profoundly for the constructive comments. Below, we provide point-by-point responses to specific comment.\n\nQuestion1:\n\nYes. In the revised version of paper, we have supplemented experiments on the discrete compositional data, which is naturally simplex-structured, and provide Figure: 'Comparison of true probability mass function (PMF) and empirical PMFs of three methods—Gauss ELBO, Dirichlet ELBO, and Dirichlet PHK-computed. Each empirical PMF is depicted with solid dots.' Each empirical PMF is computed based on 100k data generated by the model trained after 400k iterations with 200 Numbers of Function Evaluations (NFE).\n\nBecause Rebuttal does not permit links or images, we can only describe the results here: In Gauss ELBO, the average discrepancy between empirical PMF and true PMF is approximately 11.02\\%. In Dirichlet ELBO, the average discrepancy between empirical PMF and true PMF is approximately 9.81\\%. In Dirichlet PHK, the average discrepancy between empirical PMF and true PMF is approximately 2.17\\%. The lower discrepancy between empirical PMF and true PMF signifies superior generation quality.\n\nIt shows that, Dirichlet ELBO and Dirichlet PHK achieve better performance in the task where data is naturally simplex-structured. Dirichlet ELBO is better than Gauss ELBO. Dirichlet PHK is better than Dirichlet ELBO.The figure that includes both empirical PMF and true PMF would more visually demonstrate this result; however, the rebuttal page does not permit links or images.\n\nQuestion2:\n\nYes, your comment is highly precise. In addition to the FID scores, we provide an intuitive demonstration of the qualitative strengths of Dirichlet diffusion. As external links or embedded images are not permitted in the Rebuttal, please refer to Figure 3 in our paper. Comparing the sub-images in the first row, third and fourth columns of Figures 3(a) and 3(b)—which correspond to the 'car' and 'deer' classes, respectively—our model yields markedly more photorealistic images.\n\nSpecifically, the 'car' sub-image in the first row, third column of Figure 3(a) appears highly blurred, whereas the corresponding sub-image in Figure 3(b) is sharp and detailed. Both were generated under identical settings except for the underlying model; the clearer, more realistic result in Figure 3(b) is produced by our Dirichlet model. This stark contrast is equally evident for the 'deer' sub-image. Such side-by-side comparisons visually underscore our model’s superior handling of image structure, texture, and color distribution.\n\nIn the revised version of the paper, we have added some one-to-one comparative displays of these clearly distinguishable sub-images like what have been mentioned above, so as to more clearly highlight the superior generation quality of our model. Thank you for the insightful suggestion to highlight these distinctive advantages.\n\nQuestion3:\n\nThank you for your attention to hyperparameter selection. Yes, we can provide additional details about our hyperparameter tuning procedure.\n\nTo further address the concerns regarding the ablation analysis of critical hyperparameters $\\eta$ and $B$, we have conducted additional ablation experiments on the concentration parameter $\\eta$ and the mini-batch size $B$.  \n\nThe Frechet Inception Distance (FID) scores for negative ELBO, KLUB, PHD, and PHK-optimized Dirichlet Diffusion on the CIFAR-10 image dataset with varying Numbers of Function Evaluations (NFE) under several different combinations of the concentration parameter $\\eta$ and mini-batch size $B$ are shown in the following table:\n\n\\begin{array}{ccc|cccc}\n\\hline\nLoss & \\eta \\times 10^{-4} & B & NFE= 20 & 50 & 200 & 500 & 1000 & 2000 \\\\\\\\\n\\hline\n-ELBO & 1 & 512 & 15.81 & 6.53 & 4.49 & 4.33 & 4.28 & 4.31 \\\\\\\\\n-ELBO & 1 & 288 & 16.02 & 6.65 & 4.68 & 4.42 & 4.42 & 4.39 \\\\\\\\\nKLUB & 1 & 512 & 16.62 & 6.16 & 3.46 & 3.39 & 3.32 & 3.25 \\\\\\\\\nKLUB & 1 & 288 & 16.35 & 6.23 & 3.58 & 3.46 & 3.40 & 3.21 \\\\\\\\\nPHD & 1 & 512 & 16.84 & 6.48 & 3.85 & 3.72 & 3.52 & 3.51 \\\\\\\\\nPHD & 1 & 288 & 16.95 & 6.93 & 4.43 & 4.23 & 4.19 & 4.21 \\\\\\\\\nPHK & 1 & 512 & 15.92 & 5.87 & 3.20 & 2.92 & 2.84 & 2.82 \\\\\\\\\nPHK & 0.5 & 512 & 17.08 & 6.60 & 3.55 & 3.09 & 3.10 & 3.14 \\\\\\\\\nPHK & 0.1 & 512 & 20.67 & 9.67 & 5.66 & 4.54 & 4.58 & 4.53 \\\\\\\\\nPHK & 1 & 288 & 16.53 & 6.15 & 3.29 & 2.99 & 2.91 & 2.81 \\\\\\\\\nPHK & 0.5 & 288 & 16.94 & 6.71 & 3.61 & 3.09 & 3.03 & 3.02 \\\\\\\\\nPHK & 0.1 & 288 & 20.33 & 9.56 & 5.71 & 4.77 & 4.75 & 4.72 \\\\\\\\\nPHK & 1 & 128 & 16.85 & 6.14 & 3.25 & 2.83 & 2.79 & 2.78 \\\\\\\\\nPHK & 0.5 & 128 & 16.32 & 6.40 & 3.57 & 3.23 & 3.09 & 3.06 \\\\\\\\\nPHK & 0.1 & 128 & 20.22 & 9.73 & 6.00 & 5.00 & 4.93 & 5.00 \\\\\\\\\n\\hline\n\\end{array}\n\n\nFor the hyperparameters $S_{hift}$ and $S_{cale}$, the two parameters are not the focus of this work; we adopted the settings established in prior work and provided the corresponding references.In Appendix B 'Hyperparameter Settings', we elucidate the rationale of $S_{hift}$ and $S_{cale}$: 'These settings are predicated on the research conducted by Kingma et al. [6] and Zhou et al. [41]'. The appendix G of Ref [41] provides a comprehensive exposition on the selection and analysis of the hyperparameters $S_{hift}$ and $S_{cale}$.*Mingyuan Zhou,et al.Beta Diffusion.NeurIPS 2023.*\n\nTo further address the concerns regarding the analysis of loss weight coefficients $\\delta$ and $\\epsilon$, we have conducted additional ablation experiments on the key loss weight coefficients $\\delta$ and $\\epsilon$.  \n\nThe Frechet Inception Distance (FID) scores for PHK-optimized Dirichlet Diffusion on the CIFAR-10 dataset with varying Numbers of Function Evaluations (NFE) under several different combinations of the key loss weight coefficients $\\delta$ and $\\epsilon$ with $B=288$ are shown in the following table:\n\n\\begin{array}{cc|ccccc}\n\\hline\n\\delta & \\epsilon & NFE=20 & 50 & 200 & 500 & 1000 & 2000 \\\\\\\\\n\\hline\n\\delta=0.5 & \\epsilon=-0.1 & 16.62 & 6.38 & 3.78 & 3.56 & 3.42 & 3.35 \\\\\\\\\n\\delta=0.5 & \\epsilon=0 & 16.14 & 6.17 & 3.66 & 3.43 & 3.34 & 3.29 \\\\\\\\\n\\delta=0.5 & \\epsilon=0.1 & 16.71 & 6.16 & 3.25 & 2.92 & 2.82 & 2.84 \\\\\\\\\n\\delta=0.5 & \\epsilon=0.15 & 16.74 & 6.06 & 3.20 & 3.00 & 2.86 & 2.78 \\\\\\\\\n\\delta=0.5 & \\epsilon=0.2 & 16.53 & 6.15 & 3.29 & 2.99 & 2.91 & 2.81 \\\\\\\\\n\\delta=0.5 & \\epsilon=0.25 & 16.58 & 6.25 & 3.47 & 3.12 & 2.98 & 2.93 \\\\\\\\\n\\delta=0.5 & \\epsilon=0.5 & 16.07 & 6.23 & 3.61 & 3.39 & 3.35 & 3.31 \\\\\\\\\n\\hline\n\\end{array}\n\nWe have been presented these above results in the revised version of the paper.\n\nQuestion4:\n\nThank you for your insightful comment. Yes, as mentioned in the question, there is indeed the possibility to distill or accelerate our framework. We will explore this in our future work. To date, we have considered several potential directions, as detailed below:\n\nProgressive Distillation: To address the drawback of requiring thousands of evaluations, we are exploring progressive distillation. Tim Salimans et al. proposed a progressive distillation method suitable for diffusion models at ICLR. *Salimans, Tim, et al. Progressive Distillation for Fast Sampling of Diffusion Models. ICLR 2022.*\n\nNeural Operators: To accelerate sampling speed while maintaining generation quality, we are introducing Fourier Neural Operators (FNOs). Zheng, Hongkai, et al. presented Neural Operators for accelerated sampling of diffusion models (DSNO) at ICML. *Zheng, Hongkai, et al. Fast sampling of diffusion models via operator learning. ICML 2023.*\n\nDistributed Parallel Inference: Leveraging the parallel computing capabilities of modern hardware, the computation of diffusion models can be divided into multiple independent parts. Li, Muyang, et al. introduced DistriFusion at CVPR, which employs displaced patch parallelism for parallel inference of diffusion models. *Li, Muyang, et al. DistriFusion: Distributed Parallel Inference for High-Resolution Diffusion Models. CVPR 2024.*\n\nSupplement1:\n\nThe training-time cost of KLUB is substantially more expensive. When all other conditions are identical and $B=512$, KLUB incurs approximately a 15 \\% additional training-time cost compared with PHD. The training time cost of the KLUB, PHD, and PHK-optimized Dirichlet Diffusion models on the unconditional CIFAR-10 image dataset across two mini-batch sizes $B$ with four Nvidia L40s GPUs are shown in the following table:\n\n\\begin{array}{c|cc}\n\\hline\nB&KLUB&PHD&PHK\\\\\\\\\n\\hline\n512&70h&61h&64h\\\\\\\\\n288&73h&64h&68h\\\\\\\\\n\\hline\n\\end{array}\n\nSupplement2:\n\nFour NVIDIA L40 GPUs are available for most time in this work. One complete CIFAR-10 training run requires roughly 68 hours, whereas AFHQ-64×64 would demand about 11 days, and FFHQ-256×256 exceeds the GPUs’ memory capacity. The training time cost of the PHK-optimized Dirichlet Diffusion models on the unconditional CIFAR10-32x32, AFHQ-64x64, FFHQ-64x64 image datasets across two mini-batch sizes $B$ with four Nvidia L40s GPUs are shown in the following table:\n\n\\begin{array}{c|cc}\n\\hline\nB&CIFAR10&AFHQ&FFHQ\\\\\\\\\n\\hline\n512&64 hours&10 days&11 days\\\\\\\\\n288&68 hours&11 days&13 days\\\\\\\\\n\\hline\n\\end{array}\n\nSupplement3:\n\nwe have supplemented experiments on class-conditional CIFAR-10 and achieves an FID score of 2.43. The FID scores for PHK-optimized Dirichlet Diffusion on class-conditional CIFAR-10 with varying NFE:\n\\begin{array}{ccc|cc}\n\\hline\nLoss&cond/uncond&B&NFE=20&50&200&500&1000&2000\\\\\\\\\n\\hline\nPHK&cond&512&13.21&5.45&2.92&2.67&2.52&2.43\\\\\\\\\n\\hline\n\\end{array}\n\nWe sincerely appreciate the time and efforts of all reviewers and ACs, and believe that the additional experiments, analysis, and explanations significantly improve the quality of our mission. We hope that this provides sufficient reasons to raise the score."}, {"review_id": "Cty7UEZ0p3", "reviewer": "Reviewer_CCfR", "summary": "This paper proposes a novel framework for diffusion model to solve high dimensional simplex modeling. Compared to the previous works such as Gaussian-based prior DDPM, this paper claims that the improved Dirichlet distribution modeling for forward/reverse process, and the improved training loss based on Holder divergence can successfully address this issue.", "questions": "1. For forward process, only marginal distribution is given. Could you also provide the conditional transistion?\n2. The reverse kernel is proposed without Bayes derivation, which lacks correctness verification. \n3. How does the Shift and Scale chosen in the experiment? How do they be proposed? There is no description in the text.\n4. Could you please also address the SDE form given the new formulation?\n5. HPD impropriety counter-example seems wrong. Chosen p, q do not satisfy p^{\\alpha} = cq^{\\beta}.\n6. PHD strict-propriety proof omits boundary cases \\gamma = 1 or \\alpha -> 1.\n7. The bound in Lemma 2 Appendix A seems wrong. It misses a scaling factor based on \\lambda\n8. The Theorem 2 in Appendix A seems weak. Also, there is no empirical check. \n9. In Algorithm 1, why Shift and Scale are applied twice?\n10. In the main experiment, it seems that FID=2.81 of the method is obtained by 2,000 sampling steps, which may be different from DDPM, potentially causing an unfair comparison. \n11. In line 112-114, why Beta diffusion can be extended to K-dimansional by directly switching the representations? \n12. What is the difference between HPD and PHD in experments? Any analysis?\n13. Why there is not any theoretical analysis towards discrete data? The whole proof in Appendix A seems meaningless and trivial. \n14. What is the difference of this method compared to the paper Dirichlet Flow Matching with Applications to DNA Sequence Design?", "limitations": "Yes.", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strength: \n\n1. This paper presents a new perspective to understand and enhance diffusion framework, which is clear with theoretical analysis. Also, the proposed result demonstrated that the improved framework works. \n\nWeakness: \n\n1. One of the motivation--why Gaussian-based prior methods are struggling to solve high dimensional simplex data modeling--is not well-discussed as well as without proper theoretical guarantee. \n\n2. The experimental result is not comprehensive so that it can not prove 1) forward/reverse process is effective (please refer to Table 2 -ELBO B=512 Step=1000) since the FID is higher than DDPM. 2) this framework works for high dimension simplex (since there is only one dataset comparison.) 3) the framework can be extended to another modality/low dimension data. \n\n3. The theory, including the formulation and the proof in Appendix A are not well-established. Corresponding detailed questions are proposed in \"Question\" section.", "was_revised": true, "final_justification": "The authors' justification is still not convencing. The proposed framework still lacks extensive experiments to demonstrate the application values of the algorithm. Currently, there still exists inconsistency between the experimental results and the authors' claims. \n\nAlso, I don't believe the current experimental benchmark is fancy. Many existing baseline is not involved. For example, there are PFGM [1], also addresses high simplex and shares the similar story with this work. But published in 2023 and obtained better results. \n\nIn summary, I lean to reject this paper.\n\nReference: \n\n[1] Xu, Y., Liu, Z., Tegmark, M., & Jaakkola, T. (2022). Poisson flow generative models. Advances in Neural Information Processing Systems, 35, 16782-16795.", "rebuttal": "Thank you profoundly for the constructive comments. Below, we provide point-by-point responses to specific comment.\n\nQ1:\n\nBesides the marginal $q(z_t|x_0)=\\rm{Dir}(\\eta\\alpha_t x_0)$, the forward chain is Markov with a closed‑form conditional. Let $\\kappa_t=\\eta\\alpha_t$ and $\\Delta_{s\\leftarrow t}=\\kappa_s-\\kappa_t>0$. Sample $u\\sim\\rm{Dir}(\\Delta_{s\\leftarrow t}x_0),z_s=z_t+(1-\\rm{1}^\\top z_t)u,$ which preserves $z_s\\in S_K$. Equivalently, with $u=\\frac{z_s-z_t}{1-\\rm{1}^\\top z_t},q(z_s|z_t,x_0)=\\frac1{(1-\\rm 1^\\top z_t)^{\\Delta_{s\\leftarrow t}-1}}\\rm{Dir}(u|\\Delta_{s\\leftarrow t}x_0)\\rm{1}_{\\{z_s\\succeq z_t,\\rm1^\\top z_s=1\\}}.$     \n\nFor discrete steps $t_{j-1}<t_j$, replace $(s,t)$ by $(t_{j-1},t_j)$ and $\\Delta_{s\\leftarrow t}$ by $\\eta(\\alpha_{t_{j-1}}-\\alpha_{t_j})$.\n\nQ2:\n\nOur reverse kernel is exactly the Bayes posterior of the conditional bivariate Beta law, following(Beta diffusion). \nFix $0<s<t<1$, $x_0\\in(0,1)$ and $\\eta>0$. Let $a_u=\\eta\\alpha_ux_0,b_u=\\eta(1-\\alpha_ux_0),\n\\Delta_{s,t}=\\eta(\\alpha_s-\\alpha_t)x_0,$ for $u\\in\\{s,t\\}$. The joint density of $(z_s,z_t)$ (with $0<z_t<z_s<1$) is $$q(z_s,z_t|x_0)\n=\\frac{\\Gamma(\\eta)}{\\Gamma(a_t)\\Gamma(b_s)\\Gamma(\\Delta_{s,t})}\\;\nz_t^{a_t-1}(1-z_s)^{b_s-1}(z_s-z_t)^{\\Delta_{s,t}-1}.$$\nThe marginals are Beta:\n$$q(z_s|x_0)={\\rm Beta}(a_s,b_s),q(z_t|x_0)={\\rm Beta}(a_t,b_t).$$\nBayes derivation of the reverse kernel.\nBy Bayes’ rule, $q(z_s|z_t,x_0)=\\frac{q(z_s,z_t|x_0)}{q(z_t|x_0)}.$\nApply the change of variables $u=\\frac{z_s-z_t}{1-z_t}\\in(0,1),z_s=z_t+(1-z_t)u,|\\frac{\\partial z_s}{\\partial u}|=1-z_t,$ One obtains the scaled Beta form\n$$q(z_s|z_t,x_0)=\\frac1{1-z_t}{\\rm Beta}(u;\\Delta_{s,t},\\eta(1-\\alpha_sx_0)),u=\\frac{z_s-z_t}{1-z_t}.$$\nSymmetrically, using $y=z_t/z_s$, $q(z_t|z_s,x_0)\n=\\frac1{z_s}{\\rm Beta}(y;a_t,\\Delta_{s,t}),y=\\frac{z_t}{z_s}.$\nIt is straightforward to verify that $q(z_s,z_t|x_0)=q(z_t|z_s,x_0)q(z_s|x_0)=q(z_s|z_t,x_0)q(z_t|x_0),$\nso both kernels are exact Bayes factorizations of the same joint, ensuring correctness.\n\nOur model: We set $p_\\theta(z_s|z_t)=q(z_s|z_t,\\hat x_0=f _\\theta(z_t,t)),$ i.e., the Bayes posterior with $x_0$ replaced by its estimator. Thus the reverse transition is not ad hoc but the Bayesian conditional derived from the proven joint, as in Beta Diffusion.\n\nQ3:\n\nIn Appendix B Hyperparameter Settings, we elucidate the rationale of $S_{hift}$ and $S_{cale}$: These settings are predicated on the research conducted by Kingma et al. [6] and Zhou et al. [41]. The appendix G of Ref [41](Beta Diffusion, NeurIPS 2023) provides a comprehensive exposition on the selection and analysis.\n\nQ4:\n\nLet $x_t\\in\\Delta_{K-1},\\Sigma(x)={\\rm Diag}(x)-xx^\\top,s_t^\\star(x)=\\nabla_x\\log p_t(x)$, and $\\alpha_0(t)=\\sum_i\\alpha_i(t)$ so that $\\tfrac{\\alpha(t)}{\\alpha_0(t)}:=(\\alpha_i(t)/\\alpha_0(t))_i$. Then\n$$\n\\begin{cases} \n\\rm{d}x_t=\\frac12\\beta(t)(\\frac{\\alpha(t)}{\\alpha_0(t)} - x_t)\\rm{d}t+\\sqrt{\\beta(t)}\\Sigma(x_t)^{1/2}\\rm{d}W_t, Forward & (\\rm dt>0) \\\\\\\\\n\\rm{d}x_t=[\\frac12\\beta(t)(\\frac{\\alpha(t)}{\\alpha_0(t)} - x_t) - \\beta(t)\\Sigma(x_t)s_t^*(x_t,t)]\\rm{d}t+\\sqrt{\\beta(t)}\\Sigma(x_t)^{1/2}\\rm{d}\\bar{W}_t,Reverse&(\\rm{d}t<0)\n\\end{cases}\n$$\nEquation above follows from Anderson’s time-reversal formula applied to the forward Itô SDE.\n\nQ5:\n\nHPD satisfies $D_\\alpha^{H}(p:q)=0\\Longleftrightarrow\\frac{p(x)^\\alpha}{\\int p^\\alpha}=\\frac{q(x)^\\beta}{\\int q^\\beta},\\beta=\\frac{\\alpha}{\\alpha-1}\\Rightarrow p^\\alpha=cq^\\beta,$ for some $c>0$ (Nielsen et al.Entropy, 2017).  \nLet $p=(p_1,\\dots,p_K)$ be any on-uniform probability vector and fix $\\alpha>1$ (thus $\\beta=\\alpha/(\\alpha-1)$). Define $q_i = \\frac{p_i^{\\alpha-1}}{\\sum_{j=1}^K p_j^{\\alpha-1}},i=1,\\dots,K.$ Then $p\\neq q$, yet $p_i^\\alpha = cq_i^\\beta,\nc=(\\sum_{j=1}^K p_j^{\\alpha-1})^{\\beta},$ so $D_\\alpha^{H}(p:q)=0$ despite $p\\neq q$. This normalized construction confirms HPD is improper. We will replace the earlier example with this one and add the proper citation.\n\nQ6:\n\nAssumption: For $\\gamma>0$, if $0<\\gamma\\leq1$, no extra condition is needed since $p^\\gamma,q^\\gamma\\in L^1$ automatically. If $\\gamma>1$, assume $\\alpha_i(\\theta) > 1-\\frac1\\gamma$ for all $i$, ensuring integrability.\n\nTheorem: Let $\\alpha,\\beta>0$ with $\\frac1\\alpha+\\frac1\\beta=1$ and $\\gamma>0$. Then HPD is improper (exists$p\\neq q$with$D^H_\\alpha(p:q)=0$), while PHD is strictly proper for$\\alpha>1: D^H_{\\alpha,\\gamma}(p:q)=0\\iff p=q(\\nu-a.e.)$ Strict propriety also holds at the boundary $\\gamma=1$ without extra conditions, and extends continuously to the right-limit $\\alpha\\to1^+:D^H_{1^+,\\gamma}(p:q):=\\lim_{\\alpha\\to1^+} D^H_{\\alpha,\\gamma}(p:q)$exists and implies $p=q$. Detailed proofs including these boundary cases will appear in the updated manuscript.\n\nQ7:\n\nLemma 2: Let $Z\\sim Dir(\\alpha)$ on the $(K-1)$-simplex $S_K$, with $\\alpha=(\\alpha_1,...,\\alpha_K)$ and $\\alpha_0=\\sum_{i=1}^K \\alpha_i$.  \nFix an index $i\\in\\{1,...,K\\}$ and any $\\lambda>-1$.  Define $\\mu:=\\psi(\\alpha_i)-\\psi(\\alpha_0),$ where $\\psi$ is the digamma function. Then:\n$$\nI(\\lambda)=\\int_{S_K}Z_i^\\lambda(\\log Z_i-\\mu)^2 Dir(z|\\alpha)\\rm{d}z=\\frac{\\Gamma(\\alpha_0)\\Gamma(\\alpha_i+\\lambda)}{\\Gamma(\\alpha_i)\\Gamma(\\alpha_0+\\lambda)}\\{\\psi'(\\alpha_i+\\lambda)-\\psi'(\\alpha_0+\\lambda)+[\\psi(\\alpha_i+\\lambda)-\\psi(\\alpha_0+\\lambda)-\\mu]^2\\}.\n$$\nMoreover, using $\\psi'(x)\\le1/x$ for $x>0$ and the mean–value bound $|\\psi(u)-\\psi(v)|\\le|u-v|/\\min\\{u,v\\}$, we obtain the upper bound:\n$$\nI(\\lambda)\\le\\frac{\\Gamma(\\alpha_0)\\Gamma(\\alpha_i+\\lambda)}{\\Gamma(\\alpha_i)\\Gamma(\\alpha_0+\\lambda)}\n\\cdot\\frac{2\\lambda^2}{(\\alpha_i+\\lambda)(\\alpha_0+\\lambda)}.\n$$\nDetailed proofs including these boundary cases will appear in the updated manuscript.\n\nQ8:\n\nAssumption 1(Smooth Dirichlet model)\nThere exists $\\overline{c}>0$ such that $\\sup_{\\theta,i}\\|\\partial_\\theta\\alpha_i(\\theta)\\|\\le\\overline{c}$.  \n\nAssumption 2(Admissible $\\gamma$)\nIf $\\gamma>1$, then $\\alpha_i(\\theta)>1-\\frac1\\gamma$ for all $i$ and $\\theta$. No additional condition is needed for $0<\\gamma\\le1$.  \n\nTheorem 2(Uniform variance domination of PHD over HPD)\nLet $p$ be the target density, and $q_\\theta(z)=Dir(z|\\alpha(\\theta))$ the model, with $\\alpha_i(\\theta)>0$,$\\alpha_0(\\theta)=\\sum_{i=1}^K\\alpha_i(\\theta)$, satisfying Assumptions 1-2. Let $1/\\alpha+1/\\beta=1$ with $\\alpha,\\beta>0$, and $\\gamma\\ge1$. Denote $g_\\theta(z)=\\nabla_\\theta\\log q_\\theta(z)$. Then, for every $\\theta$, ${\\rm Var}_p[\\nabla _\\theta D _{\\alpha,\\gamma}^H(p: q _\\theta)]\\le\\frac{\\gamma^2}{\\alpha \\beta}\\frac{\\alpha _0(\\theta)+1}{\\alpha _0(\\theta)+\\gamma}{Var}_p[\\nabla _\\theta D _\\alpha^H(p:q _\\theta)].$\nMoreover, $\\frac{\\alpha_0+1}{\\alpha_0+\\gamma}\\le1$ decreases monotonically to zero as $\\gamma\\to\\infty$, implying strict improvement for all $\\gamma>1$. Detailed proofs including these boundary cases will appear in the updated manuscript.\n\nQ9:\n\nIn Algorithm 1, the Scale and Shift operations are applied twice. The first application occurs on the initial data, while the second is applied to the generated data.\n\nQ10:\n\nRegarding the comparison with benchmark methods, we have referred to prior works. For instance, in Zhou et al.'s Beta Diffusion(NeurIPS 2023), the reported FID score of 3.06 was achieved using 2000 sampling steps, and their comparison with DDPM was conducted under the same conditions.\n\nQ11:\n\nThe move from Beta to Dirichlet is a principled dimensional lift, not a heuristic switch.\n\n1. Beta is the $K{=}2$ Dirichlet. ${\\rm Beta}(a,b)\\equiv\\rm{Dir}(a,b)$ on the simplex $\\{(x,1-x)\\}$, so the 1D case is already a Dirichlet instance.\n\n2. Shared exponential-family / Gamma structure. Replacing the scalar $x_0$ by $\\rm{x}_0\\in\\Delta_{K-1}$ yields $z_t |\\rm{x}_0 \\sim \\rm{Dir}(\\eta\\alpha_t\\rm{x}_0),$ preserving closed-form log-normalizers, moments, and our Hölder/KL objectives.\n\n3. Reverse kernel keeps the same algebraic form.\nBayes gives the multivariate analogue of the Beta update: $z_{t_{j-1}} = z_{t_j} + (1 - z_{t_j})p_{(t_{j-1}\\leftarrow t_j)},p_{(t_{j-1}\\leftarrow t_j)} \\sim \\rm{Dir}(\\eta(\\alpha_{t_{j-1}}-\\alpha_{t_j})\\hat{\\rm{x}}_0),$ so simplex constraints, scheduling behavior, and tractability are unchanged.\n\nQ12:\n\nIn our initial experiments, we observed a suboptimal FID of 9.72 at 2000 NFE when using HPD, as HPD performs worse than PHD. Consequently, we shifted our focus to the analysis and experiments involving PHD and PHK. This observation is noted in line 132 and further detailed in Appendix A, where the superiority of PHD is thoroughly analyzed.\n\nQ13:\n\nDiscretized forward Dirichlet Diffusion chain:\n$$\nq(z_{t_{1:T}}|x_0)=\\prod_{j=1}^Tq(z_{t_j}|z_{t_{j-1}},x_0)=\\prod_{j=1}^T\\frac1{z_{t_{j-1}}}Dir(\\frac{z_{t_j}}{z_{t_{j-1}}};\\eta\\alpha_{t_j} x_0,...,\\eta\\frac1{K-1}(\\alpha_{t_{j-1}}-\\alpha_{t_j})x_0)\n$$\nThe discretized forward chain is reversible when knowing x0.Detailed analysis will appear in the updated manuscript.\n\nQ14:\n\nDFM (Stärk et al. ICML 2024) learns a deterministic ODE flow with cross‑entropy path matching for DNA design; we learn a stochastic Dirichlet diffusion with a Bayes‑derived reverse kernel, optimized by strictly proper Hölder–Kullback divergences in closed form. Sampling differs (ODE integration vs. discrete Dirichlet updates), as do targets (sequences vs. high‑dimensional images). Complimentary, but method, objective, and application all differ.\n\nSupplement: In revised version, we have supplemented many experiments about parameter, training cost, discrete simplex data, loss weights, and more. Due to 10000 characters limit, please refer to the pages of other reviewers for these supplemented results.\n\nWe sincerely appreciate the time and efforts of all reviewers and ACs, and believe that the additional experiments, analysis, and explanations significantly improve the quality of our mission. We hope that this provides sufficient reasons to raise the score."}]

Author rebuttal:

Meta-review: This paper proposes Proper Hölder–Kullback Dirichlet Diffusion, a non-Gaussian diffusion framework based on the Dirichlet distribution and two new divergence measures: Proper Hölder Divergence (PHD) and Proper Hölder–Kullback (PHK). The approach generalizes Beta diffusion and achieves strong empirical performance within the standard of non-Gaussian diffusion models (e.g., FID 2.78 on CIFAR-10). The work is original, introduces new theoretical results, and the authors were highly responsive in the rebuttal, adding ablations and experiments on compositional data, class-conditional CIFAR-10, and subsets of FFHQ and CVUSA.

The main concerns across reviews are the limited empirical scope (with most results still concentrated on CIFAR-10), and unclear practical advantages beyond FID improvements. One reviewer remained skeptical about the strength of the empirical evidence, and the AC also noted that the Dirichlet diffusion was applied to the latent space rather than directly to observations, a point that requires clearer explanation. Applying the method to real-world compositional domains, such as natural language or biological sequences, would make the paper substantially stronger.

While there remain concerns about scope and clarity, the originality of the contributions, combined with the authors’ thorough responses, provide sufficient reasons to recommend acceptance.
Decision: Accept (poster)