ItDPDM: Information-Theoretic Discrete Poisson Diffusion Model
Authors: Sagnik Bhattacharya
Abhiram Rao Gorle
Ahsan Bilal
Connor Ding
Amit Kumar Singh Yadav
Tsachy Weissman
Venue: NeurIPS 2025 poster ()
Keywords: poisson diffusion
information-theoretic framework
discrete modeling
negative log likelihood
Primary area: other
TL;DR: This paper introduces ItDPDM, a Poisson-based diffusion model that directly models discrete data without relying on continuous embeddings or variational approximations, achieving improved likelihood estimation on music and image benchmarks.

Abstract: Generative modeling of non-negative, discrete data, such as symbolic music, remains challenging due to two persistent limitations in existing methods. Firstly, many approaches rely on modeling continuous embeddings, which is suboptimal for inherently discrete data distributions. Secondly, most models optimize variational bounds rather than exact data likelihood, resulting in inaccurate likelihood estimates and degraded sampling quality. While recent diffusion-based models have addressed these issues separately, we tackle them jointly. In this work, we introduce the Information-Theoretic Discrete Poisson Diffusion Model (ItDPDM), inspired by photon arrival process, which combines exact likelihood estimation with fully discrete-state modeling. Central to our approach is an information-theoretic Poisson Reconstruction Loss (PRL) that has a provable exact relationship with the true data likelihood. ItDPDM achieves improved likelihood and sampling performance over prior discrete and continuous diffusion models on a variety of synthetic discrete datasets. Furthermore, on real-world datasets such as symbolic music and images, ItDPDM attains superior likelihood estimates and competitive generation quality—demonstrating a proof of concept for distribution-robust discrete generative modeling.

ItDPDM: Information-Theoreii Discrete Poisson Diffusion Model
Sagnik Bhattacharya Abhiram R. Gorle' Absan Bilal?, Connor Ding', Amit Kumar Singh Yadav Tsachy Weissman Department of Electrical Engineering, Stanford University DDeparmeent of Computer Science, Oklahoma University Schooo of Electrical and Computer Engineering, Purdue University, West Lafayette, IN, USA
Abstract
Generative modeling of non-negative, discrete data, such as symbolic music, re- mains challenging due to two persistent limitations in existing methods. First, most approaches rely modeling continuous embeddings, which are not well- suited for inherently discrete data distributions. Second, they typically optimize variational lower bounds instead of the true data likelihood, leading to inaccurate likelihood estimates and degraded sampling quality. While recent ceffusion-basse models have addressed these issues individually, we tackle them jointly. In this work, we introduce the CnermaaiionThhooelle Discrete Poisson Diffusion Model (ItDPDM, inspired by photon arrival processes, unifying exact likelihood esti- mation with discrete-state generative modeling. Central to our approach an information-lee -hhooretic Poisson Reconstruction Loss (PRL) that admits provable, exact relationship with the true data likelihood. ItDPDM achieves improved like- lihood and sampling performance over prior discrete and continuous diffusion models on a variety of synthetic discrete datasets. Furthermore, on real-world datasets such as symbolic music and images, IdDPDM attains superior likelihood estimates and competitive generation quality, demonstrating a proof of concept for principled, distribution-robust discrete generative modeling.
Introduction and Background
Denoising diffusion models have advanced generative modeling, outperforming GANs in image synthesis ① and autoregressive models in likelihood-based tasks 22 Their flexibility enables broad
industrial usc-frrm open-ended eet-to-imaae generation [ to audio [6] and medical imaging
7
Diffusion has also been extended to multimodal and structured tasks, including video synthesis
cross-modal retrieval [9) and molecular modeling 10
Limitations of Existing Works: Diffusion models can be classified by timestep type: discrete (DT) or continuous (CT) and latent space: discrete (DS) or continuous (CS), forming four classes: DTDS, DTCS, CTDS, and CTCS, as shown in Figiue I DTCS (c.g.. VDM (2]) and CTCS (c.g.. IT-Gaussian diffusion 12) are effective in continuous domains 2
131, but suboptimal for inherently discrete non-Gaussian data distributions. As shown in Figure the continuous-state models map discrete data
continuous state space via z-soring 14] tail normalization [5). or uniform dequantization [12] However, these fail to close the discretization gap (e.g., 127.5 for images), and lead to learning suboptimal probability density functions (pdf) instead of probability mass functions (pmf) [12]. Figure 3 shows how continuous DDPMs miss the second mode in the evidently bimodal NYC Taxi distribution [6]. Moreover, discretizing outputs during post-processing introduces train-test mis- match 22 17 18). Recent discrete-state state models directly operate in the discrete domain, addressing these limitations by avoiding embedding into continuous spaces altogether.
* Equal contribution; Correspondence to eagnikbdstanford.com edu. Our implementation available here
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Continupus Space
Diffusion [14g
Hormaization
Port pi procesiin
Da co-opanttaaiio
CTCS leg. Info-Theorete Gaurdaan [Diffucnnn X)]
I-MIEEE
intleraal
Dieeete Data
Discreti
pro prodessing
DTDS leg masking models, D3PM]La
Variational ELBO Objective
No past processing
Disorete State Space
CTDS (KDrCM)
Reconstr action Lowa (PRL)
Figure 1: Classification of diffusion models based on latent state-space (DS/CS) and timesteps (DT/CT)), resulting combinations DTCS, CTCS, DTDS, and CTDS
Discrete-time discrete-state (DTDS) models 15 operate natively in the discrete domain and outperform variational Gaussian-bais based methods, but often ignore ordinal structure of integer-valued data and need post- post-procsssing Searning-to-Juma (LTJ) [20]. recent DTDS method using binomial thinning and variational objective, improves generation on non-negative, skewed data. However, LTJ has two drawbacks: (1) its evidence lower bound (ELBO) (ELBO)-based training uses variational relative entropy loss, which lacks an exact relation the true data likelihood, yielding suboptimal likelihood and degraded generation quality; (2) denoising requires careful calibration of T (e.g.. 1000), the number of discrete denoising timesteps, without any flexibility to skip or subsample.
Main Contributions. To address these limitations, we introduce a novel information-thcoredt Discrete Poisson Diffusion Model (ItDPDM). As shown Figure contrary Gaussian diffusion, 1tDDDM directly models discrete negativve data using Poisson process, avoiding the necessity for any soft discretization or dequantization. Contrary to variational DTDS models like LTJ [20]. ItDPDM provides improved, closed- form likelihood estimates while maintaining competitive generation quality.
Our main contributions are summarized as follows:
We propose ItDDDM, novel generative framework based on a Poisson diffusion process for modeling non-negative discrete data. Unlike prior approaches relying on variational ELBO objectives, 1tDPDM enables a sieeihood-ccossisenn objective with tractable 1-D quadrature, bypassing the limitations of variational inference.
We introduce the intommttion-hhooretii Poisson Reconstruction Loss (PRL), Bregman divergence (21 tailored to Poisson processes and establish its exact relation to negative log- likelihood (NLL) via the I-MPRL identity in Eq. (17), enabling onnvaaiitional optimization discrete probability mass functions (PMFs).
Experiments on synthetic datasets with varied data distributions show that ItDPDM out- performs earlier baselines in Wasserstein-1 distance and log- ligeliidld (NLL). ItDPDM's discrete Poisson-basse diffusion generalizes well beyond Poisson distributed data.
Empirical
Gaussian DDPM
10-1
Density
1
10-2
100
120
Pickup rate
Figure 3: Gaussian diffusion fails to accurately leam the discrete probability density
Figure 2: Unconditional image samples generated by 1DDPDM

---

Gaussian Diffusion: s,- NiGlrlinyi (ei-taybn)
Gaussian Denoising
7g
y
3y
Poisson Diffusion:
Poisson Denoising
=
Tor
Figure 4: Comparison of Gaussian (top) and Poisson diffusion processes (bottom).
We also provide closed-form upper bounds the negative log-likelihood (NLL) and an importance-samplee estimator for efficient training, ensuring scalability to high- dimensional settings. Empirically, ItDPDM achieves significantly lower NLL estimates CIFAR-10 and Lakh MIDI datasets while maintaining competitive generation quality.
This work presents proof-of-concept for information-theoreet discrete Poisson diffusion models, showing initial gains over baselines modeling discrete, positive-values data. serves as a first step toward principled diffusion modeling in discrete domains, not state-of-the-an solution.
2 Information-Thenn -hhereeti Diffusion
We briefly revisit the Information- Theoretic Gaussian Diffusion (ITDiff) framework from 12 which helps us draw parallels to 1DPDM in Sec
The Gaussian noise channel is defined as
Ey Vix + e, e ~ (O,
with signal-to-noise ratio (SNR) parameter 'y" and data distribution p(z).
Relating Minimum Mean Square Error (MMSE) to Mutual Information
The "I-MMSE" relation
22 links mutual information with minimum mean square error (MMSE):
d (x;; 2n) mmse(7),
1
(1)
d-
where the MMSE is defined as: mmse(r) min n) Ep(z,,)) [[|e 2113) generalization of Eq. (1) to the KI. divergence is as follows:
pointwise
d DKL(p(z)lz) Il p(2+)) 2 mmse(x,1) dy
(2)
From here, the following discrete probability estimator is derived through an exact formulation of the variational lower bound (VLB) for diffusion models [2]
log P(x) = mmse(æ, 7) dy
3 ItDPDM: Cnormattio-Theorele Poisson Diffusion
Poisson Noise Channel: We define the canonical Poisson noise channel: given a non-negative input
x 2 0, the output 2, is drawn from P(yx), where 7 denotes the SNR. The conditional PMF
P(z)1æ) = ( 7 2g No.
2,!
where P(-) denotes the Poisson distribution. This setup is motivated by Poisson channels arising in direct-detection optical systems 24]
24), where photon counts follow Poisson process with rate determined by combination of signal intensity and device-induced dark current (25).

---

Diffusion with Poisson Noise: We propose an information thhooretththeeeeeeeeo Poisson diffusion process, where source x p(x) is corrupted at SNR Y via by P(yz), producing discrete, non-negative integers at each step. Unlike Gaussian noise, Poisson corruption is non-additive and not source- source-separable, making denoising more challenging. Figure contrasts Gaussian and Poisson diffusion: Gaussian begins from white noise, whereas Poisson diffusion starts from a black image with zero photons.
Poisson Reconstruction Loss (PRL): The function (z) z loog x I + > (where log denotes the natural logarithm) is the convex conjugate of the Poisson distribution's log moment generating function (proof App. D.1) and often arises naturally in the analysis of continuous and discrete-time jump Markov processes [20 26 and for mutual information estimation in the Poisson channel [27] Building on this, we define the poisson reconstruction loss (2x, 2) as:
(z, 2) flo(z/0) z log
(5)
Analogous to the MMSE, we also define the minimum poisson reconstruction loss (MPRL) as:
mprl(?)= min Ep(z,, (x ((,, 2xx,,7))
(6)
where 2 (Zy 7) denotes the denoiser. The optimal denoiser 2 is the conditional expectation E|X] Zy] using the fact that the Poisson reconstruction loss is Bregman divergence [28] (proof App.D.1).
2 (Z,, 7) arg min m7) P(-l(())
(7)
The analytical solution is typically intractable due to the need to sample from the Poisson noise channel's 's posterior. We next highlight key properties [29] of this loss, showing is natural fit for evaluating reconstruction of non-negative data, analogous to squared error in the Gaussian case.
Lemma (Poisson Reconstruction Loss). The loss function (xx .) satisfies the following properties:
1. Non-negativity: ix,, 2 O, with equality if and only if ât
2. Convexity; (x, 2) is convex in à for each fixed and in for each fixed i.
3. Scaling: For any a >> O, l(ax, at) al(x, 2).
4. Unboundedness for underestimation: For any x > lim Ux, 8) 20.
5. Optimality of Conditional Expectation: For any non-negative random variable X with E[Xlog' X| < oc, the conditional expectation E uniquely minimizes the expected loss (XI 3)
Convexity makes the loss amenable to gradient- based methods. Property penalizes underes- timation, making 1(x, 2) well-suited for non- negative data, unlike common loss functions (absolute/squared error). Figuure Sjillustrates the behavior the proposed PRI., As per Lemma]] the conditional expectation Y uniquely minimizes the expected "mprl" loss function.
F11,
1.0
0.5
0.0
Figure 5: Poisson Reconstruction Loss (PRL): (a) vs. denoised pixel é. for fixed ground truth pixel (b) ground truth pixel æ, for fixed denoised output 1.
Conditional Expectation for Poisson Channel We define the angle bracket operator X as con- ditional expectation given Z,; (X) E Zzy Unlike the linear Gaussian case, Poisson has a non-linear (X), making boisson-based denoising fundamentally more complex. Nevertheless, it becomes linear under certain conditions, let (X)x
denote (X) evaluated at Z,, = z then:
Lemma 2 (Linearity in Poisson Channel). Let Z, P(7X). Then, (X): az + b, if and only if and b 0.
X ~ Gam (- () for any <
Though Poisson analysis is complex, it simplifies here since the Gamma distribution is its conjugate prior [30], yielding a linear conditional variance (see App. D.2). This contrasts with the Gaussian case, where conditional variance is constant. We now revisit the squared error loss Esu .)) (x 2)) which satisfies for any fiiieevvriance X:
El(sp( (X,)) Ellsn(X E[XD] ESp( (
(8)
Our Poisson reconstruction loss (PRL) has similar property stated below ([9

---

Lemma 3. For any non-negative random variable X with E log" X] < oc, and any i E [0. oo),
E. [1((,,)) = F. 11 (x, F[X]] + (F(
A result that immediately follows from Lemma 3b, when combined with the non-negativity property in Lemma is that [[X] uniquely minimizes E (XX, .)] over all f::
min E [[X, a)] = E [1 (X, E[X]] = E X log X] [[ log E[X].
(10)
Interestingly, in Poisson channels, this estimator depends only on the marginal distribution of Zn, a property formalized by the BaeeggGGod-Roooe
formula [i 32]. This result closely relates the Discrete Universal Denoiser (DUDE) (33, which estimates discrete signals from noisy observations.
Lemma (Optimal Estimator in Poisson Channel). Let Z, = P(xX). Then, for every ? >
(x)2 1 (= 1)P2, (x+
2 = 0,
(11)
7
(2)
The PRL objective provides a principled objective for modeling non-negative discrete data, directly modeling PMFs and avoiding quantization artifacts inherent squared error loss, which assumes continuous outputs. The pointwise denoising relation for the Poisson channel is: (proof in App.F)
DKL [P(z)|z) || P(zy)] = (x,
(12)
dy
where p(z,) = p(2, (o (p( or dx the marginal distribution, and pointwise MPRL defined as: mpri(z, 7) 1x) [ (az, 50 (2,,
(13)
The pointwise MPRL is the MPRL evaluated at a fixed x, and its expectation over p(x) recovers
the total MPRL. Taking expectation wrt x: in Eq. (12) recovers the I-MPRL. relation in Eq. (14).
Moreover, for mismatched denoiser (29), integrating the excess pointwise loss over y equals the KL.
divergence between the true and mismatched channel outputs (via Eq. 12)
I-MPRL identity:
Following the foundational result of [22). which relates the mutual information
derivative to MMSE in Gaussian channels, [12] leverages this identity in generative modeling.
Analogously, we establish the I-MPRL identity for the Poisson channel, as follows:
d d; [x; (4) mprl(7).
(14)
similar result holds for the derivative with respect to the dark current in a general Poisson channel. Using the incremental channel technique from [22], we derive both results in App. D.3] This enables
exact relations between the proposed PRL objective and the likelihood, offering an information- theoretic justification for Poisson diffusion. Detailed proofs of Lemmas 7 Mare provided in App. D.2] along with Lemmas 5 and 6, stated and proved therein.
The pointwise denoiser yields log-likelihood expression akin to the variational bound. Unlike traditional methods that rely on expensive sampling, diffusion models leverage the structure of the noise for efficient sampling at arbitrary noise levels
Thermodynamic Integration for Variational Bound:
Letting (zz, ~ P(tx), using thermodynamic integration method from [35] 36 yields: d
D (P((P((z)a)) II P(z+))
mprl(ar, 7)) dr,
(15)
dy
where (xrl(2 7) = E,(,,11> [Z (a, in (2 7))] is the pointwise MPRL for Poisson denoising. The true (exact) log- likelioooo is given by:
- log P(x) = DKL (P (x) || P(en)) Ep(- log P(x
mpri(x,)) dx
(16)
Prior loss
Reconstruction loss
Difftsiio loss
We also outline possible extension of the proposed Continuous- -Tim Discrete-State Poisson Diffusion to a Continuous- Time Continuous-Shre equivalent of [12] in App.
Discrete Probability Estimation via MPRL: We derive novel discrete probability estimator in the Poisson channel setting, where x ~ P(x) and Z, ~ P(yx). In the limits 70 - oo and m -+ 0, both the prior and reconstruction loss vanish, which yields the following tractable expression:
5

---

DKL[P(2,,1E))P + Ep(e log P(x12200) = 0.
and, therefore Eq. 16 yields exact likelihood relation:
log P(x) / mprl(z,))) dry log P(x))
/ mprl(?) dr
(17)
In practice, we use importance-sampling quadrature for this, yielding an unbiased estimate. Conven- tional diffusion incurs two levels of approximation: a) the ELBO (surrogate) loss replaces the true log-likelihood. and b) Monte-Carlo quadrature (or the integral). Our formulation eliminates the first level work with the true data likelihood.
To obtain an expression resembling the variational bound, taking an expectation in x ~ P(x) gives:
log P(x)] = EE 1. mpri( 7) dh
% EI(X,X*) dy
(mmpi(2t 7)
- log - log E|X dy E X log E|X
X
dy
here, 7)) E| denotes the optimal estimator. This section establishes a tractable, non-variational estimator for discrete distributions the Poisson channel by connecting the MPRL objective to the true data likelihood. We also present in App. F.I an equivalent score matching formulation using a Poisson-adappe version of Tweedie's 's formula denoising. Additionally, App. provides a comprchensive comparison between ItDDDM (ours) and LTJ 20)
Extension Multivariate settings:
While the current framework considers a univariate setting. this can naturally be extended to the vector Poisson channel under mild regularity. Let X, X R R and Z
Poissoo)((7 + >x)- In this case, the I-MPRL identities (Eq. 14) hold component-wise [37]. Consequently, all results in this work effectively carry over to the muluvariate setting by replacing scalars with vectors and sums with inner products (or traces). For empirical validation, we also provide toy 2D experiments in App. C.5
4 Numerical Details
MPRL. Upper Bound:
key challenge is the inaccessibility of the posterior distribution in Eq. 7
To bound the intractable marginal likelihood, we compare our (suboptimal) neural denoiser X (Z,,, 7) with the intractable optimal conditional expectation X This reformulates the expected loss entirely
in terms of X (Zy, 7) from the Poisson diffusion model: (proof in App. A.1
E log P(z)] P(z) /.-0 mprl(7) dy S
/ E[((X, X)] dy
(18)
is important to note that the likelihood (NLL) upper bound in Eq. (18) is empirical, capturing the suboptimality of the learned neural denoiser. Eq. (17) yields exact theoretical expression for the likelihood (NLL), unlike variational diffusion models, which introduce two layers of approximation: first via the ELBO, and then through upper bound the denoiser.
Parametrization: ensure stability across SNR levels, we reparameterize the Poisson observation Z, P()X) to mitigate mean and variance explosion. Instead of feeding Z., directly into the neural
network, define the normalized 7, /,1(1 + +)), keeping it within [0, X] with high probability. This transformation preserves interpretability: at high SNR, E|Z,,] X, while at low SNR (? - 0), it tends to zero, aligning with Poisson behavior. We input (2,,7) into the network in place (Z,, 7). Adopting the log-St log-SNR parameterization
log get:
F( Fo log P(x)]
/ 6 mprl((a) do
/ eth E|1XX, da.
(19)
For details on efficient numerical integration of this expression, see App. A.2
MPRL Tail Bounds: Since the integration the RHS of Eq. 19 is intractable, we identify finite
integration range (ag, on) beyond which the contribution becomes negligible. The RHS of Eq. (19 thus be written in terms of 'a" as:
(/00 +
(mprl(a)dd
"mmll (a) da <
e EIEI da f(ao, a1)
a0

---

Algorithm/ 1tDPDM Training
Algorithm ItDPDM Sampling
Require: Dataset (1: ) INR-SNR samples
Require: Trained model fo. reverse steps T
S, SNR range ("min: 7mmax), denoiser fo
Compute {Te) } (e.g. spaced in log-SNR)
1: for -1,.... 1,..., S do
2: Initialize 2gyr f- 0
2: Sample mini-batch B from {x;}
3: fort =
1,...: do
3: Sample a Logistic, 7 exp(a)
4: So f fo casttatrranffrmmie
7e)
4:
Sample 2n
Poisson(T
5: Sample =nc-1 Poisson ( (7- 20)
1+7
end for
5:
in
ccdata_transform(( (2,), ?)
7: return .00
6: t LiE Eien PRL( â.), L L g(a)
7: Update by gradient descent on
8: end for
9: return
We analytically derive upper bounds for the left and right tail integrals, denoted by f(ag, 01) above in App. E and show that their contributions decay rapidly outside the relevant integration range. Algorithm and represent the pseudocode used for ItDPDM training and generation respectively.
Gap between the exact and Monte Carlo (MC) estimators: Consider the canonical binary-input Gaussian channel Y ==v=X- + N,, where E {+1} with equal probability and N ~ ~N 1). The I-MMSE identity gives (although intractable in most practical cases)
- mmmse(7), so that ((X = ///000 mmns((7) d7 In
((x+)) =
For this binary setting, the MMSE admits closed form,
mmse(7) = 1 "[tannh((/5 (vRX)),
since the posterior mean is E[X tanh( When approximated using an n-sample Monte Carlo (MC) estimator, the resulting error decays at the rate (1//VR, consistent with the central limit theorem (CLT)) In practice, this corresponds to an error of order 10-3 for n > 1000. similar behavior holds for MPRL estimators: for instance, when X ~ Gamma (2 3). its MC estimator exhibits the same (1/V/n) convergence rate, with an error magnitude of order 10- 2 for
1000. These results help illustrate that empirical estimators closely track theoretical values even for finite, moderate sample sizes, motivating our design choice of using an importance-sampling quadrature.
5 Experiments
We begin by evaluating on synthetic datasets exhibiting sparsity, skewness, and overdispersion: settings where Gaussian diffusion models often underperform, along with extreme distributions like Zipf where LTJ 200 underperforms. These experiments help empirically validate 1DDDDM by i) recovering ground- -tuth likelihood (NLL) and ii) improving modeling of discrete, non-negative data. We then evaluate ItDPDM on real-world domains like CIFAR10 (images) and Lakh MIDI (symbolic music), where discrete structure inherent. ItDPDM consistently achieves superior likelihood estimates and competitive generation quality, evidenced by domain-specific metrics.
5.1 Synthetic Data
We consider various synthetic distributions containing univariate non-negative data a grouped into two broad categories: discrete E N, and continuous x E [0, co] to mimic distributions exhibiting either sparse, heavy-tailed, skewed, zero-inflated or overdispersed behaviour.
Discrete counts (x E N): We generate six synthetic distributions capturing key real-world behaviors: PoissMix (airport arrivals), ZIP, NBinomMix (forum activity), BNB, and two heavy-tailed laws: Zipf and Simon (word frequencies). These cover bimodality, overdispersion, and long tails. Distribution parameters are listed in Table in Appendix, with design details App. C.I
Continuous non-negative E [0, oc)): also include six skewed continuous densities -aatma, Log-Normal, Lomax, Half-Cauchy, Half-t, and
in App. C.4
7

---

Table I: Metrics synthetic datasets (L lower is better). Bold indicates best.
WD
NLI
Distribution
DDPM
1DDPDM
LTT
DDPM
RDDDNM
PolasMix
3.76 = 0.32
0.99 = 0.15
1.21 - 0.30
3.50
4.24
3.72
3.69
ZIP
2.31 = 0.66
0.56
= 0.43
0.69
- 0.21
2.13
1.67
2.22
2.30
MBinonMix
4.89 0.39
1.39
1.15
1.84
BKB
1.89 1 0.45
0.67
1 0.23
0.65 1 0.32
2.06
2.56
1.87
2.01
Zipr
1.51 + 0.53
0.13
0.73 - 0.25
1.57
1.34
1.70
YS
0.32 = 0.12
0.14
0.03
0.17 = 0.06
0.94
1.39
0.79
0.76
The neural denoiser model for all (discrete, continuous) cases uses a similar architecture tionaMMP)) to ensure fair comparison: a 3-laayr MLP with 64 hidden units, LayerNorm, Leaky-ReL.U activations (slope = 0.2). Further training details can be found App. C.2 To maintain computational tractability, most distributions are truncated at 50. For each distribution, we draw 50,000 i.dd. samples to form the training data and generate 50,000 samples for each run.
Model Architecture:
Metries and results: We report Wasserstein-1 distance (WD) and negative log-likelihood (NLL) between empirical distributions of generated and test samples (see App. C.2). Table summarizzes these metrics for ItDPDM and all baselines. To illustrate the quality of PMF modeling, Figure 6 overlays the true and generated PMFs across all discrete datasets. As shown, ItDPDM consistently outperforms DDPM (trained with MSE) across all datasets, achieving lower WD and NLL estimates that closely align with the true values. It further outperforms L.TJ in 4 out of datasets, demonstrating strong generalization of ItDPDM across diverse distributions, beyond just Poisson-mixture datasets. In contrast, LTJ performs well primarily on binomial- datasets, which are well-suited to its variational think-thickening loss. More details on PMF estimation are in App. C.3
PolasonMic PMF comparison (smooth)
ZIP: PMF comparison (smooth)
NeinomMix PMF comparison (smosth)
05
1075H
03
0.0
20
100
350
340
000
23
50
300
325
150
175
200
0.0
10
35
25
35
40
(smouth)
Zipt: PMF comparison
YJlsSmmn:: PMF comparison
Tue
Tue
The
0.2
(2)
0.3
03
Value
Ve
Figure 6: Comparison of true and generated probability distributions
5.2
World Data
We evaluate ItDPDM on two discrete datasets: CIFAR- 10 images and Lakh MIDI (LMD) sym- bolic music and compare against existing baselines: Improved DDPM (IDDPM) 38 information- theoretic Gaussian diffusion (ITDiff) [12]. discrete masking- (D3PM) [7.. and Cearning-to-juma (LTJ) 120). CIFAR-10 comprises 60,000 color images (32 x 32) across 10 classes 33 LMD con- tains 444 symbolic music sequences of 1024 integers: 0 (rest), 1 (continuation). and 2-89 representing note pitches 40]. Unlike [1]. which fine-tunes pre-trained models, the absence of pre- trained models in our setting necessitates training from scratch. Denoiser architectures (U-Net [4].
ConyTransformer 17]. DenseDDPM 42 are discussed in App. B.3
5.3 Performance Comparison: Negative Log Likelihood (NLL)
Two architectural variants from DDPM 13 and IDDPM 38 are used. Table 2 reports test-set
NLLs on CIFAR-10 subset, comparing ItDPDM to relevant baselines: (1) ITDiff 12 which fine- tunes pretrained Gaussian DDPM/DDPM models, and (2) Gaussian + MSE, where DDPM/IDDMM models are trained from scratch using the ITDiff objective, ensuring a fair comparison. ItDPDM (Poisson + PRL) consistently achieves the lowest NLL across both backbones, with IDDPM slightly outperforming DDPM. These results underscore the effectiveness of Poisson diffusion and PRI. for modeling discrete, non-negative data without requiring dequantization. Figure 7 shows denoising
8

---

Nošsing Objective
DDPM
IDDPM
Noising Objective
NLL (total data)
ID
2.97
0.86
Gaussion MSE
0.51
Goussion MSE
0.44
0.48
1DPDM: Poisson + PRL
4.61 x 10-5
Goussian + PRL
0.27
0.32
Noising Objective
NLL (without rests)
Poisson MSF
0.23
0.22
Gaussion MSE
1.41
I:DPDM: Poisson + PRI
0.18
0.17
RDPDM: Poisson + PRL
0.23
"chockpoint models provided by [2 directly used
Table 2: (a) (Left) CIFAR10 (image) test-sct NLL; (b) (Right) LMD (music) test-sct NLL.
Teakling 1444 Curne
Piowes BBOC
303V.
MLL
1000
Log
Log
Figure 7: (a) Test MSE vs. logSNR for Gaussian diffusion; (b) Test PRL vs. logSNR for ItDPDM; (c) Training loss under Gaussian noise vs. PRL for ID musie; (d) Training loss under Gaussian noise vs. PRL for 2D images
loss across SNRs: MSE for ITDiff and Gaussian MSE (Figure 7 71), and PRL for ItDPDM (Figure 7b. PRL remains lower at low SNRs, consistent with the NLL improvements observed in symbolic music (Talle 2p), where ItDPDM achieves even larger NLL reductions, further demonstrating its suitability for discrete generative modeling.
Table 2 Similar trends
5.4
Performance Comparison: Generation Quality
Next, for evaluating generation quality of the generated images and music, we use domain-specific metrics: Structural Similarity Index Measure [43], and Fréchet Inception distance (FID) 44 for generated images: Fréchet Audio distance (FAD) 445, Consistency (C) [46], Mee-Spectrogram Inception Distance (MSID) [47] and Wasserstein Distance (WD) [48) for generated music. As shown in Figure 2 1DPDDM can generate realistic-looking natural images. Due to the limited computational budget available for training, the raw metrics for all models are lower than their reported values in IDDPM [8 and LTJ (20) The relative performance of the models gives the necessary insights: for image generation, IDDPM ( achieves the best FID, with 1tDPDM ranking second. In symbolic music, D3PM with categorical masking obtains the lowest FAD. ItDPDM outperforms LTJ for both image and symbolic music cases, by virtue of our exact likelihood estimation, as opposed to LTJ's variational relative entropy loss. Further details along with generated piano rolls in App. B.
Table 3: Domain speaii- generative quality metrics. Image: FID. SSIM: Audio: FAD, C. MSID. WD. FID values indicate dB increase (worse) from DDPM baseline
Baseline
Image
Audio
FID (dB)
SSIM
FAD
MSID
WD
DDPM [1
0.93
0.89
0.91
0.82
2.83
LTJ 20
0.30
0.90
0.66
0.92
0.71
2.23
D3PM 17
2.93
0.86
0.61
0.98
0.59
1.99
1DPDMM
0.18
0.91
0.64
0.94
0.67
2.14
5.5 Cross Training Paradigm
To isolate the benefits of Poisson diffusion and PRL objective, we perform cross-training: Gaussian + PRL and Poisson MSE. As shown in Table a), IDDDDM (Poisson PRL) yields the best NLL, confirming PRL's suitability for Poisson diffusion. Notably, Gaussian PRL also outperforms Gaussian MSE, suggesting PRL's broader effectiveness
discrete data. Moreover, ItDPDM converges faster and reaches lower loss than its Gaussian counterpart, as shown in (Figure dp d). We further validate the I-MPRL identity (Eq. 14) by comparing area-unde srea-under-los curves and final losses, finding close numerical similarities between the Poisson and Gaussian models, and aligning with the theoretical formulation.

---

Limitations and Future Work
Despite its theoretical strengths, performance benefits diverse discrete distributions, and competi- tive empirical results, ItDPDM remains a proof of concept and does not yet match state-of- oftie--f-th-hann performance in real-world generative tasks. As discussed in [5.4, these performance gaps are partly due to limited training and architectural tuning, with details provided in App. B. Additionally, logistic sampling parameters fixed priori without extensive hyperparameter tuning. We believe that longer training schedules (3000+ epochs), systematic hyperparameter sweeps (e.g., number of log- SNR steps), and largeted ablations could substantially improve ItDPDM's performance. Following prior (related) work [12 13
we also limit evaluation to unconditional generation on the training distribution; exploring conditional generation and robustness under prior misspecification or OOD generation remains promising direction for future work.
7 Related Work
Diffusion models are widely used in generative and restoration tasks 38
50
grounded in denois-
ing autoencoders [51), variational inference [2], and score-based SDEs
49
Recent works add
Nlormation-theorette insights [12), linking mutual information and MMSE
22
likelihood bounds.
Non-Gaussian extensions via annealed score matching [52 53 and score score-based SDEs [44] enhance theoretical rigor. In the discrete setting, Blackout Diffusion 551 and Beta Diffusion(56) use ir- reversible priors without tractable likelihoods. SEDD 57 uses score-entropy loss for token-level modeling but inherits ELBO-based approximations and lacks exact likelihood. LTJ[20] employs binomial thinning but is non-reversible and discrete-time. Our method overcomes these by us- ing reversible Poisson process, enabling bidirectional corruption, exact likelihood, and efficient continuous- -ie sampling. A more detailed discussion is provided in App. K.
8
Conclusion
We introduce ItDPDM, a diffusion framework for non-negative discrete data that combines a Poisson noising process with principled Poisson Reconstruction Loss (PRL), enabling exact likelihood estimation and discrete sampling without dequantization. ItDPDM achieves lower NLLL on both synthetic and real data, enhances modeling quality varied synthetic distributions, and delivers competitive results in image and symbolic music generation. Though a proof-of-concepl, ItDPDM lays strong foundation for distribution-robe discrete generative modeling, with applications in symbolic music, low-light imaging, and other count-based domains.
Broader Impact
ItDPDM provides a principled framework for modeling wide range of discrete, non-negative data distributions, including symbolic music, images, and skewed count data, without assuming any Gaussian or Sinomial-relatie structure. Its grounding in reversible Poisson process enables tractable likelihood estimation and efficient sampling, offering a robust alternative for domains with inherently discrete structure. This work may inspire future applications in low-light image reconstruction, scientific count data modeling, and generative modeling in constrained-de regimes.
References
[1] Prafulla Dhariwal and Alexander Quinn Nichol. Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems, 34:8780-8799 2021.
[2] Durk P Kingma, Tim Salimans, Jonathan Ho, and Xi Chen. Variational diffusion models. Advances in Neural Information Processing Systems, 34:21696 2021.
[3] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv 06.00.00....... 2022.
[4] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, Soroosh Mahdavi, Raphael Lopes, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Information Processing Systems, 12302-1112,,, 2022.
10

---

[5] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 10684-10695, 2022.
[6] Zhifeng Kong, Wei Ping, Jingguang Huang, Kexin Zhao, and Bryan Catanzaro. Diffwave: A versatile diffusion model for audio synthesis. arXiv preprini arXiv:2009.09761, 2021.
[7] Zahid Mulyukov, Geoffrey Dinsdale, Ben Glocker, and Michiel Schaap. Medical diffusion: Denoising diffusion probabilistic models for 3d medical image generation. arXiv preprint arXiv:2211.02386. 2022.
[8] Jonathan Ho, Chitwan Saharia, William Chan, Saurabh Saxena, Aravind Srinivas, Seyed Kamyar Seyed Ghasemipour, Raphael Lopes, Huimin Hu, Ariel Barzelay, Raphael Gontijo- Lopes, et al. Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv:2210.02330, 2022.
[9] Oni Avrahami, Oshri Tal, Shai Shahar, Sagie Benaim, and Tal Hassner. Retrieve first, generate later: Towards retrieval-ane retrieval-augmentee augmented text-to-image generation. arXiv preprint arXiv:2209. 15189, 2022.
[10] Bowen Jing, Jakob S Eismann, Victor Garcia Satorras, and Fabian B Fuchs. Torsional diffusion for molecular conformer generation. arXiv preprini urXiv:2206.01722 2022.
[11] Brian L Trippe, David Baker, Regina Barzilay, and Tommi Jaakkola. Diffusion probabilis- tic modeling of protein backbones in 3d for the motif-scaffolding problem. arXiv preprint arXiv:2206.04111 04112996..411, 2022.
[12] Xianghao Kong, Rob Brekelmans, and Greg Ver Steeg. Informatiin-theorrtiic diffusion, 2023. URL http:///axxxvv....aaaaao
03792
[13] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6851, 2020.
[14] Nanxin Chen, Yu Zhang, Heiga Zen, Ron J. Weiss, Mohammad Norouzi, Najim Dehak, and William Chan. Wavegrad 2: Iterative refinement for text-to-speech synthesis, 2021. URL https://arxiv org/ abaaaaas/...///// 09660
[15] E. Hoogeboom al. Categorical diffusion with gumbel-softmax.
arXiv preprint
arXiv:2107.08444 2021.
UBLLtttpp///rxxv.oor///assss 08447
[16] New York City Taxi and Limousine Commission. New york city taxi trip data, 2009-2018, 2019. URL htt.hhtts://// ptttps://www.iqqee icpsr. www.mmmchhh.m.... httbb/edd/eebbiisseeeee http:////e.psseaaaas//////////// sttddess/dees5
[17] Jacob Austin, Daniel Johnson, Jonathan Ho, Daniel Tarlow, Rianne van den Berg, and Pieter Abbeel. Structured denoising diffusion models in discrete state-spaces. Advances in Neural Information Processing Systems, 34: 14:17981-17993, 2021.
[18] Cindy M. Nguyen, Eric R. Chan, Alexander W. Bergman, and Gordon Wetzstein. Diffusion in the dark: A diffusion model for low-light text recognition, 2023. URL httph:ttt///a:ttt org/ abs bas//303.000 04291
[19] Matthias Plasser, Silvan Peter, and Gerhard Widmer. Discrete diffusion probabilistic models for symbolic music generation, 2023. URL http::ttts:///ttt org/ abs/ 09489.
[20] Tianqi Chen and Mingyuan Zhou. Learning jump: Thinning and thickening latent counts for generative modeling, 2023, URL https://arxii: http//awwwi..aaaaaaa///////oo/aa httrg/abs/2305.. 18377
[21] Arindam Banerjee, Srujana Merugu, Inderjit S. Dhillon, and Joydeep Ghosh. Clustering with bregman divergences. Journal of Machine Learning Research, 6: 1705-1749, 2005.
[22] Dongning Guo, Shlomo Shamai, and Sergio Verdú. Mutual information and minimum mean- square error gaussian channels. IEEE Transactions on Information Theory, 1614)122 2005.
11

---

[23] Sergio Verdú. Poisson communication theory. 2004. URL xtg/CorpusDD:2950 :297862
[24] I. Bar-David. Communication under the poisson regime. IEEE Transactions on Information Theory, 15(1):31-37, 1969. doi: 10. 109/TIT 196..14423.0 1054238.
[25] S. Shamai and A.D. Wyner. A binary analog to the entropy- inequality. IEEE Transactions on Information Theory, (116)):428-110
1990. doi: 10. 1.0.11000 (09/18.5992)
[26] Paul Dupuis and Richard S. Ellis. The large deviation principle for general class of queucing systems i. Transactions of the American Mathematical Society, 347(8)2668--755, 1995. ISSN 00029947. URL htthhttp:// /www wwwpwwww...t..ooooooo..p....www. jstor. org/ http//saaleee1111/
[27] R. S. Liptser and A. N. Shiryayev. Statistics of Random Processes II: Applications, volume 6 of Applications of Mathematics. Springer, Berlin, Heidelherg, 2nd edition, 2001. ISBN 978-3- Part of the Stochastic Modelling and Applied
540-21292-6. doi: 10.1007/978-3-11 3-66----6--000 6-6666577--4 Probability series.
[28] A. Banerjee, Xin Guo, and Hui Wang. On the optimality of conditional expectation as a bregman predictor. IEEE Transactions Information Theory, 51(7):2664-266)) 2005. doi: 10. 00.10099711:200588511555
[29] Rami Atar and Tsachy Weissman. Mutual information, relative entropy, and estimation in the poisson channel, 2010. URL BRLL/tttss///rxxx.orr////////// wwtp:////.bbbbbbbbbaaa/aaaaaaaaa 0302
[30] Persi Diaconis and Donald Ylvisaker. Conjugate priors for exponential families. The Annals of Statistics, 7(2):269-28)) 281, 1979. ISSN 00905364, 21688966. URL http: /tp:://wwww jstor org/ sttable29588008
[31] J. J. Good. The population frequencies of species and the estimation of population parameters. Biometrika, 40(3-4):237-22 1953. doi: 10.1093/biomee440.3-.222
[32] Herbert Robbins. An empirical bayes approach statistics. In Proceedings of the Third Berkeley Symposium on Mathematical Statistics and Probability, volume 1, pages 157-163. University of California Press, 1956.
[33] Tsachy Weissman, Erik Ordentlich, Gad Seroussi, Sergio Verdu, and Marcelo J. Weinberger. Universal discrete denoising: Known channel. IEEE Transactions on Information Theory, 51 (1):5-28, 2005.
[34] Rob Brekelmans, Vaden Masrani, Frank Wood, Greg Ver Steeg, and Aram Galstyan. All in the cxponential family: Bregman duality in thermodynamic variational inference, 2020. URL https://arxiv. org/
00642
[35] Yosihiko Ogata. A monte carlo method for high dimensional integration. Numer: Math., 55(2):137-157, March 1989. ISSN 0029-599X. doi: 10.1007/BF01406551.. URL https //do1.org/10. 1007
wttp://...000/BOOO0 1007/BF01406611
[36] Andrew Gelman and Xiao-Li Li Meng. Simulating normalizing constants: From importance sampling to bridge sampling to path sampling. Statistical Science, 1382):163-185, 1998. ISSN 088834237, 21 2888455. hRL /tttp://www... jstor. org/ sttss/aabbeee//6//
[37] Liming Wang, Miguel Rodrigues, and Lawrence Carin. Generalized bregman divergence and gradient of mutual information for vector poisson channels. In 2013 IEEE International Symposium Information Theory, pages 454 458, 2013. doi: 10. 10199///T.21133666000000
[38] Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. International Conference on Machine Learning, pages 8162-8171, 2021.
[39] Alex Krizhevsky, Learning multiple layers of features from tiny images. sttps://wwzr toronto. wwwioo..ddddrriireee cifar httl 2009.
[40] Christoph Vogl, Jesse Ranzijn, Ian Millwood, and Gabriel Essl. The lakh midi dataset. https
Cccolimaafeel..compprjeeecc....e 2017.
12

---

[41] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. ArXiv, abs/1505.04597, 2015. URL https://api semanticscholar
[42] Gautam Mittal, Jesse Engel. Curtis Hawthorne, and Ian Simon. Symbolic music generation with diffusion models, 2021. URL BtRL/https.//axii.orrr/////
16091
[43] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Ecro Simoncelli. Image quality assessment: From error visibility to structural similarity. IEEE Transactions Image Processing, 13(4): 600-612,40 2004.
[44] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge a local nash equilibrium. In Advances in Neural Information Processing Systems (NeurIPS), volume 30, 2017.
[45] Kevin Kilgour, Robin Clark, Karen Simonyan, and Matt Sharifi. Fréchet audio distance: reference-free metric for evaluating music enhancement algorithms. In Proceedings of Interspeech, pages 2350-2354, ISCA, 2019.
[46] Gautam Mittal, Jesse Engel, Curtis Hawthorne, and Ian Simon. Symbolic music generation with diffusson models. In Proceedings of the 22nd International Society for Music Information Retrieval Conference, 2021. htt..httss::/// //axxivv org/ abs/ wbsp///3.... 16091
[47] Li-Chia Yang, Szu- Chou, and Yi-Hsuan Yang. A study of evaluation metrics for music generative models. Proceedings of the 21 International Society for Music Information Retrieval Conference (ISMIR), pages 175-182, 2020.
[48] Gabriel Peyré and Marco Cuturi. Computational optimal transport. Foundations and Trends in Machine Learning, ((5-6))355-600) 2019.
[49] Yang Song, Jascha Sohl-Dickstein, Durk P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint 1.Xi::2011.134500 2020.
[50] Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Luc Van Gool. Repaint: Inpainting using denoising diffusion probabilistic models. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11461-11471, 2022.
[51] Pascal Vincent. connection between score matching, denoising autoencoders and density estimation. Advances in Neural Information Processing Systems, 24, 2011.
[52] A. Hyvärinen. Estimating the gradients of the log-density. In Advances in Neural Information Processing Systems (NeurIPS), 2005. Utt.httsss://ppeers wwappperr..iis...mmmmme cecips.ccppppe wwwpppppe//000////
[53] Y. Song and S. Ermon. Generative modeling by estimating gradients of the data distribution. In Proceedings of the 32nd Conference on Neural Information Processing Systems (NeurIPS 2019), 2019. URL https: //tp:er//ppp.....e nips cc/ wwtpppppp//ptt///eceee 2019
[54] Thomas Dockhorn, Hyungjin Chung. Stefano Ermon, and Gunnar Rätsch. Score-based genera- tive models for high-dimensional inverse problems. Advances in Neural Information Processing Systems, 35:17506-17516 2022.
[55] Javier E. Santos, Zachary R. Fox, Nicholas Lubbers, and Yen Ting Lin. Blackout diffusion: Generative diffusion models in discrete-state spaces. In Proceedings of the 40th International Conference on Machine Learning (ICML) volume 202 of Proceedings of Machine Learning Research, Honolulu, Hawaii, USA, 2023. PMLR. URL httph: TtLLhhtts:///roooooo htttp:://proceedinggs www.. press/ v202/ 20002santos23a.. html
[56] Mingyuan Zhou, Tianqi Chen, Zhendong Wang, and Huangjie Zheng. Beta diffusion, 2023.
URL
07867
[57] Aaron Lou, Chenlin Meng, and Stefano Ermon. Discrete diffusion modeling by estimating the ratios of the data distribution, 2024. http:s http///axvvvv..
httpaaawwiiaaaaaaaaaaa/aa wwrg/bb///aa 16834.
13

---

[58] FluidSynth Developers. Fluidsymth: Software synthesizer based on the soundfont 2 specifica- tions, htttps://www..r http://wwww.uuiddttt.tt. org 2025. Accessed: 2025-05-19.
[59] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreason- able effectiveness of deep features perceptual metric. CVPR, pages 586-595, 2018.
[60] Jesse Engel, Lee Hantrakul, Chenjie Gu, and Adam Roberts. Gansynth: Adversarial neural audio synthesis. International Conference on Learning Representations (ICLR), 2019.
[61] Alex Dytso and H. V. Poor. Estimation in poisson noise: Properties of the conditional mean estimator. IEEE Transactions on Information Theory, 66(7):4304-4323, 2020. doi: 10.1109/ TIT.2020.2979778.
[62] Volodymyr Braiman, Anatoliy Malyarenko, Yuliya Mishura, and Yevheniia Anastasiaa Rudyk. Properties of shannon and rényi entropies of the poisson distribution as the functions of intensity parameter, 2024. https://:
org/ abs /203.. 08805
[63] Kenneth S. Miller. Mathematical Statistics: Basic Ideas and Selected Topics. Pearson Education, 2nd edition, 2006. ISBN 9780131520780.
[64] M. Abramowitz and A. Stegun. Handbook of Mathematical Functions with Formulas, Graphs, and Mathematical Tables. Dover Publications, 1964.
[65] H. Chernoff, Measure of Asymptotic Efficiency for Statistical Tests. Annals of Mathematical Statistics, 1952.
[66] M. Davis. Markov Models and Optimization. Chapman & Hall, 1993.
[67] T. Duncan. On the calculation of mutual information. SIAM Journal on Applied Mathematics, 1970.
[68] D. Snyder. Filtering and detection for doubly stochastic poisson processes. IEEE Transactions on Information Theory, 1972.
[69] Haoran Sun, Lijun Yu, Bo Dai, Dale Schuurmans, and Hanjun Dai. Score-based continuous-time discrete diffusion models. arXiv preprint arXiv:2211.16755 2022.
[70] Mingyuan Zhou Tianqi Chen, Zhendong Wang. and Huangjic Zheng. Beta diffusion. In
Advances Neural Information Processing Systems, 2023.
[71] Emiel Hoogeboom and Tim Salimans. Blurring diffusion models. arXiv preprint 20095572 2022.
[72] Jente Vandersanden, Sascha Holl, Xingchang Huang, and Gurprit Singh. dige-prreserving noise for diffusion models. arXiv preprint arXiv:2410.01540, 2024.
14

---

Appendix
A Numerical Details
A.1 MPRL Upper Bound
This part delves into the derivation of Eq. (18). We first express the expected loss terms of the optimal estimator (using shorthand notation subsequently::
X
X
X
E|((X, (XX, 7))] E
X Xoog
X+ x(X,)) = Xlog
E
X log
X
X
((,,))
X
X
(20)
Using the law of iterated expectation gives:
E|((X, = mprl(7) + ETU(X",
The second term above denotes the estimation gap, and rearranging the terms, we get:
E P(=)
/
(EV((X,)) - EX(("", dy.
Using Jensen's is inequality here (based on the properties mentioned in Lemma 1.. we have:
= X)] v - F. = log E|X"1
E[X" X -I([[X)),))
(21)
X
Now, using the relation from Lemma 2 gives us:
mpri(7) dr <
EI((X, B[X])] dr.
(22)
We obtain a more elegant bound in terms of our suboptimal neural denoiser by dropping the negative term:
log P(x)] P(=) / /oa mprl(7) dy S / E(((XX8)] dy.
(23)
A.2 Numerical Integration.
This section outlines the effective computation of integral from 19 We first use importance sampling rewrite the integral as expectation over distribution, ((7). allowing for unbiased Monte Carlo estimation. This leads to our final numerical approximation of the loss function Ep(s) log) S L, where
C= 1 (((x)) (Ta) 1(,,,,,
We propose paradigms for numerical integration: Logistic and Uniform Integration, respectively.
Logistic Integration. In Gaussian diffusion models, the log-SNR integral is approximated via importance sampling with truncated logistic distribution. The integrand, shaped by mixture of logistic CDFs influenced by data covariance eigenvalues X; is captured by matching the empirical mean y and variance of - log A; with integration bounds [μ 4s, f + 4s]. Samples drawn via the logistic quantile function are weighted by 1/g(cz (a) to prioritize critical regions, reducing variance.
Uniform Integration. This simpler numerical method discretizes the log-SNR range [a1 a2] into uniform grid, applying trapezoidal or Riemann- sum integration without assuming an underlying distribution. While simple, efficiency depends on grid density for broad ranges, favoring ease over optimal sampling. The predefined range is [-28, 37) with uniform sampling.
B Experimental Details
B.1 Training Details (contd.)
For fair comparison, we train both CIFAR and LMD models from scratch for 600 epochs. The training starts with learning rate of 2 x 10-5 using the Adam optimizer. We adopt an 80-20 train-test
15

---

split for evaluating likelihoods. For image generation, we use a UNet-based model 41]. while for music generation, we employ the DenseDDPM and wnwvolutionaltrrnsfooooo 17 based models for the continuous embeddings (DDPM-style) and discrete domain (D3PM 17]) respectively. The training procedure ensures consistency across both domains, facilitating a meaningful comparison of their performance. It is to be noted that we train all of the models from scratch, owing to lack of pre-trained Poisson diffusion baselines, to ensure fair comparison. Because of compute resource constraints, we train the models upto 600 epochs, which falls short of the usual amount of training required to achieve peak performance (e.g., (2) trains their models for 3600 epochs), We also restrict ourselves to 100 logSNR values per image/ / music sample, and restrict the number of denoising steps used in the DDPM/ / D3PM baselines to 100 as well (instead of 1000), to ensure fair comparison. Thus, although the relative performance of the models is preserved, the absolute values of the metrics underperform those presented in DDPM 13 and [0]
B.2 Data and Model Normalization
We experimented with various schemes for data (Dn) before passing it through the noisy channel and for model inputs (Mn) post-noising. CIFAR-I0 10 data is normalized to [0, 11, 2], [0, [-5],[-1,, Lakh MIDI to [0, 1], [1, 2]. [0, 90), [-1, 1j. Poisson channels cannot handle negatives and since zero inputs yield zeros, we shift imputs by 10-6, For Gaussian noising, model normalization used [0. or [-1, 11, while Poisson noising used only [0. 1]. The best results were achieved with [-1, 1] (Gaussian) and [1, 2] (Poisson) for Dn, and [-1, 1] (Gaussian) and [0. 1] (Poisson) for Mn. Among the integration paradigms used, logistic integrate yielded the best empirical results, and the loc and scale parameters obtained for the mid-integral range were (6, 3) for Gaussian noising and (-1, for Poisson noising.
B.3 Denoiser Architecture
For CIFAR-10 images, we employ U-Net architecture 41 with residual blocks and self-attention layers. The encoder comprises four downsampling blocks (convolution - GroupNorm - SiLU) that reduce spatial resolution from 32 32 to 4 x 4. followed by a bottleneck with self-attention at 8 x 8 resolution. The decoder mirrors the encoder via transposed convolutions and skip connections. For symbolic music synthesis Lakh MIDI, we use DenscDDPMI33] -basedd architecture and a convolutional transformer 19 19] -based model, for the continuous DDPM modeling and the discrete [9 modeling respectively. For the continuous modeling, we adapt the DenseDDPM architecture from 42 It first projects the input latent vector to an MLP hidden size (default 2048) with single Dense layer, then runs it through 3 residual MLP blocks whose weights are modulated by 128ddimensional sinusoidal embedding of the diffusion timestep t. After these conditioned residual blocks, it applies LayerNorm and final Dense layer that maps back to the original latent dimensionalityy yielding the denoised output. For the discrete modeling, we adapt an NCSN++ backbone (53) with Convolutional Transformer encoder [9). The architecture includes a 512-dimensional embedding layer, six transformer layers with multi-head attention (8 heads) and positional encodings, and dime-dependent noise conditioning.
B.4 Symbolic Music Dataset Cleanup
We utilize the cleaned Lakh MIDI dataset 40) loading note sequences from .npy files with original shape (Ir, 1024). For training, sequences are partitioned into individual ID vectors of shape (1,1024), representing discrete musical events. So, our method directly models symbolic music as discrete 1D note sequences using Poisson diffusion, avoiding hybrid architectures or preprocessing.
B.5 Domain-Specific Metrics
To evaluate the generation quality of our model across image and audio domains, utilize established domain-specific metrics that quantify fidelity, diversity, and structural realism. Below, we provide deseriptions and implementation details for each metric employed in our evaluation.
Image Metries All image- saaggegenneration metrics were computed on 40,000 randomly selected ground- truth images from the CIFAR-10 10 test split and 40,000 model-generated samples. Fréchet Inception Distance (FID) was evaluated with the PyTorch torch-fidde fidelity package (Inception-v3 network, pool3 activations).
16

---

Structural Similarity Index Measure (SSIM) 43 SSIM measures the similarity between two images by comparing their luminance, contrast, and structure. It is defined as:
(2p- 220 02) SSIM v) = (12 m ( + +22
where j and denote mean and standard deviation over local image patches. Higher SSIM indicates better perceptual similarity.
Fréchet Inception Distance (FID)
FID evaluates the distance between real and generated image distributions in the feature space of a pretrained Inception network. is calculated
47
FID = ||,r - 1gll + Σ,
-(E, -gE,)
where (jer, 2x) and (Mg, Mn) are the means and covariances of the feature embeddings of real and generated samples.
Audio Metrics. All audio-based metrics are computed using 10,000 ground-truth samples and 10,000 generated samples per model. enable consistent audio evaluation, we first convert model-generated .npy files to MIDI format using the pretty_me midii library. These MIDI files are then rendered to WAV audio using FluidSynth (58) with the FluidR3_ GM soundfont, ensuring uniform timbre across all samples. All tools and dependencies are managed within automated evaluation pipeline. This standardized conversion procedure ensures reproducibility and fair comparison of audio metrics across all models.
Analogous FID, FAD computes the Fréchet distance between embeddings of real and generated audio, extracted via VGGish model pretrained for audio classification. reflects pereeptual similarity in the feature space and calculated similarly to FID.
Fréchet Audio Distance (FAD) 45
Consistency (C): To evaluate sequence-lever realism, employ framewise self-similarity based on overlapping Gaussian approximations of pitch histograms. Specifically, we use the overlapping area (OA) from [46), applied pitch only (since duration is fixed in setup). For sliding 4-measure windows with 2-measure hop:
#1 V2
c- M2
OA(K 1) = erf
+ erf
222 02
The resulting pitch OA values are compared to ground-truth sequences via:
|Mo - MCT]
C = max
0,
PGT
0, lo &arl &e
Var max
Consistency (C) measures global similarity ground truth, while variance (Var) captures generation diversity. High implies structured, music-like pitch transitions.
Mel-Spectrogram Inception Distance (MSID) 47 MSID adapts FID for audio by computing the Fréchet distance over features extracted from Mel spectrograms. The key steps include:
Convert generated .npy files to MIDI and synthesize audio using FluidSynth. Compute 128-band Mel spectrograms (16kHz, FFT=2048, hop=512), as outlined in
Extract features using a VGG16-based architecture trained on audio (VGGish).
Compute MSID using: MSID || - 11yl22 Tr(E, E, - -(E,,,
MSID captures both spectral and perceptual differences, correlating with human audio quality judgments.
17

---

Wasserstein Distance (WD) 48 WD quantifies the distance between the token distribu- tions of real and generated symbolic music. We compute weighted Wasserstein distance that prioritizes important token types (e.g., binary onsets or active pitches):
W (p, q)
inf yell(p.g)
Paaw)oo (c(x, y) w(x v)
Weights are assigned based on token values: 0.2 for Os, 0.5 for 1s. 1.0 for others. Tokens normalized and reshaped as needed. Lower WD values indicate better alignment of pitch activation distributions.
In addition to the core domain-speciff metrics described in Appendix B.5 we include the following complementary metrics used for additional analysis presented in Table These metries help analyze fine-grained perceptual and structural properties of the generated data.
Images:
Learned Perceptual Image Patch Similarity (LPIPS) [59 LPIPS measures perceptual similarity by computing the distance between deep features extracted from pretrained vision networks (e.g.. VGG, AlexNet). It is defined as:
LPIPS(I, Σ E
1 H.W,
Σ
1ew 2 (of (h, w))
where of and of are feature activations from layer 1, and wy are learned weights. Lower LPIPS values indicate higher perceptual similarity between generated and reference images.
Audio:
Spectral Convergence (SC): SC quantifies the relative difference between the magnitude spectra of real and generated audio:
|
Seeel |F
SC
S|er
||
where Sgen and Sief are the STFTs (Short-Time Fourier Transforms) of generated and reference audio, and || denotes the Frobenius norm. Lower SC suggests higher spectral alignment.
Log Mean Spectral Distance (LMSD): LMSD captures differences in log-scaled spectral magnitudes and is defined as:
LMSD
1 T
Σ log(e + [t)) log(e + (Sar(t))))
where is a small constant to ensure numerical stability, and the summation is over time frames Lower LMSD implies improved perceptual quality in frequency response.
Variance (Pitch Histogram Diversity):
42 As described in Appendix B.5 we also
compute the pitch variance metric (Var) to measure structural diversity in symbolic music:
1oo0 carl a
Var max 0.
Higher variance indicates greater distributional diversity while maintaining similarity to ground truth statistics. Together, these metrics offer comprehensive, multi-faceted eval- uation of image and audio generation quality, balancing fidelity, diversity, and perceptual structure.
Mel Spectrogram Computation Parameters:
For the listed audio-based metrics (FAD, MSID, SC, LMSD), we first convert generated symbolic music into waveform discussed earlier 58 and compute Mel spectrograms with the following parameters:
18

---

Sampling rate: 16 kHz chosen to balance temporal resolution and frequency coverage for symbolic music.
FFT size: 2048 - defines the window size for frequency analysis. This size gives sufficient frequency granularity (=7.8 Hz per bin at 16zz.).
Hop length: 512 determines the stride between successive STFT windows, corresponding to 32 ms hop (suitable for music temporal structure).
Mel bands: 128 provides perceptually motivated representation of frequency, empha-
sizing resolution in lower frequency ranges where musical structure is denser.
These parameters consistent with best practices in neural audio synthesis [60] and ensure compatibility with pretrained perceptual models like VGGish.
Additional Metries:
Table 4: Auxiliary generative quality metries, Image: LPIPS; Audio: SC, LMSD, LPIPS (Mel), Var
Baseline
LPIPS (Img)
SC
LMSD
LPIPS (Mel)
Var
IDDPM 38
0.17 + 0.05
1.56
9.99
0.38 + 0.10
0.81
20
0.18 +
1.51
9.81
0.33 + 0.10
0.87
D3PM 1V
0.29 + 0.09
1.41
9.63
0.28 + 0.09
0.90
IDPDM
0.18 + 0.08
1.49
9.71
0.30 + 0.09
0.85
B.5.1 Visualizing generated music samples:
Individual ItDPDM Samples: To examine local model behavior, we present isolated piano roll visualizations of individual samples (see Figure 8). Each plot shows the temporal and pitch structure of a single sequence, with color indicating note velocity. These visualizations enable detailed inspection of rhythmic patterns, pitch range, note density, and artifacts.
For example, ItDPDM-generrtted samples exhibit consistent pitch contours and relatively uniform spacing. occasionally disrupted by outlier notes sparse regions. Such plots help diagnose issues like over/under gentrrrignnrrtion, discontinuities, or anomalies, and complement the broader comparisons across models.
Prisow
Figure 8: Isolated piano roll visualizations of four ItDPDM-generated samples. Each plot shows pitch over time, with note velocity indicated by color intensity.
Qualitative comparison: To qualitatively observe the generative performance of our models, we visualize representative samples piano rolls in Figure Each row presents different generated
19

---

sequence, with columns corresponding to different models: DDPM (left), ASD3PM (center), and IdDPDM (right). Each piano roll plot depicts note pitch (vertical axis) over time (horizontal axis), with intensity indicating note onset.
DDPM (left): Samples from DDPM display high variability in pitch and rhythm, with note events appearing scattered and less structured. While diverse, these outputs often lack recognizable musical motifss or rhythmic regularity, indicating that the model struggles capture long-range musical structure.
ASD3PM (center): ASD3PM outputs, derived from perturbed ground truth MIDI sequences, exhibit strong rhythmic and melodic coherence. These samples closely mirror the structure of real music, featuring sustained motifs, consistent phrasing, and regular timing. This visual consistency aligns with the model's s design, which prioritizes fidelity to the data manifold.
ItDPDM (right): Samples from 1tDPDM demonstrate improved musical structure over DDPM. While some randomness remains, many outputs show rhythmic grouping, pitch contours, and repeating patterns, suggesting the model's ability learn and replicate fundamental elements of musical organization. Overall, the visualizations highlight key differences in generative behavior. ASD3PM achieves the highest structural fidelity, followed by ItDPDM, which balances diversity with coherence. DDPM produces varied outputs but lacks the structured rhythmic and melodic features observed in the other methods. These qualitative findings complement our quantitative results, offering insight into how each model captures musical dependencies in time and pitch.
Figure 9: Piano roll visualizations of generated samples from DDPM (left), ASD3PM (middle), and ItDPDM (right). Each row corresponds to a particular random sample. Higher vertical positions represent higher pitches.
To further assess how the generated music matches the statistical properties of the training data, we also compare the generated pitch distributions with the ground truth. Figure [O] shows the histogram of MIDI pitch values for ItDPDM generated sequences alongside the empirical distribution from the training data with a close alignment indicating that the model captures global pitch statistics, such as register, range, and note density. Another observation is that in the generated samples, the note velocity is slightly amplified in comparison to the ground truth distribution.
C Synthetic Benchmark Details
C.1 Diserete benchmark details
We evaluate model performance a suite of synthetic univariate discrete distributions designed to challenge generative models with features such as overdispersion, multimodality, sparsity, and skewness. All distributions take values in No and non-negative.
20

---

Pitch Distribution
16000
Ground Truth
Generated
14000
12000
10000
0000
8000
2000
60
80
100
Pitch (MID) note number)
Figure 10: Comparing pitch distributions for ground truth and 1dDDDM generated samples
Poisson Mixture (PoissMix): This is bimodal mixture of Poisson distributions:
Poisson (A 1) + 0.9 Poisson(A = 100),
producing highly skewed and dispersed distribution with modes at both low and high counts, simulating tasks where most values are large but minority remain near zero.
Zero-Inflated Poisson (ZIP): To simulate data with an excess of zeros, we use a zero-inflated Poisson distribution: which samples zero with probability Top and otherwise follows Poisson distribution:
To (1
To) -
==0
P(K) - mp)
A
with To = 0.7, X = 5.
k>0
This models structured sparsity common in count data with dropout.
Negative Binomial Mixture (NBinomMix): This is mixture of two negative binomial distribu- tions: 0.8 NB 0.9) 0.2 NB((1,, 0.1), where the first mode has high probability near zero, while the second exhibits broader dispersion. It introduces skew and multimodality in count data.
Deta--egattiveBiing Binooiall (BNB): The BNB distribution integrates Beta prior over the success probability p of the negative binomial:
P()) =
NB(k; Beta(p; b 1.5) dp, k E No-
We use parameters a 0.5, 1.5, and 5, inducing heavy-tailed count distribution with long-range dependencies.
Zipf Distribution: This power-law distribution is defined as: - ((a) x= 1.7
P(x)
where ((ar) is the Riemann zeta function. Zipf distributions model naturally occurring frequencies, such word counts node degrees.
Yule-Simon Distribution:
The Yule-Simon distribution is defined as:
P(k) = Bik, = [(k)T(p (p+1)
r(k+ p= 2.0, k E N21
21

---

where B is the Beta function and I' is the gamma function. is used to model data with power-law decay, often arising preferential attachment or reif-reinnoriing (e.g. tich-ge-rricher) processes. These distributions form challenging testbed for evaluating generative performance on discrete, non-negative data.
Table summarizes the discrete synthetic benchmarks used in our study, Each distribution is selected represent different pathological regime-bi-modality zero-inflation, overdispersion, or power-law behavior-intendle to stress PMF concentration and test model robustness. For completeness, we specify parameter values used in generation and annotate tail behaviors to clarify their impact on sample complexity and generalization.
Distribution
Parameters
Tail behaviour
Poiss.Mix
= 100)
bi-modal spike at 0 Var > power-law
Zero-Inflated Poisson
To = 0.7, =
NBinomMix
(r.p)= = ((1,0.9) (10,0.1)
BNB
0.5, b= 1.5,,
Zipf
a= 1.7
~ xx
heavier than Zipf Table 5: Specification of discrete synthetic benchmarks. All distributions are heavy-tailed, zero-
Sale-SSimon
p= 2.0
inflated, or multi-modal, stressing PMF concentration.
C.2 Training Details & Metrics
In addition to the ConditionalMLPP, a timestep embedding network additionally projects diffusion steps into 64-dimensional space using SiLU activations. Models trained for 200 epochs using the Adam optimizer (7 = 10 10-3 31 = 0.9, 32 = 0.999) with a batch size of 128. The Gaussian DDPM employs linear noise schedule 3, E [10-4,2 10 over T = 100 diffusion steps. Our ItDPDM framework adopts a linear gamma schedule 7t E [1.0, 0.0) over the same number of steps.
For Poisson diffusion, the initial sample mean set to 10.0.
Wasserstein-1 distance Wasserstein- distance 48 between two univariate distributions p and is P(x) Q(x)] dr, where π (2x y) is joint cou-
defined as: Wi (p, g) = =fx= x y/ dz v) pling of p and and P. Q are their respective cumulative distribution functions (CDFs). When p and are empirical distributions of the same size n, this reduces to: W (i q) = ||sort( () - sort (Y)l
where X, Y E R are the sorted samples from and g.
For each empirical distribution of 50,000 generated samples over 5 runs, say Pgen (with Preat denoting the empirical distribution of 50,000 test samples), we compute the Wasserstein-1 distance (WD) 48 and negative ligeliie (NLL) as:
WD Wi(Prest Poen), NLL
Σ
log Prem (2:)
(24)
Thestt
where x; denote the held-out samples.
C.3 Probability Mass Function Estimation:
For discrete distributions, estimate the empirical probability mass function (PMF) p(x) from generated samples {ar) using a histogram- approach with binning over a finite support X' 1, (0.,,..., K}:
1 N
p(x)
Σ T(x T(;=
(25)
where II() the indicator function and K is the truncation value. We set K 50 across all experiments to standardize the support. To reduce sampling noise and better visualize differences across models, we additionally compute smoothed PMF estimate using a discrete Gaussian kernel:
Pamooth (x)
Σ
Kn (x - xz:),
(26)
22

---

where (.) is Gaussian kernel defined on the integer lattice:
(x)x) = exp
(27)
Z
2/2
ensuring that K₁, sums to over the support. The bandwidth h is selected empirically per distribution to balance smoothness and fidelity to the empirical histogram. To assess variability in PMF estimation, non-parametric bootstrapping. Specifically, we generate 10 bootstrap resamples of the model outputs, re-estimate the (smoothed) PMF for each, and plot the mean = standard deviation across these resampled estimates. Fach plot includes Fig. inccldees: a) ground-truth PMF (when known), and b) the empirical unsmoothed and smoothed PMFs for each model (e.g.. ItDPDM, DDPM, LTJ), with any shaded error bands reflecting bootstrap variability.
with normalization constant Z = Er'ex exp
24
also compute error bands via
Implementation Details:
Aspect
Details
Sample size
N = 10,000 samples per model and distribution
Support Smoothing bandwidth
.. 50} for discrete; bounded æ for continuous h tuned per distribution (discrete); KDE bandwidth default resamples per model for uncertainty estimation
X {0, ......
Bootstrap
Visualization
True distribution, model estimates, and error bands plotted
Table Summary of implementation settings for PMF and PDF estimation.
Zoomed-in look at PMF plots: Building the analysis in Section 5 Figure 11 and Figure 12 provides magnified view the fule-Simon and Zipf fits produced by each model. ItDPDM exhibits the closest alignment to the target distribution, particularly in the critical low-support region.
0.2
0.2
0.1
0.1
0.0
0.0
10
0
10
Figure 11: Zoomed-in Yule-Simon fits
Figure 12: Zoomed-in Zipf law fits
C.4 Non-negative Continuous Scenarios
As stated earlier, extend our analysis six skewed continuous densities: Gamma, NormNo Lomax, Half-Cauchy, Half-t, Weibull, (along with Beta and Uniform distributions) as outlined in this section. Our goal here is to assess how well generative models capture asymmetry, concentration, and long-range dependencies in continuous data.
Descriptions and parameters:
Gamma Distribution: The Gamma distribution is defined by shape parameter 'a' and a scale parameter '0':
p(x) T(a)ou J."
n/0 x 2 0.
We use a 0.5, 0 = 2. which produces a sharp mode near zero and long right tail. Gamma distributions are commonly used to model wait times, energy release, and insurance claims-making them valuable for stress-testing the model's handling of high variance and positive skew.
Log-Normal Distribution: A log-normal distribution arises when the logarithm of variable is normally distributed:
2 exp (- (log x 11)2
p(x)=
1 0.
2.2
23

---

We use 0, 1.5, producing a distribution with significant positive skew and heavy tails. Log- normal models appear in financial returns, biological measurements, and natural language modeling. where multiplicative effects dominate.
Lomax Distribution: Also known the Pareto Type distribution, the Lomax is defined
p(x) (1 (c+1)
x x20.
We use e = 2.0, = 1.0, resulting in a fat-tailed distribution often used in reliability engineering and modeling rare, catastrophic events. It challenges models to capture high-probability mass near with occasional large outliers.
Half-Cauchy Distribution: The Half-Cauchy is the positive part of Cauchy distribution:
2
p(x) =
x' 0.
TTS 1+
With 1, this distribution has undefined mean and variance, and extremely heavy tails. It is commonly used as a prior in bierarchical Bayesian models due to its robustness to outliers. Half-t Distribution: The Half-r distribution is the absolute value of Student's r-distributed variable:
p(x) 2 ((x; v, 0, 0.
We use y 3, yielding distribution with heavy but finite tails. This is another robust prior used in Bayesian inference, particularly for variances in hierarchical models, where it prevents over-shrinkage.
Weibull Distribution: The Weibull distribution, defined by shape & and scale X, is given by:
-(z)= - e (x/A)* 2 O.
We use k = 1.5, A = L, producing distribution with increasing hazard rate and moderate skew. This widely used in survival analysis, material failure modeling, and wind speed distributions.
Beta Distribution (bounded support): Though often used on [0, 1). the Beta distribution provides diverse shapes depending on the parameters:
I(a + +) p(x) [(a)[(b) x -- (1-1))11111 x) 0 x $
We use a = 2, = 2, leading to density concentrated near zero The Beta distribution tests the model's ability learn bounded distributions with asymmetric mass concentration, relevant in probabilistic modeling and reinforcement learning. A key limitation to note here is that in case of asymmetrrc/skewed beta distributions, all the models notably fail to learn the distribution.
Uniform Distribution (flat support): The uniform distribution provides a baseline for bounded, structureless densities:
1
p(x)
b-a a<
We set 0, 1, resulting in a constant density over the unit interval. Although simple, it serves sanity check for model calibration and ability avoid mode collapse under flat distributions. Together, these distributions offer a comprehensive testbed for evaluating generative modeling under varied support, skewness, and tail behavior. They also represent common scenarios encountered in practice, ensuring relevance to real-world generative tasks.
Results:
Table 7 compares the Wasserstein distance for all the continuous cases, and in the continuous we omit NLL values as they can be overly sensitive to skewness and outliers, making them unreliable for fair comparison. More critically, whereas the true NLL in continuous distributions often be negative while our discrete estimator cannot possibly yield negative NLL.
For each distribution, we visualize the estimated PDFs from all models alongside the true density. Figure 13 summarizes the results across all eight distributions, providing qualitative comparison of how closely each model recovers the underlying data- gettrrenng process.

---

Table 7: WD for continuous cases (1 lower is better). Bold indicates best.
WD
Distribution
DDPM
1DPDD
I.T.I
Gamma
0.27
0.09
0.12 = 0.05
0.14 1 0.05
Log-Normal
2.39 0.33
1.94 0.71
1.99 0.66
Lonax
0.39 0.20
0.31 = 0.17
1.15 0.44
Half-Cauchy
6.67 2.45
6.35 1: 2.56
5.45
2.23
Half-t
0.07
0.112.002
0.22 0.04
Welbull
0.29 = 0.05
0.23 0.02
0.23 - 0.06
Beta
Uniform
0.12 0.05
0.12 1 0.03
0.12 1 0.02
Beta
Gamma
Half-Cauchy
Half-f
Log-Normal
Lomax
Uniform
Weibull
Figure Comparison of estimated PDFs for various continuous distributions in the synthetic dataset. Each plot shows the true distribution and
estimates.
PDF estimation:
For continuous non non-negatee -eggatvve distributions, we estimate the probability density function (PDF) f(x) using kernel density estimation (KDE) with Gaussian kernel:
1 N
(x xz)²²
f(x)
exp
where by default a N- 1/(1/444),
d= N
V2
2h2
(28)
with a denoting the sample standard deviation of {æ:} and N the number of samples.
We compute error bands by bootstrapping: for each model, we resample its generated samples 10 times, compute the KDE for each resample, and display the mean + standard deviation across estimates. For bounded distributions (e.g., Beta, Uniform). we clip moodll-generade distribution's support before applying KDE, Each PDF plot includes: a) ground-trus PDF, and b) the average KDE for each model, with any shaded error bands indicating bootstrap uncertainty.
samples to the
C.5 Experiments for Multivariate Setting
While most of the empirical validation focuses the univariate formulation for clarity, we addition- ally evaluate ItDPDM and other baselines on multivariate (2D) discrete distributions to demonstrate its ability capture joint count dependencies.
We first construct bivariate distributions as products of two independent univariate marginals using the same parameters in the 1D synthetic experiments (see App.C.2 above). The Wasserstein Distance (WD) metrics averaged over 5 runs are summarized in Table 8 showing that ItDPDM maintains consistent improvements both DDPM and LTJ baselines.
Independent multivariate setting:
Dependent multivariate setting: We further study two correlated count models to validate ItDPDM's flexibility in learning structured dependencies:
Bivariate Poisson: Let U ~ Pois(An). Vi ~ Pois(Ar), and V22 ~ Pois( A2) be independent;
define X, + Vi, X, = U + V2, giving marginals X, Pois(>o A1) and X₂ ~ Pois 22). For λ 3, \ 2, X 2, we obtain correlated Poisson pairs (X1, X2).
25

---

Table Independent bivariate (2D) extensions of univariate synthetic datasets.
Distribution
DDPM
ItDPDM
LTJ
PoissMix
7.90 + 0.64
2.65 + 0.23
2.96 + 0.29
ZIP
4.89 - 0.72
1.78 = 0.27
1.92 + 0.38
NBinomMix
10.83 0.88
3.53 0.57
3.26 0.48
BNB
5.07 0.72
1.82 + 0.36
1.67 0.41
Zipf
3.51 + 0.58
0.97 + 0.21
1.49 + 0.32
YS
0.87 + 0.23
0.32 + 0.05
0.36 + 0.08
Gamma-Poisson (compound Poisson): Let Gamma(k 2, 1), and draw x, Pois(@) independently for i = 1,
The WD scores for these dependent models are reported in Table showing strong performance gains for ItDPDM
Table 9: Dependent bivariate count distributions.
Distribution
DDPM
IdDPDM
I.TJ
Bivariate Poisson
3.502
0.826
1.323
Gamma Poisson
0.855
0.337
0.624
Overall, these results demonstrate that the proposed cntematiio-thhorchee Poisson diffusion frame- work (ItDPDM scales gracefully to multivariate discrete settings without additional architectural modifications.
D
Section 3 Proofs
D.1
On the Poisson Loss Function:
Here, outlined in 3.2, we establish that the function 40(x) x xlog r + serves as the convex conjugate of the Poisson distribution's log moment generating function (log MGF). We begin by deriving the log MGF of the Poisson distribution, and finally computing its convex conjugate through the Seggedre--eenhhe transform. Let X be random variable following Poisson distribution with parameter À > 0. The probability mass function (PMF) of X is given by:
P(X k))
for 0,
The moment generating function (MGF) can be evaluated as:
A
(eet
Mx(1) D[e'X]
ek P(X k)
= Me'--1
k!
k!
k=
k=
k=0
Let d(t) be the log moment generating function as shown:
<(t) = log Mx = X(e 11)
Without any loss of generality, let (since scaling does not affect the form of the conjugate), implying (+t)= ef - 1. The convex conjugate of convex function Φ R -> IR U (+oo), denoted by o² (x), is defined as:
(x) sup(xt <(t)}
This transformation maps the original function ((t) to its dual function &² (z), and then finds the supremum of linear functions subtracted by o(t).
Let (tt) e¹ be the log moment generating function (log MGF) of Poisson distribution with parameter À = 1. Then, the convex conjugate of é. denoted by o² (x), is given by:
I log +1+ if > 0,
(²(x)
+00
otherwise.
26

---

Proof. By definition: (x) = supten fzt- - o(t)} = suPEE lat- +
To find the supremum, find the value of that maximizes this expression. First-order conditions
imply: zt (zt - -e-)
e' 0 so we have & log æ. This critical point exists only if x > 0, as > 0 for all & R. From the second-order condition, condittio, we get:
d²
(xt -
- <0 EE
df2
The negative second derivative confirms that the function is concave i log z, ensuring global maximum at this point. So for t = log
(z) x:log x) e Lon + == x log
Therefore, for x > O:
6 (x) = x log x - x x
For x & O, the supremum unbounded above, leading to: x= (x) = +00 Combining these cases gives:
x log x - x if 0.
('(x)
+20
otherwise.
This establishes that lo(x) = x log x - x + 1 is the convex conjugate of the Poisson distribution's 's log moment generating function (()) e 1 and therefore, natural loss function.
Connection to Bregman Divergence
The Poisson loss function we defined [ar, .t) is member of the broader family of Bregman diver- gences, which are pivotal various domains such as machine learning, information theory, and optimization. A Bregman divergence is defined for strictly convex and differentiable function & : &
as follows:
(x, -x=) (VV)))))))V(())rr
where ( denotes the inner product in IR and ( represents the gradient of t evaluated at c.
For the Poisson loss function, the generating function vp is chosen as:
(xx) = x log x 2.
Substituting this into the Bregman divergence definition yields:
(x, = x log x - x - (# log a- - -) - (log &. (x- (e-()) +)). Simplifying the expression, we obtain:
(x(x,)) 2) = x log
x +++ +,,
which is precisely the Poisson loss function ((x,
This framework not only encapsulates the Poisson loss but also generalizes it to encompass other widely-used loss functions by merely altering the generating function y/. Well-known examples include squared error loss (choosing (x:)) = ==²²² and Itakura-Saito divergence (choosing w(xx) - -logx) x). Bregman divergences exhibit key properties that make them valuable in optimization and learning. They non-negative, vanishing only when They are also asymmetric, meaning (x .) * C (Ø, x) in general and their projection property enables efficient optimization over convex sets.
& due to the strict convexity of sh.
By leveraging the Bregman divergence framework, Poisson and Gaussian diffusion schemes can be unified under single theoretical umbrella, where squared error loss (viz) ) = 1 corresponds to Gaussian noise, and Poisson loss aligns with count-based data modeling. This unification enables extending optimization techniques across different noise models by adjusting the generating function vb. Viewing Poisson loss function as Bregman divergence thus broadens its theoretical and practical utility discrete data modelling.
27

---

Optimality of Conditional Expectation
Let Φ Rd - R be strictly convex and differentiable function. The Bregman divergence D induced by Φ is defined by
Y).
Consider a random variable X TRd and sigma-algebra (2) with = Y(Z) being any measurable function of Z. Let Y* = [ denote the conditional expectation of X given Z. The objective is show that Y uniquely minimizes the expected Bregman loss & Y)] among all measurable functions Y (Z). For any such function Y. consider the difference in expected Bregman losses:
E[D& - Y*)
E (x()) ( (x-))
- - (x (X-1)]
Simplifying, the terms involving ((X) cancel out, yielding
(XD - (TD,,,,))EEE((Xxx,)1)) = E [Ø(Y)) ((Y) V
Recognizing that Y* the conditional expectation E|X ZJ. we utilize the law of total expectation to express the above
E [ -r
= ,Y]. r)] =
Due to the striet convexity of Φ, the Bregman divergence satisfies (u(,, 2 0 for all u, E Rd. with equality if and only if u. Therefore,
EID& Y)] EID, = E\D,(Y" .Y] 2 0, with equality holding if and only Y Y almost surely. This establishes that
D E|D,(x,
for all measurable functions ((),, and thus Y E is the unique minimizer of the expected
Bregman loss (XDx(X,Y)) Y)).
D.2 Section 3 Lemma Proofs
Proof of Lemmal 1 Properties of Poisson Loss
Consider the loss function defined as ((2,
à lo 2 where lo(=) = log + 1.
1. Non-negativity: Since (o)2) achieves its minimum value of 0 at and non-negative for all
0, it follows that [xx, 8) 2 0 for all 2, 2 >> 0. Equality holds if and only if 2 1, i.e., x 2.
2. Convexity: The function (=) is convex in because its second derivative (²(2) is positive for all 0. Therefore, [(r, .)) - ( is convex in for each fixed æ. and similarly, it is convex in x for each fixed é as the composition of convex function with an affine transformation preserves convexity. (We can also directly use the Bregman divergence framework to argue its convexity)
Scaling: For any a > 0, consider scaling both arguments of the loss function:
1(xxr ocii)
az. -o ( azt
( a 1(x,5).
This demonstrates that the loss function scales linearly with
4. Unboundedness for Underestimation: For any fixed Evaluating the loss function this limit:
0, 0+, the ratio 3 - co.
1xx,²) = x- log
+1) x log
+
f
As - 0+, log ( grows without bound, causing 1xx, 4)) - oo. This shows that the loss becomes unbounded as è underestimates z.
28

---

Proof of Lemma 2 Let 2, be a Poisson random variable with parameter yxx, meaning 2,|.X Pois(y2:). Suppose the conditional expectation (X): E| Z is affine in 2,
(X); az b,
for some a and b, with 0 < a < 1/7 and b 0. We aim to show that X follows a Gamma distribution with shape a 1-ye and rate 3 8. i.e.,
Gamma 1-7a a a (
Define U X and Y Zy P(7U). Assume ElUlY 2] az b. By the law of total
expectation,
0 = EIUU (aY + bY)g
for any function y satisfying integrability. Choosing ( = e e for / >> 0, E (aY b)) 01 0.
Rewriting Y ~ P(TU), we use the known conditional Laplace transform relation for P(A) random variable y.
Ele tY U = u| = exp(u(r(e 1))).
Hence,
-1e- = Fleex(U>>(e*= 1))}
which is the Laplace transform of U evaluated at = 7(1 - e Denote
Lu((s) = Ele so that Ele = (-(1--())))
Similarly,
ElYe = d Ele-oX1.
ElUE
= 11 Lu(ss
ds
s=y(1---1)
dt
From the orthogonality condition.
EI(U (aY + bb)ee 0.
Using the above expressions,
0 -
bE(e
Substituting = =(111 e) and differentiating as needed, we obtain first-order linear differential
equation for LU (s),
-(11 a7) + (ay)) (Lu
The unique solution with (0) = is
b
Lu(s)
ya
This is the Laplace transform Gammaa 1-7a : { random variable. Hence, U X follows this
Gamma distribution. For the Gamma distribution to be well-defined with positive shape parameter, we require
1-7a > O, which holds for 0 < < 7. The rate parameter 3 > 0 requires
b > O. Under these conditions, X Gam -- d), completing one side of proof.
Converse ("if" part): The converse follows directly from Poisson-Gamma conjugacy. Suppose Po)s(yXX.. Then the posterior is Z|Z Gamma(a a + 8, 7))
X Gamma(o,5) 33) and conditional on X, we draw Z,y
with mean given by
a + 2
X 2, =
+ where 3 7
b
3+7
3+7
Hence the conditional expectation is affine z, establishing the "if" direction and completing the proof of Lemma
29

---

Proof of Lemma When X = 0 almost surely, E[ = 0, and the identity holds by convention.
Else, E[X] > O, and we have:
E [1X, []] E
X log
X + 8 = E| log X] E[X log - B[X] +
[X log X log [
+ E X + (
E [X]] +
(E
Proof of Lemma 4 yX. To determine
Consider Z, = P())X), where Z,, is Poisson random variable with parameter
(X): 2 2| for each 2 2 0, we start by applying the definition of
conditional expectation:
(X)= E X Pz, Pz, (=|x)]
Pz, (2)
Given that Z,|X Pois(yy), the conditional probability mass function is
Pz, (z|x = = x) (xz)+e
z!
Substituting this into the expression for (X)² yields
E X (xXife
21
(X)
Pz, (2)
To relate (X), Pz, (2 1), observe that
Pz, (3 + 1) E' (7x)stle->x
(xX)=e-7X
7 E' X
(2+
2 2+1
z!
Rearranging the above equation, obtain
E X (x xx
12+1)
Pz, (z 1).
2!
Substituting this back into the expression for (X)., we have
(=-1) Pz, (2 1)
(2 + 1)P2, (2
(x)
P2)(2)
(z)
This completes the proof of Lemma 4
The conditional expectation over Poisson noise channel also has other unique properties, of which are stated below. The next property is useful in showing that the conditional expectation in this case is unique for every input distribution.
Lemma 5. Let Z, P(7X). Then, for every positive integer k. and every non-negative integer z,
E [ = II E(,
Proof of Lemma 5. Let Z, P(yx)) We claim that for every positive integer k and nonnegative integer
F((78) = Ebx =
From the affine formula Lemma 4 the conditional expectation of 7 given Z,y related to the ratio of marginal probabilities. More generally, for higher-order moments,
(s k)! Pz, (2 +
El(>X)*
(29)
z!
Pz, (z)
30

---

We can also express (7X)** ) as product of xXX' terms and use the Poisson shifting property of P()X).
Applying Lemma and Eq. 29 for each shift + + gives
k-1
E(( = = II 2=0
2 =
Each factor on the right captures the conditional expectation of 7X at consecutive levels 2, 2 all higher-order moments of yX follow from the first conditional moment ZZ = z]. This completes the proof.
1..... 2 + k -
Proof Sketch of Eq. 29. The key observation behind the formula is that, for the Poisson distribution, shifting from y to y + k multiplies the corresponding probability mass by (aX+>)" Evaluating the expectation leverages the ratio of adjacent Poisson probabilities (y + k) P)/Pr()) and tracks how (a.X Λ) factors. In essence, product expansion shows how each additional factor a.X increases the count from y to y + 1, and iterating this argument recovers the moment expression. As shown in [6]] for Poisson observations Z, ~ PaaxX + A), the sequence of conditional expectations [ Z, = =|}+20 uniquely determines the input distribution Px. This supports our information- theoretic derivation and strengthens the foundation for learning in discrete-state noise models. For our Poisson setting, we also have:
Lemma 6. Let by = P()x). Then, for every y > and y = 0, 1,..,
d dy PZ,xx(12)) (Pz, x 1 [z) Pz,1x(Ulz)), ddy Pz, (v) yP2, (v) + 1)
d
where P2 (x11x(-1 Pz, (-1) =
Proof of Lemma 6. Let Z, = P(>>), where Zy is Poisson random variable with parameter 7
We first compute the derivative of the conditional probability mass function Pz, (= = x) with respect 7.
Since Z, given X = x follows a Poisson distribution with mean ya, we have
P2, = x) (1e)
y
Taking the derivative with respect to y and using product rule, we obtain:
()z))e->a = 2(7x)* 2- we nm
dr (x = x) =) dy
z!
z!
2!
Simplifying the terms, we obtain
(
(x
Pz, (z|x = x) =
dy
(z (z-1)! 1)!
z!
Notice that
(70))-
Pzz (2 - 1|X = =x),
(2 - 1)!
we can rewrite the derivative as
d =x)
1 = =x)= -2 z))
dy
This establishes the first part of the lemma.
Next, we compute the derivative of the marginal probability Pz, (z) with respect to y. By the law of total probability, we have
Pz, (v) = [P2, (=|x²)
Diffcrentiating both sides with respect to 7, we obtain
_ Pz. (x1x)
Pz, (2) E
d-
dr
31

---

Substituting the result from above, we get
d
Pz, (z) - - E (
dy
This can be expressed
d T
(x) (2 1|X)] 7E |
Noting that for Poisson distribution, E [xP2, (z|X)] = (2) and E (2- 1|x)] = Pz, (2), we substitute to obtain
d
Pz, (2) = =P - (2
dy
Thus, the second part of the lemma established.
Other properties of the Conditional Expectation
Lemma 7. Let Z, = P(yX) where X is nonnegative random variable, and > 0. Then, for every y > 0 and integer z 2 0,
24 Var =x
dy
where Var (X = -1) = O.
Proof. Fix an integer 0. Consider the conditional expectation
= 1 (I++1) - ( p(Zm=2)
Differentiating both sides with respect to y, we obtain
= 1 d = =|
( P(Z, -(2,- - P(Z,=
(a (=+ P(Z, = 12+1) P(Zy =)
dn
7 dy
Applying the quotient rule to the derivative inside the parentheses, we get
P(Z, =) 2) P(Z, = 2 + P(Z, = z + d P(Z, =
P(Z,=z+1) (PiZ,=2+1))
d7
dy -(),=2)
P(Z, = 2)
Using the properties of the Poisson distribution, specifically the identity
P(Z, =2+1) = 7X
P(Z, =
2 ++1
we can simplify the derivative expression. Substituting back, we obtain
dn t = = Var (X = ==-1) 1).
For the case 0, the derivative simplifies E 0] 0, since Var (XlZ, -1) by definition.
The result for higher moments follows similarly. For any positive integer k. differentiating E (>x))* Z., expression. This completes the proof.
with respect to y and applying the quotient rule leads to the stated piecewise
Moreover, for any positive integer k,
k x- =0
== 0,
dy E [(6X))1122 =]
(2 + k) E ((xX)= Zz, E E = E (xx)*
2 z 1.
E Z, -2
32

---

Lemma 8. Let Z, ~ P(,X). Then, for every fixed > 0 and any non non-degenraate X, the mapping
2 2 2] is strictly increasing.
Proof. To show that Z = 2] is strictly increasing, we define U = ? and consider the Poisson marginal probability:
Pz. (k)
F.
(30)
K!!
Applying the Cauchy-Schwarz inequality, we obtain
Pz, (A) S A.
E EluktLe-U) ElUi--e--O
(31)
Rewriting in terms of factorial expressions, we get
+ 1 Pz, (k + 1)P2, (k: 1).
Pz, (k) <
(32)
k
Now, substituting this bound into the Turing- Turing-Good- Good-Robbins (TGR) formula from Lemma
(= + 1) (= + +1
E Zy = 2]
(33)
Pz, (z)
we obtain the lower bound
(2 1) (z) Ey =
(34)
(2
Simplifying, this reduces
xP2,(2)
EIU|Z, 2
(35)
Pz, (z - 1)
Using the same formulation for conclude
ETUIZ, 2] 2 EIU =
(36)
Since X U/7 it follows that E[X|Z =] is strictly increasing in completing the proof.
D.3
Incremental Channel Approach to I-MPRL and related proofs::
Here, we derive interesting relations between the mutual information in Poisson noise channel and various parameters of the channel. The general distribution we consider here is Y ~ Poisson (o.X + A).
Theorem 9. Let A > 0 and let X be positive random variable satisfying EςX log < Consider the Poisson random transformation X + Zx P(X + A). Then, the derivative of the mutual information between X and Zx with respect to the dark current X is given by
[X; 2x) [log(X + log(X + +N)]
where (X x) E|X + A| =]
Proof: Let Yo = P(X) and N₂ = (A) be independent Poisson random variables with means X and X, respectively. Define Y, Yo + NA. which has the same distribution P(X A). By the definition of mutual information.
1(X;
E(L(X, Yo. Y.).
where the expectation is over the joint distribution of (X, Yo, Ys), and the log-likelihood ratio is PYOxx Pxo (k)
(rix(((=)
L(x, k, 8) = log
log Pr,
Given that Yo|X = ~ P(x) and V22 X
P(x A). the conditional probabilities are
Prix(((1))= (x x) (x+A)
Proxx (k(z))
k!
33

---

Substituting these into the log- ligeliikl ratio, we obtain
L(X, Yo. Yx)= = Yo log X - Ys log + X) + U,
where U encompasses terms involving the logarithms of the marginal probabilities. Taking the expectation, have
EIL B(X log X (X + log + {)} + BlU].
Expanding Y2 Yo+ N and leveraging the independence of Nx from Yo, we analyze the behavior of E|U] as becomes small. Through series of manipulations and applying the dominated convergence theorem, we find that
X
-(X;)) = A log
+ o(A).
(X)
Dividing both sides by À and taking the limit as + O, we obtain
d
dA (x) = [log(X X) - log X + X)],
where (X + = E|X + A|2> = 2]. This completes the proof of Lemma
Theorem 10. For every Poisson transformation Px with E log X < oc, and as 5 - 0,
1 = E log X - (x) lo) + o()).
(xx;
Proof: Consider first the case ā 0+_ Let Y P(X) and Z = P(6X) be independent conditioned
on X. Define Y8 = Y + Z. Then, the left-hand side of the lemma can be expressed as
[(x; 1z) - (X: = E Ef108 PYxLx
log (r PY(
Py, (Y,)
Expanding the log-likellnn liikliihoodd ratio, we have
E' Z log X - 6X - log EI(X' V -(+8)) [A}
E((X))x-X(1))
Here, X' is identically distributed as X but independent of Y and Z.
To analyze the expression
O, approximate A P(6X) by Bernoulli random variable that takes the value 1 with probability 6 (conditioned on X) and 0 otherwise. This approximation is valid because for small 8, the Poisson distribution P(6X) closely resembles a Bernoulli distribution.
Substituting this approximation into the previous step, we obtain
I(x:Y))-1(xxY) = Zog X - 8X - log (1 6X) E((X') -X 177+8XE((X))))
EX e [ +0(0)
(37)
Expanding e- AX' to first order in 6, we have 8x' = 1 - 5X' Therefore,
E((X))++) - + (38)
Substituting this back into the logarithm and applying the first-order Taylor expansion log(1 + c) x for small c, we obtain
log (1 5X E -X' 1X +8X (XXX
8X (X Y - x +
s log [(((') e [8)]
E((X))e-x11))
F((X')X YJ B((x))+1e-X11 + o(5),
= log((X> - 5x
where (X) E(X
34

---

Substituting this approximation back into equation 37
we get
- - I(X;Y) E 2 log X - 6x - log/()) + áx BI(X' x
(((X))XXX1) + +o(d) (39)
Noting that Z is Poisson with parameter X, we have { = X, and thus E log X) E{X log X).
Furthermore, we know that (x) = E{ and from Lemma we have
(X')++l |x] (x)e + +1).
Substituting these into equation 39 we simplify to
( ( log X - (oglo)(X)) 0(0),
Dividing both sides by 8 and taking the limit as E - O, we obtain
d
(();
= E X - (X) log(X)),
d8
16=0
where (X) EY This completes the proof of the lemma.
E
Tail Bounds
As we know the output 2y given the input æ is modeled as ~ P(xx), where a 2 0 is the non-negative input random variable, and 7 represents the signal-to-noise ratio (SNR). The negative log-likelihood when estimating 2, using is given by:
(1) log p(2ylz) = -log (- ()a))t)))) = -----y-- 2, (og((x)))) + log zy!
Zay
We define the expected negative log-likelihood M(7)=EE(a,2) [1x,, (0),,,1]] ) = Ex [E),, [(x, 2, 21] We now consider a mean constraint μ E[x] in this and our objective then determine the input distribution px (x) over æ 2 0 that maximizes the above function. To compute the expected loss, let first evaluate Ez,1 [xx zy)] and using E,, [=|]]
7ar gives:
Ez, [[,, 21)1 = E2 by:: - 2, log()z) + log 2,! = ya - log(>x) |2,]= | + Ez,, Iu (log 2,! = 7 yar log(>æ)) + Lx,, 2,!
(40)
(41)
We can write M(7) in terms of the the conditional entropy of n,, given x as:
M(7) E. [ (2y[z)), since H(2)1z) E,,1- logp(zz E2,1- 1xx 27)1.
The entropy H (2, (r) of a Poisson distribution with parameter yx is given by:
HS( = P(z, k:) log P(zy = k)
0=0
where P(zy = k))
So substituting this into the entropy expression, we obtain:
k!
((Tx))+e k!
(7x)*e-<0
HS(x)) =
log
-xx 7x log(ar)
+
log k!
kd
k!
Am
It is natural to assume that the Shannon entropy HS(A) of a Poisson distribution strictly increases with À E (0, +00). We will prove this result, as well as the concavity property of HS))), in the following lemma.
Lemma 11. The Shannon entropy HS(A), À (0, +oo), is strictly increasing and concave in X.
35

---

Proof. The Shannon entropy (S) of a Poisson distribution is as outlined above. To analyze the monotonicity and concavity of SA),, we compute its first and second derivatives with respect to First, the first derivative HS' (X) is:
A log k!
1 log k!
H''(A) - - log
A
(42)
k!
(h: 1)!
k=2
A log(k (k + 1)!
2
\ log k!
= - =llog X +
(43)
k!
k!
k=
Simplifying, we get:
1 log(k: k + 1)
(A) = - log + e- -
km
is clear that both terms on the right-hand side of (2) are non-negative for À E (0, 1]. and the second term is strictly positive. Therefore, (A) > 0 for X E 1). Now, it remains to prove that (A) > for À > 1. Let's calculate:
A log(k +1
0
1 log(k + 1)
(A)
te-
k!
(k 1)!
1
* log(k ( + 2)
A log(k: + 1) h!
+
X
k!
k=U
k=1
00
A log
1 11 1++1
> log + E
E
k!
k=1
A log
1 K+I
1
X +
k!
km0
1
A
1
1 A++1
+e
< -- >
+e
X
(k ++1!
(k
IJ!
k=
k=
1
=0. 0.
So, (A) < 0 for all X >>0 0. Therefore, H's (A) strictly decreases in X, proving concavity and it is sufficient to prove that limx (A) 2 0 After further simplification.
lim (A) = lim log A ( (e~1 A) X-000
A log(k + 1) -1) 1).
X
k!
and it is sufficient establish that
A log(A: + +1
lim imminee (log X)-1 X
2 1.
k!
This inequality is outlined in 62). Using this, we get that (A)()) 0 for all A 2 and Hy(() < 0 for all A 2 O, hence the proof follows.
Given that (zy is increasing and concave function of x for > 0, we aim to maximize Feg (2 [z)] under the mean constraint F|x] = p. The functional to maximize is J[x((=)] = Jo
dr, subject to the normalization and mean constraints: Jo (x(x) de =
1 and Jo apx(v) (x) da =
Introducing Lagrange multipliers À and v for these constraints, the Lagrangian becomes:
36

---

(lpx(1))= _ ((z, de - - px px(o) (xx -v
(L, spx -
dx)
dr
Taking the functional derivative of L respect to (x( and setting it to zero for optimality yields: 8C
[px(æ] (2ylz) - A
Given the properties of H (2) the solution corresponds to exponential distribution. The exponential distribution with mean ju is given by:
Px Px(x) (x)
2
T 20
Maximizing the entropy of leads distribution that spreads the probability mass, thereby increasing uncertainty and consequently maximizing the mprl. Now, using this exponential prior, we will derive expression for mprl(7) which for deriving the left and right tail bounds.
Now, the prior distribution for is assumed to an exponential distribution:
fx (z)) = Ae Aar
We introduce the latent variable Z, such that:
e (yx)²
PIZ, =
z!
which follows a Poisson distribution. The conditional density of X given Z, = 2 is derived as:
[x12(<|z)= P(Z, = = (x)
PiZ, =)
[x]2(0|22) (Bx)²
de. -x-r -xre
21
(Bxx)=>e->> (2+B8)s (x-ttB3))
=1P(2nn =2)
and notice that this is Gamma distribution: X| posterior mean of given Z, is:
Gamma(z 1, + 9) The
2 2+1
E 2 = =]
(44)
A + 3
and this serves as the optimal estimate X², Now, let consider the following expectation: (where / the previously defined Poisson loss function)
X x
X X
Ex12, [[X,x*] E|X log
X X+X*
E
X log
E| (45) 2
Using integration by parts and properties of the Gamma function, if W ~ Gamma(a, 3). then: 63
W log W a 8 [1b(a 1) - -log 3
where defined the digamma function (a) as: vi(a)= d log [(a). The above results would also follow from differentiating the moment formula:
do
T(a +
E[
37

---

Applying this this result in our case gives us:
+
lox = A 3 [ + 2) - (A + a)]
We also have from Equation. 44
log( X*) log(z + - log(A + a))
Taking expectation, the first term in Eq. 45 can be written as:
2+1
+ 1 A A+a
E X log
[1b(2 + log(A + +
[lo(22 + 1) - log(A + a)]
x+
(46)
2 ++1
1:(2 2) log(z + 1)]
+ B
(47)
Now, we compute the marginal distribution as follows:
- l x'e
AB= 0
=(Z,=2)= =
z!
Using the Gamma integral property stated as follows:
l x -0+002 dr
((=+1)
(A + 8)=+1
obtain (since [[2 + 2!):
P(Z, 2) X
T(2 1)
AB=
(A+ B)++) (1 p)p², where p A A B
8)=+1
Now, the mppl(7) expression obtained as follows:
mprl(?)= = _ (1-p)p²
2
[0(2 +) log(z + 1)]
A (A+ 3)²
Σ
(2+1)p [+/( +2) - -lo((z + 1)].
x B
E.1 Left Tail Bound
In case of 11) being the relevant range of integration, the left tail integral is defined as: 00 mprl(7) dy First, we interchange the sum and the integral:
(1+7)2 / (x777² dy.
mprl(7) dy
(2 + 1) + 2) - loo(z +1]
==0
We define the inner integral as
X
?
1 Jo
dr.
(A+ + 7)2
Substitute > + 7. which implies 7 \ and dr du. The bounds change accordingly: M when 7 = 0 and u = X + to when = 70. The integral becomes
= (u X)=
u=+2 du.
Next, using the substitution v = 2 leading to u = I=p and du (I- dr. The bounds
A_
A
transform to 0 when À and X4 when 7o. Substituting these into the integral yields
J =
0² du.
38

---

The integral I can be evaluated as
20 x450
2++
7
2 ++1
2
Substituting I back into the expression for the expectation, gives:
x+1
mprl(n) dr 2) log(z -
+1]
A 70
Let the above sum be which we use in the sections below. By re-indexing the sum with = 2 + 1, the final result can more elegantly be expressed as:
l
mprl(7) dr = Elo(++)) 1|lg(()))
(
70
( 70
We aim to establish an upper bound for the sum
= Σ (2+ 1) 2) log(2 + +1]
Yo λ + 70
2=0
where & denotes the digamma function, Yo > 0, and 0.
Let us define x = X4
Given that " > and > 0, follows that x < 1. From, 64], we recall the expansion of the digamma function:
v0z + 2) H2+1 YE.
where H,, is the n-th harmonic number and YE is the Euler MalcceMnacheroni constant. For large 2,
Hz+1 = log(= + 1) + 7F + 2(2 11)
12(= + 1)2
Substituting this into the expression for cb= + 2) yields:
1
( + 2) log(= + 1) 2( +1
12(2 1)2
From this expansion, it is evident that
< + 2) log(z + 1) < 2(z + 1)
for all z 2 0, since the higher-order terms 12(=+1y² +... contribute negatively, thereby decreasing the overall value. Consequently, each term in the sum satisfies
1 (z + 1) [x(2 + - log(z + 1)] xtt1 < gost1
Summing over z from to co, we obtain
1
S<
<
x
z+1
2
==0
Using the simplification of the geometric series
Eee T. 8+1
x
S
<
21-6
==0
Substituting back x
7ta X
we have
Yo
À
X+
%
Jo
x
70
À
+
70
1 -
X
A
X+
39

---

Putting this into the inequality for S, we obtain
1 70
S <
2
Hence, the upper bound for the sum in the scalar case (for single input-output realization) is
2+1
(2+1) (= 2) log(z 1)]
70
70
+ 70
2X
(Note: This is different from the 2, notation used throughout the paper.)
Extending this result to the vector case, consider a d-dimensional random vector x E X C 2d with covariance matrix Σ, whose eigenvalues are yd all positive. Assuming the problem separable across the eigenbasis of Σ, each dimension can be treated independently.
For the vector case, the sum becomes
d ΣΣ
70
Syector
[+ 2 log(z + 1)]
A3 + 70
2 2
Applying the scalar bound to each eigenvalue As. we have
(1,
Σ (= 1)
log(= + 1)]
70
A5 70
2
Summing over all from 1 to d, the vector sum satisfies
Svector
70
70
Σ
2
X;
i=1
1=1
In the special where the covariance matrix E is isotropic, meaning all eigenvalues A; À for i = 1, ,,...,d, d, the bound simplifies to
dio
Svector
2X
This concludes the derivation of the left tail bounds for both the scalar and vector cases.
E.2 Right Tail Bound
In case of (70, 71) being the relevant range of integration, the right tail integral is defined as: f mprl(7) dy
Consider a discrete variable x = (#1, .2....,.......... x.)) E X C Zd, where each component x; belongs to a discrete set { Ali Z). Observations modeled 2, PP(ya;) for large signal-to-noise ratio (SNR) parameter y, The estimator (z,,,)) is typically the maximum likelihood estimator (MLE), implemented by rounding , the nearest bin {k A).
The loss function per component defined as
24
Lx,, xz log
Iz
Li.
and the mprl(7) is given by [[[ 2.)] over the randomness of 2y,a. The right-tail integral of interest is
E Σ L(x,, Lo, (((2)))))) dr.
IR =
which we aim to upper bound.
At high SNR (7 xc). the noise is relatively small compared to æ; but rare rounding errors of size j A can still occur. Focusing on single component ws, an error size jA happens
f z iA
2y,5
(x jA 0.5A), (x= jA + 0.5A)).
40

---

For 2 ~ ppisson(p) with jr y2 the Poisson Chernoff bound 65 provides that the probability
of such deviation most exp(-ca. ( where Ci.j > is constant dependent on A, Zi. and the shift jA. Hence,
P(error of size jA) S ou47
The per-component contribution to the mean MLE loss is
(1) = Ezy,, [L 3i (2()))))
When the estimation error is jA, the loss becomes
L. (zi Bi.
jA) 3y log
33 (ote -A)).
24 - -
Therefore, the mean loss satisfies
Jmax
(7) &
Ty log
E2
+ (xt - j4) eaiis.
jA
Summing over all components i = 1,.... d, we obtain
fiav Σ Σ
mprl(7) Σ mprl, (7) S
3j log
21
24 (z; - - e a0s
24 - iA
The right-tail integral I thus be bounded
IR mmr((7) dy v
Jiaae
xee log
24
I3 + (0t :1) / a
Fi jA
Evaluating the integral, we find
e c dy = 0-am
Ca.
Leading to the final right-tail bound
In
E
L(x, 8.) dy 1
Jmnx
2t log
Ri
jA
é cu
F7 jA
Ca.i
71
In the above expression, Cij 0 represents the Chernoff-type exponent from the Poisson large- deviation bound for the event causing an error of size jA in component i. We determine these parameters empirically, and the parameter Jmax indicates the largest error shift considered, which typically small in practice and can be tuned empirically. For empirical purposes, it might also be worthwhile note that the bracketed term in Eq. 47 can be approximated as the sum over a few starting a beyond which it effectively dies out as illustrated Figure 14
Figure 14: Approximating the Digamma term
41

---

Proof Sketch of Pointwise Poisson Denoising Relation
For Poisson channel defined earlier, we derive the pointwise denoising relation:
Theorem 12, The KL. divergence derivative satisfies:
dy Dx](P(2)|[))IP(E))) = mprl(z,7)
where the pointwise MPRL
mprl(s, 7) (p(,, (zx "* (z,))
with (z, x²) x log 2 x² x² and (2,) [2
Proof. For the Poisson channel 2 ~ Pois()æ:) define
(=(a) py(2) (=,7)) F|X|Z, Z, =}, ((x, log x 6.
Rx(7))= (pr() log
#20
Differentiate the series using the product rule get
d R₂ -- (z))(() log
(=(2))
- -pr(2) Or (og 0, (ogp((z)) TT T2.
dy
(2(
Term T2: For the Poisson distribution with mean y
(=(2/=)) = --v - (7x)*
Taking the derivative,
d dy
log Pay (2/xx) x + 2
7
Hence, for the conditional Poisson law, we have
02 logoop (z|z) 7
Similarly for the marginal,
Py(2)= = / (((c))))(
Taking the log- derivative,
d log 1 /cxt ? (2(
()
dy
Identifying this conditional expectation gives:
0, log (2 (=) = F. - X - 2
7 =
7
Hence
2
T2 = Ep.()1))
E|
7
Term Ti: Let ret) log (=(r))) and = 7x. Since (2(x) = pp - -x), p.(x]
(7) -))(()) -[(
Ti = (xlo)
Now, we state and prove what call the "Poisson-Stein identity.
42

---

Lemma 13 (Poisson-See Cooisson-Steinto identity). For Z, ~ Pois(A), we have:
E[(2, A)h( [,) = x X [M 2, + 1) 1(27))
Proof sketch. Let (k) PiZz, k} e x /kt for & 0. Then, for any h such that Ei(2, A)h(Z,)] < 00.
20
El(Zy 2)m(2,)1 Σ
(k: P)((
(a) A Σ
h(k) Px
k=
Using k (x) = Apx 1) for k 2 1 and reindexing.
Σ k h = (kk) - 11)=
hi + 1))x
j=0
so that
E [i ()m - (( +1) - h((,)]
j=0
This discrete integration-by--arts argument establishes the Poisson-Stein identity. Now with h in Lemma 13, we have
T Flr(Z, 1) +
Using the ratio formulas (from Lemma 4) gives:
(= 1|x)
(2 +
7 (X)2
(X)x == EI = =].
(=|x)
z+1' (2(
x+1
we get
Ti = x log x
r(= + ---1)(()) = log
(x)=
(X)
Now, combining both the terms gives:
R. = x E log x
+ E (X)z,
7,
dy
(X),,)
Since E|2,|X- = =]== ya, the second expectation equals El(X) z.l1 x, and hence
x log d R2 (alp)
I (x)z, x + (X)
((x,
d7
This equation can also be derived special case of Lemma 4.2 from 29
Link to the MPRL Loss: We already defined the loss function:
(xx, a*) log
If E|X [ny] is the estimator of x given Z,,, then by standard properties of conditional expectation,
Ep(e, ux) [+)] E(E 2, E[X] = (if x is deterministic, replace E[X] by x:).
Hence,
Ep(z, mn) [1(x,, (0)] E[x logx log 3 x x+] x log x x x [=oglog B [[**).
Since E(6") x.
((( [x)x=')) = z(log x - Ellog T*1).
One can show (by comparing with the final expression in the KL derivative) that this expectation
aligns with Ep(e, [E[X|2,) $11. thus establishing the link between the MPRL and the derivative of the KI. divergence. We can generalize this relation to any loss function that belongs to the class of Bregman divergences in a Poisson channel using the framework described in [37].
43

---

F.1
Tweedie's ' for Poisson Denoising
well-known result in Gaussian denoising Tweedie's Formula, which expresses the conditional expectation of the latent variable in terms of the derivative of the log-pdf of noisy observation. [32] Specifically, for Zy V + with (0, I), we have:
2 =
+ Vlog Jz. (z),
(48)
V
In the Poisson setting, we cannot directly take derivatives of log Pz, (2) with respect to discrete since they are undefined. Instead, the forward difference of the log of the marginal PMF serves as discrete analog. This culminates in the Yuring-Good-Rooie Lemma
(TGR) formula, already presented in
Hence, just like Tweedie's Formula in the continuous Gaussian case, TGR expresses the conditional mean (X)= purely in terms of the marginal distribution Pz, (=), bypassing any need compute the conditional distribution Px z., In effect, the ratio Y (X); plays the role of score function for the Poisson channel, analogous to the logarithmic derivative in the Gaussian case. This discrete variant underpins our Poisson diffusion framework, allowing efficiently compute the optimal denoiser E X Z.,] directly from the marginal PMF
G
Continuous Extension of ItDPDM
We extend the continuous-time channel with discrete states (CTDS) to continuous states through the following construction:
Definition 14 (Continuous- Time Channel with States (CTCS)). Let {Xih220 be right-cootinuouu state process with left limits (càdlàg) taking values in I The output process (Y2)220 is a counting process satisfying:
(L X,ds)
Yi y==P
(49)
where P--) denotes a Poisson counting measure. For measurable intensity X,, the output increments also satisfy:
Y Ye~ Yc~P
X,do):
v1,820
(50)
with (Y,, - -Y Je independent given Xjo,z] for any finite partition {4x}.
The mutual information between state and observation processes over [0, 7) is given by:
dPyr
1(x=;TT)= = log
(51)
dP
The key connection discrete-tinee -ime systems emerges through infinitesimal discretization:
Lemma 15 (Mutual Information Rate). For the CTCS in Definition [4] the mutual information rate satisfies:
lim 1 T T ((xx;;"))
= lim 1 ((x;: 6-0 _
(52)
where X8 = (10.8) and Y. = V8 Yn corresponds to the discrete-time channel P(XX).
Proof Sketch. Consider time partitions 0 to <t < tn =T =T with max 1+++1 tul S 8. By the chain rule of mutual information:
1(X
((T;YT)
YT)
I(X*x Y;Yee IY*))
k=
Σ
[(((ee ((ederr) + Eal
k
44

---

where captures residual dependence between time intervals. Using the Markov property of Poisson counters [66] and taking the residual terms vanish by the Asymptotic Equipartition Property (AEP) for Poisson processes 667 The result follows from Lemma CommaaPapplees applied to each infinitesimal interval.
The continuous-time counterpart of the derivative relationship becomes:
Lemma 16 (Information Rate Derivative). For the CTCS system, the time derivative of mutual information satisfies:
d di
[(X' = =E(, log X₁ log(())
(53)
where (Xx) = E| is the causal MPRL estimator.
Proof.
From Lemma (x':)'))
15 and the DTCS derivative, we have:
d
lim 1
[(1446 Yo45 [(xe: xx) Yi)
dt
6 6-0
1
lim E (6x, log X. o(X.) log (Xx)) o(1)
8-0
The result follows by dominated convergence and the tower property of conditional expectation. This continuous-time formulation preserves the essential duality between information and estimation seen in discrete time, with the Poisson channel's inherent noise characteristics governing both regimes. The CTCS framework enables analysis of real-time filtering and prediction [68] through differential versions of the key discrete-ting time identities.
H Detailed comparison of ItDPDM vs. Learning to Jump (LT.J, [20]
Table 10 shows detailed comparison below:
In the Leeaming-to-Jume (LTJ) framework [20), the per-step training loss written as Do(x, fo(2,, t)).
where
x E N is the true discrete count.
24 the noisy observation at step obtained by binomial thinning of 2t-1.
fo(z,, is the denoising network (parameterized by 0), which takes (zx, 1) and outputs an estimate &, of x.
(u v) is the Bregman divergence induced by a convex generator o: (z(u, = d(u) σ(v) - (Vó(v), u v). For the Poisson channel one uses ø(u) log u, yielding (o à log - i x, i.. the Poisson-Bres Bregman (relative-entropy) loss.
I Noised and Denoised Image Comparison
Figure 19 presents comparison of noisy and denoised images under Gaussian and Poisson noise conditions at a logSNR of 4.01. The left column displays the input images corrupted by Gaussian (Figure 17 and Poisson noise (Figure 15 while the right column shows the corresponding denoised outputs (Figures 16 and Figures 16p. Notably, the Poisson noise case exhibits a higher level of degradation than the Gaussian noise case, making recovery more challenging. However, the denoising process effectively reconstructs meaningful image structures both cases, demonstrating the model's robustness to varying noise distributions.
J Theoretical Runtime Analysis of ItDPDM Architecture
We present theoretical runtime analysis the proposed Information- Theoretic Discrete Poisson Diffussio Model (ItDPDM), focusing the components contributing to its computational cost during training and inference.
Poisson Noise Sampling
45

---

Table 10: Side-by-side comparison of our IdDPDM the Learning-to-Jumm (LTJ) framework.
We note that both methods employ a Poisson-Bregni 1tDPDM and D& for LTJ but they diverge sharply in how that loss is used and how connects to likelihood, as summarised below.
(eeative-entropy) loss denoted PRL for
Aspect Forward "moising"
IDPDMM (ours)
Boaning--o-JJupg (LTJ) [20]
Single-shot Poisson channel Z,,
Binomial thinning chain
Pois(?:X) with continuous SNR y E (0, 00)
2, 2. fort=1,....
Reverse generation
sampling operates in log-SNR space via continuous-time re- SDE ODE; sampling can flexibly subsample the SNR con- tinuum (e.g. 20-50 steps) without quality loss, in contrast to fixed- step chains
"Count-thickeming Markov chain with shifted- jumps; sam- pling requires executing all T dis- crete steps with no flexibility to skip or subsample, so the full T-- step chain is incurred for every generated sample
Bounds NLL
Informationn extends the classic I-MMSE identity to Poisson channel. giving the ex- act relation: -log p(æ)
Variational ELBO, multi-term -l.-divergence sum with bi- nomial/Poisson factors; yields only an approximate bound on -(z)
1 (P)L(
Training Loss
log($/z) + x, integrated over continuous 7. producing an exact upper bound and provides analytic tail bounds & an importance-samphiee estimator; empirically yields lower NI.I than all baselines.
PRL: ((x,0)
relative-entropy [z( fo(2, inside an ELBO with an identical Bregman form, but summed over discrete T' only with no closed-form link between total loss and true likelihood.
Per-step
Scheduling
Choose only a continuous SNR grid (e.g.. 1000-point logistic); no 0n or 1'' happe-yppraeeeeee parameters.
Must hand-design thinning sched- ule {ax]²= and pick T (typically (0=0..)..
Likelihood evaluation
Exact tail bounds importance sampling; likelihood (NLL) (in bits-per-dim) on real-world data, both WD & NLL synthetic data evaluated
Likelihood nor estimated; evalu- ation solcly via Wasserstein dis- tance (WD) of histograms.
Sampling speed
Compatible with fast ODE solvers (20-50 steps) due to continuous 7,
Must all T thickening steps.
Theoretical extensions
Poisson-Tweedie Tweedie mutual-information CTCS extension.
identity;
derivative;
The forward diffusion process in ItDPDM is governed by a Poisson noise channel By ~ Poisson(yæ)) where æ E RD denotes the input data vector and y the signal-to-noise ratio (SNR). Sampling from a Poisson distribution can be performed in ((1) per element using rejection sampling or table-based methods, resulting in a total cost (D) per data point.
Neural Denoising
The denoiser is instantiated neural network, such U-Net (for images) Transformer encoder (for symbolic music). The input to the denoiser is the reparameterized form
which improves numerical stability. The forward pass of the denoiser has cost O(D) per data point, assuming conventional convolutional or attention-based layers.
Poisson Loss Function Evaluation
The proposed loss function is based on Bregman divergence tailored to Poisson noise:
{ .) = x log
46

---

Noisy Image (Poisson)
Denoised Image (Poisson)
250
200
150
150
100
100
50
Figure 15: Noisy Image (Poisson Noise)
Figure 16: Denoised Image (Poisson Noise)
Noisy Image (Gaussian)
Denoised Image (Gaussian)
250
200
W
150
150
100
100
50
Figure 18: Denoised Image (Gaussian Noise) Figure 19; Comparison of noisy and denoised images for poisoned and Gaussian noise birds with logsnr=4.01.
Figure 17: Noisy Image (Gaussian Noise)
which convex, differentiable, and evaluated pointwise. The cost of loss evaluation and gradient computation is O(D) per sample.
Integral Estimation over SNR
defining component of the ItDPDM framework is the estimation of the negative log-like log-liklihood using thermodynamic integration:
log P(x) / mprl( 7) dr,
where MPRL denotes the minimum mean likelihood error. In practice, this integral is approximated numerically using n log-SNR values (e.g., n 1000), obtained via uniform or importance sampling a log 7.
Each SNR point requires a forward pass through the denoiser and loss computation, yielding a total per-sample complexity of O(n D). To reduce overhead, the model uses importance sampling from truncated logistic distribution over a and closed-form tail integral bounds to truncate the SNR domain (see Eqs. (28)-29) in the main text).
Given a batch size B and number of training epochs E, the overall training complexity becomes: O(B E n- D). This is comparable to standard continuous-stad diffusion models using discretized time steps, but the Poisson-specific formulation and MPRL integral introduce unique architectural and optimization challenges that are efficiently addressed via reparameterization and sampling strategies. Additionally, in terms of wall-clock times for training/sampling, we observe that 1DDPDM is comparable standard DDPM-style models.
47

---

Component
Complexity O(D)
Description
Poisson noise sampling
Efficient per-sample noise generation
Neural denoising
O(D)
Forward pass through CNN or Transformer
Poisson loss function
O(D)
Evaluated pointwise for each data coordi- nate
Integral over SNR
On D)
Dominant cost due to repeated inference and loss evaluations
Total per-sample cost
O D)
For fixed number of SNR grid points
Table 11: Asymptotic complexity of key components in the IDDPDM training pipeline.
K
Extended Related Work
Diffusion models have evolved along two orthogonal dimensions- -nose rype and state space. Classical DDPMs corrupt continuous data with additive Gaussian noise and learn the reverse process with score matching or variational bounds [2 51) An information-theoree
viewpoint links
these objectives to mutual-information integrals 12 22 and has recently motivated non-Gaussian
extensions based on annealed score matching [52 53 and SDE formalisms [54]. Parallel work
seeks native discrete-state alternatives: masking schemes such as Blackout Diffusion employ an irreversible "black" token that blocks exact likelihood computation [55]; Learning-te -carning-to-Jump (LTJ) replaces Gaussian noise by binomial thinning/thickent
yet remains limited to discrete time and variational ELBO 20). Very recent approaches move to continuous-ting jump processes, but still approximate the likelihood: [69] devise categorical SDE whose reverse dynamics are learned by discrete score matching, while [57] estimate probability ratios rather than scores to reduce perplexity on text.
Score Entropy Diserete Diffusion (SEDD) 57 represents a significant advancement in discrete diffusion modeling. It introduces the Score Entropy loss, novel objective that extends score matching to discrete spaces by directly modeling the ratios of data probabilities. This approach addresses the challenges of applying traditional score matching to discrete data and enables the construction of discrete diffusion models that are both thcoretically sound and empirically effective. SEDD demonstrates competitive performance with autoregressive models like GPT-2 on standard language modeling benchmarks. Notably, it achieves comparable zero-shot perplexities and offers advantages in generation quality and efficiency. For instance, SEDD generate high-quality text samples with 4x lower generative perplexity when matching function evaluations and requires 16x fewer function evaluations match the generative perplexity of standard autoregressive sampling methods. Moreover, SEDD enables arbitrary infilling beyond standard left-to-right prompting, matching the quality of nucleus sampling without the need for specialized training or sampling techniques.
Concurrently, several non-Gaussian continuous diffusion models have been proposed to address the limitations of traditional Gaussian-bart based approaches, particularly in handling data with bounded support or preserving structural details in images.
Beta Diffusion (70 introduces novel generative modeling method that integrates demasking and denoising to generate data within bounded ranges. Utilizing scaled and shifted beta distributions, employs multiplicative transitions over time create both forward and reverse diffusion processes. This approach maintains beta distributions in both the forward marginals and the reverse conditionals, given the data at any point in time. Unlike traditional diffusion models relying on additive Gaussian noise and reweighted evidence lower bounds (ELBOs), Beta Diffusion is multiplicative and optimized with KL-divergence upper bounds (KLUBs) derived from the convexity of the KL divergence. Experimental results demonstrate its unique capabilities in generative modeling of range-bounded data and validate the effectiveness of KLUBs in optimizing diffusion models.
propose generalized class of diffusion models that offer the best of both standard Gaussian denoising diffusion and inverse heat dissipation. By defining blurring through a Gaussian diffusion process with non-isotropic noise, this approach bridges the gap between inverse heat dissipation and denoising diffusion. It sheds light on the inductive bias resulting from this modeling choice and demonstrates the capability to better learn the low-to-mid frequencies within datasets, which plays crucial role in representing shapes and structural information.
Blurring Diffusion Models 7
48

---

Edge-Presee Pregeerrres Noise [72] for diffusion introduces a content-awary diffusion model explicitly trained to learn the non-isotropic edge information dataset. Inspired by anisotropic diffusion in image processing, this model incorporates an edge-aware noise scheduler that varies between edge bageeprrserving and isotropic Gaussian noise. The generative process converges faster results that more closely match the target distribution and better learns the low-to-mid frequencies within the dataset, crucial for representing shapes and structural information. This edge- dige-preserving diffusion process consistently outperforms state-of-the- baselines in unconditional image generation and is particularly robust for generative tasks guided by a shape-based prior, such as stroke-to-image generation
While these models offer significant advancements in handling specific data characteristics, they still require dequantization and rely on surrogate objectives. In contrast, ItDPDM models corruption with a reversible Poisson channel, maintaining discrete latent space, supporting bidirectional perturbations, and-via the I-MPRL. identity-trane transformnng the Minimum Poisson Reconstruction Loss into an exact likelihood integral instead of a bound. This unifies the tractability of information- theoretic Gaussian diffusion with the fidelity of discrete-state models, yielding closed-form NLLs, scalable continuous-time sampling, and strong empirical performance on sparse, skewed, and over- dispersed count data
IDPDM differs fundamentally from the above lines. By modelling corruption with a reversible Poisson channel, 1DDPDM keeps the latent space discrete. supports bidirectional perturbations, and-via the I-MPRL identity-turn the Minimum Poisson Reconstruction Loss into an exact likelihood integral instead of bound. This unifies the tractability of information-theorrtic Gaussian diffusion with the fidelity of discrete-state models, yielding closed-form NLL.s, scalable continuous- time sampling, and strong empirical performance on sparse, skewed, and over-dispersed count data.
NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility. transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the
relevant information is Not Available.
Please provide a short (1-2 sentence) justificatin right after your answer (even for NA).
The checklist answers reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
integral part of your paper submission. They visible to the
The reviewers of your paper will be asked to use the checklist one of the factors in their evaluation. While "[Yes] " is generally preferable to "[No] ", it is perfectly acceptable to answer "[No] " provided a proper justification is given (c.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] " or "[NA] is not grounds for rejection. While the questions are phrased in binary way, acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT; please:
Keep the checklist subsection headings, questions/answer. and guidelines below.
49

---

Do not modify the questions and only use the provided macros for your answers.
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction accurately reflect the paper's contributions and scope. They clearly motivate the need for discrete-state diffusion models for non-negative data and introduce the Poisson diffusion process with the coormation-theoreee Poisson Reconstruction Loss (PRL). Theoretical contributions like the I-MPRL identity are matched by strong empirical results across synthetic, image, and symbolic music domains. The claims are balanced, emphasizing likelihood modeling and robustness across varied discrete data distributions over generation quality, which is appropriately acknowledger as a current limitation.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. It is fine to include aspirational goals as motivation as long it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Limitations are discussed explicitly in Section 6 of the manuscript. The paper notes that ItDPDM is proof-of-cone and currently does not match oate--of-the-ar results on real-world image and music generation tasks. The authors attribute this to limited training epochs, fixed sampling parameters, and the use of non-tuned architectures, and outline steps for improvement in future work.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "LLimitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-ted system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
50

---

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
The paper includes several theoretical results with clearly stated assumptions (e.g., Lemmas 1-4). All key results are numbered, with most them proved formally in Appendix, including proofs of various lemmas, results, tail bounds, denoising relations, CTCS extensions. The I-MPRI. identity and likelihood derivations are also supported with mathematical justifications in the main text and supplementary material.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
The paper provides detailed descriptions of training settings, datasets (synthetic, CIFAR- 10, Lakh MIDI), architectures, and loss functions (with most of the details being deferred to Appendix). Algorithms 1 and 2 outline training and sampling procedures, and all model components are described enable reproduction of the main results.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
51

---

(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily
model architecture, the paper should descrihe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open- open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the partieular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: The paper will provide open access to the full codebase in a GitHub repository, which will be made public after the review process. The repository includes all necessary scripts, hyperparameters, and instructions to reproduce the main experiments and appendix results across synthetic, image, and symbolic music datasets.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
bublic/guides.o Saaiai/guuide///eeeuuiiieiee
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ httpps..ccpubbiccc wwwps///gieesseeee http:///ggiiiee//eeuuubbbbibaai http://ooooSuumsiiiiooooo
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
The paper provides full training and evaluation details, including data splits, loss functions, optimizers, learning rates, batch sizes, and number of epochs. Architectural configurations and noise schedules are also specified, enabling clear interpretation of results.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
52

---

The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report enror bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
The paper reports error bars for experiments where significant, capturing variability across multiple random seeds. All reported error bars represent standard devia- tions, and their interpretation is stated clearly in the text and figure captions (e.g., Table and Appendix).
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: The paper reports compute details in Appendix, including the use of AWS instances, running time and memory requirements. Training times and number of epochs per experiment are specified. While the paper focuses on final experiments, the total compute for all reported results modest relative to standard diffusion models.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics http://ttw......oo//oo//////hhaa
53

---

Answer: [Yes]
Justification: The research complies with the NeurIPS Code of Ethics. It does not involve human subjects, sensitive data, or models with foreseeable misuse risks. All datasets used (CIFAR- 10, Lakh MIDI) are publicly available and ethically sourced.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The paper includes Broader Impacts section discussing potential benefits with risks being deferred to the Appendix. Positive impacts include improved modeling of structured, discrete data such as symbolic music, low-light images medical records. Risks include misuse of generative models in sensitive domains; however, the discrete nature of ItDPDM may reduce risk compared to high-fidelity continuous generators. Responsible release and domain-specine safeguards are noted as future considerations.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups). privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
The paper introduces a diffusion based generative modeling framework for discrete, non-negative data, such as symbolic music and count-valued image distributions. does not involve large pretrained language image generation models, nor does it use or release scraped web data. Therefore, we helieve poses low risk for misuse, and no special safeguards required for responsible release.
Justification:
54

---

Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All baseline models are properly cited following the original authors' guide- lines. The datasets used (CIFAR-10 and Lakh MIDI) open-source and publicly available under permissive licenses (MIT-style and CC licenses, respectively). No proprietary or scraped assets are used.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thcode. www.coo//daaa.ee com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release any new assets. All experiments use publicly available datasets and reimplement or cite existing models as baselines.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
55

---

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve srorcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: This paper does not involve crowd-sourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: LLMs only used for paper formatting or editting.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our L.M policy huttps://neurd: ce. / for what should or should be described.
56


Reviews:
[{"review_id": "k8tlGFh3lr", "reviewer": "Reviewer_XDNg", "summary": "This paper introduces a generative model for non-negative discrete data based on information-theoretic Poisson diffusion process. The main idea is based on the Poisson reconstruction loss. This model seems to improve the performance in terms of NLL and Wasserstein distance on real-world discrete datasets.", "questions": "1. The theorems are proved using the Poisson channel, which is very domain-specific, How does this generalize to multimodal data?\n2. What is the gap between the exact estimator and the Monte Carlo estimator?\n3. Is there a reason that PRL works better on Zipf and PoissMix but worse on Half Cauchy?\n4. How would this work for low SNR scenarios? how bout the choice of architecture or timesteps?", "limitations": "- It is only applicable to non-negative discrete data. \n- Lack of ablation study and experimental baseline. \n- Limited reproducibility and insufficient robustness analysis. \n- The claims about the exact likelihood are exaggerated.", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strength:\n1. It is easy to follow and read. The Poisson reconstruction loss is well justified through multiple identities. \n2. Experimental results are sufficient and they are performed over real-world datasets. \n\nWeakness: \n1. I am not sure if I understand correctly but this paper constantly claims exact likelihood computation but seems like in Eq 17 we still have an intractable integral over $SNR(\\gamma)$ and then importance sampling is used in Eq 19. How is this exact? I believe this claim is misleading. \n2. There is no comparison to recent discrete-state score-based models such as D3PM with tuning. Also, no ablation study on the SNR schedule, etc. \n3. CIFAR 10 and LakhMODO results are only trained over 600 epochs and use only 100 log SNR steps. This seems to undermine the significance of performance claims on those datasets. \n4. Also, no robustness experiment is provided meaning what happens if data is sparse or noisy. \n5. There are redundancies and inconsistencies (e.g., $\\hat{x}(z_\\gamma, \\gamma), <X>_z, x^*$ used interchangeably?  Also, in Eq 16, the log-likelihood is in terms of marginalization but the implementation relies on approximation with NNs and sampling. \n6. Despite the theoretical guarantees, it is hard to reproduce the results. No training time, or used compute is provided. \n7. The idea of replacing squared error or generalizing better than ELBO is oversold as it only applies to discrete non-negative data.", "was_revised": true, "final_justification": "I appreciate the authors' response, but based on what I provided here and other reviews, I keep my score. I believe this paper would benefit from several rounds of revision before it is ready for submission.", "rebuttal": "We sincerely thank the reviewer for their thorough feedback and valuable suggestions. We believe that many of the concerns raised might stem from misunderstandings, which we clarify below:\n\n**Weaknesses**:\n\n**1. “I am not sure if I understand correctly but this paper…..”**\n\nEq. 17 rewrites the KL-divergence between the forward Poisson $q_\\gamma$ and the learnt reverse $p_\\theta$ as an expectation over $\\gamma$. So, the integrand is analytic: after marginalising out $z_\\gamma$, **no** latent variables remain. Hence, the NLL is exactly defined and only numerical step is integral evaluation. In practice, we use importance-sampling quadrature for this, yielding an unbiased estimate. Conventional diffusion incurs two levels of approximation: a) the **ELBO/surrogate loss** replaces the true log-likelihood, b) Monte-Carlo quadrature (or the integral). Our Poisson formulation eliminates the first level to work with the true likelihood as in [1], which also refers to their method as “exact likelihood”. To avoid any ambiguity we will replace “*exact likelihood*” with “*likelihood-consistent objective with tractable 1-D quadrature*” and provide an upper bound on the estimator's variance in the final version. \n\n**2. “There is no comparison to recent discrete-state…..”**\n\nIn the case of images and symbolic music (line 255), we already compare against a state-of-the-art variant of the vanilla D3PM ([4]) from [2]. With same synthetic setup, we performed an ablation study on the top-2 gamma schedules for each baseline and the results (evaluated using WD) are as follows:\n\n| Dist. | ItDPDM (logit) | ItDPDM (uniform) | LTJ (linear) | LTJ (geometric decay) | DDPM (linear) | DDPM (cosine) |\n|--------------|----------------|------------------|--------------|------------------|----------------|----------------|\n| PoissMix     | **0.99**           | 1.30             | **1.21**         | 1.26             | 3.76           | **3.60**           |\n| ZIP          | **0.56**           | 0.71             | **0.69**         | 0.83             | 2.31           | **2.22**           |\n| NBinomMix    | **1.39**           | 1.44             | **1.15**         | 1.19             | 4.89           | **4.50**           |\n| BNB          | 0.67           | **0.58**             | **0.65**         | 0.66             | 1.89           | **1.86**           |\n| Zipf         | **0.48**           | 0.67             | **0.73**         | 0.95             | **1.51**           | 1.53           |\n| YS           | **0.14**           | 0.22             | **0.17**         | 0.21             | 0.32           | 0.32           |\n\nWe will incorporate this ablation study in our final version.\n\n**\"3. CIFAR 10 and LakhMODO results are only trained over 600 epochs…..\"**\n\nFigure 7 in the main paper shows the loss curves flattening early; thus the 600-epoch setting reported is already in a converged regime (drop in loss of $10^{-4}$ order thereafter for every 100 epochs). We limited all baselines to 100 steps, 600 epochs to keep compute budgets comparable (as in [1]). Extended training of ItDPDM to 1500 epochs earlier with larger log-SNR grid gave very little NLL gain (<0.002 bpd).\n\n**4. “Also, no robustness experiment is provided…….”**\n\nOur experiments already target **sparse/zero-inflated** and **noisy** regimes:\n\n**Synthetic experiments**: In Sec. 5.1, for sparsity, we include *ZIP* and *PoissMix*: datasets which have mass mostly at 0 and ItDPDM achieves the best performance on both. \n\n**Noisy data**: We assert that ItDPDM demonstrates strong reconstructive capabilities for noisy data. For instance, 1000 CIFAR-10 images corrupted at very low SNR (log SNR = –5) are recovered to log SNR = 4 (ref. image in App. Fig.19). This is also illustrated in Fig. 4, where reverse process recovers the image despite extensive zeroing out of pixels via Poisson noising. This highlights the robustness of ItDPDM, and we will add more reference images, including high-resolution *CelebA* images for the final version (in the Appendix). \n\n**Theoretical justification**: In addition, [3] provides guarantees for a mismatched estimator (for bounded divergence between source and “mismatched” distribution), so even for sparse/noisy data, the near-optimal denoiser in the Poisson channel has theoretical guarantees (upper bounds) on the estimation error. \n\n**5. “There are redundancies and inconsistencies……”**\n\n**Notations**: We clarify the key notations as follows: $\\hat X*$ to denote the optimal denoiser (essentially conditional expectation of $X$ given $Z_\\gamma$, which is further denoted by  $\\langle X \\rangle_z$) and $\\hat X$ to denote learnt suboptimal denoiser. We will rephrase notation to avoid any ambiguity in the final version.\n\n**Implementation**: The identity itself is exact: its integrand depends on the Bayes‐optimal estimator $\\hat x* (z_\\gamma,\\gamma)$. At training/inference time, we substitute the learned network output $\\hat x(z_\\gamma,\\gamma)$ in place of $\\hat x*$ and evaluate the resulting integral using the unbiased Monte Carlo estimator (see Sec.~4). In practice, this is computed as outlined in (1.)\n\n**6. Despite the theoretical guarantees, it is hard to reproduce……..”**\n\nWe plan to release code, configs, and pre-trained checkpoints. In terms of the wall-clock time, the training time is 3.5 minutes per epoch. The wall-clock time is 35 hours (600 epochs) for both ItDPDM and other baselines. All experiments were run on an AWS **g5.12xlarge** instance (us-west-2):\n\n| Resource | Spec |\n|----------|------|\n| **GPUs** | 4 × NVIDIA **A10G** (24 GB each, 22.3 TFLOPS FP16) |\n| **CPU / RAM** | 48 vCPUs (Intel “Ice Lake”), 192 GiB system RAM |\n\n**7. \"The idea of replacing squared error or generalizing better than ELBO is oversold as it only applies to discrete non-negative data\"**\n\nWe thank the reviewer for pointing this out, we will revise the wording as needed to accurately reflect that the improvement primarily applies to discrete non-negative data. At the same time, we would like to point out that a Poisson-consistent objective is valuable for the **large growing class of discrete, count-valued, zero-inflated data**. For detailed evidence, examples and exposition, we refer the reviewer XDNg to our **response to \"Weaknesses\" section to reviewer 7eDB** [a]. (due to lack of space here)\n   \n**ItDPDM works for any continuous data**: App. Sec. C.4 demonstrates how ItDPDM generalises even to continuous distributions with ItDPDM outperforming DDPM in all cases except one (App. Table 7). (some preprocessing would be necessary for general distributions, this is again discussed in [a])\n\n**Questions**:\n\n**1. “The theorems are proved using the Poisson channel….”**\n    \nThe *Poisson noising* is analogous to Gaussian noising in DDPM: it defines how we corrupt samples, not what *underlying data distribution* is. All the theorems make no assumption on $p(x)$, so the theory naturally extends to any data modality. Our experiments include strongly multimodal data (PoissMix, NB-Mix, BNB); along with real-world datasets that are highly multimodal.\n\n**2. “What is the gap……”**\n\n**Example**: pick a binary $\\mathbf{X} \\in \\{\\pm 1\\}$ with equal probability (BPSK) through a Gaussian channel \n$Y = \\sqrt{\\gamma} X + N$, where $N \\sim \\mathcal{N}(0,1)$. \nThe classic I-MMSE identity gives:\n$\n\\frac{d}{d\\gamma} I(X;Y) = \\frac{1}{2} \\mathrm{mmse}(\\gamma)\n \\Rightarrow \n\\frac{1}{2} \\int_0^\\infty \\mathrm{mmse}(\\gamma) \\ d\\gamma = I(X;Y)\\big|_0^\\infty.\n$\n\nAt $\\gamma = 0$, $I = 0$; as $\\gamma \\to \\infty$, $I \\to H(X) = \\ln 2$ (nats). \nTherefore, the exact value is: $I = \\frac{1}{2} \\int_0^\\infty \\mathrm{mmse}(\\gamma) d\\gamma = \\ln 2$.\n\nFor this BPSK case, a convenient form of the MMSE is: $\\mathrm{mmse}(\\gamma) = 1 - \\mathbb{E} \\left[ \\tanh^2\\left(\\sqrt{\\gamma} Y \\right) \\right],$\nwhich follows from the posterior mean: $\\mathbb{E}[X |Y] = \\tanh(\\sqrt{\\gamma} Y)$. Using an $n$-sample Monte-Carlo (MC) estimator here shows a $\\frac{1}{\\sqrt n}$-error decay (this can also be proven using CLT) and is of the order of $10^{-3}$ for $n > 1000$. Similarly, for MMLE, a Gamma prior on $X$ gives an exact closed-form linear estimator (Lemma 2 in the paper) and the MC estimator again shows a $\\frac{1}{\\sqrt n}$-decay with increase in $n$. For example, if $X \\sim \\text{Gamma}(2,3)$, the MC error is of the order of $10^{-2}$ for $n > 1000$.\n\n**3. “Is there a reason that PRL works better on.”**\n\nThe paper emphasizes PRL as an objective primarily for non-negative data and Zipf/PoissMix datasets exhibit sparse, heavy-tailed behaviour suited PRL. Half-Cauchy is an over-dispersed continuous case where LTJ’s thinning process has an extra quadratic-tail KL term to better fit those outliers. In this case, PRL is only second to LTJ and better than DDPM, while remaining superior in other continuous cases.\n    \n**4. “How would this work……”**\n\nWe refer the reviewer to: point (4.) in Weaknesses for low SNR scenarios. and point (2.) for log SNR-based ablation. We use SoTA architectures (UNet for images, ConvTransformer for music) while tuning the 1) training objective and 2) diffusion modeling.\n\nWe once again thank Reviewer XDNg for their feedback and for giving us the opportunity to address their concerns. We truly hope the above response helps address and clarify the mentioned weaknesses/questions.\n\n**References**: \n\n[1] Xianghao Kong, Rob Brekelmans, and Greg Ver Steeg. Information‑Theoretic Diffusion. ICLR 2023.\n\n[2] Matthias Plasser, Silvan Peter, and Gerhard Widmer. Absorbing‑State Discrete Denoising Diffusion Probabilistic Models for Symbolic Music Generation. (2023).\n\n[3] Rami  Atar and Tsachy  Weissman. Mutual Information, Relative Entropy, and Estimation in the Poisson Channel. IEEE Transactions on Information Theory (2012).\n\n[4] Jacob Austin, Daniel D. Johnson, Jonathan Ho, Daniel Tarlow, and Rianne van den Berg. Structured Denoising Diffusion Models in Discrete State‑Spaces. NeurIPS 2021."}, {"review_id": "CNYodmI3rW", "reviewer": "Reviewer_7eDB", "summary": "By employing the Poisson Reconstruction loss (PRL), ItDPDM addresses two key challenges: discrete data distribution estimation and accurate likelihood estimation, without requiring post-processing (soft discretization) or pre-processing (dequantization). Experiments on synthetic datasets demonstrate ItDPDM’s strong capability in estimating and generalizing over discrete data distributions. However, its performance on real-world datasets is suboptimal, likely due to limited training and insufficient hyperparameter tuning.", "questions": "The theoretical part seems good to me, but I am confused about the experimental part:\n\nQ1. All the experiments in this paper predict $\\hat{x}$ rather than $\\hat{\\epsilon}$. Could the authors clarify how reparameterization was handled in the experiments? For example, the MMSE calculation in Figure 7 is based on $x$ instead of $\\epsilon$.\n\nQ2. In the synthetic data experiments, DDPM does not seem to be a suitable baseline, since it is a continuous diffusion model. Did it use Poisson Reconstruction Loss (PRL) as the training objective? Additionally, the results of ItDPDM and LTJ are quite similar (see Figure 6, Figure 13, Table 7, and Table 1). Are the results in Table 1 averaged over multiple runs, and were experimental variances considered?\n\nQ3. For the real-world data experiments, Table 2 shows that the FID of IDDPM (Gaussian version trained from scratch) is higher than that of DDPM. Can the authors explain this result?", "limitations": "Since almost all experiments in this paper are conducted with models trained from scratch, but with limited training, architectural tuning, and hyperparameter searching, I hope the authors can provide improved results on real-world datasets during the rebuttal period.", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths\nThe proposed Info+Poisson noise closed-form discrete diffusion framework is novel. Figure 1 clearly summarizes four different diffusion methods based on training objectives and state space, and ItDPDM effectively fills the gap in accurately estimating discrete data distributions.\n\n## Weaknesses\nThe method can only estimate distributions for non-negative data and tends to produce high-biased estimates. In the case of image data, this may lead to generated samples that appear overly bright or overexposed (as illustrated in the example in Figure 4), which in turn results in higher FID scores.", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thorough, constructive feedback, and appreciate that they found our work novel. We also believe that the concerns raised mostly stem from misunderstandings, which we clarify below:\n\n“**Weaknesses**: The method can only estimate distributions for non-negative data and tends to produce high-biased estimates………”\n\n**On non-negativity & domain suitability**: **ItDPDM** is explicitly designed for non-negative discrete data (e.g., images, counts, symbolic sequences), so the non-negativity constraint is a *feature*, not a limitation. Many real-world data domains (pixel intensities, photon counts, MIDI tokens, word frequencies, genomics, etc.) are naturally non-negative and discrete with no pre/post-processing needed. Below are some examples of application areas supporting this argument: (DS = discrete-state)\n\n- **Images & audio** are ultimately stored as *integers*: 8-bit RGB or 16-bit PCM; discrete-state diffusions already show gains here ([1], [2], [3])\n- **Symbolic music** (note numbers, durations), **MIDI** event streams: state-of-the-art models use discrete diffusions ([1], [4])\n- **Photon-limited imaging, low-light vision** record integer photon counts ([5], [6]), \n- **Single-cell RNA-seq, metagenomics**, and other omics assays produce gene–count matrices ([7]), \n- **Neuromorphic event cameras** output per-pixel spike counts [8]\n- **Word-frequency** bags in NLP (Zipf distribution), **Poisson processes** in queueing are classic count data [9].\n\nWe also refer the reviewer to the last App. Sec. K on related work that highlights the growing need for handling non-negative discrete data carefully. \n\n**ItDPDM also works for any continuous data**: App. Sec. C.4 demonstrates how ItDPDM generalises even to continuous distributions with ItDPDM outperforming DDPM in all cases except one (App. Table 7). We note that preprocessing would be necessary for general continuous distributions to bring them to a non-negative range after which ItDPDM can be directly used.\n\nFor most continuous data in practice, a simple shift/rescale (e.g., mapping $[0,1]$ to $[0,\\infty)$) or more generally, an **invertible transform** would suffice and does not affect results with regards to likelihood computation, denoising or generation.\n\n**Regarding the “over-bright” appearance in Fig.~4:** This snapshot shows a result of using a few denoising steps ($T=10$) starting at very low SNR ($\\log \\gamma \\approx –5$), where the Poisson prior contributes near-zero photons. Since intensity reconstructions start from black and accumulate photons, some intermediate frames appear brighter. However, final samples (after all reverse steps) are well-balanced, as evidenced by comparable or better FID/SSIM on CIFAR-10 (see Table~3) and App. Fig. 19. For this particular image from the CIFAR10 dataset, we empirically verify the denoised image by the proposed ItDPDM to be closer to the ground truth (in terms of mean squared error) than the one denoised by the Gaussian DDPM baseline. This eliminates any concerns regarding over brightening of image.\n\nFigure 10 in App. C.1 compares generated vs. ground truth pitch histograms for symbolic music, showing **no** systematic amplification bias (or a shift). To further clarify, the final Gaussian denoised image is not the ground truth, with the ground truth in fact being closer to the Poisson output. We totally understand that this could have caused confusion, and will add the ground truth image for reference in the final version.\n\n**On the “high-biased” NLL estimate**: Our NLL estimator is an analytic upper bound (Eq.~18), reflecting the gap between the learned denoiser $\\hat{x}$ and the Bayes-optimal estimator $\\hat{x}^*$. This is a principled design choice: we report a conservative likelihood rather than a variational lower bound like ELBO. In practice, this gap shrinks with training and is $<0.02$ bits/dim on CIFAR-10. \n\n**Questions**:\n\n**1. “All the experiments in this paper predict $\\hat x$ rather than $\\hat \\epsilon$.....\"**\n\nAll models in the paper *output the clean signal estimate* $\\hat x(z_\\gamma,\\gamma)$ (without any explicit reparametrization); firstly, we do not train an $\\hat\\varepsilon$-network for the Poisson channel, because $\\varepsilon$ has no natural analogue in this case. For the Gaussian channel, the two parameterisations are interchangeable, as shown below: (starting from the Gaussian channel introduced in the main paper)\n\n\\begin{align*}\n\\varepsilon - \\hat{\\varepsilon}\n&= z_{\\gamma} - \\sqrt{\\gamma} x - (z_{\\gamma} - \\sqrt{\\gamma}\\hat{x}) = \\sqrt{\\gamma} (x - \\hat{x})\n\\Longrightarrow\\\n\\|\\varepsilon - \\hat{\\varepsilon}\\|_2^{2}\n   = \\gamma\\\\|x - \\hat{x}\\|_2^{2}.\n\\end{align*}\n\nThis implies:\n$\nMMSE_{x}(\\gamma) = \n\\mathbb{E}\\bigl[\\lVert x - \\hat{x}(z_\\gamma,\\gamma) \\rVert_2^2\\bigr],\n$\nand therefore\n\n$MMSE_{\\epsilon}(\\gamma) =\n\\mathbb{E} \\bigl[\\lVert \\varepsilon - \\hat{\\varepsilon}(z_\\gamma,\\gamma) \\rVert_2^2\\bigr] = \\gamma MMSE_{x}(\\gamma)\n$\n\nTherefore, under the squared-error loss, the two MMSEs differ only by a scale factor. We also verify this empirically, with both the variants of DDPM yielding similar performance. For the Poisson channel, we optimise the Poisson Reconstruction Loss (PRL), which is typically expressed in $\\hat x$-space. Therefore, reporting MSE in $\\hat x$-space keeps metrics consistent across Gaussian and Poisson baselines.\n\n**2. “In the synthetic data experiments, DDPM does not….”**\n\n* Table 1 numbers are mean ± s.d. (standard deviations) over five random-seed runs (50k samples each); error bars are printed in the table and these details are outlined in Appendix Sec. C.2.\n* DDPM is trained with both MSE and PRL objectives (results shown below), with the PRL training exhibiting better performance. This emphasizes **two takeaways**: \n\n(i) proposed PRL objective improves performance *regardless* of channel and \n\n(ii) using poisson channel + PRL (i.e. ItDPDM) is the **best overall**.\n\n| Distribution | DDPM (PRL) | DDPM (MSE) | ItDPDM | LTJ |\n|--------------|-----------:|-----------:|-------:|----:|\n| PoissMix     | 3.74       | 3.76       | **0.99** | 1.21 |\n| ZIP          | 2.19       | 2.31       | **0.56** | 0.69 |\n| NBinomMix    | 4.45       | 4.89       | 1.39 | **1.15**|\n| BNB          | 1.84       | 1.89       | 0.67 | **0.65** |\n| Zipf         | 1.40       | 1.51       | **0.48** | 0.73 |\n| YS           | 0.31       | 0.32       | **0.14** | 0.17 |\n\n- ***Regarding the apparent similarity to LTJ***: a zoom-in of the PMFs (refer Fig.11–12, App.C.3 “Zoomed-in look at PMF plots”) and the full metric table shows that ItDPDM outperforms LTJ on:\n\na) **9/12** distributions in Wasserstein distance (WD) (including continuous distributions) and \n\nb) **4/6** distributions in negative log-likelihood (NLL). \n\nThe synthetic suite therefore highlights (a) the limitations of Gaussian DDPM on skewed or zero-inflated counts and (b) the edge of our exact-likelihood ItDPDM over the variational LTJ.\n\n**3. “For the real-world data experiments, ……….”**\n\n- We would like to **clarify** that Table 2 compares the *test-set NLLs* of various schemes (not the FID). All models are trained under a fixed compute budget (600 epochs, 100 denoise steps). While the IDDPM backbone has ~2x more parameters and uses DDIM-style self-conditioning; its convergence under this shallow schedule is slightly slower than DDPM, resulting in marginally higher NLLs. With extended training upto 2,000 epochs or sampling with 1000-steps, IDDPM can regain the expected (small) advantage (upto 0.02 bits/dim).\n- The key takeaway remains that: PRL improved NLL significantly (**by 35-40%**) irrespective of architecture, and the Poisson+PRL combination (ItDPDM) yields the lowest NLL overall. Thus, the results here underscore our central claim: that a Poisson-consistent, PRL-based objective is the best for modelling discrete, non-negative data, while backbone choice only fine-tunes performance once sufficient training budget is available.\n\n**Further real-world experiments:**\n\nWe also ran experiments on the **CelebA** (64x64) (image) dataset for 500 epochs (30 hours wall-clock time) with 100 logSNR (denoising) steps per image, for proposed ItDPDM and Gaussian DDPM baseline. ItDDPM has an absolute 7 point **lower** FID score (or 0.21 dB better) compared to the Gaussian DDPM baseline.\n\nWe once again thank reviewer 7eDB for their thoughtful feedback and we truly hope the above mentioned points provide further clarification regarding the experimental parts.\n\n**References**:\n\n[1] Matthias Plasser, Silvan Peter, and Gerhard Widmer. Absorbing‑State Discrete Denoising Diffusion Probabilistic Models for Symbolic Music Generation. (2023).\n\n[2] Jacob Austin, Daniel D. Johnson, Jonathan Ho, Daniel Tarlow, and Rianne van den Berg. Structured Denoising Diffusion Models in Discrete State‑Spaces. NeurIPS 2021.\n\n[3] Mingyuan Zhou, Tianqi Chen, Zhendong Wang, and Huangjie Zheng. Beta Diffusion. NeurIPS 2023.\n\n[4] Gautam Mittal, Jesse Engel, Curtis Hawthorne, and Ian Simon. Symbolic Music Generation with Diffusion Models. IJCAI 2021.\n\n[5] Cindy M. Nguyen, Eric R. Chan, Alexander W. Bergman, and Gordon Wetzstein. Diffusion in the Dark: A Diffusion Model for Low‑Light Text Recognition. 2024.\n\n[6] Shlomo Shamai and Aaron D. Wyner. A Binary Analog to the Entropy‑Power Inequality. IEEE Transactions on Information Theory, 36(6): 1428–1430, 1990.\n\n[7] Valentine Svensson and Lior Pachter. Droplet scRNA‑seq is not zero‑inflated. Genome Biology, 21(1): 35. 2020\n\n[8] Xu Zheng, Yexin Liu, Yunfan Lu, Tongyan Hua, Tianbo Pan, Weiming Zhang, Dacheng Tao, and Lin Wang. Deep Learning for Event‑based Vision: A Comprehensive Survey and Benchmarks.\n\n[9] I. Bar‑David. Communication under the Poisson regime. IEEE Transactions on Information Theory, 1969."}, {"review_id": "gfwjlvsebP", "reviewer": "Reviewer_3Fg2", "summary": "While diffusion based on Gaussian noise has become popular for images, recent work has explored how different noise models may be more effective for different data types (masking noise for text, e.g.). \nThis paper introduces a new approach for modeling non-negative discrete or continuous data based on denoising in a Poisson channel. \nPoisson noise is a natural and well-studied channel. The authors are able to transfer a wealth of information theory results leading to an elegant relation between probability and Poisson denoising. \nThe authors discuss practical implementation and demonstrate proof of concept results across diverse types of distributions and data modalities.", "questions": "These are just minor comments / thoughts. \n\n- I get the point of Fig. 3, but it looks strange at a glance. I didn't feel I fully understood it. I guess Fig. 6 is supposed to show that you fixed this problem? \n- Eq. 11 is an intriguing expression that reminds me of your reference [56]. I believe you could write a first order ODE for P_gamma(z), then use this to define sampling dynamics (based on adding counts in a random order autoregressive way).  This would avoid the need for gamma noise scheduling. \n- Did you need to define a gamma noise schedule? I missed that detail. It would be nice in general if there were space to discuss the sampling more. \n\nAnother nice paper that I was reminded of when you gave Lemma 2\n- Dytso, A., & Poor, H. V. (2020). Estimation in Poisson noise: Properties of the conditional mean estimator. IEEE Transactions on Information Theory, 66(7), 4304-4323.", "limitations": "yes", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 4, "strengths_and_weaknesses": "- There was a good overview of connections between probability and denoising coming from information theory literature. \n- Detailed exposition of properties of Poisson channel and loss. \n- Fig. 4 is a great illustration to understand the fundamental difference in the noising scheme in a glance. \n- Lots of great detail on theory and experimental details in supplementary material \n- The sampling algorithm, Alg. 2, seems simple too.\n\nExperiments \n- Interesting and diverse selection of datasets for capturing different tail behaviors, and diverse modalities is a major plus\n- Baselines seemed adequate for proof of concept. In synthetic case, DDPM and a recent paper that extends denoising to count data. For real data, a variety of diffusion-based approaches were compared. \n- Table 2(b). Very cool, it seems that it particularly excels with exactly modeling zero-inflated values compared to continuous approaches, if I'm reading it right.", "was_revised": true, "final_justification": "I maintained my accept score. \n- Authors did a good job responding to questions on derivations and statement of results. I generally found the derivations clear. (But was already familiar with some of the information theory results they used.)  \n- There was some pushback on results, but I thought the authors justified their choices and provided many of the things requested. The native formulation for count data is interesting and novel. I don't see the need for large-scale experiments as this is not a paper that is based on improving a standard benchmark with incremental modification of an existing method.", "rebuttal": "We sincerely thank the reviewer for their constructive feedback on our work regarding the novelty of our method, choice of experiments and for their intriguing follow-up questions. We have addressed the reviewer’s comments below:\n\n1. \"I get the point of Fig. 3, but it looks strange at a glance. I didn't feel I fully understood it. I guess Fig. 6 is supposed to show that you fixed this problem?\"\n\nFig. 3 (the diagnosis) contrasts a *continuous* Gaussian DDPM (green) fit with the true density (blue) on the NYC-Taxi pick-up data. The underlying distribution is sharply bimodal. The top panel overlays the *density estimate* produced by DDPM (since it maps discrete counts to a continuous latent), which smooths the second peak into a long shoulder on the discrete histogram; therefore totally failing to capture the second peak.\n\n*Fix shown*: Yes, Fig. 6 plots the learned PMF of each model for all six synthetic datasets, demonstrating that ItDPDM (orange) now places correct mass on the modes while DDPM (green) still flattens the tail. In the revision: we will also (i) label the two modes explicitly in the caption, and (ii) add a sentence pointing the reader forward to Fig. 6 for the quantitative confirmation, avoiding any ambiguity.\n\n2. \"Eq. 11 is an intriguing expression that reminds me of your reference [56]. I believe you could write a first order ODE for P_gamma(z), then use this to define sampling dynamics (based on adding counts in a random order autoregressive way). This would avoid the need for gamma noise scheduling.\"\n\nExactly (great observation)! Eq. 11 is actually a closed form solution to the following *first order ODE*:\n\n$$\n\\partial_\\gamma P_\\gamma(z) = \\frac{(z+1)P_\\gamma(z+1) - (\\gamma + z)P_\\gamma(z)}{\\gamma}\n$$\n\nSolving this Kolmogorov-forward ODE forward (or its time-reversal) yields a *continuous-time counting process* akin to the discrete-state SDE in [56]. We choose the log-SNR schedule plus importance sampling for two practical reasons:\n\n(i) it let's us reuse standard CT diffusion solvers (Euler, Heun, DPM-Solver), and\n\n(ii) it helps us build our implementation on top of standard Gaussian baselines. \n\nThe autoregressive “add counts in random order” interpretation is a promising idea; we will note this connection in Sec. 3 and include a brief discussion in Appendix Sec. G for future work.\n\n3. \"Did you need to define a gamma noise schedule? I missed that detail. It would be nice in general if there were space to discuss the sampling more.\"\n\nAn explicit/nuanced “noise schedule” is not necessary: $\\gamma$ is treated as a continuous random variable. During **training**, we sample:\n$$\n\\alpha = \\log \\gamma \\sim \\mathrm{Logistic}(\\mu=0,s=1);\\text{  } \\gamma = e^{\\alpha},\n$$\nclipped to the tail‐bound interval [$\\gamma_{\\min} \\approx 10^{-5},\\,\\gamma_{\\max}\\approx 10^{5}$].\n\nThis importance‐sampling procedure places more draws where the PRL integrand has high variance and provides an *unbiased* Monte Carlo estimate of the integral (Eq. 19).\n\nFor **generation** (Alg. 2) we integrate the reverse process from $\\gamma_{\\max}$ down to $\\gamma_{\\min}$.\nBy default we discretize that range into **100 log-spaced points**, but because the formulation is continuous-time, one can freely (i) use fewer points for a speed-up or (ii) plug in any adaptive ODE solver without retraining.\n\nWe additionally refer the reviewer to R1 response point (2.) under the *Weaknesses* section for an ablation on the gamma schedule.\n\n4. “Another nice paper that I was reminded of when you gave Lemma 2: \nDytso, A., & Poor, H. V. (2020). Estimation in Poisson noise: Properties of the conditional mean estimator. IEEE Transactions on Information Theory, 66(7), 4304-4323.”\n\nThank you for recalling our attention to this reference, we will cite this in our final version. We are once again grateful and sincerely appreciate that reviewer 3Fg2 found our work interesting."}, {"review_id": "aZf6kRUOoT", "reviewer": "Reviewer_Ptu5", "summary": "This paper presents ItDPDM, a diffusion-like model that combines exact likelihood estimation with discrete-state modeling. To this end, the paper also introduces poisson reconstruction loss (PRL). Some theoretical properties of this loss are then shown. The proposed ItDPDM is evaluated on both synthetic and real-world data, with a particular focus to compare against LTJ.", "questions": "Please see the weakness section.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 1, "clarity": 1, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n1. A new diffusion-like algorithm that enables exact likelihood estimation\n\n2. A new Poisson-based loss with some theoretical properties\n\n\nWeaknesses:\n\n1. Wrong proof of Equation (12): This equation plays a key role for exact likelihood calculation (e.g., Eq. (15)). However, in the proof of this equation (Lemma 12 in Appendix F), the differentiation is directly moved inside the expectation operator w.r.t. p(z_\\gamma|x) (line 1546 in the supplementary). I don't think it's valid because the conditional measure p(z_\\gamma|x) depends on \\gamma. (Please see the correct way to handle it in [1, Appendix A]). This calls the rest of the likelihood estimation results into question.\n\n2. Missing proof: In the proof of Lemma 2, I only see the \"only if\" part, but I couldn't find the \"if\" part. However, the main text seems to be highlighting the \"if\" part (see the sentence above Lemma 2), which lacks a solid proof.\n\n3. Weak numerics: In Table 1, among the six distributions chosen for likelihood evaluation, LTJ beats ItDPDM only on three. (Also note here that the boldfaces are very misleading, as they are not highlighting the best values, as they are supposed to be.) This calls into question whether the proposed method is advantageous numerically for likelihood evaluation. Plus, the generation quality of ItDPDM is not as good as shown in Table 3.\n\n4. Simplified univariate setting: Both the sampler and the loss are defined only for the univariate distributions. While this is as expected since Poisson distributions are primarily designed for count data. Nevertheless, it calls into question whether both of them are useful for multi-dimensional data. Also, it is not clear how the model and loss are evaluated for image data such as CIFAR-10.\n\n5. Unclarities and typos: I list the following for consideration, which indeed hindered my reading. in Lemma 4, P_{Z_\\gamma} as the marginal of z is not clearly specified. In Eq. (12), \\calP should just be P. In Algorithm 1, line 4 is inconsistent with Figure 4, line 6 contains a q(\\alpha) which is not defined, and it is not clear what data_transform function is in line 5. In Algorithm 2, \\hat{x}_t should be \\hat{x}_0, and why is [12] cited here?\n\n[1] Kong et al., Information-theoretic diffusion, ICLR 23.", "was_revised": true, "final_justification": "The authors' revised argument makes sense to me, and the experiments show some merit in the model. 2D synthetic experiments, though insufficient to me, are okay, given that there are other real-data experiments to back up. Although there might be a need to revamp the current draft (for both the contents and proofs), I lean towards the value of the contribution this paper brings.", "rebuttal": "We thank the reviewer for their thorough feedback and valuable suggestions. We have addressed the reviewer’s concerns below:\n\n**Weaknesses**:\n\n**1.\" Wrong proof of Equation (12): This equation plays a key role for exact likelihood........\"**\n\nWe sincerely thank the reviewer for pointing this out and bringing it to our attention. We now have an updated proof-sketch stated below, which will also be included in more detail in the revised version:\n\n**Setup.**\nFix $x\\ge0,\\ \\gamma>0,\\ \\lambda=\\gamma x$.\nWith $Z_\\gamma\\mid X=x\\sim\\mathrm{Pois}(\\lambda)$ define\n\n$$\nR_x(\\gamma)=\\sum_{z\\ge0} p_\\gamma(z|x)\\\n\\log\\frac{p_\\gamma(z\\mid x)}{p_\\gamma(z)} .\n$$\n\nWe can split this using the product-rule for derivatives:\n\n$$\n\\frac{\\mathrm d}{\\mathrm d\\gamma} R_x = T_1 + T_2,\n$$\nwhere \n$$\nT_1 = \\sum_{z}\\partial_\\gamma p_\\gamma(z|x)\n\\log \\frac{p_\\gamma(z|x)}{p_\\gamma(z)},\\text{ }\nT_2 = \\sum_{z}p_\\gamma(z|x)\\\n\\bigl(\\partial_\\gamma\\log p_\\gamma(z|x)-\\partial_\\gamma\\log p_\\gamma(z)\\bigr).\n$$\n\n$\\partial_\\gamma\\log p_\\gamma(z|x)=z/\\gamma-x$. \n\nTaking the required conditional expectation drives this term to zero. \n\nFor the marginal, we have:\n\n$$\n\\partial_\\gamma\\log p_\\gamma(z)=\n\\mathbb{E}\\left[-X+\\frac{Z_\\gamma}{\\gamma}|\\Big| Z_\\gamma=z\\right].\n$$\n\nHence:\n\n$$T_2=\\mathbb{E}\\bigl[\\mathbb{E}[X|Z_\\gamma]-Z_\\gamma/\\gamma\\bigr]$$\n\nNow, let $r(z)=\\log\\frac{p_\\gamma(z\\mid x)}{p_\\gamma(z)}$.\nWe use the Poisson-Stein identity:\n\n$$\n\\mathbb{E}\\bigl[(Z_\\gamma-\\lambda)h(Z_\\gamma)\\bigr]\n=\\lambda\\mathbb{E}\\bigl[h(Z_\\gamma+1)-h(Z_\\gamma)\\bigr]\n$$\n\nwith $h=r$, this gives\n\n$$\nT_1=\\frac{1}{\\gamma}\\mathbb{E}[(Z_\\gamma-\\lambda)r(Z_\\gamma)]\n      =x\\\\mathbb{E}[r(Z_\\gamma+1)-r(Z_\\gamma)].\n$$\n\nWe now have these ratios:\n$\\frac{p_\\gamma(z+1\\mid x)}{p_\\gamma(z\\mid x)}=\\frac{\\gamma x}{z+1}$ and\n$\\frac{p_\\gamma(z+1)}{p_\\gamma(z)}=\\frac{\\gamma\\\\langle X\\rangle_{z}}{z+1}$\nwith $\\langle X\\rangle_{z}:=\\mathbb{E}[X|Z_\\gamma=z]$ (from Lemma 4) yield\n\n$$\nr(z+1)-r(z)=\\log\\frac{x}{\\langle X\\rangle_{z}}, \n\\text{ and }\nT_1=x\\\\mathbb{E}\\\\left[\\log\\frac{x}{\\langle X\\rangle_{Z}}\\right].\n$$\n\nCombine these two terms now, to get:\n\n$$\n\\frac{d}{d\\gamma}R_x =\nx\\\\mathbb{E}\\\\left[\\log\\frac{x}{\\langle X\\rangle_{Z}}\\right]\n+\\mathbb{E}\\\\left[\\langle X\\rangle_{Z}-\\frac{Z_\\gamma}{\\gamma}\\right].\n$$\n\nSince $\\mathbb{E}[Z_\\gamma\\mid X=x]=\\gamma x$, the second expectation equals\n$\\mathbb{E}[\\langle X\\rangle_{Z}]-x$.  Therefore:\n\n$$\n\\frac{\\mathrm d}{\\mathrm d\\gamma} R_x\n=\\mathbb{E}\\\\bigl[x\\log\\frac{x}{\\langle X\\rangle_{Z}}-x+\\langle X\\rangle_{Z}\\bigr]\n=\\mathbb{E}\\\\bigl[\\ell(x,\\widehat x^{*})\\bigr],\n$$\n\nwhere $\\widehat x^{*}=\\langle X\\rangle_{Z}$ and\n$\\ell(u,v)=u\\log(u/v)-u+v$.\nThis is Eq. (12).\n\n\n\n**2. \"Missing proof: In the proof of Lemma 2, I only see the \"only if\".......\"**\n\n**On Lemma 2 (\"if\" part)**: We are once again grateful to the reviewer for catching this. The appendix currently proves the \"only-if'\" direction (linear conditional mean $\\Rightarrow$ Gamma prior). We will add the converse, which follows directly from Poisson--Gamma conjugacy: if $X \\sim \\mathrm{Gamma}(\\alpha, \\beta)$ and $Z_\\gamma \\mid X \\sim \\mathrm{Pois}(\\gamma X)$, then $X \\mid Z_\\gamma = z \\sim \\mathrm{Gamma}(\\alpha + z, \\beta + \\gamma)$, hence $\\mathbb{E}[X \\mid Z_\\gamma = z] = (\\alpha + z)/(\\beta + \\gamma) = az + b$ with $a = 1/(\\beta + \\gamma)$, $b = \\alpha/(\\beta + \\gamma)$. This completes both the \"**if**\" and \"**only if**\" directions of the proof.\n\n**3. \"Weak numerics: In Table 1, among the six distributions chosen......\"**\n\n**Note**: We sincerely request the reviewer to have another glance at Table 1.\n\nThere is a column for *“true NLL”* and boldfaces here reflect whichever baseline yields the closest estimate to the ground-truth NLL, thereby becoming the “best values”. We acknowledge that this may not have been immediately clear, and we will revise the caption of this table to clarify our intent.\n\nOn this metric, ItDPDM outperforms LTJ on **4 out of 6** distributions. The remaining two cases (BNB, NBinomMix) have an implicit bias towards LTJ due to the binomial thinning process that underlies LTJ’s noising scheme. Overall this emphasizes the numerical advantage of ItDPDM: it consistently yields likelihood estimates that more closely align with the true NLL across a diverse set of discrete distributions.\n\n**Regarding generation quality**: This is apparent currently due to CIFAR10's 32x32 resolution: we also refer the reviewer to Appendix Fig. 19 for comparing a denoised CIFAR-10 sample from DDPM and ItDPDM. In the context of symbolic music, Appendix Fig. 10 shows that the pitch histogram from ItDPDM-generated symbolic music closely matches the ground truth distribution. Additionally, Fig. 9 (App. B.5.1) includes piano roll visualizations for unconditionally generated samples. We will include: 1) more image generation examples for reference, including some high-resolution images and 2) generative fidelity metrics for a high-resolution image dataset (like *CelebA*) in the final version for further strengthening our work. \n\n**4. \"Simplified univariate setting: Both the sampler and the loss........\"**\n\n**Experiments for multivariate setting**: We acknowledge that the results presented in the paper are restricted to a univariate setting for simplicity. To further demonstrate ItDPDM’s ability to model multivariate discrete data, we extended our synthetic experiments to a bivariate (2D) joint distribution. First example: we define the joint distribution as a product of two independent univariate marginals using the same parameters from the 1D synthetic experiments. We report the WD metrics in this case as follows: (with the same training settings as in Appendix C.2.)\n\n| Distribution | DDPM           | ItDPDM         | LTJ           |\n|--------------|----------------|----------------|----------------|\n| PoissMix     | 7.90 ± 0.64    | **2.65 ± 0.23**     | 2.96 ± 0.29     |\n| ZIP          | 4.89 ± 0.72    | **1.78 ± 0.27**   | 1.92 ± 0.38     |\n| NBinomMix    | 10.83 ± 0.88    | 3.53 ± 0.57     | **3.26 ± 0.48**      |\n| BNB          | 5.07 ± 0.72    | 1.82 ± 0.36     | **1.67 ± 0.41**      |\n| Zipf         | 3.51 ± 0.58    | **0.97 ± 0.21**      | 1.49 ± 0.32     |\n| YS           | 0.87 ± 0.23    |**0.32 ± 0.05**     | 0.36 ± 0.08     |\n\nSecondly, in addition to the above simple multivariate setting, we also consider two dependent 2D count distributions:\n* **Bivariate Poisson:** Let $U\\sim\\mathrm{Pois}(\\lambda_0)$, $V_1\\sim\\mathrm{Pois}(\\lambda_1)$, $V_2\\sim\\mathrm{Pois}(\\lambda_2)$ be independent; define $X_1=U+V_1$, $X_2=U+V_2$. Marginals: $X_1\\sim\\mathrm{Pois}(\\lambda_0+\\lambda_1)$, $X_2\\sim\\mathrm{Pois}(\\lambda_0+\\lambda_2)$. Setting $\\lambda_0 = 3, \\lambda_1 = 2, \\lambda_2= 2$, we draw $U,V_1,V_2$, and thereby $(X_1,X_2)$.\n\n* **Gamma–Poisson:** Let $\\Theta\\sim\\mathrm{Gamma}(k = 2,\\beta = 1)$; given $\\Theta$, we draw $X_i\\mid\\Theta\\sim\\mathrm{Pois}(\\Theta)$ independently for $i=1,2$.\n\nIn these two settings, the evaluated WD scores are: \n| Distribution | DDPM           | ItDPDM         | LTJ           |\n|--------------|----------------|----------------|----------------|\n| Bivariate Poisson  | 3.502| **0.826**     | 1.323     |\n| Gamma–Poisson  | 0.855    | **0.337**   | 0.624     |\n\n**Theoretical extension to multi-variate settings**: Additionally, we would like to call attention to [2] from ISIT’13, which generalizes the I-MMLE identities for vector poisson channels; this would imply that all the results of this paper naturally extend to a multi-variate setting. (along with the updated proofs stated above)\n\nWe will incorporate these extensions in the appendix to emphasize the natural extension and benefits of ItDPDM to multivariate settings. Model/loss are evaluated *pixel-by-pixel* for our proposed method and all the baselines: we will also add this note in the final version. We also refer the reviewer to the pseudocode in Algorithm 1, step 6 for clarification regarding this.\n\n**5. \"Unclarities and typos: I list the following for consideration,...\"**\n\nThank you for pointing this out, we understand that this could have caused confusion. Line 132 states that in Lemma 4, we express the optimal estimator in terms of the marginal, which P_{Z_\\gamma} was supposed to capture. In Eq. (12), we acknowledge that this is a typo and we will replace \\calP with P. \n\nRegarding Algorithm 1, line 4: we thank you for pointing out this typo, we will replace it with the correct form from Figure 4:\n\n$z_\\gamma \\sim \\frac{\\text{Poisson}(\\gamma x)}{1+\\gamma}$\n\nWe hope this clarifies that it is no longer inconsistent with figure 4. For clarification regarding Algorithm 1, line 6: ($q$… is importance sampling) we refer the reviewer to Appendix Sec. A. 2. We will also mention this in the main paper for final version. Details regarding the data transform function can be found in Appendix Sec. B. 2. We also acknowledge the typo in Algo 2, line 5, and we will change this to $\\hat{x}_0$ for the final version. And, finally, we understand that there is no substantial reason to cite [12] here, and will omit it from the pseudocode in the final version. \n\nWe sincerely thank the reviewer again for their thoughtful and critical feedback. We believe the points mentioned above: both theoretical and empirical, meaningfully strengthen the paper and address your concerns. \n\n**References**:\n\n[1] Xianghao Kong, Rob Brekelmans, and Greg Ver Steeg. Information‑Theoretic Diffusion. ICLR 2023.\n\n[2] Liming Wang, Miguel Rodrigues, and Lawrence Carin. *Generalized Bregman Divergence and Gradient of Mutual Information for Vector Poisson Channels*. IEEE International Symposium on Information Theory ISIT 2013."}]

Author rebuttal:

Meta-review: The paper introduces ItDPDM, a Poisson-channel diffusion framework with an information-theoretic Poisson Reconstruction Loss (PRL) that yields a likelihood-consistent training objective (avoiding ELBO gaps) and natively models non-negative discrete data. The work cleanly connects denoising and likelihood via Poisson I-MMLE/Poisson–Stein identities and demonstrates strong results on synthetic count distributions and competitive performance on real data (symbolic music, images), with clear advantages on zero-inflated regimes.

Reasons to accept:
- Conceptual novelty & rigor: A principled discrete diffusion built on the Poisson channel; transparent derivations linking PRL to data likelihood. The rebuttal strengthens theory (completing Lemma 2, clarifying Eq. 12 via Poisson–Stein) and aligns terminology (“likelihood-consistent objective”).
- Empirical substance: Broad synthetic suite covering heavy tails/zero inflation where ItDPDM reliably improves NLL/WD; real-data evidence on music/images; denoising robustness across extreme SNR; ablations (SNR schedules, DDPM+PRL) showing PRL helps beyond the Poisson channel. Added 2D/dependent toy studies and clarified per-pixel image modeling.
- Significance: Provides a native treatment for discrete, count-valued modalities (music, photon-limited imaging, event data, genomics counts), an area underserved by Gaussian diffusions; the method is simple to sample and compatible with standard backbones.

Weaknesses (camera-ready suggestions):
- Report seeds/SDs consistently and release code/checkpoints/configs.
- Keep “exact likelihood” toned down; include the promised finite-sample bias/variance discussion.
- Expand multivariate/correlated evaluations and add clearer qualitative comparisons (with ground truth) and a short note on OOD robustness scope.

The paper offers a well-motivated, theoretically grounded alternative for discrete generative modeling with promising empirical evidence and clear potential impact.
Decision: Accept (poster)