Heavy-Tailed Diffusion Models
Authors: Kushagra Pandey
Jaideep Pathak
Yilun Xu
Stephan Mandt
Michael Pritchard
Arash Vahdat
Morteza Mardani
Venue: ICLR 2025 Poster (2025)
Keywords: Generative models
Diffusion models
Heavy-Tailed Distributions
Primary area: generative models
TL;DR: We present diffusion models for modeling heavy-tailed data distributions.

Abstract: Diffusion models achieve state-of-the-art generation quality across many applications, but their ability to capture rare or extreme events in heavy-tailed distributions remains unclear. In this work, we show that traditional diffusion and flow-matching models with standard Gaussian priors fail to capture heavy-tailed behavior. We address this by repurposing the diffusion framework for heavy-tail estimation using multivariate Student-t distributions. We develop a tailored perturbation kernel and derive the denoising posterior based on the conditional Student-t distribution for the backward process. Inspired by $\gamma$-divergence for heavy-tailed distributions, we derive a training objective for heavy-tailed denoisers. The resulting framework introduces controllable tail generation using only a single scalar hyperparameter, making it easily tunable for diverse real-world distributions. As specific instantiations of our framework, we introduce t-EDM and t-Flow, extensions of existing diffusion and flow models that employ a Student-t prior. Remarkably, our approach is readily compatible with standard Gaussian diffusion models and requires only minimal code changes. Empirically, we show that our t-EDM and t-Flow outperform standard diffusion models in heavy-tail estimation on high-resolution weather datasets in which generating rare and extreme events is crucial.

Published as a conference paper ICLR 2025
HEAVY-TAILED DIFFUSION MODELS
Kushagra Pandey 1,,,, Jaideep Pathak', Yilun Xu', Stephan Mandt?, Michael Pritchard 12, Arash Vahdat") Morteza Mardani NVIDIA', University of California, Irvine (pandeyk1 mandt ) euci.edu, (jpathak, yiluu,, www.t mipann,
avahdat, mmardani) @nvidia. com
ABSTRACT
Diffusion models achieve sante-of-the-aee art generation quality across many applications, but their ability to capture rare or extreme events in heavy-tailed distributions remains unclear. In this work, we show that traditional diffusion and flow-matching models with standard Gaussian priors fail to capture heavy-tailed behavior. We address this by repurposing the diffusion framework for heavy-tail estimation using multivariate Student-t distributions. We develop a tailored perturbation kernel and derive the denoising posterior based on the conditional Student-t distribution for the backward process. Inspired by 7-divergence for heavy-tailed distributions, we derive training objective for heavy-tailed denoisers. The resulting framework introduces controllable tail generation using only single sealar hyperparameter. making it easily tunable for diverse real-world distributions. As specific instantiations our framework, we introduce 1-EDM and r-Flow, extensions existing diffusion and flow models that employ Student-t prior. Remarkably, approach is readily compatible with standard Gaussian diffusion models and requires only minimal code changes. Empirically,
show that 1-ED and f-Flow outperform standard diffusion models in heavy-tail estimation on high-resolution weather datasets in which generating rare and extreme events crucial.
INTRODUCTION
In many real-world applications, such weather forecasting, rare extreme events-like hurricanes heatwaves-Con can have disproportionately larger impacts than more common occurrences. Therefore, building generative models capable of accurately capturing these extreme events critically important (Gründemann et al., 2022). However, learning the distribution of such data from finite samples is particularly challenging, the number of empirically observed tail events is typically small, making accurate estimation difficult.
One promising approach is to use heavy-tailed distributions. which allocate more density to the tails than light-tailed alternatives. In popular generative models like Normalizing Flows (Rezende & Mohamed, 2016) and Variational Autoencoders (VAEs) (Kingma & Welling, 2022), recent works address heavy-tail estimation by learning a mapping from a heavy-tailed prior to the target distribution (Jaini et al., 2020; Kim al., 2024).
While these works advocate for heavy-tailed base distributions, their application to real-world, high- dimensional datasets remains limited, with empirical results focused on small-scale or toy datasets. In contrast, diffusion models (Ho et al., 2020; Song et al., 2020; Lipman et al., 2023) have demonstrated excellent synthesis quality in large-scale applications. However, is unclear whether diffusion models with Gaussian priors effectively model heavy-tailed distributions without significant modifications.
Work during an internship at NVIDIA
Equal Advising

---

Published as conference paper ICLR 2025
Genarated Samples
Ganarated Hists
Original Hists
21
Ground Truth Density (Log-Scale)
Gaussian Diflusion
I-Difusiio 4,20)
I-Diffusion (v 7,20)
I-iifusion 10,
Figure 1: Toy Illustration. Our proposed diffusion model (I-Diffusion) captures heavy-tailed behavior more accurately than standard Gaussian diffusion (see histogram comparisons the top panel, x-axis). The framework allows for controllable tail estimation using a hyperparameter v, which can adjusted each dimension. Lower y values model heavier tails, while higher values approach Gaussian diffusion (Best viewed when zoomed in; see App. C.3 for details). Brighter colors indicate deghitenity regions
this work, we first demonstrate through extensive experiments that traditional diffusion models even with proper normalization, preconditioning, and noise schedule design (see Section 4)- to accurately capture the heavy-tailed behavior in target distributions (see Fig. for toy example). We hypothesize that, in high- dimensional spaces, the Gaussian distribution in standard diffusion models tends concentrate narrow shell, thereby neglecting the tails. To address this, we adopt the multivariate Student-t distribution the base noise distribution, with its degrees of freedom providing controllability over tail estimation. Consequently, we reformulate the denoising diffusion framework using multivariate Student-t distributions by designing a tailored perturbation kernel and deriving the corresponding denoiser, Moreover, we draw inspiration from the "-power Divergences (Eguchi, 2021; Kim et al., 2024) for heavy-tailed distributions to formulate the learning problem for our heavy-tailed denoiser.
spherical
We extend widely adopted diffusion models, such EDM (Karras et al., 2022) and straight-line flows (Lipman et al., 2023; Liu al., 2022), by introducing their Student-t counterparts: (-EDM and i-Flow. We derive the corresponding SDEs and ODEs for modeling heavy-tailed distributions. Through extensive experiments on the HRRR dataset (Dowell et al., 2022), we train both unconditional and conditional versions of these models. The results show that standard EDM struggles capture tails and extreme events, whereas 1-DD performs significantly better in modeling such phenomena. To summarize, we present,
Heavy-tailed Diffusion Models. We repurpose the diffusion model framework for heavy-tail estimation by formulating both the forward and reverse processes using multivariate Student-t distributions. The denoiser is learned by minimizing the 7-power divergence (Kim et al., 2024) between the forward and reverse posteriors.
Continuous Counterparts. We derive continuous formulations for heavy-tailed diffusion models and provide principled approach to constructing ODE and SDE samplers. This enables the instantiation of (-EDM and i-Flow as heavy-tailed alternatives to standard diffusion and flow models.
Empirical Results. Experiments on the HRRR dataset (Dowell et al., 2022), a high-resolution dataset for weather modeling, show that 1-EDM significantly outperforms EDM in capturing tail distributions for both unconditional and conditional tasks.
Theoretical Connections. To theoretically justify the effectiveness of our approach, we present several theoretical connections between our framework and existing work in diffusion models and robust statistical estimators (Futami et al., 2018).
2

---

Published as a conference paper ICLR 2025
2 BACKGROUND
As prerequisites underlying our method, we briefly summarize Gaussian diffusion models (as introduced in (Ho et al., 2020; Sohl-Dickstein et al., 2015)) and multivariate Student-t distributions.
2.1 DIFFUSION MODELS
Diffusion models define a forward process (usually with an affine drift and no learnable parameters) to convert data X p(xo), Xo E IRd to noise. A learnable reverse process is then trained to generate data from noise. the discrete-time setting, the training objective for diffusion models can be specified as,
Eg D p(xx)) + Σ D xo) pe(x log po (Xu|xae)
(I)
L
La
where T denotes the trajectory length while At denotes the time increment between two consecutive time points. DKL. denotes the Kullback-Leibler (KI.) divergence defined as, DRi(g () Sa(x) log 2(x) dx.
p(x)
In the objective in Eq. 1, the trajectory length T is chosen to match the generative prior p(x.) and the forward marginal ((x))x)). The second term in Eqn. 1 proposes to minimize the KL divergence between the
forward posterior ((x;-A1) X,, Xo) and the learnable posterior (x (x) which corresponds to learning the denoiser (i.e. predicting a less noisy state from noise). The forward marginals, posterior, and reverse posterior are modeled using Gaussian distributions, which exhibit an analytical form of the KL divergence. The discrete-time diffusion framework can also be extended to the continuous time setting (Song et al., 2020; 2021; Karras et al., 2022). Recently, Lipman al. (2023); Albergo al. (2023) proposed stochastic interpolants (or flows), which allow flexible transport between two arbitrary distributions.
2.2 STUDENT-T DISTRIBUTIONS
The multivariate Student-t distribution (,( E, 1/) with dimensionality d, location μ,, scale matrix Σ and degrees of freedom defined as,
ta(u, E, Co.d Cua(11+
1 (x x v
(2)
where Co.d is the normalizing factor. Since the multivariate Student-t distribution has polynomially decaying density, can model heavy-tailed distributions. Interestingly, for / 1, the Student-t distribution analogous to the Cauchy distribution. As distribution. A Student-t distributed random variable x can be reparameterized as (Andrews Mallows, 1974), x u + " /F., with z [O, Ia). K (v(v/v where x² denotes the Chi-squared distribution (See Fig. 8 for an illustration of the pdf of the x² distribution).
x0. the Student-t distribution converges to the Gaussian
3 HEAVY-TAILED DIFFUSION MODELS
We now repurpose standard diffusion models using multivariate Student-t distributions. The main idea behind our design the use of heavy-tailed generative priors (Jaini et al., 2020; Kim al., 2024) for learning transport map towards potentially heavy-tailed target distribution. From Eqn. 1 we note three key requirements for training diffusion models: choice of the perturbation kernel ((x,)())) form of the target denoising posterior q(x:-A= Xx, Xo) and the parameterization of the learnable reverse posterior P (x.). Therefore, we begin our discussion in the context of discrete-tima diffusion models and later extend to the continuous regime. This has several advantages in terms of highlighting these three key design choices, which
3

---

Published as conference paper ICLR 2025
might be obscured by the continuous-timo framework of defining a forward and reverse SDE (Karras et al., 2022) while at the same time leading to a simpler construction. Lastly, without loss of generality, we assume a scalar for subsequent analysis due to mathematical convenience.
3.1 NOISING PROCESS DESIGN WITH STUDENT-T DISTRIBUTIONS.
Our construction of the noising process involves three key steps.
1. Firstly, given two consecutive noisy states X, and Xt- we specify a joint distribution ((x,, X
2. Secondly, given q(x,, ((x,,X(-<((x)) {q(x; (x)))(x)))xxx- which can be used as a noising process during training.
we construct the perturbation kernel g(x:|xo)
3. Lastly, from Steps 1 and 2, we construct the forward denoising posterior ((x,_seXx, xo) g(xx:x)-A(lx)) We will later utilize the form of ((x:x(xx)) Xt, Xo) to parameterize the reverse posterior.
((X,(Xo)
It is worth noting that our construction of the noising process bypasses the specification of the forward transition kernel (xx Xx x)-ar). This has the advantage that we can directly specify the form the perturbation kernel parameters A4 and ar as in Karras et al. (2022) unlike Song et al. (2020); Ho et al. (2020). We next highlight the noising process construction in more detail.
Specifiying the joint distribution gx,,, x We parameterize the joint distribution g X;-Ai Xx) as a multivariate Student-t distribution with the following form,
q(x,, x t2a(1s E, v), p Me-a)ao0 Σ
la.
021
oi-be/
where fee ot, (1), 21 (1) are time-dependent scalar design parameters. While the choice of the parameters 14 and or determines the perturbation kernel used during training, the choice of 122()) and 021 (t) can affect the ODE/SDE formulation for the denoising process and will be clarified when discussing sampling.
Constructing the perturbation kernel q(x:) xx): Given the joint distribution q(x,, XI- (Xo) specified multivariate Student-t distribution, it follows that the perturbation kernel distribution g(xe xx) is also a Student-t distribution (Ding, 2016) parameterized as, (xx, ((xx)x)) = tt(pe o Jd, 1/) (proof in App. A. We choose the scalar coefficients μe and a such that the perturbation kernel at time T converges to (a(0, Ia v) to
standard Student-t distribution. Later, we will set our generative prior p(xr) = g(xr|
instantiate sample generation. We discuss practical choices of μ4 and a2 in Section 3.5.
Estimating the reference denoising posterior: Given the joint distribution g(x,, X;- At (xo) and the pertur- bation kernel ((xx,Xx)) the denoising posterior be specified as (see Ding (2016)),
F di
q(x, ((xx,AilX1,, Xn) = La(p
o. Id, v + d).
(4)
+ d
0 of P.. A1
2 (1 oं o o7
(x, M,X0).
(5)
of
where d1 J Ix pexol 2 Next, we formulate the training objective for heavy-tailed diffusions.
3.2 PARAMETERIZATION OF THE REVERSE POSTERIOR
Following Eqn. 4, we parameterize the reverse (or the denoising) posterior distribution as:
Po (x( ta(Mo t), o² I d),
(6)
4

---

Published as a conference paper ICLR 2025
where the denoiser mean (x t) is further parameterized as follows:
( t) = 22 of
+ M4-2i -
22 (t) of Me (o),
(7)
We discuss alternative posterior parameterizations in App. A.2. It worth noting that while the noising process defined in Eq. 3 is non-mankee non-markovian, our parameterization of the posterior is still Markovian. However, similar Song et al. (2022a), this choice works well empirically (see Section 4). Moreover, when parameterizing the reverse posterior scale, we drop the data-dependent coefficient (v + d1)/(v + d). This choice is primarily inspired by simplicity in deriving preconditioners (Sec. 3.5) and developing continuous-time sampling methods (Sec. 3.4) for heavy-tailed diffusions, resulting in models that require minimal implementation overhead over standard diffusion models during training and sampling (see Fig. 2). However, heteroskedastic modeling of the denoiser is possible in our framework and could be an interesting direction for future work. Next, we reformulate the training objective Eqn. for heavy-tailes diffusions.
3.3 TRAINING WITH POWER DIVERGENCES
The optimization objective in Eqn. primarily minimizes the KL-Divergence between given pair of distributions. However, since we parameterize the distributions Eqn. using multivariate Student-t distributions, using the KL-Divergence might not be suitable choice of divergence. This is because computing the KL divergence for Student-t distributions does not exhibit a closed-form expression. An alternative is the 7-Power Divergence (Eguchi, 2021; Kim et al., 2024) defined as,
D (1) -[c,(o.)) H.) 7E (-1,)) (0,
7
14,(1) = - 1Pl1, = -- p(x)
C, - - g(x)
p(x) |pl |pl
dx.
where, like Kim et al. (2024), set w+d for the remainder of our discussion. Moreover, H, and C., represent the 7-power entropy and cross-entropy. respectively. Interestingly, the 7-Power divergence between two multivariate Student-t distributions, 9u = tu( Eg. 1/) and Pv = ta(11,, E1, can be tractably computed in closed form and is defined as (see Kim et al. (2024) for a proof),
D.laulled] I..ii (1 A
Eol 214 (1 +
d
w-2
(8)
E1 ममन.
ا [ΣT -o) + (H a1) ' (mO " ((po-mi)) mi))
Therefore, analogous to Eqn. 1, we minimize the following optimization objective,
Eg (a(((xx(x)) p(x))) + Σ (((((xx-aixxxxx | (o(xx- log po (xo|xi)
(9)
1-1
Here, we note couple of caveats. Firstly, while replacing the KL-Divergence with the 7-Power Divergence in the objective in Eqn. might appear to be due to computational convenience, the y-power divergence has several connections with robust estimators (Futami et al., 2018) in statistics and provides tunable parameter 7 which can be used to control the model density assigned at the tail (see Section 5). Secondly, while the objective in Eqn. 1 is valid ELBO, the objective in Eq. 9 is not. However, the following result provides connection between the two objectives (see proof in App. A.3),
Proposition 1. For arbitrary distributions q and p. in the limit of 0, D2 p) converges to DKL(q || p). Consequently, for a finite- dimite-dimensoonal dataset with X E IRd and y v4d under the limit of the objective in Eqn. 9 converges to the objective in Eqn. I.
5

---

Published as conference paper ICLR 2025
Component
Gaussian Diffusion
(Ours) t-Diffusion
Perturbation Kernel (g(x,) xx))
M(uxo, Ia (a,, La
ta(pxxo Ia talpe, u-di oi Falaa +d) atd fa(po(x,. tf d) D. (0 Ia)
Forward Posterior (g(Xe- Ar Xt, Xu))
Reverse Posterior (Po (X==ar(X))
(No((x, T)) DKL(q l
Divergence Measure
Generative Prior (p(xT))
(0,
Table 1: Comparison between different modeling components for constructing Gaussian vs Heavy-Tailed diffusion models. Under the limit of v - on, our proposed I-Diffusion framework converges to Gaussian diffusion models.
Therefore, under the limit - O, the standard diffusion model framework becomes a special case of our proposed framework. Moreover, for y 22 d), this also explains the tail estimation moving towards Gaussian diffusion for increasing (See Fig. 1 for illustration). Lastly, the divergence-basee interpretation of the ELBO loss in Eq. has also been considered in prior work in generative models (Xiao al.; Kim al., 2024) and is also commonplace in M-estimators (Futami et al., 2018) used in robust statisties. Therefore, our choice of the training objective in Eq. quite principled.
Simplifying the Training Objective. Plugging the form of the forward posterior q(x,- At Xr, xo) in Eqn. 4, the reverse posterior pol (t-Aa xx) the optimization objective Eqn. 9, we obtain the following simplified training loss (proof in App. A.4).
((0)
E. Berp(t) E. N. Ex²(e) De or
VK 00)
Xo
(10)
Intuitively, the form of our training objective is similar to existing diffusion models (Ho et al., 2020; Karras et al., 2022). However, the only difference lies in sampling the noisy state x, from a Student-t distribution based perturbation kernel instead of Gaussian distribution. Next, discuss sampling from our proposed framework under discrete and continuous time settings.
3.4 SAMPLING
Discrete-time Sampling. For discrete-ting time settings, we can simply perform ancestral sampling from the learned reverse posterior distribution Po (X,-a)/X,) Therefore, following simple re- parameterrization, an ancestral sampling update can be specified as,
X At = (o(x,,t) t) + 0,21/ / z ~~ N(O, K~ d)/(v + d),
Continuous-time Sampling. Due to recent advances in accelerating sampling in continuous-time diffusion processes (Pandey et al., 2024a; Zhang & Chen, 2023; Lu et al., 2022; Song al., 2022a; Xu al., 2023a), we reformulate discrete-time dynamics in heavy-tailed diffusions to the continuous time regime. More specifically, we present family of continuous-time processes in the following result (Proof in App. A.5).
Proposition 2. The posterior parameterization po (x(x ta t), La
d) induces the
following comtinuou--lime dynamics,
dx; = 1r
ft Xt
01)
+ 14 1t
(x. 1i DoDo dt ( ((()g(or,
(1I)
where f R R+ and R x R - are user-specified functions, 3r E Ie+ is a scaling f( At,, where jus 5t denote the first-
coefficient such that diatt order time-derivatives of the perturbation kernel parameters μt and a, respectively and the differential ds, t,0, dt, d).
-
(ož a.))
6

---

Published as a conference paper ICLR 2025
Algorithm I: Training (t-EDM)
Algorithm 2: Sampling (t-EDM) (14, 0z
repeat
1: sample
2: Xa p(Xo)
2: for (O...., - do
3: o ~ LogNormal (Tria Tmems Wan)
3: d, - (x, (x((xttt)))
4:
n, La(0, Ia)
4: Xi+1 - X; + (4-1 - lidd
5: ==ov VovvvllV
5:
if (.+1 then
6:
Do(x,0)
6:
d: - (x141 14(x441:444))/4
( + (C Croise (0))
7: Xi+1 x4 + (l++1 - (:) dd)
7: Ma) = Coui (o)
8: end if
8: Take gradient descent step
Vol ((o)
xoll*
9:
end for
10: return XN
10:
until converged
Training and Sampling algorithms for 1-EDM (v > 2). Our proposed method requires minimal code updates (indicated with blue) over traditional Gaussian diffusion models and converges to the latter y/ o0.
Figure 2:
Based on the result in Proposition 2, is possible to construct determmiiistic/soccasti samplers for heavy- tailed diffusions. It worth noting that the SDE in Eqn. 11 implies adding Student-t stochastic noise during inference (Bollerslev, 1987). This intuitive since the denoising distribution pe (x-- (xx)) is modeled as a Student-t distribution. Next, we provide specific instantiations of the generic sampler Eq. 11.
Sampler Instantiations. We instantiate the continuous-tin time SDE in Eqn. 11 by setting g( ot) = 0 and 012(t) o.. Consequently, f(āz, σz) = de. In this case, the SDE in Eqn. 11 reduces to an ODE,
dxt
jig #4 ae
(x: (x),(x),
(12)
dt
Summary, Overall, we compare between Gaussian diffusion and heavy-tailed diffusion models in Table 1.
3.5 SPECIFIC INSTANTIATIONS: T-EDM
Karras al. (2022) highlight several design choices during training and sampling, which significantly improve sample quality while reducing sampling budget for image datasets like CIFAR-10 10 (Krizhevsky, 2009) and ImageNet (Deng et al., 2009). With similar motivation, reformulate the perturbation kernel ((x:(x)) = (a(s(1)xo: s(1)aa (1)² (I)aIaa v) and denote the resulting diffusion model as 1-EDM.
Training. During training. set the perturbation kernel, q(x:/xo), parameters s(t) 1, a(t)
LogNormal( (I mmaan Pas). We parameterize the denoiser De(x,, oτ) similar to Karras et al. (2022) with the difference that coefficients like cout additionally depend on v. We include full derivations in Appendix A.6. Consequently, our denoising loss can be specified as follows:
(()) E Eolnn (00.0 [Ma, v)] Do(x + n, 0) X 10
(13)
where M(a, v) is weighting function set to X(a, v) 1/Cou(a,v))
Sampling. Interestingly, it can be shown that the ODE in Eqn. 12 is equivalent to the deterministic dynamics presented in Karras al. (2022) (See Appendix A.7 for proof). Consequently, we choose s(t) 1 and (() during sampling. further simplifying the dynamics in Eqn. 12 to dx./dl (x, Do xx, 1)/L We adopt the timestep discretization schedule and the choice of the numerical ODE solver (Heun's 's method (Ascher & Petzold, 1998)) directly from EDM. Figure 2 illustrates the ease of transitioning from Gaussian diffusion framework (EDM) to t-EDM. Under standard settings, transitioning to t-EDM requires as few as two lines of code change, making our method readily compatible with existing diffusion implementations. Similarly, we can also construct heavy-tailed flows (Albergo et al., 2023; Lipman et al., 2023). We denote the resulting model i-Flow (see App. B for details).
7

---

Published as a conference paper at ICLR 2025
EDM (+INC)
EDM (+PCP)
t-EDM
EDM (+RVC) EDM Vest Set
EDM [+PCP) EDM
I-SOM REW Test Set
Rest Set
12-1
12
Densiry
12.0
127
32-7
Figure 3: Sample 1-d histogram comparison between EDM and t-EDM on the test set for the Vertically Integrated Liquid (VIL) channel. t-EDM captures hcavy-tailed behavior more accurately than other baselines. INC: Inverse CDF Normalization, PCP: Cer-Chnneel Preconditioning
4 EXPERIMENTS
Next, we demonstrate the effectiveness of the proposed heavy-tailed diffusion models on real-world weather data for both unconditional and conditional generation tasks. We include full experimental details in App. C.
Datasets. We adopt the Righ-RResolution Rapid Refresh (HRRR) (Dowell et al., 2022) dataset, which is an operational archive of the US km-scale forecasting model. Among multiple dynamical variables in the dataset that exhibit heavy-tailed behavior, based on their dynamic range. we only consider the Vertically Integrated Liquid (VIL) and Vertical Wind Velocity at level 20 (denoted as w20) channels (see App. C.I for more details). It is worth noting that the VIL and w20 channels have heavier right and left tails, respectively (See Fig. 6).
Tasks and Metrics. We consider both unconditional and conditional generative tasks relevant to weather and climate science. For unconditional modeling, we aim to generate the VIL and w20 physical variables in the HRRR dataset. For conditional modeling, we aim to generatively predict the hourly evolution of the target variable for the next lead-time (T + 1) hour-ahead evolution of VII. and w20 based on information only at the current state time see more details in the appendix and see Pathak et al. (2024) for discussion of why hour-ahead, km-scale atmospheric prediction is a stochastic physical task appropriate for conditional generative models. To quantify the empirical performance of unconditional modeling, we rely comparing 1-d statistics of generated and train/test set samples. More specifically, for quantitative analysis, we report the Kurtosis Ratio (KR), the Skewness Ratio (SR), and the Solmooorov-Smmirnv (KS)-2 sample statistic (at the tails) between the generated and train/test set samples. For qualitative analysis, we compare 1-d histograms between generated and train/test samples. For the conditional task, we adopt standard probabilistic prediction score metrics such as the Continuous Ranked Probability Score (CRPS), the Root-Mean Squared Error (RMSE), and the skill-spread ratio (SSR); see, e.g., Mardani al. (2024); Srivastava et al. (2023). A more detailed explanation of our evaluation protocol is provided in App. C.
Methods and Baselines. In addition to standard diffusion (EDM (Karras et al., 2022)) and flow models (Albergo et al., 2023) based on Gaussian priors, we introduce two additional baselines that are variants of EDM. To account for the high dynamic-range often exhibited by heavy- tailed distributions, we include Inverse CDF Normalization (INC) as an alternative data preprocessing step to z-score normalization. Using the former reduces dynamic range significantly and can make the data distribution closer to Gaussian. We denote this preprocessing scheme combined with standard EDM training as EDM + INC. Alternatively, instead modulate the noise levels used during EDM training as a function of the dynamic range of the input channel while keeping the data preprocessing unchanged. The main intuition is use more heave-tailed noise for large values. We denote this modulating scheme as Per-Channa Preconditioning (PCP) and denote the resulting baseline as EDM PCP. We elaborate these baselines in more detail in App. C.1.2
could
8

---

Published as a conference paper ICLR 2025
VIL (Train)
VIL (Test)
W20 (Train)
w20 (Test)
Method
KR L
SR 4
KS
KR J
SR 1
KS
V
KR 1
SR J
KSS
KR L
SR J
KS
EDM
210.11
10.79
0.997
45.35
5.23
0.991
50
12.59
0.89
0.991
5.01
0.38
0.978
Baselines
+ICC
11.33
2.29
0.987
1.70
0.74
0.95
50
1,80
0.909
0.23
+PCP
2.12
0.72
0.800
0.31
0.09
0.522
00
2.17
0.70
0.838
0.40
0.24
0.648
-ED
1,06
0.43
0,431
0.54
0.23
0.114
3
2.44
0.65
0.683
0.52
0.21
0.286
Ours
L EDM
29.66
4.07
0.955
5.73
1.68
0.888
5
8.55
1.77
0.895
3.22
1.03
0.774
FDM
24.35
4.14
0.959
4.57
0.908
7.03
2.55
0.622
Table 2: t-EDM outperforms standard diffusion models for unconditional generation on the HRRR dataset. For all metrics, lower better. Values in bold indicate the best results in column.
VII. (Train)
(II.(Tes))
w20 (Train)
w20 (Test)
Method
KR 1
SRL
KSL
KR
SR 1
KSL
KR
SR
KS
KR
SR
KS
Gaussian Flow
Baselines
8
0.46
0.09
0.897
0.67
0.52
0.704
8
2,03
0.36
0.294
0.34
0.01
0.384
t-Flow
3
1.39
0.37
0.711
0.47
0.27
0.275
1,08
0.333
0.07
0.42
0.512
Ours
FFFlow
5
3.30
0.75
0.857
0.05
0.07
0.633
7
3.24
0.36
0.259
0.87
0.01
0.300
t. FFow
3.36
0.84
0.844
0.04
0.02
0.603
0.478
0.034
Table 3: t-Flow outperforms standard Gaussian flows for unconditional generation on the HRRR dataset. For all metrics, lower better. Values in bold indicate the best results in column.
4.1 UNCONDITIONAL GENERATION
We assess the effectiveness of different methods on unconditional modeling for the VIL and w20 channels in the HRRR dataset. Fig. 3 qualitatively compares 1-d histograms of sample intensities between different methods for the VIL channel. We make the following key observations. Firstly, though EDM (with additional tricks like noise conditioning) can improve tail coverage, t-EDM covers a broader range of extreme values in the test set. Secondly, in addition better dynamic range coverage, t-EDM qualitatively performs much better in capturing the density assigned to intermediate intensity levels under the model. We note similar observations from our quantitative results in Table 2, where 1-EDM outperforms other baselines on the KS metric, implying our model exhibits better tail estimation over competing baselines for both the VIL and w20 channels. More importantly, unlike traditional Gaussian diffusion models like EDM, t-EDM enables controllable tail estimation by varying v/ which could be useful when modeling a combination of channels with diverse statistical properties. Lastly, we note that while EDM combined with improved preprocessing (INC) preconditioning (PCP) outperforms t-EDM in some cases, these techniques can also be readily integrated with t-EDM. Therefore, the main point of comparison in Table 2 should be primarily between t-EDM and the standard EDM baseline (row 1). We present similar quantitative results for t-Flow in Table 3 where the advantages of t-Flow over Gaussian Flows for heavy-tailed modeling are quite apparent. We also present additional results for unconditional modeling App. C.1.7.
4.2 CONDITIONAL GENERATION
Next, we consider the task of conditional modeling, where we aim to predict the hourly evolution of a target variable for the next lead time (r 8T) based on the current state time 7 with 8r 1hr Table 4 illustrates the performance of EDM and t-EDM on this task for the VIL and w20 channels. We make the following key observations. Firstly, for both channels, t-EDM exhibits better CRPS and SSR scores, implying better probabilistic forecast skills and ensemble than EDM. Moreover, while t-EDM exhibits under-dispersion for VIL, while is well-calibrated for w20, with its SSR close to an ideal score of 1. On the contrary, the bascline EDM model exhibits under-dispersion for both channels, thus implying overconfident predictions. Secondly, in addition to better calibration, t-EDM is better at tail estimation (as measured by the KS statistic) for the underlying conditional distribution. Lastly, we notice that different values of the parameter v optimal for
9

---

Published as a conference paper ICLR 2025
w20 (Test)
Method
CRPS J
RMSE L
S-R 1)
KS
CRPS
RMSE 1
SSR (-+ 1)
KS J
Baselines
EDM
1,696
4.473
0.203
0.715
20
0.304
0.664
0.865
0.345
LEDM
1.649
4.526
0.255
0.419
3
0.295
0.734
1,045
0.111
LEDM
1,6099
4.361
0.305
0.665
0.301
0.674
0
0.323
Table t-EDM outperforms EDM for conditional next frame prediction for the HRRR dataset. Values bold indicate the hest results each column. note that VII. has higher dynamic range over w20, and thus, the gains for VII. are more apparent (sec hist plots in Fig. 6). (- 1) indicates values near are better.
different channels, which suggests a more data-driven approach learning the optimal directly. We present additional results for conditional modeling in App. C.2.
5 DISCUSSION AND THEORETICAL INSIGHTS
To conclude, we propose a framework for constructing heavy-tailed diffusion models and demonstrate their effectiveness over traditional diffusion models on unconditional and conditional generation tasks for a high- resolution weather dataset. Here, we highlight some theoretical connections that help gain insights into the effectiveness of our proposed framework while establishing connections with prior work.
Exploring distribution tails during sampling. The ODE in Eq. 12 can re- re-formulated as,
dx: jixx, + o. di
d' + d 1e
iu
Vx logr(xe;(), 1), (14)
14t
oe
where d' = (1/a7)|lx, nr W. (x1, ox) By formulating the gain some intuition our model in modeling heavy-tailed
ODE in terms of the score function, into the effectiveness distributions. Figure 4 illustrates the variation of the mean and variance of the multiplier (v + d/)((v + d) along the diffusion trajectory across IM samples generated from
Office drnndtp
110
Figure 4: Variation of the mean and standard deviation of the ratio (v + di)/(v d) with y across diffusion sampling trajectory for the toy dataset. As decreases, the mean ratio its standard deviation increase, leading to large score multiplier weights.
toy models. Interestingly, as the value of v decreases, the mean and variance of this multiplier increase significantly, which leads large score multiplier weights. We hypothesize that this behavior allows our proposed model to explore more diverse regions during sampling (more details in App. A.10).
Enabling efficient tail coverage during training. The optimization objective in Eq. 9 has several connections with robust statistical estimators. More specifically, it can be shown that (proof in App. A.11),
Po(x) [po 1+>
)
V DoDaall - - ((x)
Vo po()) - EEp Vo Voloooo(x)]] dx,
where q and py denote the forward (g(x;-Arl) Xt, xo)) and reverse diffusion posteriors (pel (xo(x;-Ax)) ax)=)) respec- tively. Intuitively, the coefficient weighs the likelihood gradient, Vo log (x)), and can be set accordingly to ignore or consider outliers when modeling the data distribution. Specifically, when y > 1. the model would leamn to ignore outliers (Futami et al., 2018; Fujisawa & Eguchi, 2008; Basu et al., 1998) since data points on the tails would be assigned low likelihood. On the contrary, a negative value of 7 (as the case in this work since set ? 2/(1v / + d)), the model can assign more weights to capture these extreme values.
We discuss some other connections to prior work in heavy-tailed generative modeling and more recent work in diffusion models in App. F.1 and some limitations of our approach in App. F.2.
10

---

Published as conference paper ICLR 2025
ACKNOWLEDGEMENT
Stephan Mandt acknowledges support from the National Science Foundation (NSF) under an NSF CAREER Award UIS-2047418 and IIS-2007719, the NSF LEAP Center, by the Department of Energy under grant DE- SC0022331, the IARPA WRIVA program, the Hasso Plattner Research Center UCI, the Chan Zuckerberg Initiative, and gifts from Qualcomm and Disney.
REPRODUCIBILITY STATEMENT
We include proofs for all theoretical results introduced in the main text in Appendix A. We describe our complete experimental setup (including data processing steps, model specification for training and inference, description of evaluation metrics, and extended experimental results) in Appendix C.
ETHICS STATEMENT
We develop generative framework for modeling heavy-tailed distributions and demonstrate its effectiveness for scientific applications. In this context, we do not think model poses risk of misinformation other ethical biases associated with large-scale image synthesis models. However, we would like to point out that similar other generative models, our model sometimes hallucinate predictions for certain channels, which could impact downstream applications like weather forecasting.
REFERENCES
Michael S Albergo, Nicholas M Boffi, and Eric Vanden-Ein Dijeen-n,jddn.... Stochastic interpolants: A unifying framework for flows and diffusions. arXiv preprint arXiv:2303.0877 2023.
D. F. Andrews and C. L. Mallows. Scale mixtures of normal distributions. Journal of the Royal Statistical Society. Series B (Methodological), 36(1)) 36(1):99-110) 102, 1974. ISSN 00359246. URL http: www. jstor. org/ wwtaabeee994777
Uri M. Ascher and Linda R. Petzold. Computer Methods for Ordinary Differential Equations and Differemtial-Algebree Equations. Society for Industrial and Applied Mathematics, Philadelphia, PA, 1998. doi: 97811117781111 URL httttt:s:///ppe 9781611971392
siam. org. btlgg/ddi/ane wttp:////////.....oooo
Ayanendranath Basu, Ian R Harris, Nils I. Hjort, and MC Jones. Robust and efficient estimation by minimising a density power divergence. Biometrika, 85(3):549 -599, 1998.
Tim Bollerslev. A conditionally heteroskedastic time series model for speculative prices and rates of return. The Review of Economics and Statistics, 69(3):542-547, 1987. ISSN 00346535, 15309142. URL http:/ www. jstor. wwtp//taaeeeeee////o/o 9225546
Steve Brooks, Andrew Gelman, Galin Jones, and Xiao-Li Meng. Handbook of Markov Chain Monte Carlo. Chapman and Hall/CRC, May 2011. ISBN 9780429138508. doi: 10.1201/b10905. URL
http:/ /ttp://dxx dxx doi. 10.
Tianfeng Chai and Roland R Draxler. Root mean square error (rmse) or mean absolute error (mae)?-are) -argumenss against avoiding in the literature. Geoscientific Model Development, (3)):127-12551 2014.
11

---

Published as a conference paper ICLR 2025
Hyungjin Chung, Jeongsol Kim, Michael Thompson Mccann, Mare Louis Klasky, and Jong Chul Ye. Diffusion posterior sampling for general noisy inverse problems. In The Eleventh International Conference on Learning Representations, 2022.
Jacob Deasy, Nikola Simidjievski, and Pietro Liò. Heavy-tailed denoising score matching. arXiv preprint arXiv:21 12.09788, 2021.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 JEEE Conference Computer Vision and Pattern Recognition, pp. 248-255, 2009. doi: 10.1 10.11000V 55206888.
Peng Ding. On the conditional distribution of the multivariate t distribution, 2016. URL https://///axx org/ hat//ab///t 200s/1604 6044 00561.
Tim Dockhorn, Arash Vahdat, and Karsten Kreis. Score-based generative modeling with critically-damped langevin diffusion, 2022. URL https: /ttp/aawivvv...///a///../.a.aa wwwpxiivvoog/aaooooo aba////s/22/ /2122. 07068.
Alexey Dosovitskiy and Thomas Brox. Generating images with perceptual similarity metrics based deep networks, 2016. URL https: arxiv. www.wawwww.aa..aa.a.......a..ett abs/ 02644.
David C. Dowell, Curtis R. Alexander, Eric P. James, Stephen Weygandt, Stanley G. Benjamin, Geoffrey S. Manikin, Benjamin T. Blake, John M. Brown, Joseph B. Olson, Ming Hu, Tatiana G. Smirnova, Terra Ladwig. Jaymes S. Kenyon, Ravan Ahmadov, David D. Turner, Jeffrey D. Duda, and Trevor I. Alcott. The high-resolution rapid refresh (hrrr): An hourly updating convection-allowing forecast model. part i: Motivation and system description. Weather and Forecasting, 37(8):1371 - 1395, 2022. doi: 10.1175/ WAF-D-21-01511 URL https:/ //oorraallsss ametsac.....................a... org/ wtwggvvviie wwwe//iwww//e//eeeeeeeeeee www.unass/wweo//////ooeoe/eaoeee 21 xml.
Shinto Eguchi. Chapter 2 pythagoras theorem information geometry and applications to generalized linear models. In Angelo Plastino, Arni S.R. Srinivasa Rao, and C.R. Rao (eds.), Information Geometry, of Handbook of Statistics, pp. 42. Elsevier, 2021. doi: https://bb.oo///0..0111bss.sssss
Geomert volume 45 URL https: www. www.sciiencccee rect www.scccenaeeeocccoeeecccec....e swwsnceeeeccee///e//ee//eeeecccs wttaaaeeepppisaeaaeeeeaeeeeeeeee http:///rr...eee////////7/121110
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, Dustin Podell, Tim Dockhorn, Zion English, Kyle Lacey, Alex Goodwin, Yannik Marek, and Robin Rombach. Scaling rectified flow transformers for high-resolution aawaiiiiiii..io org abs 2403. 03206.
image synthesis, 2024. URL https: http::::/axx.v..////pp
Hironori Fujisawa and Shinto Eguchi. Robust parameter estimation with small bias against heavy contami- nation. Journal of Multivariate Analysis, 990)):00332088 2008. ISSN 0047-259X. doi: https://doi.or// 10. 10.1016/j jmvvaa/jjmva..... jmv.. 20..20008.00000 URL https::////. www. scwenced rect, com/ www.enceeeeateiiccoooooe
725/000047259x0000445
Futoshi Futami, Issei Sato, and Masashi Sugiyama. Variational inference based on robust divergences, 2018.
URL https:tt///.m /ttp///aaavvv/aa///a//eeeea.weva org/ abs 1710. 06595.
Gaby Joanne Gründemann, Nick de Giesen, Lukas Brunner, and Ruud van der Ent. Rarest rainfall events will see the greatest relative increase in magnitude under future climate change. Communications Earth &; Environment, 3(1), October 2022. ISSN 2662-4435 doi: 10.1038/442247-22-0055533 8.
URL
http: / dx. doi. org/ 10. 1038/ 103.1/133342444 443247-0222 00077702220055-50
H. Guo, J.-C. Golaz, L. J. Donner, B. Wyman, M. Zhao, and P. Ginoux. Clubb as a unified cloud pa- rameterization: Opportunities and challenges. Geophysical Research Letters, 42(1 1):4540-4544 2015. doi: http://dciiagg///..0022221.66160 htt./o///bas/1..100/2222200
URL //t.attsss//gguuuuu..om on wtw.eeerrrrrr.eeeee...........e.
12

---

Published as a conference paper ICLR 2025
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler. and Sepp Hochreiter, Gans trained by a two time-scale update rule converge to a local nash equilibrium, 2018. URL https: //arxiv. org/ abs/ 7006. 08500.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840 6851, 2020.
M.F. Hutchinson. A stochastic estimator of the trace of the influence matrix for laplacian smoothing splines. Communications in Statistics Simulation and Computation, 19())433-450, 1990. doi: 10.10 036109190088 18666. URL https: / /oiiiiiooo//o 10t.1000000000011111110010000100
Priyank Jaini, Ivan Kobyzev, Yaoliang Yu, and Marcus Brubaker. Tails lipschitz triangular flows, 2020.
URL httph:tt/////m /awxiv..vvvvv////// orooor///aoooo.. http//bs/1900 04481.
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models, 2022. URL htttpps//arxiv.orggg/.
220./222....00 00364.
Juno Kim, Jachyuk Kwon, Mincheol Cho, Hyunjong Lee, and Joong-Ho Won. 3/33-variation autoencoder: Learning heavy-tailed data with student's t and power divergence. In The Twelfth International Conference on Learning Representations, 2024. URL https: : / openreview net/ woo.mooddd
Diederik P Kingma and Max Welling. Auto-encoding variational bayes, 2022. URL htttt:///aaiii org/abs 1112.111222 6114.
Alex Krizhevsky. Leanning multiple layers of features from tiny images. pp. 32-33, 2009. URL https: h/www. www.wwws.troot.... www.oorrttto.uuueeeeoooooe Lelezzlleaann learning features 2009-TR, pdf.
Mike hiaszkkiewicz,,haaaae Lederer, and Asja Fischer. Marginal tail-adaptive normalizing flows, 2022. URL https: /ttp////a/iii/ http://wwwwww........o.o// 2206. 10311.
Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maximilian Nickel, and Matthew Le. Flow matching
for generative modeling. In International Conference on Learning Representations, 2023. URL https: //ppnneviewww.... net/ forum? Porrumiid-PPvVMDCTTTTT
Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow, 2022. URL https: /aaravivvvv/v/////////aa////aivv org/ abs 2209. 03003.
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. 2022. URL https: arxiv. org/ abs / 2206.2206600 00927.
Morteza Mardani, Jiaming Song, Jan Kautz, and Arash Vahdat. variational perspective on solving inverse problems with diffusion models. In The Twelfth International Conference on Learning Representations, 2023.
Morteza Mardani, Noah Brenowitz, Yair Cohen, Jaideep Pathak, Chieh Yu Chen, Cheng-Chin Liu, Arash Vahdat, Mohammad Amin Nabian, Tao Ge, Akshay Subramaniam, Karthik Kashinath, Jan Kautz, and Mike Pritchard. Residual corrective diffusion modeling for km-scale atmospheric downscaling, 2024. URL https: http///avvvv//e org. abs 2309. 15214.
Frank J. Massey. The kolmogonov-smimov test for goodness of fit. Journal of the American Statistical Association, 46(253) 1951. ISSN 01621459, 1537274X. URL http: /ww.. jsttrrstooooooo...ooo..o. .rg stable/2280095.
13

---

Published as conference paper ICLR 2025
Radford M. Neal. Slice sampling. The Annals of Statistics, 31(3):705 767, 2003. doi: 10.12///aos/ 1056562461. URL https: / doi org/ 1214 /at.//11555562666
Kushagra Pandey and Stephan Mandt. A complete recipe for diffusion generative models, 2023. URL fetps://arxive http://www.rr/b///33333333 01748.
Kushagra Pandey, Avideep Mukherjee, Piyush Rai, and Abhishek Kumar. Diffusevae: Efficient, controllable and high-fidelity generation from low-dimensional latents, 2022. 2201. 00306.
/abs//
Kushagra Pandey, Maja Rudolph, and Stephan Mandt. Efficient integrators for diffusion generative mod- els. In The Twelfth International Conference Learning Representations, 2024a. URL https: /
//openreview.ro net forum? dd= fox05Gf.
Kushagra Pandey, Ruihan Yang, and Stephan Mandt. Fast samplers for inverse problems in iterative refinement models, 2024b. URL https://p///......c..tt www.aaavvvv.o.....aaaaaaa.aaaaaa www. /ttgg/aa////e/ abs 2405 17673.
Jaideep Pathak, Yair Cohen, Piyush Garg, Peter Harrington, Noah Brenowitz, Dale Durran, Morteza Mardani, Arash Vahdat, Shaoming Xu, Karthik Kashinath, and Michael Pritchard. Kilometer-scale convection allowing model emulation using generative diffusion modeling, 2024. URL /tttts://arxiv. org/
abs/2 abs/2408 10958.
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis, 2023. URL
org/ abs /200. 01 152.
Danilo Jimenez Rezende and Shakir Mohamed. Variational inference with normalizing flows, 2016. URL https: / org/ abs 1100. 05770.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. High-resolution image synthesis with latent diffusion models, 2022. URL https: :/axxii wwwsxivoorgg org/ httgg//b///// 1ttp//21...001111
Francisco R Ruiz, Titsias RC AUEB, David Blei, al. The generalized reparameteriization gradient. Advances in neural information processing systems, 29, 2016.
Mehdi SM Sajjadi, Olivier Bachem, Mario Lucic, Olivier Bousquet, and Sylvain Gelly. Assessing generative models via precision and recall. Advances in neural information processing systems, 31, 2018.
Dario Shariatian, Umut Simsekli, and Alain Durmus. Denoising lévy probabilistic models, 2024. URL
https: //taaiv///vvvvee/// orgl abs 2407 18609.
Raghav Singhal, Mark Goldstein, and Rajesh Ranganath. Where diffuse, how diffuse, and how get /2302.
back: Automated learning for multivariate diffusions, 2023. URL 07261.
John Skilling. The Eigenvalues of Mega-dimensional Matrices, pp. 466- Netherlans,,,,, Dordrecht, 1989. ISBN 978-94-015-7860-5 doi: 10.10007978-994-055-78888 48. URL httpsthtt////: 1007 9978 94 78600---------8 8_48.
Netherlands,
Jascha Sohl-Dickstein Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamies. In International Conference on Machine Learning, pp. 2256-2265. PMLR, 2015.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models, 2022a. URL http:///wwww..oo////////////// 02502.
14

---

Published as conference paper ICLR 2025
Jiaming Song, Arash Vahdat, Morteza Mardani, and Jan Kautz. Pscudoinverse-guidee diffusion models for inverse problems. In International Conference on Learning Representations, 2022b.
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2020.
Yang Song, Conor Durkan, lain Murray, and Stefano Ermon, Maximum likelihood training of score-based diffusion models, 2021. URL wtwpt:tt///t/...... http:///axvvvoooo http://rrrgaa/aaaaa//a//oa/aae/ abw//basl.00 09258.
Prakhar Srivastava, Ruihan Yang, Gavin Kerrigan, Gideon Dresdner, Jeremy McGibbon, Christopher Brether- ton, and Stephan Mandt. Precipitation downscaling with spatiotemporal video diffusion. arXiv preprint arXiv:2312.06071, 2023.
Pascal Vincent. A connection between score matching and denoising autoencoders. Neural Computation, 23 (7):1661-1674, 2011. doi: 10. 10.1162/NECO_a a 00142..
Daniel Wilks. Statistical methods in the atmospheric sciences, volume 100. Academic press, 2011.
Zhisheng Xiao, Karsten Kreis, and Arash Vahdat. Tackling the generative learning trilemma with denoising diffusion gans. In International Conference on Learning Representations.
Yilun Xu, Mingyang Deng, Xiang Cheng, Yonglong Tian, Ziming Liu, and Tommi S. Jaakkola. Restart sampling for improving generative processes. In Shirty-seventh Conference on Neural Information Processing Systems, 2023a. URL
net met/forunm ?id=wE uemocyH2.
Yilun Xu, Ziming Liu, Yonglong Tian, Shangyuan Tong, Max Tegmark, and Tommi Jaakkola. Pfgm++: Unlocking the potential of physics-inspired generative models. In International Conference on Machine Learning, pp. 38566-38591. PMLR, 2023b.
Eunbi Yoon, Keehun Park, Sungwoong Kim, and Sungbin Lim. Score-based generative models with lévy processes. Thirth-seventh Conference on Neural Information Processing Systems, 2023. URL https: /tttps://operrett ew net / forum? (itpmmmid000000 d=00R33HX00Gm
Qinsheng Zhang and Yongxin Chen, Fast sampling of diffusion models with exponential integrator, 2023.
URL httpt::t////aatttt org/ abs/ http:///w...rra///22./.....2 13902.
CONTENTS
1
Introduction
2 Background
2.1 Diffusion Models
3
2.2
Student-t Distributions
3 Heavy-Tailed Diffusion Models
3
3.1 Noising Process Design with Student-t Distributions.
3.2
Parameterization of the Reverse Posterior
4
15

---

Published as conference paper ICLR 2025
3.3
Training with Power Divergences
5
3.4 Sampling
6
3.5
Specific Instantiations: t-EDM
4
Experiments
8
4.1
Unconditional Generation
9
4.2 Conditional Generation
5 Discussion and Theoretical Insights
10
A Proofs
17
A.1 Derivation of the Perturbation Kernel
17
A.2 On the Posterior Parameterization
17
A.3 Proof of Proposition 1
19
A.4 Derivation of the simplified denoising loss
21
A.5 Proof of Proposition 2
22
A.6 Deriving the Denoiser Preconditioner for t-EDM
24
A.7 Equivalence with the EDM ODE
................................
25
A.8 Conditional Vector Field for t-Flow
................................
26
A.9 Connection Denoising Score Matching
................................
26
A.10 ODE Reformulation and Connections to Inverse Problems.
27
A.11 Connections Robust Statistical Estimators
28
B Extension to Flows
30
Experimental Setup
31
C.I Unconditional Modeling
31
C.I.1
HRRR Dataset
31
C.1.2
Baselines
31
C.1.3
Evaluation
33
C.1.4 Denoiser Architecture
................................
34
C.I.5 Training
34
C.1.6 Sampling
................................
35
C.1.7 Extended Results on Unconditional Modeling
35
C.2 Conditional Modeling....
35
16

---

Published as conference paper ICLR 2025
C.2.1
HRRR Dataset for Conditional Modeling
35
C.2.2
Baselines
36
C.2.3
Denoiser Architecture
36
C.2.4
Training.
36
C.2.5
Sampling
36
................................
C.2.6
Evaluation
37
C.2.7
Extended Results Conditional Modeling
37
C.3 Toy Experiments
38
Optimal Noise Schedule Design
39
D.I Design for EDM
40
D.2 Extension t-EDM
41
D.3 Extension to Correlated Gaussian Noise
41
E
Lok-Likelihood for t-EDM
42
F Discussion and Limitations
43
F.1 Related Work
43
F.2 Limitations and Future Work
45
A PROOFS
A.1 DERIVATION OF THE PERTURBATION KERNEL.
Proof. The proof is adaptation from Ding (2016) and included here for completeness. We define the joint state x = X Consequently,
fe(t)
q(x xo) = ( E, = [Pe: P4-A: E = (2)
Ia.
o
By re- ce-parameterization of the joint Student-t distribution ((x(xo), we have,
x r + / z ~~ No(,, and K (2)/2v This implies that the conditional distribution qxx(x,, K) MN, E/x). Therefore, following properties of Gaussian distributions, q(x,) Xx, x) ((ux0,, of I.) Therefore, from reparameterization,
(15)
x = 1480 + o.z/vR
(16)
which implies that qxx (x)) La(uxxo, of La, v). This completes the proof.
A.2 ON THE POSTERIOR PARAMETERIZATTIN
The perturbation kernel g(xx|xx)) for Student-t diffusions is parameterized as,
q(x,/xo) (a4x40, of La
(17)
17

---

Published as a conference paper ICLR 2025
Using ce-parameterization.
X = 14X0 + oi VR' La) 22(\/v
(18)
During inference, given a noisy state X,, we have the following estimation problem.
x = 4 x,] + o [x,]
(19)
Therefore, the task of denoising can be posed either estimating E[xp] x5] or 18] VR [x] With this motivation, the posterior pe (x (x)) can be parameterized appropriately. Recall the form of the forward posterior
V d1 (x xb) ta(fu:: = d
(20)
Tm (t) (xt - piX0),
oi = 0 oi- - o (t)oR((22))
ir = Pt
(21)
of
of
where di 3/ [xx - pexol22 2 Further simplifying the mean Pr,
2 (t)
Xo, t) = 11-A100 +
(xi
-4200)
(22)
of
(1 (t)
22 (1)
= M1AA100
X4
1480
(23)
0
of
2
2
-(1-S1 14
22
Xo
+
(24)
o
Therefore, the mean μe (xx,,)) of the reverse posterior
xx)
can be parameterized as,
22 (t)
B[xo|x,1 + (2) (1) 0z) +
(x t) = (PI-2e-1 -M
of
of X
(25)
(1) (M-SI-- p4 of
(1) of x;
(26)
where E[x,|x,] is learned using parametric estimator Dx ot). This corresponds to the xy-prediction parameterization presented in Eq. 7 in the main text. Alternatively, From Eqn. 19,
0z. (1)
(1) o
( t)
11-A1
u4
E. E[xo x+
Xt
(27)
of
03 (1) )(
(1)
14 14-A1
(x.- R.
VK [x1))
X1
(28)
Pt
of
of
Me-Ai
Xt:
1t- - 14
02 (t)
|5|
[x]]
(29)
1t
1t
of
V
0 (t)
Ht-At. At
Xx
11-A1
co co(xi, 01)
(30)
Ht
μt
o?
Xx] is learned using parametric estimator co(x1, This corresponds to the e-prediction parameterization (Ho et al., 2020).
where I8
18

---

Published as a conference paper ICLR 2025
A.3 PROOF PROPOSITION
We restate the proposition here for convenience,
Proposition 1. For arbitrary distributions q and p. in the limit - 0, Dr(4 p) converges to DRI.g |I p). Consequently, the objective in Eqn. 9 converges to the DDPM objective stated in Eqn. 1.
-
Proof. We present our proof in two parts:
1. Firstly, we establish the following relation between the y-Power Divergence and the KI.-Divergence between two distributions and p.
(4 p) DxL(q p) + O())
(31)
2. Next, for the choice of 7/= 2-4-d- we show that in the limit of y - 0, the optimization objective in Eqn. converges to the optimization objective Eqn.
Relation between ( p) and D,(q p). The 7-Power Divergence as stated in (Kim et al., 2024)
assumes the following form:
Da p) [Cs(a.P) -C,((0.))- H6,(1)
(32)
where y (-1,0) U (0, oo), and,
(v) - 11pl12t (aatti---((((x)) ((nrxx))
ntG Ca(a,)) - -f((x) (((x) p(x)
dx
(33)
For subsequent analysis, we assume y - O. Under this assumption, we simplify Hg(9) as follows. By definition,
H.)(g) -- -1ll147 ( [(xx)++dxx dx
ihe
(34)
( [(xx)((2))xx
+
(35)
( (xx) (xg i
(36)
(J [(x) + [ logg(x)) + O(72)]xx
(37)
(s [(xxxxxx++ + ((x)) (o8((xxx + 0(73) i2,
(38)
1 y g(x) logg(x)dx + 0 i
(39)
Using the approximation (1 + 6z) + adx for small ò in the above equation, have,
H1) - 14112477
1 1 [f ((x) log (x))xx + [xx]
7
(40)
1 + 7(1 7)) [/ ((x)) log q())( (x()))
(41)
19

---

Published as conference paper ICLR 2025
where we have used the approximation + 11-1 y in the above equation. This is justified since ? assumed be small enough. Therefore, have,
I+y
(g) - 14l14, -- -(1+ 1 S ((x) log ((x) + 0(13)
(42)
Similarly, we now obtain an approximation for the power-cross entropy as follows. By definition,
p(x) dx
Cg(pp) = g(xx
(43)
|pll147
J ((xx) (1 (1+ y log ( p(x)
1pll1t7 ) +0(73) dx
(44)
(J (faxx)dx +> ((xx) log
p(x) (1(x)) + 0(13))
(pllzt7
(45)
= -(1+ 1 S ((x)) logr(x)xx - & ((xx) log lell dx + 0(13))
(46)
From Eqn. 42, it follows that,
p(x) (1==7 +
lel Ipl
(47)
Therefore,
log 1pl1 log 1 & (oxx)) +073)
(48)
~7 log) p())(
(49)
where the above result follows from the logarithmic series and ignores the terms of order O()2) or higher. Plugging the approximation in Eqn. 49 Eqn. 46, have,
C.(ap)) = -
1 (1 7 ((x)) log p(x)xx x 1 ((x)) log lell,t, dx 0(92))
(50)
--(1 1+ (ox) 0(92)
(51)
Therefore,
11 (c((..)) Hly(1)
(52)
((xx) o(xx)x x(x)xx +xxxloogxxxxxxx+(3))
(53)
-f((x) ((x) log ((x) dx + 0(7)
(54)
p(x)
DKI,(q l1 p) + (())
(55)
This establishes the relationship between the KL and P-Poeer divergence between distributions. Therefore, for two distributions and p. the difference in the magnitude of DKL(q q || p) and (2 || p) is the order of (7)). In the limit of 0, the (g p) DKI(q p). This concludes the first part of our proof.
20

---

Published as conference paper ICLR 2025
_ and finite-dimensional dataset Equivalence between the objectives under 7 O. For 7 7+d
with IRd it follows that y O implies - cc. Moreover, in the limit of v oc, the multivariate Student-t distribution converges to Gaussian distribution. As already shown in the previous part, under this limit, (g p) converges to (x p). Therefore, under this limit, the optimization objective in Eqn. 9 converges to the standard DDPM objective in Eqn. 1. This completes the proof.
A.4 DERIVATION THE SIMPLIFIED DENOISING LOSS
Here, we derive the simplified denoising loss presented in Eq. 10 in the main text. We specifically consider the term (,(g (g( Xt- ar|xx, Xn) || po( (xxxxxxxll -x)))))))) in Eq. 9. The 7-power divergence between two Student-t distributions is given by,
Dy Dyllll
1ot (1+ x nd
17r [ भपे (1 Az)
(56)
(1 - (M) Σo) + ( m1) 1
(HO (omimi) mi)
E शाकन
_2 where == v+d
Recall, the definitions of the forward denoising posterior,
/ d1 4 d
((x1-ar talftt
2
La, v + dd)
(57)
(1) (xi mix0),
2. (t)
P1. Mi
of
2 0i-it
(58)
of
of
and the reverse denoising posterior,
po(x--A1 x1) ta(uo (x., f), o, Ia d)
(59)
where the denoiser mean μ (Xe, t) is further parameterized as follows:
(x,, o. (t) xx + Me-ar
22 (t) of Me Do (x(
(60)
of
Since we only parameterize the mean of the reverse posterior, the majority of the terms in the 7-power divergence are independent of and can be ignored (or treated as scalar coefficients). Therefore,
(on(xx--xxx,xx)) Xo) po (x:xx-aa (x)x ) (pe mo(xx, t)) (ft: m)(xx,t))
(61)
(62)
(1) 12 11x0 Do Do(x,, 01)1
14-21 of 14
(63)
For better sample quality, it is common to ignore the scalar multiple in prior works (Ho et al., 2020; Song et al., 2020). Therefore, ignoring the time-dependent scalar multiple,
Dx( xo) po xx xxlx) xo - Do(xi, ot)1
(64)
Therefore, the final loss function Lg can be stated as,
((I)= = Exg~r(xo, Br Ban (x((w) Do + c 0.)
.1)) -xo
(65)
VK"
21

---

Published as a conference paper at ICLR 2025
A.5 PROOF OF PROPOSITION
We restate the proposition here for convenience,
Proposition 2. The posterior parameterization Eqn. 6 induces the following continuous-time dynamics,
Pe jrn - [[oonn))++ 1i4e
dx: =
(x. ()Do(xi,))) di + Or))S.
(66)
Mt_
where f : IR+ x I2+ - R and R x R+ - R+ are scalar-valued functions, 3r E IR+ is a scaling coefficient such that the following condition holds,
of2(t) (
(oi ((t)g(or, -1 -(,
(67)
where juc or denote the first-order time-derivatives of the perturbation kernel parameters Pr and respec- tively and the differential ds,- (a dt, v + d).
Proof. We start by writing single sampling step from our learned posterior distribution. Recall pa xx) ta(us(X (), of La, d)
(68)
where (using the e-prediction parameterization in App. A.2),
(x t)
14-Ai
X1
1
021
2 (t)
Pt oi c)
(69)
He
σ
Mr
From re-parameterization, we have,
A1 = po((x. 0) +
Ōr b
zt
N(O, In), Rit ~ x²(v + dd/(vv + d)
(70)
Vit
X1-
11- X1
(1)
Me-1 of co(x), 0z) + σt
2t
(71)
Ht Moreover, we choose the posterior scale to be the same the forward posterior g(xe-1 Xxc, Xo) i.e.
Me
Vkz
(t) of
of of o
La
(72)
This implies,
of (oi- - -i:)
() (t) (tt
(73)
Substituting this form of o2 (t) into Eqn. 71, we have,
Xx-At
1
X,
i- - -0?)
M1 M 2
of
co(x),
(74)
H4
σe
(t)
/ki
14-Ai X1 o
1
(0-At--0
-0?)
Me-
C6 (x(1),00)
+
σt 21
(75)
H1
H4
Vkc
14-D1 X1 + ot
(t)
1
(i- - -0,) - -1+ 1
11-11
Eq E 02) +
v Vkz
ōt
zt
(76)
Mt-A1 Xx
+ T2
(Te(()
1
(o%- - - - 1
A Al ep( 0z) + σt
Zt
(77)
J4
where in the above equation we use the first-order approximation j4: following design choices:
He Pe-A0 At
Next, we make the
22

---

Published as conference paper ICLR 2025
1. Firstly, we assume the following form of the reverse posterior variance o7.
of 3(t)g(o,,
(78)
where g RR R+ R and ((t) R$ represents a time-varying scaling factor chosen empirically which can be used to vary the noise injected at each sampling step. It worth noting that a positive oz (as indicated in the definition of g) is a consequence of a monotonically increasing noise schedule o, in diffusion model design.
2. Secondly, we make the following design choice:
(2()) (o)- 02)-1 az)
(79)
where R R IR
With these two design choices, Eqn. 77 simplifies as:
(2(() (0--sr- (06.1-6)) -0;) 1 ti Al (x) + σe
At-At
X1
a
(80)
1t
1t
vist
1
X1 of o() At (x( + A(t)g(or, At
2t
(81)
14
Vka
He-At
X1 + of f(or, + ju
c +)) V At
Zt
(82)
P4.
Vkr
X1-Ar - 81
(4--Se - 1)xx + f(or,0t) +
He
ED + 0,) A
Z:
(83)
14
Me
1u
X, ((f(or10)) + me
X1-Ai
+t+ O
(84)
P4
Me
/ki
the limit of At
0:
j4 X1
oz f(or, + je
oz) + VB(tgg(on,))
dWe V/ve
dx:
(85)
11
Me
dW,
Xt 14
(0)r +
(xt pr boDo (x) di (B(L)g(or, σ.)
(86)
1
Tive
Moreover, since dW, NoO, dt) and ~ x² + d)/(v + d). the term dS, dWi/ / distributed as a Student-t random variable with dS,- -t(00, v + d). Therefore.
ju jixx, jt
or f(or, ue
j
(o)(x) de ((t))
dx;
(87)
which gives the required SDE formulation for the diffusion posterior sampling. Next, we discuss specific choices of f(a,, ()) and â(),,
On the choice of f'a dz) and g( bz): Recall our design choices:
of at
(88)
(1) (oi -7) - = O.)
(89)
23

---

Published as conference paper ICLR 2025
Substituting the value of of from the first design choice to the second yields the following condition:
- -
((t))((r,
f(a,, oz)
(90)
(1)
This concludes the proof. It is worth noting that the above equation provides two degrees of freedom, i.c., we can choose two variables among 012(t). 9: I and automatically determine the third. However, is more convenient to choose (t) and g. since both these quantities should be positive. Different choices of these quantities yield different instantiations of the SDE in Equ. as illustrated in the main text.
A.6 DERIVING THE DENOISER PRECONDITIONER FOR T-EDM
Recall denoiser parameterization,
Fo Ds(x, o) = Cakipaa, v/)x + Cout Fo (Gn(a, v/xx, Cnoise (o))
(91)
Karras et al. (2022) highlight the following design choices, which we adopt directly.
1. Derive en based on constraining the input variance
2. Derive Cskip and Com constrain output variance to and additionally minimizing Cou to bound scaling errors in o).
The coefficient On can be derived by setting the network inputs to have unit variance. Therefore,
Varx,, (Ci(o)((())))) + n)]
(92)
(a, 1)² Varx Xu + n
(93)
Cin(a, 2)2 (oaus + 2 02) _02)
v
1
(94)
Ca(o,v)
1/
σ2 + o d
(95)
1/
The coefficients Cskip and Cant can be derived by setting the training target to have unit variance. Similar to Karras et al. (2022) training target be specified as:
Furget
(xo - Cakip ())
(96)
Cour v)
Var xo. (Piurgr (No: n1;
1
(97)
Varx.. v (xo Cekip (o, v) (Xo n))
1 Cakip v) (xo n)]
(98)
Sulo Varxo.
Xo
1 Varxo Caiipla, v) n]]
(99)
Comi (a, u)²
(100)
Coul (a, 2)
Varxo. (1 Catip(a, v)) xo cakip
(101)
Comr (a, 2)2
(1- (1- Cakip(o, dd + cuiplo, 1)2 (102)
Lastly, setting Caki v) to minimize Cour Coulla, (o, v), we obtain,
Cakip v) = 2 3 2
0² + saaa
(103)
24

---

Published as a conference paper ICLR 2025
Consequently Cour(o,)) v) can be specified as:
Coul( (3, 1) V v-
1/
v aanta// V v
0² + o
(104)
A.7 EQUIVALENCE WITH THE EDM ODE
Similar to Karras et al. (2022), we start by deriving the optimal denoiser for the denoising loss func- tion. Moreover, reformulate the form of the perturbation kernel as q(xx (u of Iv. ta (()xx()xo, (1)) v) by setting b = s(1) and or = s(t)a((). The denoiser loss can then be specified
follows,
C(),o)
= Exx Bn- D(s + - xo Bor. -
(105)
Ex -p(x))
xo13] -xx0)))-xx1113
(106)
Exx
-xol3
(107)
N Σ (x(x; ((0 LN x:13]dx
(108)
where the last result follows from assuming p(xo) as the empirical data distribution. Thus, the optimal denoiser can be specified by setting VnC(D,o) = 0. Therefore,
VDL(D,o)=0
(109)
Consequently,
1 N
V
- x: dx = 0
(110)
S fa(x;
s(t) (f
xx) =
(111)
Ex((x (a - x,) 0
(112)
The optimal denoiser D(x, o), can be obtained from Eq. 112 as,
D(x, Σ, E, tatax (((x;((t)xx) s(t)x, sf Id,,v)
(1 v\x:
(113)
We can further simplify the optimal denoiser as,
D)x,0) Σ, ta(xx; s(t))(()))))) (())202 v)x:
(114)
Σ, talx: s(t)x,, s(t)2
Σ La Xi, 22 Ia u)xi
(115)
Σ, tal x X,, o² 1)
D
(116)
Next, recall the ODE dynamics (Eqn. 12) in our formulation,
dx:
Pt 14
it
X
(xt
ur Dax, (()))
(117)
dl
σe
Pt
25

---

Published as conference paper ICLR 2025
Reparameterizing the above ODE by setting Pr s(t) and or s(t)a(t),
dx+
1
δt
1
X
+
(xt - s(t) (o(x,
(118)
dt
01
Me
8(t)
8(t)
+ +
(x.- (x)ox()))(xx.oo))
(119)
s(t)
s(()
o(t)s(t)
8(()
6(1)
(xt (x(x,
(120)
s(t)
ot))
8(1) d(t)
X:
a(t) (x: sfx) ot)))
(121)
(t))
Lastly, since Karras al. (2022) propose train the denoiser Da using unscaled noisy state, from Eqn. 116, we can re-write the above ODE as,
dx:
s(t) a(t) o(t)s(t) dt st))
Do x (((t) ott)
+
att) x
(122)
o(t)
The form of the ODE in Eqn. 122 equivalent to the ODE presented in Karras et al. (2022) (Algorithm 1 line 4 in their paper). This concludes the proof.
A.8 CONDITIONAL VECTOR FIELD FOR T-FLOW
Here, we derive the conditional vector field for the t-Flow interpolant. Recall, the interpolant in r-Flow derived as follows,
txo + (1 -) VK'
( (v)/vv
(123)
It follows that,
X; = 1 X X
x
+ (1
E)
x,
(124)
Moreover, following Eq. 2.10 in Albergo et al. (2023), the conditional vector field b(x;, t) can be defined as,
b(x,, ()
E x.]
E[xo x.]
E
(125)
From Eqns 124 and 125, the conditional vector field can be simplified as,
b(x,, 1) X B Vk X1
(126)
This concludes the proof.
A.9 CONNECTION TO DENOISING SCORE MATCHING
We start by defining the score for the perturbation kernel q(x:/xo). The pdf for the perturbation kernel q(x, (xo)= = ta(pux, of Ia, v) can be expressed as (ignoring the normalization constant):
-(++d)
(xx(xx)) æ
|
(x: -1x0) ( (x: m1x0) vo?
(127)
26

---

Published as conference paper ICLR 2025
Therefore,
V d
V X, (xg)((;(Xo)
xx, log
1 +
Iixe 1ixo1
(128)
2
vo?
V + d
2
X
14x0
(129)
2
Xx MeXo
1 vo,
Denoting d1 211 pexolè for convenience, have,
V + d
1
Xx log ((x)
Xt 14x0
(130)
V d1 of
In Denoising Score Matching (DSM) (Vincent, 201 1), the following objective is minimized,
LDSM pxc-p((hxxx--ppxx)))
Vx, log (ogg(x;(xo) - so (xc, t)
(131)
for some loss weighting function 7(t). Parameterizing the score estimator so 6x(x () as:
1 so(x;; t) 1/ d1 oā
x 1o--DDo(xx,0 o.) (u++)
(132)
With this parameterization of (s (x( t) and some choice (t), the DSM objective can be further simplified as follows,
LUSM Ex p(x)) E, E,E, N10 [ (1=+11) d
di xo Doluuxo or Vx 01)
(133)
A(xe, u, 1) 00 - Doluxxo + or TR. oi) :
= Exx Br
(134)
2
ptd otd which equivalent to a scaled version of the simplified denoising loss Eq. 10. This concludes the proof. As additional caveat, the parameterization in Eq. 132 depends
where u,
di Xx+- p4x0l2. which can be approximated during inference as, dj s * pr Dx o;)
A.10 ODE REFORMULATION AND CONNECTIONS TO INVERSE PROBLEMS.
Recall the ODE dynamics in terms of the denoiser can be specified as,
dxi je Xt.
ōt
ftt
(xt
pr OiDo(x(,or))
(135)
dl
Mt
at
From Eq. 132, we have,
(o@Vud dj
X. ue Do(:: σ1) = =
Vx, (og,(x;,()
(136)
v
where di = - Ix: 4xx0
Substituting the above result in the ODE dynamics, we obtain the reformulated
ODE in Eq. 14.
dx:
f Pex, x +
1/ 2u+d dl
Pt
Vx p(x,,()
(137)
dl. 1t. v
d
Mr
27

---

Published as conference paper ICLR 2025
Since the term di data-dependent, can estimate it during inference di di f 0,)1.. Thus, the above ODE can be reformulated as,
dx, fu Xt + of v
(
V di
iu de de Vx log
(138)
dt
14
Tweedie's Estimate and Inverse Problems. Given the perturbation kernel g(x,(xo)) = ta(4x0,
we have,
X 14 + O+ VR
((0, x2(1\vv
(139)
follows that,
x; = |x
+ o,E
x
(140)
vk
[[xo|xx xt]
Xx - -
Xx
(141)
14
/
+ v +
X + of
Vx log (x),,))
(142)
H4
which gives us an estimate of the predicted Xy at any time Moreover, the framework can also extended for solving inverse problems using diffusion models. More specifically, given conditional signal y, the goal is to simulate the ODE,
dx;
P1. X4 +
1 +
d'
f4
d,
o,
Vx logg((xxly)
(143)
dt
/ d
Pt
14 X4 + of
(u+ d'
it
à.
[w Vx logp(y/x)) + Vx log (xx)
(144)
v + d
where the above decomposition follows from p(x, (y) xppp//p(/x//////p (),,) p((((,)) and the weight represents the guidance weight of the distribution p(y[x;). The term log p(y (x;) can now be approximated using existing
posterior approximation methods in inverse problems (Chung et al., 2022; Song et al., 2022b; Mardani et al., 2023; Pandey et al., 2024b)
A.11 CONNECTIONS TO ROBUST STATISTICAL ESTIMATORS
Here, we derive the expression for the gradient of the 7-power divergence between the forward and the reverse posteriors (denoted by and pe, respectively for notational convenience) i.e., VoD,(q pe). By the definition of the 7-power divergence.
Dy(2 ll po) = -
7 [[C.(a.po) - - H,WW], (-1,)) ( (0,
(145)
Hg(p) = = 1lpl = -[(((x)))tt +
C.(..)) =- ((x)) ( p(x) dx
|Pll (Ipl1197)
(146)
Therefore,
VoDa(ul1 (1 po) 1 (oC,(y
(147)
28

---

Published as a conference paper at ICLR 2025
Consequently, we next derive an expression for VgC,(g..o))
pa(x)
7
VoC,(1 -Vo [(x)
dx
(148)
|Ppo 1+7
= =-7 f ((xx)
Px(xx)
7-1
po(x) dx
Va
(149)
| |polltnn
Ippol 14
- f ((xx)
(x(x)
9-1 1pollat7 Vopo(x) - po(()) Vo lpoll1t dx
(150)
|polltn
pee 2 1+7
From the definition of [pollatv*
lpellat7 - (J ([oo(s) tx
(151)
Vo 1poll147 Vo (Spo + dxx
147
(152)
1+ (s ((o(()) (7)x) dx
17
[ (o(
(153)
I+> (s (Do(x)++) dx
(1+ vo(x)
(154)
Vo 1pell4 = (J po(x))+t dxx
S (o(x)) Vopo(x)dx
(155)
l1 117
ps (x(x)) (opo(xxx
(156)
S (po(x))
l|pol 1+4
px(x) 1+7 Vo lx)po(x)dx
(157)
(Spo(x))+)xxx
Ipolllts S
Da(x)2+7
Ve log po (x)dx
(158)
[pp(x) +
(x(x)
1polli (x)) V log Px(
(159)
where we denote (x)) =
for notational convenience. Substituting the above result in Eq.
150, we have the following simplification,
29

---

Published as conference paper ICLR 2025
Algorithm 3: Training (t-Flow)
Algorithm 4: Sampling (t-Flow)
repeat
1: sample (0( Ia. v)
2:
X1 p(xi)
2: for onriE (O....,, -1 do
3:
t Un.form((1,..... .T]
3: d - (xi - 60
je 1,00=
4: x,-1 - x; + (f++1 ft)d
Mxx1 + n,n, ta(0, v)
5: if 4 0 then
Take gradient descent step on
6: d. - (x++1 (x ())
Ve In ox)x
until converged
7: X
+ (l441 - ( 1dl)
8: end if
9: end for
10: return X.V
Figure 5: Training and Sampling algorithms for t-Flow. Our proposed method requires minimal code updates (indicated with blue) over traditional Gaussian flow models converges the latter
pe (x(x)
[ 14. Vepa(x) - pe Vo |pell1+7 dx
VoC,(a,po) = - ((x)
(160)
\1po 1+7
Ipoll +
- --7 S ((x))
Po (x)
7-1 [poll2ty Vopo(x) - (x() lIpe Ipoll2t7 Vo
dx
|pP 11t
100 11
(161)
px(x)
[xpo(x) - Eo) Vo po()) dx
= =-7 ((x))
(162)
\ppel 1477
pel 1+7
(x)
(x(x) Vo log Po
po x E (x)
Vo log px)p)
- ((xx)
dx
(163)
[Pee 147
Ipe
(164)
V VaC,(1.Po) 10p0)-=1 -07 ((x) po(x) ) (v log loggp(x) - Exo(x) Vo log dx |Politt7
(165)
Plugging this result in Eq. 147, we have the following result,
VoDz(all 1 -v. = =-
((x) -
) (( log - (x) (V log]o(x)] dx [oo 1+7
Po(x)
(166)
This completes the proof. Intuitively, the second term inside the integral in Eq. 166 ensures unbiasedness of the gradients. Therefore, the scalar coefficient controls the weighting on the likelihood gradient and can be set accordingly to ignore or model outliers when modeling the data distribution.
EXTENSION TO FLOWS
Here, discuss extension of framework to flow matching models (Albergo et al., 2023; Lipman et al., 2023) with a Student-t base distribution. More specifically, we define a straight-line flow of the form,
= tx1 + (1
E E~ E~~(00 K (v)//v
(167)
30

---

Published as a conference paper ICLR 2025
where X1
p(x)). Intuitively, at given time the flow defined in Eqn. 167 linearly interpolates between data and Student-t noise. Following Albergo et al. (2023), the conditional vector field which induces this interpolant can be specified as (proof in App. A.8)
dx: dt
b(x,,))
E. X
VR
Xx
(168)
b
We estimate E [xo] by minimizing the objective
((0) = Exo-p(xo) vin
[/ + +(1 (1-t)
t) VK"
6
2
(169)
VR
We refer to this flow setup as f-Flow. To generate samples from our model, we simulate the ODE Eq. 168 using Heun's 's solver. Figure illustrates the ease of transitioning from a Gaussian flow to t-Flow. Similar to Gaussian diffusion, transitioning to t-Flow requires very few lines of code change, making our method readily compatible with existing implementations of flow models.
C EXPERIMENTAL SETUP
C.1 UNCONDITIONAL MODELING
C.1.1 HRRR DATASET
We adopt the High-Resolution Rapid Refresh (HRRR) (Dowell et al., 2022) dataset, which is an operational archive of the US km-scale forecasting model. Among multiple dynamical variables in the dataset that exhibit heavy-tailed behavior, based on their dynamic range, we only consider the Vertically Integrated Liquid (VIL) and Vertical Wind Velocity at level 20 (denoted as w20) channels. How to cope with the especially non- non-Gaussian nature of such physical variables km-scales, represents entire subfield of climate model subgrid-scal scale parameterization (e.g., Guo et al. (2015)). We only use data for the years 2019 2020 for training (17.4k samples) and the data for 2021 (8.7k samples) for testing: data before 2019 avoided owing to non-stationaries associated with periodic version changes of the HRRR. Lastly, while the HRRR dataset spans the entire US, for simplicity, we work with regional crops of size 128 x 128 (corresponding to 384 x 384 km over the Central US). Unless specified otherwise, we perform z-score normalization using precomputed statistics as a preprocessing step. We do not perform any additional data augmentation.
C.1.2 BASELINES
Baseline I: EDM. For standard Gaussian diffusion models, the recently proposed EDM (Karras al., 2022) model, which shows strong empirical performance on various image synthesis benchmarks and has also been employed recent work in weather forecasting and downscaling (Pathak et al., 2024; Mardani et al., 2024). To summarize, EDM employs the following denoising loss during training,
((0) o Exo~p(xo) E,
Do(Xo + [o)
.) X xolø]
(170)
where the noise levels are usually sampled from LogNormal distribution, p(a) LogNormal( ( m
Baseline EDM Inverse CDF Normalization (INC), is commonplace to perform z-score normalization data pre-procsssing step during training. However, since heavy-tailed channels usually exhibit high dynamic range, using z-score normalization for such channels cannot fully compensate for this range, especially when working with diverse channels in downstream tasks in weather modeling- An alternative could be to use a more stronger normalization scheme like Inverse CDF normalization, which essentially involves the following key steps:
31

---

Published as conference paper ICLR 2025
Channel: VIL
101
Dorcky
Densty
Donaty
100
1.0
200
250
300
2.6
25
30
35
4.6 Velue
4.5
5.6
330
1.00
200
256
780
120
Value
Value
Channel: w20
107
13--
10'
12
Donoby
Densiy
22-
Dendty
33
23
33
-330
Original Samples
Samples (after INC)
Denormalized Samples
Figure 6: Inverse CDF Normalization. Using Inverse CDF Normalization (INC) can help reduce channel dynamic range during training while providing accurate denormalization. (Top Panel) INC applied to the Vertically Integrated Liquid (VIL) channel in the HRRR dataset. (Bottom Panel) INC applied the Vertical Wind Velocity (w20) channel in the HRRR dataset.
1. Compute channel-wise 1-d histograms of the training data.
2. Compute channel-wise empirical CDFs from the 1-d histograms computed in Step 1.
3. Use the empirical CDFs from Step 2 to compute the CDF at each spatial location.
4. For each spatial location with CDF value p, replace its value by the value obtained by applying the Inverse CDF operation under the standard Normal distribution.
Fig. 6 iillustrates the effect of performing normalization under this scheme. As can be observed, using such a normalization scheme can greatly reduce the dynamic range of a given channel while offering reliable denormalization. Moreover, since our normalization scheme only affects data preprocessing, we leave the standard EDM model parameters unchanged for this baseline.
Baseline 3: EDM + Per Chameel-rreconditonnn (PCP). Another alternative to account for extreme values the data (or high dynamic range) could be to instead add more heavy-tailed noise during training. This can be controlled by modulating the Tuean and Rst parameters based the dynamic range of the channel under consideration. Recall that these parameters control the domain of noise levels a used during EDM model training. In this work, channel dynamic range (denoted as d). More specifically, we set,
simple heuristic modulate these parameters based the normalized
meaa -12 + RBF d. 1.0,
(171)
32

---

Published as a conference paper ICLR 2025
where RBF denotes the Radial Basis Function kernel with radius=1.0, parameter and magnitude scaling factor a. We keep Tau = 1.2 fixed for all channels. Intuitively, this implies that higher normalized dynamic range (near 1.0) corresponds to sampling the noise levels from more heavy-taile distribution during training. This is natural since signal with larger magnitude would need more amount of noise to convert it to noise during the forward diffusion process. In this work, we set a 3.0, 3 = 2.0, which yields T "mean 1.8 and T 0.453 for the VIL and w20 channels. respectively. It is worth noting that, unlike the previous baseline, we use z-score normalization as a preprocessing step for this baseline. Lastly, we keep other EDM parameters during training and sampling fixed.
Baseline 4. Gaussian Flow. Since we also extend our framework to flow matching models (Albergo et al., 2023; Lipman et al., 2023), also compare with linear one-sided interpolant with Gaussian base distribution. More specifically,
X =x1 + (1-t)e, t)e, e~ e~N(00,
(172)
Similar to t-Flow (Section B), we train the Gaussian flow with the following objective,
Bu-u0, [ + (1 t)), -1/2]
cootxx
2
[())
(173)
C.1.3 EVALUATION
Here, we describe our scoring protocol used in Tables 2 and 3 in more detail.
Kurtosis Ratio (KR). Intuitively, sample kurtosis characterizes the heavy-tailed behavior of distribution and represents the fourth-order moment. Higher kurtosis represents greater deviations from the central tendency, such as from outliers in the data. In this work, given samples from the underlying train/test set, we generate 20k samples from our model. We then flatten all the samples and compute empirical kurtosis for both the underlying samples from the train/test set (denoted as kdma) and our model (denoted as ksim). The Kurtosis ratio is then computed as,
Ksim
KR
(174)
kgata
Lower values of this ratio imply a better estimation of the underlying sample kurtosis.
Skewness Ratio (SR). Intuitively, sample skewness represents the asymmetry of tailed distribution and represents the third-order moment. In this work, given samples from the underlying train/test set, we generate 20k samples from our model. We then flatten all the samples and compute empirical skewness for both the underlying samples from the train/test set (denoted as ddaaa) and our model (denoted as ssim). The Skewness ratio is then computed as,
SR
Ssim
(175)
Sdata
Lower values of this ratio imply a better estimation of the underlying sample skewness.
Kolmogoroy-Smirnn 2-Sample Test (KS). The KS (Massey, 1951) statistic measures the maximum difference between the CDFs of two distributions. For heavy-tailed distributions, evaluating the KS statistic the tails could provide a useful measure of the efficacy of our model in estimating tails reliably. To evaluate the KS statistic at the tails, similar to prior metrics, we generate 20k samples from our model. We then flatten all samples in the generated and train/test sets, followed by retaining samples lying above the 99.9th percentile (quantifying right tails/extreme region) below the 0.1th percentile (quantifying the left tail/extreme region). Lastly, we compute the KS statistic between the retained samples from the generated and the train/test sets individually for each tail and average the KS statistic values for both tails to obtain average KS score. The final score estimates how well the model might capture both tails. As an exception,.. for the VIL channel, we report KS scores only for the right tail due the absence of left tail in the underlying samples for this
33

---

Published as conference paper ICLR 2025
Parametens
EDM (+INC, +PCP)
LEDM
Flow/tFlow
Caiip
and o² fom
v- aad
Preconditioner
am / 102 oa
D=
Gn
V/ o² oan
" 1/v
Croeec
1 log
log
log (V mid))
1, (4(0.
Training
A(r)
(/)
(1,))
t-Flow
Loss
Eq. 170
Eq.
Gaussian Flow -q. 173
Solver
Heun's (2nd order)
Heun's is (2nd order)
Houn's 's (2nd order)
F-llow: Eg. 168, xo ~ ta(0. Ia. v)
ODF
Eq. 12, xr (0, Fa) min
Eq. 12, xT tz(0, Ta v)
Sampling
Flow: Eq. 168,
Discretization
(or w-1 1
(o + N lo mmn)))
(o + w-1 oo nnann))
Scaling: (4) Schedule:
1-t
1.0
Flow:
Hyperparaetters
VIL: [3. 5. w20: [3,5, 7)
I-Flow (J)
VIL=(3, 5,
w20= 7,9]
EDM:
EDM (+INC) (HINC)--1.. :-2,2, 1.2
Treear Red
(+PCP) (L)
-1.2, 1.2
N/A
VIL: 1.8, 1,2
w20: 0.453, 1.2
80, 0.002
80, 0.002
1.0, 0.01
NFF
7
Table 5: Comparison between design choices and specific hyperparameters between EDM (Karras et al., 2022) (+ related baselines) and E-EM (Ours, Section 3.5) for unconditional modeling (Section 4.1). INC: Inverse CDF Normalization baselines, PCP: Per- Per-Channel Preconditioning baseline, VIL: Vertically Integrated Liquid channel in the HRRR dataset, w20: Vertical Wind Velocity at level channel the HRRR dataset, NFE; Number of Function Evaluations
channel (see Fig. 6 (first column) for 1-d intensity histograms for this channel). Lower values of the KS statistic imply better density assignment at the tails by the model.
Histogram Comparisons. As a qualitative metric, comparing 1-d intensity histograms between the generated and the original samples from the train/test set can serve as a reliable proxy to assess the tail estimation capabilities of all models.
C.1.4 DENOISER ARCHITECTURE
We use the DDPM++ architecture from (Karras al., 2022; Song et al., 2020). We set the base channel multiplier to 32 and the per-resolution channel multiplier to [1,2,2,4.4] with self-attention at resolution 16. The rest of the hyperparameters remain unchanged from Karras et al. (2022), which results in model size of around 12M parameters.
C. TRAINING
We adopt the same training hyperparameters from Karras et al. (2022) for training all models. Model training is distributed across 4 DGX nodes, each with 8 A100 GPUs, with total batch size of 512. We train all
34

---

Published as a conference paper ICLR 2025
Parameter
Model Levels
Height Levels (m)
Zonal Wind (u)
1222444.66777.9,,11111111..
Meridonal Wind (v)
10
Geopotential Height (2)
None
Humidity (q)
1223:4..6.299...11.........
None
Pressure (p)
1:2.345.677899..01113......
None
Temperature (1)
12234456778910.1113....22.
2
Radar Reflectivity (refe)
N/A
Integrated
Table 6: Parameters in the HRRR dataset used for conditional modeling tasks.
models for maximum budget of 60Mimg and select the best-perforiing model in terms of qualitative 1-D histogram comparisons.
C.1.6 SAMPLING
For the EDM and related baselines (INC, PCP), we use the ODE solver presented in Karras el al. (2022). For the t-EDM models, as presented in Section 3.5, our sampler is the same as EDM with the only difference in the sampling of initial latents from Student-t distribution instead (See Fig. 2 (Right)). For Flow and t-Flow, we numerically simulate the ODE in Eq. 168 using the 2nd order Heun's solver with the timestep discretization proposed in Karras et al. (2022). For evaluation, we generate 20k samples from each model.
We summarize our experimental setup more detail for unconditional modeling in Table
C.1.7 EXTENDED RESULTS ON UNCONDITIONAL MODELING
Sample Visualization. We visualize samples generated from the t-EDM and t-Flow models for the VIL and w20 channels in Figs. 9-12
Visualization of 1-d histograms. Similar to Fig. 3, we present additional results on histogram comparisons between different baselines and our proposed methods for the VIL and w20 channels in Figs. 13 and 14.
Is PCP enough for capturing heavy tails? In Table we observe that the EDM + PCP baseline outperforms I-EDM in some scenarios. A natural question is whether a dynamic preconditioning scheme based on the dynamic range of channel is enough for modeling heavy-tailed data. To present some insight into this observation, we train an unconditional EDM + PCP model on 10 channels in the HRRR dataset selected on the basis of the top 10 channels with the highest dynamic range and heavier tails. Our EDM training setup the same other experiments presented in Section 4 with the exception that the input has 10 channels instead a single channel. The updated Wmean for all 10 channels is computed based on the formulation in Eqn. 171. Consequently, this is more difficult modeling task than presented in the main text in Table 2. We compare the generated histograms between the 10-channel run with PCP and the single-channel run with PCP for the VIL and w20 channels in Fig. 7. As can be observed, when scaling in terms of the number of channels, the performance of EDM + PCP takes a big hit. This suggests that merely adjusting the noise schedule during training using heuristics like PCP models to estimate heavy tails accurately.
sufficient to resolve the fundamental limitation of Gaussian diffusion
C.2 CONDITIONAL MODELING
C.2.1 HRRR DATASET FOR CONDITIONAL MODELING
Similar to unconditional modeling (See App. C.1.1), we use the HRRR dataset for conditional modeling the 128 x 128 resolution. We train the model HRRR forecast fields --hour post-analysis to allow for
35

---

Published as conference paper ICLR 2025
VIL
w20
10-
channels) PCP (Single channel)
(10 charnels)
PCP (Single channel)
10
10
10-3
Density
Denaity
10-1
10-3
10
-200
-150
-50
Value
Value
Figure 7: Sample 1-d histogram comparison between EDM + PCP with 10 channels and single-channel training on the test set for the Vertically Integrated Liquid (VIL) channel and Vertical Wind Velocity (w20) channel. Tail estimation suffers when scaling up training in terms of the number of channels in the input. PCP: Per-Channel Preconditioning
some model spin-up following data assimilation. More specifically, for lead time of 1hr, we sample (input, output) pairs at time т and 1, respectively. For the input, at time т, we use state vector consisting of combination of 86 atmospheric channels (including the channel to be predicted time r), which are summarized in Table 6. For the output, at time T + 1, we use either the Vertically Integrated Liquid (VIL) Vertical Wind Velocity at level 20 (w20) channels, depending the prediction task. Unless specified otherwise, we perform z-score normalization using precomputed statistics as a preprocessing step without any additional data augmentation.
C.2.2 BASELINES
We adopt the standard EDM (Karras et al., 2022) for conditional modeling baseline.
C.2.3 DENOISER ARCHITECTURE
We the DDPM+ architecture from (Karras al., 2022: Song et al., 2020). We set the base channel multiplier to 32 and the per-resolution channel multiplier to [1,2,2,4,4] with self-attention at resolution 16. Additionally, our noisy state x is channel-wise concatenated with an 86-channel conditioning signal, increasing the total number of input channels in the denoiser to 87. The number of output channels remains 1 since we are predicting only single VII./20 channel. However, the increase in the number of parameters minimal since only the first convolutional layer in the denoiser is affected. Therefore, our denoiser is around 12M parameters. The rest of the hyperparameters remain unchanged from Karras et al. (2022).
C.2.4 TRAINING
We adopt the same training hyperparameters from Karras et al. (2022) for training conditional models. Model training is distributed across 4 DGX nodes, each with 8 A100 GPUs, with total batch size of 512. We train all models for a maximum budget 60Mimg.
C.2.5 SAMPLING
For both EDM and t-EDM models, we use the ODE solver presented in Karras et al. (2022). For the t-EDM models, presented in Section 3.5, sampler is the same EDM with the only difference in the sampling
36

---

Published as conference paper ICLR 2025
of initial latents from Student-t distribution instead (See Fig. 2 (Right)). For given input conditioning state, we generate an ensemble of predictions of size 16 by randomly initializing our ODE solver with different random seeds. All other sampling parameters remain unchanged from our unconditional modeling setup (see App. C.1.6).
C.2.6 EVALUATION
Root Mean Square Error (RMSE). standard evaluation metric used to measure the difference between the predicted values and the true values Chai Draxler (2014). In the context of our problem, let a be the true target and be the predicted value. The RMSE is defined as:
RMSE
E [|| +122]
This metric captures the average magnitude of the residuals, i.c., the difference between the predicted and true values. A lower RMSE indicates better model performance, as it suggests the predicted values are closer to the true values on average. RMSE is sensitive to large errors, making it an ideal choice for evaluating models where minimizing large deviations critical.
Continuous Ranked Probability Score (CRPS). is measure used to evaluate probabilistic predictions Wilks (2011). It compares the entire predicted distribution F(#) with the observed data point æ. For a probabilistic forecast with cumulative distribution function (CDF) F, and the true value æ, the CRPS can be formulated as follows:
CRPS( .)) (F(y)-I(y - I(y > x))² dy.
where I(-) is the indicator function. Unlike RMSE, CRPS provides more comprehensive evaluation of both the location and spread of the predicted distribution. A lower CRPS indicates a better match between the forecast distribution and the observed data. It is especially useful for probabilistic models that output distribution rather than a single- point prediction.
Spread-Skill -kill Ratio (SSR). is used to assess over/under-dispersion in probabilistic forecasts. Spread measures the uncertainty in the ensemble forecasts and can be represented by computing the standard deviation of the ensemble members. Skill represents the accuracy of the mean of the ensemble forecasts and can be represented by computing the RMSE between the ensemble mean and the observations.
Scoring Criterion. Since CRPS and SSR metrics are based predicting ensemble forecasts for given input state, we predict an ensemble of size 16 for 4000 samples from the VIL/w20 test set. We then enumerate window sizes of 16 16 across the spatial resolution of the generated sample (128 128). Since the VIL. channel is quite sparse, we filter out windows with maximum value of less than a threshold (1.0 for VIL) and compute the CRPS, SSR, and RMSE metrics for all remaining windows. As an additional caveat, we note that while it is common to roll out trajectories for weather forecasting, in this work, we only predict the target the immediate next time step.
C.2.7 EXTENDED RESULTS ON CONDITIONAL MODELING
Sample Visualization. We visualize samples generated from the t-EDM and t-Flow models for the VII. and w20 channels in Figs. 9 and 10
On t-EDM stability for Autoregressive Rollouts. While we demonstrate the effectiveness t-EDM for the conditional task of predicting the next frame, in practice, it is more useful to perform autoregressive
37

---

Published as conference paper ICLR 2025
Parameters
EDM
EDM
Cskip
aadl
o² dan
dana
w-2 0 Dan
Preconditioner
Cout
Busl
V + o
4-2
a Danal
xー 0² + o
1/ az + o
bー 0² + Jann
Cuoise
b/ llogo
} *log
a s(t)
log a ~ (T Tud)
log o ~ (NTmems F
Training
X(ar
(/)
Loss
Eq. in Karras (2022)
Eq. 13
Solver ODE
Heun's (2nd order)
Heun's 's (2nd order)
Eq. 12, Xr (O, Is)
Eq. 12, xr tz(0, Ia, v)
Sampling
Discretization
N-T o Gmix
(o + N-I Min men
Scaling: s(l)
Schedule: (L)
daata
1.0
1.0
20, x2 (4, 10)
Hyperparameters
Miean Watd
-1.2, 1.2
-1.2,
Tmuxs m7nin
80, 0,002
80, 0.002
NFE P
18
18
7
Table 7: Comparison between design choices and specific hyperparameters between EDM (Karras et al., 2022) and t-DD (Ours, Section 3.5) the Toy dataset analysis Fig. NFE: Number Function Evaluations
rollouts for downstream applications like forecasting. Therefore, we include qualitative results to showcase the stability of (-EDM when performing autoregressive rollouts in Fig. 15
C.3 TOY EXPERIMENTS
Dataset. For the toy illustration in Fig. 1, we work with the Neals Funnel dataset (Neal, 2003), which commonly used in the MCMC literature (Brooks al., 2011) due its challenging geometry. The underlying generative process for Neal's i funnel can be specified as follows:
p(x1 x2) (x 0, 3)A expx)/2).
(176)
For training, we randomly generate 1M samples from the generative process in Eq. 176 and perform z-score normalization as a pre-processing step.
Baselines and Models. For the standard Gaussian diffusion model baseline (2nd column in Fig. 1), we use EDM with standard hyperparameters as presented in Karras et al. (2022). Consequently, for heavy-tailed diffusion models (columns 3-5 in Fig. 1), we use the 1-DD instantiation of our framework as presented in Section 3.5. Since the hyperparameter is key in our framework, we tune y for each individual dimension our toy experiments. We fix v to 20 for the X₃ dimension and vary it between v E (4, 7, 10) to illustrate controllable tail estimation along the x2 dimension.
Denoiser Architecture. For modeling the underlying denoiser Dø(x, σ). we use a simple MLP for all toy models. At the input, concatenate the 2-dimensional noisy state vector with the noise level a. We use two hidden layers of size 64, followed by a linear output layer. This results around 8.5k parameters for the denoiser. We share the same denoiser architecture across all toy models.
38

---

Published as conference paper ICLR 2025
Toy Dataset (Neal's Funnel)
Method
L
4
Baselines
EDM
0,909
2.334
0.234
EDD
0.393
1.584
0.051
Ours
FDM
0.795
3.671
0.194
tED
10
0.863
2.198
0.261
Table 8: Quantitative comparison between EDM and t-EDM on the Neals Funnel 2-d toy dataset. For all metries, lower is better. Values in bold indicate the best results in column.
Training. We optimize the training objective in Eq. 13 for both t-EDM and EDM (See Fig. (Left)). Our training hyperparameters are the same as proposed in Karras et al. (2022). We train all toy models for a fixed duration of 30M samples and choose the last checkpoint for evaluation.
Sampling. For the EDM baseline, we use the ODE solver presented in Karras et al. (2022). For the t-EDM models, as presented in Section 3.5, our sampler is the same as EDM with the only difference in the sampling of initial latents from a Student-t distribution instead (See Fig, 2 (Right)). For visualization of the generated samples Fig. 1, we generate 1M samples for each model.
Overall, our experimental setup for the toy dataset analysis in Fig. 1 is summarized in Table 7.
Quantitative Results Neals Funnel. We present quantitative results the Neals funnel dataset Table 8. t-EDM outperforms EDM on all three metrics, indicating better tail estimation, which also supports our qualitative findings in Fig.
D OPTIMAL NOISE SCHEDULE DESIGN
this section we discuss a strategy for choosing the parameter Fmax (denoted by a in this section for notational brevity) in more principled manner compared to EDM (Karras al., 2022). More specifically, our approach involves directly estimating a from the empirically observed samples which circumvents the need rely on ad-hoe choices of this parameter which can affect downstream sampler performance.
The main idea behind our approach is minimizing the statistical mutual information between datapoints from the underlying data distribution, Xa Pdata, and their noisy counterparts X,, p(Xx). While trivial (and non con-practica)) way to achieve this objective could be to set large enough i.e. a - oc, we instead minimize the mutual information I (xx, x,) while ensuring the magnitude of to be as small as possible. Formally, our objective can be defined as,
min o² subject
[xx,, xx) 0
(177)
As we will discuss later, minimizing this constrained objective provides a more principled way to obtain o from the underlying data statistics for a specific level of mutual information desired by the user. Next, first simplify the form of ((x,, x,), followed by a discussion on the estimation of a in the context of EDM and t-EDM. We also extend the case of non-i.i.d noise.
39

---

Published as conference paper ICLR 2025
Simplification of (xxo xo). The mutual information (xx, x. can be stated and simplified as follows,
1(xo: x,) = (plop xx) || |Ipp px(xo)((xo))
(178)
p(x0= xx) log p(xo, xo)
dxndxa
(179)
p(xo)p(xx)
p(xoo (xo) dx
p(xu: p(xu,xo) x6) log
(180)
p(x₂)
p(xa (xo) xx) log pl x xx0) dx
(181)
x(xa)
Ex x(xx)(x)) S p(xx)10)) log
p(xo (xO) does
(182)
p(xo)
Exoxp(xx
(xi (xo) p(xx))
(183)
D.1 DESIGN FOR EDM
Given the simplification in Eqn. 183, the optimization problem in Eqn. 177 reduces to the following,
min x²
2 subject to (p(x Bxx~ DKL
p(x)))
(184)
Since at mmax we expect the marginal distribution p(x,) to converge to the generative prior (i.e. completely destory the structure of the data), approximate p(x,) (0 With this simplification, the Lagrangian for the optimization problem in Eqn. 177 can be specified as,
arg min 0² + AIEx x(xx)) a²
Xo, o²) || (xxxi00,0)))
(185)
Setting the gradient w.r.t =0=0
X
1
Exp [x] xul =
(186)
which implies,
2² V AExo(xxn xol
(187)
For an empirical dataset,
(188)
This allows us to choose a max from the underlying data statistics during training or sampling. It is worth noting that the choice of the multiplier! À impacts the magnitude of Tmax- However, this parameter can be chosen in a principled manner. At 02 = = the estimate of the mutual information is given by:
Ex [x] Xo
I (X0, xx) Exo x xul =
(189)
o
X
which implies,
A Exp [2n Xo] xol
(190)
(x0, x,) )2
The above result provides a way to choose X. Given the dataset stastistics, Ex,, [x]][xx[x] xo]. the user can specify an acceptable level of mutual information T(xo, x,) compute the correponding X, which can then be used find the corresponding minimum dma required to achieve that level of mutual information. Next, we extend this analysis to t-EDM.
40

---

Published as conference paper ICLR 2025
D.2 EXTENSION T-EDM
In the case of t-EDM, we the optimization problem as follows,
arg min o² + p(xx)) Dy (ta(xxx o² La. ta(0,, 0² v))
(191)
where D4(q p) is the Gamma-power Divergence between two distributions q and p. From the definition of the 7-Power Divergence in Eqn. 8, have,
1 c vd (1+
d - - (o²) शाक -1 'Ex, |xx xo]
(t La.v) l| (0, III, v))
(192)
17
--2
-f(ud))
r( (1z)
where C..d
and 7 utd
Solving the optimization problem yields the following optimal o,
v 2 o [ d) 1- 2 ) xxo xol
2-2+d 2(u-2)+
B B[x
(193)
For an empirical dataset, we have the following simplification,
リ- 2 a² = [ d) 2 (,- d N
Σ |
2)
(194)
D.3 EXTENSION TO CORRELATED GAUSSIAN NOISE
We now extend our formulation for optimal noise schedule design to the case correlated noise in the diffusion perturbation kernel. This is useful, especially for seientific applications where the data energy is distributed quite non-uniformly across the (data) spectrum. Let R Exo p(xx) [xoXo 1 E Rdxd denote the data correlation matrix. Let also consider the perturbation kernel (O Σ) for the postitve- covaraince matrix Σ E Rd Following the steps in equation 185, the Lagrangian for noise covariance estimation be formulated as follows:
min trace Σ)
XExx (xxo)
DKL(N(Xxo Σ) | E))
(195)
min trace Σ) XI ((xo)
p(xo)
Xo Σ '
(196)
Σ
min trace Σ) + AE px(x))
trace E x-1x
(197)
min trace Σ) + À trace Σ EExo~pixol [xxxx
(198)
Σ
min trace Σ) + À Atrace(E-1n)
(199)
It can be shown that the optimal solution to this minimization problem is given by, Σ*
VRR 1/2 where
R 1/2 denotes the matrix square root of R. This implies that the noise energy must be distributed along the singular vectors of the correlaiton matrix, where the energy is proportional the noise singular values. We include the proof below.
Proof. We define the objective:
Σ trace Σ) À tracee 'R)
(200)
41

---

Published as conference paper ICLR 2025
We compute the gradient of f(z) with respect to E:
2
Vxf(E)
trace 3) + A 2r
trace Σ RR).
(201)
The gradient of the first term is straightforward:
trace Σ) I.
(202)
>>
For the second term, using the matrix calculus identity, the gradient is:
2Σ (Atrace(E- 'R)) - 'RE
(203)
Combining these results, the total gradient is:
Vxf(E) I- xΣ RE -
(204)
Setting the gradient zero to find the critical point:
I I- A- -- O.
(205)
E ER'RE-1 I.
(206)
X
which implies,
R X
(207)
Σ = 1/2 1/2
(208)
which completes the proof.
E LOG-LIKELIHOOD FOR T-EDM
Here, we present method estimate the log-likelihood for the generated samples using the ODE solver for t-EDM (see Section 3.5). Our analysis is based on the likelihood computation in continuous-time diffusion models discussed in Song et al. (2020) (Appendix D.2). More specifically, given probabiliy-flow ODE,
dx+ dl
= fo(x,, !)
(209)
with a vector field fe (x,, t), Song et al. (2020) propose to estimate the log-likelihood of the model as follows,
log)(xx) logp(xT) + {
T V fo( ()dt.
(210)
The divergence of the vector field fo(x,, t) is further estimated using the Huitting-Hutchinonn trace estimator (Skilling, 1989; Hutchinson, 1990) as follows,
fo Ep(e)le t)c
(211)
42

---

Published as conference paper ICLR 2025
where usually, NN Ld). For (-EDM ODE, this estimate can be fürther simplified as follows,
fo (x,, t) Ep(e)le V (x(,
(212)
-Bpce) c'v Xt Do(xx.1)) 1)
(213)
Id- Vo
-Eple E"
(214)
[ppee
6 6 E <xx Do Xx. 1)
(215)
- Bp(e)) E' - = Vx, <v, Do (x, 1) d Rp(c) Vx, Do(x,, t) where the data dimensionality. Thus, the log-likelihood can specified as,
(216)
(217)
log p(xo) = logp(xx) + J Hd- d
Ee(cs( VDo(x,,
(218)
i
When e ~ N(O, Ia), the above result can be re-formulated as,
7 d _ " (VD ()6) dt.
1
log p(x)) = p(x)) +
(219)
J0
Moreover, using the first-order taylor series expansion
D&(x + = Dg(x) + Dge O(E(l
(220)
For a sufficiently small a, oigher-order terms in ([]ee can be ignored since E[ = ddd. Therefore,
Do(x c) 2 Dn(x) VDge
(221)
DDe R e [Do + e) - D,(x)]
(222)
E, E V Doe R E, (Do + 6)- Do(x))]
(223)
[ V Doc] E,[ (x c)]
(224)
Therefore, the log-likelihood expression can be further simplified as,
logp(xo) = logp(xx) +
-
1
- Exce (e De t)e) dt
(225)
T 1
l(go(xo) logp(xT) Jo 7. [ - (e Do(x, + E, dt
(226)
The advantage of this simplification is that we don't need to rely on expensive jacobian-vector products in Eq. 218. However, since the denoiser now depends e, monte-carlo approximation of the expectation in the above equation could be computationally expensive for many samples
F DISCUSSION AND LIMITATIONS
F. RELATED WORK
Connections with Denoising Score Matching. For the perturbation kernel ((xx(xu) = ta( Ia the denoising score matching (Vincent, 2011; Song et al., 2020) loss, LosM: can be formulated as,
LUSM K Ex Bx M(x,, v, t) Do(Mxxo + o VR (..o))
σ1) -x,
(227)
43

---

Published as a conference paper at ICLR 2025
with the scaling factor A(x;, t) = [VV + +d/(v+ + di)]²
where d1 (1/of)]x, - ااxo| (proof in App.
A.9). Therefore, the denoising score matching loss in our framework is equivalent to the simplified training objective in Eq. 10 scaled by data- data-dependent coefficient. However, in this work, we do not explore this loss formulation and leave further exploration to future work.
Prior work in Heavy-Taile Generative Modeling. The idea of exploring heavy-tailed priors for modeling heavy-tailed distributions has been explored in several works in the past. More specifically, Jaini et al. (2020) argue that Lipschitz flow map cannot change the tails of the base distribution significantly. Consequently, they use heavy-tailed prior (modeled using Student-t distribution) as the base distribution to learn Tail Adaptive flows (TAFs), which can model the tails more accurately. In this work, we make similar observations where standard diffusion models fail to accurately model the tails of real-world distributions. Consequently, Laszkiewicz et al. (2022) assess the tailedness of each marginal dimension and set the prior accordingly. On a similar note, we note that learning the tail parameter v spatially and across channels can provide greater modeling flexibility for downstream tasks and will be an important direction for future work on this problem. More recently. Kim et al. (2024) introduce heavy-tailed VAEs (Kingma & Welling, 2022: Rezende & Mohamed, 2016) based on minimizing "-power divergences (Eguchi, 2021). This is perhaps the closest connection of our method with prior work since we rely on 7-power divergences to minimize the divergence between heavy-tailed forward and reverse diffusion posteriors. However, VAEs often have scalability issues and tend to produce blurry artifacts (Dosovitskiy & Brox, 2016; Pandey et al., 2022). On the other hand, we work with diffusion models, which are known to scale well to large-scale modeling applications (Pathak et al., 2024; Mardani et al., 2024; Esser et al., 2024; Podell et al., 2023).
Within diffusion models, Deasy et al. (2021) show that denoising score matching can be extended to Generalized Normal distributions and refer to the resulting training as heavy-tailed denoising score matching (HTDSM). Moreover, Deasy et al. (2021) rely annealed Langevin dynamics for sampling. However, an exact formulation of the reverse process is non-trivial and would involve Levy formulations of Kolmogorov's 's forward and reverse equations, which can be quite complex. In contrast, due to the conditional properties of multivariate Student-t distributions, our method circumvents such complex formulations and enables a simple framework for designing samplers for Student-t based diffusion models (see Proposition in the main text). Similarly, Yoon et al. (2023) present framework for modeling heavy-tailed distributions using a-stable Levy processes while Shariatian et al. (2024) simplify the framework proposed in Yoon et al. (2023) and instantiate it for more practical diffusion models like DDPM. In contrast, our work deals with Student-t noise, which in general (with the exceptions of Cauchy and the Gaussian distribution) is not a-stable and, therefore, a distinct category of diffusion models for modeling heavy-tailed distributions. Moreover, prior works like Yoon et al. (2023); Shariatian et al. (2024) rely on empirical evidence from light-tailed variants of small-scale datasets like CIFAR-10 (Krizhevsky, 2009) and their efficacy on actual large-scale scientific datasets like weather datasets remains to be seen.
Prior work in Diffusion Models. Our work is direct extension of standard diffusion models in the literature (Karras et al., 2022; Ho et al., 2020; Song et al., 2020). Moreover, since it only requires a few lines of code change to transition from standard diffusion models to our framework, with popular families of latent diffusion models (Pandey et al., 2022; Rombach et al., 2022) and augmented diffusion models (Dockhorn et al., 2022; Pandey & Mandt, 2023; Singhal et al., 2023). Our work is also related to prior work in diffusion models on a more theoretical level. More specifically, PFGM++ (Xu et al., 2023b) is unique type of generative flow model inspired by electrostatic theory. It treats d-dimensional data as electrical charges in D + d-dimensional space, where the electric field lines define bijection between a heavy-tailed prior and the data distribution. D is hyperparameter controlling the shape of the electric fields that define the generative mapping. In essence, their method can be seen as utilizing a perturbation kernel:
work is directly compatible
p(xx ([)xx xo112 + D)) Aed
txoo of La. D)
When setting v = D, the perturbation kernel becomes equivalent to that of t-EDM, indicating the Student-t perturbation kernel can be interpreted from another physical perspective that of electrostatic fields and
44

---

Published as conference paper ICLR 2025
1.0
df df = 2
df = 3
0.8
df = 5 df
Probability Density
df 20
0.6
0.4
0.2
0.0
10
15
25
30
Figure 8: Illustration of pdf for x" distribution with varying degrees of freedom.
charges. The authors demonstrated that using an intermediate value for D (or v\) leads to improved robustness compared to diffusion models (where D oo), due to the heavy-tailed perturbation kernel.
F.2 LIMITATIONS AND FUTURE WORK
While our proposed framework works well for modeling heavy-tailed data, it is not without its limitations.
On tuning Firstly, while the parameter offers controllability for tail estimation using diffusion models, it also increases the tuning budget by introducing an extra hyperparameter. Moreover, for diverse data channels, tuning per channel could be key good estimation at the tails. This could result in combinatorial explosion with manual tuning. We think there might be several ways to learn v from the data.
Firstly, is worth noting that the parameter v directly influences training in heavy-tailed diffusions. This is because the noising process x, Mę + v/ where (v)v)/v. Since the x² -s-uuree distribution is a special case of Gamma distribution, we can perhaps use reparameterization (Ruiz et al., 2016) to estimate the parameters (i.e. v) of this distribution using neural network to learn this parameter end-to-end. One important caveat learning end-to-end that the optimization process could just select a divergence which reduces the loss while not fitting on the data. Therefore, exploring these caveats for learning y are an important direction for further research.
also estimate from the data in separate stage and use it perturbation kernel parameter. We highlight one simple way to do this in Appendix where we attempt to derive the optimal noise schedule for EDM and (-EDM by minimizing the mutual information between the and the noisy data point X, w.r.t a. In principle, for (-EDM, we can also solve this minimization problem jointly over y and a to learn an initial from the data.
Alternatively,
On Evaluation metrics. Secondly, evaluation protocol relies primarily comparing the statistical properties of samples obtained by flattening the generated or train/test set samples. One disadvantage of this approach is that our current evaluation metrics ignore the structure of the generated samples. On this
45

---

Published as conference paper ICLR 2025
note, it may be tempting to use metrics like FID (Heusel al., 2018), precision/recall (Sajjadi al., 2018) assess the performance between the generated and dataset samples. However, a key caveat in using these metrics the reliance on a pre-trainn pre-trained feature extractor, which is commonly trained on natural images (like ImageNet). However, in this work, we focus on weather data, which has much different structure than natural images. Due to this distribution shift, is unclear if feature extractor pre-trained on natural images can extract meaningful structural features from weather datasets. Therefore, computing metries like FID using a pre-trained network trained on natural images for weather modeling dubious at best. In this spirit, training these feature extractors large-scale weather data using classification self-supervised losses that downstream metrics like FII) can be reported reliably can be an interesting direction for future work.
Applications in other domains. In this work, while explore the application of heavy-tailed diffusion models like t-EDM in the context of weather forecasting, the methods developed in this work are quite generic, and it will be interesting to apply these in the context of other domains like finance, which rely on heavy-tailed modeling of assets. Lastly, our conditional synthesis experiments are limited to the next step prediction and do not perform autoregressive rollouts like the forecasting setup. Extending t-EDM with improved preconditioning and automatic y tuning for forecasting would be an interesting research direction.
46

---

Published as conference paper ICLR 2025
Train Set Samples
t-EDM (v 3) (KS: 0.431)
EDM (KS: 0.997)
Figure 9: Random samples generated from t-EDM (Top Panel) and EDM (Bottom Panel) for the Vertically Integrated
Liquid (VIL) channel. KS: - 2-sample statistic. Samples have been scaled logarithmically for better visualization
47

---

Published as conference paper ICLR 2025
Train Set Samples
t-Flow (v = 3) (KS: 0.711)
Gaussian Flow (KS: 0.897)
Figure 10: Random samples generated from t-Flow (Top Panel) and Gaussian Flow (Bottom Panel) for the Vertically Integrated Liquid (VIL) channel. KS: Kolmogorov-Smimot Smirmov 2-sample statistic. Samples have been scaled logarithmically for better visualization
48

---

Published as conference paper ICLR 2025
Train Set Samples
215
t-EDM (v 3) (KS: 0.683)
EDM (KS: 0.991)
Figure 11: Random samples generated from I-DDM (Top Panel) and EDM (Bottom Panel) for the Vertical Wind Velocity (w20) channel. KS: Kolmogorov-Sminng Smimov 2-sample statistic
49

---

Published as conference paper ICLR 2025
Train Set Samples
t-Flow (v= 7)
Gaussian Flow (KS: 0.294)
Figure 12: Random samples generated from t-Flow (Top Panel) and Gaussian Flow (Bottom Panel) for the Vertical Wind Velocity (w20) channel. KS: Kolmogoroy-Smimov 2-sample statistic
50

---

Published as conference paper ICLR 2025
t-EDM (Channel: VIL)
5:50N (2-3))
EEDN (s
5.0M tu- 71
10'
10
50
Densio
10'
Denstty
20-1
Density
10
t0-
10"7
10-11
10-1
120 Valuee
Viltue
Valuum
107
50
Deritty
1077
Density:
20
Donsitt
10'9
to
10"1
10-1
50
II
Value
Value
Value
t-EDM (Channel: w20)
1.E0% 1o--30
I-EM tu-
5-IDM
EDM
EDM
EDM
Tain Set
Triin Set
Train Set
Dendhy
Dendty
Densty
10
18
-256
-200
-104
-200
-206
-250
-103
-52
54
-256
-000
-250
-150
56
Volue
Value
Value
HERM
5-5DM
CDM
CDM
EDM
Densty
Densty
Donaty
1273
250
200
302
200
250
Volue
Value
Valuuo
Figure 13: 1-d Histogram Comparisons between samples from the generated and the Train/Test set for the Vertically Integrated Liquid (VII., see Top Pandel) and Vertical Wind Velocity (w20, see Bottom Pancl) channels using t-EDM (with varying v).
51

---

Published as conference paper ICLR 2025
t-Flow (Channel: VIL)
0714
1071
Trin
Donsiyy
40-
Denstty
10-1
Donsitv
110
255
900
130
155
200
300
155
209
250
300
Value
Value
Value
10
10'1
10
Donstty
Density
Density
1011-
10
Bijl
359 Saiue
428
100
380
100
300
Valus
Vilrue
t-Flow (Channel: w20)
5-Tlew to-m 5)
5-fisw (+ T)
5-Tlow Eu $)
Flow
Floer
Train bet
10 Se:
train Set
Donoty
Demily
Dendlt
35
-300
10t
5-Poee
10t
t-Flow
5cflow R)
Flow
Test Set
"istt Set
Test Siet
30
30
Densty
Demily
Gondyy
10
30
397
32
11
-300
-003
-100
-200
20:
-000
200
-300
550
-200
200
400
Valuu
Figure 14: 1-d Histogram Comparisons between samples from the generated and the Train/Test set for the Vertically Integrated Liquid (VIL, see Top Pandel) and Vortical Wind Velocity (w20, see Bottom Panel) channels using t-Flow (with varying v).
52

---

Published as conference paper ICLR 2025
Ensemble Mean
Prediction
Prediction
Ensemble (gif)
tn
120
Ground Truth
120
1s
cn
3M0
Ensemble Mean
Prediction
Prediction 2
Ensemble (gif)
Ground Truth
Figure 9: Qualitative visualization of samples generated from our conditional modeling for predicting the next state for the Vertically Integrated I. Liquid (VIL.) channel. The ensemble mean represents the mean of ensemble predictions (16 in our case). Columns 2-3 represent two samples from the ensemble. The last column visualizes animation of all ensemble members (Best viewed in dedicated PDF reader). For each sample, the correspond predictions from EDM, i-EDM 3), I-EDM (v 5) from top to bottom, respectively. Samples have been scaled logarithmically for better visualization
53

---

Published as conference paper ICLR 2025
Ensemble Mean
Prediction
Prediction 2
Ensemble (gif)
Ground Truth
Ensemble Mean
Prediction
Prediction 2
Ensemble (gif)
s3c
Ground Truth
41
Figure Qualitative visualization of samples generated our conditional modeling predicting the next state for the Vertical Wind Velocity (w20) channel. The ensemble mean represents the mean of ensemble predictions (16 in our case). Columns 2-3 represent two samples from the ensemble. The last column visualizes an animation of all ensemble members (Best viewed in dedicated PDF reader). For each sample, the correspond to predictions from EDM, 5) from top to bottom, respectively.
1-EDM (v 3), and (-EDM (v
54

---

Published as conference paper ICLR 2025
Ground Truth
T=4 4
T=
T 9
I-EDM (v 3)
EDM
Figure 15: Qualitative illustration of autoregressive rollouts using 1-EDM (v 3) and EDM on the w20 channel. The conditional diffusion models initialized using HRRR validation data and predictions were made with interval of hour up lead time of hours. We do not observe any instabilities when generating trajectories using t-EDM. represents lead time with r = denoting initial state.
55


Reviews:
[{"review_id": "eUUax4RXdS", "reviewer": "Reviewer_uzb2", "summary": "The authors provide a framework, termed t-EDM, for diffusion models where the added noise is heavy-tailed, following a Student-t distribution. They build upon the EDM introduced by Karras et al 2022. Interestingly, starting with the EDM framework, replacing the Gaussian noise by the Student-t distribution and changing the KL divergence in the objective function to $\\gamma-$Power Divergence for a specific $\\gamma$ leads to same loss function as that of the EDM, except that the noisy state is now drawn from a Student-t distribution. This is of practical interest since their framework can be obtained by only performing minimal changes to previous implementations of EDM. It is of theoretical interest that since the Student-t distribution with $\\nu$ degrees of freedom converges to a Gaussian distribution as $\\nu$ goes to infinity, t-EDM is a generalization of the EDM. The authors provide experiments on weather prediction showing that t-EDM outperforms EDM for both unconditional and conditional generation for certain metrics.", "questions": "Questions\n* Since the preconditioning for EDM helps a good amount for unconditional generation (see Table 2), is not there some preconditioning for EDM in a similar spirit to the unconditional case that could help for the conditional generation case?\n* As mentioned in strengths the t-EDM has a free parameter $\\nu$ that can be tuned to predict more accurately the heavy-tail behavior for different datasets. However, the INC preconditioning for the EDM could be given a parameter by reparameterizing into a normal with variance $\\sigma^2$ instead of a standard normal. (More precisely, this means that in step 4 in Baseline 2 in Appendix C.1.2. we would replace 'standard Normal distribution' to centered normal distribution with variance $\\sigma^2.$) This change in variance is qualitatively different from a change in decay (which is what happens when we vary $\\nu$) so this approach may not be theoretically justified, but it may work in practice. The PCP preconditioning already has parameters. Is there any claim for INC/PCP that can be made about whether tuning this parameters helps or not in capturing different type of heavy-tail distributions?\n\nIn summary, I am wondering if the performance of EDM can be increased only by preconditioning to reach that of t-EDM or if t-EDM does provide a significant difference not achievable by preconditioning.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The t-EDM provides a straightforward and mathematically justified extension for diffusion models to heavy-tailed data. This is to be compared, for instance, to Yoon et al (2023) who also provide a framework for heavy-tailed diffusion models (using Levy processes), but whose implementation requires heavier mathematics and to make the model work in practice they make theoretically unjustified changes to the model. \n\nThe authors do a good job presenting the concepts as natural extensions of what was already known in the field. \n\nThe experiments make the case for t-EDM being useful for weather modeling, an application that requires capturing tails and extreme events. Although the present reviewer is not knowledgable in this application, it is clear that having an extension of EDM to heavy-tails is an important contribution to scientists working on weather forecasting and several other natural science applications. It is especially useful that this implementation requires only minimal changes over previous EDM implementation, so it is quick to implement for practitioners.\n\nOne advantage of the t-EDM is that it has a free parameter $\\nu$ that can be tuned to model different datasets (see results in Table 4), providing a flexible framework useful for practitioners. \n\nYoon, E. B., Park, K., Kim, S., & Lim, S. (2023). Score-based Generative Models with Lévy Processes. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, & S. Levine (Eds.), Advances in Neural Information Processing Systems (Vol. 36, pp. 40694–40707). Curran Associates, Inc.", "weaknesses": "The experiments for unconditional generation in Table 2 show that the EDM (using preconditioning) outperforms the t-EDM for the KR and SR metric in the VIL channel with test data. This hinders the claim the authors make that standard diffusion models, even with preconditioning, fail to capture the heavy-tailed distributions (line 043) and that t-EDM outperforms standard diffusion models. Qualitatively, the plots in Figure 3 do show that t-EDM outperforms EDM with preconditioning in capturing the heavy-tailed behavior. However, I believe a justification of why the KR and SR metrics do not reflect this should be included. As addressed in Appendix F.2, this metrics rely on flattening the data. This may be one reason, but the metric KS also relies on flattening the data but does show that t-EDM outperforms EDM. In any case, I believe a more throrough justification or explanation is needed here.\n\nAlso see questions regarding preconditioning for EDM vs t-EDM. If these questions are successfully addressed I will raise my score.\n\nSmaller comments: \n* in line 198 there is a typo, the equal is in the subscript and it should not.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DOrjoAOPwK", "reviewer": "Reviewer_M1e3", "summary": "This paper adds to the line of work investigating training diffusion models using heavy-tailed noise. Specifically the paper investigates the setting where the increments of noise added in a discrete diffusion model are drawn from a student T distribution.\n\nNew methods are developed for training the diffusion model based on power divergences.", "questions": "- Can the authors comment on the draw backs of using student t noise discussed in \"Heavy-tailed denoising score matching\" and why this makes sense in the context proposed?\n- proposition 2 to me is not well defined. The construction of the stochastic process with student t increments. Can the authors discuss how such a process is constructed in more detail?\n- In the proof of proposition 2 the authors derive the last term converging to Brownian motion, not a student process, can they please comment on this discrepancy.\n- It seems unintuitive to define a diffusion model via its finite increments, only to show it converges to a continuous time diffusion and later make use of this perspective. Can the authors comment on why the did not simply define the noising process as a student-t diffusion in the first place and later discretise this?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The paper introduces a new way of training a score based model with heavy tailed noise, based on the student t distribution, and shows clear benefits in training.\n- The experimental results are reasonably convincing.", "weaknesses": "- The paper fails to discuss previous work, for example \"Heavy-tailed denoising score matching\" covering the subject matter.\n- The paper lacks experiments on synthetic data which is generated as heavy tailed. While not needed, I believe it would help make the case that this method definitively helps with modeling heavy tails by demonstrating it on definitively heavy tailed data.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "i9nl7J4qxe", "reviewer": "Reviewer_zbkB", "summary": "Diffusion models are widely used, but there has been little work studying how well they capture extreme events in the tail of distributions. This is especially important in environmental prediction where extreme events are often the ones that we care most about. This paper shows that standard approaches can fail to capture the extremes accurately. It develops a new form of diffusion by leveraging the multivariate Student-t distribution to form new noising and denoising processes in discrete and continuous time. It develops a method to fit these new processes using a power-divergence which avoids the intractability of the ELBO.  It then applies the new methods to modelling a pair of variables from HRRR, NOAA’s high-resolution local area system for the continental US. This product has recently been a focus of AI diffusion emulators e.g. in NVIDIA’s StormCast.", "questions": "Figure 1 - I liked this! I thought it could be improved by adding axis labels including a description of what the colour map shows (log ground truth density?), calling the original density “ground truth” (\"original hists” in the legend was slightly cryptic),  would add axis labels. \n\nI was a bit confused by the notation around the degree of freedom parameter v. Initially in figure 1 it is a vector, then in equation 2 it is a scalar, but then in the equations on line 155 we switch back to a vector with t_{2d}() being a composition of d independent 2-dimensional multivariate student-t distributions (although this isn’t explained). The issue around clarity here bleeds into the discussion of the divergences where then the divergence parameter \\gamma would depend on the dimension. Balancing notational clarity and correctness is tricky here — perhaps say that \"we’ll consider v to be constant across dimensions for the mathematical development, but the generalisation is simple\" somewhere early on to clarify these issues?\n\nThere is a lack of clarity in my mind around the noising process and the approximations made when arriving at the denoising equation (equation 5). Wouldn’t the true denoising process be non-Markov? If you consider the student-t as a Gaussian scale mixture, I believe the latent scale parameter in your noising construction is effectively shared across all time-steps (a consequence of the equations on line 155). So, when generating x_t the whole history of previous x_{t+1:T} will inform us about the latent scale and so p(x_t | x_{t+1:T}) \\ne  p(x_t | x_{t+1}). Have I misunderstood? Moreover, dropping the data dependent coefficient (v+d_1)/(v+d) is rather glossed over. In the Gaussian setting, dropping this can be somewhat justified by 1. it being an exact choice in the limit of small step sizes, and 2. at least it’s exact when the true distribution of x_0 is Gaussian. In the current case neither of these statements hold I believe. I felt that this should be discussed as the suboptimality of this choice in this case is likely far larger than in the usual case.  Finally, could you clarify how the continuous case fits in with this? I presume, both of the approximations mentioned above are made, and then we take the continuum limit?\n\nConnecting the specific divergence used to a parameters in the model / denoising process (i.e. \\gamma = 2/(v+d)) might be limiting e.g. the objective now can’t be used to learn the degree of freedom, the optimisation objective (8) cannot be used to compare between models that use different settings for v. Also, the correspondence to the traditional objective only then holds as q becomes Gaussian (ideally you’d like to recover the original ELBO without having to limit the model back to the DDPM). I’d have liked to have seen more discussion of this point. \n\nIn table 1, it might be useful to add p(x_T) for the two cases (by the way do you mention somewhere that q(x_T) is a product of student-ts and that you set p(x_T) = q(x_T))?\n\nWhy not consider surface fields, arguably where modelling extremes have most impact? \n\nHave you considered a baseline that trains the EDM and then uses a base distribution for generation which is zero-mean unit-variance student-t, sweeping the degree of freedom parameter to find a suitable one? \n\nFor the HRRR experiments, I presume that you are training off HRRR analysis (rather than, say forecasts) this should be mentioned (I didn’t see it in the main text or in the appendix). Also, is the time interval between inputs mentioned? Is it 1hr like in StormCast? \n\nThe spatial fields shown in the appendix were useful. For the forecasts results shown in figure 8 and 9, it would be useful to show the persistence forecast.\n\nFor the unconditional sampling, it looks like on the test set t-EDM outperforms the baselines 1/3 of the time (and only on the KS metric). \n\nIs the KS metric to be more trusted? Why weren’t the other metrics applied to the forecasting case too? \n\nIt would be interesting to see how t-Flow behaved during auto-regressive rollouts. I could imagine that the under-dispersion of EDM might actually help maintain stability in these cases, whereas heavier tails could lead to greater instability.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "I really liked the problem the paper was looking into — better characterisation of extremes in diffusion and flow based generative models — especially in the context of environmental forecasting.  I think that this is an important practical problem. I also liked the use of a simple example to analyse how well diffusion models capture tail behaviour. (In fact I would have liked to have seen more investigation to toy examples of this sort to pin down the failure models of diffusion more generally.) \n\n I thought the writing was very clear and struck a good balance between communicating technical content and the high level idea. \n\nThe technical contribution is interesting. I liked the fact that the authors provide both the discrete-time version and the continuous-time version of the new approach. The use of the power-divergence for producing a more tractable training objective was creative. \n\nHRRR was a sensible domain to test the new method on. A range of sensible metrics that are used by practitioners e.g. CRPS for the forecasting experiments. I liked the simple modifications to EDM using the inverse CDF transform and per-channel preconditioning.", "weaknesses": "I would have liked to have seen a bit more clarity on the formulation of the denoising process (the sub-optimalities here seem significantly more severe than in the Gaussian case, see questions below for more detail). The use of the power-divergence for training was creative, but connecting the parameters of the model to the divergence parameters also seemed sub-optimal as it does not allow the framework to learn the degree of freedom parameter in the student-t distribution. \n\nThe experimentation was a little light in that only two variables were considered in the conditional and unconditional settings, there were no experiments that considered forecasting multiple variables at one time, or autoregressive rollouts.\n\nSetting the degree of freedom parameter via a parameter sweep is a bit painful. It’s a shame that it can’t be learned directly, especially if it is going to be different for different dimensions of x  (locations / variables / pressure levels etc.).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "guFVzUbxEs", "reviewer": "Reviewer_Nnew", "summary": "The paper presents a novel approach to adapting diffusion models for heavy-tail estimation by leveraging multivariate Student-t distributions. This method addresses the limitations of traditional Gaussian-based diffusion models in capturing rare or extreme events, which is particularly relevant for applications such as weather forecasting. The authors introduce extensions of existing diffusion and flow models, termed t-EDM and t-Flow, which utilize a Student-t prior to enable controllable tail generation through a single scalar hyperparameter. The framework is compatible with standard Gaussian diffusion models and requires minimal code changes. Empirical validation on high-resolution datasets, such as the HRRR weather dataset, demonstrates that the proposed t-EDM outperforms existing diffusion models in accurately modeling heavy-tailed distributions.", "questions": "1.  Can the authors discuss potential applications in other domains where rare events or extreme values are important, such as finance, seismology, or network traffic analysis. This would help demonstrate the broader impact of the method beyond just weather forecasting.\n2. Do you think it possible to extend your framework into other heavy-tailed distributions beyond the Student-t, such as the Cauchy or Lévy distributions, or even unifying all these classes of perturbation kernels? Are there any theoretical or practical challenges you foresee in such extensions?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. The use of multivariate Student-t distributions for diffusion models is innovative, allowing the model to better capture rare or extreme events with controllable tail behavior through a hyperparameter. It is a very brilliant idea. \n2. The overall presentation of this paper is very clear and understandable. \n3. As an extension of previous EDM and Rectified Flow, the authors established a new framework for Student-T distribution and flow, which is able to capture the rare events more effectively. \n4. The authors provide theoretical backing for their approach, linking it to robust statistical estimators and demonstrating its foundation in existing heavy-tailed distribution literature.\n5. Choosing the weather dataset to verify the effectiveness is a great idea, and is novel.", "weaknesses": "I like this paper a lot and I believe there are no major weakness. The only thing is that, there are some duplicated references in the reference list.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5ZqzhSGwCO", "reviewer": "Reviewer_e3KL", "summary": "The paper proposes a novel generative model that, in a sense, mirrors traditional diffusion models (DM) by substituting Gaussian noise in the noising process with noise following a Student’s t-distribution.\n\nMore precisely, the authors introduce a sequence of joint conditional distributions $q_{t-\\Delta t, t}(x_{t-\\Delta t, t} \\,|\\, x_0)$ over $\\mathbb{R}^{2d}$, which are Student distributions with mean $[\\mu_{t - \\Delta t} x_0, \\mu_t x_0]$, covariance matrix $\\Sigma_t$, and degrees of freedom parameter $\\nu > 1$. Leveraging properties of conditional Student distributions, they demonstrate that the marginals of $q_{t-\\Delta t, t}(x_{t-\\Delta t, t} \\,|\\, x_0)$ are also Student-distributed, as are the conditional distributions. Given a family of transitions \n$p_{t-\\Delta t \\, |\\, t}$, the authors fit this family by minimizing a modified DM loss, where the standard Kullback-Leibler divergence is replaced with a power divergence. They then incorporate their chosen conditionals $q_{t-\\Delta t, t}(x_{t-\\Delta t, t} \\,|\\, x_0)$. Finally, the developed methodology is benchmarked against standard EDM and several of its variations for generating dynamical variables in the High-Resolution Rapid Refresh dataset.", "questions": "See weakness", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The approach appears interesting and addresses a relevant problem. Also, the methodology can be straightforwardly implemented using existing DM frameworks.", "weaknesses": "In my opinion, there are several key points the authors should address.\n\nFirst, a major concern is the soundness of the proposed method. In particular, it is unclear to me that the sequence of conditional distributions $ q $ introduced by the authors genuinely defines a Markov noising process. As such, the loss function they employ does not appear to serve as a meaningful metric. More precisely, it remains uncertain why this loss function would serve as a good proxy for measuring discrepancies between the model marginal $p_0^{\\theta}$ and the data distribution. The authors acknowledge that the loss function is not a proper ELBO and claim that the limiting diffusion model is a special case of their method, but further justification of the loss function is needed.\n\nMy second concern relates to the results and mathematical formalism of the paper. The mathematical treatment seems quite weak. For instance, the proof in the first section of the appendix contains several typos and appears incorrect in its current form. The reference to Ding (2016) would be sufficient, as it already includes the result. I also found Proposition 2 challenging to interpret, and the accompanying proof does not clarify matters. For example, it is unclear what $f$ and $g$ represent, whether they are any functions satisfying the proposition's conditions, and whether such functions exist. Similarly, the term $ dS_t \\sim t_d(0, dt, \\nu + d)$ lacks clarity. I am also unclear on the main conclusions to be drawn from Proposition 2.\n\nMy third concern is related to the experiments and the metrics used. First, there is a large gap in performance when using $\\nu = 3$ versus $\\nu = 5$, raising questions about the robustness and reliability of the method. This gap suggests that the chosen metrics (KR, KS, SR) might be sensitive only to the tails of the target distributions. Additional metrics such as precision, recall, and density coverage could provide a more balanced assessment. Moreover, the claims regarding t-EDM's performance appear somewhat overstated, given the tables and the likely margin of error. In my opinion, the experiments should include toy examples for a deeper comparison with existing diffusion models.\n\nFinally, the authors should provide a comprehensive literature review on extensions of diffusion models, acknowledging previous relevant works. The current literature review is too limited.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aO0V0BeoCj", "reviewer": "Reviewer_EPoh", "summary": "This paper proposes a method for adapting denoising diffusion models to sample from heavy tailed distributions. The core idea is to replace the Gaussian prior and transition distributions from standard DDPM with Student-t distributions. The authors show how to construct a heavy-tailed diffusion model by defining the relevant noising process in terms of the Student-t distribution, calculating the corresponding reverse noising process, showing how it can be appropriately parameterized in terms of a prediction of $x_0$, and deriving a training objective defined in terms of the $\\gamma$-Power Divergence by analogy to the standard KL objective used to train diffusion models. They then demonstrate the empirical success of their method on distributions from the HRRR dataset, showing that it outperforms Gaussian-based diffusion models in most settings they study.", "questions": "- How do we tune the parameter $\\nu$ in practice? It looks from Table 3 like the choice matters a lot, so it might be nice to say something about how one should choose a suitable value of $\\nu$ or know that one has found a good value.\n- How confident are you that the 1-dimensional summary statistics you are looking in Table 2 (and maybe Table 3 also) at appropriate? From looking at Figure 3, t-EDM seems clearly qualitatively superior to the other methods. The fact that this is not reflected in the summary statistics in Table 2 makes me suspect the summary statistics are not capturing the most relevant aspects of the distributions.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "My overall feeling is that this is a very technically solid paper which takes a relatively simple core idea - exchanging the Gaussian distributions in diffusion models for Student-t distributions in order to have a heavy-tailed sampling distribution - and executes on it methodically and very well. The presentation and flow of the argument is clear throughout, and all the obvious questions (parameterization of the reverse process, tractable training objective, continuous-time sampling with SDEs and ODEs, empirical performance) are dealt with comprehensively in my opinion.\n\nThe problem of sampling from heavy-tailed distributions is one of significant interest to the field, and I'm not aware of other work providing such a nice and integrated framework for heavy-tailed diffusion modeling, so I consider the contribution of an appropriate size for the conference.\n\nParticular highlights were the connection of the diffusion modeling loss to the $\\gamma$-Power Divergence, showing that this choice of divergence makes the loss tractable and reduces to a denoising $L^2$ loss similar to that of DDPM, and the discussion in Section 5 on various theoretical intuitions and insights the authors have concerning their framework.", "weaknesses": "I do not think this paper has any major weaknesses. The addition I would be most keen to see is a more thorough discussion of the results in Tables 2 and 3 (empirical results in the unconditional setting). For example, it would appear from Table that in the test setting for VIL, their method actually underperforms compared to EDM (+PCP) and compared to EDM (+INC) in the other setting, and the authors do not discuss this in the text as far as I can tell. (Though, Figure 3 tells a somewhat different story, and I can't work out how to reconcile the two - see question below.) The authors also don't really discuss the results in Table 3 in the main body, and a short commend about what we should take from those results might be appropriate.\n\nSecondly, I also feel that many of the derivations in the work are somewhat routine once one has the core idea of using Student-t distributions to parameterize the forward process, and the novelty of the work is therefore not outstanding. Nevertheless, the problem is of sufficient interest and the execution of the authors is sufficiently good that I believe this still clearly merits acceptance to the conference.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)