Risk-Sensitive Diffusion: Robustly Optimizing Diffusion Models with Noisy Samples
Authors: Yangming Li
Max Ruiz Luyten
Mihaela van der Schaar
Venue: ICLR 2025 Poster (2025)
Keywords: Diffusion Models
Noisy Data
Risk-Sensitive SDE
Generative Models
Primary area: generative models
TL;DR: We introduce Risk-Sensitive SDE, a framework that generalizes current diffusion models to learn the underlying distribution from noisy data optimally.

Abstract: Diffusion models are mainly studied on image data. However, non-image data (e.g., tabular data) are also prevalent in real applications and tend to be noisy due to some inevitable factors in the stage of data collection, degrading the generation quality of diffusion models. In this paper, we consider a novel problem setting where every collected sample is paired with a vector indicating the data quality: risk vector. This setting applies to many scenarios involving noisy data and we propose risk-sensitive SDE, a type of stochastic differential equation (SDE) parameterized by the risk vector, to address it. With some proper coefficients, risk-sensitive SDE can minimize the negative effect of noisy samples on the optimization of diffusion models. We conduct systematic studies for both Gaussian and non-Gaussian noise distributions, providing analytical forms of risk-sensitive SDE. To verify the effectiveness of our method, we have conducted extensive experiments on multiple tabular and time-series datasets, showing that risk-sensitive SDE permits a robust optimization of diffusion models with noisy samples and significantly outperforms previous baselines.

Published as a conference paper at ICLR 2025
RISK-SENSITIVE DIFFUSION: ROBUSTLY OPTIMIZING DIFFUSION MODELS WITH NOISY SAMPLES
Yangming Li, Max Ruiz Luyten, Mihaela van der Schaar Department of Applied Mathematics and Theoretical Physics University of Cambridge y18 11877 cam. uc
ABSTRACT
Diffusion models are mainly studied on image data. However, non-image data (e.g., tabular data) also prevalent in real applications and tend be noisy due to some inevitable factors in the stage of data collection, degrading the generation quality of diffusion models. In this paper, we consider novel problem setting where every collected sample is paired with a vector indicating the data qual- ity: risk vector. This setting applies to many scenarios involving noisy data and we propose risk-sensitive SDE, a type of stochastic differential equation (SDE) parameterized by the risk vector, to address it. With some proper coefficients, risk-sensitive SDE can minimize the negatíve effect of noisy samples on the op- timization of diffusion models. We conduct systematic studies for both Gaussian and non- nonn-Gassiian noise distributions, providing analytical forms of risk-sensitive SDE. To verify the effectiveness of our method, we have conducted extensive ex- periments on multiple tabular and time-series datasets, showing that risk iiskssenittive SDE permits a robust optimization of diffusion models with noisy samples and significantly outperforms previous baselines.
INTRODUCTION
Prevalence of noisy non-image data. Current studies on diffusion models (Sohl-Dickstell et al.
2013 Ho al. 2020) (or score-based generative models (Song & Ermon 2019 Song et al.) 2021)
have primarily focused on high-quality image data, achieving promising performance (Dhariwal & Nichol 202T) in image synthesis. However, non-image data (e.g., tabular data and time series) in fact more popular in real applications (c.g.. medicine (Johnson et al., 2016) and finance (Takabashi al. 2019)). A survey conducted by Kaggle (Kaggle 2017| van Breugel et al., (2023) revealed that 79% of the data scientists are mainly working on tabular data. Importantly, while image datasets are commonly of high quality, non-image data contain noisy samples in most cases. For example, sensor data susceptible measurement errors (Steinvall & Chevalier 2005), and such noise can significantly degrade the performance of diffusion models.
Introduction of risk vectors. In this work, are interested in novel problem setup where every sample in the dataset is associated with a vector indicating the sample quality: risk vector. The robustly
purpose of setting this vector is to provide information that potential method can optimize diffusion models in the presence of noisy samples. While this setup might seem artificial, it applies many real scenarios involving noisy data. For example, tabular data often contain missing values (Barnard & Meng 1999), and practitioners typically impute those values before using the data. During this preprocessing step, many imputation methods can provide confidence values (i.e., risk information) for their predictions. Even when such risk vectors are not directly accessible, a class of methods known as uncertainty quantification (Angelopoulos & Bates] (021) can offer viable alternatives. In Appendix D we provide a detailed discussion and more real-world examples, showing the broad applicability of our proposed setup.
Principled method: risk-sensitive diffusion. To address the problem setup: noisy samples paired with risk vectors, we first study the negative impact of noisy samples on optimizing diffusion mod- els, with conclusion that such samples mainly cause marginal distribution shift in the diffusion

---

Published as a conference paper at ICLR 2025
process. In light of this finding, we introduce quantifies the negative effect of noisy samples, and propose seskiseni SDE, a type of stochastic differential equation (SDE) parameterized by the risk vector, with the aim to minimize the instability measure. For both Gaussian and general non-Gaussian noise perturbation, we determine the optimal coefficients of risk-sensitive SDE, and prove that, in the case of isotropic Gaussian noises, that type of negative impact can be fully eliminated. In experiments, we show that our method is still very effective in the scenario with non-Gaussian (e.g., Cauchy) noises.
measure called perturbation instability, which
Contributions. In summary, the contributions of this paper are as follows:
Conceptually, we are the first to introduce risk vectors to robustly optimize diffusion models with noisy samples, with principled method: risk-sensitive SDE, to incorporate such vector, reducing the negative impact of noisy samples: perturbation instability;
Technically, we solve the analytical forms of risk-sensitive SDE for both Gaussian and non-Gaussian noise distributions, with a notable conclusion that, in the case of Gaussian perturbation, the negative impact noisy samples can be fully reduced;
Empirically, experiment results on multiple tabular and time-series datasets show that risk- sensitive SDE can effectively handle noisy samples, even when the noise distribution is mis-specified or non- non-Gaussian, and notably outperform previous baselines.
We have publicly released the code at https:/ggthha..com//eeelsssssi/e
2 PRELIMINARIES
In this section, we first briefly introduce the background of diffusion models, with basic terminolo- gies and notations that will also be used later. Then, we present the motivation and formulation of problem setup: noisy samples paired with risk vector.
2.1 BACKGROUND OF DIFFUSION MODELS
While diffusion modes (or score-based generative models) have different versions and variants, we adopt the formulation of [Sogg et al. (20TT). which generalizes DDPM (Ho et al., 2020 SMLD Song & Ermon 2019), VDM (Dhariwal & Nichol 2021), etc.
At the core of diffusion models lies a diffusion process, which drives data samples x(0) Po (x))
(i.e., finite-dimeniional vector) towards noise x(7) ~ (x()( at time T E IRt, and can be expressed through stochastic differential equation (SDE) (Itō) 1944):
dx( [(()x(()dt + 9(()dw((),
(I)
where w(t) is standard Wiener process, [(()x(t)= R x Rd function that specifies the drift coefficient, and g({) R R is predetermined scalar-valued function that specifies the diffusion coefficient. We call pr, T oo the prior distribution, which is fixed and retains no information of po via proper design of coefficients f(t), gt)..
TRd is a predefined vector-valued
Interestingly, the reverse process (i.e., reverse version of the diffusion process) also follows an SDE. For process of the form as Eq. @ shapes as:
dx(()) (f())(()= - V x(e)
V (x))() + g(t)dw())
which runs another standard Wiener process w(t) backward in time. For generative purposes, we can sample randomly from the prior distribution Pr and use the reverse process to map such samples into data samples that will follow po. that is, the distribution of inputs. The challenge is to determine the expression Vx In (x() in the backward process (known as the score function) since the term is analytically intractable in most cases. A common practice (Song & Ermon 2019) is to use an approximation called the score-based model Sø so(x, t), for instance, a neural network.
To optimize the score model towards the score function, previous works (Song & Ermon 2019
Song et al. 2021 derived the following score-matching loss:
L (((....).)) [A(1) (0) (x()),t) Vx(o In (x)))))))
2

---

Published as a conference paper at ICLR 2025
where the weight X(t) [0, R* is generally set uniformly. Importantly, it is common (Song et al. 2021) to adopt an upper bound of the above loss to fit the model so sx(((t), with the kernel Pojol x(0)) (i.e., the density of x(t) conditioning x(O)).
2.2 PROBLEM SETUP
For standard diffusion models, the observed sample x(0) R is implicitly assumed be without noise perturbation. However, this simplification does not apply to many real applications. Fig. T shows an example of medical time series, which consists of irregularly spaced observations. To apply diffusion models to such data, one will first fill in the missing values with some interpolation method (Rubanova et al. 2019), resulting in noises in the form of interpolation errors.
Misguidance effect of Noisy Samples. Noisy sample x(0) intuitively has negative impact the optimization of score-based model (x, t), degrading the generation quality of diffusion models. For this point, solid explanation is below.
Remark 2.1. In the standard case with only clean sample x(0), the score- basee model sø(x, (x, t) is optimized to match the score function Vx In (x). Since noisy sample x(0) has different initial distribution Po (x) from that (x()) of clean sample x(), their marginal distributions (x), pe (x) at time step will also be different, with the same diffusion process (i.e., Eq. ()) As result, we have Vx In Px(x) * Vx In (x), indicating that noisy sample x0) causes a wrong objective Vx In (x) for optimizing the model So (x(t),t)
In short, we say that noisy sample x() misleads the diffusion models in training.
Introducing risk information. Although noisy samples are in- escapable in some situations, they are usually with additional infor- mation, estimating the potential risk of using such samples. Fol- lowing the previous example, a Gaussian process (MacKay et al.) 1998) that interpolates the missing samples in Fig- naturally pro- vides uncertainty information (i.e., confidence intervals) for each prediction. We can thus pair every possibly noisy sample x(0) (0), (0) [0)] with its risk information r, available for free. While risk defined in very general way in Defini- tion 3.1. its concrete form depends on the noise type. For example, the case of non-isotropic Gaussian perturbation, the risk vector r [T)(0) (0), 72 (0). [O)] of the same D dimensions as the sample x0). indicating its entry-wise data quality. The closer to the value in each entry т; E IR U(0} of r is, the higher the ex- pected quality, where r; = 0 indicates that entry (0) is clean.
Normatood value
Timestamp (hr)
Figure I: A segment of noisy time series from MIMIC (Johnson et al. 2016). The data points out- side the orange region (i.e., 95% confidence intervals) are observed, and a Gaussian process interpolates the ones within the area.
Provided with the risk vector draw information from both risky (x(0), * 0) and clean sam- ples (x(0) x(0), 0), and importantly, this model only optimized towards the distribution of clean samples: Po(x).
ideal generative model could
3 METHOD: SES-SENSITIVE DIFFUSION
To reduce the misguidance of noisy samples optimizing diffu- sion models, we present principled method: risk-sensitive SDE, a type of SDE parameterized by risk vector r. In the following, first define risk-sensitive SDE and some other useful concepts, and then solve its optimal coefficients for different noise perturbations. Finally, we present the training and inference algorithms for diffusion models under the framework of risk-sensitive SDE,
3.1 BASIC DEFINITIONS
Risk vectors and noise distribution families. Intuitively, risk information represents the data quality of a noisy sample x()). To formalize this concept in a more rigorous way, we provide the below definition, which still aligns with the intuition.

---

Published as a conference paper at ICLR 2025
Definition 3.1 (Risk Vectors). The risk information shapes as a vector that is element blement-wise non- negative and controls a family of continuous noise distributions:
P. pr((e) : IR - RR,, (r(c)dde - * 0):
(4)
with each one perturbing clean sample x(0) ~ po (0)) into noisy sample x(0) x(0) + c, which pol (x())))((((()) x(0))dx(0). For zero risk
is with respect to a distribution as Po. (x())
r = O, means the sample (0) = x(0) is nrese-free.
Remark 3.1. This definition might not seem intuitive. For better understanding, let us take isotropic Gaussian perturbation as an example. In this case, the risk vector be simplified as scalar and the family of noise distributions Pe is as {N' rI) 00. 0).
Remark 3.2. The operation of noise perturbation can be regarded as a form of "local averaging", which is typically not reversible. Even suppose that the reverse operation was possible, recovering the potential clean sample x(O) from noisy sample x(), would require knowledge of the probabilis- tic densities of samples, which are not accessible in practice.
Motivation and definition of risk-sensitive SDE. In light of the misguidance effect of noisy sam- ple x(0), we aim to seek an alternative diffusion process parameterized by the risk r, such that noisy sample (x(0), r) under this process has the same distribution (x) at some iteration in [0, T] as that of clean sample x(0) under the ordinary diffusion process: ps (x). For iteration t where the equality (x) (x) holds, the score function of noisy samples: V. In (x), can be used to safely optimize model (x, t). The new process chosen in this spirit is a specific choice of SDE whose parameterization includes the risk vector r. We name such an SDE as risk-sensitive SDE, with a strict definition as follows.
Definition 3.2 (Risk-sensitive SDE). For a noisy sample x(0) with risk vector r, the risk-sensitive SDE is type of SDE that incorporates the risk r into its coefficients, extending sample vector x(0) into a dynamics (x(t))<(O)T] as
(ix (f(r,t) (()) + g(r,t) dw((),
(5)
where O stands for the Hadamard product, and the coefficient functions f(r,t), g(rg t) are every- where continuous with right derivatives.
Remark 3.3. For zero risk r = O, the above SDE is fed with clean sample x(0), and thus corre- sponds to standard diffusion model with uisk-er coefficients t), g(0, t). We refer to this particular case as mask-uawaaeeaa SDE.
Remark 3.4. One might notice that risk-sensitive SDE is more expressive than the ordinarily defined diffusion process (i.e.. Eq. (I): The risk-sensitive coefficients f(r, t), g(r, are vectors (i.e. non- isotropic), while risk-unaware coefficients f(t), g(t) are just scalars. In Theorem 3.2 we will see this setting is essential for non-isotropic perturbation.
Error measure: perturbation instability. As previously discussed, we aim to find type of risk- sensitive SDE that satisfies nice property at some time step I: -x. = pr (x), which we define as perturbation stability. While this condition is indeed possible to reach for Gaussian noises, we will see in Theorem 3.1 that is not achievable in the case of non-Gaussian perturbation. Therefore, we have to introduce new "criterion" that generalize the stability condition, measuring how much it is violated. With this type of criterion, we can score all the coefficient candidates of a risk-sensitive SDE and search for the best candidate, which minimizes the stability violation.
Because probability densities are uniquely determined by their cumulant- cumullant-generatingg functions (i.c., log-characteristies functions) (Chung. 2001), obvious way to define the criterion is measure the mean square error (Weisberg 2005) between the cumulant- that of Pt (x). A formal definition is the following.
function of pt, and
Definition 3.3 (Measure of Perturbation Instability). For given risk vector and time step the perturbation instability S, of a risk-sensitive SDE (as defined in Eq, (5)) measures the discrep- ancy between its marginal density (x) for a noisy sample x(D) and that of the ordinary diffusion process (x() for a clean sample x(0) as:
S((r) sup (S px(
Mxy ((.))/ - x((r))(ds)
(6)
4

---

Published as a conference paper at ICLR 2025
where (yy) : IR R$ is positive weight function and the complex modulus. In par- ticular, (y) Xx(y) respectively stand for the cumulant- -eeneratnng functions (Chung 2001 of (t.r(x),, Px()), which both depends on the distribution of real samples: Po(x).
say risk-sensitive SDE achieves perturbation stability at time step if and only it also satisfies (r(r) = 0. The forward direction of this claim is obvious and the reverse is proved in the appendix: Lemma [.I] Importantly, we will see in the next section that such stability is not always reachable. In that case, we say a risk-sensitive SDE, which achieves the infimum of (r)) has the property of minimum instability.
Remark 3.5. Extending
terminology,
Remark 3.6. The significance of perturbation stability is that, when this property holds, then the desired equality Vx In (x) Vx In pt (x) will also hold. In this situation, the score-based model sø(x, t) can be robustly optimized with noisy samples (x(0), 0).
One might adopt another way define the instability measure, considering that there are many other methods (e.g., KI. divergence (Shlens 2014)) to quantify the discrepancy of two probability distributions. However, we find that our defined measure S2(r) leads meaningful theoretical results and performs well in experiments. We remain the explorations of other possible measures and their implications for future work.
3.2 MAIN THEORY
In this part, we aim answer the following three questions:
1. In what conditions is there a risk-sensitive SDE that facilitates perturbation stability? For example, does this depend on specific noise types or sample distribution Po(x)?
2. If the stability property is not reachable, is there a possibility to have a analytical solution that minimally violates the stability property?
3. In the above two situations, what are the actual forms of risk-sensitive SDE? Is it general- izable to extend the current diffusion models for application?
To improve readability, we present simplified theoretical results while preserving the key ideas. The complete theory and detailed proofs can be found in Appendices G
Answer to the 1st question. Our theorems provide satisfactory answer as follows.
Theorem 3.1 (Simplified and Reinterpreted from Theorem E.I and Proposition F.I.. The necessary and sufficient conditions for risk-sensisitve SDE to achieve perturbation stability: Pt,x(x) = (x)),, include: 1) the noisy sample x(0) is perturbed by a diagonal Gaussian noise and the risk indicates its variance; 2) the time step is within the stability interval (rr).
In particular, suppose the Gaussian noise isotropic, then it suffices to represent the risk vector as a scalar I and the form of sensitvne SDE under this condition is as
f(r.t)= = dln dl)u(()
Vt Vt [0,T]
dt
g(r,1) =r()) d (ol..09))
di ).VLE VVLE T(r), g(r. VLe T(r)
where u(t), t(r,t) are continuous functions with right derivatives, satisfying v(r,t)2 = max(v(0, t)2 (u((t)),))
(8)
and T(r) E [0, ((r,,)= > 0} is defined the stability interval. For zero risk 0, the above equations reduce to ordinary risk-unaware diffusion model.
We can see that the ideal situation with perturbation stability reachable if and only if the noise distribution is Gaussian and the time step is within the stability interval. This conclusion is also very intuitive from two perspectives: Firstly, since the backbones of risk-sensitive SDE and diffusion model are in fact drifted Brownian motion, it is not likely that our tool can reduce the impact of noise distribution beyond Gaussian; Secondly, noisy sample x(0) is surely less informative than the clean sample x(0), it is reasonable that noisy samples cannot be used correctly optimize the basee model Sp (x, t) at every time step t. In Theorem E. of the appendix, one can also find more general conclusion for non-isotropic Gaussian noises.
5

---

Published as a conference paper at ICLR 2025
Algorithm 1 Training
A Algorithm 2 Sampling
1: repeat
1: time points (far T, 11--1-- 12,11
2: Sample (R((). from the training set
2: Set zero risk r = und (tax))tr)) ~ (x)
Sample é from nability interval ((r)
3: for M. M 1.....2do
4: 7 ())
4: (g(x((s))
g(r,t,) 3 (
5: &(1) u(() xx(0)) + v(r, Q) 7
5: b(x(t;;; f(r,%)) (x)())))))
Updaze 0 with V v(r.t) 2i (R(0).))
6: " NN(, 4 1)1)
7: converged.
x(t,-1) x(t;) b(-)(t - 14-1)
gir.t) On
8: for
Answer to the 3rd question for Gaussian noises. We have the following corollary that extends VP SDE (Song et al.) (021) (i.e., the continuous relaxation of DDPM) to risk-sensifive VP SDE, which supports a robust optimization with isotropic Gaussian noises.
Under the setting of isotropic Gaussian perturbation, the risk-sensitive SDE VP SDE is parameterized as follows
Corollary 3.1 (Risk-sensitiv VP SDE, Simplified from Corollary
1 3())) g(r,t) = 1(1 (1+ ()+)))((tt)))))))))))))))))))))) B(t).
f(r,t)
(9)
where 1(-) is an indicator function and the coefficient a(t) is defined as a(t) exp(- ((s)ds).
The stability interval in this case Trr) {t [0, T] a(t)-1 > 1 r2). As expected, for the special case with zero risk O, the sens-iensitvee SDE reduces to an ordinary VP SDE, with f = - ((t), g(0,t) f) = v B()). and T = [0.
Risk-sensitive VP SDE is the same as vanilla VP SDE for optimization with clean sample (x(0), 0), otherwise will adopt a different coefficient g(r, t) and a restricted set of sampling time steps T(r) to reduce the negative impact of noisy sample (x(0), > 0). We will discuss this point more in the next section, with detailed optimization and sampling algorithms. Corollary G.2 in the appendix also provides its version for non-isotropic Gaussian noises.
Answer to the 2nd question. This question is very important, considering that the perturbation distributions in the real world might be non- non-Gaussian. As shown below, we can always find the optimal parameterization of risk-sensitive SDE that minimizes the negative impact of an arbitrarily complex noise distribution on the optimization of model so (x, L).
Theorem 3.2 (General Stability Theory, Simplified from Theorem F.T. Suppose the risk vector r is element-wist positive and controls a family of continuous noise distributions, with each one formulated as Pr(E) .RD - R.., pr(()dd = 1, then the optimal coefficients for the resk-sensitive SDE to minimize the instability measure (x)r) satisfy the following equality:
v(r,t)² = .max ((0, + ((((t),))
((((()))= = (000===((())))))
((y)yy J'dy) -1 (s ((y)) In
(10)
ln|exp exp ((((())))))ll
where the vectorized coefficients u(t), v(r, come from the formal definition of risk-sensitive SDE (i.e., Definition 3.2) and the new terms x,())))()) are basic elements that defines the instability
measure S((r) (i.e., Definition 3.3.
We can see that the general form of perturbation distribution (r(E) incurs very complex expres- sion & (u(t), r) in the optimal coefficient v(r, t). In particular, if the noise e follows an isotropic Gaussian (E) = N(E;O,r1L then we can verify that (u((),r) = r²u(t)² regardless of the weight function P(y), which is consistent with our previous conclusion: Theorem 3.1 Another complication of non-Gaussian noise is that: even for some isotropic noise distribution (e). the term V ((((). r) appearing in coefficient v(r. t) might have different expressions in different dimen- sions. Therefore, distinct from our Theorem 3.1 the vectorized coefficients u(t), v(r, t) cannot be simplified into scalar functions (e.g., u(t), v f)) for isotropic noise perturbation.

---

Published as a conference paper at ICLR 2025
Clan PSSt W1 301
Sampres with Mic por.atttr S0E
0.25 IDifeemnt Ceralies)
Time: csame Dana Best
(DePeren Denalies)
isame Best
Isame Denailiesi
Figure 2: Comparison between the diffusion process of standard VP SDE for clean samples (i.e., the upper 5 subfigures) and its alternative: risk-sensitive SDE, for Gaussian-corrupted samples (i.e., the lower 5 subfigures). With the proper risk-sensitive coefficients, the clean and noisy samples will have the same marginal densities in the stability interval: e [0.26, 1].
Answer to the 3rd question for non-Gaussian noises. To finally answer this question for non- Gaussian noises, we have the below corollary that extends VE SDE (Song et al.) (02I] (a type of diffusion model also used in [Song et (2023)) to risk-sensitivv VE SDE, which supports robust optimization with Cauchy noises.
For some properly de- fined weight function 12(y) and an isotropic Cauchy perturbation specified by scale r as (r(c) = -[]=1(aT( (f/r))= c 1c, ",,,,,,"", the minimally-unstabbe risk-sensitive SDE for VE SDE has coefficients as
Corollary 3.2 (Risk-sensitive VE SDE, Simplified from Corollary G.3
f(r, 0, 9(r,1) (0(1)² -0))
D D 2
dt dt
(11)
D D+5
Notably, for the setting with no risk r = risk-sensitive VE SDE reduces to the ordinary risk- unaware SDE, which has fixed coefficients f(), = 0, (0, = (I3
With heavy tail in the distribution, Cauchy noise has high probability drift clean sample far away, exhibiting a very distinct behavior from Gaussian noises. In Sec. our numerical experiments (e.g., Fig. 4 show that risk-sensitive VE SDE rarely generates outliers, indicating that the optimal risk-sensitive SDE is very effective in reducing the negative impact of Cauchy-corrupte samples. Corollary G.3 in the appendix stands as its version for non-isotropic Cauchy noises.
3.3 OPTIMIZATION AND SAMPLING
Similar to the score matching loss of standard diffusion models, the loss function under the frame- work of risk-sensitive SDD shapes as
Ler Ex- [|] [x, 0) V (x)
(12)
Proposition G. in Appendix G shows that this loss function for noisy sample (x, is equal to the
score matching loss for clean sample (x, 0) within the stability interval T(r), and has another form for computation in practice.
Proposition 3.1 (Risk-free Loss, Simplified from Proposition G.1
The loss function Li, for risky
sample (x(0), r 0) is equivalent to the below expression:
(x) m - v(r,)) + sa(u(() C x(0) v(r, O 0)13]
(13)
up to constant. Here u(t), v(r t) are vectorized versions terms u(t), v(r, 1) that appear in Eq. 77. with their formal defimitions in Theorem
We respectively show the training and sampling procedures Algorithm [ Algorithm 2 We also highlight in blue the terms that differ from vanilla diffusion models. For the optimization algorithm, when r = 0, the algorithm reduces to the optimization procedure of a vanilla diffusion model, with trivial stability interval of T(r) [0, 7]. When the random variable is non-zero, the risk-sensitive

---

Published as a conference paper at ICLR 2025
aubben 2000
3600
number: LUoe
number: 503
nurbber: 1932
1932
1o
-50
-20
66
15
45 40 05
(a) Training data, polluted (b) Samples from stan- (c) Samples from the risk- (d) Samples from our risk- by Gaussian noises.
dard diffusion model. conditional bascline.
sensitive VP SDE.
Figure 3: Comparison Gaussian mixture data (Fig.
(3a) three-sigma regions as ellipses), with
part of Gaussian-cor Cooustia-corrupted samples. Our model (Fig. ((d) mostly samples within the ellipses, while
the samples from standard diffusion model (Fig. ((b) typically fall out of them, and conditional generation leads to an unbalanced generation distribution (Fig. ((c)).
coefficient v(r, t) and interval T(r) will guarantce that Vx IxInp:(>x) = Vx In (x.)) for E T(r), such that the noisy sample (x(0), 0) can be used to safely train the model sx t).
For the sampling algorithm, by setting zero risk r = O, the coefficients f(r, ())gg)rr)) t) become compatible with the model sø(x.t) t) and together generate high-quality sample x(0). Our model will generate only clean samples (x(0), r = 0), but it was already able to capture the rich distribution information contained in noisy sample (x(0), of 0) during optimization.
4 RELATED WORK
Similar setups. To our knowledge, are the first study the problem setup of pairing noisy samples with risk vectors in the field of diffusion models. Some previous works (Ouyang et al, 2023 Kim et al., 2024) also focused diffusion models with noisy data, though under different settings. For example, Unbiased Diffusion Model (Kim et al.) 2024)) considered the presence of both biased dataset and clean dataset and, thus, tackled particular case of our setting: assigning risk 1 to the samples of biased dataset and risk O to those of the clean dataset. However, this model cannot be adapted the common situation where different noisy samples might have different risks. Ambient Diffusion (Daras et al.) 2024) aimed to handle situation where the images are with missing pixel patches, which largely differs from our setting. As discussed in Appendix [ missingness also typical use case of our method: risk-sensitive SDE. Another related work is Na et al. (2024). which considered noisy labels, instead of noisy samples.
Potential risk-conditional baseline. Our proposed risk-sensitive SDE is the first method to ad- dress the problem setup of this paper. An alternative way to adopt conditional diffusion mod- els (Dhariwal & Nichol, 2021: Ho & Salimans 2021]. though there is surely no such work in the literature. One can treat the risk vector as that "conditional information" and apply these techniques to guide diffusion models to generate low-risk samples. We name this method as risk-conditional baseline this paper and provide three different implementations in Appendix [A] The main prob- lem with cinniconditional diffusion models is that it might lead to biased sampling distribution. To understand this point, note that conditional models essentially learn a joint distribution of samples and risk vectors. If one applies risk-conditional generation, which means a preference is imposed towards less noisy samples during generation, then the regions that are correlated with high noise level in the sampling space tend to be ignored, yielding unbalanced distribution of generated samples. In Sec. 5) our experiment results (e.g., Fig. 3 confirm this claim.
EXPERIMENTS
In this section, we provide two groups of empirical results: one is to verify the validity our theo- rems in practice and the other is to apply method: risk-sensitive diffusion, to real datasets. Due to the limited space, we put other experiment results in Appendix B which involve more baselines (e.g., Unbiased Diffusion Model), different evaluation metric, and noisy images.
8

---

Published as a conference paper at ICLR 2025
2030
rumbe:
MAY 3000
(a) Training data, polluted (b) Samples from stan- (c) Samples from the risk- (d) Samples from risk-
by Cauchy noises.
dard diffusion model. conditional baseline. sensitive VE SDE.
Figure 4: Comparison Gaussian mixture data (Fig- 4()). with part of Despite minimal instability, our model still recovers the potential sample distribution (Fig. 4(d).
samples.
while both baselines (Fig. 4 Fig. 4(b) incorrectly produce many outliers.
5.1 PROOF-OF-CONCER STUDIES
Existence of stability interval. With 1,8(1) = 0.1 + 19.91, Fig. 2 shows an experiment, where VP SDE runs for clean samples (x(0),
while its risk-sensitive SDE (i.e., Corollary 3.1) operates on Gaussian corrupted samples (x(0), = =1). We can see that the clean and noisy samples follow the same distributions for step in the stability interval T(r) [0.26, 1]. This experiment verifies that Theorem 3.1 is effective in practice.
Risk-sensitive VP SDE under Gaussian perturbation. Fig. shows a comparison between model (i.c., Corollary 3.2) and the risk-conditional baseline Gaussian corrupted dataset Fig.3(a)) ((a). The conditional model underrepresents (Fig. ( the upper-right component at low-risk generation because contains many more (i.e., 95%) noisy samples than other components. Instead, the generated samples of our model (Fig. 3(d) are mostly unbiased, with no preference for spe- cific mixture component. This experiment confirms the weakness of the risk-conditional baseline and verifies that our model is more robust in practice.
Risk-sensitive VE SDE under Gauchy perturbation. With heavy tails, Cauchy distributions can usually distort clean sample far away, exhibiting a distinct behavior from Gaussian noises. Fig. shows an experiment on Gaussian-mixture data (Fig. 3(a)). but with Cauchy noises corrupting samples. While risk-sensitive SDE cannot achieve perturbation stability in this case, our model still nicely recovers the distribution of clean samples and is robust to outliers (Fig. 4(d).. In contrast, the
generated distributions of both standard (Fig. [ and conditional models (Fig. 4(b) are seriously
biased by outliers. This experiment highlights the flexibility of risk-sensitive SDEs and indicates that it can still be very effective under minimally instability.
5.2 APPLIED STUDIES
We now assess the Gaussian versions of risk-sensitive SDE (c.g., Corollary 3.1 on multiple real- world non-image datasets. We will find that our models still perform very well even when the data is highly noisy and the perturbation noise in fact not Gaussian.
Noisy time series. depicted in Fig. [ time-series data might have irregularly spaced observa- tions. To reshape such data into proper training samples for diffusion models, common practices are to first interpolate missing observations, resulting in noisy training samples. For this scenario, we adopt medical time series datasets: MIMIC-II (Johsson al. 2016) and WARDS (Alaa al.) 2017). For every time series in a dataset, we extract the observations the first 48 hours and se- leet their top 5 features with the highest variance, leading to 440diimensional vector. To impute the missing values in a highly noisy manner, we apply a primitive method: Gaussian process, to interpolate them and estimate the variances, which treated as the risk information.
Noisy tabular data. Tabular data is naturally composed of fixed- dinensiienii vectors, though they usually contain missing values, and imputations of those values introduce noise. For this scenario, adopt 3 UCT datasets (Asuncion & Newman 2007): Abalone, Telemonitoring, and Mushroom.

---

Published as a conference paper at ICLR 2025
Model
MIMIC III
Fime Senes
WARDS
Abalone
Telemomitorng
Tabular Data
Musbroom
VE
10.053
9.116
1.032
5.196
VESDE w/ Risk Regresser
7.Y21
7.522T
7..797
4.563
T636
VE SDE Risk Vanialle
6.549
7.314
0.853
5.161
4.970
SDE Conditional
3.159
Our Midel: Rist-sanstrive VF SDD
1.865
2.513
0.089
1.582
0.713
Standard VPSDE
8.765
6.23%
VP w/ Risk Regressor
7.981
7.832
0.732
4.197
5.327
VP SDE Risk Variable
6.723
7.515
0.899
5.169
6.583
VP SDE w/ Risk Conditonal
5.697
6.202
0.585
3.785
4,850
Our
VF3DE
2.582
0.077
0.832
Table I: Wasserstein distances of different models on 5 datasets across tasks. Part of model performances with another metric: MMD, are in Ta Appendix [ The results not only show that our model significantly outperforms the baselines, but also indicate: when the potential noise type is unknown, the assumption of Gaussian perturbation works well in practice.
Since these datasets are initially complete, we force the missingness by randomly masking 5% of the entries in each dataset. For data instance with missing values, we first apply k-nearest neighbors (KNN) algorithm (Peterson) 2009). to find the closest samples. Then, we impute the missing value with their median and treat their absolute median deviation the risk. Admittedly, the data generated in this way will be very noisy since KNN is certainly very inaccurate.
Experiment setup and results. Following com- mon practices (Ho et al., 2020), we adopt the com- monly used Wasserstein Distance (Heusel et al.) 2017 Kolouri et al. 2019 Colombo et al. (021) to evaluate the generative models, which measures the discrepancy of two distributions. For baselines, adopt two standard diffusion models (VE SDE and VP SDE) and three risk-conditional models (details in Appendix(AN Aj. In Table[] we can see that our mod- els significantly outperform all baselines regardless of the backbone model and the dataset. For exam- ple, with VE SDE as the backbone, our model has Wasserstein distances lower than Risk Conditional by 1.577 on the Telemonitoring dataset and 4.063 on MIMIC-III.
SDE Vartable SDE
Precis on
Figure 5: PRD curves (i.e., precision and call scores) of our model and baselines on Telemonitoring dataset.
We also depict the PRD curves (Sajjadi et al. 2018 Razavi et al. 2019) of our model and two base- lines on the Telemonitoring dataset. The PRD curve is similar to the precision-recall curve (Davis & Goadrich 2006) used in testing classification models: the curves that locate more at the upper right corner indicate better performances. From Fig. 5] we can see that our model consistently achieves better recall scores than the baselines at identical precision scores. Plus, the PRD curve of our model very close to the upper right corner, indicating the generation distribution of our model is almost consistent with the distribution of clean samples.
6 CONCLUSION
In this paper, we consider a novel problem setup to robustly train the diffusion models on noisy datasets: pairing noisy samples with risk vectors. To address this setup, we propose principled method: risk-sensitive SDE, in the spirit of minimizing defined measure: perturbation instability, which measures the negative effect of noisy samples. We have studied both the Gaussian and non- Gaussian noise perturbations, providing the optimal coefficients of risk-sensitive SDE in both cases. We have conducted extensive experiments can effectively handle noisy samples and significantly outperform previous baselines, even when the potential noise distribution might be non-Gaussian mis-specifiid as Gaussian.
multiple real datasets, showing that risk-sensitive SDE
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENTSS
We sincerely appreciate the constructive and professional feedback from the ICLR reviewers, which has greatly improved this work. We also thank Jonathan Crabbe for several discussions. He gave suggestions on conditional generation baselines and time series datasets.
BIBLIOGRAPHY
Peter Ahrendt. The multivariate gaussian probability distribution. Technical University of Denmark, Tech. Rep, pp. 203, 2005.
Ahmed M Alaa, Jinsung Yoon, Scott Hu, and Mihaela Van der Schaar. Personalized risk scoring for critical care prognosis using mixtures of gaussian processes. IEEE Transactions on Biomedical Engineering, 65(1))207-218, 2017.
Brian DO Anderson. Reverse-time diffusion equation models. Stochastic Processes and their Ap-
plications, 12(3):11-336, 1982.
William J Anderson.
Continuous- Markov chains: An arpplccciioooo-rreeeeeddo apppaaer Springer Science & Business Media, 2012.
Anastasios Angelopoulos and Stephen Bates. A gentle introduction to conformal prediction and distribution-free uncertainty quantification. arXiv preprint arXiv:2107.075/1 2021.
Arthur Asuncion and David Newman. Uci machine learning repository, 2007.
John Barnard and Xiao-Li ii Meng. Applications of multiple imputation in medical studies: from aids to nhanes. Statistical methods in medical research, 8( 8(1):17-36, 1999.
Dwyane C Carmer and Lauren M Peterson. Laser radar in robotics. Proceedings of the IEEE, 84 (2):299-320, 1996.
Hyungjin Chung and Jong Chul Ye. Score-based diffusion models for accelerated mri. Medical image analysis, 80:102479 2022.
Kai Lai Chung. A course in probability theory. Academic press, 2001.
Pierre Colombo, Guillaume Staerman, Chloé Clavel, and Pablo Piantanida. Automatic text evalu- ation through the lens of Wasserstein barycenters. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 10450-10466, Online and Punta Cana, Dominican Republic, November 2021. Assoociation for Computational Linguistics. doi: (..18653/v1/2211 emnlp-main.811 817. URL htt.tttss//aatttttlllaatooo wwwp:oyy..rgg/200000 -mm
817
Giannis Daras, Kulin Shah, Yuval Dagan, Aravind Gollakota, Alex Dimakis, and Adam Klivans. Ambient diffusion: Learning clean distributions from corrupted data. Advances in Neural Infor- mation Processing Systems, 36, 2024.
Anirban DasGupta and Anirban DasGupta. Characteristic functions and applications. Probability for Statisties and Machine Learning: Fundamentals and Advanced Topics, pp. 293-322, 2011.
Jesse Davis and Mark Goadrich. The relationship between precision-recall and roc curves. In Proceedings of the 23rd international conference on Machine learning, pp. 233 240, 2006.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. Ieee, 2009.
Prafulla Dhariwal and Alexander Nichol. Diffusson models beat gans image synthesis. Advances in neural information processing systems, 34:8780-8794 2021.
Lawrence Evans. An introduction to stochastic differential equations, volume 82. American Mathematical Soc., 2012.
11

---

Published as a conference paper at ICLR 2025
Yarin Gal and Zoubin Ghahramani. Dropout as bayesian approximation: Representing model uncertainty in deep learning. In international conference on machine learning, pp. 1050-1059.
PMLR, 2016.
lan Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, 63(11):139-144, 2020.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In NewrlPS 202/ Workshop on Deep Generative Models and Downstream cooonstream.Apppicatiosss 2021. UKLitttpss://pperrer//e
net f Eorum iid 1wWK
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6851, 2020.
Lei Huang, Hengtong Zhang, Tingyang Xu, and Ka-Chun Wong. Mdm: Molecular diffusion model for 3d molecule generation. In Proceedings of the AAAI Conference Artificial Intelligence, volume 37, pp. 1112-5112, 2023.
Kiyosi It6. 109, stochastic integral. Proceedings of the Imperial Academy, 20(8):519-524, 1944.
Xiaodong Jia, Ming Zhao, Yuan Di, Qibo Yang, and Jay Lee. Assessment of data suitability for ma- chine prognosis using maximum mean discrepancy. IEEE transactions on industrial electronics, (5(7):5872-5881,200 2017.
Alistair EW Johnson, Tom J Pollard, Lu Shen, Li-wei Lehman, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G Mark. Mimic-iii, freely accessible critical care database. Scientific data, 3(1):1-9, 2016.
Kaggle. 2017 kaggle machine learning & data science survey, 2017.
Ivan Kapelyukh, Vitalis Vosylius, and Edward Johns. Dall-e-bot: Introducing web-scale diffusion models to roboties. IEEE Robotics and Automation Letters, 2023.
Alex Kendall and Yarin Gal. What uncertainties do we need bayesian deep learning for computer vision? Advances in neural information processing systems, 30, 2017.
Yeongmin Kim, Byeonghu Na, JoonHo Jang, Minsang Park, Dongjun Kim, Wanmo Kang, and chul Moon. Training unbiased diffusion models from biased dataset. In The Twelfth Interna- tional Conference on Learning Representations, 2024. URL httph::t///to //opr/rv//erreeewww net/
Forum?id= 1dd.dc333P.111 BBd.
Youngjoo Kim, Hyochoong Bang, et al. Introduction to kalman filter and its applications. Introduc- tion and Implementations of the Kalman Filter, 1:1-16, 2018
Martti Kirkko-Jaakkola, Jussi Collin, and Jarmo Takala. Bias prediction for mems gyroscopes. IEEE Sensors Journal, 12(6):2157-2163 2012.
Soheil Kolouri, Kimia Nadjahi, Umut Simsekli, Roland Badeau, and Gustavo Rohde. Generalized sliced wasserstein distances. Advances neural information processing systems, 32, 2019.
Steven G Krantz. Partial differential equations and complex analysis. CRC press, 2018.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Arbind Kumar and Jagdeep Kaur. Primer based approach for per amplification of high ge content gene: Mycohacterium gene model. Molecular biology international, 2014, 2014.
12

---

Published as a conference paper at ICLR 2025
Martin F Laursen, Marlene D Dalgaard, and Martin Bahl. Genomic ge-content affects the accuracy of 16s mma gene sequencing based microbial profiling due to per bias. Frontiers microbiology, 8:287367, 2017.
Wei-Chao Lin and Chih-Fong Tsai. Missing value imputation: review and analysis of the literature (2006-2017). Artificial Intelligence Review, 53:1487-1509, 2020.
David JC MacKay al. Introduction gaussian processes. NATO ASI series F computer and systems sciences, 168: 108:133-166, 1998.
Byeonghu Na, Yeongmin Kim, HeeSun Bae, Jung Hyun Lee, Se Jung Kwon, Wanmo Kang, and chul Moon. Label-noise robust diffusion models. In The Twelfth International Confer- ence on Learning Representations, 2024. URL https:://opereviiii hewprreeiieeeeeeeeeeeeeeeeeeeeee fortmff
HXWTXXEHN
Bernt Oksendal. Stochastic differential equations: an introduction with applications.
Springer
Science & Business Media, 2013.
Bernt Øksendal and Bernt Øksendal. Stochastic differential equations. Springer, 2003.
Yidong Ouyang, Liyan Xie, Chongxuan Li, and Guang Cheng. Missdiff: Training diffusion models on tabular data with missing values. arXiv preprint arXiv:2307.00466 2023.
Leif E Peterson. K-nearest neighhor. Scholarpedia, 4(2):1883, 2009.
Ali Razavi, Aaron van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with vq-vae-2. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buu E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL btttps://procesd/// http::/ss.nniiirrne..eeceeeeeeee pdf.
files/
Yulia Rubanova, Ricky TQ Chen, and David K Duvenaud. Latent ordinary differential equations for irreuularyy-smppled time series. Advances in neural information processing systems, 32, 2019.
Mehdi SM Sajjadi, Olivier Bachem, Mario Lucic, Olivier Bousquet, and Sylvain Gelly. Assessing generative models via precision and recall. Advances in neural information processing systems, 31, 2018.
Jonathon Shlens. Notes on kullback-leibler divergence and likelihood. arXiv: arXiv: 1404.2000. 2014.
arXiv preprint
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics.. In International Conference on Machine Learn- ing. pp. 2256-2265. PMLR, 2015.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019.
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. Interna- tional Conference Learning Representations, 2021. URL https://epenrevier net/
[Formm?ddPPxIG1220
Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models. In Proceedings of the 40th International Conference on Machine Learning, ICMI:23. JML.R.org, 2023.
Ajay Sridhar, Dhruv Shah, Catherine Glossop, and Sergey Levine. Nomad: Goal masked diffu- sion policies for navigation and exploration. In First Workshop on Ous-of-Disrrbutton General- ization in Robotics at CoRL 2023, 2023. URL htttas///ppnneeriw..nnt//oooo FhoRJw71h5
Ove Steinvall and Tomas Chevalier. Range accuracy and resolution for laser radars. Electro- Optical Remote Sensing, volume 5988, pp. 73-88. SPIE, 2005.
13

---

Published as a conference paper at ICLR 2025
Daniel Stekhoven and Peter Bühlmann. Missforest pon-mmmmetriic missing value imputation for mixed-type data. Bioinformatics, (8 2012.
Chenxi Sun, Shenda Hong, Moxian Song, and Hongyan Li. A review of deep learning methods for irregularly sampled medical time series data. arXiv preprint arXiv:2010. 0eT000.1249. 12493, 2020.
Shuntaro Takahashi, Yu Chen, and Kumiko Tanaka-Ishii. Modeling financial time-series with gen- erative adversarial networks. Physica A: Statistical Mechanics and its Applications, 527:121261, 2019.
Qijian Tang, Xiangjun Wang, and Qingping Yang. Scale factor model analysis of mems gyroscopes. Microsystem Technologies, 23:1215-1219, 2017.
Boris Breugel, Nabeel Seedat, Fergus Imrie, and Mihaela der Schaar. SYNG4ME: Model evaluation using synthetic test data, 2023. URL
J7CTp- JNyJ
Sanford Weisberg. Applied linear regression, volume 528. John Wiley Sons, 2005.
Margaux Zaffran, Aymeric Dieuleveut, Julie Josse, and Yaniv Romano. Conformal prediction with missing values. In International Conference Machine Learning, pp. 40578 40604. PMLR, 2023.
14

---

Published as a conference paper at ICLR 2025
Appendix
A Risk-conditional diffusion models
16
A.1 Risk as the Variable Risk the Conditional
16
B Additional Experiments
17
C Backward Risk-sensitive SDE
18
D Wide Applications
18
E Stability Gaussian Perturbation
Risk Resk-sensitive Kernel
20
E.2
Solution for Gaussian Perturbation
21
F
Minimum Instability for General Noises
23
F.1
Spectral Representation
23
F2
Necessary Condition for Achieving Stability
24
F3 Solution for General Noises
25
G Applications to Diffusion Models
26
G.1 Extensions under Gaussian Perturbation
26
G.2 Optimization and Sampling
28
G.3 Extension under Cauchy Perturbation
30
15

---

Published as a conference paper at ICLR 2025
A RISK-CONDITIONAL DIFFUSION MODELS
An obvious way to adapt current diffusion models to the extra risk information is conditional generation (Dhariwal & Nichol 2021). The main drawback of conditional diffusion models that it might have a biased sampling distribution. For example, suppose regressor-guides diffusion model (Dhariwal & Nichol 202T) I is trained on a dataset composed of blurry pictures of dogs and clear pictures of cats, then the conditional model will generate very few images of dogs, given that they are associated with a higher risk than cats. the following, we present three different implementations under this scheme.
A.1 RISK AS THE VARIABLE
A naive implementation is first to let diffusion models learn the joint distribution of samples and risk vectors: po(x, r), and then regularize the reverse process for drawing samples of a low risk: r z 0, from the trained model.
Risk variable. In the optimization stage. we concatenate the sample and risk vectors z(0) = (0) ① in column-wise manner, with Eq. (T) and Eq. to train a vanilla diffusion model. We draw low-risk samples from the trained model at inference time through improved backward SDE. Considering the technique of classifier guidance (Dhariwal Nichol, 202I), we set a parameter- frree regressor- -11 lla: the minus square norm, which takes the last D entries of variable ZD+12D(() == r(t) as the input and has a derivate as -rre [(()]l2 = - (rt)//ll(()] With this regressor, the backward process (i.e., Eq. (2) is updated as follows:
dz(1) (f(z(O))) () (V)) 200) In (x()()()) - V(OO)1((112) (())d(())
()(z((),)) ()()) (000)((Vnn.(2 (Vlnp(z(()))
|r((]l2) + g())d(())
r(t)
(14)
(f(z(t), g(t)² (s)(z(t),)) [ )) a(t)dw(t),
r(t)
where some redundant parts are omitted (-). In practice, the gradient -Vro)llr()la is re-scaled
with a positive coefficient 7, which trades diversity for quality. Intuitively, the regressor -|| 12 gradually reduces the norm of r(t) as decreasing iteration such that the final sample x(0) (0) will be paired with low risk r(0).
A.2 RISK AS THE CONDITIONAL.
Another type of implementation treats the risk vector generation conditional for diffusion models. Ideally, we can draw clean samples from trained model by setting r O.
Risk conditional. There are two types of conditional diffusion models. The easier one is classifier- free (Ho & Salimans] 2021). which adds the risk vector as input to the score-based model: sø(x, t, r). Eq. 41 and Eq. are the same to train the new model, but the input r is randomly masked with dummy variable Ø to permit unconditional generation. For inference, the score function Vx(o) In pi (x(()) in the backward SDE (i.e., Eq. (2) is replaced with
(1 (x()) (), fr r- 0) (x)( 2),
(15)
where 7 is non-negative number that plays a similar role to the first model.
Risk regressor. The other one is just classifier-guided sampling. While the diffusion model re- mains the same, we separately train regressor h to predict the risk of sample:
f
In(1 1 exp(h(x)),
(16)
where the SoftPlus function exp(1+In( (n))) is to ensure that the final output is positive and variable either raw sample x or its noisy version a(t), > 0 (obtained by Eq. (T). For implementation,
the regressor h can be any neural network and is optimized with square Toss: |r flß. To have
16

---

Published as a conference paper at ICLR 2025
Dataset
Abalone
Telemonitoring
Standard VP SDE VP SDE w/ Risk Conditional
0.01056
0.01667
0.00766
0.01267
Our Model: VP SDE w/ Risk-sensitiv Diffusion
0.00198
0.00717
Table 2: Model performances measured by another metric: MMD.
a scalar outcome, we wrap the regressor as (x), h2(xx,.". ... [x(() and derivative
- In(1 + exp(h;(x)),, with expansion h(x) =
D Vx Σ In(1 + ex)(h,((x)) = -1 (h(())) (h,(x)
D
D
Oh,
|",
Σ
a())
(17)
dxi
OID
where a is the Sigmoid function. Similar our first model, we apply this derivate to regularize the backward process of trained diffusion model as
dx(t) R (f(x((),)) -(0)))0((()))()))))))))))))))))) - - ->Vx (vx((-inn)) In(-)) + (())d((),
(18)
where 7 R+ U(0) is non-negative.
B ADDITIONAL EXPERIMENTS
We have performed extra experiments to further confirm the effectiveness of our method: risk- sensitive SDE, including comparisons with other baselines (e.g., Ambient Diffusion), the introduc- tion of another evaluation metric: MMD, and a study on the image dataset.
Another evaluation metrie: MMD. Recognizing the importance of diverse evaluation criteria, we introduced the Maximum Mean Discrepancy (MMD) metric (Jia et al. (017) in our analysis. This additional metric further affirms the great effectiveness of our method to bridge the gap be- tween generated and real distributions across different noisy datasets. As shown in Table 2 our model still significantly outperforms the baselines in terms of the new metric: MMD. Overall, given the consistent best results of our method across different datasets and evaluation metrics (includ- ing Wasserstein Distance, Frecision-Recall Curves, and MMD), we believe the effectiveness of our method: risk-sensitive diffusion, is well justified.
Other baselines from similar settings. While the works of Ambient Diffusion (Daras et al. 2024) and Unbiased Diffusion Model (Kim et al.) 2024) also discuss the training of diffusion models on imperfect data, their settings are either a particular case of ours or differ altogether. In the following, we compare the settings and purpose of each of these works sisk-sensi diffusion and empirically compare our method to these baselines:
Ambient Diffusion addressed the setting of images with missing pixels, which, although is one of the applications of risk-sensitive diffusion, our method more broadly applicable to the general case of "noisy pixels";
Unbiased Diffusion Model considered the presence of both a biased dataset and a clean dataset and, thus, tackled a particular case of our method. If one attributes risk 1 to the biased dataset and risk O to the clean dataset, then our method can be used to learn an unbiased diffusion model. However, our method encompasses much more broad range of settings. In particular, this method does not admit varying risks for different features or risk distributions, risk-sensitive diffusion does.
The experiments of our paper include tabular data with missing values, which constitute valid benchmark for our method against the baselines. Table summarizes the obtained new results. The performances of Ambient Diffusion and Unbiased Diffusion Model are comparable our baseline VP SDE w/ Risk Conditional, and our method: risk-sensitive diffusion, significantly outperforms all of them, thereby showing the value of our contribution.
17

---

Published as a conference paper at ICLR 2025
Dataset
Abalone
Telemonitoring
Standard VP SDE
0.925
9.935
Ambient Diffusion
0.482
4.253
Unbiased Diffusion Models VP SDE w/ Risk Conditional
0.679
4.991
0.585
3.785
Our Model: VP SDE w/ Risk-sensitive Diffusion
0.077
1.462
Table 3: Comparison between our method and the baselines from similar settings.
Model
20% noisy images
40% noisy images
Standard VP SDE
8.31
13.29
Unbiased Diffusion Model VP SDE w/ Risk Conditional
5.37
8.53
6.57
9.15
Our Model: VP SDE w/ Risk-sensitive Diffusion
4.89
6.97
Table 4: Model performances CIFAR-IO with certain numbers of noisy images.
Study on noisy images. We also explored the performance of our risk-sensitive diffusion frame- work on image data: CIFAR-10 10 (Krizhessky et al., 2009) images with pixel noises. Specifically, perturb certain portions of CIFAR-IO images with Gaussian noises and compare method with two baselines (i.e., risk-conditional diffusion model and Unbiased Diffusion Model). Table 4 contains the results in terms of FID scores, which show that risk-sensitiv diffusion outperforms other methods in this setting, too. Our finding reveals that our method outperforms conventional and recent approaches even domains outside our primary focus, further underscoring its general applicability and robustness.
C BACKWARD RISK- SESSSIINS SDE
Although the backward risk-sensitive SDE is not necessary for our method and theorems, we still provide it for reference. Let risk-sensitive SDE be of the form
dx(t) (f(t) dt + diag(g(r, t)),w(t),
where both coefficient functions f()), g(r, 1) are everywhere continuous with right derivatives. Ac- cording to (Anderson) 1982), we can get the corresponding backward SDE as
dx (((t) x(t) v diag(g(r, diag(g(r, V Inpi( r) r)) dt (i)g(g(r,t))dw/
= -f(1))x(1) - g(r, Ix Vz di g(r, (
(19)
While g(r,t) t) = is possible for e (t e [1.7] g) f(r,t)² * g(r, t) our above conclusion still applies and the backward SDE is the same the forward one in that case.
D WIDE APPLICATIONS
The problem formulation of our paper: noisy sample (0) paired with risk vector r (i.e., accessible information of data quality), only rarely seen in the field of generative models, but also highly motivated by real-world applications.
Data with Accessible Risks. In many cases, data naturally born with information indicating their quality. Here are some examples from the biological and sensor domains:
Polymerase chain reaction (PCR) is widely used In DNA sequencing to produce genomic data. Since the accuracy of PCR is largely affected by the Guanine-Cytosse content (GC- content), researchers typically regard this information as primary indicator the data
quality (Kumar & Kaur 2014 Laursen al. 2017;
18

---

Published as a conference paper at ICLR 2025
Laser radars resort to laser beams to generate data, indicating the spatial positions of phys- ical objects. This type of sensor data tends to be very noisy, so the radars also provide the engineers with other data sources, such as light strength (Steinvall & Chevalier 2005) and device states (e.g., excessive voltage and temperature) (Carmer & Peterson 1996). which reflect the data quality;
Gyroscopes are commonly used in navigation and robotics, which measure the angular velocity of an object. This type of sensor can inherently estimate the data quality provide engineers with more information, including bias (offset from true value) (Kirkko-Jaakkola et al. 2012) and scale factor (deviation from the expected sensitivity) (Tang al. 2017).
Recently, there a growing trend towards applying generative models to scentific (c.g., Al for Science) (Chung & Ye, 2022 Huang et al. 2023) and industrial data (e.g., Smart Manufactur- ing) (Kapelyukh et al.) 2023; Sridhar et al., 2023). Including the above examples, those types of data are generally noisy and will play a key role.
with risk information, where proposed risk-sensitive SDE
Data without Available Risks. There also situations where the risk information for noisy sample (() is not available. However, since our definition of the risk vector r not limited, is find an alternative to the vector in low-effort manner, without resorting to manual annotation and expert knowledge. Typical examples are time series and tabular data in the medical domain (e.g.. MIMIC dataset (Johnson al. 2016)). Specifically, because these two types of medical data either are irregular (Sun et al. (2020) or have missing values (Lin & Tsal (2020). one will preprocess the data with interpolation and imputation before using them. Such preprocessing techniques are commonly not fully accurate, leading the final data to be noisy. In this situation, there are least two very efficient ways to harvest the risk information:
very likely that
Some interpolation and imputation models can inherently quantify the uncertainties of their predictions. For example, Gaussian Process (MacKay et al. 1998) and MissFor- est (Stekhoven & Bühlmann, 2012). The uncertainties provided by these models can be treated as the risk information;
There are a number of approaches (e.g.. Bayesian Dropout (Gal & Ghahramani 2016)) in the field of Bayesian Deep Learning (Kendall & Gal 2017 which estimate the prediction uncertainty of black-bor box model. If a preprocessing tool provides no extra information for output, one can apply such a method to construct the risk vector.
Even for high-quality image data, the concept of risk information still applies and convenient to find the risk vector. For example, images in the ImageNet dataset (Deng et al.) (2099) are of various sizes. To train a deep learning model (c.g., GAN (Goodfellow et al. 2020g) on that dataset, one has to first let all images have the same shape. In this way, clear image of small shape H x W will be expanded to a fuzzy image big shape H' x This type of sample is certainly noisy for the model and one can regard this ratio / /H'WW)/ (HW) - as the risk information.
Determination of the Noise Types. A question might arise: How for applying the risk-sensitive SDE? In some cases, we can infer it based on the mechanism that generates risk vectors. For example, the arrival time of unobserved sample in continuous-time Markov Chain (Anderson] 2012) has an exponential distribution. generating mechanism unknown, we can suppose the noise Gaussian, similar to the treatments in Conformal Prediction (Zaffran et al. 2023 Angelopoulos & Bates] (021) and Kalman Filter (Kim)
determine the noise type
other scenarios where the risk-
al. 2018). In Appendix our numerical experiments (e.g.. Table 1 show that this assumption works quite well.
E STABILITY FOR GAUSSIAN PERTURBATION
In this section, aim to find the optimal risk-sensitiv coefficients f(r, t), g(r, that let the risk- sensitive SDE achieves stability under Gaussian perturbation. We will first prove lemma about the kernel of risk-sensitive SDE and then dive into the main theorem.
19

---

Published as a conference paper at ICLR 2025
E. RISK-SENSITIVE KERNEL
For analysis purpose, we provide a lemma that determines the form of kernel P x())) (i.e.. the density of x conditioning on noisy sample x(()) for given risk-sensitive SDE.
Lemma E.1 (Kernel of Risk-sensitive SDE). Suppose we have risk-sensitive SDE defined as shapes as (x(1)((((1) x(0)) N () diag(((r,,)
Eq. (5) then its associated kernel (x()) x())
f(r,t) d (lnf(r,4) dt
(20)
d 1(r
g(r,t)2 f(r,,)² dt f(r,t)²
where f(r, 0) 1, 1(, 0) 0 and operation diag expands a vector into a diagonal matrix.
Proof. For SDE, its kernel is a Gaussian distribution and the first moment is also affine if the drift term is affine (Evans 2012: Oksendal 2013). Based on these facts, we can suppose that the kernel Pejo.r (0) of risk-sensitive SDE has the following form:
(x(1)(((()) ()) (t) F(r, ())(0), G(r, ()²),
(21)
where F(r,4), Grr, 1) are undetermined functions that output diagonal matrices.
Considering a corner case where /. - 0, we can infer that F(r, 0) I and (r,,0) For / 0, assume 5t 0 and 5t O, then we have
Pe+0110,- (X(t 8t) xx))) (
P1+..t.10o. (X(t
5t). xt)) (()))
(22)
= -fBeToLA (x(t + 5t) {(t)) (x(() (((0))x(()
Note that Pa+ 8e (x(( 8t) x(t), x()) = on (x(t 5t) x(t)) because of the Markov property. For notational convenience, we represent the risk-sensitive SDE as
(x(() Frr, ()) G(r, ()dw(l).
(23)
where F(r, t) = dfag(f(r, t)) and G(r,t) diag(g(r, t)), According to Eqs. 21 and 23 we
can have the following equation:
( + 8t) F(r, (1) G(r, t St)e1
((t) F(r, + G(r,t)e;
(24)
x + 8t) = (r. (t) + (r,
where E,, E2, 63 ~ (N, I) are independent Gaussian noises. Combining the last two equalities, we have the following equality:
x 8t) (I (r. Fr. t)x(0) ((I+ 8tF ()) G(r,t) VoGG(r,
(25)
Comparing the above two equations, we have:
F 5t) (tF(r, (r,
(26)
G(r,t+ 51)² = (I 56F(r, (f SIG(r,
Let öt - 0, this equation can be converted into differential form:
F(r, () dn F(r,t)
G(r, dG f)2 d F(r, t) di
(27)
dt.
dt
If G(r, t) is only continuous but not differentiable, then the term d Frr, t)/dt indicates its right derivative. Now, by converting all matrix-valued functions F(r, f), G(r, f), F(r,t), G(r, () into their vector forms f(r, ((r.))).r.. 1), frr 1), g(r 1), we have
f(r, t) d dln f(r,
d g (r)
g(r, 1),,)3
(28)
dt
di f(r,t)
where operations C and - respectively denote element- wise product and division. The initial condi- tions for F(r, (), G () can also be directly transferred f(r, (), (r,
20

---

Published as a conference paper at ICLR 2025
The above lemma is very useful. We will also see it in Sec. G
E.2 SOLUTION FOR GAUSSIAN PERTURBATION
Provided with Lemma E.1 the following theorem gives sufficient condition for letting the risk- sensitive SDE achieve perturbation stability for Gaussian noises. In Sec. F we will also see that this condition both sufficient and necessary.
Theorem E.1 seisiisen SDE for Gaussian Perturbation). Suppose that have Gaussian family of perturbation distributions: P. = (O diag(r")) # 0), then the risk-sensitive SDE
defined in Eq. (5) parameterized as below:
((r, dln u(t)
V4 [0, T]
dt
v(r,t)²
g(r, u
VL T(r)
(29)
di u(t)2
g(r, t) o, vt [0, T(r)
has the property of perturbation stability (i.e., (r)r) for any t in
T(r) = e [0, 7J t)r, r O v(r,0)3)
(30)
regardless of the weight function Sily). Here T(r) represents the complement T(r) and u(l), v(,),v(r.)) 0) are arbitrary functions that are everywhere continuous with right derivatives.
In particular, for risk the equations correspond the associated risk-unaware diffusion process for clean sample (x(0) = x(0), r 0).
Proof. According to Lemma E.1 the kemnl of risk-sensitive SDE shapes
P.jo (x()) x()) ((((()); + diag)/ia +))),
(31)
where coefficients f(r, t), g(r, t)² are defined in Eq 20 indicates the entry-wise product, and diag converts vector into diagonal matrix.
Let x() E IP be a real sample that is without noise and we perturb it as x = x(0) + e N(O. I). We aim first find the relation between risk-unan eak-unawrre kernel transition Pr ox x(0)) and the expected risk-sensitiv transition:
[x(0)
(P x())]
(32)
While the risk-unawe unaware transition is simply multivariate Gaussian:
Nxx: f(0, t) O x(),, diag(g(0, ))))
(33)
we can expand the expected risk-sensitive transition as
[(((0)); x0), diag()))))))))))) di))) |(0))(X(0))
(34)
{ (0)
f(r, O x(0), diag, ()))
The second Gaussian distribution in the above equation can be reformulated as
N(x:; ) (2T) D/2 diag(gg( t)²))
(agggg(rr)))))))
exp
( --
クタ (x f(r, ( O din)))igi)
diag(g(r, t))) ()))((((,,))))))((()
(2m)- D/2
((r, 1)2
112
1 (x(0) f(r,t) t)
g(r, 1)² -1
diag
exp
diag
Idiag( f(r, t))l
f(r, t)2
f(r,t)²
1
g(r, t)r,
(x(0),
diag
= |diag( ((rr
f(r, t)
f(r, t)2
(35)
21

---

Published as a conference paper at ICLR 2025
where means the determinant of a matrix. According to the product rule of multivariate Gaus- sians Ahrendt 2005), we can simplify the form iessiicnte transition as
S N N [x(0);; x(0) diag(r"))
[())
x
diag g(r,t)² f(r,t)²
dā(()
diag(f(x
t))l
f(r,t) t)
(f(ag((((,tt)) (
x f(r,t) x()). diag
r²
g(r,t)² f(r,
N (x)):) (0)):
(36)
t)2
(x ((r, O x(0), ding(f(r 1)2 o +
To let the two transitions equal, one must achieve the following two conditions:
f(r,t) f(O,t), t),
frr,
g(,,t)²
(37)
The first condition indicates that the term f is independent of risk r, while the second condition implies that there might exist some iteration / that the two transitions are not identical. Plus, because this term g(r, t)² is always non non-negative, we have
grr, 1)² max max 0)2
(38)
which means the risk-sensitive SDE has an initial period of pure contraction, but after that, its transition kernel is equal to the real one. Note that operation max is applied in an element-wiss manner. g(r, t)² might not be differentiable everywhere, but we can either locally smooth the curve or take its right derivative.
With the above derivation, we see that the following equation holds:
[x x(0)) (x00 [Pro.r x(0)),
(39)
if Eq. 37 holds. For the left hand, we then have
(x [P:jox) x(0))] = po (o(x(())))H(x x(0))xx((0) = px()..
(40)
Similarly, apply the expectation operation Ex(o) to the risk-sensitive transition:
x(0))]
pn(x(0))p(((()) x00)) ((0))x((0)dx(()
_x(1) 22(0)
(41)
Por (x.))((())))).xxxx ())) = (x).
x((0)
Therefore, we finally get Pr.r. (x) = (x) (i.e., perturbation stability) for t in
T(r) = { [0, T] f(r, t) f(0, t), f(r, of g(0.)))
(42)
Now, if we replace f(0, t). g(r, t) by utt), v(r, t). then we get the theorem proved.
Remark E.I. From this conclusion, we can easily see that, for a fixed coefficient unt), the setup v(r, t)² = max (v(r, 0)2 - r O u(t)² 0²) (where max is an element- wise operation) maximizes the period perturbation stability: [T()), leading to optimal coefficients.
Remark E.2. is apparent that stability interval T(r) shrinks as the risk r increases, indicating that noisier sample is less valuable for training. Therefore, under Gaussian perturbation, the ratio [T(r) reflects how much information is contained in noisy sample (x)
Remark E.3. If one limits the perturbation noise to be isotropically Gaussian, then the risk vector reduces scalar r, with P.- (A rI) r 0}. Eq. (29) and Eq. (30 the theorem can also be simplified in terms of r1, u()) m((() v(r, v(
In Sec. G apply the above theorem to diffusion models (e.g.. Risk-sensitive VP SDE in Corol- lary G.2 and develop tools (e.g., simplified loss in Proposition G.II for efficient optimization. In Appendix 5 numerical experiments confirm the validity of the theorem (i.e., Fig. 2 and show that its suggested coefficients let diffusion models be robust to Gaussian C samples (i.c.,
Fig. 3
22

---

Published as a conference paper at ICLR 2025
F MINIMUM INSTABILITY FOR GENERAL NOISES
The goal of this section to find the optimal coefficients f(r, t) g(r, t) of risk-sensitive SDE for general noise perturbation. Importantly, S,(r) = 0 not achievable in the case of non-Gaussian perturbation. We will first prove two other conclusions and then provide the main theorem.
will see that the property of perturbation stability:
F.1 SPECTRAL REPRESENTATION
The first conclusion is to study the form of cumulant-generating fiunction (tr((w). Lemma F.1 (Spectral Form of the Marginal Density). Suppose that have risk-sensitive SDE as defined in Eq. ()). then the cumulant cumulant-generation generating function of its marginal density (x)((x) at time step t [0, :]: (t,) has a form as
Xer((w)= = (o.r(f(r,t)Quu
Σ w (r,
(43)
2
where terms ((r,t),g(r,d are defined in Eq. 20
Proof. Based on Fokker-Planne equation Oksendal & Øksendal 2003), the marginal distribution (x)((x) at time step satisfies the following partial differential equation (PDE):
Opr.(() ôt
- -- f)
Σ tr,
(x))
1 D
gr t)² (x)) (44)
+ Σ
dz;
2
Ox²
i=1
Because (x) belongs to the function space L (IR {h: 1R
E
[1(xx) dx 000, we can
apply the continuous-time Fourier transform Krantz 2018).
F(M()))( = (x(x))xx((ff exp(iw WER
where is the imaginary unit, to the PDE as
(x)))
D Σ (((r (())
δt
(45)
Σ t)
D (w) = - dw.
(Prr(x)) (w)) (w)
2 2 t)
6 pe.((())
(w)
dx;
ox?
According to some basic properties of the Fourier transform, we have (
(x) I
(F
(w)) (w))))))))))))) à ow;
(w;F(Fi((x))
dxc,
dx,
(46)
2 (x)) (w)) w ((F(())()))())
Ōx?
By denoting FlPr. xx) (w) as (0r4r((r))) we can cast Eq. 44 as
@w)rr(w)
1 wifi(r,t) (sper(() ow;
( Σ w grr t)² (wr(ww)
(47)
ōt
In terms of the characteristic curves, we ordinary differential equations (ODE) D o (r..
dt
dw: -w---i wif,( [L, D1, dip = -- ds
ds = 1,
(48)
ds
where 8 and N respectively dummy variable and the set of all natural numbers. Considering the initial condition: ((0) = 0, (0) = ((0) (P)x(E), E = [2, _ the solutions to these ODE are formulated as
1 D
(),
t(s) = wr(s)= = (r,
(p(s) = (5) exp (-
Σ
(49)
2 i-1
23

---

Published as a conference paper at ICLR 2025
where coefficient functions fi(r, s), ge(r, 8) are of the forms
f((r,s) s) exp ( fi(r, sddss) g( o1
So (/)
ds'
(50)
Based on the above results, we can get the solution of (prr)(w) as
((r((w) Po. (f(r, 1) ()
Σ w (r,
(51)
2
in which
t(r,t) [g (r, t), g2 t), gD(r,t) of the equation.
[f f), fe(r, t), (r ()] and g(r,t) The lemma is proved by taking logarithms on both sides
We can also get a similar conclusion for the term xi(w) by setting r 0.
F.2 NECESSARY CONDITION FOR ACHIEVING STABILITY
The second conclusion is about the necessary condition to achieve perturbation stability.
Proposition F.1 (Necessary Condition for Perturbation Stability). Given the definition (i.e., Eq. (5)
of risk-sensitive SDE, then a necessary condition for to have the property of perturbation stability is that the perturbation results from diagonal Gaussian noises.
Proof. Let the noise distribution be of a free form g(E) S {q(E)de= = 1;9(E) > 0, V R,, then the distribution of clean data Px( will be perturbed into a noisy one
(x) = f po(x')g(x - -x') x')dx' = (pol-) * (po(())((-)))xx))
(52)
where represents the convolution operation. Through Fourier transform, have
(w) = (()).r(x))(()) = (x)))(()))(w) F(g(x))(w)
(P.r(())((())
(53)
where (ww) = (w)( Here we also suppose that g(E) e L (I
With the above results and applying the Lemma F.I
we can get the form of risk-unaware marginal
distribution (x) in the frequency domain as
1 Σ "
Xi(w) xo(P(0,
(0,1),
(54)
2
and the one for risk risk-sensitive marginal distribution (x. as
D (.r((w) = Xo.-(((r, () + x,(((r,))w) - Σ (r,
(55)
where (-)-) are the cumulant-gennerating function of noise distribution q(e).
Because the Fourier transform FF is injective in the domain of definition L (RR the prop- erty (x) = (x()) is equivalent to the condition X((w) = Xt.((w). Considering function Po(x) E L (RR) and variable w e I are arbitrarily selected, the above two equations are equiva- lent indicate that the below two conditions
satisfied:
F(O,t) f(r,t), ((w) exp (- (- J/w J diag (0,+)) f(0, 1)2 f(r, 1)2 (rr0))wa ww)
g(r,t)3
54
(56)
The shape of characteristic function ((w) indicates that its form q(x) in the spatial domain is multivariate Gaussian (DasGupta & DasGupta 2011). with the following moments:
Ex~g(x) [x] = O, diag(r") = xxxx = diag 2)
g(r, 1)2
(57)
f(0,
f(r,t)
24

---

Published as a conference paper at ICLR 2025
Therefore, perturbation stability is only possible for Gaussian noise .V diag(r R.. To achieve this stability, we have to pick up a risk-sensitive SDE of the form:
dx(t) (dln((t)
C x(()) dt [f(t)² d ((r, ) ddw(()
dt
dt f(t)2
(58)
g(r,1)² ma - 1 ((()))))))
which is risk-unaware for iteration in e [1, g t)² -
Paired with Theorem this proposition indicates that the necessary and sufficient conditions for risk-sensitive SDE to achieve perturbation stability (x) - (x(x) are: 1) noises follow diagonal Gaussian distributions; 2) the time step within the stability interval.
F.3 SOLUTION FOR GENERAL NOISES
Provided with former two conclusions,
prove the below main theorem.
Theorem F.1 (General Theory of Perturbation Stability). Suppose we have a family of continuous noise distributions: P. {pr(E) IR lR+ -
(pr(e)de 1|r + 0}, controlled by a risk vector
then the optimal coefficients of the risk-sensitive SDE (as defined in Eq. (5) that minimizes the perturbation instability Sx(r) satisfy
v(r (), max (0, v(0,)) (((()),)))
(59)
in which the term (.) is defined as
(u((),r) - (xxy)
② ((fo()) exp (xx(u(() (y))]yy]dy
(60)
where & stands for a matrix multiplication, Xr(y) is the sumulant-generatinn fuuntionnnguunctionn distribution pr(e), and u(t), v(r,t) satisfy the same conditions as in Eq. (29). Importantly, the property of perturbation stability Sy(r) = -0 is only possible to achieve for Gaussian perturbations of diagonal form: pr(c) =:(])=]((00)) diag(r)))
function of the noise
can see there is no appropriate risk-sensitive SDE to fully neutralize the negative impact of non- Gaussian noise distribution q(c). Therefore, we aim to find optimal (though not perfect) risk- SDE in this regard. Note that po(x) is arbitrarily selected in space ()) so condition f(r, f(0, i) still needs to hold. For g)0, we first consider the objective function to optimize:
Proof. Based Lemma F.1 and Proposition F.
O, = (((w)xxrr((a x((w)) dw,
(61)
where |. represents the magnitude of complex number and ?(w) RDD R+ is a predefined weight function. Considering Eq. (54) and Eq. (55) we have
O, [sww>]
Σ (0,
1 Σ
wfg.(r.t) +x.(f. dw
1 /
w² (00 g(r,
t)²) t)r,
ln (f(r, +i +i.arg (((f(r..t)))) (dd do
2
[a(w] 1 w2 ((0 t)² g(r, t)² + In t, dw ((()) arg (.) dw.
(62)
where arg(.) and <.. respectively represent the argument of complex number and the inner product two vectors. Plus, X-(+) and (.) are respectively the sumulant-generatin and charac- teristic functions of noise distribution pr(e).
Now, we denote g(0, 1)2 ( t)2 as a dummy variable y = [Vv, U2. yp T. Then, we compute the derivative of objective O, with respect every input entry yj.J E [1, DJ:
dO: dy; - (M(w) w/ (w)))))) +
)w) de.
(63)
25

---

Published as a conference paper at ICLR 2025
Through setting dOr/dy, 0 for every j in [], can get
(I] ((W))
iE[1,D] x) (w) In (r 1) O w)
(64)
where LI represents some column vector. By combining all results, we have
[]
(WW)) [
2 {[ ((w) In ( (f(r, 1) O w) of (E(L)
(65)
where []]jE[1,D] represents some matrix. For notational convenience, we denote
MWWWFFWfdw
[(w)[ww dF
i.je[I.]]
[]] ((w) In ( ( wdd [E(1.D]
(66)
((w)) In 1 (+
Considering that (8001) g(r, 2) and g(r, 1)2 is always non-negative, we get
g(r, 1)2 max ((.,)) 2 {{ (ww)(ww ( For simplification, we introduce a new symbol L as
-1 [( ((w))) In (w) )o).
(67)
(f(r, r). = 2( [(w][ww (P)w))))TTT))) " ((w))) (f t) (w) |w)
(68)
=2(foW))ww [ww (w))ww) )adw) (( (((w)) In exp (xr(F(r, () (w)) [w]]
As result, the optimal coefficient g(r, can be formulated as
g(r, 1)2 = max ()(0,))2+ (f(((r ())),))
(69)
which finally proves the theorem.
With the above theorem, we can find the optimal risk-sensitive SDE for non-Gaussian perturbation,
though its coefficient g(r, () might have rather complicated form. In Sec. G we apply this theorem
to Cauchy perturbation under architecture iments in Appendix also show that such risk-sensitiv VE SDE very robust for optimization with Cauchy- corruted samples (i.c., Fig. 4
SDE (i.e., Corollary G.3). Our numerical exper-
G APPLICATIONS TO DIFFUSION MODELS
In this section, aim apply risk-sensitive SDE and our developed theorems the practical implementations diffusion models. We will show how to extend a vanilla diffusion model to its risk-sensitive versions under different noise perturbations and provide efficient algorithm for optimizing the score-based model with risk-sensitive SDE.
G.1 EXTENSIONS UNDER GAUSSIAN PERTURBATION
In terms of Theorem E.1 we have the following corollaries that extend two widely used diffusion models to their risk-sensitive versions. One of them is VE SDE (Song et al. 2021).
Corollary G.1 (Risk-sensitive VE SDE for Gaussian Noises). Under the assumption of Gaussian perturbation, the risk-sensitive SDE (as defined by Eq. (5)) for VE SDE, a commonly used risk- unaware diffusion model, is parameterized as follows:
do dt
f(r,t) 0. g(r,t) ((0))
26

---

(t)² Published as a conference paper at ICLR 2025
where 1(.) is an element-wiss indicator function, represents the element-wiss greater-than sign. The stability interval T (x) in this case is
(r) { [0, o(t)²
max - - 0 ie|1.DI
In particular, for the case of zero risk r = O, the above risk-sensitive SDE reduces to the vanilla VE SDE, with f(0, t) = O, g t) v (t)2 and T [0.
Proof. For SDE, its risk-unaware kernel as
Pro (x(()(() x()) (x((((); ((() O diag
f(t)
g(0, f)2 (a(1)²
where o(t) [0, T] IR+ is an exponentially increasing function. Considering our previous conclusions (i.e., Eq. (37) and Eq. (28) for Gaussian noises, the risk-unaware and risk-sensitive SDEs for VE SDE are parameterized as follows:
dx(() = g(0, b) 2 dw g(0,t) = IV do(t)²
dl
dx(t) = g(r, t) g(r,t) ((o o (0)
dt
where 1(-) is sement-wise indicator function. The sensisene SDE of VE SDE of perturba- tion stability at iteration E [0, 7 iff the vector o r is positive in each entry.
The other is VP SDE (Song et al. 2021), the continuous version of DDPM (Ho et al. 2020).
Corollary G.2 (Risk-sensitive vP SDE for Gaussian Noises). Under the assumption of Gaussian perturbation, the risk-sensitive SDE for VP SDE is parameterized as follows:
f(r, t) = 3011, g(r,t) = 1(1 (1+ (1+r))((t)) B(t),
where (t)) = (xp== So (ss)
The stability interval ((x) in this case is
T(r) {t [0, a(t)
1+ max r}]. LE
As expected, for the situation with zero risk r = 0, the risk-sensitive SDE reduces to an ordinary VP SDE, with f(0, f) = - 1, g0,, f) 1t)
Proof. For VP SDE, its risk unaware kernel is as (r)o(x(t) x())) = =N(x(t): Ox(0), diag(g) t))) f(t) 1exp (- J Jo
B(s)ds)
g(0, l)2 1 eexp (- B(s)ds)
where (()) [o. T IR+ is a predefined curve. Similar to our discussion about VE SDE, by using
Eq. (37 and Eq.
28 the risk-unaware and risk-sensitive SDEs for VP SDE are
dx(t) = (f(t) 3 x 1) g(0 dw()), = -,,, 3()),,
3()1
(x(t) (f(+) O g(r, dw g(r,t) 1 ( ( (1
exp (-f. ((s)ss)) 3(1)
exp
The stability interval T(r) is as (t [0. exp(S) (s))ss) > 1 max
11
One can apply the two corollaries to isotropic Gaussian noises by simply setting r = r1, such that coefficients (()),((rt) will reduce to scalar functions. In Appendix our numerical experiments confirm that Risk-sensitive VP SDE can achieve stability under Gaussian perturbation (i.c., Fig. 2 and it is indeed robust to Gaussian-comupted samples (i.e., Fig. 3)
27

---

Published as a conference paper at ICLR 2025
G.2 OPTIMIZATION AND SAMPLING
In this part, we provide essential elements for applying risk-sensitive diffusion models in practice, including the loss function, optimization algorithm, and sampling algorithm. Before diving into the details, we will first prove a lemma about the stability condition.
Lemma G.1 ("Not Instable" means "Stable"). Provided with the definition of instability measure S,(r) in Eq. (6) and suppose that distributions px(x), pr(e) are continuous. If we have S,(r) 0. then (y) Xc(y), Vy E TRD and (x,(()) = (x(),, Vx E RD both hold.
Proof. We prove the first equality by contradiction. Suppose that Xxr(yy * Xcly) for some point y' RD,, then there is closed region y' E U RD that satisfies |0] > 0 and (y. of Xi(y), Vy e U because probabilistic densities (t.r(y),xelll are both continuous every- where. Considering that the region U is closed set, have the below inequalities
C1 = min (y) - xx(y)) > O, C2 = min Myy) > yeU
(70)
yEU
With these results, we further have
Sr(r)= > [. (y)))xrr()) x:(y) S Se - 10K > 0,
(71)
which contradict the precondition (r)r) = Hence, we have (v, = ,y Vy IR., We can also immediately get the second equality proved since probability densities are uniquely determined by their cumulant-generatinn functions.
trivial that stability condition (x) (x(x) indicates zero instability measure S((r) O. Therefore, an implication of the above lemma is that the two conditions are in fact equivalent if the sample (x( and noise distributions (r((c) are both continuous.
Risk-free loss for noisy samples. In the following proposition, we derive the loss function for risk-sensitive SDE to robustly optimize the score-based model so(x, t) with noisy sample (X)
We also further simplify the loss for efficient computation.
Proposition G.1 (Risk-free Loss for Robust Optimization). Suppose that the risky sample (x(0) x() c. r) is generated from clean sample (x(0). 0) with some perturbation noise Pr (r(c), then the loss of standard (i.e., risk-unawrre) diffission models at time step t:
fi,==0 (x) l (x, Vx In In)x((()))
equal to the below new loss for non-zero risk r 0:
Cir = (x) [lss xx t) - V ln
(72)
if the noise distribution pe(c) is Gaussian and the time step t is within the stability interval T(r). Importantly, the alternative loss Lir has another form:
Li.r = C. + Ex(o) mlmm / v(r,t) + so(u(t) x + v(r, C n. nn.)) [3
(73)
where x(0) Po.x(x), coefficients u(t), v(r, t) are as defined in Eq. 29 1 I., and C, is constant that does not contain the parameter 0.
Proof. Based on Theorem E.
and Lemma G.I we know that the stability condition pt.r(x) = pr(x)
is achieved in this setup. Therefore, we can derive
C.o = (x 1) - Vx In (x)]l = [[]|so(x,)) () - Vx In Pn.), Lr,--
(74)
Therefore, the first claim of this theorem is proved.
Secondly, we aim to derive another form of the loss C.r make it computationally feasible. To begin with, by expanding the definition of Lir, we have
L., (x(1) (R(e)) (x((),,), - Inp..
(75)
[s( 1 V In (x(O)) 2so (x()),) VxO (n(r..(((())
28

---

Published as a conference paper at ICLR 2025
Considering the following transformations:
V x(() Ia (x(1)) Vx(1)Pr.xx((( Vx(1) SPo....... (()))
De. (x())
(x,)((
Vx(0) Spo. (X(0)) (x(1) x(0))d((0)
(t)r(
(x(.t.(((()), x()) x())
x()) In ( (()))(((0)
(76)
pt.
iooj. (x(0) [ k()) In P. (X(t) ((0))
(x00)))) xx()) In (x(()) {()))]
Combining the above two equations, we have
Crr =
Ex(e) [|sa())] + Vx(e) In (x(1)) 2sg (x(1), t) [x(o)\<<()) ( {
(x(0).x(1)
(-)
+ V x(() In Prioor (-) 2so (S((), t) Vx(1) on (x(t) X ())) + C.
(o, x()) (1)00))000..00............ P..o.
{(O))
So (x((),,)) - Vx( In ( x + C..
(77)
where C2 is constant that does not contain parameter A:
C. = E2000 V In.).. - VV on (x0))(
(78)
Finally, we only have to simplify the derivative term:
Vx(1 P (x((t) x(0))= = Vx(1) (00
D In (r, - - (1) (79)
= -x ( (- (
ln(2 -
j=1
f(r, O ( x x(t)
g(r,))
To conclude, the risk-free loss has a simplified form as
(x()),) f(r, t) @ x(0 x(t) 2
C.r = (E(00) (00))
+ Cr. (80)
g(r, l)
Similar to DDPM (Ho et al. 2020). we can further simplify this equation by Gaussian reparam- eterization. With the reparameterization x(1) f(r, O x(0) + g(r, t) G,, (0, I), we get
|s
Cer (x00))
g(r,t) 2 C,, (81)
where time step belongs to stability period T.
The expression of risk-free loss in Eq. 73 permits efficient computation in practice. Importantly
but as anticipated, for the case of zero risk r = 0, the term reduces to the loss function of ordinary risk-unaware diffusion models: Lr,0, for clean sample x(O).
Optimization and sampling. We respectively show the optimization and sampling procedures in Algorithm and Algorithm We also highlight in blue the terms that differ from vanilla diffusion models. For the optimization algorithm, when the risk is 0, the algorithm reduces to the opti- mization procedure of a vanilla diffusion model, with a trivial stability interval of T(r) = [o. T7. When the risk is non-zero, the risk seskssensitive coefficient v(r, t) and interval T(r) will guarantee that Vx In (x) = V. In pr. for E T(r), such that the noisy sample (x(0), r * 0) can be used to safely optimize the score-based model 6o(x, t).
29

---

Published as a conference paper at ICLR 2025
Algorithm 3 Optimization Algorithm
1: repeat
2: Sample (x(0), F) from the dataset
Sample time step from stability interval T (x)
4: x(t) ut) O (0) + v(r, On 10)
5: Update 0 with V v(r,t)
(1)
6: until converged
Algorithm Sampling Algorithm
1: Set time points (tM- T, T, .22
2: Set zero risk
3: x(tar) (x) 2 (x: 0,
4: for M, 2,.....a do
5: t(x(t)),t)) = f(r,t,) @x((t)) g(r,t,)² C (s((((t)),t)
6: n~ NoO, (4, [)
7: x(4-1) x(4:) - b(x(tt(x (:; 4-1) g(r,4) On
8: end for
9: return x()
For the sampling algorithm, by setting zero risk r 0, the coefficients f(r, L), g(r, L) become compatible with the model sg(x, t) and together generate high-quality sample (0). Our model will generate only clean samples (x(0).r = 0), but it was already able to capture the rich distribution information contained in noisy sample (x(0), r * 0) during optimization.
G.3 EXTENSION UNDER CAUCHY PERTURBATION
In terms of Theorem F.I we provide corollary that applies risk-sensitive diffusion VE SDE, letting it be robust to Corcht-corrupted samples.
Corollary G.3 (Risk-sensitive VE SDE for Cauchy Noises), For the weight function f2(y) = 1x and Cauchy noise distribution pr(c) that is specified by scales (Le., risk vector):
D
)
(r(E)
nr,
11
the minimally-unstaleee risk-sensitive SDE (defined by Eq. (1) for VE SDE has drift coefficient as f(r, t) 0 and a diffussoon coefficient as
do
g(r,t) = 1(o(t) 1( b(())
dt
where the term (x) is defined as
(1r)= (r (r-2 + diag(5r (iiag(5r1)) (1-1T11) 2r
The vector r is element-wise non-negative. For r = 0, the risk-sensitive SDE reduces to VE SDE, with f(0, t) O, (0, t) = dt)
Proof. Based on Theorem F.1 we aim to derive risk-sensitive SDEs for the noises sampled from multivariate Cauchy distribution. We first suppose noise [e1, .eD]] with every dimension j E [1, D] being independent and following a univariate Cauchy distribution py(e,) that parameterized by scale ky:
1
(5(w,) exp ( Kkylun (-Kylwy)).
P)(e,) TH (1+ (1+23/2)))
(82)
30

---

Published as a conference paper at ICLR 2025
where (يار is the characteristic function of distribution py(c)). Then, because random variables (1,52 CD are mutually independent, their joint distribution p(c) equal to 1 P}(C)) and we can derive its characteristic function ((w) as
((w) EE-ete T w)l f p(c) exp(ic w)de
(83)
(f (fells) (xp((es)s))) j=1
II
exp (-x (wl)
where N [x1, N2: :. KDD] Now, we can convert Eq. (66) into the following form:
[s (WW)wRFfdw (-[STW) (PW)O()))) ww) =
DE (((0,))-- g(r, t)², = [[(w) p(f(t) wf (a
T
2 2[ʃ [ww)) j.je(i, (1()))))
(84)
We set the weight function l(w) as the magnitude of the characteristic function [o(w]] since it indicates the importance of value w. In this regard, consider the below five integrals:
S exp(-Nylus|) Ny )dws
2
Texp(=xslee (u)
4 x33
2 [ [exp(- (xp(-Kylus)))wsjle ldw;
Kig
f expl - Hiy (wzg)//
48 K3
exp( Hy nyررارا Du Idw,
12
(85)
we can solve that linear equation as
- - 0)² = 1
1 21 =
(1())=))
(86)
[. Kin]
[.Je[I.D]
For demonstration purposes, let's consider the case of D 1:
g(r, = g(0, 8)2 - f(t)²²
(87)
For D 1, we can only have the below form with an inverse matrix:
g(r, 1)) g( - f(t) O (nN 2xx (nx-2(E 2) + diag(5x (5) -x 1m=-' +22x+=x7)
1xxx 9))
(88)
n where & [MT R22 .
For risk-sensitive VE SDE under Cauchy Perturbation, the concept of stability interval does not apply, though its coefficient g(r, is optimal in sense that the instability measure S,(r) is mini- mized. For optimization, one can simply set T(r) = [o. T] and apply Algorithm 3 In Appendix numerical experiments show that the risk-sensitive VE SDE are very robust for optimization with Cauchy-corrupted samples (i.c., Fig. 4p.
31


Reviews:
[{"review_id": "LYNlunbDkF", "reviewer": "Reviewer_8F9f", "summary": "This paper introduces a method called risk-sensitive SDE to improve diffusion model performance on noisy non-image data, like tabular and time-series datasets, by pairing noisy samples with risk vectors. Applying the standard diffusion process on noisy samples causes a marginal distribution shift and degrades generation quality. To mitigate this, the risk-sensitive SDE aims to minimize noise impact and improve robustness by minimizing a stability measure, termed Perturbation Instability. If perturbation stability holds—which is achievable in cases where the noise distribution is Gaussian—the noisy sample will have the same distribution as that of the corresponding clean sample at some iteration t in the Stability Interval and this could be used to optimize the score-based model. When this condition does not hold, the model seeks to minimize perturbation instability. The authors provide sufficient and necessary conditions for achieving perturbation stability and conduct extensive experiments demonstrating the effectiveness of risk-sensitive SDE over baseline models in both Gaussian and non-Gaussian noise settings.", "questions": "1. As mentioned above, while this paper focuses on noisy non-image data, such as tabular data and time series, noisy image data is also a common challenge in real-world applications. How would the proposed method perform on image datasets with high levels of noise?\n\n2. Could you provide more details on how you derive the sensitivity interval in practice and what are the challenges you might encounter in deriving this?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-written and well-structured, addressing an important problem and clearly articulating its contributions and findings.\n\n2.  The work provides a well-developed mathematical foundation, including solutions for both Gaussian and non-Gaussian noise, enabling a robust approach to noisy data.\n\n3. The paper includes well-designed experiments that validate the theoretical results and demonstrate the effectiveness of the proposed approach in handling noisy samples.", "weaknesses": "1. The paper focuses primarily on tabular and time-series data, which may limit insights into the method's performance on other data types, especially noisy image data. Extending experiments to image datasets could demonstrate the method’s generalizability across diverse noisy data contexts.\n\n2. The effectiveness of the proposed approach heavily relies on accurate estimations of the risk vector. If these estimations are inaccurate—either overestimating or underestimating noise levels—the model may apply inappropriate adjustments, which could hinder its ability to adequately suppress noise. This reliance on accurate risk estimation introduces a vulnerability to the approach.\n\n3. The paper could benefit from a discussion on the limitations of the proposed method and the areas that could be improved in future work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aMkfh7jdLb", "reviewer": "Reviewer_Rd4g", "summary": "This paper extends diffusion models to non-image data, like tabular data, which often contains noise that degrades model performance. The authors introduce a \"risk vector\" paired with each sample to indicate data quality and propose a risk-sensitive stochastic differential equation (SDE) that leverages this vector to minimize the effects of noise during model optimization. With specifically chosen coefficients, the risk-sensitive SDE supports more stable diffusion model training, accommodating both Gaussian and non-Gaussian noise types.", "questions": "Please see weakness.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1 This paper ntroduces the novel concept of a \"risk vector\" to improve the robustness of diffusion models against noisy samples. This approach uses a principled method, risk-sensitive SDE, to incorporate the risk vector and reduce the adverse effects of noise, specifically perturbation instability.\n\n2 This paper provides analytical solutions for risk-sensitive SDE with both Gaussian and non-Gaussian noise. A key finding is that Gaussian noise can be fully mitigated, eliminating its negative impact on model performance.\n\n3 Extensive experiments on tabular and time-series datasets demonstrate the model's effectiveness in handling noisy samples, even when noise is mis-specified or non-Gaussian, outperforming prior baseline models.", "weaknesses": "1. The second section suggests that noise interference can cause a bias in the neural network's estimation of the score function. A visual experiment could be added to illustrate the extent of this bias.\n\n2. The construction of Risk-Sensitive Diffusion inherently requires extensive prior information about noise. Could it be possible to denoise the data directly instead? The article lacks relevant exploration and comparative experiments on this aspect.\n\n3. The Risk-Sensitive Diffusion approach is limited to Gaussian and Cauchy noise. However, is this limitation applicable to most real world datasets? The article lacks an explanation regarding this point. Although the experiments in the article show that their method can be applied to unknown noise under the Gaussian assumption, there is a lack of experiments demonstrating whether the noise itself actually has Gaussian characteristics.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7pOxC0n3Mo", "reviewer": "Reviewer_pAqa", "summary": "This paper introduces a framework for optimizing diffusion models in the presence of noisy data by using risk-sensitive stochastic differential equations (SDEs) guided by a “risk vector” for each data sample. This risk vector quantifies sample quality, allowing the model to adapt its optimization process to noisy conditions, minimizing instability and improving robustness. The authors derive analytical solutions for risk-sensitive SDEs under both Gaussian and non-Gaussian noise and validate the framework’s effectiveness through experiments on tabular and time-series data, showing significant improvements over baseline diffusion models.", "questions": "I am willing to reconsider my scores if my concerns are addressed. \n\n* Lines 56-57: The claim that isotropic Gaussian noise can completely eliminate negative impacts is maybe a bit of an over-statement. Could you provide a brief explanation (the data being already noised, we juste use it after some noising time)\n* Lines 93 and 119 contain redundant expressions such as “the reverse process (i.e., reverse version of the diffusion process)” and the entire “Remark 2.1.” Simplifying these would improve readability.\n* Definition 3.1: The phrase “The risk information $r$ shapes as a vector” is mathematically unclear, alike the definition of the set in (4). The use of $P_{\\epsilon}$ for a family of distributions that should depend on $r$ is confusing. Furthermore, on line 172, $P_r$ is used instead of $P_{\\epsilon}$, while it still does not depend on $r$ (all isotropic centred Gaussian distributions?). This is frustrating to follow.\n* In Theorem 3.1, it would be nice to add some interpretations for the mathematics. What can we expect the stability interval to look like? Are we sure $u(t)$ or $f(t) (= u' / u)$ does not diverge at $t=0$ or on the boundary of the stability interval? What can we say about the fact that $f(r, t)$ does not depend on $r$? You can then articulate it better with Corollary 3.1, where you can focus on explaining that, essentially, a noised sample $\\Tilde{x}(0)$ can be used in the training process at the moment when the noising process would have added \"equivalent\" noise from a clean sample. Is this not a good interpretation?\n* Line 340: For non-Gaussian noise, it would be interesting to specify any required conditions on the noise distribution for the derivation of tractable expressions.\n* Experiments (Tabular Data): Given that the imputed data is noisy (lines 506-507 \"the data generated in this way will be very noisy since KNN is certainly very inaccurate.\"), it will mostly not be used to train the model, since the stability interval will be a small set around $T$. To properly assess the technique, you would need an experiment where you train a model only on the clean samples. \n* Experiments (Noisy Time Series): In the same way, we would need more information on how the interpolation is constructed, what is the kernel used for the Gaussian process, on which part of the dataset is the generative performance measured etc., so we can fully understand the performance gap. A comparison based on your customized evaluation methods, which appear designed to highlight the strengths of your approach, may not fully capture a fair performance assessment.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 1, "contribution": 3, "strengths": "* The paper provides a comprehensive characterization of each case encountered within the risk-sensitive framework\n* The proposed methodology intrinsically outperforms approaches that condition the score function solely on the risk value or vector, demonstrating better handling of noise.\n* Simple, low-dimensional experiments effectively illustrate the method’s properties: stability intervals for Gaussian-corrupted data, enhanced robustness to class imbalance relative to a risk-conditional baseline, and flexibility of the framework with \"minimal instability\".", "weaknesses": "* The paper would benefit from improved mathematical precision, as some definitions and notations create ambiguity, which affects readability. The structure can feel confusing. Some theoretical claims could provide interpretative insights, which would strengthen the theoretical exposition.\n* The experimental setup appears tailored to emphasize the proposed method’s strengths, which raises questions about the generalizability and fairness of the evaluation. For instance, it is likely that the proposed method includes noisy imputed data only very late in its training process, thus excluding them in practice, which would explain most of the performance gap.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QsGKT8vTTr", "reviewer": "Reviewer_YG5L", "summary": "When we train a diffusion model, what if not all samples from the training dataset were made equal? Some contain no noise - risk of zero - and some are noisy and come with so-called risk vectors that indicate how corrupted they are. The paper proposes modifications to the standard diffusion training and sampling algorithms that account for this noise in the training data. Samples generated by the resulting model are [almost] the same as if the model were trained on entirely clean/uncorrupted data. The authors show that their modifications are grounded in theory and are expected to work in many realistic scenarios.", "questions": "I don't have questions - everything in the paper is pretty clear. I suggest improving empirical results on the real-world data. A low-hanging fruit would be promoting the vision results from the Appendix to the main body: this methods seems very general to me and needs not be confined to just the time series or tabular data setting. Generally, authors should think about/look for settings where this risk/corruption is front and center. Perhaps from vision and video domains. Or from finance - you mention applicability, but don't attempt to solve any finance-flavored problems. Perhaps volatility can proxy for noise/risk. Right now, this paper is more like a [rather elegant] solution looking for a problem it can solve.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "I buy into the concept, I think that this is an interesting twist on the original idea. Even though the changes to the diffusion algorithms are rather minor, the theory section persuades me that this is indeed the right modification in the presence of noise. The toy examples are equally persuasive - it's pretty clear that the proposal works.", "weaknesses": "I'm less impressed with the empirical results, as they too seem rather artificial and closer in spirit to the synthetic data than to a real-world problem being solved.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)