Free Hunch: Denoiser Covariance Estimation for Diffusion Models Without Extra Costs
Authors: Severi Rissanen
Markus Heinonen
Arno Solin
Venue: ICLR 2025 Poster (2025)
Keywords: diffusion model
conditional generation
inverse problems
denoiser covariance estimation
Primary area: generative models
TL;DR: We propose a new, efficient method for denoiser covariance estimation in diffusion models, which can be used for conditional generation and inverse problems

Abstract: The covariance for clean data given a noisy observation is an important quantity in many training-free guided generation methods for diffusion models. Current methods require heavy test-time computation, altering the standard diffusion training process or denoiser architecture, or making heavy approximations. We propose a new framework that sidesteps these issues by using covariance information that is available for free from training data and the curvature of the generative trajectory, which is linked to the covariance through the second-order Tweedie's formula. We integrate these sources of information using (i) a novel method to transfer covariance estimates across noise levels and (ii) low-rank updates in a given noise level. We validate the method on linear inverse problems, where it outperforms recent baselines, especially with fewer diffusion steps.

Published as a conference paper at ICLR 2025
FREE HUNCH: DENOISER COVARIANCE ESTIMATION FOR DIFFUSION MODELS WITHOUT EXTRA COSTS
Severi Rissanen Aalto University severi. rissaneneaalto..r fi
Markus Heinonen Aalto University
Arno Solin
Aalto University
ABSTRACT
The covariance for clean data given a noisy observation is an important quan- tity in many training-free guided generation methods for diffusion models. Cur- rent methods require heavy test-time computation, altering the standard diffusion training process or denoiser architecture, or making heavy approximations, We propose a new framework that sidesteps these issues by using covariance informa- tion that is available for free from training data and the curvature of the generative trajectory, which is linked to the covariance through the second-order Tweedie's s formula. We integrate these sources of information using (i) novel method transfer covariance estimates across noise levels and (ii) low-rank updates in a given noise level. We validate the method on linear inverse problems, where it outperforms recent baselines, especially with fewer diffusion steps.
INTRODUCTION
Diffusion models (Sohl-Dickstein et al., 2015; Ho et al., 2020; Song et al., 2021) have emerged as a robust class of generative models in machine learning, adept of producing high-quality samples across diverse domains. These models function by progressively denoising data through an iterative process, learning to reverse a predefined forward diffusion process that systematically adds noise. Conditional generation extends the capabilities of diffusion models by allowing them to generate samples based on specific input conditions or attributes. This conditioning enables more controlled and targeted generation, making diffusion models applicable to-image synthesis or linear inverse problems such as deblurring, inpainting, or super-resolution.
wide range of tasks such as text-
A strand of recent research has concentrated applying pretrained diffusion models accommo- date user-defined conditions, enhancing the flexibility and control of a single model to an arbitrary number of tasks. These methods guide the sampler towards regions whose denoisings p(æ0 x.) are compatible with the condition or constraint, which requires efficient denoising mean E [g x.a and covariance Cov(z, 2x) estimates (Ho et al., 2022; Song et al., 2023a;b; Boys al., 2023; Peng et al., 2024). While estimating the mean is straightforward through the denoiser, accurately deter- mining the covariance has proven more challenging. Consequently, efficient approaches have been proposed with heavy approximations (Chung et al., 2023; Song et al., 2023a).
ORIGINAL
MEASUREMENT
DPS
HIGDM
FH (OURS)
Figure 1: Comparison of different conditional diffusion methods for deblurring, with low number of solver steps (15 Heun iterations). DPS (Chung et al., 2023) and IIGDM (Song et al., 2023a) work well with many steps, but accurate covariance estimates matter more for small step counts.

---

Published as a conference paper at ICLR 2025
(a)
(b)
(c)
(d)
Figure 2: (a) distribution p(zu) represented by a pretrained diffusion model, and a Gaussian likeli- hood p xo). (b) The (exact) posterior p(zo y) ~ p(æ.) pp æo). (c) Generated samples from model with heuristic diagonal denoiser covariance Σo ((x)). and generative ODE trajectory with approximated p(æol (x)) shapes represented ellipses along the trajectory. (d) Generated samples with our denoiser covariance.
In this paper, propose Hunch (FH). The name stems from the core insight that much the required guiding covariance information is, in fact, freely available from the training data and the generative process itself. FH significantly improves accuracy over baselines, it is directly applicable to all standard diffusion mod- els and does not require significant additional compute. This is achieved by integrating of information into a unified framework: (i) the covariance of the data distribution and (ii) the im- plicit covariance information available in the denoiser evaluations along the generative trajectory itself. We apply the method to linear inverse problems, where we show mathematically that ac- curate covariance estimates are crucial for unbiased conditional generation, and achieve significant improvements over recent methods (see Fig. 1). In summary, our contributions are:
method for denoiser covariance estimation, which we refer to Free
sources
We propose novel, efficient method for estimating denoiser covariances in diffuusin models. (i) does not require additional training, (ii) avoids the need for expensive score Jacobian computations, (iii) adapts to the specific input and noise level, and (iv) is applicable to all standard diffusion models.
Methodological:
Analytical: We give theoretical analysis of why accurate covariance estimation is crucial for reconstruction guidance in linear inverse problems.
Practical: Our improved covariance estimates result in significant improvements over baselines in linear inverse problems, especially with small diffusion step counts.
BACKGROUND
Diffusion models are a powerful framework for generative modelling. Given a data distribution p(xo), consider the following sequence of marginal distributions:
p(a:)) = N(xx e [o (()))))((()))) dao.
and corresponding reverse processes (Song et al., 2021; Karras et al., 2022)
Reverse SDE: PF-ODE:
dæ; (t) logp(z;) dt + 2 (t)o dwr, dæ; = (t) p(x; dr.
Here, the d(t) = (t) and wa is Brownian motion. The score Vz. log p(x;) can be learned through score matching methods (Hyvärinen & Dayan, 2005; Vincent, 2011; Song al., 2021). Starting at sample 2, ~ N(x; Zp, of I) at sufficiently high max and integrating either differ- ential equation backwards in time, we recover the data distribution p(z)) the score accurate.
di.
In conditional generation, we need to define the conditional score
V an log, (1) Va, log:( + Vn, log mx);
2

---

Published as a conference paper at ICLR 2025
which decomposes into an unconditional score and the conditional adjustment through Bayes' rule. If we train a classifier to estimate the condition y given the noisy images Tt, we get classifier guid- ance (Song et al., 2021; Dhariwal & Nichol, 2021). Using additional training compute for each con- ditioning task, however, may be prohibitive in some applications. A more modular way to do con- ditional generation is to define constraint ply æn) only the clean data points æo, and estimate
V log xx) = Va, log
p(y) 20 p(ro atx) dac
V log Eptso w.) [p(y] mo)]--
(5)
constraini
demoise
That is, we need to integrate over all possible denoisings of æt and their constraints. We want to avoid costly repeated sampling Zy 22 from the reverse and seek practical approximations to p(aol [+)). A common approach is a Gaussian approximation
p(a0o ax)) (aro Jr, Σ),
which is appealing since the so-called Tweedie's 's formula (Efron, 2011) links the score function Va, log p(x,) to exact moments of the posterior p(ao xx.) p(mo (imo)p(( [ma)
p(w,)
E|wo [x]] = + o x. log p(a;),
Cov(zo [x = of (o. V2. log p(pp( +1)
(8)
The correct mean (Eq. (7)) of the denoiser is directly implied by the score function, as long our es- timate of the score accurate. Estimating the Tweedie covariance through the full Hessian Eq. (8) is very expensive for high-dimennional data, however, and multiple methods have been proposed.
2.1 RELATED WORK
Denoiser covariance estimation in diffusion models Previous attempts to improve denoiser co- variance estimation in diffusion models can be broadly categorized into four categories:
1. Heuristic methods: Many methods (Ho et al., 2022; Song et al., 2023a;b) use a heuristic scaled identity covariance or can be seen as a special case where the covariance is zero (Chung et al., 2023). The methods are simple to implement but may result in biases in the conditional distribu- tions.
Training- based methods: These involve training neural networks directly output covariance estimates (Nichol & Dhariwal, 2021; Meng et al., 2021; Bao et al., 2022a: Peng et al., 2024). While potentially powerful, these approaches are not directly applicable to many existing diffu- sion models.
3. Gradient-based methods: These techniques estimate covariances by computing gradients of the denoiser (Finzi al., 2023; Boys et al., 2023; Rozet et al., 2024; Manor Michaeli, 2024b). However, getting the full covariance is computationally expensive and memory-intensive, mak- ing challenging to apply to nigh-dimensional data without additional approximations. Manor & Michaeli (2024b) propose method to compute the covariance along the top principal compo- nents efficiently.
4. Post-hoc constant variance methods: These approaches optimize constant variances for each time step based on pre-trained diffusion model scores (Bao et al., 2022b; Peng et al., 2024). While they do not require training or significant extra compute, they limited in their ability to adapt to different inputs.
Diffusion models for inverse problems and training-free conditional generation Recent reviews can be found in Daras et al. (2024); Luo al. (2024). Many works explicitly train conditional diffusion models for different tasks (Li et al., 2022; Saharia et al., 2022; Whang et al., 2022).
Many other methods adapt pre-trained diffusion models for inverse problems at inference time. DPS (Chung et al., 2022), IIGDM (Song et al., 2023a), TMPD (Boys et al., 2023) and Peng et al. (2024); Song et al. (2023b); Ho et al. (2022) use backpropagation to explicitly approximate Eq. (5). We focus on these methods because they share a common framework with varying covariance approximations, enabling straightforward comparisons. Other methods modify the generative process to reduce the
3

---

Published as a conference paper at ICLR 2025
residual y Az, in linear inverse problems (Song et al., 2021; Jalal al., 2021; Choi al., 2021). DDS (Chung et al., 2024) and DiffPIR (Zhu et al., 2023) determine guidance direction by optimizing for an 2o that balances proximity to both the measurement and denoiser output. DDNM (Wang et al., 2023) projects the denoised Zn to the null-space of the measurement operator during sampling. Peng al. (2024) show that DDNM and DiffPIR be framed in similar framework. Rout et al. (2024) use second-orde corrections to reconstruction guidance to mitigate biases in first- order Tweedie. Kawar al. (2021; 2022) decompose the linear measurement operator with SVD to create specialized conditional samplers. Methods based on variational inference optimize for zo that match with the observations while having high diffusion model likelihood (Mardani al., 2024; Feng et al., 2023). Ben-Hamu et al. (2024); Wang et al. (2024b) optimize the noise latent æT such that it matches with the observation. The methods by Wu et al. (2024); Dou & Song (2024); Trippe et al. (2023) frame conditional generation and inverse problems with Bayesian filtering perspective, giving asymptotic guarantees with increasing compute.
Other applications of Hessians and denoiser covariances Linhart et al. (2024) demonstrate that Gaussian approximation of p(æ. x3) enables compositional generation between diffusion models. Higher-order ODE solvers leverage the Hessian V2 log p(z;) for efficient sampling (Dockhorn et al., 2022), while Sanchez et al. (2022) use it for causal discovery. Lu et al. (2022) improved model likelihoods by explicitly matching higher-order score gradients. Song & Lai (2024) identified the Hessian as equívalent to Fisher information for measuring step informativeness in conditional generation. Recent advances include efficient Hessian computation using training data (Wang al., 2024a) and unsupervised audio editing via perturbing generation along denoiser covariance principal components (Manor& & Michaeli, 2024a).
3 METHODS
We present our framework for incorporating prior data covariance information with curva- ture information observed during sampling, We define Moj (Zt) and Σg ((x)) as our approxi- mations of IE [2 xy] and Cov(xo æe] at time and location æ. As we move from point (æ, t) (x Ax, + At) in the diffusion process, the denoiser covariance changes but remains similar for small steps. We develop methods to transfer this information across time steps (Sec. 3.1,, incorporate additional curvature in- formation (Sec. 3.2), and combine these up- dates (Sec. 3.3). For high- nigh-dimensional data, propose efficient algorithm using diago- nal and low-rank structures (Sec. 3.4). We dis- cuss covariance initialization (Sec. 3.5) and in- troduce reconstruction guidance with a linear-Gaussiae observation model (Sec. 3.6). Finally, we analyze why diagonal denoiser covariance overestimates guidance for correlated data at large diffu- sion times and demonstrate this issue with image data, showing that the problem is resolved with correct covariance estimation (Sec. 3.7).
Directly measunable during sampling
0 Online estimate available
Obtoined with Gaussian approx.
Obtained low-raok update
Alyy (
samplig stop
pln.t) oo Wi ((a.))
BEGS
Pajien eu (i)
Majrsdil (a Ams
Σg i- Az
2:
Figure 3: Sketch of our method during sampling.
Notation In the following, we interchangeably use p(æ, t) in place of p(æ;) where we want to emphasise the possibility to change either i or i, but not the other. However, in contexts where we talk about the posterior, we use p(xx)) and p(Zo ax)) to emphasise the difference between the two random variables Tg and xt:
3.1 TIME UPDATE
Our goal is to obtain the evolution of the denoiser moments Mo| (1;) and Σg (x)( (Eqs. (7) and (8)). The evolution of the moments under the diffusion process is characterised by the Fokker- Planck equation, and in practise intractable. We approximate the evolution with second-order Taylor expansion of log p(x:) around point æt, which leads to a Gaussian form for p(æ:):
p(ar)) ss N m( 1), C(x,,
4

---

Published as a conference paper at ICLR 2025
where (temporarily dropping out the subscript from æt for clarity)
m( æ. t) = a V logg t) ' Va log log,(x
(10)
C( 1) = - logp(æ,()) [_]p(,))]]
(11)
The evolution of the Gaussian (Eq. (9)) under the linear forward SDE (Eq. (1)) has closed form (Särkkä & Solin, 2019). With the forward process induced by Eq. (1), this results in
m(x,t+ At) m(x, t), C(e,I+ A1) = ((x, () A
(12)
(13)
where Ao² o²(t+ At) (t).. That is, the forward keeps the mean intact while increasing the covariance. Using the above equations derive Tweedie moment updates:
Po)e+ar(2,)) = x, o(t At)²
Ao² Eo 20) o(t+ At)
(Ho) x1). (14)
o(t)
Eo)( = (Eo))(Z1)) Do 10--1)
(15)
The complete derivations can be found in App. A. As At approaches zero, the Gaussian approximation becomes increasingly accurate. This is because the solution the Fokker-Planck equation (which simplifies to the heat equation for variance-explodingg diffusion) is a convolution with small Gaussian [x 0, (t) The integral f p(xx) (xx, z, 0, a (t) dæ, is dominated by values near Z₁. as the Gaussian rapidly diminishes further away.
where 2 -t) o(t)
3.2 SPACE UPDATE FOR ADDING NEW LOW-RANK INFORMATION DURING SAMPLING
We take inspiration from quasi- quasi-Newton methods (e.g., BFGS, see Luenberger et al., 1984) in optimization, where repeated gradient evaluations at different points used for low-rank updates to the Hessian of the function to optimize. The diffusion sampling process is similar: we gather gradient evaluations æz log p(æ;) different locations, and could use them to update the Hessian V2 log p(æ)). The Hessian is then connected to the denoiser covariance via Eq, (8). Here, we derive even more convenient method update Σn| Eojr(a) directly.
To use update rules like BFGS, Eo (z)) should be the Jacobian of some function. Thankfully, notice that Cov (xo æ.] is proportional to the Jacobian of expectation IF/co ær]:
2 (t)² (Va,, w, logp(z;)o + (t)? V E E (t))()))) (V2. log log (t) Cov(x, mz)-
E Ja
(Eq. (7))
(16)
(Eq. (8))
(17)
We can then directly formulate the finite difference update condition equation for our estimate Eaje(e))
a (1) Aa) Mo ((x)) Rs Eo c(x + Ar))Ar.
(18)
This allows to use BFGS-like update procedure for the covariance and inverse covariance
Egj (x Ax) Eo]r(z)) Eo Ax Ea)r(x)
AeAeT Ae Ax'
(19)
Ax Eo
Eg (x + Ax) = --rAAe DEoje(a -1())-((11 AeA + Ar
(20)
where
Ae = o cax Ax) (o)r(x)) and
(21)
Ae Ar
While other update rules exist, BFGS has the advantage that it preserves the positive-definiteness of the covariance matrix. We provide further discussion in App.
5

---

Published as a conference paper at ICLR 2025
Algorithm 1: Time update
Algorithm 2: Space update
Input: En (())) at + (t),
(t)) Mo (2))
Input: Σn (xt). Moi c(x Ax),
Ho e (z). (t)), A.
A(o- = att -
201HAr(Z)) = En (2)))) A (()00))))) 7)I
Ae = Eq. (21)
20le+Ae(x)) = (20((+ (2))
2 y = Eq. (21)
2 + Ax) Eq. (19)
M0)14Ar(z) Eq. (14)
return Eg ( + Ax)
return Eo)etar(Z), Mo (o)++Ar()))
3.3 COMBINING THE UPDATES FOR PRACTICAL SAMPLERS
Note that the update step requires Mo Mo| (xx) evaluated two æ locations but with the same t. Usually, we only have Poj e (æ) at different during sampling, however. The solution is that we can combine the time updates with the space updates in any diffusion model sampler as follows: Let's 's say we have two consecutive score evaluations V, log p(æ, t) and Va log p(æ Az, t At). We first update the denoiser mean and covariance with the time update to get estimates of Eo 1 1+t2e (x) and M4 | (+ Then we can update En e+ (+)) with the BFGS update since we have Mo ++)r(2)) from the time update and to + Ax) from the second score function evaluation and Eq. (7). This is visualized in Fig. 3, and the algorithms for updating the covariance given in Alg. and Alg. 2.
3.4 PRACTICAL. IMPLEMENTATION FOR HIGH-DIMENSIONAL DATA
While the method described so far works well for low-dimensional data, storing entire covariance matrices memory difficult for high nigh-dimensional data. Luckily, this is necessary since we only perform low-rank updates to the covariance matrix. In practice, we keep track of the following representation of the denoiser covariance:
Eo { D UUT VV
(22)
where D is diagonal and U, V low-rank N x l matrices. This structure comes from the two outer products in the the BFGS update (positive and negative). The vectors
and
vāe Av
Eo (m))m
VAx Ax)= become new columns in U and V respectively. In App. B, show that inverting
this matrix structure yields another matrix of the same form: Eoj((x) D' D'+U'" -VV'' Using two applications of the Woodbury identity, this computation only requires inverting matrices rather than N x N ones, enabling efficient caleulation of both Σo + Az) and the time update inverse.
3.5 INITIALISATION OF THE COVARIANCE
Having established methods for representing and updating denoiser covariances, we address initialization. While one might consider the limit / + x:: where p(æ;) - N(x, 0, (1)=T) and V2, lo:.(x:)) + =(1)= this is suboptimal: although the Hessian approaches identity at high i, the denoiser covariance approaches the data covariance. We estimate this from the data and initialise the covariance to For high-dimensional data, we approximate this covariance as diagonal in the DCT basis: (x,,) r Dr DCT- This is justified by natural images being approximately diagonal in frequency bases (Hyvärinen et al., 2009). While alternatives like PCA could used, we found the DCT-based method sufficient. We provide additional discussion on the DCT basis in App. I.
3.6 GUIDANCE WITH A LINEAR- GAUASIAN OBSERVATION MODEL
the observation model p(y wo) is linear-Gaussian, the reconstruction guidance becomes
V æ, logp(y xx) s V log
NN Axo, 1) Ho (xx)), En (xx)) dat
A yy App)((z:) (A A + -1 AV Mo e (xt).
(23)
where A is the linear measurement operator (e.g., blurring). f (xx) obtained using Tweedie's formula, and Eo (#x) Eojz is assumed constant with respect to 2s when taking the derivative.

---

Published as a conference paper at ICLR 2025
The Gaacs-G setting is valuable it both represents many real-world problems (deblurring, inpainting) and provides analytic insights into Σnle choices. We will show that simplistic denoiser covariance approximations lead to severe overestimation of the guidance scale in Eq. (23).
3.7 ISSUES WITH DIAGONAL DENOISER COVARIANCE
In this section, we will focus on DPS (Chung et al., 2023) and IIGDM (Song et al., 2023a) for the linear inverse problem case. Both can be cast as using the same formula with Eajz = vi and different post-processing steps on the resulting v., log p(y xx) approximation:
{o y Amoll
1. In DPS, r 0. The resulting V Va, log æx) is scaled with
(E is hyperparameter).
a(0)² 2. In HIGMM, r I+a(1)²
The resulting Va, logp(y ax)) is further scaled with rī.
Importantly, in the where the resulting E E/zg 32 y) 3; + a (t)) æ. p(x; y approxima- tion is outside the data range [-,, the modified score in both DPS and IIGDM is clipped to keep the denoiser mean within this range. Other choices include r? (t)² (Ho et al., 2022).
The mismatch between simplistic covariance and the denoiser Jacobian Notice that according to Eq. (8), Mo x Covlao mi] where Cov(zo [t]] is the real denoiser covariance. For real data like images, this denoiser covariance is highly non-diagonal due pixel correlations. This creates tension with the inverse (AEo :A + r in Eq. (23), which assumes diagonal Σo
a(t)²
toy model For the denoising task color), giving Cov [oo] = J where J is all ones. As + oc, Cov [#o [.,] + Cov[zo) Assume that the observation y and the denoiser mean No r(x,) are similarly vectors of ones I scaled by constant, and thus y (o)e(2z) = aI. Note that V (e.Mo|/(Z))) a())- J Now, the guidance terms =()² with r = [+o read:
consider images with perfectly correlated pixels (same
J
aN
2. logply x..) = aI
IT
(24)
1 T+ o()2
(1 + +
Here N is the data dimensionality. Two key issues emerge: (1) the per-pixel guidance term scales with the total pixel count, and (2) for typical values (a
of < 1), the guidance becomes implausibly large. For a 1000 1000 image, ((t)² m; logp(y x.) R NI. yielding values around 10 per fixel-far beyond the [- data range. This issue even worse in DPS where r 0. For IGDMM, clipping the denoiser mean to [-1 prevents trajectory blow-up but loses information and introduces biases. The scaling factors in DPS also reduce but do not eliminate the problem.
Solution with the correct covariance In contrast, the same issue does not occur if we use the correct denoiser covariance in the formula:
Σ2 0
10a
E,
Tie Dr
| aa (xx)
V me log p(y æ.) = (y Mo (xx))) (Cov(aro mx]]
10a
+ : Cov|wo [d]
(25)
of
10
Clearly, if oy + 0, the covariances cancel out. Thus, the scale of the calculated guidance does not cause issues. In App. C, we repeat this analysis without assuming Cy 0.
10a
10
10a
10a
Noise schedule, (t))
Fig. 4 showcases the issue in practice for a Gaus- ImagetNet 256x256 and denoiser from Dhariwal & Nichol (2021). In comparison, the problem is less severe for a more sophisticated DCT-diagonal covariance approxima- tion. However, even the DCT-diagonal method does cause the adjusted denoiser mean diverge at high noise levels.
sian blur operator
Figure
4: Norm of (o)(()) + (t) log ply x;) for different co- variance estimation methods 256x256. Values > indicate overestima-
ImageNet
tion since the data is normalized [-1, 1].

---

Published as a conference paper at ICLR 2025
J5 Div: 0.278
JS Div: 0.252
Div: 0.256
IS Div. 0.171
JS Div: 0.142
(a) DPS
(b) ITGDM
(c) [TGDM (no : scaling)
(d) FH (Ours)
(c) Optimal cov
Figure 5: Different methods for posterior inference in the example in Fig. and Jensen Shannon divergences to the true posterior.
Approximating V. Moje ( Given that the problem stems from the mismatch between Vo, and our covariance approximation, solution needs to harmonize these two terms. Thus, propose to further approximate V ( by our denoiser covariance estimate when the scale the guidance by calculating the full Jacobian is too large. In practice, we first calculate the adjusted denoiser covariance estimate (x))) and fall back to approximating
Nog (xxe) s would push the trajectory in a direction that outside the data range 1))
Eo))(as) in case our initial approximation [|a(()) log æt)ll (which
o())
Full algorithm. In App. D, we show full algorithms for linear inverse problems with our covariance estimation method, one with the Euler ODE solver and another that works with any solver.
EXPERIMENTS
We validate our method using synthetic Gaussian mixture model data and compare it against base- lines on linear imaging inverse problems. Our experiments demonstrate that our more sophisti- cated covariance approximations reduce bias and improve results, particularly at lower diffusion step counts. We use a linear schedule (t) t. as advocated by Karras et al. (2022), and follow their settings for our image diffusion models otherwise as well. For the image experiments, we used max 80 and a max 20 for the synthetic data. We use simple Euler sampler for the synthetic data experiments and 2nd order Heun method (Karras et al., 2022) for the image experiments.
SYNTHETIC DATA EXPERIMENTS
Toy data We first showcase the performance of different methods toy problem using mixture of Gaussians distribution, which admits a closed-form formula for the score (see App. M). The results in Fig. 5 show that method clearly outperforms DPS and IIGDM, approaching the method using optimal covariance obtained by backpropagation and Eq. (8). Note that the example favours DPS, since tuned the guidance hyperparameter for this particular task.
The effect of dimensionality and correlation In Sec. 3.7, we noticed that the guidance scale is overestimated the larger the dimensionality is. practical consequence is that the variance of the generative distribution can be underestimated. In App. E, we directly showcase this with synthetic data and show that it does happen with our method.
Approximation error in the covariance In App. G, we analyse the error in the covariance approx- imation for diweensiiionll example, and empirically show that the error approaches zero with large amount of steps and a stochastic sampler.
4.2 IMAGE DATA AND LINEAR INVERSE PROBLEMS
We experiment on ImageNet 256x256 (Deng et al., 2009) with an unconditional denoiser from Dhariwal & Nichol (2021). We evaluate the models four linear inverse problems: Gaussian deblurring, motion deblurring, random inpainting, and super-resolution. We evaluate our models with peak signal-to-noise ratio (PSNR), structural similarity index measure (SSIM, Wang et al., 2004) and learned perceptual image patch similarity (LPIPS, Zhang et al., 2018) on the ImageNet test set. We the same set of 1000 randomly selected images for all models.
8

---

Published as a conference paper at ICLR 2025
We solve the inverse in Eq. (23) using conjugate gradient, following Peng et al. (2024). Our custom PyTorch implementation uses GPU acceleration and adjusts solver tolerance based on noise levels. This optimization removes the solver bottleneck without noticeable performance loss. Details are in App. J.
0.6
Identity base covariance (DCT-diaaonal covariance)
Optimal guidance without scaling
Proposed models We introduce four new methods. The first, 'Identity". is initialized with identity covariance. "IdentityOnlline also uses the space updates. 'FH' is initialized with data covari- ance projected a DCT-diagonal basis. Finally, "FH+Online" enhances 'FH' with online updates.
0.5
Baselines We compare against several methods using Eq. (23) for linear imaging inverse prob- lems: DPS (Chung et al., 2023), IIGDM (Song et al., 2023a), TMPD (Boys et al., 2023), and two methods from Peng et al. (2024) Peng (Convert) and Peng (Analytic). TMPD uses vector-Jacobian product I V. (o,))(()) for denoiser covari- ance. Convert employs neural network-output pixel- space diagonal covariance, while Analytic deter- mines optimal constant pixel-diagonal covariances per timestep through moment matching. These
0.8
0.8
1.2
Guidance strength
Figure 6: LPIPS w.r.t. guidance strength for the ImageNet validation set and the Gaussian blurring task. With a better covariance ap- proximation, the usefulness of adjusting the approximated guidance Væ, with post-hoc tricks becomes smaller.
methods were selected as they represent reconstruction guidance with different covariances, en- abling analysis of our covariance approximation approach. For DPS, we optimized guidance scale via ImageNet validation set sweeps. Non-identity covariance models used SciPy's 's conjugate gradi- ent method for solving Eq. (23). For TMPD, we adjusted tolerance higher noise levels reduce generation time (see App. J).
Scaling the guidance term We investigated how covariance approximation affects need for post-hoc changes to the estimated gradient Va, log,(yy æx,), as shown in Fig. 6. For deblurring, the cruder identity initialization required scaling slightly below 1, indicating an initial overestimation of the guidance scale. The more sophisticated DCT-diagonal covariance (FH) showed no systematic over- or underestimation, with optimal scaling at We determined optimal guidance strength for identity covariance through small sweep of 100 ImageNet validation samples different solver step counts. No scaling was applied for DCT--iagonal covariance. Additional analysis with PSNR and SSIM is provided App. K.
Baseline comparisons Our experiments focus on the low ODE sampling step regime to ensure practical applicability. Results in Table 1 show that adding online updates during sampling improves performance, with even greater gains when using DCT-diagonal covariance instead of the identity base covariance. On low step counts, our FH models consistently outperform baselines across all metrics, particularly on LPIPS scores. Visual comparisons in Fig. 7 and Fig. 9 confirm the effective fine detail preservation of FH at 15 and 30 steps. Extended results with 50 and 100 steps and the Euler solver in App. H, including comparisons DDNM+ (Wang et al., 2023) and DifIPIR (Zhu et al., 2023), show and FH+Online almost always outperforming others at low step counts and typically achieving the best LPIPS scores even at higher step counts.
non-reconstruction guidance methods
5 CONCLUSIONS
We introduced Free Hunch (FH), framework for denoiser covariance estimation in diffusion mod- els that leverages training data and trajectory curvature. FH provides accurate covariance estimates without additional training, architectural changes or ODE/SDE solver modifications, Our theoret- ical analysis showed that incorrect denoiser covariances significantly bias linear inverse problem solutions. Experiments on ImageNet demonstrated strong performance in linear inverse problems, especially low step counts, with excellent LPIPS scores and fine detail preservation.

---

Published as a conference paper at ICLR 2025
While FH introduces some additional complexity compared to simpler approaches, its efficiency and adaptability make it promising for various conditional generation tasks. Limitations our work include the focus of the theoretical and experimental analysis linear inverse problems, and future work could investigate nonlinear inverse problems and other types of conditional generation. Another open question is whether can derive error bounds the accuracy of the estimated covariance matrix in the entire process, or within individual 'time updates' or "space updates'. While our DCT-diagonal base covariance works well for image data, the application to other data domains is another open question. A low-rank estimate of the covariance matrix with a PCA decomposition seems like generally applicable approach, but this remains be validated in practice.
Code for our approach is available at hstps:: /ggtttbt.bb/...// com/ www.caot./a///ecaaacaaeeaoaoaa Aww./aalaoooaa/o streehhnch
Table 1: Comparison of image restoration methods for 15- and 30-step Heun iterations for deblurring (Gaussian), inpainting (random), deblurring (motion), and super-resolution (4x) tasks. Our method (FH) excels overall, especially in the descriptive LPIPS metric. The top scores per category are bolded, with runners-up underlined. Close scores share rankings.
Deblur (Gaussian)
Inpainting (Random)
Deblur (Motion)
Super res. (4x)
Method
PSNRT
SSIMT
LPIPS
PSNR
SSIM
LPIPS..
PSNK"
SSIM
LPIPS]
PSNR
SSIMT
LPIPS]
DPS
19.94
0.444
0.572
20.68
0.494
0.574
17.02
0.354
0.646
19.85
0,460
0.590
HGGDM
20.30
0.475
0.574
19.87
0.468
0.598
19.21
0.429
0.602
20.17
0.474
0.582
23.08
20.80
0.491
Peng (Convert)
22.55
0.563
0.490
22.23
0.579
0.489
20.46
0.475
0,556
21.92
0,541
0.517
Peng (Analytic)
22.53
0.563
0.490
22.14
0.574
0.494
20.46
0.475
0.556
21.92
0.541
0.517
1.5
Identity
22.91
0.594
0.384
18.83
0.397
0.590
20.06
0.393
0.506
22.65
0.589
0.412
Identity+online
23.08
0.606
0.385
18.86
0.397
0.590
20.31
0,418
0.492
76
0.597
0.414
FH
23.39
0.624
0.372
24.73
0.701
0.327
21.69
0.534
0.446
23.30
0.624
0.390
FH+online
23.54
0.634
0.378
25.25
0.728
0.317
21.84
0.549
0.441
23.39
0.632
0.394
DPS
21.76
0.527
0.463
24.84
0.678
0.387
18.22
0.389
0.582
23.00
0.593
0.440
ITGDM
22.27
0.559
0.468
21.24
0.518
0.517
21.16
0.508
0,503
22.11
0.553
0.478
TMPD
23.16
0.602
0.415
18.85
0.481
0.537
20.91
0.500
0.507
21.94
0.549
0.472
30 sleps
Peng (Convert)
Peng (Analytic)
23.61
23.61
0.626
0.627
0.405
23.59
0.640
0.411
21.99
21.99
0.552
0.463
23.21
0.608
0.430
Identity
23.15
0.602
0.374
18.75
0.402
0.578
20.14
0.406
0.494
22.82
0.588
0.405
sonline
23.38
0.621
0.359
20.07
0.443
0.529
20.47
0.420
0.467
23.38
0.622
0.383
FH
23.55
0.630
0.353
26.00
0.757
0.256
21.80
0.538
0.411
23.38
0.623
0.372
FHtooninnc
23.62
0.635
0.358
26.18
0.767
0.268
21.88
0.547
0.410
23.44
0.628
0.375
Condition
Forward
TMPD
Peng (Convert)
(Ours) FH +Online (Ours)
Motion Blur Gaussian Blur
Inpuinting
Crufts
Grifts
C
C
Crintiss
Criiitts
Grufts
Crufts
Super res.
Figure 7: Qualitative examples using the 15-step Heun sampler for image restoration methods for deblurring (Gaussian), inpainting (random), deblurring (motion), and super-resolution (4x) tasks. Quantitative metries in Table I. Our method manages to restore the corrupted ( ('Forward') to match well with the original ('Condition').
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
We acknowledge funding from the Research Council of Finland (grants 339730, 362408, 334600). We acknowledge CSC IT Center for Science, Finland, for awarding this project access to the LUMI supercomputer, owned by the EuroHPC Joint Undertaking, hosted by CSC (Finland) and the LUMI consortium through CSC. We acknowledge the computational resources provided by the Aalto Science-IT project.
REFERENCES
Fan Bao, Chongxuan Li, Jiacheng Sun, Jun Zhu, and Bo Zhang. Estimating the optimal covariance with imperfect mean in diffusion probabilistic models. In International Conference on Machine Learning (ICML), pp. 1555-1584, PMLR, 2022a.
Fan Bao, Chongxuan Li, Jun Zhu, and Bo Zhang. Analytic-DPM: An analytic estimate the opti- mal reverse variance in diffusion probabilistic models. In International Conference on Learning Representations (ICLR), 2022b.
Heli Ben- Omri Puny, Itai Gat, Brian Karrer, Uriel Singer, and Yaron Lipman. D-Flow: Differentiating through flows for controlled generation. In International Conference on Machine Learning (ICML), pp. 3462-3483, PMLR, 2024.
Benjamin Boys, Mark Girolami, Jakiw Pidstrigach, Sebastian Reich, Alan Mosca, and O Deniz Akyildiz. Tweedie moment projected diffusions for inverse problems. arXiv preprint arXiv:2310.06721, 2023.
Jooyoung Choi. Sungwon Kim, Yonghyun Jeong. Youngjune Gwon, and Sungroh Yoon. Ilvr: Con- ditioning method for denoising diffusion probabilistic models. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 13367-14376, 2021.
Hyungjin Chung. Byeongsu Sim, Dohoon Ryu, and Jong Chul Ye. Improving diffusion models for inverse problems using manifold constraints. In Advances in Neural Information Processing Systems 35 (NeurIPS), pp. 25683-25696. Curran Associates, Inc., 2022.
Hyungjin Chung, Jeongsol Kim, Michael Thompson Mccann, Mare Louis Klasky, and Jong Chul Ye. Diffusson posterior sampling for general noisy inverse problems. In International Conference on Learning Representations (ICLR), 2023.
Hyungjin Chung, Suhyeon Lee, and Jong Chul Ye. Decomposed diffusion sampler for accelerating large-scale inverse problems. In International Conference on Learning Representations (ICLR), 2024.
Giannis Daras, Hyungjin Chung, Chieh-Hsin Lai, Yuki Mitsufuji, Peyman Milanfar, Alexan- dros G. Dimakis, Chul Ye, and Mauricio Delbracio. A survey on diffusion models for in- verse problems. 2024. URL diffusion_ wwr.oonssuuueeueee pdf.
sdaras. github. ioc htt.ppiiicaaiiiinnioomimoccc
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 248-255. IEEE, 2009.
John Dennis, Jr and Jorge J Moré. Quasi-newton methods, motivation and theory. SIAM Review, 19(1):46-9, 1977.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Ad- vances in Neural Information Processing Systems 34 (NeurIPS), pp. 3780-8794. Curran Asso- ciates, Inc., 2021.
Tim Dockhorn, Arash Vahdat, and Karsten Kreis. Genie: Higher-order denoising diffusion solvers. In Advances in Neural Information Processing Systems 35 (NeurIPS), pp. 30150-30166. Curran Associates, Inc., 2022.
11

---

Published as a conference paper at ICLR 2025
Zehao Dou and Yang Song. Diffusion posterior sampling for linear inverse problem solving: A filtering perspective. International Conference on Learning Representations (ICLR), 2024.
Bradley Efron. Tweedie's ' formula and selection bias. Journal of the American Statistical Associa- tion, (06(496):1602-1) 1614, 2011.
Berthy Feng, Jamie Smith, Michael Rubinstein, Huiwen Chang, Katherine L Bouman, and William T Freeman. Score-based diffusion models principled priors for inverse imaging. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 10520- 10531, 2023.
Mare Anton Finzi, Anudhyan Boral, Andrew Gordon Wilson, Fei Sha, and Leonardo Zepeda-Núñezz User-defined event sampling and uncertainty quantification in diffusion models for physical dy- namical systems. In International Conference on Machine Learning (ICML), pp. 10136-10152. PMLR, 2023.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems 33 (NeurlPS), pp. 6840-685 Curran Associates, Inc., 2020.
Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David Fleet. Video diffusion models. In Advances in Neural Information Processing Systems 35 (NeurIPS), pp. 8633-8646. Curran Associates, Inc., 2022.
Aapo Hyvärinen and Peter Dayan, Estimation of non-normalized statistical models by score match- ing. Journal of Machine Learning Research, 6(4), 2005.
Aapo Hyvärinen, Jarmo Hurri, and Patrick O Hoyer. Natural Image Statistics: A Probabilistic Approach to Early Computational Vision, volume 39. Springer Science & Business Media, 2009.
Ajil Jalal, Marius Arvinte, Giannis Daras, Eric Price, Alexandros G Dimakis, and Jon Tamir. Ro- bust compressed sensing mri with deep generative priors. In Advances in Neural Information Processing Systems 34 (NeurIPS), pp. 19938-149544 Curran Associates, Inc., 2021.
Tero Karras, Miia Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- based generative models. In Advances in Neural Information Processing Systems 35 (NeurIPS), pp. 26565-26577. Curran Associates, Inc., 2022.
Bahjat Kawar, Gregory Vaksman, and Michael Elad. Snips: Solving noisy inverse problems stochas- tically. In Advances in Neural Information Processing Systems 34 (NeurIPS), pp. 21757-21769. Curran Associates, Inc., 2021.
Bahjat Kawar, Michael Elad, Stefano Ermon, and Jiaming Song. Denoising diffusion restoration models. In Advances in Neural Information Processing Systems 35 (NeurIPS), pp. 23593-23606. Curran Associates, Inc., 2022.
Haoying Li, Yifan Yang, Meng Chang, Shiqi Chen, Huajun Feng. Zhihai Xu, Qi Li, and Yueting Chen. SRDiff: Single image super-resolution with diffusion probabilistic models. Neurocomput- ing, 479:47-59, 2022.
Julia Linhart, Gabriel Victorino Cardoso, Alexandre Gramfort, Sylvain Le Corff, and Pedro LC Rodrigues, Diffusion posterior sampling for simulation-bar inference in tall data settings. arXiv preprint arXiv:2404.00 414.:2444..7593,, 2024.
Cheng Lu, Kaiwen Zheng, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Maximum likelihood training for score-based diffusion odes by high order denoising score matching. In International Conference on Machine Learning (ICML), pp. 14429-14460. PMLR, 2022.
David G Luenberger, Yinyu Ye, et al. Linear and Nonlinear Programming, volume 2. Springer, 1984.
Ziwei Luo, Fredrik K Gustafsson, Zheng Zhao, Jens Sjölund, and Thomas Schön. Taming diffu- sion models for image restoration: A review. arXiv preprint arXiv:2409. 10353, 2024.
12

---

Published as a conference paper at ICLR 2025
Hila Manor and Tomer Michaeli. Zero- shot unsupervised and text-based audio editing using DDPM inversion. In International Conference on Machine Learning (ICML), pp. 14603-34622 PMLR, 2024a.
Hila Manor and Tomer Michaeli. On the posterior distribution in denoising: Application to uncer- tainty quantification. In International Conference on Learning Representations (ICLR), 2024b.
Morteza Mardani, Jiaming Song, Jan Kautz, and Arash Vahdat. A variational perspective solving inverse problems with diffusion models. In International Conference on Learning Representa- lions (ICLR). 2024.
Chenlin Meng, Yang Song, Wenzhe Li, and Stefano Ermon. Estimating high order gradients of the data distribution by denoising. In Advances in Neural Information Processing Systems 34 (NeurIPS), pp.
Curran Associates, Inc., 2021.
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning (ICML). pp. 8162 8171. PMLR, 2021.
Xinyu Peng. Ziyang Zheng, Wenrui Dai, Nuoqian Xiao, Chenglin Li, Junni Zou, and Hongkai Xiong. Improving diffusion models for inverse problems using optimal posterior covariance. In International Conference on Machine Learning (ICML). pp. 40347-40377 PMLR, 2024.
Litu Rout, Yujia Chen, Abhishek Kumar, Constantine Caramanis, Sanjay Shakkottai, and Wen- Sheng Chu. Beyond first-order tweedie: Solving inverse problems using latent diffusion. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 9472-9481, 2024.
François Rozet, Gérôme Andry, François Lanusse, and Gilles Louppe. Learning diffusion priors from observations by expectation maximization. arXiv preprint arXiv:2405.. 137 2024.
Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David Fleet, and Mohammad Norouzi. Image super-resolution via iterative refinement. IEEE Transactions Pattern Analysis and Machine Intelligence, 45(4):4713 (743))4447777266 4726,, 2022.
Pedro Sanchez, Xiao Liu, Alison Q O'Neil, and Sotirios A Tsaftaris. Diffusion models for causal discovery via topological ordering. In International Conference on Learning Representations (ICLR), 2022.
Simo Särkkä and Amo Solin. Applied Stochastic Differential Equations, volume 10. Cambridge University Press, 2019.
Jascha Sohl-Dickstein. Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learn- ing (ICML), pp. 2256-2265. PMLR, 2015.
Jiaming Song, Arash Vahdat, Morteza Mardani, and Jan Kautz, Pseudoinverse-guidee diffusion models for inverse problems. In International Conference on Learning Representations (ICLR), 2023a.
Jiaming Song, Qinsheng Zhang, Hongxu Yin, Morteza Mardani, Ming- Yu Liu, Jan Kautz, Yongxin Chen, and Arash Vahdat. Loss-guided diffusion models for plug-and-play controllable generation. In International Conference on Machine Learning (ICML), pp. 32483-32498. PMLR, 2023b.
Kaiyu Song and Hanjiang Lai. Fisher information improved training-free conditional diffusion model. arXiv preprint arX1v:2404.18252, 2024.
Yang Song, Jascha Sohl-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In Interna- tional Conference on Learning Representations (ICLR), 2021.
Brian L Trippe, Jason Yim, Doug Tischer, David Baker, Tamara Broderick, Regina Barzilay, and Tommi S Jaakkola. Diffusion probabilistic modeling of protein backbones 3D for the motif- scaffolding problem. In International Conference on Learning Representations (ICLR), 2023.
13

---

Published as a conference paper at ICLR 2025
Pascal Vincent. A connection between score matching and denoising autoencoders. Neural Compu- tation, 23(7):1661-1674, 2011.
Gregory K Wallace. The JPEG still picture compression standard. Communications of the ACM, 34 (4):30-44, 1991.
Fangyikang Wang. Hubery Yin, Huminhao Zhu, Shaobin Zhuang, Chao Zhang, Hanbin Zhao, Hui Qian, and Chen Li. Gradient-free analytical fisher information of diffused distributions. OpenRe- view preprint, 2024a.
Hengkang Wang. Xu Zhang, Taihui Li, Yuxiang Wan, Tiancong Chen, and Ju Sun. Dmplug: A plug- in method for solving inverse problems with diffusion models. arXiv preprint arXiv:2405.16749, 2024b.
Yinhuai Wang, Jiwen Yu, and Jian Zhang. Zero-shot image restoration using denoising diffusion null-space model. In International Conference on Learning Representations (ICLR), 2023.
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero Simoncelli. Image quality assessment: From error visibility to structural similarity. IEFE Transactions on Image Processing, 13(4): 600-612, 2004.
Jay Whang, Mauricio Delbracio, Hossein Talebi, Chitwan Saharia, Alexandros G Dimakis, and Pey- man Milanfar. Deblurring via stochastic refinement. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 16293-16303, 2022.
Luhuan Wu, Brian Trippe, Christian Naesseth, David Blei, and John P Cunningham. Practical and asymptotically exact conditional sampling diffusion models. In Advances in Neural Informa- tion Processing Systems 36 (NeurIPS). Curran Associates, Inc., 2024.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness deep features as a perceptual metric. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 586-595 1886-99,777 2018.
Yuanzhi Zhu, Kai Zhang, Jingyun Liang, Jiezhang Cao, Bihan Wen, Radu Timofte, and Luc Van Gool. Denoising diffusion models for plug-and-play image restoration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1219-1229, 2023.
14

---

Published as a conference paper at ICLR 2025
APPENDICES
A FULL DERIVATIONS FOR THE TIME UPDATE
For this section, we denote p(æ,) log p( a. f) to explicitly separate the time variable from the spatial variable. This notation is useful for the derivations below, but other parts of the paper we use the In to separate it from Zo.
Recall the mean and covariance of the Gaussian approximation location æ and time
m(x; t) = a V log, t) VV log (o,,(t,)). t), C( t) = - log p(z, t)-
(26)
(27)
The evolution the Gaussian has a closed form (Särkkä & Solin, 2019). In the variance-exploding case this results in
m(x,t+ At) = m(x,t) t), C(a,++ At) = C(x, 1) + A
(28)
(29)
where Ag o + At) - (1). That is, the forward keeps the mean intact while increasing the covariance. Using the above equations we can derive:
x logp(a, + At) = (V2 æ p(a t) 20²T)
(30)
V log t At) v2 log t + At) log log V x p(æ, t)
(31)
When connected with Equation (7) and Equation (8), we can now derive the denoiser mean and covariance updates.
Mo (0))+a((z)) a + d + At logp(o, At) = x o + At) 2 log logp(x,t) -o log log p(x; t)
Hessian projection
Hessian-score product
(32)
Ao? Eo ((x))
M0)44r(()) = + At) (o(t)) (o(t))+AA0) =00)) o(t)2
(HO) (00l((())))) -x)) =x),
2
= x +o+ol + At)2 a1t+
o(1) 20)((())
(Ho)e(a)) - x),
(33)
Eg (0)())((0) Ev (x)) Ao- 11=
(34)
This result is not entirely obvious, and next we will provide detailed derivation.
Deriving the denoiser covariance update With the locally Gaussian approximation on p(æ, t), we can represent the time evolution of the p(æ, f) covariance as the following
C(x,, t) = Co + (t))2
(35)
where Cp is the hypothetical covariance when extrapolating the Gaussian time evolution to f. Then, moving back to the a notation, we have the following connections (Eq. (11))
0.
V2 log p(x.)
(Co (1)
(36)
x. V2
æ, logp(a;;)
(C
(37)
On the other hand, the denoiser covariance is Cov(æo 21 = (V', log (t) (1))))
(38)
The inverse of the denoiser covariance is then
formula):
Cov(aro [xx] = (V2 Φ: (0sppam)o((0)++
(39)
= (I - (I+
(Woodbury)
(40)
(I-
o(1)= (37)
(41)
-(++C)o00))011
(42)
Co -
(43)
15

---

Published as a conference paper at ICLR 2025
So the inverse of the denoiser covariance is simply constant term plus identity scaled with o (t) -2 This means that
Cov(aro #e+ae] Cov zo [+s] -1 (Co + Io(t + A1) (C5 Ia(t) -)
(44)
Cov(æo xctsel -1 Cov(2o xt| + Io + At) Io(l)=
(45)
Cov(2o xz (t) ²I.
(46)
And thus
Covlaro re+ad] = (Cov( (zo xx +
(o( (1)-11)-1
(47)
Deriving the denoiser mean update We want to calculate the expression for the updated mean (o))) This mean can be written as:
Mo| (o) æt (t + At)² (V2, log - V2 log w. log p xz
(48)
We aim simplify the term inside the parentheses. Starting from the observation:
(V2 log Ao²1) -1 Vzz æe log (ogp(a;))- (I Ao2 log
(49)
and using Eq. (7) and Eq. (8), we can express V log p(r)( and V., logg( as functions of Eoj (2r)) and o (t), yielding:
(o)) = 2t + olt At)² (o(0)²+
Ao? o(1)2 Eo 201 (Ho)e(())) xz)
(50)
This is the final simplified expression for (o)
B INVERTING THE EFFICIENT MATRIX REPRESENTATION
Let's say we have a positive-definite matrix in represented in the format C = D VVV where D is diagonal matrix and U and V are N x kn and N k/ matrices, respectively. N is the data dimensionality and h < N. To invert it, we use the Woodbury identity twice, first for A D UU and second for A VV The first application of the identity is:
A = )-1 D -
- D [ UIUU
D U UUD
D-! D-1
(51)
=
=D D D UUqqt(K)) TTD!
D-1
(52)
D - V'VIT
(53)
that is, when we invert A = D UUT, we get something in the form D V'V"T Note that UT D- is kn x kn matrix, instea of an N x N matrix and as such much more efficient to invert than the full N x N matrix when kn & N. Now, invert C A VV
c-t= (A -V - - V A - -1
(54)
= A -1 + A V [L sqrt VTA! A-
(55)
A - - V'v".
(56)
Note that V A-1V is again efficient compute due the low-rank structure of A-1:
VAA-VV V (D-1_ (D-I-V'v') V
(57)
VVVV-VVVV' -VID-'V (V (VTV'VVV'V) 12x
(58)
(59)
l2 k2
lzz dzz
16

---

Published as a conference paper at ICLR 2025
This leads well-behaved hS x hy matrix, and the inverse (I- V A V is also efficient to compute.
The matrix square root and complex numbers Note that in addition to the 11 K1 and k2 k2 inverses, the method requires matrix square root. One might imagine that (I + U D U)- is guaranteed be positive-defintt due to the original matrix being such, but this is not necessarily the case. (I + U SUTT U) would be, but the diagonal term D can push the eigenvalues of the matrix negative. This means that we can not use the Cholesky decomposition for the matrix square root operations, but instead we use the Schur decomposition as implemented in the scipy library. A side- effect also that have to use complex numbers represent D. U, and V our implementation. This is not an issue, since in the calculation of the covariance D + UU VV T, the imaginary components cancel out and we get real matrix.
C EXTENDED ANALYSIS OF THE TOY EXAMPLE
As recap, in the toy model, A I and all the pixels are perfectly correlated with Cov[æo] J, where J is matrix full of ones. The observation and the denoiser mean M4) (o)e( are also vectors of ones I scaled by constant, so that y Mo al.
ITGDM guidance without postprocessing
(1)2
(y Ho (2)) (I
+ 1 -V (x))
1+ o(t)
~ (y Mo)e((z)) (I o Covlwr x
(60)
(t)² rT.
J
oN
(61)
O att)
(1 o (t)
DPS guidance without postprocessing
(y - Mo (2)))
(I0 + 02 Va,Ho x(x))
Covl xzd]
ly - Mo)c((2))) 0/-
(62)
(t)
aN IT.
(63)
ot12 (t)
Here N is the data dimensionality.
For IIGDM, the gradient is scaled by T+a(ey= but this does not change the result in high noise to [-1, regularises the guidance such that the generation trajectory does not blow up. For DPS, the additional scaling results in
ott)
levels. Instead, the clipping of the denoiser
Eoy
N -i
E i=
& i= {VNi VN
(t)) P(y xx) R
(64)
1 Awol
og
which is less severe than TIGDM, but still requires additional clipping unless the scale & is set to very low values.
Solution with the correct covariance In the main text, we showed that the issue does not show up in the case o, =0 0. This resulted in:
v. logp(y) œ.) (y xo(xx)) Cov(2o xx]- Cov(xg x
(y
(x)))
(65)
of
oo
The corresponding 2n estimate is:
St + of Va, logg(x:))))))))..))))))))))))l + (y No (xx))
of
=E(zo x + y
Elxco xx]] y
(66)
17

---

Published as a conference paper at ICLR 2025
that is, the updated score points the direction of the observation for all time steps For the case t - oo and Cov(an æz] == and not assuming By = O, we can derive with the Sherman therman-Morrison
formula that
1 0 0g
(Cov|æo xx, - (J + o
I No
J.
(67)
To simplify formulas, let again assume that y =00((0)) = =
J
Va, logply x4) =al
o + Noo 1 oo
(68)
6
N
J
I
(69)
02
+ Not
o7
IT
(70)
N
N N
(71)
0f
Here, again, since the inverse term is inversely dependent on N, the dependence of the last term on N is cancelled. In the case og = O, we recover the exact same result as previously. With non-zero observation noise, the strength of the guidance becomes slightly smaller, reflecting the uncertainty about the underlying pure 20 value we have measured.
D FULL GUIDANCE ALGORRIHMM FOR THE LANEAR-GAUSSIA OBSERVATION MODEL
The algorithm Alg. 3 details an implementation the method with the Euler ODE solver. The algorithm Alg. 4 is higher-order methods like the Heun method. In it, we instantiate a class at the beginning of sampling, and whenever call the denoiser / score model is made, it is passed the class calculate Va. log p(x,) and update the covariance information. For image data, we only perform the space updates 1 < (t) < as detailed in App.
easily applicable implementation with any type of solver, including
E ADDITIONAL TOY EXPERIMENT WITH CORRELATED DATA
To examine the effect mentioned in Sec. 3.7, we constructed data p(z0) = N(xo 0, (11 where J is matrix of ones and p 0.999. We used observation with noise σg = 0.2 and varied the dimension. We plot the variance the generated samples Fig. 8. As expected, both IIGDM and DPS become overly confident dimensionality increases. In contrast, explicitly accounts for data covariance, maintains correct uncertainty calibration across dimension counts. Note that the DPS results obtained after tuning the guidance scale for this particular problem, making the comparisons somewhat favourable towards DPS.
method, which
F ADDITIONAL QUALITATIVE RESULTS
Figure 9 shows the qualitative comparison with 30 Heun solver steps.
G QUANTIFYING THE ERROR IN THE COVARIANCE ESTIMATION
We study the where directly estimate the error in the denoiser covariance estimates for low- dimensional toy data, where this directly feasible. We consider a 2D Gaussian mixture model with an likelihood function and posterior shown in Fig. 10.
We generate samples with four different methods, and compare the covariances true to the true covariance with the Frobenius norm. The methods are:
18

---

Published as a conference paper at ICLR 2025
Algorithm: 3: Free Hunch for Linear Inverse Problems (Euler solver, with diffusion parameters
from (Karras et al., 2022))
Input: Linear operator A, observation y, noise By
Input: Initial covariance Edat, score model
Input: Schedule params: min mmnn 0.002, max tmxx 80, p steps N
/* Define time discretization
= (Amax + N-T' min
(max ) for < N, IN 0 (Karras et al., 2022)
Initialize 2t (0,00
Initialize 20 (0)o(2r) (2xe) data
Initialize 1 0
transferred = null
Initialize A null do
for i I,...,N
Cun = t4s mext f++1
At = li+1 - t;
(+ New score and denoising mean evaluation Vx, logp(æ;) (1),44)
Mo (=())) Hi + deun Va, ((;)
/* Space update for covariance mansferred #null And A # null then Pol4,
11
Ae = o2 (Ho) - Mol tmnnferre)
13
7 De Am
Eg ( = Eo)e(ax) (o) T (e(((())) Ax Eo (2))
+ Aede de
14
15
end
/* Reconstruction guidance
Væ, logp(y(æ;) (y (AE
+ - AV 2. ((xr)
16
/* Fall back to approximation if guidance too large I Va, logp(y x,)]l then
17
V., loga(v() (y- Apo)((a.)) ((AE))
1 A (())) our
19
end
/* Update sample with Euler step
29
Ar - ouur Va, log ((z;) + Va, log p(gl())z
x, + Ar
/* Time update for mean
Ag² = o o2
22
23
Mo 1it1
transferred Dt +
Ag" E -o))(=)))-1(44 (Ho))(()) -xx)
xx)
/+ Time update for covariance (moving to noise level )next) ) +
2(o-2)- --2 x-2
(x)) [Z)((()) A(o-2)I
Eoje(are) = -1))
end
28
return æt
19

---

Published as a conference paper at ICLR 2025
Algorithm 4: Free Hunch Guidance Class, applicable with any solver
class FreeHunchGruidance:
/* Initialize with measurement model and data covariance +/ constructor(A, y, Jg. dan
Store A, y. yy 2 = dam
Initialize Hprev = null, Fprer = null, Opev = null
/* Process new denoiser evaluation and return guidance, to be used for updating Vz, pog)(z;) to Va, logp(x;) + Va, log p(y a4))
before using it in the solver. function process denoiser((Mnew) Unew, Onew):
if perev o null then
/* Time update from previous step
A(o new prev
Ao Ine prev
2011 201 + A(aa
10
Eo1: = (2 o
/* Transfer previous mu to new noise level
Mransterred Ppee + new (oex Ao²
E01z) (Mprev Fprev) of
11
/* Space update
12
Ax Unew Bprev
Ae 22 (Inew (reanffmood)
14
7 Ae Ax
15
E01: Σoje-
Ann Eols
Ar En Am
AedeT T Ae An
16
end
/* Calculate reconstruction guidance
v log Logp(z)(Znew) = (y Appew) (A ( A + - an
17
/* Fall back
approximation if guidance too large
if o
new Va log Loy knew)) || then
V log ply [new)= = (y Afuew) (AE A + +o Epit
19
end
Update staze variables
21
Mpryv Fnew
Eprev Fnew
23
Jprev Onew
24
return V, log pl (Feww)
20

---

Published as a conference paper at ICLR 2025
True p paz [x) HIGDM (Ul)
0.5
DPS
Standard deviation
0.4
0.3
0.1
10
12
14
16
18
20
Data dimensionality
Figure 8: The standard deviation of posterior samples from different methods for the toy data discussed in App. E, showcasing the overconfidence problem caused by overestimated Vz, log py xx).
Condition
Forward
DPS
IIGDM
TMPD
Peng (Convert)
FH (Ours) FH +Online (Ours)
Motion Blur Gaussian Buu
Impainting
Crufts
Criftss
Crufts
Crississ
Cridtes
Criitts
Crifts
Crufts
Super res.
Figure 9: Qualitative results from 30-step Heun sampler.
21

---

Published as a conference paper at ICLR 2025
Prior p(xo) with Likelihood p(y[xo)
Posterior p(xoly)
3
1.
0-
-2
3
Figure 10: The posterior distribution of the toy data discussed App. G and the prior distribution.
1. Covlan [m]] I+a(1)+ I, similarly IIGDM.
2. Cov(an [x] approximated with our method by initialising at the data covariance, but not performing space updates.
3. Cov|æo xx approximated with our method by initialising the data covariance, and per- forming space updates.
4. Cov(zo [x]] approximated with our method and with space updates. but estimating the BFGS updates by calculating V æ. log p(z, Ax) and V log p(æ,) explicitly, requiring 2 denoiser calls per step, but without error from the time updates affecting the BFGS updates. This method is also discussed in A.p.
For these experiments, we used the Maruuaaa sampler. The results are shown in Fig. 11. The IIGDM covariance approximation is the furthest from the true value. Initialising at the data covariance helps, and adding the space updates decreases the error further. However, there is a clear gap in the standard method and using two score evaluations per step. We believe that this is due to the errors from the time updates affecting the BFGS updates.
We also perform ablation comparing deterministic Fuler sampler and the stochastic Fuler- Maryama sampler with the fourth method, and varying diffusion step count. The results are shown in Fig. 12. Whereas for the deterministic sampler, the covariance estimate does not significantly im- prove after a certain point, for the stochastic sampler, the error approaches zero across the sampling steps with more steps. This because the reason the inherent curvature in the ODE path does signifcantly change after increasing the step count above certain limit. With stochastic sampler, the generative path explores a slightly different direction each step, and the BFSG updates get information from the curvature in all directions.
H ADDITIONAL RESULTS ON IMAGE MODELS
Here we list full additional results from:
1. ImageNet 256x256 results with Fuler solver for 15, 30, 50, and 100 steps, in Table 2.
2. ImageNet 256x256 results with Heun solver for 15, 30, 50, and 100 steps, in Table 3.
For each task, we use 1000 samples from the ImageNet test set. We also evaluate DDNM+ et al., 2023) and DifiPIR(Zhu et al., 2023) for more thorough to different types of methods, al- though these can not directly be interpreted as reconstruction guidance with specific covariances. For DiffPIR, use the implementation of (Peng et al., 2024), where they
22

---

Published as a conference paper at ICLR 2025
10
Average Frobenius Norm Eror
10
1
10-3
Cov(zo m:) Few(ty?
Time updates only
With space updates
Space updates without transferring score across
0.2
D.4
0.6
0.8
Sampling progress
The Frobenius norm of the difference between the true covariance and the estimated covariance for different methods for different sampling steps (0 corresponds to the maximum noise level and 1 corresponds to zero noise level). As pointed out in App. J, using the time updates to transfer scores for use with the BFGS updates can cause inaccuracies with low-dimensional data, making the curve slightly rough, although still below the with only time updates.
Figure 11:
Covariance Approximation Error (Fuler-Maryama)
Covariance Approximation Error (Fuler)
0.2
25 steps
25 steps
Average Frobenius Nomm Eror
50
Average Frobenius Norm Error
50
steps
steps
0.2
200 steps
400
0.15
400
200 steps
0.1
0.1
5-10
0.2
0.4
0.6
0.8
0.2
0.4
0.6
0.8
Sampling progress (1 diffusion time)
Sampling progress (1 diffusion time)
Figure 12: The Frobenius of the difference between the true covariance and the estimated covariance for different methods, with varying step count.
23

---

Published as a conference paper at ICLR 2025
Notes hyperparameters For DPS, Identity, Identity+Online updates, and DiffPIR, tuned hy- perparameters for each task using Gaussian blur as a baseline. We separately tuned the hyperparame- ters for the Euler and Heun solvers, and for each step count. While the optimal hyperparameters were similar for DiffPIR, Identity and Identity+Online updates, for DPS, the optimal values depended on the solver type and step count. We used 100 samples from the ImageNet validation set for tuning, and used these parameters for all experiments. The results are shown in Fig. 13,Fig. 14,Fig. 15 and Fig. 16.
I MOTIVATIONS FOR THE DCT BASIS AND THE BEGS UPDATE
The reason that we chose the DCT over, e.g., the DFT basis is that it purely real-valued, does not assume periodic boundaries, and in practice needs less coefficients to efficiently represent natural images. This is also one of the reasons for its use in the JPEG compression standard (Wallace, 1991). The BFGS update has the attractive property of preserving rositie-semiddefiiteeess (as opposed to, e.g., the symmetric rank-1 update). This combines well with performing the updates in the denoiser covariance, which is positive-definite (as opposed to the Hessian). Compared to Davidon-Fletcher- Powell (DFP), the difference is that the BFGS update minimizes a weighted Frobenius norm for the size of the update in inverse covariance (Dennis & Moré, 1977), instead of the covariance directly. In the update formula in Eq. (23), if we use A=I and the obseration noise is low, the inverse term simply the inverse covariance. Thus, could stabilise the updates across iterations, but this is more speculative, and DFP could work in practice as well.
J IMPLEMENTATION DETAILS
The solver We noticed that in large noise levels, it does not matter if the inversion in Eq. (23) is not exact, and we can set the tolerance quite high. We then defined a schedule for the tolerance such that it becomes lower towards the end. A lower tolerance towards the end of sampling is not an issue, since the covariance becomes closer to diagonal, and the required matrix inverse becomes easier to calculate. In practice, we use the following schedule:
max = 80
a min
rtol
rtolmin 1c 14
0.1
clipped max o muxx) mmin)
log factor log ( (Felipped) log10 (Omin)
(72)
log1o (mmax) - log (mmin)
log rtol
log. factor (log (itolamxx)
l logio((tolung
(73)
rtol 10
(74)
where rtolmax and rtol control the maximum and minimum relative tolerances of the solver, and Fmax and Fmin control the noise levels outside of which the tolerances are rtolmax and rtol respec- tively.
Note that scheduling the solver in this way does not improve image quality. Instead, improves inference speed considerably, to the point where the solver is not bottleneck anymore. Instead of using a standard off-the-shelf conjugate gradient implementation, we implemented one ourselves in PyTorch utilize the speedup from the GPU.
Also for TMPD, we created a schedule for the conjugate gradient since a constant low tolerance slowed down the computation quite bit. For TMPD, standard scipy implementation that
24

---

Published as a conference paper at ICLR 2025
Table 2: Results with the Euler solver. Our model performs especially well at small step sizes and remains competitive at larger step counts well. DDNM+ is designed to enforce consistency with the measurement in cases where the measurement operator has clearly defined nullspace, such as inpainting and super-resolution potentially affecting the good PSNR and SSIM results there. In contrast, DDNM+ struggles with our Gaussian blur kernels. Motion blur results are not presented, as the code of DDNM+ assumes separable kernels.
Deblur (Gaussian)
Inpainting (Random)
Dehlur (Motion)
Super res, (4x)
Method
PSNRT
SSIM
LPIPS
PSNR
SSIM1
LPIPS
PSNR1
SSIM
LPIPS
PSNRT
SSIMT
LPIPS_
DPS
19.94
0.444
0.572
20.68
0.494
0.574
17.02
0.354
0.646
19.85
0.460
0.590
IIGDM
20.29
0.474
0.574
19.87
0.468
0.598
19.21
0.429
0.602
20.17
0.474
0.582
TMPD
22.56
0.572
0.486
17.70
0.447
0.589
20.40
0.481
0.567
21.15
0.517
0.541
15 steps
Pong Convert
22.53
0.563
0.490
22.23
0.579
0.489
20.46
0.475
0.556
21.92
0.541
0.517
Peng Analytic
22.52
0.563
0.490
22.14
0,574
0.494
20.46
0.556
21.92
0.541
0.517
DDNM+
7.21
0.029
0.822
23.95
0.667
0.352
24.30
0.669
0.398
DinPIR
22.77
0.403
16.10
0.284
19.75
0.381
0.527
21.76
0.540
0.436
Identity
22,91
0.594
0.384
18.83
0,397
0,590
20.06
U.393
0.506
22.65
0.589
0.412
tomnlie
23.08
0.606
0,385
18.86
0.397
0.590
20.31
0.418
0.492
22.76
0.597
0.414
FH
23.41
0.625
0.373
24.76
0.702
0.327
21.69
0.534
0.447
23.39
0.632
0.390
FH+online
23.57
0.685
0.378
25.29
0.731
0.315
21.83
0.5-18
0.442
23.31
0.624
0.393
21.76
0.463
0.678
0.386
18:22
0.389
0.582
23.00
0.593
0.440
IIGDM
22.27
0.559
0,468
21.24
0.518
0.517
21.15
0,508
0.503
22.11
0.552
0.479
TMPD
22.92
0.591
0.451
18.27
0,465
0,563
20.71
0.495
0.538
21.59
0.536
0.507
Peng Convert
23.61
0.627
0.405
23.74
0.648
0,400
21.99
0.553
0.463
23.21
0.608
0.430
Peng Analytic
23.61
0.627
0.405
23.59
0.640
0.410
21.99
0.552
0.463
23.22
0.608
0.430
30
DDNM+
26.66
0.769
24.09
0.657
0.418
DiAPIR
22.34
0.552
0.404
15.94
0,262
0.667
19.38
0.368
0.523
21.25
0.512
0.443
Identity
23.15
0.602
0.374
18.75
0,402
0,578
20.14 14
0.406
0.494
22.82
0.588
0.405
conliit
23.38
0.621
0.359
20.07
0,443
0.529
20.47
0.420
0.467
23.38
0.622
0.383
FH
23.56
0.630
0.353
26.00
0.758
0.255
21.79
0.537
0.410
23.38
0.624
0.371
FHtooninnc
23.66
0.686
0.359
26.17
0.766
0.268
21.89
0.548
0.409
23.46
0.629
0.375
DPS
22.66
0.579
0.411
26.31
0.761
0.297
19.05
0.428
0.537
23.79
0.642
0.375
IIGDM
22.64
0.577
0.434
21.67
0,536
0.484
21.56
0.526
0.468
22.48
0.571
0.442
TMPD
23.09
0.600
0.434
18.50
0.472
0.551
20.83
U,500
0.524
21.76
0.543
0.492
50 steps
Peng Convert
23.81
0.638
0.377
24.75
0.698
0.346
22.30
0.587
0.430
23.44
0.622
0.400
Peng Analytic
23.81
0.638
0.378
24.47
0.683
0.360
22.30
0.566
0.430
23.44
0.622
0.400
DDNM+
7.83
0.038
0.806
27.15
0.771
0,301
24.05
0.655
0.424
DiAPIR
22.10
0.539
0.407
15.82
0.251
0.670
19.17
0.525
21.00
0.498
0.448
Ideatity
23.18
0.602
0.360
19.64
0,436
0,536
19.92
0.373
0.497
23.11
0.600
0.385
Ldentityoonline
23:47
0.620
0.370
19.46
0.409
0.552
20.74
0.453
0.453
23.20
0.607
0.399
FII
23.43
0.622
0.348
25.94
0.760
0.238
21.56
0.523
0.406
23.21
0.614
0.366
Fitanline
23.59
0.631
0.353
26.08
0.770
0.252
21.71
0.534
0.406
23.33
0.620
0.369
23.36
0.615
0.379
26.61
0,800
0.229
20.05
0.473
0.492
23.36
0.622
0.366
HIGDM
22.76
0.585
0.408
21.97
0.551
0.459
21.71
0.533
0.440
22.63
0.580
0.416
TMPD
23.18
0.604
0.423
18.67
0.477
0.543
20.90
0.502
0.514
21.88
0.548
0.480
Peng Convert
23.74
0.638
0.358
24.89
0.706
0.332
22.36
0.570
0.407
23.46
0.625
0.379
Peng Analytic
23.73
0.637
0.358
24.67
0.694
0.345
22.36
0.570
0.407
23.46
0.625
0.379
100
DDNM+
8.77
0.054
0.786
28.28
0.809
0.278
23.55
0.630
0.450
DiMPIR
21.88
0.527
0.410
15.69
0,241
0.672
18.95
U.345
0.529
20.78
0.485
0.451
Ideatity
23.11
0.596
0,361
19.66
0.439
0.528
19.72
0.358
0.501
23.04
0.594
0.384
tonliity
23.43
0.616
0.373
19.65
0.420
0,538
20.77
0.459
0.447
23.08
0.599
0.403
FII
23.24
0.613
0.346
25.71
0.755
0.233
21.35
0.510
0.407
23.03
0.604
0.364
FHvonline
23.32
0.616
0.356
25.73
0.764
0.243
21.37
0.509
0.417
23.17
0.609
0.370
25

---

Published as a conference paper at ICLR 2025
Table 3: Results with the Heun solver. Our model performs especially well at small step sizes and remains competitive larger step counts well. DDNM+ is designed to enforce consistency with the measurement in cases where the measurement operator has clearly defined nullspace, such as inpainting and super- super-resolution, potentially affecting the good PSNR and SSIM results there. In contrast, DDNM+ struggles with our Gaussian blur kernels. Motion blur results are not presented, as the code of DDNM+ assumes separable kernels.
Deblur (Gaussian)
Inpainting (Random)
Dehlur (Motion)
Super res, (4x)
Method
PSNRT
SSIM
LPIPS
PSNR
SSIM1
LPIPS
PSNR1
SSIM
LPIPS
PSNRT
SSIMT
LPIPS
DPS
19.94
0.444
0.572
20.68
0.494
0.574
17.02
0.354
0.646
19.85
0.460
0.590
IIGDM
20.30
0.475
0.574
19.87
0.468
0.598
19.21
0.429
0.602
20.17
0.474
0.582
TMPD
23.08
0.597
0.420
18.99
0.481
0.539
20.80
0.491
0.514
21.88
0.545
0.476
15 steps
Pong Convert
22.53
0.563
0.490
22.23
0.579
0.489
20.46
0.475
0.556
21.92
0.54
0.517
Peng Analytic
22.53
0.563
0.490
22.14
0,574
0.494
20.46
0.556
21.92
0.541
0.517
DDNM+
7.21
0.029
0.822
23.95
0.667
0.352
24.30
0.669
0.398
DinPIR
22.77
0.403
16.10
0.284
19.75
0.381
0.527
21.76
0.540
0.436
Identity
22,91
0.594
0.384
18.83
0,397
0,590
20.06
U.393
0.506
22.65
0.589
0.412
tomnlie
23.08
0.606
0,385
18.86
0.397
0.590
20.31
0.418
0.492
22.76
0.597
0.414
FH
23.30
0.624
0.372
24.73
0.701
0.327
21.69
0.534
0.446
23.30
0.624
0.390
FH+online
23.54
0.634
0.378
25.25
0.728
0.317
21.84
0.5-19
0.441
23.39
0.632
0.394
21.76
0.463
0.678
0.387
18:22
0.582
23.00
0.593
0.440
IIGDM
22.27
0.559
0,468
21.24
0.518
0,517
21.16
0,508
0.503
22.11
0.553
0.478
TMPD
23.16
0.602
0.415
18.85
0,481
0.537
20.91
0.500
0.507
21.94
0.549
0.472
Peng Convert
23.61
0.627
0.405
23.74
0.648
0.4003
21.99
0.553
0.463
23.22
0.608
0.430
Peng Analytic
23.61
0.626
0.405
23.59
0.640
0.411
21.99
0.552
0.463
23.21
0.608
0.430
30
DDNM+
26.66
0.769
24.09
0.657
DiAPIR
22.34
0.552
0.404
15.94
0.262
0.667
19.38
0.368
0.523
21.25
0.512
0.443
Identity
23.15
0.602
0.374
18.75
0,402
0,578
20.14
0.406
0.494
22.82
0.588
0.405
conliit
23.38
0.621
0.359
20.07
0,443
0.529
20.47
0.420
0.467
23.38
0.622
0.383
FH
23.55
0.630
0.353
26.00
0.757
0.256
21.80
0.538
0.411
23.38
0.623
0.372
FHtoolinn
23.62
0.685
0.358
26.18
0.767
0.268
21.88
0,547
0.110
23.44
0.628
0.375
DPS
22.66
0.578
0.412
26.31
0.761
0.296
19.05
0.428
0.537
23.80
0.642
0.375
IIGDM
22.64
0.577
0.435
21.67
0,536
0.484
21.56
0.526
0.468
22.48
0.571
0.442
TMPD
23.20
0.605
0.414
18.83
0,482
0.536
20.93
U,502
0.504
21.96
0.550
0.471
50 steps
Peng Convert
23.81
0.638
0.377
24.75
0.698
0.346
22.30
0.587
0.429
23.44
0.622
0.400
Peng Analytic
23.80
0.638
0.378
24.47
0.683
0.360
22.30
0.566
0.430
23.44
0.622
0.400
DDNM+
7.83
0.038
0.806
27.15
0.771
0,301
24.05
0.655
0.424
DiAPIR
22.10
0.539
0.407
15.82
0.251
0.670
19.17
0.525
21.00
0.498
0.448
Ideatity
23.18
0.602
0.360
19.64
0,436
0,536
19.92
0.373
0.497
23.11
0.600
0.385
Ldentityoonliee
23.47
0.620
0.370
19.46
0.409
0.552
20.74
0.453
0.453
23.20
0.607
0.399
FII
23.44
0.623
0.348
25.95
0.760
0.237
21.58
0.523
0.406
23.22
0.614
0.367
Fitanline
23.60
0.631
0.353
26.10
0.772
0.250
21.73
0.535
0.406
23.31
0.619
0.369
23.36
0.615
0.378
26.61
0,800
0.228
20.05
0.473
0.492
23.36
0.622
0.366
HIGDM
22.76
0.585
0.408
21.97
0.551
0.459
21.71
0.533
0.440
22.63
0.580
0.416
TMPD
23.22
0.606
0.412
18.83
0.482
0.536
20.95
0.502
0.504
21.98
0.551
0.469
Peng Convert
23.74
0.638
0.358
24.89
0.706
0.332
22.36
0.570
0.407
23.46
0.625
0.379
Peng Analytic
23.73
0.637
0.358
24.67
0.694
0.345
22.36
0.570
0.407
23.46
0.625
0.379
100
DDNM+
8.77
0.054
0.786
28.28
0.809
0.278
23.55
0.630
0.450
DiMPIR
21.88
0.527
0.410
15.69
0,241
0.672
18.95
U.345
0.529
20.78
0.485
0.451
Ideatity
23.11
0.596
0,361
19.66
0.439
0.528
19.72
0.358
0.501
23.04
0.594
0.384
tonliity
23.43
0.616
0.373
19.65
0.420
0.538
20.77
0.458
0.447
23.08
0.599
0.403
FII
23.25
0.613
0.346
25.71
0.755
0.233
21.35
0.509
0.408
23.02
0.604
0.364
FHvonline
23.35
0.616
0.357
25.75
0.765
0.242
21.40
0.512
0.414
23.15
0.608
0.371
26

---

Published as a conference paper at ICLR 2025
15 steps.
15 steps,
Fullr solver Best LPIPS scale: 100 Best SSIM scale: 30.00
Heun solver Best LPIPS scale: 80 00 Best SSM scale: 80 00
0725
0044
0.0
0675
0.40
0650
0.15
0.36
0.30
0.54
0575
0550
Conditoning
Conditionin Scale
Euler solver LPIPS scale: 500.00 00 SSIM scale: 500.00
solver Best LPIPS scale: 300.00 SSIM scale: 500.00
0.62
0.60
0400
0475
D375
0.45
0:425
0.25
0500
0.50
0:375
0.68
0450
038
0.47
837
057
0.46
055
0.45
055
0.45
854
0 0.44
035
0.44
0.43
033
0.43
052
0:42
651
0.42
2000
4000
5000
8000
30000
2000
4000
5000
3000
10000
Condilaming
Conditiining
100 steps.
100 steps.
Euler solver Best LPSS scale: 2000 00 Best SSM scale: 2000 00
Heun solver Best UMPS stale: 2000 00 Best 5S0M scale: 2000 00
0.46
060
0.02
0.45
0.64
0.43
0
2258
0:42
0.41
0.54
0.40
039
052
2000
4000
0000
0000
32200
0
2000
4003
0200
3000
10000
Conditioiin Scale
Conditioing Scale
Figure 13: LPIPS and SSIM metrics across different solver steps and conditioning scales for DPS. The optimal LPIPS values are used the experiments.
27

---

Published as a conference paper at ICLR 2025
15 steps, euler solver
15 steps, houn salver
Best LRPS a: 30.00, Best SSM 30.00
Best LPPS 20.00, Best SSIM 30.00
1795
1785
050
0.65
50M
0.55
500
0.35
0.55
0.5
0.40
steps, nnter salver
steps, houn salver
LMPS a: 50,00, Best
LPPS 50.000 00. 30.00
035
0.65
500
050
0.50
040
0.55
0.55
0.30
0.50
0.30
0.0
015
0.40
DITT1
culer solver
solver
LRPS
30.00,
LPIS
00
055
050
0.65
045
0.50
042
0:50
0.0
0 0.55
033
0.30
0.45
0.0
015
DMTRR
stops.
stops.
030
030
0.60
040
040
3.55
0.55
0.50
025
0.0
025
0.40
015
035
20
22
50
D3T
Figure 14: LPIPS and SSIM metrics across different solver steps and À values for DiffPIR. The optimal LPIPS values are used in the experiments.
28

---

Published as a conference paper at ICLR 2025
15 steps, culer solver
15 steps, houn solver
Best LPIPS scale: 8D, Best SSIM scale 0.70
Best LPIPS scale: 8000 Best SSIM scale: 0.70
LPPS
015
UPPS
034
0.0
500
0.48
046
0.6
0.4
0.45
0.0
0.65
0.2
Conditioing Scale
Scale
cullr solver
houn solver
0.70.
scale:
scale:
UPPS
UPP
50M
052
048
041
0.46
0.64
0.44
0.44
0:05
1.0
0.5
1.0
0.48
0.47
0.47
0.46
055
1995
0.45
0.10
0.44
0.5
0.6
0.5
1.0
0.5
36
0.5
1.0
Conditaming
Canditioiigg
100 steps, culer solver
100 steps, heun solver
Best LPIPS scale:
Best SSIM scale: 0.60
Best UMPS /0, SSIM scale:
18P5
050
050
0.47
048
0442
0.45
0.45
0.4
0.44
0.3
0.5
0.6
07
0.3
1.0
3.5
0.5
0.6
0.5
0.
3.0
1.3
Conditioing Scale
Conditioning Scale
Figure 15: LPIPS and SSIM metrics across different solver steps and conditioning scales for our method with identity base covariance. The optimal LPIPS values are used in the experiments.
29

---

Published as a conference paper at ICLR 2025
15 steps, culer solver
15 steps, houn solver
Best scele: Best SKIM
Best LPPSS 880
scale
058
055
0.47
054
0.43
0.45
852
0.42
0.44
0.4
244
042
039
2.66
0.5
2.6
07
0.8
033
1.0
11
0.
0.
0.5
0.3
11
Conditooiin Scale
Conditooingg
30 stops, culer solver
30 stops, houn solver
Best UPIPS scale: 3.0, Best SSIM scale: 0.00
Best UMPS scale: 0.70, Best SSIM scale: 0.70
055
0.43
058
054
044
0.42
0.42
2.48
044
0.5
0.6
07
0.5
0.9
1.0
11
0.5
0.6
0.7
0.5
0.9
1.0
1.
Conditiorang Scale
Conditionag
50 steps, euler solver
50 steps, heun solver
LMPS
0.455
0.46
0.400
652
0.400
0.44
050
0.435
043
0.45
0.41
0.
06
077
0.5
03
1.0
0.
06
0.7
0.5
0.9
Conditioiin
Conditioning
100 steps, culer solver
100 steps, houn solver
Best UMIP scale: /0, (Best SSIM scale: 80
Best UMPS scale: /0, Best SS3M scall: 0.70
0.48
0.45
0.47
0.46
0.45
0.4
641
043
2.02
0:48
042
0.5
0.
0.9
06
Conditioiig
Conditionig
Figure 16: LPIPS and SSIM metrics across different solver steps and conditioning scales for our method with the identity base covariance and online updates. The optimal LPIPS values used in the experiments.
30

---

Published as a conference paper at ICLR 2025
is parameterized that is parameterized in terms of
max 80
min
tol
toim
1c 14
0.05
0 clipped
max(min(o,, o mmux) ) Omin)
logio ( (Felipped) logiu ((Bin))mm))) (Fmin)
log factor
(75)
logio maax - (m
log rtol log factor (logıo (ftolmax) (rtolmin)) login (rtolmin)
(76)
rtol 10
(77)
We tried to choose the schedule such that this does not degrade the performance noticeably, but also allows us run experiments in reasonable time.
Range for the BFGS updates In practice, we do the space/BFGS updates for range of (t) values for image data, particular S (t) < A motivation for this choice that we noticed the finite differences to not be numerically accurate at high noise levels, where the time updates At and the space updates A are large. For low noise levels, is also unnecessary, given that the covariance approaches of ( I in any case. How to apply the space updates in the optimal way is interesting direction for future research.
The solver We noticed that in large noise levels, it does matter if the inversion in Eq. (23) is not exact, and we can set the tolerance quite high. We then defined a schedule for the tolerance such that it becomes lower towards the end. A lower tolerance towards the end of sampling is not issue, since the covariance becomes closer to diagonal, and the required matrix inverse becomes easier calculate.
Details on the low-dimensional experiments In the Gaussian mixture experiments, we did not apply the time updates for μ4 (x)... but instead evaluate Mo (o]t+Ar(z)) explicitly before applying the BFGS update. The reason is that on low-dimensional data, some of the prior samples are close to the actual data distribution. In that region, the time evolution is complex enough from the start that the denoiser mean time update coupled with the BFGS update in the next step sometimes causes numerical instability. To avoid additional score function evaluations, we could devise a schedule for when to apply the space updates.
Measurement operators. We obtained the measurement operator definitions from (Peng et al., 2024), which in turn based on the operators in (Chung et al., 2023), We use noise level o = 0.1 for all measurement models (data scaled to [-1,1]).
K ADDITIONAL RESULTS ON OPTIMAL GUIDANCE STRENGTH
Figure 17 shows the PSNR, SSIM and LPIPS scores for the Gaussian deblurring task on Ima- geNet 256x256 with different post-hoc guidance scales the initially calculated guidance term V log pyy x.)).
L COMPUTATIONAL REQUIREMENTS
The sweep to obtain the results in Table 1 was done with multiple NVIDIA V100 GPUs in a few hours, and be obtained with single V100s in less than a day of compute. For some of the methods, the matrix inversion in Eq. (23) can slow down generation considerably, although this is not significant overhead with our tolerance-optimized conjugate gradient implementation.
31

---

Published as a conference paper at ICLR 2025
22
PSNR
SSIM
LPIPS
0.3
0.4
0.6
0.8
1.2
1.4
0.6
0.8
1.2
1.4
0.8
0.8
1.2
1.4
Guidance strength
Guidance strength
Guidance strength
Figure 17: Different metries with respect to guidance strength for 100 images from the Imagenet validation set, using an identity base covariance (blue) and DCT-diagonal covariance (orange). With better covariance approximation, the usefulness of adjusting the resulting guidance with post-hoc tricks becomes smaller.
M OPTIMAL Var æt pog y) FOR GAUSSIAN MIXTURE DATA AND GAUSSIAN OBSERVATION
In this section, we derive the optimal gradient V log p(x; y for situation with Gaussian mixture data and Gaussian observation model. This is useful for performing toy experiments without having to retrain the model. Note that we can get the unconditional score from the end result by setting the observation noise My infinity.
Model Definition
We begin with the following components:
1. Prior Distribution: The prior æo is a Gaussian mixture model:
p( = Σ wyN(aa pss E,),
(78)
where w; are the mixture weights, μi; are the mean vectors, and Sigma,, are the covariance matrices for each mixture component.
2. Likelihood: The observation model Gaussian:
p xo) NNy æo. Σ,)
(79)
where Ey is the observation noise covariance.
3. Transition Model: The transition from ætp to æ;, modeled as:
p(z, (x) N(x, æn. It)
(80)
where σ² I is isotropic Gaussian noise with variance (t)?
M.1 POSTERIOR DISTRIBUTION
Given these components, the posterior distribution p(zo Ltt y) is also a Gaussian mixture:
p( Ft b) Σ w/N(ca p1: Σ/
(81)
where
E-- = (0(1) Σ-1 n/ ()((((1))1) Ey y + E m.) w/ x wpN(a, us, (1) + E,W us: Ey E,)
(82)
(83)
(84)
M.2 CONDITIONAL EXPECTATION
The conditional expectation of æo given 8; and is:
Cy æt, y] Σ mlpl
(85)
32

---

Published as a conference paper at ICLR 2025
M.3 DERIVATION OF THE GRADIENT
Now, let's derive the gradient Vn, logp(xr (y):
1. We start with:
V log (1) = Vn, log / p(az ( y) da
(86)
2. Applying the chain rule and moving the gradient inside the integral:
V v. logp(ze y) J Va, zu)p( y dxz
(87)
J p( zo)p(20 y dx
3. Given p(a: xv) = Nxx Do, It) we have:
V Va,p(rs xo) (t)2 ( mo)p(aa a0)
(88)
4. Substituting this back:
1 [α, z0)p((x zo))((())) y) dxo S p((r, xo wo)p y) da
xe log (0t
(3)
(89)
o1)²
1
I popp(z) wr)pp(m y) dæy)
(x)
(90)
(t)2
I p(æ: (o)o y) dæ
(a0: R|aro T y))
(91)
(t)2
M.4 FINAL FORM OF THE GRADIENT
Therefore, the final form of the gradient is:
1
V m. pog () σ2 (ac, Ble00 x.,, y))
(92)
M.5 DETAILED FORMULA FOR E[zy| xt. y)
Let's expand the formula for
B|wv (er y]:
1. We start with the posterior distribution:
p xt. y) = Σ W/N 1k4 E
(93)
2. The expectation of this mixture is the weighted sum of the means:
E [xo Ct, y) Σ whuk
(94)
3. Expanding pt:
=²= = ((=)(((())) + Ey Eyyy y + E1 u1)
(95)
4. Substituting this into the expectation formula:
E E It. yl = Σ [ + Σ y + 1 (1)
(96)
5. Rearranging:
E[ [oo xe. y] = E (/EE + Σ w "E "1 + Σ w W 4s (97)
33

---

Published as a conference paper at ICLR 2025
Let's define:
= Σ ((
(98)
B Σ W. "
(99)
c Σ w p.:
(100)
Then we can write the final formula as:
E/zp Dt, y)
Ax, + By + c
(101)
where:
/ x w M4s o (t) E, Mi, Ey + E,),
(102)
Σ = ((a(t) + x -
(103)
This formula shows that Ex xt, yj is linear combination of x; and y, plus constant term. The matrices and B determine how much the expectation depends on i and y respectively, while c represents a constant offset based on the prior distribution.
34


Reviews:
[{"review_id": "HdNzpFBtni", "reviewer": "Reviewer_ei58", "summary": "The authors introduce a new covariance estimation method which makes use of covariance information in denoiser and the trajectory curvature without introducing significant additional compute. Moreover, to make the approach suitable for high-dimensional data (e.g. covariance matrices storage issue), the low-rank updates is proposed.  The authors validate their approach on linear inverse problems, demonstrating its effectiveness compared to baselines under four metrics.", "questions": "- The authors assume that the observation model p(y|x0) is linear gaussian. Is there any other real world problems can be expressed in this form? \n- Continue with weakness section: Can you provide more experimental results? \nFor example, does the performance changes when using more Heun steps? does the improvement also happen to other type of discretization methods, e.g. Euler? Also there are many other ODE samplers and SDE samplers, if you change the ODE samplers with SDE ones, does the introduced stochasticity make any difference when using your covariance estimate method?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- Novel method, proposed a covariance estimate method via separately updating along time and position/space. Also provide a practical implementation for the proposed method. \n- The paper provides theoretical insights into why accurate covariance estimation is crucial for unbiased conditional generation, which is supported by experimental results on inverse problems.", "weaknesses": "- The experiment section is not as strong as methodology section: only one dataset and one ODE denoiser is tested under low NFE setting. The design of the experiment and the corresponding results analysis can be improved. \n- Only one ODE sampler (Heun) with steps=15, 30 is tested. While indeed results show the importance of accurate covariance estimate, it’s hard to conclude that this is applicable to all standard diffusion models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "q72Wg3Xyib", "reviewer": "Reviewer_D6UH", "summary": "The authors proposed a new methodological framework for estimating covariance in the generative methods of diffusion models.  The estimation of the covariance in the reverse diffusion process is purely based on the existing samples through a two-step updating scheme. The examples show the proposed method outperforms other existing methods in denoising.", "questions": "1. In the image data results, adding the online update steps to either identity covariance or the FH covariance does not always improve the performance. Is there any explanation or further investigation on why it could happen? \n2. Table 1 shows that in most cases, FH+online outperforms FH. But in Figure 7, only the denoised image from FH is demonstrated. Is there any vision difference between FH and FH+online in terms of the images in Figure 7?\n3. Notational /exposition issues:\n    1. In Eq. (1), the \"noising forward process\" should be defined in the joint distribution of the stochastic paths or a stochastic differential equation. Eq. (1) gives the margin distribution of the state at time t, which cannot formally \"define\" a process.\n    2. In Eqs. (2) and (3), $\\dot\\sigma$ and $\\omega_t$ are undefined. \n    3. Above Eq. (7), $p_{0|t}$ is undefined. \n    4. In Eq. (11), add a bracket to make the inverse clear. I.e., $[\\nabla_{\\mathbf{x}}^2\\log p(\\mathbf{x},t)]^{-1}$.\n    5. The last few sentences in Sec. 3.1. confused me. The integral $\\int p(\\mathbf{x}')\\mathcal N(\\mathbf{x}'\\mid \\mathbf{0}, \\sigma^2\\mathbf{I})d\\mathbf{x}'$ is not a convolution. \n    6. In Eq. (18), should the right-hand side be $\\left[\\mathbf{\\Sigma}_{0\\mid t}(\\mathbf{x}+\\Delta\\mathbf{x})\\right]\\Delta\\mathbf{x}$.\n    7. In Eq. (23), use either $\\nabla_{\\mathbf{x}}$ or $\\nabla_{\\mathbf{x_t}}$ consistently throughout the paper.\n    8. In Eqs. (23) and (24), $\\mathbf{A}$ is undefined.\n    9. Above Eqs. (25), $\\mu_{0\\mid t}$ should be in boldface. (and many other occurrences throughout the paper).\n    10. In Sec. 4.2., \"we noticed that the guidance scale is overestimated the larger the dimensionality is, leading to overconfidence.\" does not read well --- the authors probably left over some words from a previous edition. \n    11. Throughout the paper and the appendix, please only label the equations you refer to later.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 2, "contribution": 3, "strengths": "Originality and significance: The proposed method is relevant to the diffusion models and is novel in that it provides a fast and efficient algorithm for estimating a broader class of covariance structures. \n\nQuality and clarity: The paper clearly connects the proposed method to existing literature on similar problems.", "weaknesses": "The exposition can be improved as the current manuscript contains many inconsistent notations, undefined variables, and many typos. I would say it is a lovely work with appealing results, but unfortunately, the manuscript should be significantly revised and proofread to be easier to follow.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LS7loVWTC2", "reviewer": "Reviewer_nmFX", "summary": "This paper introduces a new method to estimate the covariance during the denoising process in diffusion models. The key idea is using Tweedie's formula to relate the covariance of the denoiser with the Hessian of $\\log(p(x_t))$, then using the gradient of $\\log(p(x_t))$ (i.e. the score function) evaluated during each denoising step to estimate this Hessian. Starting with an estimate of the covariance of the data distribution, the algorithm performs low-rank updates to this covariance matrix, inspired by quasi-Newton methods, using the score function estimates given by the neural network. This method is then applied to fast sampling for solving inverse problems, showing improvements over other methods.", "questions": "1. Since the data covariance is estimated in the DCT basis where the covariance is approximately diagonal, have you considered training a model to directly estimate the denoiser covariance in this basis, or with a diagonal + low-rank representation?\n\n2. The numerical results in Table 1 seems to be worse than that in the DDNM paper, is this because the number of sampling steps is limited?\n\n3. What is the computational overhead for estimating the data covariance and doing the low-rank updates, compared to the cost of evaluating the neural network?\n\n4. What is the variance exploding case first mentioned in Line 213? This doesn't seem to be explicitly defined in the paper.\n\n5. What is the difference between $\\mu_{0\\mid t}(x)$ and $\\mu_{0 \\mid t}(x_t)$, as well as $\\sigma(t)$ and $\\sigma_t$?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. It is difficult to efficiently estimate the covariance of the denoiser due to its high-dimensionality, and methods that attempt to estimate it has have to use reduced-complexity representations. Using the DCT-diagonal basis to estimate the initial data covariance and a low-rank update inspired by quasi-Newton methods is a novel approach.\n\n2. Because this method does not require training an additional neural network, it can be applied to existing pre-trained diffusion models to be used for solving inverse problems.\n\n3. The experimental results seem promising and improves on previous methods on both qualitative examples and quantitative metrics.", "weaknesses": "1. There needs to be more theoretical or empirical justification for some of the algorithmic choices made, such as the choice of quasi-Newton method or the DCT basis used to approximate the data covariance.\n\n2. The notation in the paper is inconsistent and the definition of some quantities are not clear, for example are $p(x, t)$, $p(x_t)$, $p_t(x)$ referring to the same quantity? Also, it is not clear what exactly are the full algorithms used for solving linear inverse problems; Algorithms 1 and 2 in the paper only updates the covariance estimate.\n\n3. The experimental improvements for solving linear inverse problems seems to be only for the few-sample-step regime. It is unclear if this covariance estimation method can be helpful in other applications such as unconditional sampling, or with non-linear guidance terms.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VMpqvRosbk", "reviewer": "Reviewer_SLRN", "summary": "The paper introduces a new framework for estimating covariance in diffusion models, addressing issues in current methods such as high test-time computation and heavy approximations. The proposed method utilizes readily available covariance information from training data and the curvature of the generative trajectory. The authors also present a method for transferring covariance estimates across noise levels.", "questions": "1. refer to weaknesses.\n2. in Figure 6, these setting of y is what? inpating transform or deblurring? Does this guidance meaningful across different y|x_0 and dataset?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. It is novel and reasonable to design low-dim expression and update rules for covariance approximation.\n2. There are abundant math derivations to support the methods", "weaknesses": "1.  To handle linear inverse problems using diffusion models, there are a lot of classes of methods [1]. The experiments in this work only compare some of them, arthors should at least mension why take them out of comparison.\n2. There lack of a experiment to back up whether the low-dim covariance of the proposed method can indeed match the underlying true one. I understand this cannot be done under high dimensional case, mayba a 2D or 10D suffice.\n3. There is other applications of diffusion models would need covariance like causual reasoning [2], likelihood-evaluation [3][4] and adjoint guidance[5], maybe you could add a discussion to these.\n\n\n[1] Daras, Giannis, et al. \"A survey on diffusion models for inverse problems.\" arXiv preprint arXiv:2410.00083 (2024).\n\n[2] Sanchez P, Liu X, O'Neil A Q, et al. Diffusion models for causal discovery via topological ordering[J]. arXiv preprint arXiv:2210.06201, 2022.\n\n[3] Lu C, Zheng K, Bao F, et al. Maximum likelihood training for score-based diffusion odes by high order denoising score matching[C]//International Conference on Machine Learning. PMLR, 2022: 14429-14460.\n\n[4] Anonymous. Gradient-Free Analytical Fisher Information of Diffused Distributions.\n\n[5] Song K, Lai H. Fisher Information Improved Training-Free Conditional Diffusion Model[J]. arXiv preprint arXiv:2404.18252, 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "v4v2uEVack", "reviewer": "Reviewer_d5jC", "summary": "This paper introduce a L-BFGS-like method to maintain the covariance information of denoiser along the inference path to better solve the linear inverse problems via diffusion models.", "questions": "see Weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 1, "contribution": 2, "strengths": "1. It is novel to leverage L-BFGS way to approximate the covariance in a low-dim manner in DMs\n2. the mathematical derivation is rigorous, though it is quite similar to the math of BFGS optimization\n3. The design of initial covariance to be the data covariance is reasonable.", "weaknesses": "1. The presentation of the paper could be improved for clarity. The term 'conditional generation' is often associated with 'text-guided' or 'label-guided' generation. However, in this paper, it refers to conditions based on partially corrupted samples. While this interpretation is not incorrect and indeed falls under the classifier-guided framework, it might be confusing for readers at first glance. \n\n2. There are typographical errors present in the paper, for instance, in equation (98). These need to be corrected for accuracy and better understanding.\n\n3. The paper lacks a theoretical bound on the approximation error of the covariance using the proposed method. Both Equation 6 and the low-rank approximation involve approximation errors.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)