Efficiently Access Diffusion Fisher: Within the Outer Product Span Space
Authors: Fangyikang Wang
Hubery Yin
Shaobin Zhuang
Huminhao Zhu
Yinan Li
Lei Qian
Chao Zhang
Hanbin Zhao
Hui Qian
Chen Li
Venue: ICML 2025 poster (2025)
Keywords: diffusion models; fisher information; optimal transport
Primary area: deep_learning->generative_models_and_autoencoders
TL;DR: This paper reveals diffusion Fisher's outer-product structure, develops efficient algorithm, and verifies PF-ODE map's optimal transport.

Abstract: Recent Diffusion models (DMs) advancements have explored incorporating the second-order diffusion Fisher information (DF), defined as the negative Hessian of log density, into various downstream tasks and theoretical analysis.
However, current practices typically approximate the diffusion Fisher by applying auto-differentiation to the learned score network. This black-box method, though straightforward, lacks any accuracy guarantee and is time-consuming.
In this paper, we show that the diffusion Fisher actually resides within a space spanned by the outer products of score and initial data.
Based on the outer-product structure, we develop two efficient approximation algorithms to access the trace and matrix-vector multiplication of DF, respectively.
These algorithms bypass the auto-differentiation operations with time-efficient vector-product calculations. 
Furthermore, we establish the approximation error bounds for the proposed algorithms.
Experiments in likelihood evaluation and adjoint optimization demonstrate the superior accuracy and reduced computational cost of our proposed algorithms.
Additionally, based on the novel outer-product formulation of DF, we design the first numerical verification experiment for the optimal transport property of the general PF-ODE deduced map.

Efficiently Access Diffusion Fisher: Within the Outer Product Span Space
Fangvikang Wang 1'
Hubery Yin
Shaobin Zhuang 11 Huminhao Zhu
Yinan Li" Lei Qian
Chao Zhang'
Hanbin Zhao'
Hui Qian
Chen Li²
Abstract
bution, especially in generating high-quality images (Ncchll
al. [022: Dhariwal & Nichol [22 la Saharia et al. 2022
Recent Diffusion models (DMs) advancements have explored incorporating the second-order dif- fusion Fisher information (DF), defined as the negative Hessian of log density, into various down- stream tasks and theoretical analysis. However, current practices typically approximate the diffu- sion Fisher by applying outo-differentiatinn to the learned score network. This black-box method, though straightforward, lacks any accuracy guar- antee and is time-consuming. In this paper, we show that the diffusion Fisher actually resides within a space spanned by the outer products of score and initial data. Based on the outer- prouuct structure, we develop two efficient approximation algorithms to access the trace and matrix-vector multiplication of DF, respectively. These algo- rithms bypass the auto-differentiation operations with time-efficient vector-produc calculations. Furthermore, we establish the approximation error bounds for the proposed algorithms. Experiments in likelihood evaluation and adjoint optimization demonstrate the superior accuracy and reduced computational cost of our proposed algorithms. Additionally, based on the novel outer-product formulation of DF, we design the first numerical verification experiment for the optimal transport property of the general PF-ODE deduced map.
Ramesh et al., 2022 Rombach et al. 2022 Ho et al.) 2022)
The training process of DMs can be seen as employing a neural network to match the first-order diffusion score Vx log ge(xr) at varying noise levels.
There has been growing trend recognize the impor- tance of the diffusion Fisher (DF) in DMs, defined as the negative Hessian of the diffused distributions' log den- sity, VV log (xx). The DF provides valuable second- order information on DMs and plays a crucial role in downstream tasks, e.g., likelihood evaluation (Lu et al. 2022a Zheng et al. 2023), adjoint optimization (Pan et al. 2023a Blasingame & Liu 2024). and the optimal trans- port analysis of DMs (Zhang al.. 2024a). It is straight- forward to access DFs by applying auto-differentiation to the learned first-order diffusion score. Nevertheless, the repeated auto-differentiation operations would be time- consuming for large-scale tasks, even when incorporating the cestor-Jacohian-prodde (VJP) and Hutchinson's esti- mation technique (Song & Lai| 2024). Recently, certain re- search efforts investigated the time-evolving property of DM and demonstrated that the DF can be explicitly expressed in terms of the score and its covariance matrix (Benton et al., 2024 Lu et al. 2022a). However, these results predom- inantly concentrate on the theoretical aspects and fail to enable efficient access to the DF.
In this paper, we derive novel form for the diffusion Fisher, which is composed of weighted outer-product sums. This innovative formulation uncovers that the diffusion Fisher in- variably lies within the span of a set of outer- snter-proddcc bases. Notably, these bases solely depend on the initial data and the noise schedule. Leveraging this outer- anter-product structure, we devise two efficient approximation algorithms to access the trace and matrix-vecte vector multiplication of DF, respectively. Moreover, we design the first numerical verification exper- iment to investigate the optimal transport property of the diffusion Oattsioon-DDEEE deduced map under diverse types of initial data and noise schedules. The main contributions of our paper are listed as follows:
1. Introduction
The emerging diffusion models (DMs) Dickstinn et al.
2015
Ho et al. 2020 Song & Ermon 2019 (oong al. 2020), generating samples of data distribution from initial noise by learning reverse diffusion process, have been proven to be an effective technique for modeling data distri-
'Zhejiang University. "WeChat Vision, Tencent Inc. "Shanghai Jiao Tong University. "Equal contribution. *Wok done interns WeChat Vision, Tencent Inc. Correspondence to: Chao Zhang
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 20255 by the author(s).
We derive an analytical formulation of the diffusion Fisher, which is the weighted summation of the outer

---

Efficiently Access Diffusion Fisher: Within the Outer Product Span Space
product, under the setting when the initial distribution is sum of Dirac. The derivation process involves ap- plying the consecutive partial differential chain rule to the data- data-dependent form of marginal distributions. Sub- sequently, we extend this formulation to a more general setting. In this general case, we only assume that the initial distribution has a finite second moment. These novel formulations reveal that the diffusion Fisher in- variably lies within the span of set of outer-produc bases.
to the Lebesgue measure: J [()()) When the integral is with respect to a different measure 14, explicitly write f gdd.. When clear from context, we sometimes abuse nota- tion by identifying a measure μ with its Lebesgue density, We also use 5(-) to denote the Dirac Delta function. For a vector v E IR we denote the d x matrix vo as the outer-product of
2.1. Diffusion Models and Diffusion SDEs
Suppose that we have a d-dimensional random variable vo E Id following an unknown target distribution qo (zo). Diffusion Models (DMs) define a forward pro- (E).T) with T > starting with æn, such that the distribution of conditioned on zo satisfies
Based on the outer- auter-product structure of DF, we propose two efficient approximation algorithms, each tailored to access the trace and matrix-vector multiplication of the DF, respectively. When it comes to evaluating the trace of the diffusion Fisher, we introduce parameter- ized network to learn the trace, significantly reducing the time complexity of trace evaluation from quadratic to linear w.r.t. the dimension. In situations where we need to access the matrix-vector multiplication of the diffusion Fisher, we present training- frreeng-frre method that simplifies the complex linear transformation into sev- eral simple vector- -roductt calculations. Furthermore, we rigorously establish the approximation error bounds for these two algorithms.
(n mo) = (ax;; (t))oo, (t))),,
(1)
where a--), E C([O,T), 12) have bounded derivatives, and we denote them as ag and og for simplicity. The choice for (N and or is referred to as the noise schedule of DM.
According to (Kingma al. 202T; Karras et al. 2022). with some assumption on (+)) and o (+), the forward process
can be modeled linear SDE which is also called the CrnsteinUUllenbed process:
Utilizing the novel outer- piterrproduct formulation of DF, we derive corollary for the optimal transport (OT) property of the diffusion-ODD deduced map. Subse- quently, we design the first numerical experiment to examine the OT property of any general diffusion ODE deduced map based the corollary. Our numerical experiments indicate that for all commonly employed noise schedules (VE (Ho et al. 2020). VP (Song & Ermon 2019), Sub-VP (Song et al. 2020), and EDM Karras et al. 2022) ) the diffusion ODE map exhibits the OT property when the initial data follows a single Gaussian distribution or has an affine form. Conversely, it does not exhibit this property when the initial data is non-affine. These numerical results not only validate the conclusions presented in (Khrulkov et al. 2023 Zhang et al. 2024a] Lavenant & Santambrogio 2022)
dæ, = [()) + g(t)dB,
(2)
where Bg is the standard d-dimensional Brownian Motion (BM), f(t) dingg and g²(() dož 2 dloogo of. Under some regularity conditions, the above forward SDE equa- tion 2 have reverse SDE from time T to 0, which starts from o, (Anderson) 1982):
dt
dæ. = [f())= (oseeeg22())) ( a, log logg(2z,
0) dt + gt(dd
where B, is the reverse-timg time Brownian motion and q t) is the single-time marginal distribution of the forward pro- cess. In practice, DMs (Ho et al. 2020 Song et al. 2020) use (2,,,, t) to estimate at) m. log ((x,,t) and the pa- rameter 0 is optimized by the following loss:
C= E. {X.Ezo. [[]=o(xe,)) 1) - V., logp(zt, f/20 0)]11)
(4)
but also open up new avenues for the exploration of more general cases.
where 89 represents the parameterized score, i.e. so (xt, f) = co(m..t) This familiar parameterization is called e-prediction. There are also y-prediction and u- u-prediction (Salimans Ho 2022). The correspond- ing loss equal to replace the term je Eæ t) with ai Bu--yo(a: f) and a vy (2), The learned En (æx, 1) can be also transformed to y-prediction form by go (0,,,, t) (aro))
We evaluate our DF access algorithms on likelihood evalua- tion and adjoint optimization tasks. The empirical results demonstrate our DF methods' enhanced accuracy and re- duced time cost.
2. Preliminaries
2.2. Diffusson Models Inference as Neural ODE
Notation. The Euclidean norm over Rd is denoted by I|-II, and the Euclidean inner product is denoted by (-I-). Through- out, we simply write g to denote the integral with respect
It is noted that the reverse diffusion SDE in equation 3 has an associated probability flow ODE (PF-ODE, also called diffu-
2

---

Efficiently Access Diffusion Fisher: Within the Outer Product Span Space
sion ODE), which is deterministic process that shares the
Theoretical
Practical
Theoretical Error Bound
same single-time marginal distribution (Song al. 2020
Access
Methods
Time-cost
Time-cost (s)
VJP (eq.
15
qu + ozd"
2195.48
x
Trace
DF-TM (Ours)
2qdd
0.072
V(Prop. 7
1 dæ = [100=
2 - (t)
log qe (2 t) dt. (5)
VJP (cq.
19
e + ead
0.155
X
Operator
DF-EA (Ours)
20yd
0.066 4
/(Prop. 8
By replacing the score function in equation 5 with the noise predictor Eø. the inference process of DMs can be con- structed by the following neural ODE:
(VJP) and our proposed efficient access methods terms per-iteration theoret- ical time-cost, practical time-cost, and approximation error bound, The theoretical time cost is based on assumptions of a network access cost of cnd time and backpropagation on network cost of Ozd time (o2 z 4c2). The practical time-cost is tested using the SD-v1.5 model over 10k COCO prompts. Here, our VJP baseline calculates every element of the trace, and discussion on its approx- imation is deferred to Appendix C.5
Table 1: Comparison of cattor-Jacobia-pppee
dæ = he (Ie, t) ;= f(t) + (t(t) di
Eo (2t, t),
(6)
20,
where initial æT ~ N (0, 1)
2.3. Fisher Information Diffusion Models
is theoretical guarantee that the diffusion Fisher can be accurately accessed through the VJP. Moreover, the VJP fails to provide any theoretical insight into the diffusion Fisher.
The diffusion Fisher information matrix in DMs is defined as the negative Hessian of the marginal diffused log-density function, which takes the following matrix-valued form
(Song et al. 2021: Song & Lai 2024):
3. Diffusion Fisher in Outer-Products Span
0²
F1 () = log de (at,,) t)
dz?
Accessing diffusion Fisher via the VJP shown in equa- tion straightforward, but it does not take advantage of any inherent structure of the diffusion Fisher. Though cer- tain recent research efforts (Lu et al. 2022a| [enton et al., 2024) demonstrated that the DF can be explicitly expressed in terms of the score and its covariance matrix. Neverthe- less, their formulations cannot directly facilitate efficient access to the diffusion Fisher. In this section, derive novel form for the diffusion Fisher, which is composed of weighted outer producp sums. We first conduct this under a simplified scenario, assuming that the initial distribution in is a sum of Dirac distributions. Subsequently, we general- ize this outer-product span form to a more general setting. Significantly, the outer outer-product span form of the diffusion Fisher obtained in both settings does not require any gradi- ent calculations and depends solely the initial data and the noise schedule. Notably, there exist fast methods for accessing the trace and matrix-vector multiplication of an outer-produce matrix. This advantageous characteristic al- lows for the development of novel algorithms for efficiently accessing the diffusion Fisher. We start with simple setting where we assume that the initial distribution is character- ized as sum of Dirac distributions composed of the set of samples in the dataset. If we suppose the dataset is denoted as { J then the initial distribution follows
The current technique typically approximately accesses the diffusion Fisher by accessing the scaled Jacobian matrix of the learned score estimator network Eg:
0
F. F. () = axz
log (115
dæ.
c0(
1 des (x). Ox dx.
Dx.
Tt
The full diffusion Fisher matrix within DMs cannot be ob- tained due to dimensional constraints. For instance, the Stable Diffusion-1.5 model (Rombach et al. 2022) features a latent dimension of d = x 64 x 64 = 16384, result- ing in a diffusion Fisher matrix of 16384 x 16384. Fortu- nately, for applications that only need to access the trace or multiplication of diffusion Fisher, it is feasible use the (VJP) to access diffusion Fisher. For any d-dimensional vector v. the approximation of v left multiplied by Fr Fi t) using VJP is as follows:
Nector-Jacobian-preee
dep( t)
(VJP)
e FT,,,,
v
dx,
1 a KEa (2r, |))
0x Ox.
1
(Dirac Setting)
Σ 8(x y.). ((=, N
The VJP is time-consuming process due to its requirement for gradient calculations within the neural network. In ad- dition, empirical evidence from synthetic distributions, as demonstrated in Lu et al. (2022a), shows that the approxi- mation results from the VJP significantly deviate from the true underlying diffusion Fisher. To our knowledge, there
(10)
where exists a < D, < 20 such that ||3 S D, holds true for every i. In this Dirac setting, we derive the following formulation of diffusion Fisher, which a weighted outer- product sum devoid of gradients and composed solely of the
3

---

Efficiently Access Diffusion Fisher: Within the Outer Product Span Space
3360
Proposition
3.
Let
define v( y)
EDV
vl Fe - 201
exp
and w(aa,, y) as
ol00,.f.0) Jed vi(t (. (1o
E R. 90 takes form as
equation 12 the diffusion Fisher matrix of the diffused distribution for é (0, 1] can be analytically formulated
Stoge
follows:
(a)
(b)
at o²
dgu [/ [wlvivor 'gu
Figure 1: (a) The training loss of DF-TM for SD-1.5 5 and SD- 2base. It demonstrates commendable convergence behavior. (b) The trade-off curve of NLL and Clip score of SD-1.5 and SD-2base across various guidance scales [1.5, 2.5, 12.5, 13.5]
Fi(x;;, t) o²
(f (wwv)rddn)) ) ( ((s))argo)
(13)
where we simply write w(a,, y) as w(y). as long it does not lead to any contusion.
initial distribution and the noise schedule.
1. Defines (z( 1) as and wy as R. If 4o takes the form in equation equation [O] the diffusion Fisher matrix of the diffused distribution for / E (0, 1] can be analytically formulated as follows:
Proposition
We further ascertain that the (()))ydgo())) component in equation 13 can be effectively approximated by the score network in the form y-prediction, as demonstrated the following proposition.
exp a..l"" )
201
v.(
Proposition 4. Given the diffusion loss in equation and if gu conforms the form in equation 12 then the optimal ye (z,, t) can accurately estimate w (y)yddy())))))
a?
1) Fi(1,,
77 00
I
Σ waynn
o:
The derivation of the outer outer-product form of diffusion Fisher under the general setting is akin to the Dirac case, but in an integral form. For the remainder of the paper, we will focus on developing our method based on the Dirac setting diffusion Fisher. However, the same results can be naturally extended the general setting.
(11)
Σ wigri ( wayss
where have simplified (,, t) wi, as it does not lead to any confusion.
4. Diffusion Fisher Trace Matching
We also find that the Ej way; component in equation 11 can be effectively approximated by the trained score network in the form of y- u-prediction, as demonstrated in the following proposition.
The likelihood evaluation of DMs would require access to diffusion Fisher's trace. In this section, we introduce net- work to learn the trace, thus facilitating effective likclihood evaluation in DMs.
Proposition 2. Given the diffusion training loss in equation 4 and if go conforms to the form presented in equation T0, then the optimal ga(,, 1) can accu- rately estimate Σ, w.Yk:
Log-Likelihood in DMs Log-likelihood is classic and significant metric for probabilistic generative models, exten- sively utilized for comparison between samples or models [Bengio et al. 2013 Theis et al. 2015). According to Chen et al. 2018) 8); Song et al. 2021), I), the log-likelihood sam- ples generated by PF-ODE equation 66 from DMs can be computed through connection to continuous normalizing flows as follows:
The General Setting We further extend the diffusion Fisher formulation in equation II to a more general set- ting, where we only assume that the initial distribution go is probability measure on gd with finite second moments.
(General Setting)
To E P2(RR))
(12)
i log qi (z., t) (
0 dee
(f(I)a) 1
59 (1)a.. log
which means that 4/ P(RR), fEa (ell)g(()ddd < co. In this general setting, we derive the following form of diffusion Fisher, which a weighted outer proouct integral, which resides in a space spanned by an infinite set of outer products.
((10)) -000t) ax,²
0²
log log
=-I()d -²((), (F((((.))
(14)

---

Efficiently Access Diffusion Fisher: Within the Outer Product Span Space
fire hydrant stands the sidewalk by the street.
Large matorcycle sitting grassy line.
A bird resting outside of boat window.
picture of dog laying on the ground.
white stove top oven sitting inside of kitchen.
yellow commuter train parked at train station.
Prompts
(High quality)
NLL 27.84
NLL 25.54
NLL 27.36
NLL 25.74
NLL 28.59
NLL 28.57
NLL = 30.76
NLL = 27.31
NLL = 29.13
NLL = 29.37
NLL 30.09
NLL = 30.00
High (Low quality)
NLL 30.93
NLL 29.04
NLL=31.11
NLL 29.96
NLL 32.06
NLL 30.41
Figure Our DF-TM method facilitates the effective evaluation of the NLL of generated samples with varying seeds. can be demonstrated that lower NLL signifies region of higher possibility, thereby consistently indicating superior image quality.
where tr(-) denotes the trace of matrix, which defined to be the sum of elements the diagonal.
The relative emor NLL evaluation
Methods
1.0
t.8.8
t.6.6
t=
1= 0.2
t= 1.0.0
VJP (eq. 15
6.68%
5.79%
10.46%
20.13%
51.14%
70.95%
DF-TM (Ours)
3.41%
4.56%
4.13%
4.28%
5.33%
5.81%
Log-Likelihood Evaluation via VJP The current tech- nique is only capable of conducting backpropagation of scalar value the neural network. Therefore, the VJP in equation D cannot directly calculate the trace of the diffu- sion Fisher. The VJP must iterate through each dimension to compute the individual elements on the diagonal, and then sum them up as follows
Table Comparison
VJP method our DF-TM terms of the diffusion Fisher trace evaluation error across different timesteps. The error is evaluated on the 2-D chessboard data with the VB schedule.
the trace of the diffusion Fisher, we propose to directly ob- tain its analytical form and train network to match it. Note that the trace of an outer pnoturt of the vector results in the vector's 's norm. Thus, given the diffusion Fisher in Proposi- tion we can also derive its trace the form weighted norm sum, as highlighted the following proposition:
a (eo(x),)))(())
(FF((
(15)
at
Ox.
Evaluating the trace using the VJP method would be ex- tremely time-consuming due to the curse of dimensional- ity. If the time-complexity of single backpropagation is (d), then the calculation in equation 15 would have time- complexity of (dd). In practice, demonstrated in Table I evaluating the trace of diffusion Fisher on the SD-v1.5 model would require half an hour, rendering it nearly in- feasible. Some Monte-Casd Carlo type approximations can used accelerate the NIL evaluation of models, but they still fall short in per-sample NLL evaluation, see detailed discussion in C.5
In the same context as Proposition I the trace of the diffusion Fisher matrix for the diffused distribution gt, where (0, 1]. given by:
Proposition 5.
a² Σ un willerill
>>>>> wayh
tr (F;(;
o?
(16)
Log-Likelne Ligelikel Evaluation via DE trace matching To overcome the limitations of the VJP method evaluating
As demonstrated in Proposition 2 the |1 w.yll can be directly estimated by || (m (m, 1) Therefore, the only
5

---

Efficiently Access Diffusion Fisher: Within the Outer Product Span Space
at (r-nk si an Up
5045 + 3r-Ch
5043
S Score
Farm Lepn
50-2a og
5045 DI-AA (0um)
5045 up
25 +F
2045 sp
(0um)
52-2 up
Time cost in
Thes cont 90
Time cast
Them cuat
(a) SAC Aesthetic Score (T)
(b) Aesthetic Score (T)
(c) Pick- Score (
(d) Clip Loss (L)
(c) Face Loss ([)
Figure 3: Comparison hetween our DF method and the VJP method on adjoint guidance sampling across five objective scores: SAC/AVA
aesthetic score, Pick-Score, clip loss, and Face ID loss. Notably, our DF consistently achieves superior scores with less time expenditure.
unknown element in equation 16 is Σ, woll Conse-
Proposition 7. Assume the approximation error on to(æ,, 1) is 81 and on E, () is 62, then the ap- proximation error of the approximated Fisher trace in equation 17 is at most 2/ o, 3
quently, we suggest learning this term using scalar-valued network, as per the following training scheme:
o?
Algorithm Training of DF-TM Network
1: Input: data space dimension d, initial network to(:-)) IRd x R + IR, noise schedule }or} and {oz}.
Experiments We conduct toy experiments to show the accuracy and commercial-Ievel experiments to show the efficiency of our DF-TM method. In toy experiments, as shown in Table 2 we trained a simple DF-TM network on 2-D chessboard data and then evaluated the relative error of the trace estimated by VJP and DF-TM. It is shown that our DF-TM method consistently achieves more accurate trace estimation of DF compared to the VJP method.
2: repeat
3: Wi ~ 9o (#0)
4: t t~ Uniform (1....., T))
5: E g~ I)
6: 2t 0, + o,E
7: Take gradient descent on Vo to), ||
8: until converged
9:
Output: (=(::))
In commercial-level experiments, we trained two DF-TM networks for the SD-1.5 and SD-2base pipeline the Laion2B-en dataset (Schuhmann et al. 2022), which con- tains 2.32 billion text-image pairs. Our to follows the U-net structure, similar to stable diffusion models, but with an added MLP head to produce a valued output. We utilize the AdamW optimizer (Loshchilov & Hutter (019)) with a learning rate of le-4. The training executed across 8 V100 chips with batch size of 384 and completed after 150K steps. In Figure Ia we demonstrate that the train- ing loss of DF-TM nets converges smoothly, indicating the robustness of the DF-TM training scheme in Algorithm [
The training scheme detailed in Algorithm can in- deed enable to( to estimate the weighted norm term E, (,, 1 This is substantiated by the conver- gence analysis Proposition o
as presented below.
Proposition ((z,,t) t) E Rd to (a,, l)s trained by the objective in Algorithm D are equal to 1 Σ, w,( (o [yall?
x Rzo, the optimal
we evaluate the average NLL and Clip score of samples generated by various SD models, using 10k ran- domly selected prompts from the COCO dataset (Lin et al.) 2014). A lower NLL suggests more realistic data generation, while higher Clip score indicates a better match between the generated images and the input prompts. The results imply that the NLL and the Clip score form a trade-off curve across different guidance scales. This phenomenon, previously hypothesized in theory (Wu al., 2024), is now confirmed in the SD models, thanks to the effective NLL evaluation via the DF-TM method.
In Figure lb
Once we have obtained tg. fusion Fisher efficiently, as illustrated below. This approach 2
evaluate the trace of dif-
is a straightforward result of equation 16]and Propositions
and 6, which we refer to as DF trace matching (DF-TM).
d of = of of
tr (Fi
( to 1100 e)l
(17)
To estimate the trace using DF-TM in equation 17 we sim-
ply need one access to and Eg. DF-TM enables us to effectively evaluate the log-likelihood in a gradient-free manner with linear time complexity. We can further sub- stantiate the theoretical approximation error bound when using DF-TM to calculate the trace of the diffusion Fisher, as illustrated in the Proposition 7
In Figure 2 we display images with varying NLL under the same prompt with 10 steps on SD-1.5 using DDIM. It's clear that images with lower NLL exhibit greater visual realism, while those with higher NLL often contain deformed ele- ments (emphasized by the yellow rectangle). Our proposed
6

---

Efficiently Access Diffusion Fisher: Within the Outer Product Span Space
A bathroom sink and Prompts mirror reflecting shower and bathitu.
girl astronout exploring cosmos, floating amang planets and stars.
motor home.
detailed full-body depiction griffin.
An abandoned city with ruined buildings.
frezen cosmic rosz.
Original Images
Acs: 5.93
Acs: 4,95
Pic: 17.68
Pic: 18.92
Pie: 19.13
Pic: 18.46
VJP Adjoint
Aes: 68
402.6c 6a
Aes: 5.53
404.18
Pic: 19.12
194.4s
Pic: 19.23
194.7s
Pic: 20.24
1 194.0s
Pic: 18.98
3 193.5%
DE-EA Adjoint (Ours)
Aes: 6.38
3 285.0s
Aes: 5.68
1 2222.74
Pic: 20.27
3 163.3s
Pic: 19.80
3 164.6s
Pic: 20.63
162.5s
Pic: 19.29
( 162.4s
Figure 4: Visual comparison DF-EA (Ours) and VJP the adjoint improvement task on (left) SAC aesthetic score and (right) Pick-Score. DF-EA consistently generates images with better visual effects and reduced time expenditure.
NLL evaluation method proves to an effective tool for automatic sample selection.
Adjoint ODE via VJP. Regardless of the ODE solver being used, necessary to compute the right-hand side of equation 18 equivalently, F( t) A This compu- tation be interpreted applying the diffusion Fisher matrix as linear operator to the adjoint state A,, from functional analysis perspective (Yosida 2012). Current prac- tices utilize the VJP technique to approximate this linear transformation operation as follows:
5. Diffusion Fisher Endpoint Approximation
The adjoint optimization of DMs would require access to the matrix-vector multiplication of the diffusion Fisher, which can also be seen as applying diffusion Fisher as a linear op- erator. this section, we present training-fres method that simplifies the complex linear transformation calculations, thus enabling faster and more accurate adjoint optimization based on the outer-product form diffusion Fisher.
1 DEg (xt, t)
F(x,,t)
X ~
A
Ot dx.
(19)
(Ee (æt, 1))
Gt Ox:
Adjoint optimization sampling. Guided sampling tech- niques are extensively utilized in diffusion models to fa- cilitate controllable generation. Recently, to address the inflexibility of commonly used classifier-based guidance (Dhariwal & Nichol 202Tb) and classifier-free guidance (Ho& & Salimans 2022), series of adjoint guidance meth- ods have been explored (Pan al. 2023a
This process involves computationally expensive neural net- work auto-diiferrnniaaionn,, and the approximation errors introduced by the VJP technique have theoretical bound.
Adjoint ODE via DF-EA. As previously discussed in Section the DF inherently doesn't it require gradients, sug- gesting that we could potentially apply the diffusion Fisher as linear operator in a gradient-free manner. The chal- lenging part in equation is E; w.y.y, which represents weighted form of outer outer-products of data. Based on the definition of w,, the closest y; to æ6 will dominate - 0. with single final
Consider optimizing scalar-valued loss function L(-) Rd + IR, which takes in the data space as input. Adjoint guidance is implemented by applying gradient descent on at in the direction of (c:=) The essence of adjoint guidance is to use the gradient at = and follow the
This makes it intuitive to replace this sample outer-prood nuter-product Z00a%] It's also important to note that the adjoint guidance itself needs to compute Co at each guid- ance step, eliminating the need for additional computation to obtain zn- Given that we utilize the endpoint sample Zo. we refer to this approximation technique as DF Endpoint
adjoint ODE (Pollini et al.) 2018: Chen et al.] 2018 to compute A = (L(o)(())) for > O.
dæ:
d>, -A Tho t2), dar
a[(w))
No Day
(18)
dl

---

Efficiently Access Diffusion Fisher: Within the Outer Product Span Space
Approximation (EA). The formulation for DF-EA in the
optimal transport. Zhang et al. (2024a) has demonstrated that affine initial data suffices for the PF-ODE map to be optimal transport. However, the OT property of the general PF-ODE map remains an open question.
adjoint ODE is as follows:
F (Fee: t) Xx
1 aj
Σ wiyiyi (xr, (1)0))))0(( (w0s (1
A
In this section, we propose the first numerical OT verifica- tion experiment for a general PF-ODE deduced map based on the outer- outer-product form diffusion Fisher we obtained in section 3 We first derive the following corollary for the OT property of the PF-ODE deduced map.
1
1n (ar (x () A.
A
0i (mra X1) mo (To( Ar) (2r,, t)
af
Corollary 1. Denote the diffeomorphism deduced by the PF-ODE in equation 5 as follows
(20)
The DF-EA approximation leads to a scalar-weighte combi- nation of X, To. and (2t,t), which importantly, does not involve any gradients. Additionally. we derive the theoreti- cal approximation error bound of the DF-EA in Proposition To measure the accuracy of DF-EA as a linear operator, we opt to use the Hilbert-Schmidt norm (Gohberg et al.. 1990) for measurement, as follows:
Te. R R" F. De,
AA 2 8 > 0. (21)
The diffcomorphism Tur is Monge optimal transport map if and only if the normalized fundamental matrix for ((t) = (t,, xx)) at s is s.p.d. for every PF-ODE chain that starts from ar RR. where
(4)
aigg(() 200
Proposition 8. Assume that the approximation error ( t) is 62, the approximation error of the DF- EA linear operator, as referenced in 20 is at most
B(t, = [(t)
20? I
Σ wi w:y.ys
Σ wiy:
Σ
ways
oi 2DD + vd82) when measured in terms of the Hilbert-Schmide norm.
(22)
The definition of the normalized fundamental matrix is deferred to Appendix A.10
Experiments on DF-EA. As depicted in Figure 3 we conducted experiments comparing our DF-EA and VJP methods in adjoint guidance sampling, using five different scores and two different base models. DF-EA consistently achieves better scores due to its bounded approximation er- ror. Furthermore, DF-EA requires less processing time as it eliminates the need for time-consuming gradient operations. DF-EA and VJP are compared under the same guidance scales and schemes across various numbers of steps. Details regarding the score function can be found in Appendix B.2
Note that Ts.; is well-posed, guaranteed by the global ver- sion of the Picard-Lindelöf theorem (Amann 2011 Zhang] ct al.
2024a). Detailed proofs can be found in Appendix A.10 We then design the numerical OT verification experi- ment for a given noise schedule and initial data as shown in the Algorithm 2 The detailed version of Algorithm 2 can be found in Appendix B.3
Algorithm 2 Numerical OT test for PF-ODE map
As depicted in Figure our DF-EA consistently generates samples with higher scores with a reduced time cost com- pared to VJP. All samples are generated within 50 steps, with adjoint applied from the 15th to the 35th step. Details on hyperparameters can be found in Appendix B.2
1: Input: initial data {y, ) noise schedule far) and fach. discretization steps M.
2: Initialize Av = I, aM (0.
3: for = M, M 1,... ,1 do
4: di = f4-1 - t.
5: Calculate B, by equation 22
6: A A. + dt A B.
(solve fundamental matrix.)
7:
ai-1 PF-ODE Solver(ær..) i)
6. Numerical OT Verification of DMs
8:
end for
9: Output:
The result fundamental matrix.)
There is an increasing trend towards analyzing the proba- bility modeling capabilities of DMs by interpreting them from an optimal transport (OT) perspective (Albergo al.] 2023 Chen al. 2024). The foundational concepts of optimal transport can be found in Appendix A.9 One of the central questions is whether the map deduced by the PF-ODE could represent an optimal transport. Khrulkov ct al. (202)) have proven that, given magle-Gaussian initial data and a VE noise schedule, the PF-ODE deduced map is
In Table we numerically tested the OT property of the PF-ODE map under different nose schedules and initial data. We tested four commonly used noise schedules, VE (Ho et al. 2020). VP (Song & Ermon 2019), sub-VP (Song et al. 2020 and EDM (Karras et al.) 2022) on 2-D data. We first tested when the initial data is single-Gaussian affine, we found that the result normalized fundamental
Gaussian and
8

---

Efficiently Access Diffusion Fisher: Within the Outer Product Span Space
Initial Data
Single Gaussian
affine
comments. Fangyikang Wang wishes to express gratitude Pengze Zhang from ByteDance, Binxin Yang, and Xinhang Leng from WeChat Vision for their insightful discussions on the experiments.
Noise Schedule
Asym.
OT
Asym,
OT
Asym.
OT
VB
0.00%
0.00%
V
25.28%
Song Ermon
2019
(Ho et al. 2020
VP
0.00%
0.00%
/ 23.36%
X
sub- VP
0.00%
0.00%
/
13.84%
X
Song al. 2020
BDM
Impact Statement
Karras et
2022
0.00%
0.00%
27.09%
The development of accurate diffusion Fisher, as discussed in this paper, holds significant potential for several domains, including machine learning, healthcare, environmental mod- eling, and economics. However, while this research holds great potential for positive impacts, it is also important to consider potential negative societal impacts. The enhanced ability of generative models given by DF could potentially be misused. For instance, could be exploited to create aesthetically improved deepfakes, leading misinforma- tion. In healthcare, if not properly regulated, the use of synthetic patient data could lead to ethical issues. Therefore, it is crucial to ensure that the findings of this research are applied ethically and responsibly, with necessary safeguards in place to prevent misuse and protect privacy.
Table 3: Comparison of numerical OT verification results of four commonly used noise schedulers with different initial data.
matrix is p.s.d., which means the PF-ODE map is optimal transport. These results coincide with the previous proposed theoretical results in (Khrulkov et al. 2023) and (Zhang et al. 2024a). We further tested an extremely simple non- affine case, that is, there are only three initial data points, (0,0), (0,0.5), and (0.5,0). We showed that all noise sched- ules result in an obvious asymmetric fundamental matrix, which means that the PF-ODE map fails to be OT in these cases. Here, use the Frobenius norm of the difference of A and its transpose divided by the Frobenius norm of A to measure its asymmetric rate. Based on this finding, we hypothesize that the PF-ODE map can only be OT in the affine initial data (Single-Gaunl Gaussian is special case of affine data), but fails to be OT in general non-affine data. Our hypothesis coincides with the statement that the flow map of a Fokker-Planck equation is not necessarily OT as proposed in (Lavenant & Santambrogio 2022). (Note that PF-ODE is a special class of Fokker-Planck equations)
References
Albergo, M. S., Boffi, N. M., and Vanden-Eijnden, E. Stochastic interpolants: unifying framework for flows and diffusions. arXiv preprint arXiv:2303.08797, 2023.
Amann, H. Ordinary differential equations: an introduction nonlinear analysis, volume 13. Walter de gruyter, 2011.
7. Conclusions
This paper derives an outer-product span formulation of the diffusion Fisher and introduces two approximation algo- rithms for different types of access to the diffusion Fisher: diffusion Fisher trace matching (DF-TM) and diffusion Fisher endpoint approximation (DF-EA). Both methods are theoretically guaranteed in terms of approximation error bounds, and offer improved accuracy and reduced time- cost compared to the traditional VJP method. We also designed the numerical OT property verification experi- ment for the PF-ODE map based on the outer-product form of DF. This work not only improves the efficiency of diffusion Fisher access but also widens our understand- ing of the diffusion models. Please refer to further dis- cussions in appendix The code is available at https: //github. com/ zi /ituttut/iiffusiionttt isher
Anderson, B. D. Reverse-time diffusion equation models. Stochastic Processes and their Applications, 12(3):313- 326, 1982.
Bao, F., Li, C., Sun, J., Zhu, J., and Zhang, B. Estimating the optimal covariance with imperfect mean in diffusion probabilistic models. arXiv preprint 2xXx:22066073099 2022a.
Bao, F., Li, C., Zhu, J., and Zhang, B. Analytic-dpm: an ana- lytic estimate of the optimal reverse variance in diffusion probabilistic models. arXiv preprint Stti:22011005033 2022b.
C
Bengio, Y., Yao, L., and Cho, K. Bounding the test log-likelihood of generative models. arXiv preprint arXiv: 2013.
Acknowledgments
This work was supported in part by the National Natural Science Foundation of China under Grants 62206248 and 62402430, and the Zhejiang Provincial Natural Science Foundation of China under Grant LQN25F020008. We would like to thank all the reviewers for their constructive
Benton, J. et al. Nearly d-linear convergence bounds for dif- fusion models via stochastic localization. In The Twelftth International Conference on Learning Representations, 2024.

---

Efficiently Access Diffusion Fisher: Within the Outer Product Span Space
Blasingame, Z. W. and Liu, C. Adjointdeis: Effi- cient gradients for diffusion models. arXiv preprint arXiv:2405.15020, 2024.
Ho, J., Jain, A., and Abbeel, Denoising diffusion proba- bilistic models. Advances in Neural Information Process- ing Systems, 33:6840-6851, 2020.
Brenier, Y. Décomposition polaire et réarrangement mono- tone des champs de vecteurs. CR Acad. Sci. Paris Sér. Math., 305:805-88 8088 1987.
Ho, J., Saharia, C., Chan, W., Fleet, D. J., Norouzi, M., and Salimans, T. Cascaded diffusion models for high fidelity image generation. Journal of Machine Learning Research, 23(47):1-33, 2022.
Brenier, Y. Polar factorization and monotone rearrangement of vector-valued functions. Communications on pure and applied mathematics, 44(4):375 1991.
Kantorovich, I., V. Mathematical methods of organizing and planning production. Management science, 6(4):366 422, 1960.
Chen, R. T., Rubanova, Y., Bettencourt, J., and Duvenaud, D. K. Neural ordinary differential equations. Advances in neural information processing systems, 31, 2018.
Karras, T., Aittala, M., Aila, T., and Laine, S. Elucidating the design space of diffusion- based generative models. arXiv preprint arXiv:2206. 00364, 2022.
Chen, Z., Li, H., Wang, F., Zhang, O., Xu, H., Jiang, X., Song, Z., and Wang. E. H. Rethinking the diffusion models for numerical tabular data imputation from the perspective of wasserstein gradient flow. arXiv preprint arXiv:2406. 15762, 2024.
Khrulkov, V., Ryzhakov, G., Chertkov, A., and Oseledets, I. Understanding DDPM latent codes through optimal trans- port. In International Conference on Learning Represen- net/
tations, 2023. URL
Forum ?id 6PIrhAx1j43
Cheng, F., Li, H., Liu, F., Van Rooij. R., Zhang, K., and Lin, Z. Empowering ilms with logical reasoning: A com- prehensive survey. In Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence, 2025.
Kingma, D., Salimans, T., Poole, B., and Ho, J. Variational diffusion models. In Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang P., and Vaughan, W. (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 21696-21707 Curran Associates, Inc.,
2021. URL http:://proceednngs http:/pprooeeddngggeee.rrssss
Dhariwal, and Nichol, A. Diffusion models beat gans image synthesis. In Advances in Neural Information Processing Systems, volume 34. pp. 8780-8794, 2021a.
CC
fi httpppeesiieeeeppee//////2/
b5 7578 77782255220222077fffff2222222220 Pape Paper.
pdf
Dhariwal, P. and Nichol, A. Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems, 34:8780-8794, 2021b.
Lang, S. Fundamentals of differential geometry, volume 191. Springer Science & Business Media, 2012.
Lavenant, H. and Santambrogio. F. The flow map of the fokker-planck equation does not provide optimal trans- port. Applied Mathematics Letters, 133:108225, 2022.
Feng, Q., Zhao, H., Zhang, C., Dong, J., Ding, H., Jiang, Y.-G., and Qian, H. Pectp: Parameter-efficien cross- task prompts for incremental vision transformer. arXiv preprint ar arXhv:2407.03813, 2024a.
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ra- manan, D., Dollár, P., and Zimick, C. L. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pp. 740- 755. Springer, 2014.
Feng, Q. Thou, D., Zhao, H., Zhang, C., and Qian, H. Lw2g: Learning whether to grow for prompt-based continual learning. arXiv preprint aoXiv: 18860, 2024b.
Fu. H., Zhao, H., Dong, J., Zhang, C., and Qian, H. lap: Improving continual learning of vision-language models via instance-aware prompting. arXiv preprint arXiv:2503.20612 2025.
Lipman, Y., Chen, R. T., Ben-Hamu, H., Nickel, M., and Le, M. Flow matching for generative modeling. arXiv 2022.
preprint arXiv:2210.022.
Gohberg, I., Goldberg, S., Kaashock, M. A., Gohberg, I., Goldberg, S., and Kaashoek, M. A. Hilbert-schmidt oper- ators. Classes of Linear Operators Vol. I, pp. 138-147, 1990.
Liu, I.,, Ren, Y., Lin, Z.,, and Zhao, 7. Pseudo numeri- cal methods for diffusion models on manifolds. arXiv preprint arXiv:220..09778. 2022.
Loshchilov, I. and Hutter, F. Decoupled weight decay reg- ularization. In International Conference on Learning Representations (ICLR), 2019.
Ho, J. and Salimans, T. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12599, 2022.
10

---

Efficiently Access Diffusion Fisher: Within the Outer Product Span Space
Lu, C.,, Zheng, K., Bao, F., Chen, J., Li, C., and Zhu, J. Max- imum likelihood training for score-based diffusion ODEs by high order denoising score matching. In Chaudhuri, K., Jegelka, S., Song, L., Szepesvari, C., Niu, G., and Sabato, S. (eds.), Proceedings of the 39th International Confer- ence on Machine Learning, volume 162 of Proceedings of Machine Leurning Reseurch, pp. 14429-14460) PMLR, 17-23 Jul 2022a. URL bttps: htt.:latttpppcceeeeeeeeee /ttp://ecoceeengsgoeee mlr. www.ppres...m 1 162 1u22E/lu2E... html
Rockafellar, R. T. Convex analysis<pmss-88) 2015.
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. High-resolution image synthesis with latent diffusion models, In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pp. 10684-10695, 2022.
Saharia, C., Chan, W., Saxena, S., Li, L.,, Whang, J., Denton, E. L.,, Ghasemipour, K., Gontijo Lopes, R., Karagol Ayan, B., Salimans, T., Ho, J., Fleet, D. J., and Norouzi, M. Photorealistic text-to text-to-image diffusion models with deep language understanding. In Advances in Neural Informa- tion Processing Systems, volume 35, pp. 11479-36494, 2022.
Lu, C., Zhou, Y., Bao, F., Chen, J., Li, C., and Zhu, Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. Advances in Neural Information Processing Systems, 35:5775-555 -78777 2022b.
Lu. C., Zhou, Y., Bao, F., Chen, J., Li, C.,, and Zhu, J. Dpm- solver++: Fast solver for guided sampling of diffusion probabilistic models. arXiv preprint arXiv:2211.01095, 2022c.
Salimans, T. and Ho. Progressive distillation for fast sampling of diffusion models. arXiv preprint urXiv:2202.00512,222 2022.
Masuyama, H. Limit formulas for the normalized fun- damental matrix of the northwest-corner truncation of markov chains: Matrix-infinite- taari---iinnit--rouuu----ooo product-form solutions of block-hessenberg markov chains. arXiv preprint 2016.
Santambrogio,, F.
Optimal transport for applied mathemati- cians. Birkätseer NY, 55(58-63):94, 2015.
Schuhmann, C., Beaumont, R., Vencu, R., Gordon, C. W., Wightman, R., Cherti, M., Coombes, T., Katta, A., Mullis, C., Wortsman, M., Schramowski, P., Kundurthy, S. R., Crowson, K., Schmidt, L., Kaczmarezyk, R., and Jitsev. J. LAION-5b: An open large-scale dataset for training next generation image-text models. In Advances in Neural Information Processing Systems (NeurIPS), pp. 25278- 25294, 2022.
Monge, G. Mémoire sur la théorie des déblais et des rem- blais. Mem. Math. Phys. Acad. Royale Sci., pp. 666-704, 1781.
Nichol, A. Q. Dhariwal, P., Ramesh, A., Shyam, P., Mishkin, P., Megrew, B., Sutskever, I., and Chen, M. GLIDE: Towards photorealistic image generation and editing with text-guided diffusion models. In Proceed- ings of the 39th International Conference on Machine Learning, volume 162, pp. 17784-16804 2022.
Skorski, M. Modern analysis of hutchinson's 's trace estimator. In 2021 55th CISS. IEEE, 2021
Sohl-Dickstein, J., Weiss, E., Maheswaranathan, N., and Ganguli, S. Deep unsupervised learning using nonequi- librium thermodynamics. In International Conference on Machine Learning. pp. 2256-2265. 2265. PMLR, 2015.
Osler, T. J. Leibniz rule for fractional derivatives generalized and an application to infinite series. SIAM Journal on Applied Mathematics, 18(3):658-615 674, 1970.
Song, K. and Lai, H. Fisher information improved training-fres conditional diffusion model. arXiv preprint arXiv:2404.1825 2024.
Pan, J., Liew, J. H., Tan, V. Y., Feng, J., and Yan, H. Adjoint- dpm: Adjoint sensitivity method for gradient backpropa- gation of diffusion probabilistic models. arXiv preprint STtV:2307.107111 2023a.
Song, Y. and Ermon, S. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019.
Pan, J., Yan, H., Liew, J. H., Feng. J.. and Tan, V. Y. Towards accurate guided diffusion sampling through symplectic adjoint method. arXiv preprint arXiv:2312. 12030, 2023b.
Song, Y., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Er- mon, S., and Poole, B. baore-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011. 1.001.11446,, 2020.
Pollini, N., Lavan, O., and Amir, O. Adjoint sensitivity analysis and optimization of hysteretic dynamic systems with nonlinear viscous dampers. Structural and Multidis- ciplinary Optimization, 57:2273-225 2229,222,,222 2018
Song, Y., Durkan, C., Murray, I., and Ermon, S. Maxi- mum likelihood training of score- based diffusion models. Advances in neural information processing systems, 34: 14185-128,, 2021.
Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., and Chen, M. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.0 02/55, 2022.

---

Efficiently Access Diffusion Fisher: Within the Outer Product Span Space
Theis, L., Oord, A. v. d., and Bethge, M. A note the evaluation of generative models. arXiv preprint arXiv:1511.0188 2015.
Zhang, P., Yin, H., Li, C., and Xie, X. Formulating discrete probability flow through optimal transport. Advances in Neural Information Processing Systems, 36, 2024a.
Tu, J., Fu, H., Yang, F., Zhao, H.,, Zhang, C., and Qian, H. Texttoucher: Fine-grained text-to-touch generation. In Proceedings of the AAAI Conference on Artificial Intelli- gence, volume 39, pp. 7455-7463, 2025a.
Zhang, P. Yin, H., Li, C., and Xie, X. Tackling the singular- ities at the endpoints of time intervals in diffusion models. arXiv preprint arXiv:2403. 08381, 2024b.
Zheng, K., Lu, C., Chen, J., and Zhu, J. Improved tech- niques for maximum likelihood estimation for diffu- sion ODEs. In Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato, S., and Scarlett, (eds.), Pro- ceedings of the 40th International Conference Ma- chine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 42363-42389, PMLR, 23-29 Jul 2023.
Tu, J., Ji, W. Zhao, H., Zhang, C., Zimmermann, R., and Qian, H. Driveditfit: Fine-tuning diffusion transformers for autonomous driving data generation. ACM Transac- tions on Multimedia Computing, Communications and Applications, 21(3):1-29, 2025b.
Villani, et al. Optimal transport: old and new, volume 338. Springer, 2009.
press/
Zhu, H., Wang, F., Ding, T., Qu, Q., and Zhu, Z. Analyzing and improving model collapse in rectified flow models. arXiv preprint arXiv:2412.08/75, 2024a.
Wallace, B., Gokul, A., and Naik, N. Edict: Exact diffusion inversion via coupled transformations. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22332-2254,, 2023.
Zhu, H., Wang, F., Zhang, C., Zhao, H., and Qian, H. Neural sinkhorn gradient flow. arXiv preprint vrXiv22401.14069 2024b.
Wang, F. Yin, H., Dong. Y.J.,, Zhu, H., Zhao, H., Qian, H., Li, C., al. Belm: Bidirectional explicit linear multi-step sampler for exact inversion in diffusion models. Advances in Neural Information Processing Systems, 37:461 18- 46159, 2024a.
Wang, F., Zhu, H., Zhang, C., Zhao, H., and Qian, H. Gad- pvi: A general accelerated weight particle-based variational inference framework. Entropy, 26(8):679, 2024b.
Wang, F., Yin, H., Qian, L., Li, Y., Zhuang. S., Zhu, H., Zhang, Y., Tang. Y., Zhang, C.,, Zhao, H., et al. Unleash- ing high-quality image generation diffusion sampling using second-order
arXiv
preprint ar arXiv:2505.24220 2025.
Wu, Y., Chen, M., Li, Z., Wang, M., and Wei, Y. Theoretical insights for diffusion guidance: A case study for gaussian mixture models. arXiv preprint arXiv:2403. 0/6.99,,0,, 2024.
Yang, Z., Feng, R., Zhang, H., Shen, Y., Zhu, K., Huang, L., Zhang, Y., Liu, Y., Zhao, D., Zhou, J., et al. Lips- chitz singularities in diffusion models. In The Twelfth International Conference on Learning Representations, 2023.
Yosida, K. Functional analysis, volume 123. Springer Science & Business Media, 2012.
Zhang, G., Lewis, J. P., and Kleijn, W. B. Exact diffusion inversion via bi-directional integration approximation. arXiv preprint orXiv:2307. 10829, 2023.
12

---

Efficiently Access Diffusion Fisher: Within the Outer Product Span Space
Appendix
A. Proofs and Formulations
A.1. Proof of Proposition !
Notice that, in the subsection, we can do interchange of sum and gradient; this is due Leibniz's rule (Osler 1970) and the boundness condition set in equation I0 Before give the proof of Proposition we would like establish two technical lemmas. The first lemma is about the first partial derivative of os (mx, () with respect to Zt.
Lemma 1.
(- [22 Cce 1 20
dv, (xc,)
eexp
dv.
Pa.t
1
[1 - -0:3:! 12
(23)
(Et (x.9)) exp
di
20
(t - a434) (0(3)))((211,))
The second lemma about the Jacobian of 3; we (20,, t) ys w.r.t. xt. Lemma 2.
i Σ wi (at, yi
dx
Ows (Ies t)
Σ
dx.
T
a
U/ (xe,
Σ
Dg Ui (2re
(ae.))))))
Ex Uk (2re, t)] - Us (zie, 1) Bha. Eg My (ase, 1)
T
Σ yi
8m.
E> Un (ate, 1))²
T
Σ
1
(xt a1yi) Us Σ Uk us (x).
(- (-o²
Σ (ace - assss)
Ex vk)
yi
0,
0,²
(24)
1
Σ
Viyli
(x: aryj) Σ Uk
(x. (4:31a) Uj
Cx )x)
o
Σ Vigl
Uk Ot
Uj
Ex 0k)
Σ, U.y [E uk
Σ Dilli
0,
CE )x)
En ke UN
En k Us
Σ wayfi Σ wiys ]
Ott
Σ we W.U.
o?
Now we are ready to give the Proof of Proposition !
Proof. According the initial distribution (equation 10 and the diffusion kernel (equation 1 the marginal distribution at
13

---

Efficiently Access Diffusion Fisher: Within the Outer Product Span Space
some time / > 0 would be
p (o, 1)
Σ (2ma?)
-
exp
[r
Ox.yi 2
(25)
2
Thus, the log-density has the following analytical formulation
[xt
oityi 12
log pogp( 1) = log
(2TO²)
Σ exp
N
20,
[tt - a+ 12
= log
Σ exp
+C
(26)
207
log
Σ Us (0tr, t)
+C
The can be expressed follows
log p (zt. f) =
log Σ 2s (2t,,)
Sæ.
a
D (xte, t]]
Σ; (Itt
1
E, (mt O4tll) 03
(27)
o7
Σn Us (z)..)
2t au Σ Wy (xt t)
of
The Fisher information we want can then be calculated by further applying a gradient on the score.
θ i
(x
log (z)).t)
0xc
0
1
D1 Ct Σ wy (ar 1) Ms
7
07
(28)
1
θ
we (are, 1) y
I
o7
o7
dxx
I
a?
Σ
wiysy!
Σ wayss
Σ waytt
(by Lemma 2
di
o
This proof is inherently the calculation the Hessian of log-convolution of density, we provide the detailed derivation here for completeness.
A.2. Proof of Proposition
Proof. Given fixed (z, t). L a quadratic form of ye. To obtain the optimal ye. we differentiate and set this derivative equal zero, resulting in the following
14

---

Efficiently Access Diffusion Fisher: Within the Outer Product Span Space
ac
1
(2TT²)
oa t) of
(o( 3/
dyo (z),
dre (xr,
N
at 2A
2z ())o y1))
(29)
o
which yields
Uk (x t)
y6 (1 1) Σ
Σj U1 (1t Yh = Σ wayi
(30)
A.3. Proof of Proposition 3
Notice that, the subsection, we can do an interchange integral and gradient; this is due to Leibniz's rule (Osler 1970 and the bounded moments condition set in equation [2] Before we give the proof of Proposition [ would like to establish two technical lemmas. The first lemma is about the first partial derivative of v( y) w.r.t. Zt.
Lemma 3.
a eexp
le, 10.0 207
Ov(wi, 4,y)
Oæ:
Ox:
at a,3
(31)
[z; - a:yy exp
207
(x: aay))((ai,tt.))
Lemma 4.
Jga un(ar, t, y)
Ow (2os, t 8)
(go(y)
2d
vlare, (. [ u(x t, y ddoo (2r") 1)² dgo(zr)
Oxc
a. [[ vares i, 3 (dgo()/)) (z/')] v (2rt, 3) Dmg [No vfar,, v ddoo (3/') dgo(2))
Uae u(20,, t, y" dd)""("""")
(xt aur) v(y) / u(2' )doo(s)) - v(y) ( (- / (as ary' viy' / {
-(()) 0 / o(s) (z:: (zo--ar) ary) / ((s')ddu(xr))
o('')doo(zz)) )dao(ss) deo(s)) (32)
}
of
(#1- ais') uis' (oo'rioss } dgu(y)
/ n(uiee -ory / r( ((e'')dgolv')) ar Mo'''''d' dgo(v)
[S 2
[ ((y)yy dgo ' ol Adgloe))
( (
( (((u)rrgu()))
of
[] (fo())")dgl (1r))
[ "wy)yy' dgo(y) (/ (((y))rreo()))
(ʃ (()))dg((()))
Proof. According the initial distribution (equation 12p and the diffusion kernel (equation []. the marginal distribution at
15

---

Efficiently Access Diffusion Fisher: Within the Outer Product Span Space
some time / 0 would be
exp (- Jtt a,yy
p (1t, ()
(2TO?)
dgo(y)
(33)
20
Thus the log-density has the following analytical formulation
(- |st a,y"
log qe (2t; t) log
(2mo?)
- exp
dgole)
20,
2t anyl
log
exp
dou(y)
+C
(34)
20,p2
Ra
log
C v (zr, 4) dgo(3))
Rd
The score can expressed as follows
0
2 log Ue (xi, 2) (g)(y)
log p (Ir,
a Sæe.
[ (x t, 3) (z)
Sza (zr,,t, t, y) dgo(z)
JR 800i (xx, t, x) (g)
(by and Leibniz integral rule)
(35)
Jna (20,, 1, (g)
of Ja (xr. a.yy) u (a L, y) (g)
JR v (I,, t, 4) dgo
of / w(a,, yddo
The Fisher information we want then be calculated by further applying gradient the score.
θ
Fx
log (z.,
dx,
0
It at { w(x,, t, y)
}
dx.
ii
7
Rd
0Ja w(a t, ()
(36)
I
o7
o
dx.
00
2
w,yy ddo(z)) ( w.ydgoos)) ) (ʃ w.yrdqu())
]
I
(by Lemma 4
0
0
This proof also inherently the calculation of the Hessian of a log-convolution of a density, we provide the detailed derivation here for completeness.
A.4. Proof of Proposition 4
Proof. Given fixed (z, t), L a quadratic form of ye. To obtain the optimal ye. we differentiate and set this derivative equal zero, resulting in the following
16

---

||00/| 2 Efficiently Access Diffusion Fisher: Within the Outer Product Span Space
0C dge (xt,
of dÿs (2e, () Jxe N (2mо?)- u( t, y)
(0 t) yll yll'dgo (3)
o7
= 2A at of / 1, (iola - y)dgoly)
(37)
which yields
(0r, t) [ Spe v "do (y)) y/ doo (y)) = - w(x y y')y/ (y'dqo(y')
v( t, t²)
(38)
A.5. Proof of Proposition 5
Lemma 5. Given vector E Rd the trace of the outer-product matrix of this vector is precisely equal to the square of its 2-norm. This can be shown as follows:
(vv*)
Σ (ve) )r,-
21 * 02
|| 2
(39)
Proof.
Then we can start to give the derivation of Proposition 5
tr (F,(,, 1)) = tr
at
T
Σ w.y.y, -
Σ with
Σ
wiys
((I ways
02
(I)
Σ
watr (3:21
ways
(40)
o
d
a
Σ
wil
|
Σ
wiy/
of
o
A.6. Proof of Proposition 6
Proof. The objective in Algorithm
obviously equals to:
arg min (1o
to( to
(41)
d
By expressing the expectation Equation equation in the form marginal distribution, we can transform the objective follows:
arg min
N (2
to(
(42)
17

---

Efficiently Access Diffusion Fisher: Within the Outer Product Span Space
The optimal lg must satisfy the condition that the gradient of the loss equals 0. Therefore, have:
13 ( t)
Σ N (2TO7) 4 (xe,)) t)
0 = (2s,)))
d
E (20, (F) 1Beel
13:.11
A. Σ (1 ()t(((e;))))) A. Σ wx
d
Thus
Ar L: 12 (2 1) 1lve 1
t0,,, t)
A E, Ug (2(e))
U/ (2
13
(43)
En (x t)
d
Σ wi (2ee, |191111
We have successfully completed the proof that the optimal (x,,, 8). trained by Algorithm D is equivalent
A.7. Proof of Proposition 7
The approximation error of the estimated trace equation 17 will be its difference from the true Fisher information trace equation T6. We use consecutive Cauchy-Schwarz and triangle inequality to get the bound the approximation error:
æe - a.sa(arr () 1112)
Σ us 11vil1 -
Σ Wi
d
to tz(
o
o
a2
wil 11il" || Σ wiyk
23 0.50 (2t t) 12)
Lo (o(201)))
04
(44)
Σ wc 1 2
to (xx,, t)
+ Σ
wiy:
23 0:E0 (z 1)
a4
03
61 +
07
01
18

---

Efficiently Access Diffusion Fisher: Within the Outer Product Span Space
A.8. Proof of Proposition g
]
- 0t
ot
Σ w:y.
Σ wiyk
Σ
ag "o 10o (zze, (0re 1)T)
wyys
01
of
HS
Σ w;y.y!
Σ
woys
Σ
wiys
(o go ()00());))))))))))))))))))
HS
y:
Σ ii yt
Σ
wi
go ())0((0(;)) 8)
o
(45)
HS
HS
T
Σ
wi 19:21
x
11
+
at
Σ
Wiyli
wiyi
go (xx,
0
0
HS
at
Σ
wi max ||4.
2020
+
Σ
Σ
v0 t)
Σ
wiy: Vo
o
11
0
[2 vdd2)
A.9. Preliminaries on Optimal Transport
The optimal transport is the general problem of moving one distribution of mass to another as efficiently as possible. The optimal transport problem can be formulated two primary ways, namely the Monge formulation (Monge 178T) and the Kantorovich formulation (Kantorovich) 1960). Suppose there are two probability measures μ and v on (IR", B), and cost function e T x TRn (.0) The Monge problem is
inf { ((,,,(())) ((z)) Tel TEM= V T
(MP)
(46)
The measure T is defined through ((A) (A) for every A E B and is called the pushforward of μ through T.
It is evident that the Monge Problem (MP) transports the entire mass from particular point, denoted x, to a single point T(z). In contrast, Kantorovich provided more general formulation, referred to as the Kantorovich problem:
inf (Eexace cdy: E M11,v))
(KP)
(47)
where (7,,, v) is the set of transport plans, i.e.,
(1 = (7 P(R" I²) : (Tm)47 = 44, (Ty)e v},
(48)
where n, and Ny are the two projections of R2" x Ra onto R", For measures absolutely continuous with respect to the Lebesgue measure, these two problems are equivalent (Villani et al., 2009). However, when the measures are discrete, they entirely distinct the constraint of the Monge Problem may never be fulfilled.
A.10. Proof of Corollary 1
To prove the Corollary D we first introduce two theorems to transform the problem of whether the PF-ODE mapping is a Monge map into the task of deciding the convexity of the potential function of Tg.T-
(Santambrogio/ 2015 Theorem 1.48) Suppose that μ is probability measure on (Ik", B) such that ʃ | (p))) < 2o and that u E. + R {+oo} is convex and differentiable μ-a.e. Set T = Vu and suppose [(p)))) oc. Then T is optimal for the transport cost c(x, y) z/1 w/2 between the measures μ and
Theorem
V' T.p.
19

---

Efficiently Access Diffusion Fisher: Within the Outer Product Span Space
Theorem 2. The Brenier's Theorem. (Santambrogio 2013 Theorem 1.22) (Brenier 1987: 1991) Let ja, be probabilities over Rd and c(z, y) x y/.. Suppose J | da. lw| dy < +oo, which implies min(KP) < +000 and suppose that gives no mass to (d 1) surfaces of class C² Then there exists, uniquely, an optimal transport map from i to v, and it is of the form T = Vu for a convex function u.
To ensure the existence of the potential function, we need to leverage the following
Theorem 3. The Poincaré's Theorem. (Lang 2012 Theorem 4.1 of Chapter V, $4) Let U be open ball in I= and let w be differential form of degree > 1 on U such that du 0. Then there exists differential form Φ on U such that dő w.
Remark 1. The conclusion remains valid when the open ball U is substituted with the entirety of R
For is clear that dor(ar) is non-singular, and therefore, it satisfies the requirements of Brenier's Theorem leading
(1,)
the existence of unique optimal transport map. According to Theorem if we can establish that the potential function
of T convex, then the PF-ODE mapping will indeed be Monge map. Notice that the existence of the potential map guaranteed by Poincaré's Theorem
We can now convert the condition of the potential function of T. being convex into the condition that its Jacobian,
8T, r(.) is positive semi-definite, as per the following theorem in convex analysis. ds.
Theorem 4. (Rockafellar 2015 Theorem 4.5) Let f be a twice continuously differentiable real-valued function an open convex set C' in R?" Then f is convex on C' if and only if its Hessian matrix
f
(z)(2) = dE, Qx = (q)(())),
(EI.....En) Enn)
(49)
is positive semi-definite for every x E C.
If we denote that
(Tr)(())
A(t)
(50)
Or:
obviously, ATT) = is p.s.d., our goal is to answer when A() p.s.d.. We try to answer this to set up connection
between At) and A(T) I. We can derive that:
d!(() lim c-0
A(t <) A(L)
di
A(I+ A(L+ ax te (s())=
-000) Van. log, (mx)) + O
2
-lim +
(ftt)((t + c) 200
cA(t + Va, log, (x())) + O(22)
lim
-
e
=f())(1)) + (2))
A(I)Vm, log, (1xz)
2
=((()A()) g2())
1- of
[
"(y)yy ((y)yyTdqo0z)) dgo(l)) (/ w(t))dgo())) (
}
A(t)
w (((t)ydgo()))
2
00
of
(t)
+ a (t)
-A(1)
1(1)
Σ w.y.y. Σ w.ys
Σ wifi
20
201
B(O)
(51)
20

---

Efficiently Access Diffusion Fisher: Within the Outer Product Span Space
Notice that the above ODE starts from T.
According to the Solution Matrices theory (Masuyama 2016 let denote the C(t) is the normalized fundamental matrix
at T for (T), which implies (tt) is the solution to the following ODE:
C'(t) (()))((())())C)))))))) =,
(flow from T to
(52)
Then we can deduce that
(T,)(1))
A(I)
C(I)A(T)
(53)
= =C(II
CC()
Thus, the diffeomorphism T. is Monge optimal transport map if and only if C(s) is semi-positive definite. Note that the above requirement needs to be satisfied for every PF-ODE chain 2,, t [].
B. Experiments Details
B.1. Evaluation of NLL
For toy experiments, we follow the experiment design in Lu al. (2022a) and adopt the same network architecture for the score network and our DF-TM network. We calculate the true diffusion Fisher with 5,000 data samples from the cheeseboard distribution.
For commercial-levee experiments, we employ the explicit Euler method to compute the NLL, excluding the final step near == 0, for reasons discussed in Appendix C.1 We evaluate the NI.L across 10 steps throughout the timeline of the PF-ODE. The DF-TM network we trained uses the 1oat-pointl6 data type.
Network architectures In terms of network architecture, we employ SD Unet structure with an additional MLP head. However, we believe that lighter network could potentially be sufficient for DF-TM.
Training cost of DF-TM For training the DF-TM network, approximately spend 24 hours using 8 Tesla V V100 chips. Given the size of the Laion2B-en dataset (which contains 2.32 billion images), this is quite efficient speed. Additionally, the convergence behavior of the training loss is robust, as illustrated in Figure Ia We also hypothesize that our network design has redundancy, suggesting that we could further reduce costs by opting for lighter networks. We will provide more details about training costs in our revised manuscript.
B.2. Adjoint Guidance Sampling
For Figure 3 we examined varying numbers of adjoint guidance, ranging from 0 to 20, under full inference number of 50. The adjoint guidance scale was grid-searched by the VJP method. For experiments on Pick-Score, we use NVIDIA V100 chips, and the rest experiments use Tesla V100 chips.
Base method for DF-EA Several variants of iijiiit-optimizztiion algorithms exist, such as AdjoinDDPM (Pan et al. 2023a), AdjointDES (Blasingame & Liu 2024), and SAG (Pan et al.) (023b). However, while these algorithms differ in their design of solvers for the adjoint ODE, they all utilize VJP when accessing the diffusion Fisher. We selected SAG as our base method due to state-of-the-art performance. We believe that replacing VJP with DF-FA could also enhance the performance of algorithms like AdjointDPM and AdjointDES.
The Design of Score functions
Aesthetic Score (SAC/AVA/Pick-Scong
Definition in http://ptw//..aaaaa//o/////////r
suffice result here.
21

---

Efficiently Access Diffusion Fisher: Within the Outer Product Span Space
For the aesthetic score predictor faes Rd IR, the adjoint optimization target is simply Saes itself.
((xo)= = facs((20)
(54)
Clip Loss
Following the implementation of (Pan et al. 2023a), we use the features from the CLIP image encoder as our feature vector. The loss function is Lg-norm between the Gram matrix of the style image and the Gram matrix of the estimated clean image.
((x)) = ||lip((x)elip(aa)) T
(eee)) "I
(55)
FaceID Loss
Following the implementation of (Pan et al. 2023b), use ArcFace to extract the target features of reference faces to represent face IDs and compute the l2 Euclidean distance between the extracted ID features of the estimated clean image and the reference face image as the loss function.
((x)) ||ArcFace(zo) ArcFace((rre))
(56)
Hyperparameters For the hyperparameters adjoint-guided sampling, we ensure fair comparison between the VJP and DF-EA methods. For most hyperparameters, we directly adopt the settings from previous works (Pan al. 2023b) for both the baseline VJP method and our DF-EA method. For the guidance scale, tune the value for the VJP method and use the same value for our DF-EA method. The DF-EA method does not introduce additional hyperparameters, and for mutual hyperparameters, our DF-EA method uses the exact same values as the VJP method. We adopt this strategy because our DF-EA method solely improves the approximation of the diffusion Fisher linear operator, without altering the adjoint sampling mechanism. Therefore, the suitable hyperparameters should remain unchanged, and we simply use the same parameters from the VJP method for our DF-EA method. For all experiments, we set the number of sampling steps to T' 50. Adjoint guidance is applied starting from steps ranging from 15 to 35 and ending at step 35, with one guidance per step. Thus the only parameter tune is the guidance strength. We determine this value for the VJP method via grid search from 0.1 to 0.5 with step size of 0.1 and find that the optimal guidance strength for VJP is 0.2. We then use this value for our DF-EA method. Notice that, we apply a normalization to the guidance gradient for both VJP and DF-EA methods, making our optimal guidance strength consistent across different scores. The tuning is conducted on 1k COCO prompts, and the computational budget for tuning is * 5 # GPU hours (4 tasks 5 grids 3 hours per single test) on Tesla V100 chips.
B.3. Numerical OT Experiments
The detailed algorithm The detailed version of the Algorithm 2 is presented in Algorithm 3
We use all the data points in the initial dataset to calculate every quantum we need in the sampling trajectory to validate the condition discussed in Corollary
Initial data In Table 3 we adopt numerical verification of OT on 2-D synthesized data. For the single-Gaussian case, we fix the one mode central in (0.5,0.5) and test at s = 0.1; For the affine data case, we set three data points as (0.2,-0.4), (0.2,0.0), and (0.2,0.9), and test at 0.0; For the non-affine data case, we set three data points as (0.0.0.5), (0.0,0.0), and (0.5,0.0), and test at 0.0.
The definition of Asymmetric rate In Table 3 we need the asymmetry rate to point out that the PF-ODE map in very simple non-affine data cannot be OT. Let A be an n X n matrix. The asymmetry rate I is defined as:
| ATI
V2||A||F
where ||
En Σn= mi, is the Frobenius norm of the matrix M. The denominator /2|[A|] is used to normalize the index to the range of [o, 1). The asymmetry rate of the non-affine data in Table 3 has significantly deviated from zero
22

---

Efficiently Access Diffusion Fisher: Within the Outer Product Span Space
Algorithm 3 Detailed numerical OT test for PE-ODE map
1: Input: initial data {y; 1==1- noise schedule (o(()) and (o()), discretization steps M.
2: Initialize AM = 2M ~ [0, orl).
for .M, M 1, ... do
4: a₁ a(4;), 01 (14)
5: dt - t4
6: fi dlog
7: = do? 2 d loga, o,? dt
8: for j 1.... do
9: calculate Uy via Ui exp
20
10: end for
11: for j 1, N do
12: calculate wy via wij = En Vk
13:
end for
o
En wyyyy} - -(E, wyys)
(E m,y,) }
14:
B.
fl
20
15:
A
Ai
dl. A Bi
(solve fundamental matrix)
-
E,
16:
17: 0:-1
+ [fitz
2 df
18:
end for
19: Output:Ao.
B.4. Pretrained Models
All of the pretrained models used in research are open-sourced and available online as follows:
stable-difiusionn-1155
stable-diffusion-2-baee
SAC-aesthetic scone predictor
AVA-aesthetic score predictor predcccoo
are32llooo-111-LnnaaMMh.hhth
ArcFace ID loss
Clip loss
Pick-Score
23

---

Efficiently Access Diffusion Fisher: Within the Outer Product Span Space
C. Discussions
C.1. Singularity of Fisher information at
Previous studies (Yang et al. 2023; Zhang et al. 2024b) have shown that the diffusion model, particularly when learned in c-prediction, can encounter a singularity issue at = 0. Our DF in equation TT reaffirms this issue, as this formulation becomes ill-formed
0 due to division by zero (ar)) Consequently, our formulation does not describe the behavior == 0. The deep theoretical exploration of the singularity problem remains an open question in the diffusion model field. However, as it is not the primary focus of this paper, we will not discuss the DF at t = 0.
C.2. Statistical Caliber of Negative Log-Likelihood
When dealing with high-dimensional data such as images, direct likelihood comparisons may encounter scaling issues due to the dimensionality. In this study, unless explicitly indicated otherwise, we adopt the approach of (Zheng et al., (2023) and typically use Negative Log- nigeliied (NLL) to refer to Bits Per Dimension (BPD).
log Po (2:)
BPD = Eggrggo
(57)
d log 2
C.3. The ODE Solvers
To compute the numerical solutions for the PF-ODE in equation 5 the likelihood ODE in equation 14 and the adjoint ODE in equation 18, we require ODE solvers. In our paper's experiments, we consistently use the explicit Euler method (referred to as DDIM when applied to PF-ODE). However, it's important to note that our approach is not dependent on a specific
ODE solver. We can also utilize alternatives like fast ODE solvers (Lu et al. 2022b Liu et al. 2022) or exact inversion ODE solvers (Wallace et al. 2023 Zhang et al. 2023).
C.4. The relation of our outer-product form DF Covariance in DMs
We note that there is a series of studies aiming learn the covariance of the reverse diffusion Stochastic Differential Equation (SDE) (Bao al., [2022b a). In Bayesian statisties, Fisher information defined as the covariance of the score. These studies derive their formulation by analyzing the covariance of the score and obtaining the Fisher information in terms of the score. However, our DF is derived directly from the marginal distribution and is composed solely of the initial distribution and noise schedule. Furthermore, our application is unique; we are the first to replace the use of VJP with DF, while the focus of these studies to enhance the performance of Diffusion Models (DMs) with analytical covariance.
very similar form of the Fisher information proposed in Lemma 5 of (Benton et al. 2024). The difference that our Proposition 3 presents the specific form of Fisher information in terms of data distribution, which is not included in Lemma 5 of (Benton al., 2 This distinction is crucial it facilitates the derivation of our new algorithms. Also, we adopt the currently more commonly used a; o, notation to represent the noise schedule. Instead, (Benton et al. 2024) or - We gave out this detailed formulation to avoid unnecessary misunderstandings in the development of the subsequent training scheme.
C.5. Comparison of DF-TM to Hutchinson trace estimator
We notice that the naive trace calculation of the VJP method can be accelerated by the Hutchinson trace estimator. We have not conducted experiments using the Hutchinson estimator, as it is a Monte-Carlo type estimation, unlike the direct evaluation methods like the full-VJP baseline and our DF-TM. Furthermore, the VJP method, with the help of the Hutchinson method and its variants, is still considerably more expensive than our method, and its applicability is also more restricted.
The Hutchinson method is much more costly:
To attain relative error less than with probability of 1 - 6, the Hutchinson method requires 2(1- 2(1-ge) log samples (Skorski) 2021). Assuming that the goal is to obtain an estimation with relative error of less than 10% with a probability exceeding 90%, a minimum of 351 NFEs is required. This equates to I or 2 minutes on SD-v1.5 for a single trace estimation. For a complete NLL estimation of an image with 20 steps, this would take 20 minutes, which entirely impractical in any business context. In contrast, our DF-TM only requires NFE for a single trace estimation, needing merely 10 seconds for the full NIL. estimation of an image.
2²
24

---

Efficiently Access Diffusion Fisher: Within the Outer Product Span Space
The application of the Hutchinson method is more restricted:
Due to its Monte-Carlo characteristics, the Hutchinson method more appropriate for contributing to the computation of certain training objectives, as in (Lu al., 2022a), where the unbiased property sufficient, and large variance may be absorbed into network training. However, the Hutchinson method may encounter difficulties with accurate per-sample trace estimation. Our method can accommodate both scenarios, including per-sample computation.
There are already attempts to use the Hutchinson method to expedite the VJP of trace estimation in diffusion models (Lu et al. 2022a). Nonetheless, due to Hutchinson's 's limitations, these practices are restricted to relatively small DMs (CIFAR-I0 most). We will add discussions on the Hutchinson method our revision.
C.6. Discussions on the theoretical bound of DF-EA
We notice that the error bound in Proposition does not vanish as the training error decreases. From the rigorous theoretical perspective, considering the error bounds in Propositions 7 and [ the DF-EA method is less valid than the DF-TM method, as we currently cannot establish vanishing bounds for it. However, from an empirical perspective, our DF-EA outperforms the naive VJP method in terms of score improvement across various tasks and pretrained models, demonstrated in Figure 4. Thus, the DF-EA approximation proves to be valid in practical sense. The replacement E: y:y, ss xo0g originates from the observation that W/ is weighting of the summation equal to 1, and as approaches 0, the closest to zu will dominate due to the diffusion kernel. Therefore, this approximation is intuitively reasonable near = O, which is precisely where we apply adjoint guidance.
C.7. Limitations
This paper does not explore the integration of DF into accelerated ODE solvers (Wang et al. 2025 Lu et al. 2022b), or
exact inversion ODE samplers (Wang et al. 2024a). This paper is constrained in the scope of DMs, but similar second-orde -oddr
information may also exist in flow-matching generative models (Lipman et al. 2022 Zhu et al. 2024a) or variational inference models (Zhu et al., 2024b; Wang et al. 2024b). Our technique may also have downstream applications in automated driving (Tu et al. 2025b), muuch-geneationn (Tu et al. 2025a), draasf-r (Feng et al. 2024b a), language
generation (Fu et al. 2025), and missing data imputation (Chen al. 2024). The DF may also contribute to a more effective
inference method, like L-BFGS, which we did not explore. The complex second-order information in auto-regressive models
can also be analytically explored (Cheng et al. 2025).
25


Reviews:
[{"review_id": "A6ve8QUc2F", "reviewer": "Reviewer_V5LL", "summary": "This paper addresses the challenge of efficiently accessing the diffusion Fisher information (DF) in diffusion models (DMs). Based on the analytical formulation of the diffusion Fisher, the authors propose two novel algorithms: DF Trace Matching (DF-TM) for efficiently estimating the trace of the DF, and DF Endpoint Approximation (DF-EA) for efficient matrix-vector multiplication with the DF. DF-TM trains a neural network to estimate the diffusion Fisher while DF-EA approximates the drift term through the learned score function. The paper provides theoretical error bounds for these approximations and showcases their superior accuracy and reduced computational cost in experiments on likelihood evaluation and adjoint optimization. Furthermore, leveraging their outer-product formulation, the authors show that their approach can be used to numerically verify the OT property of the diffusion-ODE derived map under various conditions.\n\n## Update after rebuttal\n- Most concerns have been clarified during the rebuttal. \n- As an empirical likelihood estimation method, the proposed method showed improved accuracy based on low-dimensional experimental results. However, the practical significance of the method, based on the presented empirical evidence (Fig 2-4), remains somewhat unclear to me.\n- The error bounds appear vacuous due to singularity. However, I admit that this issue originates from a fundamental limitation inherent to diffusion models—specifically, the singularity associated with the denoising score matching objective—rather than a flaw unique to the proposed method.\n- Overall, I view this paper as borderline. However, considering its potential to inspire future research in the community, I would be supportive of acceptance if the other reviewers believe it's worth publishing.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback! We will answer your weaknesses/questions one by one.\n\n> **Weaknesses 1:** The bound in Proposition 5 is vacuous when t is small...\n\nIt is true that as $t$ approaches $0$, the bound in Proposition 5 will be ill-defined and blow up due to division-by-zero. This concern is also raised by Reviewer a1EQ. We argue that this problem, which has been referred to as the **singularity issue** in previous literature [1][2], stems from the inherent characteristics of diffusion models rather than flaws in our derivations. A deep exploration of the singularity problem remains an open question, which we will investigate in our future works.\n\nPractically, we adopt the approach in [3], where the diffusion sampling path ended at $t_{\\min} = 0.005$ instead of $t_{\\min} = 0$. This circumvents potential blow-up effects of singularity issues and results in a well-defined bounded NLL.\n\nWe briefly discussed this singularity issue on line 1242 of the original paper. In our revision, we will provide a more detailed discussion and clarification of our handling.\n\n[1] Zhang, P, et al. Tackling the singularities at the endpoints of diffusion models. CVPR 2024.\n\n[2] Yang, Z, et al. \"Lipschitz singularities in diffusion models.\" ICLR 2024.\n\n[3] Song, Y, et al. Consistency Models ICML 2023.\n\n\n> **Weaknesses 2:** In the Dirac setting, the likelihood evaluation becomes trivial...\n\nWe derive the theory within the Dirac setting to simplify the formulations and make them easier to follow. We also present the derivations under the general setting in the appendix. Remarkably, both the Dirac and general settings yield the same forms for the DF-TM and DF-EA methods. \n\nAll derivations within the Dirac setting can be straightforwardly adapted to the general setting by replacing discrete summations with general integrals. The underlying ideas and mechanisms are identical.\nThe Dirac case can be regarded as a collection of finite Monte-Carlo samples drawn from a general distribution. In the mean-field limit, the Dirac case converges to the general case.\n\nTo prevent misunderstandings, we will revise all propositions in the main body of the paper to the general setting.\n\n> **Weaknesses 3:** In adjoint task, the visual quality...\n\nIn Fig. 4, the DF-EA method contributes to (left) a higher Laion-Aes score, as evidenced by a smoother visual effect, and (right) a higher pick-score, reflected in enhanced colorfulness and more detailed features. Since visual effects can be subjective, we also present a quantitative demonstration of the score improvement brought about by DF-EA in Fig. 3.\n\n\n> **Suggestions:** Line 145...Line 153...Eq. (7)...\n\nWe apologize for the oversights. We have now corrected the typographical errors and adopted standard Hessian notation for Eq. 7:\n $\\left(F_t(x_t,t)\\right)_{i,j} := - \\frac{\\partial^2 \\log q_t(x_t, t)}{\\partial x_t^{(i)} \\partial x_t^{(j)}}$, where $x^{(i)}$ denotes the i-th element of vector $x$.\n\n> **Question 1:** ...Can the authors extend Proposition 4,5,6 for this setting?\n\nYes, our Proposition 3,4,5,6 can be extended to the general setting and result in **the same DF-TM and DF-EA method**. Suppose we have $m$ samples from a general initial distribution $q_0$.\n\n- **Proposition 3 (general version)**: *The trace of the diffusion Fisher for the diffused distribution $q_t$, where $t\\in(0,1]$, is given by: \n$\n\\mathrm{tr}\\left(F_t(x_t,t)\\right) = \\frac{d}{\\sigma_t^2} - \\frac{\\alpha_t^2}{\\sigma_t^4}\\left[ \\int w(y) ||y||^2dq_0(y) - \\left\\|\\int w(y) y dq_0(y)\\right\\|^2 \\right] \n$*\n\n- **Proposition 4 (general version)**: *$\\forall (x_t, t) \\in \\mathbb{R}^d \\times \\mathbb{R}^{+}$, the optimal $t_\\theta(x_t, t)$ trained by Algo. 1 are equal to $\\frac{1}{d}\\int w(y) ||{y}||^2 dq_0(y)$.*\n\n- **Proposition 5 (general version)**: *The approximation error of DF-TM is at most $\\frac{\\alpha_t^2}{\\sigma_t^4}\\delta_1 +\\frac{1}{\\sigma_t^2}\\delta_2^2$.*\n\n- **Proposition 6 (general version)**: *The approximation error of the DF-EA linear operator, is at most $\\frac{\\alpha_t^2}{\\sigma_t^3}(2\\mathcal{D}_y^2+ \\sqrt{d}\\delta_2)$*\n\nNote that, in Proposition 5 & 6, the approximation error induced from the finite number of $m$ samples is absorbed into $\\delta_1$ and $\\delta_2$.\nWe'll include the general-version propositions and their proofs in the revision.\n\n> **Question 2:** In the toy problem, how do you compute the ground truth NLL from the dataset?\n\nIn the toy experiment, we adopt the method in [1] and approximate the true NLL following Eq. 13. We use Euler discretization over 1000 timesteps for simulation and 5000 data samples per timestep to approximate the right-hand side term of Eq. 13. This setup aims to ensure NLL approximation accuracy.\n\nWe apologize that the description in line ~1096 is rather simplistic. In the revision, we will provide a more comprehensive description.\n\n[1] Lu, Cheng, et al. Maximum likelihood training for score-based diffusion odes by high order denoising score matching. ICML 2022.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes, it makes sense.", "theoretical_claims": "Yes, I checked the proofs for Proposition 1 & 2.", "experimental_designs_or_analyses": "- In the toy problem reported in Table 2, it is unclear how the ground truth NLL is obtained.", "relation_to_broader_scientific_literature": "This paper proposes a novel approach to estimate the diffusion Fisher.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "**Strength**\n\n- The proposed DF-TM is efficient and effective in terms of estimating the likelihood of an observed sample.\n- The proposed method enables OT verification of diffusion models, which is an interesting application.\n\n**Weaknesses**\n\n- The bound in Proposition 5 is vacuous especially when t is small. NLL evaluation requires integration over [0, T] which means the overall approximation error bound of NLL will blow up.\n- In the Dirac setting that the authors focus on, the direct likelihood evaluation becomes almost trivial. The likelihood of an observed $x$ is zero if $x$ is not one of the $y_i$;  is 1 / N i f $x$ is one of the $y_i$. Therefore, there is a gap from the typical challenges in generative modeling where the data distribution is general and we have only finite data samples. The paper should focus on the actual setting in the generative modeling.\n- In the adjoint improvement task, the visual quality gain from proposed DF-EA adjoint is negligible.", "other_comments_or_suggestions": "- Line 145: “outer-product sums We first” needs a period → “outer-product sums. We first”.\n- Line 153: “in (Lu et al., 2022a)”. Please use inline citation when it is part of the sentence.\n- Eq. (7) uses the second derivative to represent Hessian which is confusing. Please check out the standard the notation for [Hessian matrix](https://en.wikipedia.org/wiki/Hessian_matrix).", "questions_for_authors": "- The valuable setting in generative modeling is where we have finite samples from the general data distribution. Can the authors extend Proposition 4,5,6 for this setting?\n- In the toy problem, how do you compute the ground truth NLL from the dataset?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ngwSR7kzdS", "reviewer": "Reviewer_mnN8", "summary": "This paper introduces a novel formulation of the diffusion Fisher (DF) information in diffusion models by expressing it as a weighted sum of outer products of the score function and initial data, thereby revealing that DF lies in a space spanned by specific outer-product bases dependent solely on the initial distribution and noise schedule. This formulation enables the development of two efficient approximation algorithms—one for computing the trace of DF and another for its matrix-vector multiplication—with rigorously established error bounds, significantly reducing computational costs compared to traditional auto-differentiation methods. Additionally, the paper derives a corollary for the optimal transport property of the diffusion-ODE deduced map and validates this property through numerical experiments on various noise schedules, demonstrating enhanced accuracy in tasks like likelihood evaluation and adjoint optimization.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback! We will answer your concerns/questions one by one.\n\n> **Concern 1:** I have a problem with the paper's main claim that JVP cannot be calculated effectively, they claim that the time complexity is $O(d^2)$...\n\n\nPlease allow me to humbly clarify that we didn't claim a single JVP's time complexity to be $O(d^2)$. Instead, we stated that calculating the diffusion Fisher trace using JVPs has an $O(d^2)$ complexity. \nThe reason is that to obtain each diagonal element of the diffusion Fisher matrix, one JVP operation is needed. As computing the full trace demands all $d$ diagonal elements and each JVP has $O(d)$ complexity, the total complexity for $d$ JVPs is $O(d^2)$.\n\nIn the context of accessing the diffusion Fisher trace, previous methods rely on $d$ JVP operations. In contrast, our DF-TM method only requires one forward pass through the learned trace network to access the diffusion Fisher trace, which is notably more efficient than the traditional multiple-JVP approach. \n\nIn the adjoint scenario, previous methods use a JVP to compute the adjoint term. Our DF-EA approximation method eliminates this JVP, making it marginally more efficient than the traditional method.\n\nFurthermore, accessing the diffusion Fisher via JVP is a black-box approach without any accuracy guarantee. Conversely, our DF-TM and DF-EA methods exploit the analytical structure of the diffusion Fisher and offer theoretical guarantees.\n\n\n> **Concern 2:**  how about approximating it with numerical methods?...\n\nWhile numerical methods for JVP and trace evaluation are inaccurate and inefficient in high-dimensional scenarios, as discussed in line ~1281.\nWe appreciate your suggestion and agree that incorporating these as baselines enhances the clarity of our experiments.\nSo, we've added two numerical methods, Finite Difference and Hutchinson's Trace Estimation, as baselines in the toy NLL evaluation experiment:\n\n**Table 2: The relative error of NLL evaluation.**\n|Methods|t = 1.0|t = 0.8|t = 0.6|t = 0.4|t = 0.2|t = $t_{\\min}$|\n|-|-|-|-|-|-|-|\n|Finite Difference|20.54%|44.57%|60.68%|79.53%|83.08%|94.87%|\n|Hutchinson|11.28%|10.01%|16.12%|23.79%|53.02%|71.85%|\n|JVP|6.68%|5.79%|10.46%|20.13%|51.14%|70.95%|\n|DF-TM ​(**Ours**​)|​**3.41%​**|​**4.56%​**|**4.13%​**|​**4.28%​**|​**5.33%​**|**5.81%​**|\n\nIt is shown that our DF-TM outperforms these numerical methods.\n\n> **Concern 3:** Training a separate nontrivial network for the use cases listed in the paper feels too heavy...\n\n- In adjoint applications, our DF-EA is training-free.\n- In NLL evaluation, our DF-TM incorporates an additional trace matching network, only bringing moderate costs:\n    - The trace network is scalar-valued and demonstrates a much faster convergence compared to the main network, as shown in Fig. 1. In Laion-2B dataset, the main network of SD-1.5 needs 100,000 GPU hours, while the trace network only 3000, just **3%** of the main network's training cost.\n    - The trace network shares the same input as the diffusion network, allowing it to use a shared backbone of $\\epsilon_\\theta$ for feature extraction, and only necessitates the training of an output head. We plan to explore and implement this promising approach in our future research.\n    \n    Thus, DF-TM can leverage off-the-shelf pretrained diffusion models and only needs to train a small trace network as a \"plug\".\n\n> **Question 1:**  Is it true that the prior methods all use the highly inefficient way of calculating the JVP...\n\nIn prior adjoint methods [1][2], a JVP operation is required at each step. Usually, these methods use the official PyTorch auto-differentiation tools to compute the JVP.\nOur DF-EA approximation method circumvents the need for this JVP calculation, though the efficiency gain is limited.\n\nThe main advantage of DF-EA in adjoint optimization lies in its accuracy, leading to better scores and visual effects, as demonstrated by the qualitative comparison in Fig. 4 and the quantitative comparison in Fig. 3.\n\nIn the revision, we'll add more details on prior adjoint methods and their applications.\n\n[1] Pan, Jiachun, et al. \"Towards accurate guided diffusion sampling through symplectic adjoint method.\" arXiv preprint arXiv:2312.12030 (2023).\n\n[2] Blasingame, Zander W., and Chen Liu. \"AdjointDEIS: Efficient gradients for diffusion models.\" NeurIPS 2024.", "claims_and_evidence": "I have a problem with the paper's main claim that JVP cannot be calculated effectively, which motivates the authors to train an additional network. More specifically, they claim that the time complexity is $O(d^2)$ and infeasible. However, both Pytorch and JAX provide forward-mode autodiff, which can be used to calculate JVP very efficiently (e.g. Pytorch with torch.func). Loosely, it requires only 2 forward passes, which is faster than a single optimization step (1 forward, 1 backward at least) on the same network. If the model evaluation is required anyway, then the overhead is just 1 forward pass. Thus, training a network to calculate the JVP seems unfavorable and unscalable. Even without forward-mode autodiff, how about approximating it with numerical methods? They may not be accurate, but I think it is a reasonable baseline that needs presenting.", "methods_and_evaluation_criteria": "They make sense to me.", "theoretical_claims": "I checked the theoretical claims, though not their proofs in detail. I do not find any problems.", "experimental_designs_or_analyses": "I do not have problems with them.", "relation_to_broader_scientific_literature": "Authors proposed a new way to calculate the trace of diffusion Fisher.", "essential_references_not_discussed": "Not that I know of.", "other_strengths_and_weaknesses": "Even without considering the problems listed in Claims And Evidence. Training a separate nontrivial network for the use cases listed in the paper feels too heavy. I suggest the authors justify the additional cost better with more practical and popular use cases.", "other_comments_or_suggestions": "Given my concerns stated before, I cannot recommend acceptance of this paper. However, there are claims that I enjoy learning, e.g. Prop 1, and the paper is well-written and easy to follow.", "questions_for_authors": "1. The adjoint optimization is an interesting topic, which I do not know very much about. Is it true that the prior methods all use the highly inefficient way of calculating the JVP? I wish the authors could go into more detail here, as I think this setting/use case has the most potential in practice.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FlozTQducO", "reviewer": "Reviewer_a1EQ", "summary": "The diffusion Fisher information matrix (or just 'diffusion Fisher') provides useful quantitative information about the sensitivity of diffusion model log-likelihoods to small changes in state, and can be exploited to (among other things) evaluate sample likelihoods and guide diffusion models to generate higher-quality samples.\n\nThe authors observe that a special property of the diffusion Fisher allows it to be computed more efficiently than is possible with the naive auto-differentiation-based approach. Moreover, they show that their approach is much faster than an existing alternative approach for computing (the trace of) the diffusion Fisher, which involves taking the Jacobian-vector product (JVP).", "questions": "", "limitations": "", "rating": 5, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your appreciation of our work. We will answer your questions one by one regarding these suggestions/questions.\n\n> **Answer to Suggestion 1:**  The authors' evaluations are easy to understand and clear, although it would be helpful if more examples (along the lines of Figs 2 and 4) were shown in the SI.\n\nWhile we consider the theoretical analysis of the diffusion Fisher to be the primary contribution of this work, we agree that additional visual illustrations, such as those in Figures 2 and 4, would further enhance clarity. We will include more examples in the Supplementary Information (SI) as part of our revision.\n\n\n> **Answer to Suggestion 2:** line ~ 208, \"detail discussions\" -> \"a detailed discussion\"\n\nWe apologize for this oversight. We have now corrected the syntax error and conducted a thorough proofreading of our work.\n\n\n> **Answer to Question 1:** The authors' proposed methods involve some approximation (see, e.g., line 375). Are there cases where these approximations are expected to break down, and this method is not expected to give good results? \n\nIn the DF-EA method, we introduce an approximation technique for the adjoint term (around line 375). These approximations are likely to be inaccurate in the following scenarios:\n- When $t$ is very large, the $x$-prediction part in the approximation is less accurate because of the large noise scale.\n- When $t$ is very small, the approximation error bound stated in Proposition 6 will increase due to the singularity issue, and the approximation will lack an accuracy guarantee. We will discuss on this singularity issue in detail in Question 3\n\nIn practice, we apply DF-EA in the range from $t=0.7$ to $t=0.3$. This range also aligns with the recommended adjoint region of the original adjoint methods.\n\n> **Answer to Question 2:** In general, are there any obvious issues associated with using or scaling up these methods for computing the diffusion Fisher (or certain related quantities, like its trace)?\n\n- Obvious issue: Our methods are derived exclusively on the classical diffusion process and thus cannot be directly extended to settings where the diffusion process is modified, such as cases involving additional consistency-model-type distillation [1], Schrödinger-Bridge-type endpoint alignment [2], non-Gaussian transition kernels [3], or flow rectification [4].\n\n- Scaling-up:\nTheoretically, both the training and sampling phases of our ​DF-TM and ​DF-EA exhibit ​linear complexity growth with respect to dataset scale and data dimensionality.\nExperimentally, our approach performs well on commercial-grade diffusion models such as SD2-base and large-scale datasets like Laion-2B. Therefore, our method indeed has scalability.\n\n[1] Frans, Kevin, et al. \"One step diffusion via shortcut models.\" ICLR 2025.\n\n[2] De Bortoli, Valentin, et al. \"Diffusion schrödinger bridge with applications to score-based generative modeling.\" NeurIPS 2022.\n\n[3] Yoon, Eun Bi, et al. \"Score-based generative models with Lévy processes.\" NeurIPS 2023.\n\n[4] Liu, Xingchao, et al. Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow. ICLR 2022.\n\n\n> **Answer to Question 3:** As a related point, Proposition 6 appears to show potentially bad behavior when $\\sigma_t \\approx 0$.\n\nNote that Proposition 6 will be ill-defined due to division-by-zero when $t=0$ since $\\sigma_0 = 0$. This concern is also raised by Reviewer V5LL. We argue that this problem, which has been referred to as the **singularity issue** in previous literature [1][2], stems from the inherent characteristics of diffusion models rather than flaws in our derivations.\n\nTheoretically, the diffusion model has an inherent singularity property when $t_{\\min} \\to 0$. This implies that many quantities associated with diffusion models become ill-defined in the vicinity of $t_{\\min} = 0$. This phenomenon, previously explored in [1] and [2], is once again recognized in the context of diffusion Fisher through our analysis. A deep theoretical exploration of the singularity problem remains an open question within the diffusion model field.\n\nPractically, we implement the approach used in [3] and [4], where the diffusion sampling path is ended at $t_{\\min} = 0.005$ instead of $t_{\\min} = 0$. This method circumvents potential bad effects resulting from the singularity issue.\n\nWe briefly discussed this singularity issue on line 1242 of the original paper. In our revision, we will provide a more detailed discussion and clarification of our handling of this issue.\n\n[1] Zhang, Pengze, et al. \"Tackling the singularities at the endpoints of time intervals in diffusion models.\" CVPR 2024.\n\n[2] Yang, Zhantao, et al. \"Lipschitz singularities in diffusion models.\" ICLR 2024.\n\n[3] Song Y, Dhariwal P, Chen M, et al. Consistency Models ICML 2023.\n\n[4] Lu, Cheng, et al. \"Maximum likelihood training for score-based diffusion odes by high order denoising score matching.\" ICML 2022.", "claims_and_evidence": "Yes. The authors' work is based on a simple theoretical idea, which they clearly prove. They support their claim of practical usefulness with a variety of numerical experiments, which involve both evaluating likelihoods and guiding the output of diffusion models to be higher-quality.", "methods_and_evaluation_criteria": "Yes, the authors' methods and evaluations (principally, using the diffusion Fisher for what it's usually used for, and showing that their computations are fast compared to the previous SOTA) make sense.", "theoretical_claims": "The authors' theoretical claims are simple and easy to verify.", "experimental_designs_or_analyses": "The authors' experiments appear sound and yield reasonable-looking results.", "relation_to_broader_scientific_literature": "The authors' work relates to the diffusion Fisher, which has various well-known applications to evaluating likelihoods and guiding diffusion models. More broadly, it relates to a large body of work concerning evaluating the quality of the samples of diffusion models, and guiding them to produce better samples.", "essential_references_not_discussed": "No references come to mind.", "other_strengths_and_weaknesses": "The paper is well-written and the main theoretical idea is clear. The authors' evaluations are easy to understand and clear, although it would be helpful if more examples (along the lines of Figs 2 and 4) were shown in the SI.", "other_comments_or_suggestions": "line ~ 208, \"detail discussions\" -> \"a detailed discussion\"", "questions_for_authors": "The authors' proposed methods involve some approximation (see, e.g., line 375). Are there cases where these approximations are expected to break down, and this method is not expected to give good results? In general, are there any obvious issues associated with using or scaling up these methods for computing the diffusion Fisher (or certain related quantities, like its trace)? As a related point, Proposition 6 appears to show potentially bad behavior when $\\sigma_t \\approx 0$.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper addresses the challenge of efficiently accessing the diffusion Fisher information (DF) in diffusion models (DMs). 
The authors derive an outer-product span formulation of the diffusion Fisher and introduce two approximation algorithms for different types of access to the diffusion Fisher: diffusion Fisher trace matching (DF-TM) and diffusion Fisher endpoint approximation (DF-EA).
Theoretical error bounds are provided, along with numerical experiments. Furthermore, using their outer-product formulation, the authors show that their approach can be used to numerically verify the optimal transport (OT) property of the diffusion-ODE map under various conditions.

Reviewers generally appreciate that the proposed approach is novel and interesting. Reviewers V5LL and a1EQ both share concerns regarding the singularity issue when t=0, in which case the theoretical bounds in Propositions 5 and 6 would be vacuous. In their rebuttal, the authors pointed out that this issue originates from a fundamental limitation inherent to diffusion models and not their proposed approach; Reviewer V5LL acknowledged this and Reviewer  a1EQ increased their score.

Overall, I believe that this would be an interesting contribution and recommend acceptance.
Decision: Accept (poster)