Gradient-Free Analytical Fisher Information of Diffused Distributions

Authors: Fangyikang Wang, Hubery Yin, Huminhao Zhu, Shaobin Zhuang, Chao Zhang, Hanbin Zhao, Hui Qian, Chen Li
Primary area: generative models
Keywords: diffusion models; Fisher Information; analytical formulation

Abstract:
Diffusion models (DMs) have demonstrated powerful distributional modeling capabilities by matching the first-order score of diffused distributions.
Recent advancements have explored incorporating the second-order Fisher information, defined as the negative Hessian of log-density, into various downstream tasks and theoretical analysis of DMs.
However, current practices often overlook the inherent structure of diffused distributions, accessing Fisher information via applying auto-differentiation to the learned score network. 
This approach, while straightforward, leaves theoretical properties unexplored and is time-consuming. 
In this paper, we derive the analytical formulation of Fisher information (AFI) by applying consecutive differentials to the diffused distributions.
As a result, AFI takes a gradient-free form of a weighted sum (or integral) of outer-products of the score and initial data.
Based on this formulation, we propose two algorithmic variants of AFI for distinct scenarios.
When evaluating the AFI’s trace, we introduce a parameterized network to learn the trace.
When AFI is applied as a linear operator, we present a training-free method that simplifies it into several inner-product calculations.
Furthermore, we provide theoretical guarantees for both algorithms regarding convergence analysis and approximation error bounds.
Additionally, we leverage AFI to establish the first general theorem for the optimal transport property of the diffusion ODE deduced map.
Experiments in likelihood evaluation and adjoint optimization demonstrate the superior accuracy and reduced time-cost of the proposed algorithms.

Decision: reject
Reviewer scores: 3, 3, 6, 6

Reviews and discussion:
<thread>
creation date:
1729663775889

thread_type:
official_review

<thread_object>
creator: reviewer_ihvf

<content>
summary:
A method for efficiently evaluating Fisher information is developed for diffusion models. The method is based on its analytic formula, which is derived by consecutive differentials to the diffused distributions. Due to the Gaussian nature of the diffusion process, the formula is expressed as a weighted sum of outer-products of the score and initial data, which is free from computing gradients. Based on the formula, an efficient algorithm for assessing trace of Fisher information matrix is developed using neural network learning. Another efficient algorithm is also proposed for using Fisher information as a linear operator for the adjoint optimization of  diffusion models. Several theoretical guarantees are provided for the algorithms. The analytical knowledge on Fisher information is also utilized for deriving a theorem that allows mapping deduced by the probability flow ordinary differential equation to possess the optimal transportation property.

strengths:
Computing Fisher information matrix for high dimensional distributions is computationally very hard. This paper develops efficient methods to compute its summarized quantities without directly computing the matrix. Not only that, the authors manage to provide a theorem that guarantees the optimal transportation property of the probability flow ODE utilizing the analytical knowledge of the Fisher information.

weaknesses:
Training cost of AFI-TM is not fully described. As for AFI-EA, adjoint guided sampling generally requires hyper-parameters. Since how the parameters were tuned is not written, it is unclear whether the obtained gain in the performance is actually due to AFI-EA or not.

questions:
1. What are the minimum computational requirements for training AFI-TM?
2. What was the total wall-clock training time for AFI-TM?
3. For the adjoint guided sampling experiments, please provide details on:
    i) Specific hyperparameters tuned,
   ii) Hyperparameter ranges explored, 
  iii) Hyperparameter optimization method used, 
  iv) Computational budget for tuning
</content>
</thread_object>

</thread>
<thread>
creation date:
1730415240781

thread_type:
official_review

<thread_object>
creator: reviewer_jsgi

<content>
summary:
In this paper, 
1. the authors derive exact form of the Fisher Information (FI) matrix of the diffused target distribution in diffusion models context. 
2. Based on the formula, they propose an estimation of FI's trace by the trained diffusion model and an additionally trained model;
    - they also analyze its estimation errors,
    - they demonstrate the estimation makes speedup of the NLL computation.
3. Based on the formula, they propose an estimation of the matrix product of adjoint FI and certain vector only by the trained diffusion model;
    - they also analyze its estimation errors,
    - they apply the estimation to adjoint guidance sampling and show its effectiveness.
4. They provide a condition for the solution of a probability-flow ODE to become an optimal transport map.

strengths:
- originality+
    - Focusing on the estimation of Fisher Information sounds novel and interesting at least for me.
- quality+
    - The paper is easy to follow, and the statements are clear, except for the theorem 1.
- significance+
    - Speedup of the NLL estimation would be useful even if it is not exact calculation. In addition, the AFI endpoint approximation seems also to be useful.

weaknesses:
- clarity-
    - There were some unclear points. I will summarize them in the following questions.

questions:
**Typo?**
1. In eq (1), $\alpha_t$ may be better to be replaced by $\alpha(t)$?
2. In eq (4), the symbol $s_\theta$ suddenly appeared without any explanation. I know it is related to $\epsilon_\theta$ and $\bar{y}_\theta$, but it would be better to leave some comments on them.
3. In eq (18), the symbol $h_\theta$ suddenly appeared without any explanation. I have no idea what this symbol stands for.

**Questions**
1. On JVP for trace, is it different from the Hutchinson's trace estimator?
2. On Proposition 7 and 8, the 2nd term and the 1st term of the errors seems to be related to data itself, and seems to be out of control, at least it cannot be reduced only with the improving its training. Does it cause any issue in practice?
3. In eq (20), I could not understand why the replacement $\sum_i w_i y_i y_i^\top \approx x_0 x_0^\top$ is valid.
4. In theorem 4, what distributions does this map define optimal transport between?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730576616384

thread_type:
official_review

<thread_object>
creator: reviewer_r2vx

<content>
summary:
This paper provides an analytical formula for the Fisher information, and uses jacobian-vector product tricks to obtain fast evaluation of likelihood evaluations. They use their formula to make a case for when, in fact, the diffusion ODE map /is/ the optimal transport map between two distributions. They have several experiments that use their approach for computing the negative loglikelihood (NLL) which appears to be correlated with high quality text-prompt/image generation.

strengths:
This paper proposed a very natural solution to an otherwise computationally expensive problem.

weaknesses:
The weaknesses of this paper are largely in the exposition, and in the sparsity of the literature review. 

On the writing, I strongly recommend having a native English speaker make a pass through the article if possible. There are many basic syntax errors throughout the text that ought to be corrected before submission.

The formula for the Hessian of a log-convolution of a density is well-known. This is a property that simply arises because this is a log-partition function, and the covariance appears when one takes two derivatives. 

I think the paper also tries to "do too much" by also trying to mention optimal transport maps --- I'm not sure what the connection is to this and other parts of the paper. I think this is also an interesting contribution to the field, but I'm not confident putting these ideas in the same work is best for clarity purposes.

questions:
N/A
</content>
</thread_object>

</thread>
<thread>
creation date:
1730691344987

thread_type:
official_review

<thread_object>
creator: reviewer_qlo1

<content>
summary:
This paper proposed a way to approximate the Jacobian of the score function in a diffusion model without taking gradients. Based on expressing the Jacobian by a conditional covariance, the paper proposed some methods to estimate the Jacobian, the trace of the Jacobian,  and matrix-vector products with Jacobian. In the end of the paper, a criterion of whether the probability flow is an optimal transport map was also claimed.

strengths:
This paper is clearly written. Core ideas are presented in a very readable manner, and there are no unnecessary complications.

weaknesses:
While the clarity of writing in this paper is greatly appreciated, I regret to say that this paper has some major issues that resist a more positive evaluation.

1. The key result, Proposition 3 in the paper, is well-known. Most of the theory papers on diffusion contained a version of it, e.g., Benton et al., ICLR 2024 had exactly the same result.

2. Theorem 1 is wrong, and the proof in the appendix apparently did not prove it. Instead, it was only proved that positive semi-definiteness of some unknown matrix $C(t)$ implies the flow map is an optimal transport (which is yet incorrect; should be $C(t)C(T)^{-1}$). 

3. The analysis of the computational complexity of estimating trace by backpropagation did not take into account much more efficient trace estimators requiring only logarithmically many backpropagations, cf. Hutchinson estimator and its modern variants. 

4. Overall, the paper lacks technical depth or insight if viewed as a theory paper, due to the reason stated above. On the other hand, it lacks comprehensive experimental validation if viewed as an empirical paper. The proposed algorithm was only evaluated on a single task, and even in this task, it can be seen from Figure 5 that the proposed algorithm has worse detail (though taking less time).

questions:
I have no particular questions.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732092849757

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
To All Reviewers

comment:
We sincerely appreciate the time and effort you have dedicated to reviewing our paper! Your valuable feedbacks have been carefully considered, and we will provide point-to-point responses to your reviews in respective reply. We remain open to any additional feedback you may have. If you feel it is appropriate based on our responses, we would be extremely grateful if you could consider raising our score.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733302483498

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Concluding Response

comment:
Dear Reviewers, AC, SAC, and PC,

We would like to begin by expressing our sincere gratitude for your engagement throughout the rebuttal process, which has significantly enhanced the quality of our paper.

Our work introduces two theoretically guaranteed practical algorithms, AFI-TM and AFI-EA, and a theoretical theorem AFI-OT. These are all based on the analytical formulation of the diffusion Fisher information. These algorithms enable us to perform NLL evaluation and adjoint guidance in a much more computationally efficient manner. These tasks were computationally very hard using naive methods.

In response to the feedback from reviewers, we have made the following revisions to our manuscript:
- **Clarification on proofs:** We have added detailed explanations for our derivations [QLo1].
- **Discussion on Hutchinson's trace estimator:** We have added a comparison and discussion on Hutchinson's trace estimator [QLo1][jSGi].
- **Discussion on related works:** We have added more discussion on related works that connect to our analytical formulation of Fisher in DMs [r2vX][QLo1].
- **Connection of AFI-OT to AFI:** We have added more discussion on the connection of our Theorem 1 to other parts of our paper [r2vX].
- **Syntax errors and typos:** We have revised typos and syntax errors as kindly pointed out [jSGi][r2vX].
- **Tighter bounds:** We derive tighter error bounds in Proposition 7 [jSGi].
- **Intuition of AFI-EA:** We add explanations on the intuition of our AFI-EA method [jSGi].
- **Experimental details:** We have added experimental details on the training costs and hyperparameters as kindly pointed out [iHvF].

We are immensely grateful for your feedback and suggestions for improving our manuscript. We kindly request your generous consideration of these points in the final evaluation of our paper.

Sincerely,

Authors of Submission 383
</content>
</thread_object>

</thread>


Metareview:
metareview:
**Summary of Discussion:**  
The paper addresses the analytical formulation of Fisher Information (FI) in diffusion models and proposes algorithms for likelihood evaluation and adjoint guidance sampling. While the reviewers appreciated the potential of these contributions, key concerns remained.  

**Key Concerns:**  

1. **Limited Theoretical Novelty:**  
   - Proposition 3 and related theoretical results were found to be extensions or adaptations of prior work, limiting their impact.  
   - Theorem 1, while corrected during rebuttal, offered minimal practical significance.  

2. **Experimental Scope:**  
   - Experiments lacked breadth and did not convincingly demonstrate the utility of the proposed algorithms across diverse scenarios.  
   - Insufficient initial detail on training costs and hyperparameter tuning raised concerns about reproducibility.  

3. **Presentation:**  
   - Clarity and organization issues, particularly around linking theory to algorithms, reduced the paper's accessibility.  
   - Language improvements during rebuttal were noted but were not sufficient to fully address concerns.  

**Conclusion:**  
While the paper tackles a relevant problem and proposes promising algorithms, the limited novelty, narrow experimental scope, and clarity issues ultimately do not meet the bar for acceptance. With expanded experiments and a more focused presentation, the work could contribute to future advancements in this area.