Diffusion-PINN Sampler

Authors: Zhekun Shi, Longlin Yu, Tianyu Xie, Cheng Zhang
Primary area: probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords: posterior sampling, multi-modal sampling, mixing proportion identification, diffusion model, physics-informed neural network

Abstract:
Recent success of diffusion models has inspired a surge of interest in developing sampling techniques using reverse diffusion processes. However, accurately estimating the drift term in the reverse stochastic differential equation (SDE) solely from the unnormalized target density poses significant challenges, hindering existing methods from achieving state-of-the-art performance. In this paper, we introduce the Diffusion-PINN Sampler (DPS), a novel diffusion-based sampling algorithm that estimates the drift term by solving the governing partial differential equation of the log-density of the underlying SDE marginals via physics-informed neural networks (PINN). We prove that the error of log-density approximation can be controlled by the PINN residual loss, enabling us to establish convergence guarantees of DPS. Experiments on a variety of sampling tasks demonstrate the effectiveness of our approach, particularly in accurately identifying mixing proportions when the target contains isolated components.

Decision: reject
Reviewer scores: 5, 8, 3, 6

Reviews and discussion:
<thread>
creation date:
1729586444507

thread_type:
official_review

<thread_object>
creator: reviewer_ndls

<content>
summary:
This paper introduces a novel diffusion-based sampling algorithm called the Diffusion-PINN Sampler (DPS). The DPS estimates the drift term in the reverse diffusion process by solving the governing partial differential equation of the log-density of the underlying stochastic differential equation (SDE) marginals using physics-informed neural networks (PINN). The authors prove that the error of log-density approximation can be controlled by the PINN residual loss, which allows them to establish convergence guarantees for the DPS. Experiments on sampling tasks demonstrate the effectiveness of this method.

strengths:
1. The paper is well-composed and the concepts are clearly articulated, making it easy to comprehend. 
2. The innovative approach of learning a diffused log-density via the log-density-PF equation is novel.
3. The math derivation is rigorous.

weaknesses:
1. When the score of the target distribution is accessible, we can perform sampling using MCMC[1] and ParVI[2][3][4]. Renowned algorithms such as HMC and SVGD should be considered for comparison. It would be interesting to see if PINN-Sampling can surpass these methods.

2. The training objective in equation 14 requires gradients of the neural network. This could potentially be time-intensive as the dimension increases. I am skeptical about the scalability of this method. Could you provide a complexity analysis of the training process in relation to dimensionality?

3. Assumption 1 is not plausible, as the suggested scenario is unlikely to occur. Consequently, it's not feasible to bound the error within a constrained domain. It's probable that the subsequent theorems will also hold when $\Omega = \mathbb{R}^d$ and $\nu_t$ are light-tailed distributions.

4. In Example 1, as $\tau$ approaches zero, the variance of $\pi^M$ and $\hat{\pi}^M$ becomes unbounded. However, in real-world applications, the variance of our data distribution is always bounded. Could you give an example where the data distribution has a bounded variance but still has an arbitrarily small $\tau$?

5. The scope of the experiments appears to be rather limited, being mostly synthetic and low-dimensional. It would be beneficial to see applications to higher-dimensional cases, Gaussian processes, Bayesian Neural Networks, and real-world data experiments.

References:

[1] Neal R M. MCMC using Hamiltonian dynamics[J]. arXiv preprint arXiv:1206.1901, 2012. 

[2] Liu Q, Wang D. Stein variational gradient descent: A general-purpose Bayesian inference algorithm[J]. Advances in neural information processing systems, 2016, 29. 

[3] Liu C, Zhuo J, Cheng P, et al. Understanding and accelerating particle-based variational inference[C]//International Conference on Machine Learning. PMLR, 2019: 4082-4092. 

[4] Wang F, Zhu H, Zhang C, et al. GAD-PVI: A General Accelerated Dynamic-Weight Particle-Based Variational Inference Framework[C]//Proceedings of the AAAI Conference on Artificial Intelligence. 2024, 38(14): 15466-15473.

questions:
see Weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730581806468

thread_type:
official_review

<thread_object>
creator: reviewer_x3wa

<content>
summary:
The paper studies sampling via time-reversing a diffusion process that starts in the target distribution, i.e. it considers diffusion-based generative modeling, where, however, no data samples from the target distribution are available, but the (unnormalized) target density can be evaluated. The authors suggest to learn the score function that is necessary to sample accurately by approximating the underlying Fokker-Planck PDE (in log-space) with Physics-informed neural networks (PINNs). In particular, it is argued that solving the Fokker-Planck PDE and taking derivatives afterwards is advantageous over solving a corresponding PDE for the score directly. The paper presents theoretical analyses that bound the error of the approximations of the log-density and the score, respectively, by the (weighted) PINN objective. Leveraging previous work, this in consequence allows to bound the sampling error by the PINN objective. Some numerical evaluation on synthetic sampling problems in moderate dimensions is provided, demonstrating a proof of concept of the method.

strengths:
Sampling via learned diffusion processes is an interesting and active field of research, which is arguably more challenging than the typical generative modeling task since no samples from the target are available. The connection to the underlying PDEs is interesting, however, not novel (see weaknesses). Still, the presentation is sound and offers a detailed and interesting theoretical analysis.

The paper is mathematically well written, however, some (practical) implications and motivations could be made clearer for the convenience of the reader.

weaknesses:
In my opinion, the paper has the following main weaknesses.

1. **Novelty.** The idea of employing PINNs for diffusion-based sampling has already been suggested in [1], [2], [3] and [4]. While [1] and [2] have been cited, only a preliminary version of [3] seems to be mentioned. In fact, in [3] (almost) the same algorithm has been presented (without relying on sampling from an MCMC algorithm) and multiple numerical experiments (including also variants of the suggested method, such as e.g. deterministic evolutions, general evolutions, annealed/prescribed evolutions) have been conducted.

2. **Numerical evaluation.** While the numerical evaluation in the paper showcases good performance of the suggested algorithm, the considered examples are rather synthetic and of moderate dimension. The question of how the algorithm scales to real-world problems and higher-dimensional examples (say, $d > 100$, which might be challenging for PINNs due to computation of higher-order derivatives) remains open.

3. **Practical relevance of theoretical results.** While the theoretical analyses look sound and are nice from a mathematical perspective, I am not certain about their practical relevance. It is clear that in principle a zero PINN loss implies zero score error and (up to discretization and up to the prior error) perfect sampling. At the same time, the provided bound seems to not allow for quantitative statements since constants cannot be computed and it is (due to the Grönwall inequality) probably far from being tight. Furthermore, it seems that the theoretical results heavily rely on earlier works by [5] and [6].

Some additional comments are the following:

4. It is unclear to me how generating collocation points with MCMC-like algorithms (e.g. LMC) is sufficient. It is known that those algorithms converge exponentially slowly for multimodal targets where the modes are sufficiently separated. Therefore, some exploration strategy seems to be necessary in order to get collocation points that cover regions containing distant modes.
5. You write that trajectory-based alternatives lead to "computational complexity associated with differentiating through SDE solvers". However, this is not necessarily true since off-policy training can be chosen also for trajectory-based methods, e.g. by using divergences other than the KL-divergence, see, e.g., [7], [8].
6. The related work section barely lists any of the many alternative methods for diffusion-based sampling that have been suggested in the last 1-2 years.
7. Concurrently to the work of [9], [1] has also derived the PDE for the log-density (and in fact, also approached it via PINNs, see above).
8. You write about "a linear interpolation (i.e., annealing) path between the target distribution and a simple prior", however it’s linear only in log-space.
9. It would be helpful to define the Fisher divergence in the main text (and not only in Appendix A.2).
10. Sampling quality can usually not be measured well by only one metric and therefore it would be even better to have more metrics (e.g. Wasserstein, ELBO, ESS, comparison to expectation reference values etc.).

I apologize in case I misunderstood certain aspects of your paper and am looking forward to potential corrections.

[1] Julius Berner, Lorenz Richter, and Karen Ullrich. An optimal control perspective on diffusion-based generative modeling. Transactions on Machine Learning Research, 2024.

[2] Bálint Máté and François Fleuret. Learning interpolations between Boltzmann densities. Transactions on Machine Learning Research, 2023.

[3] Sun, Jingtong, et al. "Dynamical measure transport and neural PDE solvers for sampling." arXiv preprint arXiv:2407.07873 (2024).

[4] Albergo, Michael S., and Eric Vanden-Eijnden. "NETS: A Non-Equilibrium Transport Sampler." arXiv preprint arXiv:2410.02711 (2024).

[5] Sitan Chen, Sinho Chewi, Jerry Li, Yuanzhi Li, Adil Salim, and Anru Zhang. Sampling is as easy as learning the score: theory for diffusion models with minimal data assumptions. In The Eleventh International Conference on Learning Representations, 2023b. 

[6] Teo Deveney, Jan Stanczuk, Lisa Maria Kreusser, Chris Budd, and Carola-Bibiane Schönlieb. Closing the ode-sde gap in score-based diffusion models through the fokker-planck equation. arXiv preprint arXiv:2311.15996, 2023. 

[7] Lorenz Richter and Julius Berner. Improved sampling via learned diffusions. In International Conference on Learning Representations, 2024.

[8] Sendera, Marcin, et al. "On diffusion models for amortized inference: Benchmarking and improving stochastic control and sampling." arXiv preprint arXiv:2402.05098 (2024).

[9] Chieh-Hsin Lai, Yuhta Takida, Naoki Murata, Toshimitsu Uesaka, Yuki Mitsufuji, and Stefano Ermon. Fp-diffusion: Improving score-based diffusion models by enforcing the underlying score fokker-planck equation. In International Conference on Machine Learning, pp. 18365–18398. PMLR, 2023.

questions:
1. If I understand it correctly, you claim that Example 1 shows that the Fisher divergence can be arbitrarily small, while the KL divergence stays large at the same time. However, clearly, if the Fisher divergence goes to zero, so does the KL divergence (by the definition of a divergence). Therefore, your statement seems to mean that "the KL divergence can be (much) larger than the Fisher divergence", is this correct? This implies the following questions:
    - a) Since the "scale" of any divergence is rather arbitrary, why should the scale of the KL divergence be better suited than the scale of the Fisher divergence? (That said, I think it could be helpful to consider log-scales in Figure 1.)
    - b) Does your observation generalize to distributions that are not Gaussian?
    - c) Are you aware of any bounds that directly relate the KL divergence to the Fisher divergence? (I’m thinking of something similar to the Poincare inequality.)
2. I find the numerical experiments in Figure 4 more enlightening than the statement in Example 1. Do those observations carry over to other (more complicated, high-dimensional) examples? It seems to me that Lai et al. (2023) have found that using the score FPE (as a regularizer) seems to work well. Can you comment on this finding, given your argument against the score FPE? Also, what happens if you use more gradient steps in the example from Figure 4 (both losses seem to not have converged yet)? Does the score FPE eventually converge sufficiently? (Actually, how do you define "score error"? Is it some kind of $L^2$ error?)
3. If one approximates the log-density as you do, one later must take gradients to get a score approximation, which might incur errors. Can you comment on potential numerical implications?
4. Did you use Hutchinson’s gradient estimator that you introduce in lines 262 and following? While it removes a derivative, it is known to increase the variance of gradient estimators. 
5. In (20) one considers $L_\mathrm{PINN}(t_k; C_1(\varepsilon))$ for multiple $t_k$, however, the integral in $L_\mathrm{PINN}$ always starts at $t=0$. Does this mean that the larger $N$ the larger is the sum? (I would not have expected the term to increase with finer time discretization.)

Some comments on notation and typos:
1. The notation seems to be not consistent all the time, e.g. regarding the time index: $g(t), f(x,t)$, but $p_t(x)$ and not $p(x, t)$; also: $u_t(x)$ vs. $u_\theta(x, t)$.
2. Why do you write $\nabla_x$, but not $\Delta_x$ or $\nabla_x \cdot f$? Also, you write $\nabla_x u_\theta(z, T)$ (e.g. in l. 254). Shouldn’t it be $\nabla_z u_\theta(z, T)$? (Alternatively, you should explain your notation. Suggestion: Writing $\nabla$ instead of $\nabla_x$ solves the issue.)
3. Some citations are not correctly formatted (e.g. "monte carlo" instead of "Monte Carlo"), some versions are old.
4. Inconsistent writing: Fokker-Planck vs. Fokker Planck.
5. 83: Definition of norm: "+" is missing.
6. 83: "Let ν denotes".
7. 199: "until the every end".
8. 320: Assumption -> Assumptions (this typo appear multiple times throughout the paper).
9. 326: After (17) replace "." with ",".
10. 333: (Arjovsky et al. (2017)).
11. 355: "Let $\hat{\pi}_T$ denotes".
12. In the appendix, there are often commas or periods missing after equations, see, e.g. (22), (23) etc.
13. 750: "Let $p(x)$ denotes".
</content>
</thread_object>

</thread>
<thread>
creation date:
1730662163941

thread_type:
official_review

<thread_object>
creator: reviewer_rbag

<content>
summary:
The paper presents a new approach for diffusion sampling. Inspired by existing methods that use physics-informed neural networks, they address the problem of solving the reverse diffusion process using the Fokker Plank equation which models the evolution of marginals $p_t(x)$ in the forward process. Specifically, they identify flaws of existing methods and address them. The first one is the failure of the score FPE and the second is the lack of theoretical analysis of proposed methods. They end up with a method called DPS (Diffusion-PINN sampler) that solves for log-density FPE. They show promising results and consistent improvement over other methods, providing, in addition, convergence analysis of the method.

strengths:
The paper is very well written! It was pleasant to read and easy to follow.

- The authors provide observations of failure of score FPE and propose a motivated and illustrated solution : solving the log-density FPE and plugging it in the reverse process afterwards.

- The authors provide a good amount of experiments and parsimonious illustrations (just what is needed to illustrate their claims).

- The authors assess performances of their methods with good metrics. It could seem naive to say, but a lot of people miss the use of probabilistic measures for probabilistic problems. The choice of KL/Fisher divergence is welcomed along with the $L^2$ of the log-density.

- The use of PINN for such a problem is relevant. It bases the method on well-established (and timely) fields which is nice!

- The overall structure of the work is clean and each part is self-sufficient.

- Analysis, performed on toy problems at first, allows for good interpretation of the results.

- The proposed sampling methods show big improvement over other baselines for considered problems which further motivates it.

weaknesses:
I'm really sorry, but I think you will have to change the name (at least the reduced name) of your method! DPS is already a well known and well established sampling method for posterior problems [1]. I think it would be beneficial to avoid being shadowed by existing method. Moreover, it's more than just a sampling method that you present. It's a conjugate training and sampling algorithm. Your main change is about what your network is trained for (the log density) and not about the FPE which is not new in itself (Although the training and the sampling make a whole). 

Otherwise, I point below a few questions and small concerns.

**Small details**

- Line 60: *to ensure convergence guarantee*, it's nitpicking but isn't it a bit redundant?

**Performance concerns**

- I'm a little bit concerned about the performances of the method both in time and resources. It would be nice to, at least discuss them and at best provide some measures in Appendix for example. I detail below my concerns.

- You model the log-density, then you have to take its gradient in order to compute the score that you plug into the reverse process, right? What is the cost of this compared to direct score matching? Indeed, I presume that taking the grad at every step is not without cost. 

- The PINN loss involves gradient and Laplacian operators. What is the cost of computing such a loss compared to simple denoising score matching (which, at the end, is not more complex than an mse)? Does it slow down training? Do you use optimized tricks like jvp/vjp when possible? Could you comment a bit on that please?

- In table 1, isn't it misleading to report KL only on first dimensions for Funnel and Double-well? Can't you report either on several sub-combinations of dimensions or use another measure? Is it due to the cost of the KL in higher dimension? What about more tractable divergence metrics such as the Sinkhorn divergence?

**Sampling method**

- From your text and as the first requirement in **Algorithm 1** you speak about access to the initial condition which derives from the unnormalized density. It is still not clear to me how you access to such density? 

- You present a sampling method, and even more, for diffusion models. It would be nice to have a higher dimensional problem (I agree that it is not necessary to cover your claims here). This method should be studied in more realistic settings also (with images, high dimensional data, ...) and I'm worried about the cost of computing explicitly the score from the log-density or the cost of the loss in such settings. At least comment on that in the main paper.

- Echoing with the previous comment: What about conditional sampling? Posterior problems. Technically your method can extend easily. You compute the prior score as for now and plug the likelihood to guide the sampling. It would be nice (and I think easy) to show, in Appendix if you lack space, the conditional generation for one toy problem. For example, generating only on one mode in the 9-Gaussians or Rings problem. I'm curious to see the benefit of more accurate prior score to such problems. I'm also curious to see if you still perform much better than other considered methods. Indeed, the likelihood guiding score can sometimes help and simplify a bit the problem.

- Nitpick: In figure 3 you say "Sampling performance". Those figures do not show any performances. Maybe change it to "Samples from different methods ...".


**Additional suggestion**

Figure 5 (center and right) are not really revealing. I know that it's hard to display samples in high dim. Could you put corner plots instead? Even if it's on a subset of dimension of the problem. It would be easier to see the differences with and without regularization if you display marginals along this 2-dimensional joint.


My current score is 6. I would gladly increase my score once my concerns have been addressed and discussed. 
Thanks for your good work!

**[1] Diffusion Posterior Sampling for General Noisy Inverse Problems - ICLR 2023 spotlight**

questions:
See weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730690911256

thread_type:
official_review

<thread_object>
creator: reviewer_yacx

<content>
summary:
This paper explores diffusion-based models for unnormalized sampling and introduces the Diffusion-PINN Sampler (DPS). Using physics-informed neural networks (PINNs), DPS directly approximates the log-density of SDE marginals, enabling more precise modeling of complex distributions. The authors provide theoretical convergence analysis and validate the method on several synthetic datasets.

strengths:
- The paper is generally easier to follow. Theoretical results in Sec 5 could be of interested for readers from other domains.

weaknesses:
- The proposed method is computationally expansive compared to other diffusion-based models to train—due to the Laplacian in PINN loss—and to sample from—due to the evaluation of gradient at every time step.
- Insufficient comparison to modern baselines — in addition to  MC methods like HMC or SMC, there’re many other diffusion/SDE based methods, such as [1,2], just to name a few.
- Experiments were only conducted on rather simple, synthetic, target. I’ll be more convinced to see some higher-dim experiments and/or real-world dataset. 
- Parametrize NN with log mu (12) seems like a strong inductive bias and can be infeasible for many practical applications (e.g., sample Boltzmann distribution for conformation generation) when querying energy functions are expansive. Can the authors provide ablation study without such parametrization? 
- Related works Section is rather short and should be extended.

[1] Particle Denoising Diffusion Sampler (ICML 2024)
[2] Improved sampling via learned diffusions (ICLR 2024)

questions:
- $\ell_reg$ seems ad-hoc. Why regress onto the boundary condition of the score instead of (11b), which seems more aligned with PINN loss?
- Given that x0 comes from LMC already, is the method computationally cheaper compared to standard MCMC methods?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732414290014

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Global response

comment:
## Ablation study

We conduct several ablation studies in this section. First, we demonstrate that a good score approximation can be achieved as long as the log-density is well approximated, as supported by our theoretical results. The corresponding results are shown in the left panel of Figure 8 in the Appendix D.3.1 of our revision. Additionally, we show that both using the parameterization trick and including the initial loss in the objective lead to a good score approximation, as evidenced by the results in the right panel of Figure 8 in the Appendix D.3.1 of our revision. 

## Complexity analysis

We propose an **unbiased** Hutchinson gradient estimator to reduce the computational cost of evaluating the Laplacian in the PINN loss. We examine its impact on training time and report the sampling time in the following table. Notably, without this estimator, training time increases significantly as the dimensionality grows. In contrast, our approach effectively mitigates this issue. Furthermore, once the log-density approximation is obtained, sampling can be performed in a remarkably short time, taking only $2$ seconds. Moreover, we compare the sampling time using direct score estimation versus taking the gradient of the approximated log-density. Our results show that sampling time is halved when score estimation is directly employed. Nonetheless, the sampling time of our method is already highly efficient. For instance, sampling 10,000 points over 1,000 time steps for 100-dim tasks takes less than 2 seconds.

Table 6: Complexity analysis for our method.

| **Time (s)**                              | 10d       | 20d       | 30d       | 40d       | 50d       | 60d       | 70d       | 80d       | 90d       | 100d      |
| -------------------------------------- | --------- | --------- | --------- | --------- | --------- | --------- | --------- | --------- | --------- | --------- |
| **Per iteration training (without trick)** | 0.037     | 0.062     | 0.086     | 0.111     | 0.135     | 0.160     | 0.184     | 0.210     | 0.235     | 0.260     |
| **Per iteration training (with trick)**   | **0.018** | **0.018** | **0.018** | **0.018** | **0.018** | **0.018** | **0.018** | **0.018** | **0.018** | **0.018** |
| **Sampling (taking gradient)**      | 1.537     | 1.534     | 1.538     | 1.551     | 1.558     | 1.567     | 1.570     | 1.581     | 1.587     | 1.596     |
| **Sampling (using score directly)**    | 0.683     | 0.685     | 0.701     | 0.731     | 0.736     | 0.742     | 0.759     | 0.775     | 0.794     | 0.812     |

---



References:

[1] Improved sampling via learned diffusions (ICLR 2024)

[2] Blessing D, Jia X, Esslinger J, et al. Beyond ELBOs: A Large-Scale Evaluation of Variational Methods for Sampling.

[3] Sun, Jingtong, et al. "Dynamical measure transport and neural PDE solvers for sampling." arXiv preprint arXiv:2407.07873 (2024).
</content>
</thread_object>

</thread>
<thread>
creation date:
1732414334269

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Global response

comment:
## More evaluation metrics

Additionally, we provide results using additional evaluation metrics, including the $W_2$-distance and EMC value proposed in [2], for specific sampling tasks. These supplementary evaluations further highlight the superior sampling performance of our method, while PIS-LV and DIS-LV exhibit comparable performance, consistent with our main evaluation results.

Table 3: $W_2$-distance to the ground truth obtained by different methods

| **$W_2$-distance ($\downarrow$)** | **$9$-Gaussians**                 | **Rings**                         |
| --------------------------------- | --------------------------------- | --------------------------------- |
| **LMC**                           | $5.0064_{\pm 0.076}$              | $3.4096_{\pm 0.075}$              |
| **RDMC**                          | $3.1329_{\pm 0.084}$              | $1.2195_{\pm 0.102}$              |
| **SLIPS**                         | $0.9884_{\pm 0.143}$              | $2.2280_{\pm 0.064}$              |
| **DIS**                           | $5.4442_{\pm 0.056}$              | $3.2716_{\pm 0.078}$              |
| **PIS**                           | $5.0341_{\pm 0.052}$              | $3.7429_{\pm 0.070}$              |
| **SVGD**                          | $3.9018_{\pm 0.054}$              | $2.0395_{\pm 0.127}$              |
| **HMC**                           | $4.5945_{\pm 0.075}$              | $2.3255_{\pm 0.111}$              |
| **PIS-LV**                        | $4.5839_{\pm 0.047}$              | $\mathbf{0.7146_{\pm 0.076}}$     |
| **DIS-LV**                        | $1.0534_{\pm 0.214}$              | $\mathbf{0.7078_{\pm 0.118}}$     |
| **DPS (ours)**                    | $\boldsymbol{0.7794_{\pm 0.127}}$ | $\boldsymbol{0.7726_{\pm 0.099}}$ |

Table 4: EMC value obtained by different methods

| **EMC ($\uparrow$)** | **9-Gaussians**                   | Rings                         |
| :------------------- | --------------------------------- | ----------------------------- |
| **LMC**              | $0.3562_{\pm 0.005}$              | $0.3844_{\pm 0.007}$          |
| **SLIPS**            | $0.9822_{\pm 0.005}$              | $0.7434_{\pm 0.008}$          |
| **RDMC**             | $0.6195_{\pm 0.014}$              | $0.8754_{\pm 0.009}$          |
| **PIS**              | $0.3341_{\pm 0.003}$              | $0.3328_{\pm 0.004}$          |
| **DIS**              | $0.2988_{\pm 0.003}$              | $0.4081_{\pm 0.007}$          |
| **SVGD**             | $0.5095_{\pm 0.008}$              | $0.7093_{\pm 0.067}$          |
| **HMC**              | $0.4139_{\pm 0.075}$              | $0.4235_{\pm 0.008}$          |
| **PIS-LV**           | $0.3208_{\pm 0.001}$              | $\mathbf{0.9976_{\pm 0.002}}$ |
| **DIS-LV**           | $\mathbf{0.9942_{\pm 0.002}}$     | $\mathbf{0.9983_{\pm 0.001}}$ |
| **DPS (ours)**       | $\boldsymbol{0.9965_{\pm 0.002}}$ | $\mathbf{0.9928_{\pm 0.002}}$ |

## Higher-dimensional experiments

Furthermore, we extend our experiments on the Double-Well target to $50$ dimensions with $32$ separated modes for several baselines, following the setup in [3]. The results are presented in the table below. Notably, our target distribution features challenging mixing proportions, making it particularly difficult to sample effectively.

Table 5: Sliced KL divergence to the ground truth and mixing proportions estimation error obtained by different methods.

| **Method**     | **Sliced KL divergence  ($\downarrow$)** | **Mixing proportions estimation error ($\downarrow$)** |
| -------------- | ---------------------------------------- | ------------------------------------------------------ |
| **LMC**        | $0.321_{\pm 0.009}$                      | $0.0681_{\pm 0.0053}$                                  |
| **SLIPS**      | $0.745_{\pm 0.018}$                      | $0.1323_{\pm 0.0062}$                                  |
| **RDMC**       | $2.769_{\pm 0.022}$                      | $0.1641_{\pm 0.0048}$                                  |
| **PIS**        | $0.362_{\pm 0.012}$                      | $0.0203_{\pm 0.0021}$                                  |
| **DIS**        | $0.973_{\pm 0.017}$                      | $0.0808_{\pm 0.0051}$                                  |
| **SVGD**       | $24.349_{\pm 0.067}$                     | $0.1710_{\pm 0.0046}$                                  |
| **HMC**        | $0.245_{\pm 0.057}$                      | $0.0212_{\pm 0.0038}$                                  |
| **DPS (ours)** | $\boldsymbol{0.101_{\pm 0.011}}$         | $\boldsymbol{0.0008_{\pm 0.0003}}$                     |
</content>
</thread_object>

</thread>
<thread>
creation date:
1732414398474

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Global response

comment:
# Global response

We sincerely thank all reviewers for their constructive feedback. In response, we present additional experimental results as follows.

## More baselines

We provide a comprehensive comparison with additional baselines, including MCMC methods (LMC, HMC), particle-based VI methods (SVGD), diffusion-based sampling approaches (RDMC, SLIPS, DIS, PIS), and recent variants of diffusion-based sampling methods (PIS-LV, DIS-LV) as outlined in [1]. The results demonstrate that our method consistently achieves superior sampling performance across various tasks. Notably, recent variants of diffusion-based sampling methods (PIS-LV, DIS-LV) exhibit comparable performance to ours on certain tasks but tend to be less stable, as evidenced by the issue such as missing several modes in the 9-Gaussians example for PIS-LV.

Table1: KL divergence to the ground truth obtained by different methods.

| KL divergence ($\downarrow$) | $9$-Gaussians                  | Rings                          | Funnel                         | Double-well                    |
| ---------------------------- | ------------------------------ | ------------------------------ | ------------------------------ | ------------------------------ |
| **LMC**                      | $1.6568_{\pm 0.0189}$          | $2.4754_{\pm 0.0302}$          | $0.1908_{\pm 0.0156}$          | $0.1915_{\pm 0.0122}$          |
| **RDMC**                     | $1.0844_{\pm 0.0132}$          | $0.7487_{\pm 0.0073}$          | $2.0250_{\pm 0.0364}$          | $1.5735_{\pm 0.0162}$          |
| **SLIPS**                    | $0.0901_{\pm 0.0071}$          | $0.4127_{\pm 0.0144}$          | $0.1971_{\pm 0.0133}$          | $0.4840_{\pm 0.0145}$          |
| **DIS**                      | $2.2758_{\pm 0.0240}$          | $2.3433_{\pm 0.0275}$          | $0.2383_{\pm 0.0169}$          | $0.6796_{\pm 0.0139}$          |
| **PIS**                      | $2.0042_{\pm 0.0203}$          | $2.6985_{\pm 0.0290}$          | $0.4377_{\pm 0.0199}$          | $0.0969_{\pm 0.0114}$          |
| **SVGD**                     | $0.9712_{\pm 0.0153}$          | $0.1608_{\pm 0.0119}$          | $0.1006_{\pm 0.0188}$          | $1.3768_{\pm 0.0683}$          |
| **HMC**                      | $1.3814_{\pm 0.0171}$          | $1.0258_{\pm 0.0297}$          | $\mathbf{0.0289_{\pm 0.0266}}$          | $0.1546_{\pm 0.0474}$          |
| **PIS-LV**                   | $2.1301_{\pm 0.0224}$          | $\mathbf{0.0124_{\pm 0.0204}}$ | $0.1521_{\pm 0.0230}$          | $0.0478_{\pm 0.0280}$          |
| **DIS-LV**                   | $0.0682_{\pm 0.0081}$          | $0.0369_{\pm 0.0178}$          | $0.0362_{\pm 0.0167}$ | $0.0358_{\pm 0.0256}$          |
| **DPS (ours)**               | $\mathbf{0.0131_{\pm 0.0093}}$ | $\mathbf{0.0176_{\pm 0.0059}}$ | $0.0846_{\pm 0.0122}$ | $\mathbf{0.0273_{\pm 0.0113}}$ |

Table 2: Mixing proportions estimation error obtained by different methods.

| **Mixing proportions estimation error ($\downarrow$)** | $9$-Gaussians                  | Rings                          | Double-well                    |
| ------------------------------------------------------ | ------------------------------ | ------------------------------ | ------------------------------ |
| **LMC**                                                | $0.5199_{\pm 0.0159}$          | $0.6005_{\pm 0.0251}$          | $0.0673_{\pm 0.0082}$          |
| **RDMC**                                               | $0.1313_{\pm 0.0099}$          | $0.0537_{\pm 0.0035}$          | $0.2154_{\pm 0.0075}$          |
| **SLIPS**                                              | $0.0018_{\pm 0.0005}$          | $0.2471_{\pm 0.0144}$          | $0.1645_{\pm 0.0113}$          |
| **DIS**                                                | $0.7268_{\pm 0.0146}$          | $0.5233_{\pm 0.0194}$          | $0.0684_{\pm 0.0035}$          |
| **PIS**                                                | $0.4893_{\pm 0.0110}$          | $0.8016_{\pm 0.0194}$          | $0.0044_{\pm 0.0011}$          |
| **SVGD**                                               | $0.2098_{\pm 0.0097}$          | $0.1767_{\pm 0.0355}$          | $0.2400_{\pm 0.0174}$          |
| **HMC**                                                | $0.3766_{\pm 0.0154}$          | $0.4994_{\pm 0.0200}$          | $0.0159_{\pm 0.0046}$          |
| **PIS-LV**                                             | $0.4217_{\pm 0.0009}$          | $0.0010_{\pm 0.0013}$          | $\mathbf{0.0005_{\pm 0.0004}}$ |
| **DIS-LV**                                             | $0.0013_{\pm 0.0004}$          | $\mathbf{0.0007_{\pm 0.0004}}$ | $0.0012_{\pm 0.0009}$          |
| **DPS (ours)**                                         | $\mathbf{0.0006_{\pm 0.0003}}$ | $\mathbf{0.0006_{\pm 0.0006}}$ | $\mathbf{0.0004_{\pm 0.0002}}$ |
</content>
</thread_object>

</thread>
<thread>
creation date:
1733282367672

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Global response

comment:
# Global response

---

We sincerely thank the reviewers for their valuable and constructive feedback. In response, we emphasize the novelty of our work, address shared concerns, and provide further clarification of our theoretical contributions as outlined below.

---

**Novelty**: Recently, sampling algorithms using PINN have garnered significant attention in research. However, we would like to highlight the unique contributions and novelty of our work in comparison to existing studies. Our approach leverages the reverse process of diffusion models for sampling, distinguishing it from related works ([3,4,5]) that rely on alternative paths, such as annealing path. Additionally, we identify the limitations of using the score Fokker-Planck equation for sampling from multi-modal distributions, specifically in **identifying mixing proportions for multi-modal distributions**, which is known as the blindness of score matching ([6]). This limitation demonstrates the importance of selecting the appropriate equation (log-density or score Fokker-Planck equation) for diffusion-based sampling. Moreover, our work provides a **comprehensive theoretical analysis of the proposed method and considers more challenging multi-modal distributions with unequal mixing proportions**, highlighting the uniqueness of our contributions in comparison to related studies ([7,8]). Notably, our work is concurrent with [8] and was submitted to NeurIPS 2024, where it received strong feedback, achieving an average review score of 6 and narrowly missing acceptance.

---

**Scalability**: The scalability of our approach primarily hinges on the performance of PINN in high-dimensional settings. Encouragingly, recent advancements ([1,2]) have introduced various methods that enhance PINN’s capacity to solve high-dimensional partial differential equations effectively. These advancements can be seamlessly integrated into our sampling framework, ensuring that our algorithm remains scalable and efficient even in higher-dimensional scenarios.

---

**Practical relevance of our theoretical results**: We clarify that our theoretical results for PINN demonstrate that if the PINN residual loss is sufficiently small, the approximation errors for both the log-density and the score will also be small. This establishes a solid theoretical foundation for our method, showing that minimizing the PINN residual to approximate the log-density can also yield a reliable approximation of the score.

---

References:

[1]. Zheyuan Hu, Zekun Shi, George Em Karniadakis, and Kenji Kawaguchi. Hutchinson trace estimation for high-dimensional and high-order physics-informed neural networks. Computer Methods in Applied Mechanics and Engineering, 424:116883, 2024a.

[2]. Zheyuan Hu, Khemraj Shukla, George Em Karniadakis, and Kenji Kawaguchi. Tackling the curse of dimensionality with physics-informed neural networks. Neural Networks, pp. 106369, 2024b.

[3]. Máté B, Fleuret F. Learning interpolations between boltzmann densities[J]. arXiv preprint arXiv:2301.07388, 2023.

[4]. Fan M, Zhou R, Tian C, et al. Path-Guided Particle-based Sampling[C]//Forty-first International Conference on Machine Learning.

[5]. Tian Y, Panda N, Lin Y T. Liouville Flow Importance Sampler[J]. arXiv preprint arXiv:2405.06672, 2024.

[6]. L. K. Wenliang. Blindness of score-based methods to isolated components and mixing proportions.
arXiv preprint arXiv:2008.10087, 2020.

[7].  Julius Berner, Lorenz Richter, and Karen Ullrich. An optimal control perspective on diffusion-based generative modeling. Transactions on Machine Learning Research, 2024.

[8]. Sun, Jingtong, et al. "Dynamical measure transport and neural PDE solvers for sampling." arXiv preprint arXiv:2407.07873 (2024).
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper uses physics-informed neural networks to establish a diffusion-based sampling algorithm. There are two salient concerns from the reviewers about this submission. One is that physics informed neural networks (PINN) have already been used in diffusion-based samplers. It is not yet clear that this paper's modification / improvement is significant enough (measured in terms of effectiveness against existing benchmarks) to grant it a publication at ICLR. Another concern is about the scalability of PINN. I would suggest that the authors take another pass to address the above mentioned problems.