Title: Preconditioned Langevin Dynamics with Score-based Generative Models for Infinite-Dimensional Linear Bayesian Inverse Problems
Venue: NeurIPS 2025
Accepted: yes
Keywords: theory

score-based generative models

error analysis

hilbert space

langevin dynamics

preconditioning
Abstract: Designing algorithms for solving high-dimensional Bayesian inverse problems directly in infinite‑dimensional function spaces – where such problems are naturally formulated – is crucial to ensure stability and  convergence as the discretization of the underlying  problem is refined. In this paper, we contribute to this line of work by analyzing a widely used sampler for linear inverse problems: Langevin dynamics driven by score‑based generative models (SGMs) acting as priors, formulated directly in function space. Building on the  theoretical framework for SGMs in Hilbert spaces, we give a rigorous definition of this sampler in the infinite-dimensional setting and derive, for the first time, error estimates that explicitly depend on the approximation error of the score. As a consequence, we obtain sufficient conditions for global convergence in Kullback–Leibler divergence on the underlying function space. Preventing numerical instabilities requires preconditioning of the Langevin algorithm and we prove the existence  and form of an optimal preconditioner. The preconditioner depends on both the score error and the forward operator and guarantees a uniform convergence rate across all posterior modes. Our analysis applies to both Gaussian and a general class of non‑Gaussian priors. Finally, we present examples that illustrate and validate our theoretical findings.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe work analyzes Score-based Models for Bayesian inference of linear inverse problems, extending to the case that posterior is supported on an infinite-dimensional space. It is shown that a trace-class preconditioning operator is required for the Langevin diffusion. Sufficient conditions for global convergence in KL divergence are derived and the existence and the form of an optimal preconditioner which depends on the score approximation error and the forward operator are also proved. The analysis covers both Gaussian and non-Gaussian priors.\n\nSTRENGTHS AND WEAKNESSES:\nLifting the Bayesian inference algorithms to infinite-dimensional space is well-motivated. The work provides the convergence analysis for both Gaussian and general non-Gaussian priors. It has good amount of originality. \n\nHowever, the work has some restrictive assumptions and the authors do not justify these well and thus the significance is not obvious to me. For example, how can the non-singularity of $A_N$ and $A_N^\\top A_N$ be guaranteed in practical problems? Besides, the sufficient conditions of the optimal preconditioned is not well discussed to provide enough insights. \n\nOverall, it will be better if the authors can provide some examples which satisfies the derived sufficient conditions and thus make the significance of the work more clear and further improve the quality of the work. If the sufficient conditions are easy to validate, then the work will be more of importance.\n\nQUESTIONS:\n1. In Fig 1, the first row of simulation is described as having growing variance. However, it seems to me that the variance stay the same when the mode index is above 60 or so. Can the authors provide further explanation regarding this?\n\n2. Can the author justify the assumption that $A_N$ is non-singular?\n\n3. How plausible are the conditions, $p_j^{-1} \\epsilon_j^a(\\tau) = O(\\tau)$ and $\\lambda_j^{-1} \\epsilon_j^b(\\tau) = O(1)$ in Theorem 3.1? Can the authors provide some insights on these conditions?\n\n4. The assumption that $A_N^\\top A_N$ is diagonal in Proposition 4.1 and Theorem 4.1 seems to be rather restrictive.\n\n5. The mean reversion rate $\\kappa$ seems to be undefined in Proposition 4.1.\n\n6. Assumption 1 is on the score approximation error, can the authors provide some comments on how this error can be guaranteed to be small as later results depend on this error?\n\n7. In line 280, $|\\epsilon^a_j| \\ll \\lambda_j$ requires approximate error to be small ($\\epsilon^a_j$ need to be small) if the scale of the Brownian motion ($\\lambda_j$) is small. It will be better if the authors can present some examples for validating such sufficient conditions.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors analyze Langevin dynamics driven by learned scores in the infinite-dimensional (function) space setting. In particular, the authors studied how the approximation error of the learned score impacts the stationary distribution.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n+ Using score-based models for sampling in inverse problems (rather for generating new data) is an important research direction. The authors provide convincing examples why inverse problems coming from PDE settings should be studied in the infinite-dimensional setting, which is largely ignored in the more common \"generative modeling\" setting.\n\n+ The analysis leads to actionable insights, namely a preconditioner. Numerical results (e.g., Figure 3) demonstrate that using the preconditioner stabilizes the inversion. \n\n+ The authors also treat the case of non-Gaussian priors, as long as they are abs continuous to a Gaussian reference (standard assumption).\n\nWeaknesses:\n- The analysis is present only for the case that A^T A is diagonalizable. The authors comment on this limitation and state that more technical argument can handle the more general case but do not provide details. \n\n- The computational costs of obtaining the preconditioner are not discussed, especially what information about the forward model and the learned score are needed.\n\nQUESTIONS:\n- What system knowledge is required to compute the preconditioner and what are the computational costs? For example, it seems a complete diagonalization of the forward operator is needed, which can be costly. \n\n- On line 198 the authors discuss the bias introduced by the learned score. How large is this in the numerical examples? For which types of inverse problems is this a major issue and does it give insights when learned priors are useful and when not?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies diffusion modeling in the setting of infinite-dimensional linear inverse problems. While the importance of pre-conditioning has been noted for other solutions of inf-dim inverse problems, it has been absent from SGMs, leading to pathological results when the discretization is refined.\n\nThis paper notes the importance of pre-conditioning by starting from a Gaussian setting and establish the existence of an optimal pre-conditioner. The paper then extends this result in the non-Gaussian setting and present similar results. The paper then validates the result from 2 linear inverse problem and illustrates that pre-conditioning is important in robustly recovering the true source of the linear inverse problems.\n\nSTRENGTHS AND WEAKNESSES:\nI am quite unfamiliar to the field of Bayesian inverse problems, so I find it quite difficult to assess the novelty of this work with little knowledge of related work. Therefore, while I recommend acceptance for this paper, my confidence level is low. \n## Strengths\nWithin the scope of the linear inverse problem, this paper does a good job of explaining and demonstrating the role of the preconditioner. \n\n- The paper is well-written with an easy-to-follow Gaussian example that illustrates to readers who might not be familiar with the linear inverse problem (such as myself) to understand. \n- The paper's illustrations seem to support its robustness towards discretization, as well as the failure of vanilla Langevin. \n- The idea of an optimal preconditioner seems to be of wider interest \n\n## Weaknesses\n- The numerical illustrations are quite difficult to follow. For example, Figure 4 comes with very little explanation and I find it hard to understand the purpose of the figure. \n- The illustrations are largely synthetic example that does not include learned score functions, but instead rely on the ground truth score functions with perturbations. \n- The assumptions of joint diagonalizability and a diagonal $A^\\top A$ seem quite restrictive.\n\nQUESTIONS:\n- The paper studies a non-Gaussian setting, but the 2 illustrations seem to both stem from Gaussians. Is this an accurate characterization? Is it possible to illustrate a non-Gaussian example?\n- Is it possible to obtain the optimal pre-conditioner outside of illustrative examples? If not, does this paper point to a possible solution to the problem of choosing a sensible preconditioner in a practical setting?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers an infinite-dimensional version of the Langevin diffusion sampler to solve Bayesian inverse problems of form $y = AX_0 +n$, for $A$ a linear operator, $X_0\\sim \\mu$ an unknown Hilbert space valued random variable, and $y, n$ finite-dimensional, with $n$ given as Gaussian observation noise. Under this setup, the aim is to find the posterior of $X_0$ given the observations. In the infinite-dimensional case the diffusion process is driven by a $C$-Wiener process. This $C$ acts as a preconditioner and the aim of the paper is to understand how this preconditioner influences the error of the learned score. To start the analysis, it is assumed that the prior of $X_0$ is Gaussian. Under this assumption the paper provides the stationary distribution of the preconditioned Langevin diffusion in terms of the error of the score approximation as well as the KL-divergence between the true and approximate posteriors. It also considers how to choose an optimal preconditioner $C$. Finally, the paper generalises these results to the case where the prior is absolutely continuous with respect to a Gaussian reference measure.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. The submission is well-written and structured, making it easy to follow. Moreover, the mathematics is well presented with all assumptions clearly stated.\n\n2. The analysis provides insights into the optimal preconditioner $C$ in the case of Langevin samplers which I think is a valuable result. It is also shown how the preconditioner influences the convergence. \n\n3. I enjoyed the remarks and comments surrounding the statements of the main theorems and propositions. I found them useful for interpreting the results.\n\n\nWeaknesses:\n\n1. Pidstrigach et al. [48] also provide a similar result about the optimal $C$ based on the Wasserstein-2 distance between the true data distribution $\\mu$ and the learned sample distribution (see Theorem 14 and Section 6). Although the problem setting is a bit different, in that they are studying time-reversals rather than Langevin-based samplers for inverse problems, the results seem very related. Could you provide some more details on the similarities and differences to this result?\n\n2. It is assumed that the data distribution is absolutely continuous with respect to a Gaussian measure. Could you shed some light on how restrictive this assumption is in practice? For example, by providing some examples of applications where this would hold.\n\n3. In the appendix, Theorem 3.1. is also generalised to the non-Gaussian case, however I don’t think this result is mentioned in the main paper. I think it would be nice to include this somewhere; if not the whole statement then perhaps a sentence linking to the result in the appendix.\n\n4. The experiments provided are quite toy, consisting of experiments on Brownian sheets and the heat equation. However, I only see this as a minor weakness, since the main objective of the paper is theoretical and in my view the experiments serve more as an illustration.\n\n5. There’s a small mistake in Line 318, which currently reads “…about nonlinear inverse problem?”. This is minor, and does not affect my judgment of the paper.\n\n\nIn summary, I think this is a nice paper with interesting results and I recommend acceptance. However, I’m currently a little unsure on how different the results are to that in Theorem 14 of Pidstrigach et al. If the authors could make this clearer I would be willing to improve my score.\n\nQUESTIONS:\nSee the weaknesses for questions.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}