Title: Conditional score-based diffusion models for Bayesian inference in infinite dimensions
Venue: NeurIPS 2023
Accepted: yes
Keywords: score-based generative models

diffusion models

inverse problems

bayesian inference

infinite dimensions
Abstract: Since their initial introduction, score-based diffusion models (SDMs) have been successfully applied to solve a variety of linear inverse problems in finite-dimensional vector spaces due to their ability to efficiently approximate the posterior distribution. However, using SDMs for inverse problems in infinite-dimensional function spaces has only been addressed recently, primarily through methods that learn the unconditional score. While this approach is advantageous for some inverse problems, it is mostly heuristic and involves numerous computationally costly forward operator evaluations during posterior sampling. To address these limitations, we propose a theoretically grounded method for sampling from the posterior of infinite-dimensional Bayesian linear inverse problems based on amortized conditional SDMs. In particular, we prove that one of the most successful approaches for estimating the conditional score in finite dimensions—the conditional denoising estimator—can also be applied in infinite dimensions. A significant part of our analysis is dedicated to demonstrating that extending infinite-dimensional SDMs to the conditional setting requires careful consideration, as the conditional score typically blows up for small times, contrarily to the unconditional score. We conclude by presenting stylized and large-scale numerical examples that validate our approach, offer additional insights, and demonstrate that our method enables large-scale, discretization-invariant Bayesian inference.

Conditional score-based diffusion models for Bayesian inference in infinite dimensions

Abstract

Since their initial introduction, score-based diffusion models (SDMs) have been successfully applied to solve a variety of linear inverse problems in finite-dimensional vector spaces due to their ability to efficiently approximate the posterior distribution. However, using SDMs for inverse problems in infinite-dimensional function spaces has only been addressed recently, primarily through methods that learn the unconditional score. While this approach is advantageous for some inverse problems, it is mostly heuristic and involves numerous computationally costly forward operator evaluations during posterior sampling. To address these limitations, we propose a theoretically grounded method for sampling from the posterior of infinite-dimensional Bayesian linear inverse problems based on amortized conditional SDMs. In particular, we prove that one of the most successful approaches for estimating the conditional score in finite dimensions-the conditional denoising estimator-can also be applied in infinite dimensions. A significant part of our analysis is dedicated to demonstrating that extending infinite-dimensional SDMs to the conditional setting requires careful consideration, as the conditional score typically blows up for small times, contrarily to the unconditional score. We conclude by presenting stylized and large-scale numerical examples that validate our approach, offer additional insights, and demonstrate that our method enables large-scale, discretization-invariant Bayesian inference.

1 Introduction

Inverse problems seek to estimate unknown parameters using noisy observations or measurements. One of the main challenges is that they are often ill-posed. A problem is ill-posed if there are no solutions, or there are many (two or more) solutions, or the solution is unstable in relation to small errors in the observations [1]. A common approach to transform the original ill-posed problem into a well-posed one is to formulate it as a least-squares optimization problem that minimizes the difference between observed and predicted data. However, minimization of the data misfit alone negatively impacts the quality of the obtained solution due to the presence of noise in the data and the inherent nullspace of the forward operator [2, 3]. Casting the inverse problem into a Bayesian probabilistic framework allows, instead, for a full characterization of all the possible solutions [4-6]. The Bayesian approach consists of putting a prior probability distribution describing uncertainty in the parameters of interest, and finding the posterior distribution over these parameters [7]. The prior must be chosen appropriately in order to mitigate the ill-posedness of the problem and facilitate computation of the posterior. By adopting the Bayesian formulation, rather than finding one single solution to the inverse

Josselin Garnier

Maarten V. de Hoop

problem (e.g., the maximum a posteriori estimator [8]), a distribution of solutions-the posterior-is finally obtained, whose samples are consistent with the observed data. The posterior distribution can then be sampled to extract statistical information that allows for uncertainty quantification [9].

Over the past few years, deep learning-based methods have been successfully applied to analyze linear inverse problems in a Bayesian fashion. In particular, recently introduced score-based diffusion models (SDMs) [10] have become increasingly popular, due to their ability of producing approximating samples from the posterior distribution [11, 12]. An SDM consists of a diffusion process, which gradually perturbs the data distribution toward a tractable distribution according to a prescribed stochastic differential equation (SDE) by progressively injecting Gaussian noise, and a generative model, which entails a denoising process defined by approximating the time-reversal of the diffusion. Crucially, the denoising stage is also a diffusion process [13] whose drift depends on the logarithmic gradients of the noised data densities-the scores-which are estimated by Song et al. [10] using a neural network. Among the advantages of SDMs over other deep generative models is that they produce high-quality samples, matching the performance of generative adversarial networks [14], without suffering from training instabilities and mode-collapse [10, 15]. Additionally, SDMs are not restricted to invertible architectures like normalizing flows [16], which often limits the complexity of the distributions that can be learned. Finally, and most importantly to the scope of this work, SDMs have demonstrated superior performance in a variety of inverse problems, such as image inpainting [10, 17], image colorization [10], compressing sensing, and medical imaging [12, 18].

In the aforementioned cases, SDMs have been applied by assuming that the data distribution of interest is supported on a finite-dimensional vector space. However, in many inverse problems, especially those governed by partial differential equations (PDEs), the unknown parameters to be estimated are functions (e.g., coefficient functions, boundary and initial conditions, or source functions) that exist in a suitable function space, typically an infinite-dimensional Hilbert space. The inverse heat equation or the elliptic inverse source problem presented in [19] are typical examples of ill-posed inverse problems that are naturally formulated in infinite-dimensional Hilbert spaces. In addition to these PDE-based examples, other interesting cases that are not PDE-based include geometric inverse problems (e.g., determining the Riemann metric from geodesic information or the background velocity map from travel time information in geophysics [20]) and inverse problems involving singular integral operators [21]. A potential solution for all of these problems could be to discretize the input and output functions into finite-dimensional vectors and apply SDMs to sample from the posterior. However, theoretical studies of current diffusion models suggest that performance guarantees do not generalize well on increasing dimensions [22-24]. This is precisely why Andrew Stuart's guiding principle to study a Bayesian inverse problem for functions-'avoid discretization until the last possible moment' [5]-is critical to the use of SDMs.

Motivated by Stuart's principle, in this work we define a conditional score in the infinite-dimensional setting , a critical step for studying Bayesian inverse problems directly in function spaces through SDMs. In particular, we show that using this newly defined score as a reverse drift of the diffusion process yields a generative stage that samples, under specified conditions, from the correct target conditional distribution. We carry out the analysis by focusing on two cases: the case of a Gaussian prior measure and the case of a general class of priors given as a density with respect to a Gaussian measure. Studying the model for a Gaussian prior measure provides illuminating insight, not only because it yields an analytic formula of the score, but also because it gives a full characterization of SDMs in the infinite-dimensional setting, showing under which conditions we are sampling from the correct target conditional distribution and how fast the reverse SDE converges to it. It also serves as a guide for the analysis in the case of a general class of prior measures. Finally, we conclude this work by presenting, in Section 6, stylized and large-scale numerical examples that demonstrate the applicability of our SDM. Specifically, we show that our SDM model (i) is able to approximate non-Gaussian multi-modal distributions , a challenging task that poses difficulties for many generative models [25]; (ii) is discretization-invariant , a property that is a consequence of our theoretical and computational framework being built on the infinite-dimensional formulation proposed by Stuart [5]; and (iii) is applicable to solve large-scale Bayesian inverse problems , which we demonstrate by applying it to a large-scale problem in geophysics, i.e., the linearized wave-equation-based imaging via the Born approximation that involves estimating a 256 × 256 -dimensional unknown parameter.

Related works Our work is primarily motivated by Andrew Stuart's comprehensive mathematical theory for studying PDE-governed inverse problems in a Bayesian fashion [5]. In particular, we are

interested in the infinite-dimensional analysis [7, 26], which emphasizes the importance of analyzing PDE-governed inverse problems directly in function space before discretization.

Our paper builds upon a rich and ever expanding body of theoretical and applied works dedicated to SDMs. Song et al. [10] defined SDMs integrating both score-based (Hyvärinen [27]; Song and Ermon [17]) and diffusion (Sohl-Dickstein et al. [28]; Ho et al. [29]) models into a single continuous-time framework based on stochastic differential equations. The generative stage in SDMs is based on a result from Anderson [13] proving that the denoising process is also a diffusion process whose drift depends on the scores. This result holds only in vector spaces , which explains the difficulties to extend SDMs to more general function spaces. Initially, there have been attempts to project the input functions into a finite-dimensional feature space and then apply SDMs (Dupont et al. [30]; Phillips et al. [31]). However, these approaches are not discretization-invariant . It is only very recently that SDMs have been directly studied in function spaces, specifically infinite-dimensional Hilbert spaces. Kerrigan et al. [32] generalized diffusion models to operate directly in function spaces, but they did not consider the time-continuous limit based on SDEs (Song et al. [10]). Dutordoir et al. [33] proposed a denoising diffusion generative model for performing Bayesian inference of functions. Lim et al. [34] generalized score matching for trace-class noise corruptions that live in the Hilbert space of the data. However, as Kerrigan et al. [32] and Dutordoir et al. [33], they did not investigate the connection to the forward and backward SDEs as Song et al. [10] did in finite dimensions. Three recent works, Pidstrigach et al. [24], Franzese et al. [35] and Lim et al. [36], finally established such connection for the unconditional setting . In particular, Franzese et al. [35] used results from infinite-dimensional SDEs theory (Föllmer and Wakolbinger [37]; Millet et al. [38]) close to Anderson [13].

Among the mentioned works, Pidstrigach et al. [24] is the closest to ours. We adopt their formalism to establish theoretical guarantees for sampling from the conditional distribution. Another crucial contribution comes from Batzolis et al. [39], as we build upon their proof to show that the score can be estimated by using a denoising score matching objective conditioned on the observed data [17, 40]. A key element in Pidstrigach et al. [24], emphasized also in our analysis, is obtaining an estimate on the expected square norm of the score that needs to be uniform in time . We explicitly compute the expected square norm of the conditional score in the case of a Gaussian prior measure, which shows that a uniform in time estimate is not always possible in the conditional setting . This is not surprising, given that the singularity in the conditional score as noise vanishes is a well-known phenomenon in finite dimensions and has been investigated in many works, both from a theoretical and a practical standpoint [41, 42]. In our paper, we provide a set of concrete conditions to be satisfied to ensure a uniform estimate in time for a general class of prior measures in infinite dimensions.

Pidstrigach et al. [24] have also proposed a method for performing conditional sampling, building upon the approach introduced by Song et al. [12] in a finite-dimensional setting. Like our approach, their method can be viewed as a contribution to the literature on likelihood-free, simulation-based inference [43, 44]. Specifically, the algorithm proposed by Pidstrigach et al. [24] relies on a projectiontype approach that incorporates the observed data into the unconditional sampling process via a proximal optimization step to generate intermediate samples consistent with the measuring acquisition process. This allows Pidstrigach et al. [24] to avoid defining the conditional score 1 . While their method has been shown to work well with specific inverse problems, such as medical imaging [12], it is primarily heuristic, and its computational efficiency varies depending on the specific inverse problem at hand. Notably, their algorithm may require numerous computationally costly forward operator evaluations during posterior sampling. Furthermore, their implementation does not fully exploit the discretization-invariance property achieved by studying the problem in infinite dimensions since they employ a UNet to parametrize their score, limiting the evaluation of their score function to the training interval. The novelty of our work is then twofold. First, we provide theoretically grounded guarantees for an approach that is not heuristic and can be implemented such that it is not constrained to the grid on which we trained our network . As a result, we show that we effectively take advantage of the discretization-invariance property achieved by adopting the infinite-dimensional

formulation proposed by Stuart [5]. Second, we perform discretization-invariant Bayesian inference by learning an amortized version of the conditional score . This is done by making the score function depending on the observations. As a result, provided that we have access to high-quality training data, during sampling we can input any new observation that we wish to condition on directly during simulation of the reverse SDE. In this sense, our method is data-driven , as the information about the forward model is implicitly encoded in the data pairs used to learn the conditional score. This addresses a critical gap in the existing literature, as the other approach using infinite-dimensional SDM resorts to projections onto the measurement subspace for sampling from the posterior-a method that not only lacks theoretical interpretation but may also yield unsatisfactory performance due to costly forward operator computations. There are well-documented instances in the literature where amortized methods can be a preferred option in Bayesian inverse problems [45-50], as they reduce inference computational costs by incurring an offline initial training cost for a deep neural network that is capable of approximating the posterior for unseen observed data, provided that one has access to a set of data pairs that adequately represent the underlying joint distribution.

Main contributions The main contribution of this work is the analysis of conditional SDMs in infinite-dimensional Hilbert spaces . More specifically,

We introduce the conditional score in an infinite-dimensional setting (Section 3).

We provide a comprehensive analysis of the forward-reverse conditional SDE framework in the case of a Gaussian prior measure . We explicitly compute the expected square norm of the conditional score, which shows that a uniform in time estimate is not always possible for the conditional score . We prove that as long as we start from the invariant distribution of the diffusion process, the reverse SDE converges to the target distribution exponentially fast (Section 4).

We provide a set of conditions to be satisfied to ensure a uniform in time estimate for a general class of prior measures that are given as a density with respect to a Gaussian measure . Under these conditions, the conditional score-used as a reverse drift of the diffusion process in SDMs-yields a generative stage that samples from the target conditional distribution (Section 5).

We prove that the conditional score can be estimated via a conditional denoising score matching objective in infinite dimensions (Section 5).

We present examples that validate our approach, offer additional insights, and demonstrate that our method enables large-scale , discretization-invariant Bayesian inference (Section 6).

2 Background

Here, we review the definition of unconditional score-based diffusion models (SDMs) in infinitedimensional Hilbert spaces proposed by Pidstrigach et al. [24], as we will adopt the same formalism to define SDMs for conditional settings. We refer to Appendix A for a brief introduction to key tools of probability theory in function spaces.

Let µ data be the target measure, supported on a separable Hilbert space ( H, ⟨· , ·⟩ ) . Consider a forward infinite-dimensional diffusion process ( X t ) t ∈ [0 ,T ] for continuous time variable t ∈ [0 , T ] , where X 0 is the starting variable and X t its perturbation at time t . The diffusion process is defined by the following SDE:

where C : H → H is a fixed trace class, positive-definite, symmetric covariance operator and W t is a Wiener process on H . Here and throughout the paper, the initial conditions and the driving Wiener processes in (1) are assumed independent.

The forward SDE evolves X 0 ∼ µ 0 towards the Gaussian measure N (0 , C ) as t →∞ . The goal of score-based diffusion models is to convert the SDE in (1) to a generative model by first sampling X T ∼ N (0 , C ) , and then running the correspondent reverse-time SDE. In the finite-dimensional case, Song et al. [10] show that the reverse-time SDE requires the knowledge of the score function ∇ log p t ( X t ) , where p t ( X t ) is the density of the marginal distribution of X t (from now on denoted P t ) with respect to the Lebesgue measure. In infinite-dimensional Hilbert spaces, there is no natural analogue of the Lebesgue measure (for additional details, see [51]) and the density is thus no longer well defined. However, Pidstrigach et al. [24, Lemma 1] notice that, in the finite-dimensional setting

where H = R D , the score can be expressed as follows:

for t > 0 . Since the right-hand side of the expression above is also well-defined in infinite dimensions, Pidstrigach et al. [24] formally define the score as follows:

Definition 1. In the infinite-dimensional setting, the score or reverse drift is defined by

Assuming that the expected square norm of the score is uniformly bounded in time, Pidstrigach et al. [24, Theorem 1] shows that the following SDE

is the time-reversal of (1) and the distribution of Z T is thus equal to µ 0 , proving that the forwardreverse SDE framework of Song et al. [10] generalizes to the infinite-dimensional setting. The reverse SDE requires the knowledge of this newly defined score, and one approach for estimating it is, similarly to [10], by using the denoising score matching loss [40]

where ˜ S ( t, X t ) is typically approximated by training a neural network.

3 The conditional score in infinite dimensions

Analogous to the score function relative to the unconditional SDM in infinite dimensions, we now define the score corresponding to the reverse drift of an SDE when conditioned on observations. We consider a setting where X 0 is an H -valued random variable and H is an infinite-dimensional Hilbert space. Denote by

a noisy observation given by n linear measurements, where the measurement acquisition process is represented by a linear operator A : H → R n , and B ∼ N (0 , C B ) represents the noise, with C B a n × n nonnegative matrix. Within a Bayesian probabilistic framework, solving (6) amounts to putting an appropriately chosen prior probability distribution µ 0 on X 0 , and sampling from the conditional distribution of X 0 given Y = y .

To the best of our knowledge, the only existing algorithm which performs conditional sampling using infinite-dimensional diffusion models on Hilbert spaces is based on the work of Song et al. [12]. The idea, adapted to infinite dimensions by Pidstrigach et al. [24], is to incorporate the observations into the unconditional sampling process of the SDM via a proximal optimization step to generate intermediate samples that are consistent with the measuring acquisition process. Our method relies instead on utilizing the score of infinite-dimensional SDMs conditioned on observed data , which we introduce in this work. We begin by defining the conditional score, by first noticing that, in finite dimensions, we have the following lemma:

Lemma 1. In the finite-dimensional setting where H = R D , we can express the conditional score function for t > 0 as

Since the right-hand side of (7) is well-defined in infinite dimensions, by following the same line of thought of Pidstrigach et al. [24] we formally define the score as follows:

Definition 2. In the infinite-dimensional setting, the conditional score is defined by

Remark 1. It is possible to define the conditional score in infinite-dimensional Hilbert spaces by resorting to the results of [37, 38], see Appendix C.1.

For Definition 2 to make sense, we need to show that if we use (8) as the drift of the time-reversal of the SDE in (1) conditioned on y , then it will sample the correct conditional distribution of X 0 given Y = y in infinite dimensions. In the next sections, we will carry out the analysis by focusing on two cases: the case of a Gaussian prior measure N (0 , C µ ) , and the case where the prior of X 0 is given as a density with respect to a Gaussian measure, i.e.,

where C µ is positive and trace class and Φ is bounded with E µ [ ∥ C µ ∇ H Φ( X 0 ) ∥ 2 ] < + ∞ .

4 Forward-reverse conditional SDE framework for a Gaussian prior measure

We begin our analysis of the forward-reverse conditional SDE framework by examining the case where the prior of X 0 is a Gaussian measure. This case provides illuminating insight, not only because it is possible to get an analytic formula of the score, but also because it offers a full characterization of SDMs in the infinite-dimensional setting, showing under which conditions we are sampling from the correct target conditional distribution and how fast the reverse SDE converges to it. We also show that the conditional score can have a singular behavior at small times when the observations are noiseless, in contrast with the unconditional score under similar hypotheses.

We assume that Φ = 0 in (9). All distributions in play are Gaussian:

where M o = C µ A ∗ ( AC µ A ∗ + C B ) -1 and C o = C µ -C µ A ∗ ( AC µ A ∗ + C B ) -1 AC µ . By Mercer theorem [52], there exist ( µ j ) in [0 , + ∞ ) and an orthonormal basis ( v j ) in H such that C µ v j = µ j v j ∀ j . We consider the infinite-dimensional case with µ j > 0 ∀ j . We assume that C µ is trace class so that ∑ j µ j < + ∞ . We assume that the functions v j are eigenfunctions of C and we denote by λ j the corresponding eigenvalues.

We assume an observational model corresponding to observing a finite-dimensional subspace of H spanned by v η (1) , . . . , v η ( n ) corresponding to g k = v η ( k ) , k = 1 , . . . , n , where g j ∈ H is such that ( Af ) j = ⟨ g j , f ⟩ . We denote I ( n ) = { η (1) , . . . , η ( n ) } . We assume moreover C B = σ 2 B I n . Let Z t be the solution of reverse-time SDE:

We want to show that the reverse SDE we have just formulated in (14) indeed constitutes a reversal of the stochastic dynamics from the forward SDE in (1) conditioned on y . To this aim, we will need the following lemma:

Lemma 2. We define Z ( j ) = ⟨ v j , Z ⟩ , p ( j ) = λ j /µ j for all j . We also define y ( j ) = y η ( j ) for j ∈ I ( n ) and y ( j ) = 0 otherwise, and q ( j ) = µ j /σ 2 B for j ∈ I ( n ) and q ( j ) = 0 otherwise. Then we can write for all j

with W ( j ) independent and identically distributed standard Brownian motions,

Proof. The proof is a Gaussian calculation. It relies on computing ⟨ S, v j ⟩ , which yields an analytic formula. See Appendix B.

Lemma 2 enables us to discuss when we are sampling from the correct target conditional distribution X 0 | Y ∼ N ( M o Y, C o ) . We can make a few remarks:

In the limit T →∞ , we get µ ( x,j ) ( T -t ) →-1 / 2 and µ ( y,j ) ( T -t ) → 0 .

If j / ∈ I ( n ) then we have the same mode dynamics as in the unconditional case. Thus we sample from the correct target distribution if T is large or if we start from Z ( j ) 0 ∼ N (0 , Σ ( j ) 0 ) for Σ ( j ) 0 = µ j e -T + λ j (1 -e -T ) , which is the distribution of X ( j ) T = ⟨ X T , v j ⟩ given Y = y .

If j ∈ I ( n ) and we start from Z ( j ) 0 ∼ N (¯ z ( j ) 0 , Σ ( j ) 0 ) , then we find Z ( j ) T ∼ N ( ¯ z ( j ) T , Σ ( j ) T ) with

The distribution of X ( j ) 0 = ⟨ X 0 , v j ⟩ given Y = y is N ( y ( j ) q ( j ) / (1 + q ( j ) ) , µ j / (1 + q ( j ) ) . As ¯ z ( j ) T → y ( j ) q ( j ) / (1 + q ( j ) ) and Σ ( j ) T → µ j / (1 + q ( j ) ) as T → + ∞ , this shows that we sample from the exact target distribution (the one of X 0 given Y = y ) for T large.

If we start the reverse-time SDE from the correct model

then indeed Z ( j ) T ∼ N ( y ( j ) q ( j ) / (1 + q ( j ) ) , µ j / (1 + q ( j ) )) . This shows that, for any T , Z T has the same distribution as X 0 given Y = y , which is the exact target distribution. We can show similarly that Z T -t has the same distribution as X t given Y = y for any t ∈ [0 , T ] .

In the case that σ B = 0 so that we observe the mode values perfectly for j ∈ I ( n ) , then

and indeed lim t ↑ T Z ( j ) t = y ( j ) a.s. Indeed the t -1 singularity at the origin drives the process to the origin like in the Brownian bridge.

Our analysis shows that, as long as we start from the invariant distribution of the diffusion process, we are able to sample from the correct target conditional distribution and that happens exponentially fast. This proves that the score of Definition 2 is the reverse drift of the SDE in (14). Additionally, the analysis shows that the score is uniformly bounded, except when there is no noise in the observations, blowing up near t = 0 .

Remark 2. Note that, for q ( j ) = 0 , we obtain the unconditional model:

If C = C µ , the square expectation of the norm and the Lipschitz constant of the score are uniformly bounded in time: sup j,t ∈ [0 ,T ] | µ ( j ) ( t ) | = 1 / 2 .

Proposition 1. The score is S ( t, x, y ) = ∑ j S ( j ) G ( t, ⟨ x, v j ⟩ , y ( j ) ) v j , S ( j ) G ( t, x ( j ) , y ( j ) ) = ( µ ( x,j ) ( T -t ) -1 / 2 ) x ( j ) + µ ( y,j ) ( T -t ) y ( j ) and it satisfies

Proof. The proof is a Gaussian calculation given in Appendix B.

In the unconditional setting, we have E [ ∥ S ( t, X t ) ∥ 2 H ] = ∑ j e t 1+( e t -1) p ( j ) λ 2 j µ j which is equal to ∑ j λ j when C = C µ . It is indeed uniformly bounded in time. In the conditional and noiseless setting ( σ B = 0 ), we have E [ ∥ S ( t, X t , y ) ∥ 2 H | Y = y ] = ∑ j ̸∈I ( n ) e t 1+( e t -1) p ( j ) λ 2 j µ j + ∑ j ∈I ( n ) λ j 1 -e -t , which blows up as 1 /t as t → 0 . This result shows that the extension of the score-based diffusion models to the conditional setting is not trivial.

5 Well-posedness for the reverse SDE for a general class of prior measures

We are now ready to consider the case of a general class of prior measures given as a density with respect to a Gaussian measure. The analysis of this case resembles the one of Pidstrigach et al. [24] for the unconditional setting. The main challenge is the singularity of the score for small times, an event that in the Gaussian case was observed in the noiseless setting. In this section we will provide a set of conditions to be satisfied by Φ in (9), so that the conditional score is bounded uniformly in time. The existence of this bound is needed to make sense of the forward-reverse conditional SDE, and to prove the accuracy and stability of the conditional sampling.

We start the analysis by recalling that, in the infinite-dimensional case, the conditional score is (8). It is easy to get a first estimate:

The proof follows from Jensen inequality and the law of total expectation, see Appendix C. Note that (23) is indeed an upper bound of (22) since Tr( C ) = ∑ j λ j .

Note that the bound (23) is also valid for the unconditional score S ( t, x ) = -(1 -e -t ) -1 ( x -e -t/ 2 E [ X 0 | X t = x ] ) . We can observe that the upper bound (23) blows up in the limit of small times. We can make a few comments:

The bound (23) is convenient for positive times, but the use of Jensen's inequality results in a very crude bound for small times. As shown in the previous section, we know that there exists a bound (21) for the unconditional score in the Gaussian case that is uniform in time.

The singular behavior as 1 /t at small time t is, however, not artificial. Such a behavior is needed in order to drive the state to the deterministic initial condition when there are exact observations. This behavior has been exhibited by (20) and (22) in the Gaussian case when σ B = 0 . This indicates that the following assumption (24) is not trivial in the conditional setting.

For Definition 2 to make sense in the more general case where the prior of X 0 is given as a density with respect to a Gaussian measure, we will need to make the following assumption.

Assumption 1. For any y ∈ R

We are now ready to state the analogous result to Pidstrigach et al. [24, Theorem 1].

Proposition 2. Under Assumption 1, the solution of the reverse-time SDE

satisfies Z T ∼ X 0 | Y = y .

Proof. Given Assumption 1, the proof follows the same steps as the one given in [24] for the unconditional score. See Appendix C for the full proof.

Assumption 1 is satisfied under some appropriate conditions. In the following proposition, we provide a set of conditions that ensure the satisfaction of this assumption. It shows that it is possible to get an upper bound in (23) that is uniform in time provided some additional conditions are fulfilled.

Proposition 3. We assume that C µ in (9) and C in (1) have the same basis of eigenfunctions ( v j ) and we define X ( j ) t = ⟨ X t , v j ⟩ and S ( j ) ( t, x, y ) = ⟨ S ( t, x, y ) , v j ⟩ so that in (1) S ( t, x, y ) = ∑ j S ( j ) ( t, x, y ) v j . We assume an observational model as described in Section 4 and that the p ( j ) (1 + q ( j ) ) are uniformly bounded with respect to j and that C is of trace class. We make a modified version of assumption in (9) as follows. We assume that 1) the conditional distribution of X 0 given Y = y is absolutely continuous with respect to the Gaussian measure µ with a RadonNikodym derivative proportional to exp( -Ψ( x 0 , y )) ; 2) we have Ψ( x 0 , y ) = ∑ j Ψ ( j ) ( x ( j ) 0 , y ) ,

where K and L do not depend on j . Then Assumption 1 holds true.

To use the new score function of Definition 2 for sampling from the posterior, we need to define a way to estimate it. In other words, we need to define a loss function over which the difference between the true score and a neural network s θ ( t, x t , y ) is minimized in θ . A natural choice for the loss function is

however it cannot be minimized directly since we do not have access to the ground truth conditional score S ( t, x t , y ) . Therefore, in practice, a different objective has to be used. Batzolis et al. [39] proved that, in finite dimensions, a denoising score matching loss can be used:

This expression involves only ∇ x t log p ( x t | x 0 ) which can be computed analytically from the transition kernel of the forward diffusion process, also in infinite dimensions. In the following proposition, we build on the arguments of Batzolis et al. [39] and provide a proof that the conditional denoising estimator is a consistent estimator of the conditional score in infinite dimensions.

Proposition 4. Under Assumption 1, the minimizer in θ of

is the same as the minimizer of

The same result holds if we add t ∼ U (0 , T ) in the expectations.

Proof. The proof combines some of the arguments of Batzolis et al. [39] and steps of the proof of Lemma 2 in [24], see Appendix C.

Remark 3. A statement of robustness can be written as in [24, Theorem 2].

6 Numerical experiments

To put the presented theoretical results into practice, we provide two examples. The first stylized example aims at showcasing (i) the ability of our method in capturing nontrivial conditional distributions; and (ii) the discretization-invariance property of the learned conditional SDM. In the second example, we sample from the posterior distribution of a linearized seismic imaging problem in order to demonstrate the applicability of our method to large-scale problems. In both examples, in order to enable learning in function spaces, we parameterize the conditional score using Fourier neural operators [53]. Details regarding our experiment and implementation 2 are presented at Appendix D.

Stylized example Inspired by Phillips et al. [31], we define the target density via the relation x 0 = ay 2 + ε with ε ∼ Γ(1 , 2) , a ∼ U{1 , 1 } , and y ∈ [ -3 , 3] . Figure 1a illustrates the samples x 0 evaluated on a fine y grid. After training (details in Appendix D), we sample the conditional distribution on uniformly sampled grids between [ -3 , 3] , each having 20 to 40 grid points. Figures 1b and 1c show the predicted samples for grid sizes of 25 and 35 , respectively. The marginal conditionals associated with y = -1 . 0 , 0 . 0 , 0 . 5 are shown in Figures 1d-1f, respectively. The gray shaded density in the bottom row of Figure 1 indicates the ground truth density, and colored estimated densities correspond to different discretizations of the horizontal axis. The visual inspection of samples and estimated densities indicates that our approach is indeed discretization-invariant.

Linearized seismic imaging example In this experiment, we address the problem of estimating the short-wavelength component of the Earth's subsurface squared-slowness model (i.e., seismic image; cf. Figure 2a) given surface measurements and a long-wavelength, smooth squared-slowness model (cf. Figure 2b). Following Orozco et al. [54], in order to reduce the high dimensionality of surface measurements, we apply the adjoint of the forward operator, the Born scattering operator, to the measurements and use the outcome (cf. Figure 2c) instead of measured data to condition the SDM. After training, given previously unseen observed data, we use the SDM to sample 10 3 posterior

(a)

(b)

(c)

(d)

(e)

(f)

True functions

Predicted functions

Predicted functions

5

5

5

0

0

0

-5

-5

grid size 25

-5

grid size 35

-3

2

1

0

1

2

3

3

2

1

0

1

2

3

-3

2

1

0

1

2

3

y=-1.00

2

y=0.50

function

0.6

True density

20

Density function

True density

20

function

1.0

True density

20

Density

0.4

0.2

25

30

35

40

7

40

25

30

35

Density

0.5

25

30

35

40

0.0

0

0.0

-1

0

1

2

3

4

1

0

1

2

3

4

1

0

1

2

3

4

(a)

(b)

(c)

(d)

(e)

(f)

0.0

Ground-truth(unknown)image

x103

1.5

0.0

Smoothbackgroundmodel

X10-1

0.0

Reverse-time migrated image

X103

0.5

1.0

0.5

0.5

Depth (km)

1.0

0.5

Depth (km)

1.0

1.5

Depth (km)

1.0

5

1.5

0.0

1.5

1.0

1.5

0

2.0

2.0

2.0

0.5

2.5

2.5

0.5

2.5

-5

-1.0

3.0

3.0

3.0

0

1

2

3

4

5

1.5

0

1

2

3

4

5

0.0

0

1

2

3

4

5

Horizontal distance (km)

Horizontal distance (km)

Horizontal distance (km)

0.0

Conditionalmeanestimate

X103

1.5

0.0

Pointwisestandarddeviation

x101

9.0

0.0

Predictionerror

X102

1.2

0.5

1.0

0.5

0.5

1.0

Depth (km)

1.0

0.5

Depth (km)

1.0

Depth (km)

1.0

1.5

1.5

1.5

0.8

0.0

2.0

2.0

2.0

0.6

-0.5

2.5

-1.0

2.5

1.0

2.5

0.4

3.0

3.0

3.0

0.2

1.5

0.6

0

1

2

3

4

5

0

1

2

3

4

5

0

1

2

3

4

5

Horizontal distance (km)

Horizontal distance (km)

Horizontal distance (km)

samples to estimate the conditional mean (cf. Figure 2d), which corresponds to the minimum-variance estimate [55], and the pointwise standard deviation (cf. Figure 2e), which we use to quantify the uncertainty. As expected, the pointwise standard deviation highlights areas of high uncertainty, particularly in regions with complex geological structures-such as near intricate reflectors and areas with limited illumination (deep and close to boundaries). We also observe a strong correlation between the pointwise standard deviation and the error in the conditional mean estimate (Figure 2f), confirming the accuracy of our Bayesian inference method.

7 Conclusions

Weintroduced a theoretically-grounded method that is able to perform conditional sampling in infinitedimensional Hilbert (function) spaces using score-based diffusion models . This is a foundational step in using diffusion models to perform Bayesian inference. To achieve this, we learned the infinite-dimensional score function, as defined by Pidstrigach et al. [24], conditioned on the observed data. Under mild assumptions on the prior, this newly defined score-used as the reverse drift of the diffusion process-yields a generative model that samples from the posterior of a linear inverse problem. In particular, the well-known singularity in the conditional score for small times can be avoided. Building on these results, we presented stylized and large-scale examples that showcase the validity of our method and its discretization-invariance, a property that is a consequence of our theoretical and computational framework being built on infinite-dimensional spaces.

Acknowledgments

JG was supported by Agence de l'Innovation de Défense - AID - via Centre Interdisciplinaire d'Etudes pour la Défense et la Sécurité - CIEDS - (project 2021 - PRODIPO). LB, AS, and MVdH acknowledge support from the Simons Foundation under the MATH + X program, the Department of Energy under grant DE-SC0020345, and the corporate members of the Geo-Mathematical Imaging Group at Rice University. KS was supported by Air Force Office of Scientific Research under grant FA9550-22-1-0176 and the National Science Foundation under grant DMS-2308389.

References

J. Hadamard. Lectures on the Cauchy's Problem in Linear Partial Differential Equations . Yale University Press, 1923.

R. C. Aster, B. Borchers, and C. H. Thurber. Parameter estimation and inverse problems . Elsevier, 2018.

K. Ito and B. Jin. Inverse Problems . World Scientific, 2014.

M. S. Lehtinen, L. Paivarinta, and E. Somersalo. Linear inverse problems for generalised random variables. Inverse Problems , 5(4):599, 1989.

A. M. Stuart. Inverse problems: A Bayesian perspective. Acta Numerica , 19:451-559, 2010.

A. Tarantola. Inverse problem theory and methods for model parameter estimation . SIAM, 2005.

B. T. Knapik, A. W. van der Vaart, and J. H. van Zanten. Bayesian inverse problems with Gaussian priors. The Annals of Statistics , 39(5):2626 - 2657, 2011.

A. F. Bennett. Inverse Modeling of the Ocean and Atmosphere . Cambridge University Press, 2002.

A. Stuart. Uncertainty quantification in Bayesian inversion. In 2014 SIAM Conference on Uncertainty Quantification , 2014.

Y. Song, J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon, and B. Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations , 2021.

B. Kawar, G. Vaksman, and M. Elad. SNIPS: Solving noisy inverse problems stochastically. In Advances in Neural Information Processing Systems , volume 34, pages 21757-21769, 2021.

Y. Song, L. Shen, L. Xing, and S. Ermon. Solving inverse problems in medical imaging with score-based generative models. In International Conference on Learning Representations , 2022.

B. D. Anderson. Reverse-time diffusion equation models. Stochastic Processes and their Applications , 12(3):313-326, 1982.

P. Dhariwal and A. Nichol. Diffusion models beat GANs on image synthesis. In Advances in Neural Information Processing Systems , volume 34, pages 8780-8794, 2021.

D. Gnaneshwar, B. Ramsundar, D. Gandhi, R. Kurchin, and V. Viswanathan. Score-based generative models for molecule generation, 2022. URL https://arxiv.org/abs/2203. 04698 . Preprint.

H. Lee, J. Lu, and Y. Tan. Convergence of score-based generative modeling for general data distributions. In Proceedings of The 34th International Conference on Algorithmic Learning Theory , volume 201, pages 946-985, 2023.

Y. Song and S. Ermon. Generative modeling by estimating gradients of the data distribution. In Advances in Neural Information Processing Systems , volume 32, 2019.

A. Jalal, M. Arvinte, G. Daras, E. Price, A. G. Dimakis, and J. Tamir. Robust compressed sensing MRI with deep generative priors. Advances in Neural Information Processing Systems , 34:14938-14954, 2021.

M. Dashti and A. M. Stuart. The Bayesian approach to inverse problems , pages 311-428. Springer International Publishing, 2017.

G. Uhlmann and A. Vasy. The inverse problem for the local geodesic ray transform. Inventiones Mathematicae , 205(1):83-120, 2016.

A. Dynin. Inversion problem for singular integral operators: C ∗ -approach. Proceedings of the National Academy of Sciences , 75(10):4668-4670, 1978.

S. Chen, S. Chewi, J. Li, Y . Li, A. Salim, and A. Zhang. Sampling is as easy as learning the score: Theory for diffusion models with minimal data assumptions. In The Eleventh International Conference on Learning Representations , 2023.

V. D. Bortoli. Convergence of denoising diffusion models under the manifold hypothesis. Transactions on Machine Learning Research , 2022.

J. Pidstrigach, Y. Marzouk, S. Reich, and S. Wang. Infinite-dimensional diffusion models for function spaces, 2023. URL https://arxiv.org/abs/2302.10130v1 . Preprint.

S. Arora, A. Risteski, and Y . Zhang. Do GANs learn the distribution? Some theory and empirics. In International Conference on Learning Representations , 2018.

S. Lasanen. Measurements and infinite-dimensional statistical inverse theory. Proceedings in Applied Mathematics and Mechanics , 7:1080101-1080102, 2007.

A. Hyvärinen. Estimation of non-normalized statistical models by score matching. Journal of Machine Learning Research , 6(24):695-709, 2005.

J. Sohl-Dickstein, E. Weiss, N. Maheswaranathan, and S. Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In Proceedings of the 32nd International Conference on Machine Learning , volume 37, pages 2256-2265, 2015.

J. Ho, A. Jain, and P. Abbeel. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems , volume 33, pages 6840-6851, 2020.

E. Dupont, H. Kim, S. M. A. Eslami, D. J. Rezende, and D. Rosenbaum. From data to functa: Your data point is a function and you can treat it like one. In Proceedings of the 39th International Conference on Machine Learning , volume 162, pages 5694-5725, 2022.

A. Phillips, T. Seror, M. J. Hutchinson, V. D. Bortoli, A. Doucet, and E. Mathieu. Spectral diffusion processes. In NeurIPS 2022 Workshop on Score-Based Methods , 2022.

G. Kerrigan, J. Ley, and P. Smyth. Diffusion generative models in infinite dimensions. In Proceedings of The 26th International Conference on Artificial Intelligence and Statistics , volume 206, pages 9538-9563, 2023.

V. Dutordoir, A. Saul, Z. Ghahramani, and F. Simpson. Neural diffusion processes. In Proceedings of the 40th International Conference on Machine Learning , volume 202, pages 8990-9012, 2023.

J. H. Lim, N. B. Kovachki, R. Baptista, C. Beckham, K. Azizzadenesheli, J. Kossaifi, V. Voleti, J. Song, K. Kreis, J. Kautz, C. Pal, A. Vahdat, and A. Anandkumar. Score-based diffusion models in function space, 2023. URL https://arxiv.org/abs/2302.07400 . Preprint.

G. Franzese, G. Corallo, S. Rossi, M. Heinonen, M. Filippone, and P. Michiardi. Continuoustime functional diffusion processes. In Thirty-seventh Conference on Neural Information Processing Systems , 2023. URL https://openreview.net/forum?id=VPrir0p5b6 .

S. Lim, E. Yoon, T. Byun, T. Kang, S. Kim, K. Lee, and S. Choi. Score-based generative modeling through stochastic evolution equations in hilbert spaces. In Thirty-seventh Conference on Neural Information Processing Systems , 2023. URL https://openreview.net/forum? id=GrElRvXnEj .

H. Föllmer and A. Wakolbinger. Time reversal of infinite-dimensional diffusions. Stochastic Processes and their Applications , 22(1):59-77, 1986.

A. Millet, D. Nualart, and M. Sanz. Time reversal for infinite-dimensional diffusions. Probability Theory and Related Fields , 82(3):315-347, 1989.

G. Batzolis, J. Stanczuk, C.-B. Schönlieb, and C. Etmann. Conditional image generation with score-based diffusion models, 2021. URL https://arxiv.org/abs/2111.13606 . Preprint.

P. Vincent. A connection between score matching and denoising autoencoders. Neural Computation , 23(7):1661-1674, 2011.

D. Kim, S. Shin, K. Song, W. Kang, and I.-C. Moon. Soft truncation: A universal training technique of score-based diffusion model for high precision score estimation. In Proceedings of the 39th International Conference on Machine Learning , volume 162, pages 11201-11228, 2022.

T. Dockhorn, A. Vahdat, and K. Kreis. Score-based generative modeling with critically-damped Langevin diffusion. In International Conference on Learning Representations , 2022.

K. Cranmer, J. Brehmer, and G. Louppe. The frontier of simulation-based inference. Proceedings of the National Academy of Sciences , 117(48):30055-30062, 2020.

A. Lavin, D. Krakauer, H. Zenil, J. Gottschlich, T. Mattson, J. Brehmer, A. Anandkumar, S. Choudry, K. Rocki, A. G. Baydin, C. Prunkl, B. Paige, O. Isayev, E. Peterson, P. L. McMahon, J. Macke, K. Cranmer, J. Zhang, H. Wainwright, A. Hanuka, M. Veloso, S. Assefa, S. Zheng, and A. Pfeffer. Simulation intelligence: Towards a new generation of scientific methods, 2022. URL https://arxiv.org/abs/2112.03235 . Preprint.

R. Baptista, B. Hosseini, N. B. Kovachki, and Y. Marzouk. Conditional sampling with monotone GANs: From generative models to likelihood-free inference, 2023. URL https://arxiv. org/abs/2006.06755 . Preprint.

Y. Kim, S. Wiseman, A. Miller, D. Sontag, and A. Rush. Semi-amortized variational autoencoders. In Proceedings of the 35th International Conference on Machine Learning , volume 80, pages 2678-2687, 2018.

J. Kruse, G. Detommaso, U. Köthe, and R. Scheichl. HINT: Hierarchical invertible neural transport for density estimation and Bayesian inference. Proceedings of the AAAI Conference on Artificial Intelligence , 35(9):8191-8199, 2021.

S. T. Radev, U. K. Mertens, A. Voss, L. Ardizzone, and U. Köthe. BayesFlow: Learning complex stochastic models with invertible neural networks. IEEE Transactions on Neural Networks and Learning Systems , 33(4):1452-1466, 2022.

A. Siahkoohi and F. J. Herrmann. Learning by example: Fast reliability-aware seismic imaging with normalizing flows. In First International Meeting for Applied Geoscience & Energy Expanded Abstracts , pages 1580-1585, 2021.

A. Siahkoohi, G. Rizzuti, R. Orozco, and F. J. Herrmann. Reliable amortized variational inference with physics-based latent distribution correction. Geophysics , 88(3):R297-R322, 2023.

G. Prato. An Introduction to Infinite-Dimensional Analysis . Springer, 2006.

J. Mercer and A. R. Forsyth. XVI. Functions of positive and negative type, and their connection the theory of integral equations. Philosophical Transactions of the Royal Society of London. Series A, Containing Papers of a Mathematical or Physical Character , 209(441-458):415-446, 1909.

Z. Li, N. B. Kovachki, K. Azizzadenesheli, B. liu, K. Bhattacharya, A. Stuart, and A. Anandkumar. Fourier neural operator for parametric partial differential equations. In International Conference on Learning Representations , 2021.

R. Orozco, A. Siahkoohi, G. Rizzuti, T. van Leeuwen, and F. J. Herrmann. Adjoint operators enable fast and amortized machine learning based Bayesian uncertainty quantification. In Medical Imaging 2023: Image Processing , volume 12464, page 124641L, 2023.

B. D. Anderson and J. B. Moore. Optimal Filtering . Prentice-Hall, Englewood Cliffs, NJ, 1979.

D. P. Kingma and J. Ba. Adam: A method for stochastic optimization, 2017. URL https: //arxiv.org/abs/1412.6980 . Preprint.

G. Lambaré, J. Virieux, R. Madariaga, and S. Jin. Iterative asymptotic inversion in the acoustic approximation. Geophysics , 57(9):1138-1154, 1992.

T. Nemeth, C. Wu, and G. T. Schuster. Least-squares migration of incomplete reflection data. Geophysics , 64(1):208-221, 1999.

Veritas. Parihaka 3D marine seismic survey - Acquisition and processing report. Technical Report 3460, New Zealand Petroleum & Minerals, Wellington, 2005.

WesternGeco. Parihaka 3D PSTM final processing report. Technical Report 4582, New Zealand Petroleum & Minerals, Wellington, 2012.

M. Louboutin, M. Lange, F. Luporini, N. Kukreja, P. A. Witte, F. J. Herrmann, P. Velesko, and G. J. Gorman. Devito (V3.1.0): An embedded domain-specific language for finite differences and geophysical exploration. Geoscientific Model Development , 12(3):1165-1187, 2019.

F. Luporini, M. Louboutin, M. Lange, N. Kukreja, P. Witte, J. Hückelheim, C. Yount, P. H. J. Kelly, F. J. Herrmann, and G. J. Gorman. Architecture and performance of Devito, a system for automated stencil computation. ACM Transactions on Mathematical Software , 46(1), 2020.

A Probability measures on infinite-dimensional Hilbert spaces

In this section, we briefly present some fundamental notions related to probability measures on infinite-dimensional spaces, specifically separable Hilbert spaces ( H, ⟨· , ·⟩ ) . There is abundant literature on the subject. For more details we refer to Stuart [5], Pidstrigach et al. [24], Kerrigan et al. [32], Prato [51] and references therein.

A.1 Gaussian measures on Hilbert spaces

Definition 3. Let (Ω , F , P ) be a probability space. A measurable function X : Ω → H is called a Gaussian random element (GRE) if for any h ∈ H , the random variable ⟨ h, X ⟩ has a scalar Gaussian distribution.

Every GRE X has a mean element m ∈ H defined by

and a linear covariance operator C : H → H defined by

We denote X ∼ N ( m,C ) for a GRE in H with mean element m and covariance operator C . It can be shown that the covariance operator of a GRE is trace class, positive-definite and symmetric. Conversely, for any trace class, positive-definite and symmetric linear operator C : H → H and every m ∈ H , there exists a GRE with X ∼ N ( m,C ) . This leads us to the following definition:

Definition 4. If X is a GRE, the pushforward of P through X , denoted by P X , is called a Gaussian probability measure on H . We will write P X = N ( m,C ) .

Let X ∼ N ( m,C ) . We can make a few remarks:

1) For any h ∈ H , we have ⟨ h, X ⟩ ∼ N ( ⟨ h, m ⟩ , ⟨ Ch,h ⟩ ) .

2) C is compact. By Mercer theorem [52] there exists ( λ j ) and an orthonormal basis of eigenfunctions ( v j ) such that λ j ≥ 0 and Cv j = λ j v j ∀ j . We consider the infinite-dimensional case in which λ j > 0 ∀ j .

3) Suppose m = 0 (we call the Gaussian measure of X centered). The expected square norm of X is given by

which is finite since C is trace class.

A.2 Absolutely continuous measures and the Feldman-Hajek theorem

Here we introduce the notion of absolute continuity for measures.

Definition 5. Let µ and ν be two probability measures on H equipped with its Borel σ -algebra B ( H ) . Measure µ is absolutely continuous with respect to ν (we write µ ≪ ν ) if µ (Σ) = 0 for all Σ ∈ B ( H ) such that ν (Σ) = 0 .

Definition 6. If µ ≪ ν and ν ≪ µ then µ and ν are said to be equivalent and we write µ ∼ ν . If µ and ν are concentrated on disjoint sets then they are called singular; in this case we write µ ⊥ ν .

Another notion that will be used throughout the paper is the Radon-Nikodym derivative.

Theorem 1. Let µ and ν be two measures on ( H, B ( H )) and ν be σ -finite. If µ ≪ ν , then there exists a ν -measurable function f on H such that

Furthermore, f is unique ν -a.e. and is called the Radon-Nikodym derivative of µ with respect to ν . It is denoted by dµ/dν .

Remark 4. In the paper, we will sometimes refer to f as the density of µ with respect to ν .

We are finally able to state the Feldman-Hajek theorem in its general form.

Theorem 2. The following statements hold.

Gaussian measures µ = N ( m 1 , C 1 ) , ν = N ( m 2 , C 2 ) are either singular or equivalent.

They are equivalent if and only if the following conditions hold:

(i) ν and µ have the same Cameron-Martin space H 0 = C 1 / 2 1 ( H ) = C 1 / 2 2 ( H ) .

(ii) m 1 -m 2 ∈ H 0 .

(iii) The operator ( C -1 / 2 1 C 1 / 2 2 )( C -1 / 2 1 C 1 / 2 2 ) ∗ -I is a Hilbert-Schmidt operator on the closure H 0 .

If µ and ν are equivalent and C 1 = C 2 = C , then ν -a.s. the Radon-Nikodym derivative dµ/dν is given by

A.3 Bayes' theorem for inverse problems

Let H and K be separable Hilbert spaces, equipped with the Borel σ -algebra, and A : H → K a measurable mapping. We want to solve the inverse problem of finding X from Y , where

and B ∈ K denotes the noise. We adopt a Bayesian approach to this problem. We let ( X,Y ) ∈ H × K be a random variable and compute X | Y . We first specify ( X,Y ) as follows:

Prior: X ∼ µ 0 measure on H .

Noise: B ∼ η 0 measure on K , with B independent from X .

The random variable Y | X is then distributed according to the measure η x , the translate of η 0 by A ( X ) . We assume that η x ≪ η 0 . Thus for some potential Ψ : H × K → R ,

The potential Ψ( · , y ) satisfying the above formula is often termed the negative log likelihood of the problem. Now define ν 0 to be the product measure ν 0 = µ 0 × η 0 . We can finally state the following infinite-dimensional analogue of the Bayes' theorem.

Theorem 3. Assume that Ψ : H × K → R is ν 0 -measurable and define

Then where µ 0 ( ·| Y = y ) is the conditional distribution of X given Y = y .

B Proofs of Section 4

B.1 Proofs of Lemma 2 and Proposition 1

We assume that C µ in (9) and C in (1) have the same basis of eigenfunctions ( v j ) and that C µ v j = µ j v j , Cv j = λ j v j ∀ j . We define X ( j ) t = ⟨ X t , v j ⟩ , y ( j ) = ⟨ y, v j ⟩ and S ( j ) ( t, x, y ) = ⟨ S ( t, x, y ) , v j ⟩ so that in (1) S ( t, x, y ) = ∑ j S ( j ) ( t, x, y ) v j . We assume j ∈ I ( n ) so that we consider a mode corresponding to an observation. We then have

with W ( j ) standard Brownian motions which are independent for the different modes j . Note also that with C µ and C having the same basis of eigenfunctions the system of modes is diagonalized so that the X ( j ) t processes are independent with respect to mode j , both for the observed and un-observed modes. Thus we have

for η ( j ) i independent standard Gaussian random variables. We then seek

which in this Gaussian setting is the L 2 projection of X ( j ) 0 onto X ( j ) t and Y ( j ) . Thus we can write x ( j,y ) 0 = ax ( j ) + by ( j ) with ( a, b ) solving

which gives

for p ( j ) = λ j /µ j , q ( j ) = µ j /σ 2 B .

We then get in view of (8)

Note that with some abuse of notation we then have

which is important since then also the time reversed system diagonalizes. We remark that for an unobserved mode we get by a similar, but easier, calculation

which simply corresponds to setting σ B = ∞ in (31).

Consider next E [ S ( j ) ( t, x, y ) 2 ] . Note first that

We can then easily check that the score is conditionally centered E [ S ( j ) ( t, X t , Y ) | Y = y ] = 0 and we then get

which gives Proposition 1 upon summing over the mode index j , where we define q ( j ) = 0 for the unobserved modes.

C Proofs of Section 5

C.1 Discussion about an alternative approach

The following lemma is a complementary result related to Remark 1. It shows that we can actually derive the expression of the score from the results contained in Millet et al. [38]. The result is powerful, but requires the verification of technical conditions.

Lemma 3. Under the conditions stated in Proposition 3 the score is defined by

and the time reversed diffusion takes the form in (14).

Proof. Define

Then

and where we assume that C is of trace class. This is then an infinite dimensional system of the type considered in Millet et al. [38]. We proceed to verify some conditions stated in Millet et al. [38]: (i) the coefficients of the system (33) satisfy standard growth and Lipschitz continuity conditions (assumption ( H 1 , H 4) satisfied); (ii) the coefficients depend on finitely many coordinates (assumption ( H 2) satisfied); the system is time independent and diagonal (assumption ( H 5) satisfied). Moreover define ˇ x ( j ) = ( x 1 , . . . , x j -1 , x j +1 , . . . ) , then the law of X ( j ) t given ˇ X ( j ) t has for t > 0 density p t ( x ( j ) | ˇ X ( j ) t = ˇ x ( j ) , Y = y ) with respect to Lebesgue measure and so that for t 0 > 0 and each j : ∫ T t 0 E [ | ∂ x ( j ) log( p t ( x ( j ) ) | Y = y ) dt < ∞ . Then it follows from Theorems 3.1 and 4.3 in Millet et al. [38] that the time reversed problem is associated with the well-posed martingale problem defined by the coefficients in (14) for the score being:

with the convention that the right hand side is null on the set { p t ( x ( j ) | ˇ X ( j ) t = ˇ x ( j ) , Y = y ) = 0 } . It then follows for t > 0

We then get

C.2 A preliminary lemma

The following lemma is the equivalent of [24, Lemma 3]. It is used in the forthcoming proof of Proposition 2.

Lemma 4. In the finite-dimensional setting x ∈ R D , we have for any 0 ≤ s ≤ t ≤ T :

where E y is the expectation with respect to the distribution of X 0 and W given Y = y and p t,y is the pdf of X t under this distribution.

Proof. We can write

We can then deduce that

which gives

Using again that p t | s,y ( ·| x s ) = p t | s ( ·| x s ) and p t | s,y ( x t | x s ) = p ( s,t ) ,y ( x s ,x t ) p s,y ( x s ) , we get

Since ∇ log p t,y ( x t ) = ∇ p t,y ( x t ) p t,y ( x t ) and p s | t,y ( x s | x t ) = p ( s,t ) ,y ( x s ,x t ) p t,y ( x t ) we get that

where p t | s,y ( ·| x s ) is the pdf of X t given Y = y and X s = x s . It is, in fact, equal to the pdf of X t given X s = x s , which is the pdf of the multivariate Gaussian distribution with mean exp( -( t -s ) / 2) x s and covariance (1 -exp( -( t -s ))) C . Therefore

C.3 Proof of Proposition 2

The proof adapts the one of [24] to the conditional setting. The only difference is that the expectation is E y , which affects the distribution of X 0 but not the one of W . Moreover, Lemma 4 shows that the key to the proof (the reverse-time martingale property of the finite-dimensional score) is still valid. Here E y is the expectation with respect to the distribution of X 0 and W given Y = y .

To prove Proposition 2, we are left to show that the solution of the reverse-time SDE

satisfies Z T ∼ X 0 | Y = y . We recall that X t is the solution to the SDE

We first notice that X t is given by the following stochastic convolution:

For P D the orthogonal projection on the subspace of H spanned by v 1 , . . . , v D (the eigenfunctions of C ), X t D = P D ( X t ) are solutions to

where

We define X D : M t = X t M -X t D . Then

where the superscript D : M indicates the projection onto span { v D +1 , . . . , v M } . It holds that

as D →∞ , where we used Doob's L 2 inequality to bound the stochastic integral. Therefore ( X N t ) is a Cauchy sequence and converges to X t in L 2 ( P y ) . Consequently, the distribution of X N t given Y = y converges to the distribution of X t given Y = y as N → + ∞ .

Recall that and recall that

In particular, due to the tower property of the conditional expectations,

Since, by Assumption 1, the quantities E y [ X t -e -t/ 2 X 0 | X t D ] are bounded in L 2 ( P y ) and will converge to the limit, E y [ S ( t, X t , y ) | X t ] = S ( t, X t , y ) , by the Martingale convergence theorem. We get rid of the projection P D by

The first term vanishes due to our previous discussion. The second term vanishes since

We now make use of the fact that ∇ log p D t,y is a square-integrable Martingale in the reverse-time direction by Lemma 4. We therefore get a sequence of continuous L 2 -bounded Martingales converging to a stochastic process. Since the space of continuous L 2 -bounded martingale is closed and pointwise convergence translates to uniform convergence, we get that S is a L 2 -bounded martingale, with the convergence of of C D ∇ log p D t,y to S being uniform in time.

We have that

Since all the terms on the left-hand side converge in L 2 , uniformly in t , so does the right-hand side. Using again the closedness of the spaces of Martingales and Levy's characterization of Wiener process, we find that √ ( C D ) W t D converges to √ CW t . Therefore

Therefore, Z t is indeed a solution to (34) and Z T ∼ X 0 | Y = y . Using uniqueness of the solution we then conclude that this holds for any solution Z t .

C.4 Proof of (23)

C.5 Proof of Proposition 3

Note that with the assumptions in Proposition 3 with C µ and C having the same basis of eigenfunctions and the separability assumption on the Radon-Nikodym derivative for the modes, the system for the modes again diagonalizes. However, in this case the (conditional) distribution for X ( j ) 0 is non-Gaussian in general and the change of measure with respect to the Gaussian measure characterized by ψ ( j ) ( x ( j ) , y ) . We let the superscript g denote the Gaussian case with ψ ≡ 1 , then

we have:

for S ( j,g ) the mode score in the Gaussian case given in (31) and with

We have for ϕ λ the centered Gaussian density at second moment λ

for λ ( j ) t = λ j (1 -e -t ) , µ ( j ) y = µ j / (1 + q ( j ) ) and x ( j ) y = y ( j ) q ( j ) / (1 + q ( j ) ) for y ( j ) = ⟨ y, v j ⟩ . Here x ( j ) y , µ ( j ) y are respectively the mean and variance of X ( j ) 0 given y and where we used the parameterization set forth in Section B.1. We then have

We moreover have

We then find and, moreover

Consider in the Gaussian case in (22) a mode so that p ( j ) (1 + q ( j ) ) ↑ ∞ and λ j fixed, then the contribution of this mode to the score norm blows up in the small time limit. The situation with

p ( j ) (1 + q ( j ) ) ↑ ∞ would happen for instance in a limit of perfect mode observation so that σ B ↓ 0 and thus q ( j ) ↑ ∞ . Indeed in the limit of small (conditional) target mode variabilty relative to the diffusion noise parameter the score drift becomes large for small time to drive the mode to the conditional target distribution. We here thus assume p ( j ) (1 + q ( j ) ) is uniformly bounded with respect to mode ( j index), moreover, that C is of trace class. We then find that Assumption 1 is satisfied with the following bound

We remark that in the case that we do not have a uniform bound on the p ( j ) (1 + q ( j ) ) 's it follows from (23) that the rate of divergence of the expected square norm of the score is at most t -1 as t ↓ 0 with C of trace class.

C.6 Proof of Proposition 4

We start from (30):

From Definition 1 we have

We obtain that

with

that does not depend on θ . Since L ( X t | X 0 = x 0 , Y = y ) = L ( X t | X 0 = x 0 ) we finally get that

D Numerical experiments: details and additional results

In this section, we provide additional details regarding our numerical experiments. In both experiments, we parameterize the conditional score s θ ( t, x t , y ) using discretization-invariant Fourier neural operators [FNOs; 53]. This parameterization enables mapping input triplets ( t, x t , y ) to the score conditioned on y at time t . Once trained-by minimizing the objective function in equation (28) with respect to θ -weuse the FNO as an approximation to the conditional score to sample new realizations of the conditional distribution by simulating the reverse-time SDE in equation (14).

D.1 Stylized example

In this example, the conditional distribution that we approximate is defined using the relation

(a)

(b)

(c)

Predictedfunctions

Predictedfunctions

Predictedfunctions

5

5

0

0

0

-5

grid size 20

-5

grid size 30

-5

grid size 40

-3

-2

-1

0

1

2

3

3

-2

-1

0

1

2

3

3

2

-1

0

1

2

3

where ε ∼ Γ(1 , 2) and a ∼ U{1 , 1 } . Here, Γ refers to the Gamma distribution, and U{1 , 1 } denotes the uniform distribution over the set {-1 , 1 } . Having an explicit expression characterizing the conditional distribution allows us to easily evaluate the obtained result through our method-as opposed to needing to use a baseline method, e.g., Markov chain Monte Carlo.

Training data The discretization invariance of our model enables us to use training data that live on varying discretization grids. We exploit this property and simulate training joint samples ( x 0 , y ) by evaluating the expression in equation (35) over values of y that are selected as nonuniform grids over the domain [ -3 , 3] that contain 15 -50 grid points.

Architecture In this example, the FNO comprises (i) a fully connected lifting layer that maps the three-dimensional vector-including the timestep t , the y value, and the corresponding x value-for each grid point to a 128 -dimensional lifted space; (ii) five Fourier neural layers, as introduced in [53], which contain pointwise linear filters applied to the five lower Fourier modes; and (iii) two fully-connected layers, separated by a ReLU activation function, that map the 128 -dimensional lifted space back to the conditional score (a scalar) for each grid point.

Optimization details To train the FNO, we minimized the objective function in equation (28) using 2 × 10 4 training steps. At each step, we simulated a batch of 512 training pairs selected from a grid with varying numbers of discretization points. We utilized the Adam stochastic optimization method [56] with an initial learning rate of 10 -3 , which decayed to 5 × 10 -4 during optimization, following a power-law rate of -1 / 3 . Regarding the diffusion process, we followed the approach outlined in Ho et al. [29] and employed standard Gaussian noise with linearly increasing variance for the forward dynamics, which was discretized into 500 timesteps. The initial Gaussian noise variance was set to 10 -4 and linearly increased over the timesteps until it reached 2 × 10 -2 . The training hyperparameters were chosen by monitoring the validation loss over 1024 samples. The training process took approximately 7 minutes on a Tesla V100 GPU device. For further details, please refer to our open-source implementation on GitHub.

Additional results Figure 3 illustrates the predicted samples associated with the remaining testing grid sizes, whose densities were shown in Figures 1d-1f.

D.2 Linearized seismic imaging example

The inverse problem we are addressing in this example involves the task of estimating the shortwavelength component of the Earth's unknown subsurface squared-slowness model using measurements taken at the surface. This particular problem, often referred to as seismic imaging, can be recast as a linear inverse problem when we linearize the nonlinear relationship between surface measurements and the squared-slowness model, as governed by the wave equation. In its simplest acoustic form, the linearization with respect to the slowness model-around a background smooth squared slowness model m 0 -results in a linear inverse problem for the estimation of the true seismic image δm ∗ using the following forward model,

where d = { d i } n s i =1 represents a collection of n s linearized surface measurements, i.e., data wherein the zero-order term of Taylor's series has been subtracted, and J ( m 0 , q i ) denotes the linearized

Born scattering operator, which is defined in terms of the source signature q i and the background squared-slowness model m 0 . Due to noise and linearization errors, the above expression contains the term ϵ i . In addition to this noise term, the forward operator has a non-trivial nullspace due to the presence of shadow zones and finite-aperture data [57, 58]. To tackle this challenges, we set seismic imaging into a Bayesian framework and learn the associated posterior distribution via our proposed method.

Training data We generated synthetic data by applying the Born scattering operator to 4750 2D seismic images, each with dimensions of 3075 m × 5120 m . These images were extracted from the Kirchhoff migrated Parihaka-3D dataset, which contains seismic images obtained by imaging the data collected in New Zealand [59, 60]. We parameterize the Born scattering operator using a background squared-slowness model (recall Figure 2b). The data acquisition geometry involves 102 sources with a spacing of 50 m , each recorded for two seconds via 204 receivers spaced at 25 m located on top the image. The source wavelet used was a Ricker wavelet with a central frequency of 30 Hz . To replicate a more realistic imaging scenario, we add band-limited noise to the the data, obtained by filtering white noise with the source function. To create training pairs, we first simulated noisy seismic data for all the 2D seismic images based on the aforementioned acquisition design. Subsequently, we reduce the dimensionality of seismic data by applying the adjoint of the Born scattering operator to the data. We use Devito [61, 62] for the wave-equation based simulations.

Architecture In this example, the FNO is composed of (i) a fully connected lifting layer that maps the five-dimensional vector-including the timestep t , the two spatial coordinates, the data value ( y , obtained after dimensionality reduction by applying the adjoint of the forward operator), and the corresponding x value-for each grid point to a 32 -dimensional lifted space; (ii) four Fourier neural layers, as introduced in [53], which contain pointwise linear filters applied to the 24 lower Fourier modes; and (iii) two fully-connected layers, separated by a ReLU activation function, that map the 128 -dimensional lifted space back to the conditional score for each grid point.

Optimization details We train the FNO according to the objective function in equation (28) with the Adam [56] stochastic optimization method with batch size 128 for 300 epochs. We use an initial stepsize of 2 × 10 -3 , decaying to 5 × 10 -4 during optimization with a power-law rate of -1 / 3 . We use a similar diffusion process as the previous example. The training hyperparameters are chosen by monitoring the validation loss over 530 samples. The training process takes approximately two hours and 15 minutes on a Tesla V100 GPU device. For further details refer to our open-source implementation on GitHub.

Additional results Figures 4-7 illustrate more results regarding using the SDM to sample from the posterior distribution of multiple seismic imaging problem instances where the ground truth images (see Figures 4a-7a) are obtained from the test dataset. In each plot, we use the dimensionality-reduced data as the conditioning input to the FNO (see Figures 4b-7b). Through the SDM, we obtain 10 3 posterior distribution samples and use them to estimate the conditional mean (see Figures 4c-7c) and the pointwise standard deviation among samples (see Figures 4d-7d), with the former serving as a measure of uncertainty. In all cases, the regions of significant uncertainty correspond well with challenging-to-image sections of the model, which qualitatively confirms the accuracy of our Bayesian inference method. This observation becomes more apparent in Figures 4g-7g, displaying two vertical profiles with 99% confidence intervals (depicted as orange-colored shading) for each experiment, which demonstrate the expected trend of increased uncertainty with depth. Furthermore, we notice that the ground truth (indicated by dashed black lines) mostly falls within the confidence intervals for most areas. We also consistently observe a strong correlation between the pointwise standard deviation and the error in the conditional mean estimate (see Figures 4e-7e), which further asserts the accuracy of our method in characterizing the posterior distribution in this large-scale Bayesian inference problem. To prevent bias from strong amplitudes in the estimated image, we present the normalized pointwise standard deviation divided by the envelope of the conditional mean in Figures 4f-7f. These visualizations provides an amplitude-independent assessment of uncertainty, highlighting regions of high uncertainty at the onset and offset of reflectors (both shallow and deeper sections). Additionally, the normalized pointwise standard deviation underscores uncertainty in areas of the image where there are discontinuities in the reflectors (indicated by black arrows), potentially indicating the presence of faults.

(a)

(b)

(c)

(d)

(e)

(f)

(g)

0.0

Ground-truth(unknown)image x103

1.5

0.0

Reverse-timemigratedimage

×103

Profile at 0.2 km

Profile at 4.9 km

0.5

1.0

0.5

(y) yda

1.0

0.5

5

0.0

Conditional mean

%99confidenceinterval

Ground-truth image

1.5

0.0

0

2.0

0.5

2.5

2.5

-5

3.0

-1.0

3.0

0.5

-1.5

0

1

2

3

4

5

0

2

3

4

5

Horizontal distance (km)

0.0

Conditional mean estimate

×103

1.5

0.0

Pointwisestandarddeviation

×101

9.0

1.0

0.5

1.0

0.5

(y) daa

1.0

0.5

Depth (km)

1.5

0.0

1.5

2.0

0.5

2.5

-1.0

2.5

1.0

3.0

3.0

0

2

4

5

-1.5

0

2

3

4

5

0.6

2.0

0.0

Prediction error

×102

0.0

Normalized standard deviationx10-1

0.5

1.2

0.5

2.5

1.0

1.00

0.8

0.6

3.0

2.5

0.4

2.5

3.0

0.2

3.0

0

2

4

5

0

1

3

4

5

0.10

1000

0

-1000 1000

Perturbation

0

-1000

(a)

(b)

(c)

(d)

(e)

(f)

(g)

0.0

Ground-truth(unknown) image x103

1.5

0.0

Reverse-timemigratedimage

X103

Profile at 0.2 km

Profile at 4.9 km

0.5

1.0

0.5

1.0

0.5

5

0.0

Conditional mean

Ground-truth image

%99confidenceinterval

1.5

0.0

0

2.0

0.5

2.5

-1.0

2.5

-5

3.0

3.0

0.5

-1.5

0

√

2

3

4

5

0

1

2

3

4

5

Horizontal distance (km)

0.0

Conditional meanestimate

×103

1.5

0.0

Pointwise standard deviation

X101

9.0

1.0

0.5

1.0

0.5

0.5

1.0

Depth (km)

1.5

0.0

1.5

2.0

0.5

2.5

-1.0

2.5

1.0

3.0

0

1

2

3

4

5

-1.5

3.0

0

2

3

4

5

0.6

2.0

0.0

Prediction error

×102

0.0

Normalized standard deviationx10-1

0.5

1.2

0.5

2.5

1.0

1.00

0.8

0.6

3.0

2.5

0.4

2.5

3.0

0.2

3.0

0

Horizontal distance (km)

2

3

4

5

0

1

2

3

4

5

0.10

1000

Perturbation

0

-1000 1000

Perturbation

0

-1000

(a)

(b)

(c)

(d)

(e)

(f)

(g)

0.0

Ground-truth(unknown) image x103

1.5

0.0

Reverse-timemigratedimage

X103

Profile at 0.2 km

Profile at 4.9 km

0.5

1.0

0.5

0.5

1.0

5

0.0

%99 confidence interval

Conditionalmean

Ground-truth image

1.5

0.0

0

2.0

-0.5

2.5

2.5

-5

3.0

-1.0

3.0

0.5

-1.5

0

2

3

4

5

0

1

2

3

4

5

Horizontal distance (km)

0.0

Conditionalmeanestimate

×103

1.5

0.0

Pointwisestandarddeviation

×101

9.0

1.0

0.5

1.0

0.5

0.5

Depth (km)

Depth (km)

1.5

0.0

1.5

2.0

0.5

2.5

-1.0

2.5

1.0

3.0

3.0

0

2

4

5

-1.5

0

1

2

3

4

5

0.6

2.0

0.0

Predictionerror

×102

0.0

Normalized standard deviationx10-1

1.2

0.5

0.5

2.5

Depth (km)

1.00

0.6

3.0

2.5

0.4

2.5

3.0

0

1

Horizontal distance (km)

2

3

4

5

0.2

3.0

0

3

4

5

0.10

1000

0

-1000 1000

Perturbation

0

-1000

(a)

(b)

(c)

(d)

(e)

(f)

(g)

0.0

0.5

Ground-truth(unknown)image x103

1.0

1.5

0.0

0.5

Reverse-time migrated image

X103

5

Profile at 0.2 km

Conditional mean

Ground-truth image

Profile at 4.9 km

0.5

Depth (km)

1.0

0.0

%99confidenceinterval

0.0

0

2.0

Wm

-0.5

2.5

2.5

-5

3.0

-1.0

3.0

0.5

0

1

2

3

4

5

1.5

0

1

2

3

4

5

0.0

Conditionalmeanestimate

×103

1.5

0.0

Pointwisestandarddeviation

X101

9.0

1.0

0.5

1.0

0.5

Depth (km)

05

0.5

Depth (km)

1.0

Depth (km)

1.5

0.0

1.5

2.0

2.0

0.5

2.5

1.0

2.5

1.0

3.0

3.0

0

1

2

3

4

5

1.5

0

1

2

3

4

5

0.6

2.0

0.0

Predictionerror

X102

0.0

Normalized standard deviationx10-1

0.5

1.2

0.5

2.5

Depth (km)

15

1.00

0.6

2.0

2.5

0.4

2.5

3.0

3.0

0.2

3.0

0

1

Horizontal distance (km)

2

3

4

5

0

1

2

4

5

0.10

1000

0

-1000 1000

Perturbation

0

-1000

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\n- The study proposes a method to learn the posterior distribution in infinite-dimensional Bayesian linear inverse problems using amortized conditional Score-based Diffusion Models (SDMs). This extends conditional SMDs into the infinite-dimensional function space setting, as existing conditional SDMs have previously only dealt with finite-dimensional vector spaces (noting also that _unconditional_ SDMs have recently been extended to infinite dimensional vector spaces by Pidstrigach et. al).  This leads the way for applications in, for example PDEs, where the unknown parameters to be estimated take the form of functions. \n- The key technique underlying their approach is to define the _conditional_ score in an infinite dimensional setting, extending the method of Pidstrigach et. al, who defined the _unconditional_ score in the infinite dimensional setting. \n- Using their definition of the conditional score in infinite dimensions, this allows them to avoid having to solve a potentially expensive proximal optimization step, as was done by Pidstrigach et. al. \n- The authors then provide a comprehensive theoretical analysis of the use of their conditional score in SDMs and show: \n  - How this newly defined score is used as a reverse drift of the diffusion process, which leads to a generative model that samples from the correct target conditional distribution under certain conditions.\n  - That as long as you start from the invariant distribution of the diffusion process, the reverse SDE converges to the target distribution exponentially fast \n  - By explicitly computing the expected square norm of the conditional score, they shows that a uniform in time estimate is not always true for the conditional score. This leads them to provide a set of conditions to be satisfied to ensure a uniform in time estimate for a general class of prior measures.\n  - That the conditional score can be estimated via a conditional denoising score matching objective in infinite dimensions.\n  - That the conditional denoising estimator is a consistent estimator of the conditional score in infinite dimensions.\n  - That unlike the unconditional score, for noiseless observations the conditional score blows up as T->0\n- They present a small toy experiment that validates their approach by demonstrating the applicability of their method in approximating non-Gaussian multi-modal distributions.\n\nSTRENGTHS:\nI will address strengths and weaknesses across the four dimensions (Originality, Quality, Clarity, Significance) below.\n\nWEAKNESSES:\n**Originality:** \n- Are the tasks or methods new? \n  - Yes, the authors make a novel contribution to the quickly-growing diffusion model literature, by showing how to extend conditional SDMs into the infinite-dimensional vector space, thus opening the door to a wider range of applications (PDEs, etc)\n- Is the work a novel combination of well-known techniques? Is it clear how this work differs from previous contributions? \n  - The authors are very clear about: \n     1. How they are starting with the recently-proposed framework of Pidstrigach et al\n     2. The specific point where they deviate from, and then extend, Pidstrigach's work (Definition 2, eq. 8)\n  - Further, they provide a comprehensive analysis of the use of their defn 2 in SDMs in sections 4 and 5. \n  - However I should state that I am not adequately familiar with the mathematical techniques deployed in this paper to check their technical claims for correctness or novelty, so please differ to another reviewer with more expertise in this area. \n- Is related work adequately cited?\n  - Yes. \n\n**Quality:** \n- Is the submission technically sound?\n  - It appears to be, although as previously stated I am not sufficiently versed in their mathematical techniques to be 100% sure. \n- Are claims well supported (e.g., by theoretical analysis or experimental results)? \n  - Theoretically yes, experimentally no. I realize this is a theory paper and don't necessarily expect full-scale experiments on PDEs, but in Section 6 I expected to see at least 2 obvious baselines which were foreshadowed in the body of the text but not experimentally validated on the toy experiments. These are:\n    1. How does their method compare to the crude, discretization-based approach discussed in lines 56 - 61:\n        > A straightforward solution may be to discretize the infinite-dimensional input and output function spaces into finite-dimensional vectors, and apply SDMs to learn the posterior. Yet theoretical studies of current DMs suggest that performance guarantees do not generalize well on increasing dimensions [7, 9, 33]. This is precisely why Stuart’s guiding principle to study a Bayesian inverse problem for functions— “avoid discretization until the last possible moment” [41] — is more than ever critical to the use of SDMs. \n    A primary motivation for their method is \"Stuart's Principle\", which says to avoid discretization until the last possible moment. I would have liked to see experimentally why this principle is so important on the simple toy examples they have provided.\n    2. How does their method compare to the approach proposed by Pidstrigach for conditional sampling, which uses a proximal optimization step? This is a second primary motivation for their approach (and appears in the abstract) - their method is more performant because it avoids solving an optimization problem at each timestep. As far as I can tell it has not been experimentally validated that their approach is faster or more performant than the baseline method of Pidstrigach. Evidence showing their approach either gets better samples, or get samples of the same quality but more efficiently, is needed, given that a primary motivation for their approach is that the baseline method of Pidstrigach may be too computationally costly because of their use of proximal optimization. \n- Are the methods used appropriate? \n  - Yes\n- Is this a complete piece of work or work in progress? \n  - Yes, modulo the missing baselines discussed above. I believe such baselines are needed to consider this a complete piece of work, given how \"Stuart's Principle\" and \"avoid proximal optimization\" play a key role in the storyline and motivation for their technique. \n- Are the authors careful and honest about evaluating both the strengths and weaknesses of their work?\n  - Yes. In particular I found their observation, that unlike the unconditional score, for noiseless observations the conditional score blows up as t->0, particularly interesting. However in the conclusion I would have liked to read more about the authors' reflections on the strengths/weaknessness/future directions of their approach, both from a technical standpoint and from the standpoint of potential downstream applications of this work. \n\n**Clarity:** \n- Is the submission clearly written? \n  - I found all writing up until section 4 relatively easy to follow. I began to get lost around sections 4 and 5 and couldn't follow the math, but I attribute this largely to not being comfortable with stochastic differential equations. I could still follow the high-level plot in these sections, but will defer to other reviews to evaluate the technical claims. \n- Is it well organized? \n  - Yes \n- Does it adequately inform the reader? \n  - Yes \n\n**Significance:**\n- Are the results important? \n  - Yes, although I would have liked the authors to motivate the applications of their approach more thoroughly. I believe they only listed PDEs as an example for why you would want to use this approach but surely there are more applications than just PDEs, no? \n- Are others (researchers or practitioners) likely to use the ideas or build on them? \n  - Yes\n- Does the submission address a difficult task in a better way than previous work? \n  - Theoretically it appears so, but baselines are needed to experimentally validate these claims. \n- Does it advance the state of the art in a demonstrable way?\n  - Theoretically it appears so, but baselines are needed to experimentally validate these claims. \n-  Does it provide unique data, unique conclusions about existing data, or a unique theoretical or experimental approach?\n  - Yes, they build on the work of Pidstrigach in a novel way.\n\nQUESTIONS:\n- What are some other applications of this work besides PDEs? Can you include a few of them in the introduction so the reader doesn't think applications of your work is limited only to PDEs? \n\n- In the conclusion the authors say their method \"is able to perform conditional sampling directly on infinite dimensional Hilbert spaces.\" Don't Gaussian processes also allow you to perform conditional sampling directly on infinite dimensional Hilbert spaces? What are the differences between your approach and Gaussian processes? When would I want to use one vs. the other? Could a GP be used as a baseline in your Figure 1? If so it would be very interesting to see how it compares. \n\n- The authors also say \"We also show that the conditional score can have a singular behavior at small times when the observations are noiseless, in contrast with the unconditional score under similar hypotheses.\". Is this just a limiting phenomenon of theoretical interest, or do you expect this to cause difficulties in practice? Are there any applications where we should expect observations to be noiseless or does that assumption never hold in practice?\n\nLIMITATIONS:\n- There does not appear to be a \"Limitations and Broader Impacts\" statement in this work.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-26"}

comments: SUMMARY:
This paper mathematically examines linear inverse problems in infinite dimensional vector spaces. Particularly, it is proved that the conditional denoising estimator is a consistent estimator of the conditional score in infinite dimension.

STRENGTHS:
The consistency of the conditional denoising estimator in infinite dimensional vector spaces is mathematically shown. For a specific case of Gaussian prior, the forward-reverse SDEs are solved exactly, which shows an exponentially fast convergence in the reverse SDE. A sufficient condition for the success of the score-based diffusion model framework is presented for the infinite dimensional version.

WEAKNESSES:
The considered inverse problems are not of the plug-and-play type, which would limits the practical utility. Numerical example is limited to a one-dimensional toy model.

QUESTIONS:
Gaussian process (GP) is a popular method for inverse problem in Hilbert space. I wonder if the current problem would have a certain connection to GP, in particular, in the case of Gaussian prior discussed in section 4.

LIMITATIONS:
Descriptions on the motivation to examine the inverse problem in Hilbert space are lacking.  In what practical situations do inverse problems in Hilbert space come out? For instance, GP is widely used for Bayes optimization, which is an optimization scheme for black-box functions. Additional writing about possible applications of the inverse problem in Hilbert space would make the paper more attractive.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-06-29

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposed a method to deal with inverse problems in infinite dimensions using conditional-score-based models. Specifically, they propose to directly learn the posterior distribution in infinite-dimensional Bayesian linear inverse problems using amortized conditional SDMs. Moreover, this paper also discussed the robustness of the learned distribution against perturbations of the observations.  A numerical experiment is conducted to validate the efficiency.\n\nSTRENGTHS:\n1. This paper proposed an interesting method to deal with infinite-dimensional Bayesian linear inverse problems.\n\n2. It provides a detailed analysis of the forward-reverse conditional SDE framework in the case of a Gaussian prior measure.\n\n3. It provides a set of conditions to ensure a uniform in-time estimate for a general class of prior measures.\n\nWEAKNESSES:\n1. Regarding the introduced definition of the conditional score and the result that the conditional score can be estimated via a conditional denoising score matching, it seems that they are straightforward extensions of the unconditional case. For score-based models, the key is to learn a general distribution using score matching, regardless of whether it is a conditional distribution or a non-conditional distribution. I mean, there is actually no fundamental difference between a conditional distribution and non-conditional distribution, i.e., score-based matching applies to them or other general distributions equally.\n\n2. From my understanding, and also as suggested in the abstract, this paper only focuses on infinite-dimensional Bayesian linear inverse problems, rather than arbitrary inverse problems. The title of this paper is kind of misleading.  On the other hand, for linear inverse problems, the score of the likelihood term is relatively easy to obtain, compared to directly training a conditional score network. Please correct me if I am wrong.\n\nQUESTIONS:\nSome additional questions:\n\n1. What is the difference between (2) and (3)? \n\n2. The experiment part only considers a simple low-dimensional problem, is it possible to add some results of large-scale problems?\n\nLIMITATIONS:\nSee above.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nScore-based diffusion models are successful in solving inverse problems in a finite-dimensional setting, but infinite-dimensional diffusion models needs to be constructed with care, as the definitions for Lebesgue measures and densities become less clear. The authors extends the work of Pidstrigach et al. [33] of unconditional score matching in infinite dimensions to a conditional setting.\n\nSTRENGTHS:\nI recommend a strong accept for the paper because of its theoretical soundness and its approachable presentation in its explanation. \n\nThe paper presents a theoretically elegant and principled approach to solving inverse problems in infinite dimensions, as it is guided by Stuart’s principle and does not involve projection to finite vector spaces and discretization when unnecessary. The paper also analyzes a general scenario with prior distributions absolutely continuous w.r.t. the Gaussian measure, and propose analogous results to Pidstrigach et al. [33].\n\nWEAKNESSES:\nI cannot identify a specific point of weakness that has to be addressed. One can argue against its simple experiment but I see a proof-of-concept experiment sufficient for this paper.\n\nQUESTIONS:\nI have a few questions regarding the general aspects of Bayesian inference in infinite dimensions, as I am not familiar with the exact formulation. \n- Observational model in the paper occupies a finite-dimensional subspace. Is there a scenario where one cannot find an orthonormal basis such that the observation $y$ only spans a finite subspace?\n- While I understand Assumption 1 _can_ be satisfied under certain conditions given by the prior measure's Radon-Nikodym derivatives, but does this assumption suffer from finite training data? If we think about extreme settings with only a few training data, the score matching essentially tries to memorize these noiseless data points, causing an automatic violation of the assumption.\n\nLIMITATIONS:\n- Line 15 typo: it should read \"extension of ... to the *conditional* setting\". \n- Minor citation error on Line 72: The seminal score matching paper [17] has Hyvärinen as the sole author. \n- Line 293 typo: “… proposition _on_ such set of conditions”", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-06"}

comments: SUMMARY:
The author extends score based diffusion from finite dimensional processes to separable Hilbert space processes.
They demonstrate that on a non-linear toy data set that the method can work.

STRENGTHS:
The paper reads very well and is easy to follow. It is important to study what happens in general separable Hilbert spaces as many algorithms and models breaks down in the infinite dimensional setting. And even though you always do real application in the finite dimensional setting it is still important because if the method works in a separable Hilbert space setting it will not break down when you increase the precision your finite dimensional discretization.

WEAKNESSES:
The numerical experiment is not very convincing. The results for this toy problem are not very impressive. 
I suspect that if one runs multiple parallel version of a  Crank–Nicolson algorithm one would get better result.
It also would be interesting to see how the numerical method scale in practice when you increase the resolution of the grid.

QUESTIONS:
-When you in  section 4. let $(Af)_i=(v_k,f)$ where $v_k$ is a eigenvector, is it not very obvious what you get? I mean the problem is a infinite series of independent processes and the data don't couple the processes. Does it not mean that you are back to the finite dimensional case (since if $v_j \notin A$ then for that $j$ the processes is equivalent to the prior?

-In the application setting I don't get the prior on $X_0$ and when I look in the code it looks one has used $x_0$ as just a line?
- What is the operator $C$ in this example?

very very minor:
Do you really have to write infinite dimensional in each sentence?

LIMITATIONS:
.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07