Title: Variational Gaussian processes for linear inverse problems
Venue: NeurIPS 2023
Accepted: yes
Keywords: Linear inverse problems

Gaussian processes

Variational inference

Inducing variables

Asymptotics

Contraction rates
Abstract: By now Bayesian methods are routinely used in practice for solving inverse problems. In inverse problems the parameter or signal of interest is observed only indirectly, as an image of a given map, and the observations are typically further corrupted with noise. Bayes offers a natural way to regularize these problems via the prior distribution and provides a probabilistic solution, quantifying the remaining uncertainty in the problem. However, the computational costs of standard, sampling based Bayesian approaches can be overly large in such complex models. Therefore, in practice variational Bayes is becoming increasingly popular. Nevertheless, the theoretical understanding of these methods is still relatively limited, especially in context of inverse problems.

In our analysis we investigate variational Bayesian methods for Gaussian process priors to solve linear inverse problems. We consider both mildly and severely ill-posed inverse problems and work with the popular inducing variable variational Bayes approach proposed by Titsias [Titsias, 2009]. We derive posterior contraction rates for the variational posterior in general settings and show that the minimax estimation rate can be attained by correctly tunned procedures. As specific examples we consider a collection of inverse problems including the heat equation, Volterra operator and Radon transform and inducing variable methods based on population and empirical spectral features.

Variational Gaussian Processes For Linear Inverse Problems

Abstract

By now Bayesian methods are routinely used in practice for solving inverse problems. In inverse problems the parameter or signal of interest is observed only indirectly, as an image of a given map, and the observations are typically further corrupted with noise. Bayes offers a natural way to regularize these problems via the prior distribution and provides a probabilistic solution, quantifying the remaining uncertainty in the problem. However, the computational costs of standard, sampling based Bayesian approaches can be overly large in such complex models. Therefore, in practice variational Bayes is becoming increasingly popular. Nevertheless, the theoretical understanding of these methods is still relatively limited, especially in context of inverse problems. In our analysis we investigate variational Bayesian methods for Gaussian process priors to solve linear inverse problems. We consider both mildly and severely ill-posed inverse problems and work with the popular inducing variables variational Bayes approach proposed by Titsias [57]. We derive posterior contraction rates for the variational posterior in general settings and show that the minimax estimation rate can be attained by correctly tunned procedures. As specific examples we consider a collection of inverse problems including the heat equation, Volterra operator and Radon transform and inducing variable methods based on population and empirical spectral features.

1 Introduction

In inverse problems we only observe the object of interest (i.e. function or signal) indirectly, through a transformation with respect to some given operator. Furthermore, the data is typically corrupted with measurement error or noise. In practice the inverse problems are often ill-posed, i.e. the inverse of the operator is not continuous. Based on the level of ill-posedness we distinguish mildly and severely ill-posed cases. The ill-posedness of the problem prevents us from simply inverting the operator as it would blow up the measurement errors in the model. Therefore, to overcome this problem, regularization techniques are applied by introducing a penalty term in the maximum likelihood approximation. Standard examples include generalized Tikhonov, total variation and Moore-Penrose estimators, see for instance [4; 5; 9; 11; 18; 56] or a recent survey [3] on data-driven methods for solving inverse problems

An increasingly popular approach to introduce regularity to the model is via the Bayesian paradigm, see for instance [3; 7; 12; 29; 54] and references therein. Beside regularization, Bayesian methods provide a probabilistic solution to the problem, which can be directly used to quantify the remaining uncertainty of the approach. This is visualised by plotting credible sets, which are sets accumulating prescribed percentage of the posterior mass. For computing the posterior typically MCMC algorithms are used, however, these can scale poorly with increasing sample size due to the complex structure of the likelihood. Therefore, in practice often alternative, approximation methods are used. Variational

Botond Szabo

Department of Decision Sciences Bocconi University via Roentgen 1, 20136, Milano, MI, Italy botond.szabo@unibocconi.it

Bayes (VB) casts the approximation of the posterior into an optimization problem. The VB approach became increasingly popular to scale up Bayesian inverse problems, see for instance the recent papers [22; 31; 34; 42] and references therein. However, until recently these procedures were considered black box methods basically without any theoretical underpinning. Theoretical results are just starting to emerge [2; 46; 63; 64; 66], but we still have limited understanding of these procedures in complex models, like inverse problems.

In our analysis we consider Gaussian process (GP) priors for solving linear inverse problems. For Gaussian likelihoods, due to conjugacy, the corresponding posterior has an analytic form. Nevertheless, they are applied more widely, in non-conjugate settings as well. However, training and prediction even in the standard GP regression model, scales as O ( n 3 ) (or O ( n 2 ) for exact inference in the recent paper [16] leveraging advances in computing hardware) and O ( n 2 ) , respectively, which practically limits GPs to a sample size n of order 10 4 . Therefore, in practice often not the full posterior, but an approximation is computed. Various such approximation methods were proposed based on some sparse or low rank structure, see for instance [13; 14; 33; 43; 49; 50; 51; 52; 57]. Our focus here lies on the increasingly popular inducing variable variational Bayes method introduced in [57; 58].

In our work we extend the inducing variable method for linear inverse problems and derive theoretical guarantees for the corresponding variational approximations. More concretely we adopt a frequentist Bayes point-of-view in our analysis by assuming that there exists a true data generating functional parameter of interest and investigate how well the variational posterior can recover this object. We derive contraction rates for the VB posterior around the true function both in the mildly and severely ill-posed inverse problems. We then focus on two specific inducing variable methods based on the spectral features of the prior covariance kernel. We show that for both methods if the number of inducing variables are chosen large enough for appropriately tunned priors the corresponding variational posterior concentrates around the true function with the optimal minimax estimation rate. One, perhaps surprising aspect of the derived results is that the number of inducing variables required to attain the optimal, minimax contraction rate is sufficiently less in the inverse setting than in the direct problem. Therefore, inverse problems can be scaled up at a higher degree than standard regression models.

Related literature. The theory of Bayesian approaches to linear inverse problems is now well established. The study of their asymptotic properties started with the study of conjugate priors [1; 15; 20; 26; 27; 28] before addressing the non-conjugate case [25; 45] and rate-adaptive priors [26; 55]. By now we have a good understanding of both the accuracy of the procedure for recovering the true function and the reliability of the corresponding uncertainty statements. The theory of Bayesian non-linear inverse problems is less developed, but recent years have seen an increasing interest in the topic, see the monograph [36] and references therein. Some algorithmic developments for variational Gaussian approximations in non-linear inverse problems, and applications to MCMC sampling, can be found in [40; 41].

The inducing variable approach for GPs proposed by [57; 58] has been widely used in practice. Recently, their theoretical behaviour was studied in the direct, nonparametric regression setting. In [8] it was shown that the expected Kullback-Leibler divergence between the variational class and posterior tends to zero when sufficient amount of inducing variables were used. Furthermore, optimal contraction rates [37] and frequentist coverage guarantees [38; 59; 60] were derived for several inducing variable methods. Our paper focuses on extending these results to the linear inverse setting.

Organization. The paper is organized as follows. In Section 2 we first introduce the inverse regression model where we carry out our analysis. Then we discuss the Bayesian approach using GPs and its variational approximations in Sections 2.1 and 2.2, respectively. As our main result we derive contraction rates for general inducing variable methods, both in the mildly and severely ill-posed cases. Then in Section 2.3 we focus on two specific inducing variable methods based on spectral features and provide more explicit results for them. We apply these results for a collection of examples, including the Volterra operator, the heat equation and the Radon transform in Section 3. Finally, we demonstrate the applicability of the procedure in the numerical analysis of Section 4 and conclude the paper with discussion in Section 5. The proof of the main theorem together with technical lemmas and additional simulation study are deferred to the supplementary material.

Notation. Let C, c be absolute constants, independent of the parameters of the problem whose values may change from line to line. For two sequences ( a n ) and ( b n ) of numbers, a n glyph[lessorsimilar] b n means that there

exists a universal constant c such that a n ≤ cb n and we write a n glyph[equivasymptotic] b n if both a n glyph[lessorsimilar] b n and b n glyph[lessorsimilar] a n hold simultaneously. We denote by a n glyph[lessmuch] b n if | a n /b n | tends to zero. The maximum and minimum of two real numbers a and b are denoted by a ∨ b and a ∧ b , respectively. We use the standard notation δ ij = 1 i = j . For m ≥ 1 , we note S m ++ the set of positive-definite matrices of size m × m .

2 Main results

In our analysis we focus on the non-parametric random design regression model where the functional parameter is observed through a linear operator. More formally, we assume to observe i.i.d. pairs of random variables ( x i , Y i ) i =1 ,...,n satisfying

where f 0 ∈ L 2 ( T ; µ ) , for some domain T ⊂ R d and measure µ on T , is the underlying functional parameter of interest and A : L 2 ( T ; µ ) ↦→ L 2 ( X ; G ) , for the measure G = A µ on X , is a known, injective, continuous linear operator. In the rest of the paper we use the notation P f 0 and E f 0 for the joint distribution and the corresponding expectation, respectively, of the data ( X,Y ) = ( x i , Y i ) i =1 ,...,n . Furthermore, we denote by E X , P X , E Y | X , P Y | X the expectation/distribution under G ⊗ n and the law of ( Y i ) i given the design respectively. Finally, for simplicity we take σ 2 = 1 in our computations.

In the following, denoting A ∗ the adjoint of A , we assume that the self-adjoint operator A ∗ A : L 2 ( T ; µ ) ↦→ L 2 ( T ; µ ) possesses countably many positive eigenvalues ( κ 2 j ) j with respect to the eigenbasis ( e j ) j (which is verified if A is a compact operator for instance). We remark that ( g j ) j defined by A e j = κ j g j is an orthonormal basis of L 2 ( X ; G ) . We work on the ill-posed problem where κ j → 0 , the rate of decay characterizing the difficulty of the inverse problem.

Definition 1. We say the problem is mildly ill-posed problem of degree p > 0 if κ j glyph[equivasymptotic] j -p has a polynomial decay. In the severely ill-posed problem, the rate we consider is exponential, κ j glyph[equivasymptotic] e -cj p for c > 0 , p ≥ 1 , and p is the degree of ill-posedness once again.

In nonparametrics it is typically assumed that f 0 belongs to some regularity class. Here we consider the generalized Sobolev space

for some β > 0 . We note that the difficulty in estimating f 0 from the data is twofold: one needs to deal with the observational noise, which is a statistical problem, as well as to invert the operator A , which comes from inverse problem theory. As a result of the ill-posedness of the problem, recovering f 0 from the observations may suffer from problems of unidentifiability and instability. The solution to these issues is to incorporate some form of regularization in the statistical procedure. The Bayesian approach provides a natural way to incorporate regularization into the model via the prior distribution on the functional parameter. In fact penalized likelihood estimators can be viewed as the maximum a posteriori estimators with the penalty term induced by a prior. For example Tikhonov type regularizations can be related to the RKHS-norm of a Gaussian Process prior, see [35; 44] for a more detailed discussion.

2.1 Gaussian Process priors for linear inverse problems

Wefocus on the Bayesian solution of the inverse problem and exploit the Gaussian likelihood structure by considering conjugate Gaussian Process (GP) priors on f . A GP GP ( η ( · ) , k ( · , · )) is a set of random variables { f ( t ) | t ∈ T } , such that any finite subset follows a Gaussian distribution. The GP is described by the mean function η and a covariance kernel k ( t, t ′ ) . We consider centered GPs as priors (i.e. we take η ≡ 0 ). Then the bilinear, symmetric nonnegative-definite function k : T × T ↦→ R determines the properties of the process (e.g., its regularity). In view of the linearity of the operator A the corresponding posterior distribution is also a Gaussian process. The mean and covariance function of the posterior is given by

where y = ( y 1 , . . . , y n ) T , A f = ( A f ( x i ) ) i =1 ,...,n , K A f A f = E Π A f A f T ∈ R n × n with E Π denoting the expectation with respect to the GP prior Π , K T t A f = ( E Π A f ( x i ) f ( t ) ) i =1 ,...,n ∈ R n , see the supplement for the detailed derivation.

Due to the closed-form expressions for the posterior and the marginal likelihood, as well as the simplicity with which uncertainty quantification may be produced, GP regression has gained popularity [44]. Furthermore, the asymptotic frequentist properties of posteriors corresponding to GP priors in the direct problem, with A taken to be the identity operator, is well-established by now. Optimal contraction rates and confidence guarantees for Bayesian uncertainty quantification were derived in the regression setting and beyond, see for instance [10; 39; 47; 53; 48; 61; 62; 65] and references therein. In the following, we say that ε n is an L 2 -posterior contraction rate for the posterior Π[ · | X,Y ] if for any M n → 0

In our analysis we consider covariance kernels with eigenfunctions coinciding with the eigenfunctions of the operator A ∗ A , i.e. we take

where ( λ j ) j denote the corresponding eigenvalues. The asymptotic behaviour of the corresponding posterior has been well investigated in the literature both in the mildly and severely ill-posed inverse problems. Rate optimal contraction rates and frequentist coverage guarantees for the resulting credible sets were derived both for known and unknown regularity parameters [15; 26; 27; 28; 55]. These results were further extended for other covariance kernels where the eigenfunctions do not exactly match the eigenfunctions of the operator A , but in principle they have to be closely related, see [1; 20; 25; 45].

However, despite the explicit, analytic form of the posterior given in (3) and the theoretical underpinning, the practical applicability of this approach is limited for large sample size n . The computation of the posterior involves inverting the n -byn matrix K A f A f + σ 2 I n , which has computational complexity O ( n 3 ) . Therefore, in practice often not the true posterior, but a scalable, computationally attractive approximation is applied. Our focus here is on the increasingly popular inducing variable variational Bayes method introduced in [57; 58].

2.2 Variational GP for linear inverse problems

In variational Bayes the approximation of the posterior is casted as an optimization problem. First a tractable class of distributions Q is considered, called the variational class. Then the approximation Ψ ∗ is computed by minimizing the Kullback-Leibler divergence between the variational class and the true posterior, i.e.

There is a natural trade-off between the computational complexity and the statistical accuracy of the resulting approximation. Smaller variational class results in faster methods and easier interpretation, while more enriched classes preserve more information about the posterior ensuring better approximations.

In context of the Gaussian process regression model (with the operator A taken to be the identity), [58] proposed a low-rank approximation approach based on inducing variables. The idea is to compress the information encoded in the observations of size n into m so called inducing variables. We extend this idea for linear inverse problems. Let us consider real valued random variables u = ( u 1 , . . . , u m ) ∈ L 2 (Π) , expressed as measurable linear functionals of f and whose prior distribution is Π u . In view of the linearity of u , the joint distribution of ( f, u ) is a Gaussian process, hence the conditional distribution f | u denoted by Π( ·| u ) , is also a Gaussian process with mean function and covariance kernel given by

respectively, where K t u = E Π ( f ( t ) u ) ∈ R m and K uu = E Π ( uu T ) ∈ R m × m . Then the posterior is approximated via a probability measure Ψ u on ( R m , B ( R m )) by Ψ = ∫ Π[ ·| u ] d Ψ u ( u ) , which is

absolutely continuous against Π and satisfies d Ψ d Π ( f ) = d Ψ u d Π u ( u ( f )) . We note that the variables u were first considered to be point evaluations of the GP prior process before these ideas were extended to interdomain inducing variables, e.g. integral forms of the process [30; 58].

Taking Ψ u = N ( µ u , Σ) as a multivariate Gaussian, the corresponding Ψ ∝ Π( ·| u )Ψ u is a Gaussian process, with mean and covariance functions

Letting µ u and Σ be the free variational parameters, the variational family is taken as

consisting of ' m -sparse' Gaussian processes.

By similar computations as those from [37], it can be shown that Π[ · | X,Y ] is equivalent to any element of Q (they are mutually dominated) so that the KL divergence is always finite and there exists a Ψ ∗ u , corresponding to the minimizer Ψ ∗ of KL (Ψ | | Π[ ·| X,Y ]) . Furthermore, we have

where K A f ( x i ) u = E Π A f ( x i ) u T . One can observe that the parameters of the variational approximations are

for K u A f = K T A fu = E Π u ( A f ) T the m × n matrix whose j th column is K A f ( x i ) u and ( A f ) T = ( A f ( x 1 ) , . . . , A f ( x n )) . Then the explicit form for the variational posterior Ψ ∗ can be attained by plugging in the parameters (6) into the variational mean and covariance function (5). We also define Q A f A f = K T u A f K uu K u A f .

We investigate the statistical inference properties of the above variational posterior distribution Ψ ∗ . More concretely we focus on how well the variational approximation can recover the underlying true functional parameter f 0 of interest in the indirect, linear inverse problem (1). We derive contraction rate for Ψ ∗ both in the mildly and severely ill-posed inverse problem case. Furthermore, we consider both the standard exponential and polynomial spectral structures for the prior, i.e. we assume that the eigenvalues of the prior covariance kernel satisfies either λ j glyph[equivasymptotic] j -α e -ξj p or λ j glyph[equivasymptotic] j -1 -2 α for some α ≥ 0 , ξ > 0 . Finally, in view of [37], we introduce additional assumptions on the covariance kernel of the conditional distribution of f | u ensuring that the variational posterior is not too far from the true posterior in Kullback-Leibler divergence.

Theorem 1. Let's assume that f 0 ∈ ¯ H β and ‖ f j ‖ ∞ glyph[lessorsimilar] j γ for β > 0 , γ ≥ 0 .

In the mildly-ill posed problem where κ j glyph[equivasymptotic] j -p , p > 0 , if λ j glyph[equivasymptotic] j -1 -2 α for α > 0 and ( α ∧ β ) + p > 3 / 2 + 2 γ , the posterior contracts at the rate ε inv n = n -α ∧ β 1+2 α +2 p .

In the severely ill-posed problem where κ j glyph[equivasymptotic] e -cj p , c > 0 , p ≥ 1 , if λ j glyph[equivasymptotic] j -α e -ξj p for α ≥ 0 , ξ > 0 , the posterior contracts at the rate ε inv n = log -β/p n .

Furthermore, if there exists a constant C independent of n such that

where ε n = n -α ∧ β + p 1+2 α +2 p in 1., and ε n = n -c/ ( ξ +2 c ) log -β/p + cα/ ( ξ +2 c ) ( n ) in 2., Ψ ∗ contracts around f 0 at the rate ε inv n for the mildly and severely ill-posed problems i.e.

Proof. We provide the sketch of the proof here, the detailed derivation of the theorem is deferred to the supplementary material. In a first step, we derive posterior contraction rates around A f 0 in empirical L 2 -norm under fixed design. In particular, we obtain an exponential decay of the probability expectation in the form

for arbitrary M n →∞ , where ε n = n -α ∧ β + p 1+2 α +2 p in the mildly and ε n = n -c ξ +2 c log -β p + cα ξ +2 c n in the severely ill-posed problems and A n is an event on the sample space R n with probability tending to one asymptotically. In the mildly ill-posed case, this follows from results in [17; 62], while additional care is needed in the severely ill-posed case. As a second step, we go back to the random design setting. We show, using concentration inequalities and controlling the tail probability of GPs in the spectral decomposition, that the empirical and population L 2 -norms are equivalent on a large enough event. This implies contraction rate with respect to the ‖ · ‖ L 2 ( X ,G ) -norm around A f 0 , similarly to (8). In the third step, using the previous result on the forward map, we derive contraction rates around f 0 . To achieve this we apply the modulus of continuity techniques introduced in [25]. Notably, we extend their ideas to infinite Gaussian series priors in the severely ill-posed case as well. Since in all these steps we can preserve the exponential upper bound for the posterior contraction (on a large enough event), we can apply Theorem 5 of [46], resulting in contraction rates for the VB procedure. It requires a control of the expected KL divergence between these two distributions, which follows from our assumptions on the expected trace and spectral norm of the covariance matrix of A f | u , see Lemma 3 in [37] for the identity operator A .

We briefly discuss the above results. First of all, the L 2 ( T ; µ ) -contraction rate of the true posterior, to the best of our knowledge, wasn't derived explicitly in the literature before, hence it is of interest in its own right. Nevertheless, the main message is that the variational posterior achieves the same contraction rate as the true posterior under the assumption (7). Note that in the mildly ill-posed inverse problem case for eigenvalues λ j glyph[equivasymptotic] j -1 -2 β (i.e. taking α = β ), the posterior contracts with the minimax rate n -β/ (1+2 β +2 p ) . Note that the d -dimensional case directly follows from this result when one defines the regularity class (2) and ill-posedness (Definition 1) with β/d and p/d which would imply the rate n -β/ ( d +2 β +2 p ) . Similarly in the severely-ill posed case one can achieve the minimax logarithmic contraction rate. Furthermore, the choice of the eigenvalue structure in the theorem was done for computational convenience, the results can be generalised for other choices of λ j as well. Though we considered the random variables u fixed as we do not optimize them above, they could conceivably be considered as free variational parameters and selected at the same time as µ u and Σ u .

In the next subsection we consider two specific choices of the inducing variables, i.e. the population spectral feature method and its empirical counter part. We show that under sufficient condition on the number of inducing variables condition (7) is satisfied implying the contraction rate results derived in the preceding theorem.

2.3 Population and empirical spectral features methods

We focus here on two inducing variables methods, based on the spectral features (i.e. eigenspectrum) of the empirical covariance matrix K A f A f = E Π A f A f T and the corresponding population level covariance operator ( x, y ) ↦→ E Π A f ( x ) A f ( y ) .

We start with the former method and consider inducing variables of the form

where v j = ( v 1 j , . . . , v n j ) is the eigenvector of K A f A f corresponding to the j th largest eigenvalue ρ j of this matrix. Similarly to the direct problem studied in [8; 37], this results in ( K A f A f ) ij = ρ j δ ij , ( K A fu ) ij = ρ j v i j , Q A f A f = ∑ m j =1 ρ j v j v T j , and K A f A f -Q A f A f = ∑ n j = m +1 ρ j v j v T j . The computational complexity of deriving the first m eigenvectors of K A f A f is O ( n 2 m ) . This is still quadratic in n , which sets limitations to its practical applicability, but it can be computed for arbitrary

choices of the prior covariance operator and map A . We also note that this choice gives the optimal rankm approximation Q A f A f of K A f A f and it was noted in [8] that it gives the minimiser of the trace and norm terms in (7).

The second inducing variables method is based on the eigendecomposition of covariance kernel ( x, y ) ↦→ E Π A f ( x ) A f ( y ) . Let us consider the variables

Again, by extending the results derived in the direct problem [8] to the inverse setting, this results in ( K A f A f ) ij = λ j κ j δ ij , ( K A fu ) ij = λ j κ j φ i j , Q A f A f = ∑ m j =1 λ j κ j φ j φ T j , and K A f A f -Q A f A f = ∑ n j = m +1 λ j κ j φ j φ T j , where φ j = ( φ j ( x 1 ) , . . . , φ j ( x n )) T . The computational complexity of this method is O ( nm 2 ) , which is substantially faster than its empirical counter part. However, it requires the exact knowledge of the eigenfunctions of the prior covariance kernel, and therefore in general has limited practical applicability.

Corollary 1. Let's assume that f 0 ∈ ¯ H β , ‖ g j ‖ ∞ glyph[lessorsimilar] j γ for β > 0 , γ ≥ 0 and in the

mildly-ill posed case κ j glyph[equivasymptotic] j -p : take prior eigenvalues λ j glyph[equivasymptotic] j -1 -2 α for some α > 0 , ( α ∧ β ) + p > 3 / 2 + 2 γ , number of inducing variables m n ≥ n 1 (1+2 p +2 α ) and denote by ε inv n = n -α ∧ β 1+2 α +2 p .

severely ill-posed case κ j glyph[equivasymptotic] e -cj p : take prior eigenvalues λ j glyph[equivasymptotic] j -α e -ξj p for α ≥ 0 , ξ > 0 , number of inducing variables m p n ≥ ( ξ +2 c ) -1 log n , and introduce the notation ε inv n = log -β/p n .

Then both for the population (if γ = 0 in 1.) and empirical spectral features variational methods the corresponding variational posterior distribution contracts around the truth with the rate ε inv n , i.e.

Remark 1. In the mildly ill-posed inverse problem taking α = β results in the minimax contraction rate for m n ≥ n 1 1+2 p +2 α . Note that it is substantially less compared to the direct problem with p = 0 , hence the computation is even faster in the inverse problem case.

3 Examples

In this section we provide three specific linear inverse problems as examples. The Volterra (integral) operator and the Radon transformations are mildly ill-posed, while the heat-equation is a severely ill-posed inverse problem. We show that in all cases by optimally tunning the GP prior and including enough inducing variables, the variational approximation of the posterior provides (from a minimax perspective) optimal recovery of the underlying signal f 0 .

3.1 Volterra operator

First, let us consider the V olterra operator, A : L 2 [0 , 1] -→ L 2 [0 , 1] satisfying that

The eigenvalues of A ∗ A and the corresponding eigenbases are given by κ 2 j = ( j -1 / 2) -2 π -2 , e j ( x ) = √ 2 cos (( j -1 / 2) πx ) , g j ( x ) = √ 2 sin (( j -1 / 2) πx ) respectively, see [21]. Therefore the problem is mildly ill-posed with degree p = 1 and these bases are uniformly bounded, i.e. sup j ‖ e j ‖ ∞ ∨ ‖ g j ‖ ∞ < ∞ . The following lemma is then a direct application of Corollary 1.

Corollary 2. Consider the Volterra operator in (1) and assume that f 0 ∈ ¯ H β , for some β > 1 / 2 . Set the eigenvalues in (4) as λ j = j -1 -2 β . Then the variational posterior Ψ ∗ resulting from either the empirical or population spectral features inducing variable methods achieves the minimax contraction rate if the number of inducing variables exceeds m n glyph[greaterorsimilar] n 1 3+2 β , i.e. for arbitrary M n →∞

3.2 Heat equation

Next let us consider the problem of recovering the initial condition for the heat equation. The heat equation is often considered as the starting example in the PDE literature and, for instance, the Black-Scholes PDE can be converted to the heat equation as well. We consider the Dirichlet boundary condition for

u

defined on

[0

,

1]

×

[0

, T

]

,

T >

0

. For

µ

∈

L

2

[0

,

1]

,

u

(

x, t

) =

2

∞

j

=1

µ

j

e

)

,

∑

-

j

2

π

2

t

with µ j = √ 2 ∫ 1 0 µ ( s ) sin( jπs ) ds . Therefore, if A : D ↦→ D , with D := { f ∈ L 2 [0 , 1] , f (0) = f (1) = 0 } , is such that, for µ = f , A f ( x ) = u ( x, T ) , then the corresponding singular-values and singular-functions of the operator A are κ j = e -j 2 π 2 T and e j ( x ) = g j ( x ) = √ 2 sin( jπx ) . Therefore it is a severely ill-posed problem with p = 2 and c = π 2 T . We also note that sup j ‖ e j ‖ ∞ ∨ ‖ g j ‖ ∞ < ∞ . This problem has been well studied both in the frequentist [6; 19; 32] and Bayesian setting [28; 54]. Then, by direct application of Corollary 1 we can provide optimality guarantees for the variational Bayes procedure in this model as well.

Corollary 3. Consider the heat equation operator A as above in the linear inverse regression model (1) and assume that f 0 ∈ ¯ H β for some β > 0 . Furthermore, we set the eigenvalues λ j = j -α e -ξj 2 , α ≥ 0 , ξ > 0 in (4) . Then the variational approximation Ψ ∗ resulting from either of the spectral features inducing variables method with m n ≥ ( ξ + π 2 T ) -1 / 2 log 1 / 2 n achieves the minimax contraction rate, i.e. for arbitrary M n →∞

3.3 Radon transform

Finally, we consider the Radon transform [24], where for some (Lebesgue)-square-integrable function f : D → R defined on the unit disc D = { x ∈ R 2 : ‖ x ‖ 2 ≤ 1 } , we observe its integrals along any line intersecting D . If we parameterized the lines by the length s ∈ [0 , 1] of their perpendicular from the origin and the angle φ ∈ [0 . 2 π ) of the perpendicular to the x-axis, we observe

where ( s, φ ) ∈ S = [0 , 1] × [0 , 2 π ) . The Radon transform is then a map from A : L 2 ( D ; µ ) → L 2 ( S ; G ) , where µ is π -1 times the Lebesgue measure and dG ( s, φ ) = 2 π -1 √ 1 -s 2 dsdφ . Then A is a bijective, mildly ill-posed linear operator of order p = 1 / 4 . Furthermore, the operator's singular value decomposition can be computed via Zernike polynomials Z k m (degree m , order k ) and Chebyshev polynomials of the second kind U m (cos θ ) = sin (( m +1) θ ) / sin θ ≤ m +1 , see [24]. Translating it to the single index setting, we get for some functions l, m : N ↦→ N satisfying m ( j ) glyph[equivasymptotic] √ j and | l ( j ) | ≤ m ( j ) , that

if polar coordinates are used on D . Therefore, we have sup j ( ‖ e j ‖ ∞ ∨ ‖ g j ‖ ∞ ) / √ j < ∞ . Then, by directly applying Corollary 1 to this setting we can show that the variational Bayes method achieves the minimax contraction rate.

Corollary 4. Consider the Radon transform operator (13) in the inverse regression model (1) and let us take f 0 ∈ ¯ H β , β > 9 / 4 . Taking polynomially decaying eigenvalues λ j glyph[equivasymptotic] j -1 -2 β , the empirical spectral features variational Bayes method achieves the optimal minimax contraction rate if m n glyph[greaterorsimilar] n 1 / (3 / 2+2 β ) , i.e. for any M n →∞

4 Numerical analysis

We demonstrate the approximation accuracy of the variational Bayes method on synthetic data. We consider here the recovery of the initial condition of the heat condition 3.2, which is a severely

ill-posed. In the supplement we provide additional simulation study for mildly ill-posed inverse problems as well. We set the sample size n = 8000 , take uniformly distributed covariates on [0 , 1) , and let

for β = 1 . The independent- observations are generated as Y i ∼ N ( A f 0 ( x i ) , 1) , depending on the solution of the forward map A f 0 after time T = 10 -2 .

We consider the prior with λ j = e -ξj 2 for ξ = 10 -1 . In view of Corollary 3 the optimal number of inducing variables is m = ( ξ +2 π 2 T ) -1 / 2 log 1 / 2 n ≈ 6 . We consider the population spectral feature method described in (10) and plot the variational approximation of the posterior for m = 6 and m = 3 inducing variables in Figure 1. We represent the true posterior mean by solid red and the upper and lower pointwise 2 . 5% quantiles by dashed red curves. The true function is given by blue and the mean and quantiles of the variational approximation by solid and dotted purple curves, respectively.

Observe that with m = 6 , see left part of Figure 1, the variational approximation results in similar 95% pointwise credible bands and posterior mean as the true posterior, providing an accurate approximation. Also note that both the true and the variational posterior contain f 0 at most of the points, indicating frequentist confidence validity of the set. At the same time, by taking a factor of two less inducing points, i.e. m = 3 , the credible sets will be overly large, resulting in good frequentist coverage, but suboptimally large posterior spread, see the second plot in Figure 1.

The computations were carried out with a 2,6 GHz Quad-Core Intel Core i7 processor. The computation of the exact posterior mean and covariance kernel on a grid of 100 points took over half an hour (in CPU time), while the variational approximation was substantially faster, taking only 50 . 5 ms, resulting in a 3 . 68 ∗ 10 4 times speed.

posteriormean

Pointwise95%CI

fo

Variational mean(m=6)

Pointwise 95% CI (variational)

Variational mean(m=3)

2

Pointwise 95% CI (variational)

0.2

0.4

0.6

0.8

A more extensive numerical analysis is available in the appendix, considering the application of our method to the settings of Sections 3.1 and 3.3 as well. We conduct these experiments several times and compare the average Mean Integrated Squared Error (MISE), see appendix A, and compute time for different choices of m . We observe that in all our examples, while increasing m results in longer computation, the MISE does not improve after a threshold close to the one presented in our results.

Therefore, it is sufficient to include as many inducing variables as we considered in Corollary 1 to obtain better performance. More than that would would only increase the computation complexity. In the Appendix, we also provide a literature review and some justifications of how relevant these problems are in practice

5 Discussion

We have extended the inducing variables variational Bayes method for linear inverse problems and derived asymptotic contraction rate guarantees for the corresponding variational posterior. Our theoretical results provide a guide for practitioners on how to tune the prior distribution and how many inducing variables to apply (in the spectral feature variational Bayes method) to obtain minimax rate optimal recovery of the true functional parameter of interest. We have demonstrated the practical relevance of this guideline numerically on synthetic data and have shown that using less variables results in highly suboptimal recovery.

In our analysis we have considered priors built on the singular basis of the operator A . In principle our results can be extended to other priors as well, until the eigenbasis of the covariance operator is not too different from the basis of the operator A . This, however, would complicate the computation of the Kullback -Leibler divergence between the variational family and the posterior, resulting in extra technical challenges. In this setting the empirical spectral features method seems practically more feasible, especially, if the eigenbasis of the covariance kernel is not known explicitly. In the literature several different types of inducing variable methods were proposed, considering other, practically more relevant approaches of interest. Furthermore, extension to other type of inverse problems is also feasible. For instance in the deconvolution problem, when f 0 is convoluted with a rectangular kernel, the eigenvalues given by the SVD are the product of a polynomially decaying and oscillating part and the 'average degree' of ill-posedness does not match the lower and upper bounds [23]. Extension to non-linear inverse problem is highly relevant, as these problems tend to be computationally even more complex, but very challenging. One possible approach is to linearize the problem and take its variational approximation. Finally, it is of importance to derive frequentist coverage guarantees for VB credible sets. Our approach cannot directly be extended for this task. However, in the direct case, for some special choices of inducing variables, frequentist coverage guarantees were derived using kernel ridge regression techniques [38; 60]. This result, although computationally somewhat cumbersome, in principle can be extended to the inverse setting as well. One last drawback of our results is that the priors we consider are non-adaptative in the mildly ill-posed case. Minimax contraction rates are attainable only if the covariance eigenvalues are properly tuned, given the smoothness β . While this is not an issue the severely ill-posed case in our results, we keep the study of adaptation for future works as it is a much more involved question.

Funding. Co-funded by the European Union (ERC, BigBayesUQ, project number: 101041064). Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or the European Research Council. Neither the European Union nor the granting authority can be held responsible for them.

References

AGAPIOU, S., LARSSON, S., AND STUART, A. M. Posterior contraction rates for the Bayesian approach to linear ill-posed inverse problems. Stochastic Process. Appl. 123 , 10 (2013), 3828-3860.

ALQUIER, P., AND RIDGWAY, J. Concentration of tempered posteriors and of their variational approximations. Ann. Statist. 48 , 3 (2020), 1475-1497.

ARRIDGE, S., MAASS, P., ÖKTEM, O., AND SCHÖNLIEB, C.-B. Solving inverse problems using data-driven models. Acta Numerica 28 (2019), 1-174.

BERTERO, M. Regularization methods for linear inverse problems. In Inverse problems (Montecatini Terme, 1986) , vol. 1225 of Lecture Notes in Math. Springer, Berlin, 1986, pp. 52-112.

BISSANTZ, N., HOHAGE, T., MUNK, A., AND RUYMGAART, F. Convergence rates of general regularization methods for statistical inverse problems and applications. SIAM J. Numer. Anal. 45 , 6 (2007), 2610-2636.

BISSANTZ, N., AND HOLZMANN, H. Statistical inference for inverse problems. Inverse Problems 24 , 3 (2008), 034009.

BRIOL, F.-X., OATES, C. J., GIROLAMI, M., OSBORNE, M. A., AND SEJDINOVIC, D. Probabilistic integration. Statistical Science 34 , 1 (2019), 1-22.

BURT, D., RASMUSSEN, C. E., AND VAN DER WILK, M. Rates of convergence for sparse variational Gaussian process regression. In Proceedings of the 36th International Conference on Machine Learning (09-15 Jun 2019), K. Chaudhuri and R. Salakhutdinov, Eds., vol. 97 of Proceedings of Machine Learning Research , PMLR, pp. 862-871.

CANDÈS, E. J., AND WAKIN, M. B. An introduction to compressive sampling. IEEE signal processing magazine 25 , 2 (2008), 21-30.

CASTILLO, I. Lower bounds for posterior rates with Gaussian process priors. Electron. J. Stat. 2 (2008), 1281-1299.

CAVALIER, L. Nonparametric statistical inverse problems. Inverse Problems 24 , 3 (2008), 034004, 19.

COTTER, S. L., DASHTI, M., AND STUART, A. M. Approximation of Bayesian inverse problems for PDEs. SIAM J. Numer. Anal. 48 , 1 (2010), 322-345.

CSATÓ, L. Gaussian processes:iterative sparse approximations. If you have discovered material in AURA which is unlawful e.g. breaches copyright, (either yours or that of a third party) or any other law, including but not limited to those relating to patent, trademark, confidentiality, data protection, obscenity, defamation, libel, then please read our Takedown Policy and contact the service immediately., March 2002.

CSATÓ, L., AND OPPER, M. Sparse On-Line Gaussian Processes. Neural Computation 14 , 3 (03 2002), 641-668.

FLORENS, J.-P., AND SIMONI, A. Regularizing priors for linear inverse problems. Econometric Theory 32 , 1 (2016), 71-121.

GARDNER, J., PLEISS, G., WEINBERGER, K. Q., BINDEL, D., AND WILSON, A. G. Gpytorch: Blackbox matrix-matrix gaussian process inference with gpu acceleration. In Advances in Neural Information Processing Systems (2018), S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, Eds., vol. 31, Curran Associates, Inc.

GHOSAL, S., AND VAN DER VAART, A. Convergence rates of posterior distributions for non-i.i.d. observations. Ann. Statist. 35 , 1 (2007), 192-223.

GOLDENSHLUGER, A., AND PEREVERZEV, S. V. On adaptive inverse estimation of linear functionals in Hilbert scales. Bernoulli 9 , 5 (2003), 783-807.

GOLUBEV, G. K., AND KHAS' MINSKII, R. Z. A statistical approach to some inverse problems for partial differential equations. Problemy Peredachi Informatsii 35 , 2 (1999), 51-66.

GUGUSHVILI, S., VAN DER VAART, A., AND YAN, D. Bayesian linear inverse problems in regularity scales. Annales de l'Institut Henri Poincaré, Probabilités et Statistiques 56 , 3 (2020), 2081 - 2107.

HALMOS, P. R. A Hilbert space problem book , vol. 19. Springer Science & Business Media, 2012.

HEGDE, P., YILDIZ, Ç., LÄHDESMÄKI, H., KASKI, S., AND HEINONEN, M. Variational multiple shooting for bayesian odes with gaussian processes. In Uncertainty in Artificial Intelligence (2022), PMLR, pp. 790-799.

JOHNSTONE, I. M., AND RAIMONDO, M. Periodic boxcar deconvolution and Diophantine approximation. Ann. Statist. 32 , 5 (2004), 1781-1804.

JOHNSTONE, I. M., AND SILVERMAN, B. W. Speed of estimation in positron emission tomography and related inverse problems. Ann. Statist. 18 , 1 (1990), 251-280.

KNAPIK, B., AND SALOMOND, J.-B. A general approach to posterior contraction in nonparametric inverse problems. Bernoulli 24 , 3 (2018), 2091 - 2121.

KNAPIK, B. T., SZABÓ, B. T., VAN DER VAART, A. W., AND VAN ZANTEN, J. H. Bayes procedures for adaptive inference in inverse problems for the white noise model. Probab. Theory Related Fields 164 , 3-4 (2016), 771-813.

KNAPIK, B. T., VAN DER VAART, A. W., AND VAN ZANTEN, J. H. Bayesian inverse problems with Gaussian priors. Ann. Statist. 39 , 5 (2011), 2626-2657.

KNAPIK, B. T., VAN DER VAART, A. W., AND VAN ZANTEN, J. H. Bayesian recovery of the initial condition for the heat equation. Communications in Statistics-Theory and Methods 42 , 7 (2013), 12941313.

LAW, K., STUART, A., AND ZYGALAKIS, K. Data assimilation. Cham, Switzerland: Springer 214 (2015), 52.

LÁZARO-GREDILLA, M., AND FIGUEIRAS-VIDAL, A. Inter-domain gaussian processes for sparse inference using inducing features. In Advances in Neural Information Processing Systems (2009), Y. Bengio, D. Schuurmans, J. Lafferty, C. Williams, and A. Culotta, Eds., vol. 22, Curran Associates, Inc.

MAESTRINI, L., AYKROYD, R. G., AND WAND, M. P. A variational inference framework for inverse problems. arXiv preprint arXiv:2103.05909 (2021).

MAIR, B. A., AND RUYMGAART, F. H. Statistical inverse estimation in hilbert scales. SIAM Journal on Applied Mathematics 56 , 5 (1996), 1424-1444.

MATTHEWS, A. G. D. G., HENSMAN, J., TURNER, R., AND GHAHRAMANI, Z. On sparse variational methods and the kullback-leibler divergence between stochastic processes. In Proceedings of the 19th International Conference on Artificial Intelligence and Statistics (Cadiz, Spain, 09-11 May 2016), A. Gretton and C. C. Robert, Eds., vol. 51 of Proceedings of Machine Learning Research , PMLR, pp. 231-239.

MENG, R., AND YANG, X. Sparse gaussian processes for solving nonlinear pdes. arXiv preprint arXiv:2205.03760 (2022).

NASHED, M. Z., AND WAHBA, G. Generalized inverses in reproducing kernel spaces: An approach to regularization of linear operator equations. SIAM Journal on Mathematical Analysis 5 , 6 (1974), 974-987.

NICKL, R. Bayesian non-linear statistical inverse problems. Lecture Notes ETH Zurich (2022).

NIEMAN, D., SZABO, B., AND VAN ZANTEN, H. Contraction rates for sparse variational approximations in gaussian process regression. Journal of Machine Learning Research 23 , 205 (2022), 1-26.

NIEMAN, D., SZABO, B., AND VAN ZANTEN, H. Uncertainty quantification for sparse spectral variational approximations in gaussian process regression. arXiv preprint arXiv:2212.11031 (2022).

PATI, D., BHATTACHARYA, A., AND CHENG, G. Optimal Bayesian estimation in random covariate design with a rescaled Gaussian process prior. J. Mach. Learn. Res. 16 (2015), 2837-2851.

PINSKI, F. J., SIMPSON, G., STUART, A. M., AND WEBER, H. Algorithms for kullback-leibler approximation of probability measures in infinite dimensions. SIAM Journal on Scientific Computing 37 , 6 (2015), A2733-A2757.

PINSKI, F. J., SIMPSON, G., STUART, A. M., AND WEBER, H. Kullback-leibler approximation for probability measures on infinite dimensional spaces. SIAM Journal on Mathematical Analysis 47 , 6 (2015), 4091-4122.

POVALA, J., KAZLAUSKAITE, I., FEBRIANTO, E., CIRAK, F., AND GIROLAMI, M. Variational bayesian approximation of inverse problems using sparse precision matrices. Computer Methods in Applied Mechanics and Engineering 393 (2022), 114712.

QUIÑONERO CANDELA, J., AND RASMUSSEN, C. E. A unifying view of sparse approximate gaussian process regression. J. Mach. Learn. Res. 6 (dec 2005), 1939-1959.

RASMUSSEN, C. E., AND WILLIAMS, C. K. I. Gaussian processes for machine learning . Adaptive Computation and Machine Learning. MIT Press, Cambridge, MA, 2006.

RAY, K. Bayesian inverse problems with non-conjugate priors. Electronic Journal of Statistics 7 , none (2013), 2516 - 2549.

RAY, K., AND SZABÓ, B. Variational bayes for high-dimensional linear regression with sparse priors. Journal of the American Statistical Association 117 , 539 (2022), 1270-1281.

ROUSSEAU, J., AND SZABO, B. Asymptotic behaviour of the empirical Bayes posteriors associated to maximum marginal likelihood estimator. The Annals of Statistics 45 , 2 (2017), 833 - 865.

ROUSSEAU, J., AND SZABO, B. Asymptotic frequentist coverage properties of Bayesian credible sets for sieve priors. The Annals of Statistics 48 , 4 (2020), 2155 - 2179.

SEEGER, M. Bayesian gaussian process models: Pac-bayesian generalisation error bounds and sparse approximations. Tech. rep., University of Edinburgh, 2003.

SEEGER, M. Pac-bayesian generalisation error bounds for gaussian process classification. J. Mach. Learn. Res. 3 , null (mar 2003), 233-269.

SEEGER, M. W., WILLIAMS, C. K. I., AND LAWRENCE, N. D. Fast forward selection to speed up sparse gaussian process regression. In Proceedings of the Ninth International Workshop on Artificial Intelligence and Statistics (03-06 Jan 2003), C. M. Bishop and B. J. Frey, Eds., vol. R4 of Proceedings of Machine Learning Research , PMLR, pp. 254-261. Reissued by PMLR on 01 April 2021.

SNELSON, E., AND GHAHRAMANI, Z. Sparse gaussian processes using pseudo-inputs. In Advances in Neural Information Processing Systems (2005), Y. Weiss, B. Schölkopf, and J. Platt, Eds., vol. 18, MIT Press.

SNIEKERS, S., AND VAN DER VAART, A. Adaptive Bayesian credible sets in regression with a Gaussian process prior. Electronic Journal of Statistics 9 , 2 (2015), 2475 - 2527.

STUART, A. M. Inverse problems: a bayesian perspective. Acta numerica 19 (2010), 451-559.

SZABÓ, B., VAN DER VAART, A. W., AND VAN ZANTEN, H. Frequentist coverage of adaptive nonparametric bayesian credible sets. Ann. Statist. 43 , 4 (2015), 1391-1428. (with discussion).

TIKHONOV, A. N. Solution of incorrectly formulated problems and the regularization method. Soviet Math. 4 (1963), 1035-1038.

TITSIAS, M. Variational learning of inducing variables in sparse gaussian processes. In Proceedings of the Twelth International Conference on Artificial Intelligence and Statistics (Hilton Clearwater Beach Resort, Clearwater Beach, Florida USA, 16-18 Apr 2009), D. van Dyk and M. Welling, Eds., vol. 5 of Proceedings of Machine Learning Research , PMLR, pp. 567-574.

TITSIAS, M. K. Variational model selection for sparse gaussian process regression. Report, University of Manchester, UK (2009).

TRAVIS, L., AND RAY, K. Pointwise uncertainty quantification for sparse variational gaussian process regression with a brownian motion prior. NeurIPS 2023 (2023).

VAKILI, S., SCARLETT, J., SHIU, D.-S., AND BERNACCHIA, A. Improved convergence rates for sparse approximation methods in kernel-based learning. In International Conference on Machine Learning (2022), PMLR, pp. 21960-21983.

VAN DER VAART, A., AND VAN ZANTEN, H. Information rates of nonparametric Gaussian process methods. J. Mach. Learn. Res. 12 (2011), 2095-2119.

VAN DER VAART, A. W., AND VAN ZANTEN, J. H. Rates of contraction of posterior distributions based on Gaussian process priors. Ann. Statist. 36 , 3 (2008), 1435-1463.

WANG, Y., AND BLEI, D. M. Frequentist consistency of variational Bayes. J. Amer. Statist. Assoc. 114 , 527 (2019), 1147-1161.

YANG, Y., PATI, D., AND BHATTACHARYA, A. α -variational inference with statistical guarantees. Ann. Statist. 48 , 2 (2020), 886-905.

YANG, Y., AND TOKDAR, S. T. Minimax-optimal nonparametric regression in high dimensions. The Annals of Statistics 43 , 2 (2015), 652 - 674.

ZHANG, F., AND GAO, C. Convergence rates of variational posterior distributions. The Annals of Statistics 48 , 4 (2020), 2180 - 2207.

Peer reviews:
comments: SUMMARY:
This paper analyzes posterior contraction rates for the inducing variables variational Bayes approximation for Gaussian process. In particular, mildly and ill-posed inverse problem settings, which are characterized by profiles of eigenvalues of measurement operators, are considered. For the both cases, the posterior contract rates are assessed theoretically. The usefulness of the results are shown for applications to three example problems.

STRENGTHS:
The posterior contract rates are assessed analytically for the inducing variable variational Bayes approximation of Gaussian process. This is useful to determine the optimal number of inducing variables in the approximation.

WEAKNESSES:
The analysis relies on the assumption that the functional space where the target function belongs is known. However, there are many practical situations where such an assumption does not hold. The analysis is not applicable for such situations.

QUESTIONS:
What happens if the target function does not belong to the assumed functional space?

LIMITATIONS:
Yes.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-06-30

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a theoretical analysis of the posterior contraction rates for the variational posterior in the context of linear inverse problems. Specifically, the variational posterior is obtained using the sparse Gaussian processes approach, with inducing variables learned through KL divergence minimization. The authors investigate three types of operators: the Volterra operator, the heat equation, and the Radon transform, each exhibiting varying degrees of ill-posedness. The numerical analysis focuses on synthetic toy problems, and the authors demonstrate that in this case, even a small number of inducing points can yield highly accurate results, achieving significant speed-up.\n\nSTRENGTHS:\n[Originality] I am not up-to-date with theoretical results with frequentist-Bayes analysis, and how far the presented results are based on or deviates from the monograph Nickl, Richard, 2023.  \n\nNickl, Richard. \"Bayesian non-linear statistical inverse problems.\" (2023).\n\n[Quality] The theoretical results seems sound. \n\n[Clarity] The paper is well-written; however, the use of jargon limits its accessibility to a narrower audience.\n\n[Significance] The inducing-point of Gaussian processes is widely used, so the theoretical results concerned with it should be of interest to the community working on inverse problems.\n\nWEAKNESSES:\n[Quality] The experimental evaluation is limited to a synthetic toy example only.  \n\n[Significance] The authors motivate the variational Bayes approach for computational reasons. However, recent developments of scalable GP methods can reduce the complexity of exact GP from O(n^3) to O(n^2) and effectively leverage the modern computing hardware such as GPU (Gardner, Jacob, et al.  2018). In this regard, one can afford many more inducing points in practice. \n\nGardner, Jacob, et al. \"Gpytorch: Blackbox matrix-matrix gaussian process inference with gpu acceleration.\" *Advances in neural information processing systems* 31 (2018).\n\nThe paper is a theory paper, but its practical significance appears to be lacking. It would greatly enhance the paper's value if the authors could at least mention some real-world applications in which the studied inverse problem commonly arises. For instance, discussing the application of the Radon transform in computed tomography (CT) could serve as a relevant and illustrative example.\n\nQUESTIONS:\nFigure 1 shows that using m=3 is insufficient, while m=6 yields highly accurate results for the studied case, while this is a single case not quite informative. I am wondering whether it is possible to establish a \"phase-transition curve\" that can inform practitioners about the required number of inducing points for various types of inverse problems and different numbers of measurements. \n\nIf the results are not understood correctly, it might leads to poor choice on the number of inducing points,  and over-or-under estimate of the posterior uncertainty. This could be harmful in mission-critical applications such as medical imaging. How to prevent this happen? \n\nCould you please further explain why the posterior uncertainty being over-estimated while in traditional variational approximation the uncertainty usually got underestimated? How well is the uncertainty calibrated?\n\nLIMITATIONS:\nThe authors do have discussed to what extend the presented results can be extended to nonlinear cases and other types of inverse problems.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

comments: SUMMARY:
This paper analyses posterior contraction rates in Variational GP for mildly and severely ill-posed linear inverse problems. The authors prove that under certain conditions variational posterior achieves same contraction rates as the true posterior. Moreover, they show that given enough inducing points (in spectral domain) the above-mentioned condition holds.
As specific examples, the minimal number of inducing points needed for optimal contraction and contraction rate is derived for two mildly ill-posed problems (Volterra operator, Radon transformations) and one severely ill-posed problem (heat equation).

STRENGTHS:
Analysis of posterior contraction rates improves our theoretical understanding of VGP for linear inverse problems.

The presentation is well structured.

The contribution is clear and well argued for.

WEAKNESSES:
The experimental part in the main manuscript is very limited, with most experiments taken out to the appendix.  

I believe the paper is more suited for a journal format. It is very technical and requires a thorough review which is hardly possible in a conference format.

QUESTIONS:
Suggestions:
1. Increase readability of figure 1. The legend way too small.

LIMITATIONS:
The limitations are adequately addressed.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis theoretical paper addresses the problem of inferring a function $f$ when\n\n$y(x_i) = \\[Af\\](x_i) + \\epsilon_i$ with $i=1, ...,n $\n\nand $A$ is suitably defined linear operator with $\\epsilon_i$ being i.i.d Gaussian noise. In words this is a linear inverse problem.\n\nA (standard) Bayesian approach is adopted by putting a Gaussian process prior on $f$ with a carefully chosen covariance function.\n\nThen an interdomain variational inducing point approach is adopted. All this so far is standard and pre-existing.\n\nThe contribution of the paper is to derive posterior contraction rates for this setup under certain assumptions and to apply it some specific examples with different choices of the the linear operator $A$. Specifically these are a volterra operator, a diffusion equation and a Radon transform.\n\nSTRENGTHS:\nThis is an important and challenging topic. The standard regression case has been studied by Nieman et al. 2022 which corresponds to taking the $A$ to be the identity in this paper. The extension is certainly of interest since the Bayesian approach to inverse problems is a thriving and important area. \n\nThe required extension is treated well by the authors. The particular challenges of introducing an operator $A$ are clearly described and handled. The results derived require significant technical heavy lifting. \n\nI was not able to go through the proof in detail but the results and approach look sensible and I suspect it to be correct. \n\nThe examples equations chosen were interesting and informative. \n\nThe numerical simulations, if brief, were nonetheless adequate in a paper of this type and compelling.\n\nLimitations were communicated where they arose.\n\nWEAKNESSES:\n**The paper requires strong assumptions on the kernel of the Gaussian process.** To some extent these are forced by theory since a very mispecified covariance function would not contract. But, as the authors clearly concede, there is a more general set of covariance functions that would work than the ones considered.\n\n**The two methods of choosing the inducing variables are somewhat idealized.** For a mildly ill-posed problem and the empirical spectral feature method the complexity is $O(N^2 M)$. Since $M$ the number of inducing points typically needs to be an increasing function of $N$ the number of data points then this is not much of an improvement on the exact inference which is $O(N^3)$ in practice and $O(N^{2.3})$ in theory. The authors do mention this. Alternatively the population method of choosing inducing variables uses $O(NM^2)$ but requires oracle knowledge of the eigensystem of $A^*A$. Admittedly the authors do show some interesting examples where they are able to do this. The required $M$ is more benign for the strongly ill-posed system and correspondingly the heat equation example given is quite compelling.  \n\n**The inducing inputs are not optimized** This is known to make a big difference in the more practical part of the literature particularly in higher input dimensions. It is also one of the things that distinguishes the Titisias variational approach from the Deterministic Training Conditional (DTC) approach using the nomenclature of this paper https://www.jmlr.org/papers/volume6/quinonero-candela05a/quinonero-candela05a.pdf. \n\n**The literature review is not complete.** \n\nThere are two significant papers from Pinski et al. Specifically: \n\n*Kullback--Leibler Approximation for Probability Measures on Infinite Dimensional Spaces. Pinski, Simpson, Stuart and Weber. 2015*\n\n*Algorithms for Kullback--Leibler Approximation of Probability Measures in Infinite Dimensions. Pinski, Simpson, Stuart and Weber. 2015*\n\nThese papers are early, rigorous and are in a similar area to the submission. The method is distinct since they work principally with the inverse covariance operator. Importantly they also address the challenging non-linear case. The type of guarantee derived is also different since they do not derive contraction directly. Instead in the second they use their approximating for an MCMC proposal and gain asymptotic guarantees that way. Nonetheless these papers should be discussed.\n\nA historical note. The work of *Burt et al. 2019* is excellent and highly impactful but perhaps a bit too much is attributed to it here. It provides *rates* of convergence to the posterior but there was some work on convergence before this. This is largely because convergence follows straightforwardly from the variational formulation. *Titsias 2009* (the longer version) Section 2.3 has some discussion of improvement in terms of KL. This was formulated in terms similar to those used in the present submission by *Matthews et al. AISTATS 2015 On Sparse variational methods and the Kullback-Leibler divergence between stochastic processes.* The AISTATS paper mentioned also references several earlier foundational papers and the present submission would benefit from mentioning them too. For instance the early work authored by *Opper*, *Seeger* and *Csato* amongst others. None of these existing papers removes the main contribution of the current work. \n\n**Smaller points/ typos:**\n\nFor the mildly ill-posed case described in Definition 1 shouldn't the decay be polymonial rather than exponential? \n\nOn line 346 I believe the units need to be milliseconds for the description to make sense. So I think the typo is that \"50.5 m\" should be \"50.5 ms\".\n\nQUESTIONS:\nI have no questions other than the ones mentioned above.\n\nLIMITATIONS:\nThere are no negative impacts emerging from this work that I can see.\n\nLimitations are adequately discussed in the submission for the most part. The \"weaknesses\" section of this review has further discussion.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose an inducing-point variational Bayes approximation for linear inverse problems. They study the frequentist properties of the arising variational posterior approximation. Specifically, they prove contraction rates for the variational posterior, both in the mildly and severely ill posed case. Moreover, they illustrate the practical relevance of their results with a number of synthetic data experiments.\n\nSTRENGTHS:\nThe paper is well-written and its objectives and scope are clear. It provides an important contribution. As far as I am aware, frequentist properties of variational posteriors arising as inducing-point approximations in linear inverse problems have not been studied explicitly before. Such properties and contraction rates are useful in that they illustrate how the choice of the prior and the number of inducing points in the approximation impacts the statistical properties of the variational posterior. In that sense, they can provide useful guidance for the user.\n\nWEAKNESSES:\nI am not exactly sure if the comparison to the existing literature provided by the authors is substantial enough – I address this among my questions to the authors.\n\nQUESTIONS:\nIn 2020, three papers were published in the Annals of Statistics providing a comprehensive and general study of the frequentist properties of variational posteriors. Two of those papers are cited by the authors as references [2] and [56]. Another one I would add to this list is “$\\alpha$-VARIATIONAL INFERENCE WITH STATISTICAL GUARANTEES” by Yang, Pati, Bhattacharya. I wonder if the authors would explain whether the results of those papers could be applied to the variational GP approximations for linear inverse problems, as considered in the authors’ submitted work. If they could, then I would like the authors to give reasons why they decided to study this problem directly rather than applying the theory provided previously in those three papers. If they couldn’t, I would like the authors to briefly describe why.\n\nI also have a few other questions and suggestions. In Definition 1 the authors say that a problem is mildly ill-posed if $\\kappa_j$ has exponential decay. Do the authors mean polynomial decay? Also in Theorem 1, only at the end of the statement do the authors define what it means for a posterior (or a variational posterior) to contract at a certain rate. I would include a definition of what it means to contract at a certain rate before stating Theorem 1. Moreover, I’m not sure if the notation $S_{++}^m$ used at the bottom of page 4 and the notation $\\Pi_u$ used at the top of page 5 are introduced anywhere in the paper before they are used.\n\nLIMITATIONS:\nA limitation of this work concerns its practical usability. For example, the authors’ results hold under the assumption that the model is well-specified. It is hard to expect that such an assumption will be satisfied in real-life applications and for real data sets. I believe that the authors should acknowledge this limitation and perhaps comment on possible extensions to instances of model misspecification.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-26"}