Learning the optimal Tikhonov regularizer for inverse problems

1 Introduction . The aim of an inverse problem is to recover information about a physical quantity from indirect measurements . Virtually all imaging problems and modalities fall within this framework , including denoising , deblurring [ 14 ] , computed tomography [ 41 ] and magnetic resonance imaging [ 15 ] . Classical and general approaches to solve inverse problems consist in studying a variational ( minimization ) problem and can be divided into two classes . The first is based on the so-called regularization theory [ 14 ] . The aim is to recover a single , deterministic , unknown x† from noisy data y = F ( x† ) + ε by solving a minimization problem min x dY ( F ( x ) , y ) + J ( x ) ( 1 ) 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . for a fidelity term dY : Y × Y → R and a regularization functional J : X → [ 0 , +∞ ) . The latter is chosen in order to mitigate the ill-posedness of the map F , and represent some a-priori knowledge on x . For instance , in the classical Tikhonov regularization we have J ( x ) = λ‖x‖2X for λ > 0 . The second approach considers the unknown as a random variable and is based on statistical/Bayesian methods [ 23 , 45 ] . In this case one can recover the unknown using point estimators such as the maximum a posteriori ( MAP ) estimator , or extract richer information on the probability distribution of the unknown . In practice , the MAP estimator is found by solving a minimization problem of the same form as ( 1 ) . The main difference is that the fidelity term and the regularizer are tailored to the statistical properties of the unknown and the noise , which are usually assumed to be known . In recent years , machine learning techniques , and especially deep learning , have shaken the field of inverse problems by providing the basis for data-driven methods that have outperformed the state-of-the-art in most imaging modalities [ 3 , 43 ] . The most successful methods take inspiration from regularization theory [ 10 , 1 , 26 , 2 , 36 , 21 , 22 , 27 , 34 , 39 , 8 , 17 , 40 ] : the physical model given by the forward map F is assumed to be known while the regularizer ( or the gradient updates related to it ) is learned from a training set . While these approaches have shown impressive results in applications , a solid theory behind their successes is lacking . In view of the many sensitive applications where these methods are already being employed , e.g . in medical imaging , it is of utmost importance to fill this theoretical gap to better understand the strengths and limits of data-driven imaging modalities . Moreover , many inverse problems are naturally formulated in infinite-dimensional spaces [ 14 , 41 , 23 , 15 , 45 , 37 ] , and their discretization must be carefully treated due to their ill-posedness [ 25 ] . Hence , it is of main interest to provide a theoretical analysis in the infinite-dimensional setting . In this work , we consider the problem of learning a regularizer for a linear inverse problem in the framework of statistical learning theory , which is the natural setting to derive precise theoretical guarantees . This is part of the growing research area of learning an operator between infinite dimensional spaces [ 46 , 13 , 32 , 42 , 31 ] . We study the case where the measurements are modeled by a linear , possibly ill-posed , forward map , and the penalty term is a generalized Tikhonov regularizer [ 47 ] . More precisely , let A : X → Y be a bounded linear operator between the separable real Hilbert spaces X and Y . We consider the inverse problem y = Ax+ ε , ( 2 ) which consists of the reconstruction of x from the knowledge of y , where ε represents noise . We assume that x is a random variable onX with mean µ and covariance Σx , and the noise ε , independent of the variable x , is a zero-mean random process on Y with covariance Σε , ( see Section 2 for more details ) . The operator A is typically injective but its inverse may be unbounded : typical examples include denoising ( A is the identity ) and deblurring ( A is a convolution operator ) . We aim to recover the unknown via generalized Tikhonov regularization . For a quadratic fidelity term dY : Y × Y → R , the minimization problem min x dY ( Ax , y ) + ‖B−1 ( x− h ) ‖2X , ( 3 ) has a unique solution Rh , B ( y ) , called the generalized Tikhonov reconstruction . Here the pair ( h , B ) , where h ∈ X and B : X → X is a positive bounded operator , is considered as a free parameter that we call regularization pair . For example , the operator B can be a smoothing operator in L2 , as a negative power of the Laplacian , so that B−1 is a differential operator , yielding a classical regularization in Sobolev spaces . We want to characterize and learn the optimal pair ( h , B ) with respect to the expected , or mean squared , error L ( h , B ) = Ex , y‖Rh , B ( y ) − x‖2X . Our first contribution is a complete characterization of the minimizers of L. In particular we find that ( µ , Σ1/2x ) is a global minimizer ( and is unique if A is injective ) , which shows that the best regularizer is completely independent of the forward operator A and depends only on the mean and covariance of x . This is consistent with the known linearized minimum mean squared error estimator in the finite-dimensional case [ 24 ] , but it is usually not taken into account in the machine learning approaches to inverse problems mentioned above . The extension to the infinite-dimensional case is not straightforward due to the presence of unbounded operators in inverse problems . Since the computation of the expected error requires the full distribution ρ of x and y , we study how this can be approximated from a finite training set . We suppose to have access to a sample of m pairs z = { ( xj , yj ) } mj=1 drawn independently from ρ . In view of the results on the expected error , we consider two alternative ways to learn a regularizer pair ( ĥz , B̂z ) from a training set z : either by minimizing the empirical risk [ 12 ] ( supervised learning ) , or by using the empirical mean and covariance of { xj } ( unsupervised learning ) . In both cases , we prove generalization bounds for the sample error |L ( ĥz , B̂z ) − L ( µ , Σ1/2x ) | . Under some natural compactness assumptions on the class of regularization pairs , we prove that the sample error has the asymptotic behavior |L ( ĥz , B̂z ) − L ( µ , Σ1/2x ) | . 1√ m , ( 4 ) with high probability , in both the supervised and the unsupervised approaches . We stress the point that these bounds hold in the infinite-dimensional setting , or , in other words , they do not depend on the discretization of the signal and of the measurements . Finally , we complement our theoretical findings with some numerical experiments . For a 1D denoising problem ( i.e . A is the identity operator ) we replicate the asymptotic bound ( 4 ) at different discretization scales . Moreover , we find that the unsupervised approach , despite yielding the same rate ( 4 ) , clearly outperforms the supervised one . The paper is organized as follows . In Section 2 we introduce the main notation and technical assumptions that will be used throughout the paper , including several examples . Section 3 presents the main results for the minimization of the expected error , while Section 4 is devoted to the study of the sample error . Numerical experiments are the subject of Section 5 . Concluding remarks and discussions are reserved for Section 6 . 2 Setting the stage . 2.1 The random objects x and ε . As mentioned in the introduction , we formulate ( 2 ) as a statistical inverse problem , where x and ε are not deterministic but random . Let us start with the description of the prior on x . Assumption 2.1 . Let x be a random variable on a probability space ( Ω , P ) taking values in X . More precisely , x is square-integrable , so that its expectation µ ∈ X and its covariance Σx : X → X is a trace-class operator . We assume that Σx is injective . Without loss of generality we can always assume that Σx is injective ( i.e. , x is non-degenerate ) , since otherwise it would be enough to consider the inverse problem only in ( ker Σx ) ⊥ ( X . Let us consider some common examples of priors arising in inverse problems . Example 2.2 ( Gaussian random variables ) . A general class of priors arises when considering Gaussian random variables . We recall that x is a Gaussian random variable if for all v ∈ X , 〈x , v〉 is a real Gaussian random variable and , by Fernique ’ s theorem , x is square-integrable [ 7 ] . Since Σx : X → X is self-adjoint , positive and trace-class , we can write its singular value decomposition ( SVD ) as Σxv = ∑ k σ2k〈v , ek〉Xek , v ∈ X , where { ek } k is an orthonormal basis of X , ∑ k σ 2 k < +∞ and 〈x , ek〉 ∼ N ( µk , σk ) , where µ = ∑ k µkek is the mean of x . In other words , x = µ+ ∑ k σkakek , ( 5 ) where ak are i.i.d . standard Gaussian variables . This shows that , in infinite dimension , since σk → 0 , the variations of x along the direction ek become smaller and smaller as k → +∞ . This abstract construction reduces to a smoothness prior by suitably choosing the covariance operator . Example 2.3 ( Smoothing priors ) . Let X = L2 ( Td ) , where Td = Rd/Zd is the d-dimensional torus and d ≥ 1 . Let ∆ denote the Laplace-Beltrami operator on Td , which is simply the classical Laplace operator on [ 0 , 1 ] d with periodic boundary conditions . For s > d2 , the operator ( I −∆ ) −s : L2 ( Td ) → L2 ( Td ) is trace class , and can be used to define the Gaussian distribution N ( 0 , ( I −∆ ) −s ) . In the notation of Example 2.2 , the SVD of ( I −∆ ) −s is given by σ2k = ( 1 + 4π|k|2 ) −s and ek ( t ) = e2πik·t with k ∈ Zd . This enforces a smoothness prior on x , depending on the parameter s , which controls the decay of the Fourier coefficients of x ( see [ 33 , Appendix B ] and [ 7 ] ) . Let us now discuss the model for the noise ε . Assumption 2.4 . Let ε = ( ε ) v∈Y be a ( linear ) random process on Y with zero mean and such that its covariance Σε : Y → Y defined by E [ εvεw ] = 〈Σεv , w〉Y is bounded and injective . Notice that the injectivity of Σε implies that noise is present in all directions of Y , whereas the boundedness of Σε allows us to regard the random process ε as a bounded ( linear ) operator from Y into L2 ( Ω , P ) . The reader is referred to [ 16 ] and to Appendix A.1 for additional details on random processes in Hilbert spaces . We mention here some basic properties that will be needed for the following discussion . Remark 2.5 . Even if ε may not belong to Y almost surely ( see Example 2.7 below ) , it is always possible to view it as an element of a larger space , as we now discuss . Let K be a separable Hilbert space and ι : K → Y be an injective linear map such that ι ( K ) is dense in Y and ι∗ ◦ Σε ◦ ι : K → K∗ is trace-class,1 ( 6 ) where we identify Y ∗ = Y , but we do not identify K∗ with K , and we regard ι∗ as the canonical embedding Y → K∗ . The restriction of ε to K is a Hilbert-Schmidt operator from K into L2 ( Ω , P ) , hence there exists a unique square-integrable random vector ε taking values in K∗ such that εv = 〈ε , v〉K∗×K for v ∈ K. It is easy to show that the random vector ε has zero mean and its covariance operator is ι∗ ◦ Σε ◦ ι : K → K∗ , since E [ 〈ε , v〉K∗×K〈ε , w〉K∗×K ] = E [ ει ( v ) ει ( w ) ] = 〈Σειv , ιw〉Y , v , w ∈ K. ( 7 ) A random variable is always a random process , as we now describe . Example 2.6 . A simple example of this abstract construction consists of considering a random variable ε . In this case Σε is trace-class itself , so that we can choose K = Y and ι = I and ε ∈ Y almost surely . As discussed in Example 2.2 for Gaussian variables , this means that , since σk → 0 , the expected amplitude of the noise in the direction ek goes to 0 as k →∞ . For instance , the choice of Σε = ( I −∆ ) −s as in Example 2.3 corresponds to smaller noise levels for higher Fourier modes . A random process allows for considering noise that is uniformly distributed in all directions . Example 2.7 ( White noise ) . The Gaussian white noise ε is a random process on Y such that for any v ∈ Y it holds that εv is a standard Gaussian variable ( mean 0 and variance 1 ) , so that Σε = I . Heuristically , in the notation of Example 2.2 , this corresponds to σk = 1 for every k , and so by ( 5 ) ε = ∑ k akek , ak ∼ N ( 0 , 1 ) , so that ε /∈ Y with probability 1 whenever Y is infinite dimensional ( see , e.g. , [ 16 ] ) . In view of Remark 2.5 , it is possible to consider a larger space K∗ so that ε ∈ K∗ almost surely . For concreteness of explanation , we focus on the case when Y = L2 ( Td ) , a typical framework in imaging . A possible choice for the space K is the Sobolev space Hs ( Td ) with s > d/2 ( see [ 25 ] ) , so that the canonical embedding ι : Hs ( Td ) → L2 ( Td ) is a Hilbert-Schmidt operator , hence ( 6 ) is satisfied and ε can naturally be seen as an element of H−s ( Td ) = Hs ( Td ) ∗ .

Summary: The paper studies the linear MMSE in infinite-dimensional systems. Specifically, given an observation $Ax +\varepsilon$ where $A$ is a known linear operator and $\varepsilon$ is some form of noise, one wants to estimate $x$, which minimizes the mean squared error.  Focusing on regularization, the paper asks what is the optimal regularizer with respect to the mean squared error. It is shown that, when constrained to Tikhonov regularization, the optimal regularizer has a rather simple form which depends only on the statistical properties of the system and, in particular, is independent from the operator $A$.   The authors also address the problem of approximating the regularizer, based on a finite sample. They show two approximations schemes: A "supervised-learning" scheme, based on minimizing the empirical risk and an "unsupervised-learning" scheme, which is based on the particular form the optimal regularizer was found to have.  For both schemes, the authors show that as the sample size $m$ tend to infinity, the approximation error decays at the expected rate $\frac{1}{\sqrt{m}}$.