SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Variational Gaussian Processes For Linear Inverse Problems
Thibault Randrianarisoa
Department of Decision Sciences
via Roentgen 1, 20136, Milano, MI, Italy
thisbault.randrianarisoa@unibocconi.it
Botond Szabo
botond.szabo@unibocconi.it
By now Bayesian methods are routinely used in practice for solving inverse problems. In inverse problems the parameter or signal of interest is observed only indirectly, as an image of a given map, and the observations are typically further corrupted with noise. Bayes offers a natural way to regularize these problems via the prior distribution and provides a probabilistic solution, quantifying the remaining uncertainty in the problem. However, the computational costs of standard, sampling based Bayesian approaches can be overly large in such complex models. Therefore, in practice variational Bayes is becoming increasingly popular. Nevertheless, the theoretical understanding of these methods is still relatively limited, especially in context of inverse problems. In our analysis we investigate variational Bayesian methods for Gaussian process priors to solve linear inverse problems. We consider both mildly and severely ill-posed inverse problems and work with the popular inducing variables variational Bayes approach proposed by Titsias [57]. We derive posterior contraction rates for the variational posterior in general settings and show that the minimax estimation rate can be attained by correctly tunned procedures. As specific examples we consider a collection of inverse problems including the heat equation, Volterra operator and Radon transform and inducing variable methods based on population and empirical spectral features.
In inverse problems we only observe the object of interest (i.e. function or signal) indirectly, through a transformation with respect to some given operator. Furthermore, the data is typically corrupted with measurement error or noise. In practice the inverse problems are often ill-posed, i.e. the inverse of the operator is not continuous. Based on the level of ill-posedness we distinguish mildly and severely ill-posed cases. The ill-posedness of the problem prevents us from simply inverting the operator as it would blow up the measurement errors in the model. Therefore, to overcome this problem, regularization techniques are applied by introducing a penalty term in the maximum likelihood approximation. Standard examples include generalized Tikhonov, total variation and Moore-Penrose estimators, see for instance [4; 5; 9; 11; 18; 56] or a recent survey [3] on data-driven methods for solving inverse problems
An increasingly popular approach to introduce regularity to the model is via the Bayesian paradigm, see for instance [3; 7; 12; 29; 54] and references therein. Beside regularization, Bayesian methods provide a probabilistic solution to the problem, which can be directly used to quantify the remaining uncertainty of the approach. This is visualised by plotting credible sets, which are sets accumulating prescribed percentage of the posterior mass. For computing the posterior typically MCMC algorithms are used, however, these can scale poorly with increasing sample size due to the complex structure of the likelihood. Therefore, in practice often alternative, approximation methods are used. Variational
Bayes (VB) casts the approximation of the posterior into an optimization problem. The VB approach became increasingly popular to scale up Bayesian inverse problems, see for instance the recent papers [22; 31; 34; 42] and references therein. However, until recently these procedures were considered black box methods basically without any theoretical underpinning. Theoretical results are just starting to emerge [2; 46; 63; 64; 66], but we still have limited understanding of these procedures in complex models, like inverse problems.
In our analysis we consider Gaussian process (GP) priors for solving linear inverse problems. For Gaussian likelihoods, due to conjugacy, the corresponding posterior has an analytic form. Nevertheless, they are applied more widely, in non-conjugate settings as well. However, training and prediction even in the standard GP regression model, scales as  $O(n^3)$  (or  $O(n^2)$ ) for exact inference in the recent paper [16] leveraging advances in computing hardware) and  $O(n^2)$ , respectively, which practically limits GPs to a sample size  $n$  of order  $10^4$ . Therefore, in practice often not the full posterior, but an approximation is computed. Various such approximation methods were proposed based on some sparse or low rank structure, see for instance [13; 14; 33; 43; 49; 50; 51; 52; 57]. Our focus here lies on the increasingly popular inducing variable variational Bayes method introduced in [57; 58].
In our work we extend the inducing variable method for linear inverse problems and derive theoretical guarantees for the corresponding variational approximations. More concretely we adopt a frequentist Bayes point-of-view in our analysis by assuming that there exists a true data generating functional parameter of interest and investigate how well the variational posterior can recover this object. We derive contraction rates for the VB posterior around the true function both in the mildly and severely ill-posed inverse problems. We then focus on two specific inducing variable methods based on the spectral features of the prior covariance kernel. We show that for both methods if the number of inducing variables are chosen large enough for appropriately tunned priors the corresponding variational posterior concentrates around the true function with the optimal minimax estimation rate. One, perhaps surprising aspect of the derived results is that the number of inducing variables required to attain the optimal, minimax contraction rate is sufficiently less in the inverse setting than in the direct problem. Therefore, inverse problems can be scaled up at a higher degree than standard regression models.
Related literature. The theory of Bayesian approaches to linear inverse problems is now well established. The study of their asymptotic properties started with the study of conjugate priors [1; 15; 20; 26; 27; 28] before addressing the non-conjugate case [25; 45] and rate-adaptive priors [26; 55]. By now we have a good understanding of both the accuracy of the procedure for recovering the true function and the reliability of the corresponding uncertainty statements. The theory of Bayesian non-linear inverse problems is less developed, but recent years have seen an increasing interest in the topic, see the monograph [36] and references therein. Some algorithmic developments for variational Gaussian approximations in non-linear inverse problems, and applications to MCMC sampling, can be found in [40; 41].
The inducing variable approach for GPs proposed by [57; 58] has been widely used in practice. Recently, their theoretical behaviour was studied in the direct, nonparametric regression setting. In [8] it was shown that the expected Kullback-Leibler divergence between the variational class and posterior tends to zero when sufficient amount of inducing variables were used. Furthermore, optimal contraction rates [37] and frequentist coverage guarantees [38; 59; 60] were derived for several inducing variable methods. Our paper focuses on extending these results to the linear inverse setting.
Organization. The paper is organized as follows. In Section 2 we first introduce the inverse regression model where we carry out our analysis. Then we discuss the Bayesian approach using GPs and its variational approximations in Sections 2.1 and 2.2, respectively. As our main result we derive contraction rates for general inducing variable methods, both in the mildly and severely ill-posed cases. Then in Section 2.3 we focus on two specific inducing variable methods based on spectral features and provide more explicit results for them. We apply these results for a collection of examples, including the Volterra operator, the heat equation and the Radon transform in Section 3. Finally, we demonstrate the applicability of the procedure in the numerical analysis of Section 4 and conclude the paper with discussion in Section 5. The proof of the main theorem together with technical lemmas and additional simulation study are deferred to the supplementary material.
Notation. Let  $C, c$  be absolute constants, independent of the parameters of the problem whose values may change from line to line. For two sequences  $(a_{n})$  and  $(b_{n})$  of numbers,  $a_{n} \lesssim b_{n}$  means that there
exists a universal constant  $c$  such that  $a_{n} \leq cb_{n}$  and we write  $a_{n} \asymp b_{n}$  if both  $a_{n} \lesssim b_{n}$  and  $b_{n} \lesssim a_{n}$  hold simultaneously. We denote by  $a_{n} \ll b_{n}$  if  $|a_{n} / b_{n}|$  tends to zero. The maximum and minimum of two real numbers  $a$  and  $b$  are denoted by  $a \vee b$  and  $a \wedge b$ , respectively. We use the standard notation  $\delta_{ij} = \mathbb{1}_{i = j}$ . For  $m \geq 1$ , we note  $S_{++}^{m}$  the set of positive-definite matrices of size  $m \times m$ .
In our analysis we focus on the non-parametric random design regression model where the functional parameter is observed through a linear operator. More formally, we assume to observe i.i.d. pairs of random variables  $(x_{i},Y_{i})_{i = 1,\ldots ,n}$  satisfying
$$
Y _ {i} = \left(\mathcal {A} f _ {0}\right) \left(x _ {i}\right) + Z _ {i}, \quad Z _ {i} \stackrel {i i d} {\sim} N \left(0, \sigma^ {2}\right), x _ {i} \stackrel {i i d} {\sim} G \quad i = 1, \dots , n, \tag {1}
$$
where  $f_0 \in L_2(\mathcal{T};\mu)$ , for some domain  $\mathcal{T} \subset \mathbb{R}^d$  and measure  $\mu$  on  $\mathcal{T}$ , is the underlying functional parameter of interest and  $\mathcal{A}:L_2(\mathcal{T};\mu)\mapsto L_2(\mathcal{X};G)$ , for the measure  $G = \mathcal{A}\mu$  on  $\mathcal{X}$ , is a known, injective, continuous linear operator. In the rest of the paper we use the notation  $P_{f_0}$  and  $E_{f_0}$  for the joint distribution and the corresponding expectation, respectively, of the data  $(X,Y) = (x_i,Y_i)_{i = 1,\ldots ,n}$ . Furthermore, we denote by  $E_{X},P_{X},E_{Y|X},P_{Y|X}$  the expectation/distribution under  $G^{\otimes n}$  and the law of  $(Y_i)_i$  given the design respectively. Finally, for simplicity we take  $\sigma^2 = 1$  in our computations.
In the following, denoting  $\mathcal{A}^*$  the adjoint of  $\mathcal{A}$ , we assume that the self-adjoint operator  $\mathcal{A}^*\mathcal{A}\colon L_2(\mathcal{T};\mu)\mapsto L_2(\mathcal{T};\mu)$  possesses countably many positive eigenvalues  $(\kappa_j^2)_j$  with respect to the eigenbasis  $(e_j)_j$  (which is verified if  $\mathcal{A}$  is a compact operator for instance). We remark that  $(g_{j})_{j}$  defined by  $Ae_{j} = \kappa_{j}g_{j}$  is an orthonormal basis of  $L_{2}(\mathcal{X};G)$ . We work on the ill-posed problem where  $\kappa_{j}\rightarrow 0$ , the rate of decay characterizing the difficulty of the inverse problem.
Definition 1. We say the problem is mildly ill-posed problem of degree  $p > 0$  if  $\kappa_j \asymp j^{-p}$  has a polynomial decay. In the severely ill-posed problem, the rate we consider is exponential,  $\kappa_j \asymp e^{-c j^p}$  for  $c > 0, p \geq 1$ , and  $p$  is the degree of ill-posedness once again.
In nonparametrics it is typically assumed that  $f_{0}$  belongs to some regularity class. Here we consider the generalized Sobolev space
$$
\bar {H} ^ {\beta} := \left\{f \in L _ {2} (\mathcal {T}; \mu): \| f \| _ {\beta} <   \infty \right\}, \quad \| f \| _ {\beta} ^ {2} = \sum_ {j} j ^ {2 \beta} | \langle f, e _ {j} \rangle | ^ {2}, \tag {2}
$$
for some  $\beta > 0$ . We note that the difficulty in estimating  $f_0$  from the data is twofold: one needs to deal with the observational noise, which is a statistical problem, as well as to invert the operator  $\mathcal{A}$ , which comes from inverse problem theory. As a result of the ill-posedness of the problem, recovering  $f_0$  from the observations may suffer from problems of unidentifiability and instability. The solution to these issues is to incorporate some form of regularization in the statistical procedure. The Bayesian approach provides a natural way to incorporate regularization into the model via the prior distribution on the functional parameter. In fact penalized likelihood estimators can be viewed as the maximum a posteriori estimators with the penalty term induced by a prior. For example Tikhonov type regularizations can be related to the RKHS-norm of a Gaussian Process prior, see [35; 44] for a more detailed discussion.
2.1 Gaussian Process priors for linear inverse problems
We focus on the Bayesian solution of the inverse problem and exploit the Gaussian likelihood structure by considering conjugate Gaussian Process (GP) priors on  $f$ . A GP  $\mathcal{GP}(\eta(\cdot), k(\cdot, \cdot))$  is a set of random variables  $\{f(t) \mid t \in \mathcal{T}\}$ , such that any finite subset follows a Gaussian distribution. The GP is described by the mean function  $\eta$  and a covariance kernel  $k(t, t')$ . We consider centered GPs as priors (i.e. we take  $\eta \equiv 0$ ). Then the bilinear, symmetric nonnegative-definite function  $k: \mathcal{T} \times \mathcal{T} \mapsto \mathbb{R}$  determines the properties of the process (e.g., its regularity). In view of the linearity of the operator  $\mathcal{A}$  the corresponding posterior distribution is also a Gaussian process. The mean and covariance function of the posterior is given by
$$
t \mapsto K _ {t \mathcal {A} f} \left(K _ {\mathcal {A} f \mathcal {A} f} + \sigma^ {2} I _ {n}\right) ^ {- 1} \mathbf {y}, \tag {3}
$$
$$
(t, s) \mapsto k (t, s) - K _ {t \mathcal {A} f} \left(K _ {\mathcal {A} f \mathcal {A} f} + \sigma^ {2} I _ {n}\right) ^ {- 1} K _ {\mathcal {A} f s},
$$
where  $\mathbf{y} = (y_{1},\ldots ,y_{n})^{T}$ ,  $\mathcal{A}\mathbf{f} = \bigl (\mathcal{A}f(x_i)\bigr)_{i = 1,\dots,n}$ ,  $K_{\mathcal{A}\mathbf{f}\mathcal{A}\mathbf{f}} = E_{\Pi}\mathcal{A}\mathbf{f}\mathcal{A}\mathbf{f}^{T}\in \mathbb{R}^{n\times n}$  with  $E_{\Pi}$  denoting the expectation with respect to the GP prior  $\Pi$ ,  $K_{t,\mathcal{A}\mathbf{f}}^T = \big(E_{\Pi}\mathcal{A}f(x_i)f(t)\big)_{i = 1,\dots,n}\in \mathbb{R}^n$  see the supplement for the detailed derivation.
Due to the closed-form expressions for the posterior and the marginal likelihood, as well as the simplicity with which uncertainty quantification may be produced, GP regression has gained popularity [44]. Furthermore, the asymptotic frequentist properties of posteriors corresponding to GP priors in the direct problem, with  $\mathcal{A}$  taken to be the identity operator, is well-established by now. Optimal contraction rates and confidence guarantees for Bayesian uncertainty quantification were derived in the regression setting and beyond, see for instance [10; 39; 47; 53; 48; 61; 62; 65] and references therein. In the following, we say that  $\varepsilon_{n}$  is an  $L_{2}$ -posterior contraction rate for the posterior  $\Pi[\cdot |X,Y]$  if for any  $M_{n} \to 0$
$$
E _ {f _ {0}} \Pi \left[ f \colon \| f - f _ {0} \| _ {L _ {2} (\mathcal {T}; \mu)} \geq M _ {n} \varepsilon_ {n} \mid X, Y \right]\rightarrow 0.
$$
In our analysis we consider covariance kernels with eigenfunctions coinciding with the eigenfunctions of the operator  $\mathcal{A}^*\mathcal{A}$ , i.e. we take
$$
k (t, s) = \sum_ {j} \lambda_ {j} e _ {j} (t) e _ {j} (s), \tag {4}
$$
where  $(\lambda_j)_j$  denote the corresponding eigenvalues. The asymptotic behaviour of the corresponding posterior has been well investigated in the literature both in the mildly and severely ill-posed inverse problems. Rate optimal contraction rates and frequentist coverage guarantees for the resulting credible sets were derived both for known and unknown regularity parameters [15; 26; 27; 28; 55]. These results were further extended for other covariance kernels where the eigenfunctions do not exactly match the eigenfunctions of the operator  $\mathcal{A}$ , but in principle they have to be closely related, see [1; 20; 25; 45].
However, despite the explicit, analytic form of the posterior given in (3) and the theoretical underpinning, the practical applicability of this approach is limited for large sample size  $n$ . The computation of the posterior involves inverting the  $n$ -by-  $n$  matrix  $K_{\mathcal{A}f\mathcal{A}f} + \sigma^2 I_n$ , which has computational complexity  $O(n^{3})$ . Therefore, in practice often not the true posterior, but a scalable, computationally attractive approximation is applied. Our focus here is on the increasingly popular inducing variable variational Bayes method introduced in [57; 58].
2.2 Variational GP for linear inverse problems
In variational Bayes the approximation of the posterior is casted as an optimization problem. First a tractable class of distributions  $\mathcal{Q}$  is considered, called the variational class. Then the approximation  $\Psi^{*}$  is computed by minimizing the Kullback-Leibler divergence between the variational class and the true posterior, i.e.
$$
\Psi^ {*} = \arg \inf  _ {Q \in \mathcal {Q}} K L (Q \| \Pi [ \cdot | X, Y ]).
$$
There is a natural trade-off between the computational complexity and the statistical accuracy of the resulting approximation. Smaller variational class results in faster methods and easier interpretation, while more enriched classes preserve more information about the posterior ensuring better approximations.
In context of the Gaussian process regression model (with the operator  $\mathcal{A}$  taken to be the identity), [58] proposed a low-rank approximation approach based on inducing variables. The idea is to compress the information encoded in the observations of size  $n$  into  $m$  so called inducing variables. We extend this idea for linear inverse problems. Let us consider real valued random variables  $\mathbf{u} = (u_1, \ldots, u_m) \in L_2(\Pi)$ , expressed as measurable linear functionals of  $f$  and whose prior distribution is  $\Pi_u$ . In view of the linearity of  $\mathbf{u}$ , the joint distribution of  $(f, \mathbf{u})$  is a Gaussian process, hence the conditional distribution  $f|\mathbf{u}$  denoted by  $\Pi(\cdot|\mathbf{u})$ , is also a Gaussian process with mean function and covariance kernel given by
$$
t \mapsto K _ {t u} K _ {u u} ^ {- 1} u \quad \text {a n d} \quad (t, s) \mapsto k (t, s) - K _ {t u} K _ {u u} ^ {- 1} K _ {u s},
$$
respectively, where  $K_{t\pmb{u}} = E_{\Pi}(f(t)\pmb{u}) \in \mathbb{R}^m$  and  $K_{\pmb{u}\pmb{u}} = E_{\Pi}(\pmb{u}\pmb{u}^T) \in \mathbb{R}^{m \times m}$ . Then the posterior is approximated via a probability measure  $\Psi_u$  on  $(\mathbb{R}^m, \mathcal{B}(\mathbb{R}^m))$  by  $\Psi = \int \Pi[\cdot|\mathbf{u}] d\Psi_u(\mathbf{u})$ , which is
absolutely continuous against  $\Pi$  and satisfies  $\frac{d\Psi}{d\Pi}(f) = \frac{d\Psi_u}{d\Pi_u}(\mathbf{u}(f))$ . We note that the variables  $\mathbf{u}$  were first considered to be point evaluations of the GP prior process before these ideas were extended to interdomain inducing variables, e.g. integral forms of the process [30; 58].
Taking  $\Psi_{\pmb{u}} = \mathcal{N}(\pmb{\mu}_{\pmb{u}},\Sigma)$  as a multivariate Gaussian, the corresponding  $\Psi \propto \Pi (\cdot |\pmb {u})\Psi_{\pmb{u}}$  is a Gaussian process, with mean and covariance functions
$$
t \mapsto K _ {t u} K _ {u u} ^ {- 1} \boldsymbol {\mu} _ {u}, \quad \text {a n d} \quad (t, s) \mapsto k (t, s) - K _ {t u} K _ {u u} ^ {- 1} \left(K _ {u u} - \Sigma\right) K _ {u u} ^ {- 1} K _ {s u} ^ {T}. \tag {5}
$$
Letting  $\mu_{u}$  and  $\Sigma$  be the free variational parameters, the variational family is taken as
$$
\mathcal {Q} := \left\{\Psi \mid \Psi_ {u} = \mathcal {N} \left(\boldsymbol {\mu} _ {u}, \Sigma_ {u}\right), \boldsymbol {\mu} _ {u} \in \mathbb {R} ^ {m}, \Sigma_ {u} \in S _ {+ +} ^ {m} \right\},
$$
consisting of "  $m$  -sparse" Gaussian processes.
By similar computations as those from [37], it can be shown that  $\Pi[\cdot | X, Y]$  is equivalent to any element of  $\mathcal{Q}$  (they are mutually dominated) so that the  $KL$  divergence is always finite and there exists a  $\Psi_{u}^{*}$ , corresponding to the minimizer  $\Psi^{*}$  of KL  $(\Psi \| \Pi[\cdot | X, Y])$ . Furthermore, we have
$$
\begin{array}{l} \frac {d \Psi^ {*}}{d \Pi} (f) = \frac {d \Psi_ {\boldsymbol {u}} ^ {*}}{d \Pi_ {\boldsymbol {u}}} (\mathbf {u}) \propto e x p \Big (- \frac {1}{2 \sigma^ {2}} \int \sum_ {i = 1} ^ {n} (Y _ {i} - \mathcal {A} f (x _ {i})) ^ {2} d \Pi (f | \mathbf {u}) \Big) \\ \propto e x p \left(- \frac {1}{2 \sigma^ {2}} \sum_ {i = 1} ^ {n} \left(Y _ {i} - K _ {\mathcal {A} f (x _ {i}) \boldsymbol {u}} K _ {\boldsymbol {u u}} ^ {- 1} \boldsymbol {u}\right) ^ {2}\right) \\ \end{array}
$$
where  $K_{\mathcal{A}f(x_i)\pmb{u}} = E_{\Pi}\mathcal{A}f(x_i)\pmb{u}^T$ . One can observe that the parameters of the variational approximations are
$$
\boldsymbol {\mu} _ {u} ^ {*} = \sigma^ {- 2} K _ {u u} \left(K _ {u u} + \sigma^ {- 2} K _ {u A f} K _ {A f u}\right) ^ {- 1} K _ {u A f} \mathbf {y}, \tag {6}
$$
$$
\Sigma_ {u} ^ {*} = K _ {u u} \left(K _ {u u} + \sigma^ {- 2} K _ {u A f} K _ {u, A f}\right) ^ {- 1} K _ {u u},
$$
for  $K_{u\mathcal{A}f} = K_{\mathcal{A}f u}^{T} = E_{\Pi}\mathbf{u}(\mathcal{A}\mathbf{f})^{T}$  the  $m \times n$  matrix whose  $j$ th column is  $K_{\mathcal{A}f(x_i)\mathbf{u}}$  and  $(\mathcal{A}\mathbf{f})^T = (\mathcal{A}f(x_1),\ldots ,\mathcal{A}f(x_n))$ . Then the explicit form for the variational posterior  $\Psi^{*}$  can be attained by plugging in the parameters (6) into the variational mean and covariance function (5). We also define  $Q_{\mathcal{A}f\mathcal{A}f} = K_{u\mathcal{A}f}^{T}K_{uu}K_{u\mathcal{A}f}$ .
We investigate the statistical inference properties of the above variational posterior distribution  $\Psi^{*}$ . More concretely we focus on how well the variational approximation can recover the underlying true functional parameter  $f_{0}$  of interest in the indirect, linear inverse problem (1). We derive contraction rate for  $\Psi^{*}$  both in the mildly and severely ill-posed inverse problem case. Furthermore, we consider both the standard exponential and polynomial spectral structures for the prior, i.e. we assume that the eigenvalues of the prior covariance kernel satisfies either  $\lambda_{j} \asymp j^{-\alpha} e^{-\xi j^{p}}$  or  $\lambda_{j} \asymp j^{-1 - 2\alpha}$  for some  $\alpha \geq 0$ ,  $\xi > 0$ . Finally, in view of [37], we introduce additional assumptions on the covariance kernel of the conditional distribution of  $f|u$  ensuring that the variational posterior is not too far from the true posterior in Kullback-Leibler divergence.
Theorem 1. Let's assume that  $f_0 \in \bar{H}^\beta$  and  $\| f_j \|_\infty \lesssim j^\gamma$  for  $\beta > 0, \gamma \geq 0$ .
1. In the mildly-ill posed problem where  $\kappa_{j} \asymp j^{-p}$ ,  $p > 0$ , if  $\lambda_{j} \asymp j^{-1 - 2\alpha}$  for  $\alpha > 0$  and  $(\alpha \wedge \beta) + p > 3/2 + 2\gamma$ , the posterior contracts at the rate  $\varepsilon_{n}^{inv} = n^{-\frac{\alpha \wedge \beta}{1 + 2\alpha + 2p}}$ .
2. In the severely ill-posed problem where  $\kappa_{j} \asymp e^{-c j^{p}}$ ,  $c > 0, p \geq 1$ , if  $\lambda_{j} \asymp j^{-\alpha} e^{-\xi j^{p}}$  for  $\alpha \geq 0, \xi > 0$ , the posterior contracts at the rate  $\varepsilon_{n}^{inv} = \log^{-\beta / p} n$ .
Furthermore, if there exists a constant  $C$  independent of  $n$  such that
$$
E _ {X} \left\| K _ {\boldsymbol {A} f \boldsymbol {A} f} - Q _ {\boldsymbol {A} f \boldsymbol {A} f} \right\| \leq C, \quad a n d \quad E _ {X} T r \left(K _ {\boldsymbol {A} f \boldsymbol {A} f} - Q _ {\boldsymbol {A} f \boldsymbol {A} f}\right) \leq C n \varepsilon_ {n} ^ {2}, \tag {7}
$$
where  $\varepsilon_{n} = n^{-\frac{\alpha\wedge\beta + p}{1 + 2\alpha + 2p}}$  in 1., and  $\varepsilon_{n} = n^{-c / (\xi +2c)}\log^{-\beta /p + c\alpha /(\xi +2c)}(n)$  in 2.,  $\Psi^{*}$  contracts around  $f_{0}$  at the rate  $\varepsilon_{n}^{inv}$  for the mildly and severely ill-posed problems i.e.
$$
E _ {f _ {0}} \Psi^ {*} \left[ f \colon \| f - f _ {0} \| _ {L _ {2} (\mathcal {T}; \mu)} \geq M _ {n} \varepsilon_ {n} ^ {i n v} \right] \to 0, \quad M _ {n} \to \infty .
$$