SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Proof. We provide the sketch of the proof here, the detailed derivation of the theorem is deferred to the supplementary material. In a first step, we derive posterior contraction rates around  $\mathcal{A}f_0$  in empirical  $L_{2}$ -norm under fixed design. In particular, we obtain an exponential decay of the probability expectation in the form
$$
E _ {Y \mid X} \Pi \left[ f: n ^ {- 1} \sum_ {i = 1} ^ {n} \left(\mathcal {A} f - \mathcal {A} f _ {0}\right) ^ {2} \left(x _ {i}\right) \geq M _ {n} \varepsilon_ {n} ^ {2} \mid X, Y \right] \mathbb {1} _ {A _ {n}} \leq C e ^ {- c M _ {n} ^ {2} n \varepsilon_ {n} ^ {2}}, \tag {8}
$$
for arbitrary  $M_{n}\to \infty$ , where  $\varepsilon_{n} = n^{-\frac{\alpha\wedge\beta + p}{1 + 2\alpha + 2p}}$  in the mildly and  $\varepsilon_{n} = n^{-\frac{c}{\xi + 2c}}\log^{-\frac{\beta}{p} +\frac{c\alpha}{\xi + 2c}}n$  in the severely ill-posed problems and  $A_{n}$  is an event on the sample space  $\mathbb{R}^n$  with probability tending to one asymptotically. In the mildly ill-posed case, this follows from results in [17; 62], while additional care is needed in the severely ill-posed case. As a second step, we go back to the random design setting. We show, using concentration inequalities and controlling the tail probability of GPs in the spectral decomposition, that the empirical and population  $L_{2}$ -norms are equivalent on a large enough event. This implies contraction rate with respect to the  $\| \cdot \|_{L^2 (\mathcal{X},G)}$ -norm around  $\mathcal{A}f_0$ , similarly to (8). In the third step, using the previous result on the forward map, we derive contraction rates around  $f_{0}$ . To achieve this we apply the modulus of continuity techniques introduced in [25]. Notably, we extend their ideas to infinite Gaussian series priors in the severely ill-posed case as well. Since in all these steps we can preserve the exponential upper bound for the posterior contraction (on a large enough event), we can apply Theorem 5 of [46], resulting in contraction rates for the VB procedure. It requires a control of the expected KL divergence between these two distributions, which follows from our assumptions on the expected trace and spectral norm of the covariance matrix of  $\mathcal{A}f|u$ , see Lemma 3 in [37] for the identity operator  $\mathcal{A}$ .
We briefly discuss the above results. First of all, the  $L_{2}(\mathcal{T};\mu)$ -contraction rate of the true posterior, to the best of our knowledge, wasn't derived explicitly in the literature before, hence it is of interest in its own right. Nevertheless, the main message is that the variational posterior achieves the same contraction rate as the true posterior under the assumption (7). Note that in the mildly ill-posed inverse problem case for eigenvalues  $\lambda_j \asymp j^{-1 - 2\beta}$  (i.e. taking  $\alpha = \beta$ ), the posterior contracts with the minimax rate  $n^{-\beta/(1 + 2\beta + 2p)}$ . Note that the  $d$ -dimensional case directly follows from this result when one defines the regularity class (2) and ill-posedness (Definition 1) with  $\beta/d$  and  $p/d$  which would imply the rate  $n^{-\beta/(d + 2\beta + 2p)}$ . Similarly in the severely-ill posed case one can achieve the minimax logarithmic contraction rate. Furthermore, the choice of the eigenvalue structure in the theorem was done for computational convenience, the results can be generalised for other choices of  $\lambda_j$  as well. Though we considered the random variables  $u$  fixed as we do not optimize them above, they could conceivably be considered as free variational parameters and selected at the same time as  $\boldsymbol{\mu}_u$  and  $\Sigma_u$ .
In the next subsection we consider two specific choices of the inducing variables, i.e. the population spectral feature method and its empirical counter part. We show that under sufficient condition on the number of inducing variables condition (7) is satisfied implying the contraction rate results derived in the preceding theorem.
2.3 Population and empirical spectral features methods
We focus here on two inducing variables methods, based on the spectral features (i.e. eigenspectrum) of the empirical covariance matrix  $K_{\mathcal{A}f\mathcal{A}f} = E_{\Pi}\mathcal{A}\mathbf{f}\mathcal{A}\mathbf{f}^T$  and the corresponding population level covariance operator  $(x,y)\mapsto E_{\Pi}\mathcal{A}f(x)\mathcal{A}f(y)$ .
We start with the former method and consider inducing variables of the form
$$
u _ {j} = \sum_ {i = 1} ^ {n} v _ {j} ^ {i} \mathcal {A} f \left(x _ {i}\right), \quad j = 1, \dots , m, \tag {9}
$$
where  $\mathbf{v}_j = (v_j^1, \ldots, v_j^n)$  is the eigenvector of  $K_{\mathcal{A}f\mathcal{A}f}$  corresponding to the  $j$ th largest eigenvalue  $\rho_j$  of this matrix. Similarly to the direct problem studied in [8; 37], this results in  $(K_{\mathcal{A}f\mathcal{A}f})_{ij} = \rho_j\delta_{ij}$ ,  $(K_{\mathcal{A}f\mathbf{u}})_{ij} = \rho_jv_j^i$ ,  $Q_{\mathcal{A}f\mathcal{A}f} = \sum_{j=1}^{m}\rho_j\mathbf{v}_j\mathbf{v}_j^T$ , and  $K_{\mathcal{A}f\mathcal{A}f} - Q_{\mathcal{A}f\mathcal{A}f} = \sum_{j=m+1}^{n}\rho_j\mathbf{v}_j\mathbf{v}_j^T$ . The computational complexity of deriving the first  $m$  eigenvectors of  $K_{\mathcal{A}f\mathcal{A}f}$  is  $\mathcal{O}(n^2m)$ . This is still quadratic in  $n$ , which sets limitations to its practical applicability, but it can be computed for arbitrary
choices of the prior covariance operator and map  $\mathcal{A}$ . We also note that this choice gives the optimal rank- $m$  approximation  $Q_{\mathcal{A}f\mathcal{A}f}$  of  $K_{\mathcal{A}f\mathcal{A}f}$  and it was noted in [8] that it gives the minimiser of the trace and norm terms in (7).
The second inducing variables method is based on the eigendecomposition of covariance kernel  $(x,y)\mapsto E_{\Pi}\mathcal{A}f(x)\mathcal{A}f(y)$ . Let us consider the variables
$$
u _ {j} = \int_ {\mathcal {X}} \mathcal {A} f (x) e _ {j} (x) d G (x), \quad j = 1, \dots , m. \tag {10}
$$
Again, by extending the results derived in the direct problem [8] to the inverse setting, this results in  $(K_{\mathcal{A}f\mathcal{A}f})_{ij} = \lambda_j\kappa_j\delta_{ij}$ ,  $(K_{\mathcal{A}f\mathbf{u}})_{ij} = \lambda_j\kappa_j\phi_j^i$ ,  $Q_{\mathcal{A}f\mathcal{A}f} = \sum_{j=1}^{m}\lambda_j\kappa_j\phi_j^T$ , and  $K_{\mathcal{A}f\mathcal{A}f} - Q_{\mathcal{A}f\mathcal{A}f} = \sum_{j=m+1}^{n}\lambda_j\kappa_j\phi_j\phi_j^T$ , where  $\phi_j = (\phi_j(x_1), \ldots, \phi_j(x_n))^T$ . The computational complexity of this method is  $O(nm^2)$ , which is substantially faster than its empirical counter part. However, it requires the exact knowledge of the eigenfunctions of the prior covariance kernel, and therefore in general has limited practical applicability.
Corollary 1. Let's assume that  $f_0 \in \bar{H}^\beta$ ,  $\| g_j\|_\infty \lesssim j^\gamma$  for  $\beta > 0, \gamma \geq 0$  and in the
1. mildly-ill posed case  $\kappa_{j} \asymp j^{-p}$ : take prior eigenvalues  $\lambda_{j} \asymp j^{-1 - 2\alpha}$  for some  $\alpha > 0$ ,  $(\alpha \wedge \beta) + p > 3/2 + 2\gamma$ , number of inducing variables  $m_{n} \geq n^{\frac{1}{(1 + 2p + 2\alpha)}}$  and denote by  $\varepsilon_{n}^{inv} = n^{-\frac{\alpha \wedge \beta}{1 + 2\alpha + 2p}}$ .
2. severely ill-posed case  $\kappa_{j} \asymp e^{-c j^{p}}$ : take prior eigenvalues  $\lambda_{j} \asymp j^{-\alpha}e^{-\xi j^{p}}$  for  $\alpha \geq 0$ ,  $\xi > 0$ , number of inducing variables  $m_{n}^{p} \geq (\xi + 2c)^{-1}\log n$ , and introduce the notation  $\varepsilon_{n}^{inv} = \log^{-\beta / p} n$ .
Then both for the population (if  $\gamma = 0$  in 1.) and empirical spectral features variational methods the corresponding variational posterior distribution contracts around the truth with the rate  $\varepsilon_{n}^{inv}$ , i.e.
$$
E _ {f _ {0}} \Psi^ {*} \left[ f \colon \| f - f _ {0} \| _ {L _ {2} (T; \mu)} \geq M _ {n} \varepsilon_ {n} ^ {i n v} \right] \to 0, \quad M _ {n} \to \infty .
$$
Remark 1. In the mildly ill-posed inverse problem taking  $\alpha = \beta$  results in the minimax contraction rate for  $m_{n} \geq n^{\frac{1}{1 + 2p + 2\alpha}}$ . Note that it is substantially less compared to the direct problem with  $p = 0$ , hence the computation is even faster in the inverse problem case.
3 Examples
In this section we provide three specific linear inverse problems as examples. The Volterra (integral) operator and the Radon transformations are mildly ill-posed, while the heat-equation is a severely ill-posed inverse problem. We show that in all cases by optimally tunning the GP prior and including enough inducing variables, the variational approximation of the posterior provides (from a minimax perspective) optimal recovery of the underlying signal  $f_{0}$ .
3.1 Volterra operator
First, let us consider the Volterra operator,  $\mathcal{A}:L_2[0,1]\longrightarrow L_2[0,1]$  satisfying that
$$
\mathcal {A} f (x) = \int_ {0} ^ {x} f (s) d s, \quad \mathcal {A} ^ {*} f (x) = \int_ {x} ^ {1} f (s) d s. \tag {11}
$$
The eigenvalues of  $\mathcal{A}^*\mathcal{A}$  and the corresponding eigenbases are given by  $\kappa_j^2 = (j - 1/2)^{-2}\pi^{-2}$ ,  $e_j(x) = \sqrt{2}\cos((j-1/2)\pi x)$ ,  $g_j(x) = \sqrt{2}\sin((j-1/2)\pi x)$  respectively, see [21]. Therefore the problem is mildly ill-posed with degree  $p = 1$  and these bases are uniformly bounded, i.e.  $\sup_j\|e_j\|_{\infty} \vee \|g_j\|_{\infty} < \infty$ . The following lemma is then a direct application of Corollary 1.
Corollary 2. Consider the Volterra operator in (1) and assume that  $f_0 \in \bar{H}^\beta$ , for some  $\beta > 1/2$ . Set the eigenvalues in (4) as  $\lambda_j = j^{-1 - 2\beta}$ . Then the variational posterior  $\Psi^*$  resulting from either the empirical or population spectral features inducing variable methods achieves the minimax contraction rate if the number of inducing variables exceeds  $m_n \gtrsim n^{\frac{1}{3 + 2\beta}}$ , i.e. for arbitrary  $M_n \to \infty$
$$
E _ {f _ {0}} \Psi^ {*} \left[ \| f - f _ {0} \| _ {L _ {2} [ 0, 1 ]} \geq M _ {n} n ^ {- \beta / (3 + 2 \beta)} \right] \to 0.
$$
3.2 Heat equation
Next let us consider the problem of recovering the initial condition for the heat equation. The heat equation is often considered as the starting example in the PDE literature and, for instance, the Black-Scholes PDE can be converted to the heat equation as well. We consider the Dirichlet boundary condition
$$
\frac {\partial}{\partial t} u (x, t) = \frac {\partial^ {2}}{\partial x ^ {2}} u (x, t), \quad u (x, 0) = \mu (x), \quad u (0, t) = u (1, t) = 0, \tag {12}
$$
for  $u$  defined on  $[0,1]\times [0,T],T > 0$ . For  $\mu \in L_2[0,1]$ ,  $u(x,t) = \sqrt{2}\sum_{j = 1}^{\infty}\mu_je^{-j^2\pi^2 t}\sin (j\pi x)$ , with  $\mu_j = \sqrt{2}\int_0^1\mu (s)\sin (j\pi s)ds$ . Therefore, if  $\mathcal{A}: \mathcal{D} \mapsto \mathcal{D}$ , with  $\mathcal{D} := \{f\in L_2[0,1], f(0) = f(1) = 0\}$ , is such that, for  $\mu = f$ ,  $\mathcal{A}f(x) = u(x,T)$ , then the corresponding singular-values and singular-functions of the operator  $\mathcal{A}$  are  $\kappa_j = e^{-j^2\pi^2 T}$  and  $e_j(x) = g_j(x) = \sqrt{2}\sin (j\pi x)$ . Therefore it is a severely ill-posed problem with  $p = 2$  and  $c = \pi^2 T$ . We also note that  $\sup_j\| e_j\|_{\infty}\vee \| g_j\|_{\infty} < \infty$ . This problem has been well studied both in the frequentist [6; 19; 32] and Bayesian setting [28; 54]. Then, by direct application of Corollary 1 we can provide optimality guarantees for the variational Bayes procedure in this model as well.
Corollary 3. Consider the heat equation operator  $\mathcal{A}$  as above in the linear inverse regression model (1) and assume that  $f_0\in \bar{H}^\beta$  for some  $\beta >0$ . Furthermore, we set the eigenvalues  $\lambda_{j} = j^{-\alpha}e^{-\xi j^{2}}$ ,  $\alpha \geq 0$ ,  $\xi >0$  in (4). Then the variational approximation  $\Psi^{*}$  resulting from either of the spectral features inducing variables method with  $m_{n}\geq (\xi +\pi^{2}T)^{-1 / 2}\log^{1 / 2}n$  achieves the minimax contraction rate, i.e. for arbitrary  $M_{n}\to \infty$
$$
E _ {f _ {0}} \Psi^ {*} \left[ \| f - f _ {0} \| _ {L _ {2} ([ 0, 1 ])} \geq M _ {n} \log^ {- \beta / 2} n \right]\rightarrow 0.
$$
3.3 Radon transform
Finally, we consider the Radon transform [24], where for some (Lebesgue)-square-integrable function  $f: D \to \mathbb{R}$  defined on the unit disc  $D = \{x \in \mathbb{R}^2 : \| x\|_2 \leq 1\}$ , we observe its integrals along any line intersecting  $D$ . If we parameterized the lines by the length  $s \in [0,1]$  of their perpendicular from the origin and the angle  $\phi \in [0.2\pi)$  of the perpendicular to the x-axis, we observe
$$
\mathcal {A} f (s, \phi) = \frac {\pi}{2 \sqrt {1 - s ^ {2}}} \int_ {- \sqrt {1 - s ^ {2}}} ^ {\sqrt {1 - s ^ {2}}} f (s \cos \phi - t \sin \phi , s \sin \phi + t \cos \phi) d t, \tag {13}
$$
where  $(s,\phi)\in S = [0,1]\times [0,2\pi)$ . The Radon transform is then a map from  $\mathcal{A}\colon L_2(D;\mu)\to$ $L_{2}(S;G)$ , where  $\mu$  is  $\pi^{-1}$  times the Lebesgue measure and  $dG(s,\phi) = 2\pi^{-1}\sqrt{1 - s^2} dsd\phi$ . Then  $\mathcal{A}$  is a bijective, mildly ill-posed linear operator of order  $p = 1 / 4$ . Furthermore, the operator's singular value decomposition can be computed via Zernike polynomials  $Z_{m}^{k}$  (degree  $m$ , order  $k$ ) and Chebyshev polynomials of the second kind  $U_{m}(\cos \theta) = \sin \left((m + 1)\theta\right) / \sin \theta \leq m + 1$ , see [24]. Translating it to the single index setting, we get for some functions  $l,m:\mathbb{N}\mapsto \mathbb{N}$  satisfying  $m(j)\asymp \sqrt{j}$  and  $|l(j)|\leq m(j)$ , that
$$
e _ {j} (r, \theta) = \sqrt {m (j) + 1} Z _ {m (j)} ^ {| l (j) |} e ^ {j l (j) \theta}, g _ {j} (s, \phi) = U _ {m (j)} (s) e ^ {j l (j) \phi},
$$
if polar coordinates are used on  $D$ . Therefore, we have  $\sup_j(\| e_j\|_{\infty}\vee \| g_j\|_{\infty}) / \sqrt{j} < \infty$ . Then, by directly applying Corollary 1 to this setting we can show that the variational Bayes method achieves the minimax contraction rate.
Corollary 4. Consider the Radon transform operator (13) in the inverse regression model (1) and let us take  $f_0 \in \bar{H}^\beta$ ,  $\beta > 9/4$ . Taking polynomially decaying eigenvalues  $\lambda_j \asymp j^{-1 - 2\beta}$ , the empirical spectral features variational Bayes method achieves the optimal minimax contraction rate if  $m_n \gtrsim n^{1/(3/2 + 2\beta)}$ , i.e. for any  $M_n \to \infty$
$$
E _ {f _ {0}} \Psi^ {*} \left[ \| f - f _ {0} \| _ {L _ {2} (D; \mu)} \geq M _ {n} n ^ {- \beta / (3 / 2 + 2 \beta)} \right] \to 0.
$$
4 Numerical analysis
We demonstrate the approximation accuracy of the variational Bayes method on synthetic data. We consider here the recovery of the initial condition of the heat condition 3.2, which is a severely
ill-posed. In the supplement we provide additional simulation study for mildly ill-posed inverse problems as well. We set the sample size  $n = 8000$ , take uniformly distributed covariates on [0, 1), and let
$$
f _ {0} (t) = \sqrt {2} \sum_ {j} c _ {j} j ^ {- (1 + \beta)} \sin (j \pi t), \quad c _ {j} = \left\{ \begin{array}{l l} 1 + 0. 4 \sin (\sqrt {5} \pi j), & j \text {o d d}, \\ 2. 5 + 2 \sin (\sqrt {2} \pi j), & j \text {e v e n}, \end{array} \right.
$$
for  $\beta = 1$ . The independent- observations are generated as  $Y_{i}\sim \mathcal{N}(\mathcal{A}f_{0}(x_{i}),1)$ , depending on the solution of the forward map  $\mathcal{A}f_0$  after time  $T = 10^{-2}$
We consider the prior with  $\lambda_{j} = e^{-\xi j^{2}}$  for  $\xi = 10^{-1}$ . In view of Corollary 3 the optimal number of inducing variables is  $m = (\xi + 2\pi^2 T)^{-1/2} \log^{1/2} n \approx 6$ . We consider the population spectral feature method described in (10) and plot the variational approximation of the posterior for  $m = 6$  and  $m = 3$  inducing variables in Figure 1. We represent the true posterior mean by solid red and the upper and lower pointwise  $2.5\%$  quantiles by dashed red curves. The true function is given by blue and the mean and quantiles of the variational approximation by solid and dotted purple curves, respectively.
Observe that with  $m = 6$ , see left part of Figure 1, the variational approximation results in similar  $95\%$  pointwise credible bands and posterior mean as the true posterior, providing an accurate approximation. Also note that both the true and the variational posterior contain  $f_{0}$  at most of the points, indicating frequentist confidence validity of the set. At the same time, by taking a factor of two less inducing points, i.e.  $m = 3$ , the credible sets will be overly large, resulting in good frequentist coverage, but suboptimally large posterior spread, see the second plot in Figure 1.
The computations were carried out with a 2,6 GHz Quad-Core Intel Core i7 processor. The computation of the exact posterior mean and covariance kernel on a grid of 100 points took over half an hour (in CPU time), while the variational approximation was substantially faster, taking only 50.5 ms, resulting in a  $3.68 \times 10^{4}$  times speed.
Figure 1: True and variational posterior means and credible regions for Gaussian series prior (sine basis) on the initial condition  $\mu = f_0$  of the heat equation (12), for  $m = 6$  (left) or  $m = 3$  (right) inducing variables from method (10).
A more extensive numerical analysis is available in the appendix, considering the application of our method to the settings of Sections 3.1 and 3.3 as well. We conduct these experiments several times and compare the average Mean Integrated Squared Error (MISE), see appendix A, and compute time for different choices of  $m$ . We observe that in all our examples, while increasing  $m$  results in longer computation, the MISE does not improve after a threshold close to the one presented in our results.
Therefore, it is sufficient to include as many inducing variables as we considered in Corollary 1 to obtain better performance. More than that would would only increase the computation complexity. In the Appendix, we also provide a literature review and some justifications of how relevant these problems are in practice
We have extended the inducing variables variational Bayes method for linear inverse problems and derived asymptotic contraction rate guarantees for the corresponding variational posterior. Our theoretical results provide a guide for practitioners on how to tune the prior distribution and how many inducing variables to apply (in the spectral feature variational Bayes method) to obtain minimax rate optimal recovery of the true functional parameter of interest. We have demonstrated the practical relevance of this guideline numerically on synthetic data and have shown that using less variables results in highly suboptimal recovery.
In our analysis we have considered priors built on the singular basis of the operator  $\mathcal{A}$ . In principle our results can be extended to other priors as well, until the eigenbasis of the covariance operator is not too different from the basis of the operator  $\mathcal{A}$ . This, however, would complicate the computation of the Kullback-Leibler divergence between the variational family and the posterior, resulting in extra technical challenges. In this setting the empirical spectral features method seems practically more feasible, especially, if the eigenbasis of the covariance kernel is not known explicitly. In the literature several different types of inducing variable methods were proposed, considering other, practically more relevant approaches of interest. Furthermore, extension to other type of inverse problems is also feasible. For instance in the deconvolution problem, when  $f_{0}$  is convoluted with a rectangular kernel, the eigenvalues given by the SVD are the product of a polynomially decaying and oscillating part and the "average degree" of ill-posedness does not match the lower and upper bounds [23]. Extension to non-linear inverse problem is highly relevant, as these problems tend to be computationally even more complex, but very challenging. One possible approach is to linearize the problem and take its variational approximation. Finally, it is of importance to derive frequentist coverage guarantees for VB credible sets. Our approach cannot directly be extended for this task. However, in the direct case, for some special choices of inducing variables, frequentist coverage guarantees were derived using kernel ridge regression techniques [38; 60]. This result, although computationally somewhat cumbersome, in principle can be extended to the inverse setting as well. One last drawback of our results is that the priors we consider are non-adaptative in the mildly ill-posed case. Minimax contraction rates are attainable only if the covariance eigenvalues are properly tuned, given the smoothness  $\beta$ . While this is not an issue the severely ill-posed case in our results, we keep the study of adaptation for future works as it is a much more involved question.
Funding. Co-funded by the European Union (ERC, BigBayesUQ, project number: 101041064). Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or the European Research Council. Neither the European Union nor the granting authority can be held responsible for them.
[1] AGAPIOU, S., LARSSON, S., AND STUART, A. M. Posterior contraction rates for the Bayesian approach to linear ill-posed inverse problems. Stochastic Process. Appl. 123, 10 (2013), 3828-3860.
[2] ALQUIER, P., AND RIDGWAY, J. Concentration of tempered posteriors and of their variational approximations. Ann. Statist. 48, 3 (2020), 1475-1497.
[3] ARRIDGE, S., MAASS, P., Öktem, O., AND SCHÖNLIEB, C.-B. Solving inverse problems using data-driven models. Acta Numerica 28 (2019), 1-174.
[4] BERTERO, M. Regularization methods for linear inverse problems. In Inverse problems (Montecatini Terme, 1986), vol. 1225 of Lecture Notes in Math. Springer, Berlin, 1986, pp. 52-112.
[5] BISSANTZ, N., HOHAGE, T., MUNK, A., AND RUYMGAART, F. Convergence rates of general regularization methods for statistical inverse problems and applications. SIAM J. Numer. Anal. 45, 6 (2007), 2610-2636.
[6] BISSANTZ, N., AND HOLZMANN, H. Statistical inference for inverse problems. Inverse Problems 24, 3 (2008), 034009.