SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Proof. Note we can approximate  $\phi_{\ell}(x) \in L^{2}(D)$  with simple functions  $\tilde{\phi}_{\ell}(x) = \sum_{i=1}^{d} k_{i} \chi_{D_{i}}(x)$  where  $D_{i}$ 's are measurable subsets of  $D$  and  $\chi_{D_{i}}(x) = 1$  if  $x \in D_{i}$  and 0 otherwise. By the linear combination property of elliptic distributions [c.f. Theorem 2.6.3 in 19],  $\tilde{u}_{\ell} = \int_{D} u(x) \tilde{\phi}_{\ell}(x) dx \sim q - \mathrm{ED}(0, c)$  with  $c = \alpha_{d}^{-1} \mathrm{E}[\tilde{u}_{\ell}^{2}]$  to be determined. Note  $\alpha_{d} = \frac{2^{\frac{2}{q}} \Gamma\left(\frac{d}{2} + \frac{2}{q}\right)}{d \Gamma\left(\frac{d}{2}\right)} d^{1 - \frac{2}{q}}$  comes from Proposition 3.1 and the scaling  $\mathbf{u}^{*} = d^{\frac{1}{2} - \frac{1}{q}} \mathbf{u}$  in Definition 3.2. We have  $\alpha_{d} = \frac{\Gamma\left(\frac{d}{2} + \frac{2}{q}\right)}{\Gamma\left(\frac{d}{2}\right)} \left(\frac{2}{d}\right)^{\frac{2}{q}} \to 1$  as  $d \to \infty$ . Taking the limit  $d \to \infty$ , we have  $u_{\ell} = \int_{D} u(x) \phi_{\ell}(x) dx \sim q - \mathrm{ED}(0, c)$ . In general, by the similar argument we have
$$
\begin{array}{l} \operatorname {C o v} \left(u _ {\ell}, u _ {\ell^ {\prime}}\right) = \operatorname {E} \left[ u _ {\ell} u _ {\ell^ {\prime}} \right] = \int_ {D} \int_ {D} \operatorname {E} \left[ u (x) u \left(x ^ {\prime}\right) \right] \phi_ {\ell} (x) \phi_ {\ell^ {\prime}} \left(x ^ {\prime}\right) d x d x ^ {\prime} \\ = \int_ {D} \int_ {D} \mathcal {C} \left(x, x ^ {\prime}\right) \phi_ {\ell} (x) \phi_ {\ell^ {\prime}} \left(x ^ {\prime}\right) d x d x ^ {\prime} = \int_ {D} \lambda_ {\ell} \phi_ {\ell} \left(x ^ {\prime}\right) \phi_ {\ell^ {\prime}} \left(x ^ {\prime}\right) d x ^ {\prime} = \lambda_ {\ell} \delta_ {\ell \ell^ {\prime}} \\ \end{array}
$$
Thus it completes the proof.
A.6 Proof of Theorem 3.5
Before proving Theorem 3.5, we first prove the following lemma based on the conditional of elliptic distribution [12, 19].
Lemma A.1. If  $\mathbf{u} = (\mathbf{u}_1, \mathbf{u}_2) \sim \mathrm{q - ED}_d(\boldsymbol{\mu}, \mathbf{C})$  with  $\boldsymbol{\mu} = \begin{bmatrix} \boldsymbol{\mu}_1 \\ \boldsymbol{\mu}_2 \end{bmatrix}$  and  $\mathbf{C} = \begin{bmatrix} \mathbf{C}_{11} & \mathbf{C}_{12} \\ \mathbf{C}_{21} & \mathbf{C}_{22} \end{bmatrix}$ ,  $\mathbf{u} \in \mathbb{R}^d$ ,  $\mathbf{u}_i \in \mathbb{R}^{d_i}$  for  $i = 1, 2$  and  $d_1 + d_2 = d$ , then we have the following conditional distribution
$$
\begin{array}{l} \mathbf {u} _ {1} \left| \mathbf {u} _ {2} \right. \sim q - \mathrm {E D} _ {d _ {1}} \left(\boldsymbol {\mu} _ {1, 2}, \mathbf {C} _ {1 1, 2}\right), \\ \boldsymbol {\mu} _ {1 \cdot 2} = \boldsymbol {\mu} _ {1} + \mathbf {C} _ {1 2} \mathbf {C} _ {2 2} ^ {- 1} (\mathbf {u} _ {2} - \boldsymbol {\mu} _ {2}), \quad \mathbf {C} _ {1 1 \cdot 2} = \mathbf {C} _ {1 1} - \mathbf {C} _ {1 2} \mathbf {C} _ {2 2} ^ {- 1} \mathbf {C} _ {2 1} \\ \end{array}
$$
Proof. This directly follows from [Corollary 5 of Theorem 5 in 12] or [Corollary 3 of Theorem 2.6.6 in 19] for  $\mathrm{q - ED}_d(\pmb{\mu},\mathbf{C}) = \mathrm{EC}_d(\pmb{\mu},\mathbf{C},g)$  with our chosen  $g$ .
Now we prove the Theorem 3.5.
Proof. By the linear combination property of the elliptic distributions [26, 19], we have  $\mathbf{y} \sim \mathrm{q - ED}(\mathbf{0},\mathbf{C} + \Sigma)$ . Then based on the consistency, we have the joint distribution
$$
\left[ \begin{array}{c} \mathbf {y} \\ u (x _ {*}) \end{array} \right] \sim \mathrm {q - E D} \left(\mathbf {0}, \left[ \begin{array}{c c} \mathbf {C} + \boldsymbol {\Sigma} & \mathbf {C} _ {*} \\ \mathbf {C} _ {*} ^ {\top} & \mathbf {C} _ {* *} \end{array} \right]\right)
$$
Therefore, the conclusion follows from Lemma A.1.
A.7 Proposition of Conditional Conjugacy for Variance Magnitude  $(\sigma^2)$
Proposition A.2. If we assume a proper inverse-gamma prior for the variance magnitude such that  $\mathbf{u}|\sigma^2\sim \mathrm{q - ED}_d(\pmb {\mu},\mathbf{C} = \sigma^2\mathbf{C}_0)$ , and  $\sigma^q\sim \Gamma^{-1}(\alpha ,\beta)$ , then we have
$$
\sigma^ {q} | \mathbf {u} \sim \Gamma^ {- 1} \left(\alpha^ {\prime}, \beta^ {\prime}\right), \quad \alpha^ {\prime} = \alpha + \frac {d}{2}, \quad \beta^ {\prime} = \beta + \frac {\left(\mathbf {u} - \boldsymbol {\mu}\right) ^ {\top} \mathbf {C} _ {0} ^ {- 1} (\mathbf {u} - \boldsymbol {\mu})}{2} \tag {19}
$$
Proof. Denote  $r_0 = (\mathbf{u} - \pmb{\mu})^\top \mathbf{C}_0^{-1}(\mathbf{u} - \pmb{\mu})$ . We can compute the joint density of  $\mathbf{u}$  and  $\sigma^2$
$$
\begin{array}{l} p (\mathbf {u}, \sigma^ {2}) = p (\mathbf {u} | \sigma^ {2}) p (\sigma^ {q}) \\ = \frac {q}{2} (2 \pi) ^ {- \frac {d}{2}} | \mathbf {C} _ {0} | ^ {- \frac {1}{2}} r _ {0} ^ {\left(\frac {q}{2} - 1\right) \frac {d}{2}} \sigma^ {- \frac {q d}{2}} \exp \left\{- \sigma^ {- q} \frac {r _ {0} ^ {\frac {q}{2}}}{2} \right\} \frac {\beta^ {\alpha}}{\Gamma (\alpha)} (\sigma^ {q}) ^ {- (\alpha + 1)} \exp (- \beta \sigma^ {- q}) \\ \propto \left(\sigma^ {q}\right) ^ {- \left(\alpha + \frac {d}{2} + 1\right)} \exp \left\{- \sigma^ {- q} \left(\beta + \frac {r _ {0} ^ {q}}{2}\right) \right\} \\ \end{array}
$$
By identifying the parameters for  $\sigma^q$  we recognize that  $\sigma^q |\mathbf{u}$  is another inverse-gamma with parameters  $\alpha^\prime$  and  $\beta^\prime$  as given.
B ALGORITHM
Algorithm 1 White-noise Preconditioned Crank-Nicolson (wn-pCN) for Q-EP Prior Models
1: Fix  $\beta \in (0,1]$ . Choose initial state  $\mathbf{z}^{(0)}\in \mathbb{R}^d$
2: for  $k = 0, \dots, K - 1$  do
3: Propose  $\hat{\mathbf{z}}^{(k)} = (1 - \beta^2)^{\frac{1}{2}}\mathbf{z}^{(k)} + \beta \mathbf{z}',\mathbf{z}'\sim \mathcal{N}(\mathbf{0},\mathbf{I})$
4: Set  $\mathbf{z}^{(k + 1)} = \hat{\mathbf{z}}^{(k)}$  with acceptance probability
$$
\min  \left\{1, \frac {L \left(T \left(\hat {\mathbf {z}} ^ {(k)}\right)\right) \left| d T \left(\hat {\mathbf {z}} ^ {(k)}\right) \right|}{L \left(T \left(\mathbf {z} ^ {(k)}\right)\right) \left| d T \left(\mathbf {z} ^ {(k)}\right) \right|} \right\}
$$
5: or else set  $\mathbf{z}^{(k + 1)} = \mathbf{z}^{(k)}$
6: end for
C ADDITIONAL EXPERIMENTAL RESULTS
In this section, we present some additional numerical experimental results that cannot be included in the main text due to the page limit.
First, we numerically verify the equivalence between the stochastic representation (6) and the white-noise representation (17) of  $\mathrm{q - ED}_d$  random variable in Figure B.1. More specifically, we generate 10000 samples using each of these two representations and illustrate in Figure B.1 that the two samples yield empirical marginal distributions (1d and 2d) close enough to each other.
C.1 Time Series Modeling
For modeling the simulated time series and stock prices, we include the optimization trace of negative (log)-posterior densities and relative errors for the two simulations and two stocks prices in Figure C.1. As commented in the main text, these plots show that Q-EP model can converge faster to lower errors compared with GP and Besov models.
Next, we compare MAP estimates by GP, Besov and Q-EP models in Figure C.2a for simulated time series with step jumps and in Figure C.2c for the Google stock prices in 2022. We also investigate the prediction results by GP and Q-EP in these two examples in Figures C.2b and C.2d. Table C.1 summarizes the RMSE of estimated stock prices by the three models and its standard deviation for repeating the experiments 10 times independently.
Figure B.1: Comparison in sampling  $\mathrm{q - ED}_d$  using the stochastic representation (6) (organge) and the white-noise representation (17) (blue). Numerical results show their sampling distributions are indistinguishable. Empirical densities are estimated based on 10000 samples (shown as dots).
(a) Time series with step jumps.
(c) Tesla stock prices in 2022.
Figure C.1: Negative posterior densities (left) and errors (right) as functions of iterations in the BFGS algorithm used to obtain MAP estimates. Early termination is implemented if the error falls below some threshold or the maximal iteration (1000) is reached. Relative errors are compared against truth in the simulation and the actual data in the Tesla stock.
(d) Google stock prices in 2022.
C.2 Computed Tomography Imaging
In the problem of reconstructing human head and torso CT images, Table C.2 compares GP, Besov and Q-EP models in terms of relative error (RLE), log-likelihood (LL), and imaging quality metrics including PSNR, SSIM and HarrPSI. In most cases, Q-EP outperforms, or achieves comparable scores with the other two methods.
Lastly, Figures C.3 and C.4 show that the posterior standard deviations estimated by wn-pCN using GP model could be misleading because the seemingly more recognizable shape deludes the fact that they are about two orders of magnitude smaller in value compared with the other two models. This implies that GP might underestimate the uncertainty present in the observed sinograms in the CT imaging analysis.
(a) Time series with step jumps (model fitting).
(c) Google stock prices in 2022 (model fitting).
(d) Google stock prices in 2022 (prediction).
Figure C.2: (a)(c) MAP estimates by GP (left), Besov (middle) and Q-EP (right) models. (b)(d) Predictions by GP (left) and Q-EP (right) models. Orange dots are actual realizations (data points). Blue solid lines are true trajectories. Black ticks indicate the training data points. Red dashed lines are MAP estimates. Red dot-dashed lines are predictions with shaded region being credible bands.
(b) Time series with jumps (prediction).
Figure C.3: Shepp-Logan phantom: uncertainty field (posterior standard deviation) given by GP, Besov and Q-EP models. GP tends to underestimate the uncertainty values (about  $1\%$  of that with Q-EP).
Figure C.4: CT of human head (upper) and torso (lower): uncertainty field (posterior standard deviation) given by GP, Besov and Q-EP models. Note GP tends to underestimate the uncertainty values (about  $1\%$  of that with Q-EP).
Table C.1: Posterior estimates of Tesla and Google stock prices by GP, Besov and Q-EP prior models: RMSE :=  $\| \overline{u} - u \|_2$ . Results are repeated 10 times with different random seeds.
Tesla	Google
GP	Besov	Q-EP	GP	Besov	Q-EP
RMSE	171.8515	90.3086	83.8130	20.4095	25.2012	18.3597
std(RMSE)	1.8018	1.1478	2.6949	0.7115	0.1698	0.9617
Table C.2: MAP estimates for CT of human head and torso by GP, Besov and Q-EP prior models: relative error, RLE :=  $\| \hat{u} - u^{\dagger} \| / \| u^{\dagger} \|$  of MAP ( $\hat{u} = u^{*}$ ), log-likelihood (LL), PSNR, SSIM and HarrPSI.
Head	Torso
GP	Besov	Q-EP	GP	Besov	Q-EP
RLE	0.2999	0.2241	0.2224	0.2611	0.2177	0.2153
LL	-4.05e+5	-1.12e+4	-1.17e+4	-3.30e+5	-3.86e+3	-4.37e+3
PSNR	24.2321	26.7633	26.8281	23.6450	25.2231	25.3190
SSIM	0.7010	0.7914	0.8096	0.5852	0.6983	0.6982
HaarPSI	0.0525	0.0593	0.0587	0.0666	0.0732	0.07190
C.3 Noisy/Blurry Image Reconstruction
Next we consider reconstructing a  $(128 \times 128$  pixels) image of satellite shown on the leftmost of Figure 1 from a blurred observation next to it. The image itself can be viewed as a function  $u(x)$  on the square unit  $D = [0,1]^2$  taking values as the pixels. When evaluating  $u(x)$  on the discretized domain,  $u(x)$  becomes a matrix of size  $128 \times 128$ , which can further be vectorized to  $\mathbf{u} \in \mathbb{R}^d$  with  $d = 128^2$ . The true image, denoted as  $u^\dagger$ , is blurred by applying a motion blur point spread function [PSF 11] and adding  $5\%$  Gaussian noise. The actual observation,  $y(x)$ , can be written as in the following linear model:
$$
y (x) = A u (x) + \varepsilon , \quad \varepsilon \sim \mathcal {N} \left(0, \sigma_ {\varepsilon} ^ {2} I _ {d}\right)
$$
where  $A \in \mathbb{R}^{J \times d}$  is the blur motion PSF with  $J = d$  and  $\sigma_{\varepsilon} / \| Au \| = 5\%$ . Note, the blurring effect in the observed image (the second from left of Figure 1) is mainly due to the PSF operator  $A$ , not the small Gaussian noise.
Figure C.5: Image of satellite: negative posterior densities (left) and errors (right) as functions of iterations in the BFGS algorithm used to obtain MAP estimates. Early termination is implemented if the error falls below some threshold or the maximal iteration (1000) is reached.
We compare the reconstructions by MAP estimate in Figure 1. The output by GP is blurry and close to the observed image, which means that GP does not "de-noise" much. The result by Besov is much better than GP due to the  $L_{1}$  regularization but it is still not sharp enough. We can see that the Q-EP prior model produces the reconstruction of the highest quality. Figure 3 demonstrates the effect of  $q > 0$ : the smaller  $q$ , the more regularization and hence sharper reconstruction. We also compare their negative posterior densities and relative errors,  $\| \hat{u} - u^{\dagger} \| / \| u^{\dagger} \|$ , in Figure C.5. The Q-EP prior model yields the smallest error among all the three models.
(a) True initial condition (top left), and the solutions  $u(\mathbf{x},t)$  at different time points  $t$ .
(b) Spatiotemporal observations at 80 selected locations (color dots) across different time points.
Figure C.7: Advection-diffusion inverse problem: true initial condition  $u_0^\dagger$  and posterior mean estimates by GP, Besov and Q-EP models.
C.4 Advection-Diffusion Inverse Problem
Finally, we consider a Bayesian inverse problem governed by a time-dependent advection-diffusion equation [36, 30] that can be applied to heat transfer, pollution tracing, etc. The inverse problem involves inferring an unknown initial condition  $u_0 \in L^2(\Omega)$  from spatiotemporal point measurements  $\{y(\mathbf{x}_i, t_j)\}$  as
$$
y (\mathbf {x}, t) = \mathcal {G} (u _ {0}) + \eta (\mathbf {x}, t), \quad \eta (\mathbf {x}, t) \sim \mathcal {N} (0, \Sigma)
$$
The forward mapping  $\mathcal{G}:u_0\to \mathcal{O}u$  maps the initial condition  $u_{0}$  to pointwise spatiotemporal observations of the concentration field  $u(\mathbf{x},t)$  through the solution of the following advection-diffusion equation [36, 45]:
$$
u _ {t} - \kappa \Delta u + \mathbf {v} \cdot \nabla u = 0 \quad i n \Omega \times (0, T) \quad - \frac {1}{\mathrm {R e}} \Delta \mathbf {v} + \nabla p + \mathbf {v} \cdot \nabla \mathbf {v} = 0 \quad i n \Omega
$$
$$
u (\cdot , 0) = u _ {0} \quad i n \Omega
$$
$$
\nabla \cdot \mathbf {v} = 0 \quad i n \Omega
$$
$$
\kappa \nabla u \cdot \vec {n} = 0, \quad o n   \partial \Omega \times (0, T)
$$
$$
\mathbf {v} = \mathbf {g}, \quad o n   \partial \Omega
$$
where  $\Omega \subset [0,1]^2$  is a bounded domain shown in Figure C.6a,  $\kappa = 10^{-3}$  is the diffusion coefficient, and  $T > 0$  is the final time. The velocity field  $\mathbf{v}$  is computed by solving the following steady-state Navier-Stokes equation with the side walls driving the flow [36]. Here,  $p$  is the pressure, and  $\mathrm{Re}$  is the Reynolds number, which is set to 100 in this example. The Dirichlet boundary data  $\mathbf{g} \in \mathbb{R}^2$  is given by  $\mathbf{g} = \mathbf{e}_2 = (0,1)$  on the left wall,  $\mathbf{g} = -\mathbf{e}_2$  on the right wall, and  $\mathbf{g} = \mathbf{0}$  everywhere else.
To generate data, we set the true value of parameter  $u_0$  in (C.4) as  $u_0^\dagger = 0.5 \wedge \exp\{-100[(x - 0.35)^2 + (y - 0.7)^2]\}$ , illustrated in the top left panel of Figure C.6a, which also shows a few snapshots of the solutions  $u(\mathbf{x},t)$  at other time points on a regular grid mesh of size  $61 \times 61$ . Spatiotemporal observations  $\{y(\mathbf{x}_i,t_j)\}_{i=1,j=1}^{I,J}$  are collected at  $I = 80$  selected locations  $\{\mathbf{x}_i\}_{i=1}^J$  around the boundary of two inner boxes (See Figure C.6a and also Figure C.6b) across  $J = 16$  time points  $\{t_j\}_{j=1}^J$  evenly
Figure C.8: Advection-diffusion inverse problem: comparing forward predictions,  $\overline{\mathcal{G}} (\mathbf{x},t_{*})$ , based on the GP (blue dashed lines) and Q-EP (orange dot-dashed lines) prior models at three selective locations  $\mathbf{x} = (0.325,0.401)$ ,  $\mathbf{x} = (0.249,0.225)$  and  $\mathbf{x} = (0.249,0.350)$ . Blues dots are observations.
distributed between 1 and 4 seconds (thus denoted as  $\mathcal{O}u$ ) with noise variance  $\Sigma = \sigma_{\eta}^{2}I_{1280}$  where  $\sigma_{\eta} = 0.5\max \mathcal{O}u$ , i.e.  $y(\mathbf{x}_i,t_j) = \mathcal{G}(u_0^\dagger) + \eta_{ij} = u(\mathbf{x}_i,t_j) + \eta_{ij}$ .
To solve the inverse problem of finding the initial condition  $u_{0}$  in the Bayesian framework [43, 16], we impose  $u_{0}$  with GP, Besov and Q-EP priors respectively and seek the posterior  $p(u_0|y)$ . For GP and Q-EP, we adopt a covariance kernel,  $\mathcal{C} = (\delta \mathcal{I} - \gamma \Delta)^{-2}$ , defined through the Laplace operator  $\Delta$ , where  $\delta$  governs the variance of the prior and  $\gamma/\delta$  controls the correlation length [16, 30]. We set  $\gamma = 1$  and  $\delta = 8$  in this example. For Besov, we adopt 2d Fourier basis of the format  $\phi_{ij} = \cos(\pi ix_1)\cos(\pi jx_2)$  and truncate the series (1) for the first  $L = 1000$  terms.
We apply wn-pCN to this challenging nonlinear inverse problem with high dimensionality (3413) of spatially discretized  $u$  at each time  $t$ . Figure C.7 compares the posterior mean estimates of  $u_{0}$  given by these three models. Because the truth (leftmost) has clear edge at its cutoff by 0.5, Q-EP is more appropriate than GP and it indeed generates better estimate closer to the truth. Figure C.8 plots the prediction of forward mapping at a few selective locations on the left side of lower inner box by  $\overline{\mathcal{G}}(\mathbf{x}, t_{*}) = \frac{1}{S} \sum_{s=1}^{S} \mathcal{G}(u^{(s)})(\mathbf{x}, t_{*})$  with  $u^{(s)} \sim p(u_{0}|y)$ . Compared with GP, Q-EP predicts the solution path closer to the truth  $\mathcal{G}(u_{0}^{\dagger})(\mathbf{x}, t_{*})$  where the observations see more dynamical changes. More importantly, Q-EP provides proper UQ with credible bands wide enough to include the true trajectories. On the other hand, the posterior estimates by GP come with much narrower error bands that miss the truth. Again, we observe GP prior model being overconfident about less accurate estimates.