SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Bayesian Learning via  $Q$ -Exponential Process
Shuyi Li Michael O'Connor Shiwei Lan
School of Mathematical & Statistical Sciences  
Arizona State University, Tempe, AZ 85287
Regularization is one of the most fundamental topics in optimization, statistics and machine learning. To get sparsity in estimating a parameter  $u \in \mathbb{R}^d$ , an  $\ell_q$  penalty term,  $\| u \|_q$ , is usually added to the objective function. What is the probabilistic distribution corresponding to such  $\ell_q$  penalty? What is the correct stochastic process corresponding to  $\| u \|_q$  when we model functions  $u \in L^q$ ? This is important for statistically modeling high-dimensional objects such as images, with penalty to preserve certain properties, e.g. edges in the image. In this work, we generalize the  $q$ -exponential distribution (with density proportional to)  $\exp(-\frac{1}{2} |u|^q)$  to a stochastic process named  $q$ -exponential (Q-EP) process that corresponds to the  $L_q$  regularization of functions. The key step is to specify consistent multivariate  $q$ -exponential distributions by choosing from a large family of elliptic contour distributions. The work is closely related to Besov process which is usually defined in terms of series. Q-EP can be regarded as a definition of Besov process with explicit probabilistic formulation, direct control on the correlation strength, and tractable prediction formula. From the Bayesian perspective, Q-EP provides a flexible prior on functions with sharper penalty ( $q < 2$ ) than the commonly used Gaussian process (GP,  $q = 2$ ). We compare GP, Besov and Q-EP in modeling functional data, reconstructing images and solving inverse problems and demonstrate the advantage of our proposed methodology.
Regularization on function spaces is one of the fundamental questions in statistics and machine learning. High-dimensional objects such as images can be viewed as discretized functions defined on 2d or 3d domains. Statistical models for these objects on function spaces demand regularization to induce sparsity, prevent over-fitting, produce meaningful reconstruction, etc. Gaussian process [GP 38, 24] has been widely used as an  $L_{2}$  penalty (negative log-density as a quadratic form) or a prior on the function space. Despite the flexibility, sometimes random candidate functions drawn from GP are over-smooth for modeling certain objects such as images with sharp edges. To address this issue, researchers have proposed a class of  $L_{1}$  penalty based priors including Laplace random field [37, 34, 28] and Besov process [31, 15, 25, 16]. They have been extensively applied in spatial modeling [37], signal processing [28], imaging analysis [44, 34] and inverse problems [31, 15]. Figure 1 demonstrates an application of nonparametric regression models on functions endowed with GP, Besov and our proposed  $q$ -exponential process (Q-EP) priors respectively to reconstruct a blurry image of a satellite. Q-EP model generates the best reconstruction, indicating its advantage over GP in modeling objects with abrupt changes or sharp contrast such as "edges" in image.
For these high-dimensional (refer to its discretization) inhomogeneous objects on  $d^{\star}$  domains  $D\subset \mathbb{R}^{d^{\star}}$  particularly 2d images with sharp edges  $(d^{\star} = 2)$ , one can model them as a random function  $u$  from a
\*slan@asu.edu
Figure 1: Image of satellite: true image, blurred observation, and reconstructions by GP, Besov and Q-EP models with relative errors  $75.19\%$ ,  $21.94\%$  and  $20.35\%$  respectively.
Besov process represented by the following series for a given orthonormal basis  $\{\phi_{\ell}\}_{\ell = 1}^{\infty}$  in  $L^2 (D)$  [31, 15]:
$$
u: D \longrightarrow \mathbb {R}, \quad u (x) = \sum_ {\ell = 1} ^ {\infty} \gamma_ {\ell} u _ {\ell} \phi_ {\ell} (x), \quad u _ {\ell} \stackrel {i i d} {\sim} \pi_ {q} (\cdot) \propto \exp \left(- \frac {1}{2} | \cdot | ^ {q}\right) \tag {1}
$$
where  $q \geq 1$  and  $\gamma_{\ell} = \kappa^{-\frac{1}{q}}\ell^{-(\frac{s}{d^{\star}} +\frac{1}{2} -\frac{1}{q})}$  with (inverse) variance  $\kappa >0$  and smoothness  $s > 0$ . When  $q = 2$  and  $\{\phi_{\ell}\}$  is chosen to be Fourier basis, this reduces to GP [16] but Besov is often used with  $q = 1$  and wavelet basis [31] to provide "edge-preserving" function candidates suitable for image analysis. Historically, [32] discovered that the total variation prior degenerates to GP prior as the discretization mesh becomes denser and thus loses the edge-preserving properties in high dimensional applications. Therefore, [31] proposed the Besov prior defined as in (1) and proved its discretization-invariant property. Though straightforward, such series definition lacks a direct way to specify the correlation structure as GP does through the covariance function. What is more, once the basis  $\{\phi_{\ell}\}$  is chosen, there is no natural way to make prediction with Besov process.
We propose a novel stochastic process named  $q$ -exponential process (Q-EP) to address these issues. We start with the  $q$ -exponential distribution  $\pi_q(\cdot)$  and generalize it to a multivariate distribution (from a family of elliptic contour distributions) that is consistent to marginalization. Such consistency requires the joint distribution and the marginalized one (by any subset of components) to have the same format of density (See Section 3). We then generalize such multivariate  $q$ -exponential distribution to the process Q-EP and establish its connection and contrast to the Besov process. Note, if we view the negative log-density of the proposed distribution and process, Q-EP would impose an  $L_{q}$  regularization on the function space, similarly as  $L_{2}$  regularization given by GP whose negative log-density is a quadratic form of the input variable  $x$  (See Remark 2).
Connection to existing works The proposed Q-EP process is related to the student- $t$  process (TP) [41] as alternatives to GP. TP generalizes multivariate  $t$ -distribution (MVT) and is derived as a scale mixture of GP. Both TP and Q-EP can be viewed as a special case of the elliptical process [5] which gives the condition on general elliptic distributions that can be generalized to a valid stochastic process. Both papers focus on extending GP to robust models for heavier tail data, while our proposed work innovates a new Bayesian learning method on function spaces through the regularization parameter  $q$  (See Figure 3 for its effect on regularization when it varies), as is usually done in the optimization. Both our proposed Q-EP and [5] are inspired by Kano's consistency result [27], however the later focuses on a completely different process named squeezebox process. Our work on Q-EP makes multi-fold contributions to the learning of functional data in statistics and machine learning:
1. We propose a novel stochastic process Q-EP corresponding to the  $L_{q}$  regularization on function spaces.
2. For the first time we define/derive Besov process probabilistically as Q-EP with direct ways to configure correlation and to make prediction.
3. We provide flexible Bayesian inference methods based on the Markov Chain Monte Carlo (MCMC) algorithms using a white-noise representation for Q-EP prior models.
The rest of the paper is organized as follows. Section 2 introduces the  $q$ -exponential distribution and its multivariate generalizations. We propose the Q-EP with details in Section 3 and introduce it as a nonparametric prior for modeling functional data. In Section 4 we demonstrate the advantage of Q-EP over GP and Besov in time series modeling, image reconstruction, and Bayesian inverse problems (Appendix C.4). Finally we discuss some future directions in Section 5.
2 THE  $Q$ -EXPONENTIAL DISTRIBUTION AND ITS MULTIVARIATE GENERALIZATIONS
Let us start with the  $q$ -exponential distribution for a scalar random variable  $u \in \mathbb{R}$ . It is named in [15] and defined with the following density not in an exact form (as a probability density normalized to 1):
$$
\pi_ {q} (u) \propto \exp \left(- \frac {1}{2} | u | ^ {q}\right). \tag {2}
$$
This  $q$ -exponential distribution (2) is actually a special case of the following exponential power (EP) distribution  $\mathrm{EP}(\mu, \sigma, q)$  with  $\mu = 0$ ,  $\sigma = 1$ :
$$
p (u | \mu , \sigma , q) = \frac {q}{2 ^ {1 + 1 / q} \sigma \Gamma (1 / q)} \exp \left\{- \frac {1}{2} \left| \frac {u - \mu}{\sigma} \right| ^ {q} \right\} \tag {3}
$$
where  $\Gamma$  denotes the gamma function. Note the parameter  $q > 0$  in (3) controls the tail behavior of the distribution: the smaller  $q$  the heavier tail and vice versa. This distribution also includes many commonly used ones such as the normal distribution  $\mathcal{N}(\mu, \sigma^2)$  for  $q = 2$  and the Laplace distribution  $L(\mu, b)$  with  $\sigma = 2^{-1/q}b$  when  $q = 1$ .
How can we generalize it to a multivariate distribution and further to a stochastic process? Gomez [23] provided one possibility of a multivariate EP distribution, denoted as  $\mathrm{EP}_d(\pmb {\mu},\mathbf{C},q)$ , with the following density:
$$
p (\mathbf {u} | \boldsymbol {\mu}, \mathbf {C}, q) = \frac {q \Gamma (\frac {d}{2})}{2 \Gamma (\frac {d}{q})} 2 ^ {- \frac {d}{q}} \pi^ {- \frac {d}{2}} | \mathbf {C} | ^ {- \frac {1}{2}} \exp \left\{- \frac {1}{2} \left[ (\mathbf {u} - \boldsymbol {\mu}) ^ {\top} \mathbf {C} ^ {- 1} (\mathbf {u} - \boldsymbol {\mu}) \right] ^ {\frac {q}{2}} \right\} \tag {4}
$$
When  $q = 2$ , it reduces to the familiar multivariate normal (MVN) distribution  $\mathcal{N}_d(\pmb{\mu}, \mathbf{C})$ .
Unfortunately, unlike MVN being the foundation of GP, the Gomez's EP distribution  $\mathrm{EP}_d(\pmb{\mu},\mathbf{C},q)$  fails to generalize to a valid stochastic process because it does not satisfy the marginalization consistency as MVN does (See Section 3 for more details). It turns out we need to seek candidates in an even larger family of elliptic (contour) distributions  $\mathrm{EC}_d(\pmb{\mu},\mathbf{C},g)$ :
Definition 2.1 (Elliptic distribution). A multivariate elliptic distribution  $\mathrm{EC}_d(\pmb {\mu},\mathbf{C},g)$  has the following density [26]
$$
p (\mathbf {u}) = k _ {d} | \mathbf {C} | ^ {- \frac {1}{2}} g (r), \quad r (\mathbf {u}) = \left(\mathbf {u} - \boldsymbol {\mu}\right) ^ {\top} \mathbf {C} ^ {- 1} (\mathbf {u} - \boldsymbol {\mu}) \tag {5}
$$
where  $k_{d} > 0$  is the normalizing constant and  $g(\cdot)$ , a one-dimensional real-valued function independent of  $d$  and  $k_{d}$ , is named density generating function [19].
Every elliptic (contour) distributed random vector  $\mathbf{u} \sim \mathrm{EC}_d(\pmb{\mu}, \mathbf{C}, g)$  has a stochastic representation mainly due to Schoenberg [40, 12, 26], as stated in the following theorem.
Theorem 2.1.  $\mathbf{u}\sim \mathrm{EC}_d(\pmb {\mu},\mathbf{C},g)$  if and only if
$$
\mathbf {u} \stackrel {d} {=} \boldsymbol {\mu} + R \mathbf {L} S \tag {6}
$$
where  $S \sim \mathrm{Unif}(\mathcal{S}^{d + 1})$  uniformly distributed on the unit-sphere  $\mathcal{S}^{d + 1}$ ,  $\mathbf{L}$  is the Cholesky factor of  $\mathbf{C}$  such that  $\mathbf{C} = \mathbf{LL}^{\top}$ ,  $R \perp S$  and  $R^2 \stackrel{d}{=} r(\mathbf{u}) \sim f(r) = \frac{\pi^{\frac{d}{2}}}{\Gamma(\frac{d}{2})} k_d r^{\frac{d}{2} - 1} g(r)$ .
The Gomez's EP distribution  $\mathrm{EP}_d(\pmb {\mu},\mathbf{C},q)$  is a special elliptic distribution  $\mathrm{EC}_d(\pmb {\mu},\mathbf{C},g)$  with  $g(r) = \exp \{-\frac{1}{2} r^{\frac{q}{2}}\}$  and  $R^q\sim \Gamma (\alpha = \frac{d}{q},\beta = \frac{1}{2})$  [23]. Not all elliptical distributions can be used to create a valid process [5]. In the following, we will carefully choose the density generator  $g$  in  $\mathrm{EC}_d(\pmb {\mu},\mathbf{C},g)$  to define a consistent multivariate  $q$ -exponential distribution generalizable to a process appropriately.
3 THE  $Q$ -EXPONENTIAL PROCESS
To generalize  $\mathrm{EC}_d(\pmb {\mu},\mathbf{C},g)$  to a valid stochastic process, we need to choose proper  $g$  such that the resulting distribution satisfies two conditions of Kolmogorov extension theorem [35]:
Figure 2: Inconsistent (Gomez's) EP distribution  $\mathrm{EP}_d(\pmb{\mu},\mathbf{C},q)$  (left) vs. consistent Q-exponential distribution  $q - \mathrm{ED}_d(\pmb{\mu},\mathbf{C})$  (right). Both can be sampled using (6) with  $R^q \sim \Gamma(\alpha = \frac{d}{q}, \beta = \frac{1}{2})$  and  $R^q \sim \Gamma(\alpha = \frac{d}{2}, \beta = \frac{1}{2})$  respectively. Note there is significant discrepancy between the marginalization of  $\mathrm{EP}_3(\pmb{\mu},\mathbf{C},q)$  and  $\mathrm{EP}_2(\pmb{\mu},\mathbf{C},q)$ . However, the marginalization of  $q - \mathrm{ED}_3(\pmb{\mu},\mathbf{C})$  coincides with  $q - \mathrm{ED}_2(\pmb{\mu},\mathbf{C})$ . Empirical densities are estimated based on 10000 samples (shown as dots) for  $q = 1$ .
Theorem 3.1 (Kolmogorov's Extension). For all  $t_1, \dots, t_k \in T$ ,  $k \in \mathbb{N}$  let  $\nu_{t_1, \dots, t_k}$  be probability measures on  $\mathbb{R}^{dk}$  satisfying
$$
(K 1): \nu_ {t _ {\sigma (1)}, \dots , t _ {\sigma (k)}} \left(F _ {1} \times \dots \times F _ {k}\right) = \nu_ {t _ {1}, \dots , t _ {k}} \left(F _ {\sigma^ {- 1} (1)} \times \dots \times F _ {\sigma^ {- 1} (k)}\right) f o r a l l p e r m u t a t i o n s \sigma \in S (k)
$$
$$
(K 2): v _ {t _ {1}, \dots , t _ {k}} \left(F _ {1} \times \dots \times F _ {k}\right) = v _ {t _ {1}, \dots , t _ {k}, t _ {k + 1}, \dots , t _ {k + m}} \left(F _ {1} \times \dots \times F _ {k} \times \mathbb {R} ^ {d} \times \dots \times \mathbb {R} ^ {d}\right) f o r a l l m \in \mathbb {N} \tag {7}
$$
Then there exists a probability space  $(\Omega, \mathcal{F}, P)$  and a stochastic process  $\{X_t\}$  on  $\Omega$ ,  $X_t : \Omega \to \mathbb{R}^n$  such that
$$
\mathcal {V} _ {t _ {1}, \dots , t _ {k}} \left(F _ {1} \times \dots \times F _ {k}\right) = P \left[ X _ {t _ {1}} \in F _ {1}, \dots , X _ {t _ {k}} \in F _ {k} \right] \tag {8}
$$
for all  $t_i \in T$ ,  $k \in \mathbb{N}$  and all Borel sets  $F_i \in \mathcal{F}$ . (K1) and (K2) are referred to as exchangeability and consistency conditions respectively.
As pointed out by Kano [27], the elliptic distribution  $\mathrm{EC}_d(\pmb{\mu},\mathbf{C},g)$  in the format of Gomez's EP distribution (4) with  $g(r) = \exp \left\{-\frac{1}{2} r^{\frac{q}{2}}\right\}$  does not satisfy the consistency condition [also c.f. Proposition 5.1 of 23]. Figure 2 (left panel) also illustrates such inconsistency numerically. However, Kano's consistency theorem [27] suggests a different viable choice of  $g$  to make a valid generalization of  $\mathrm{EC}_d(\pmb{\mu},\mathbf{C},g)$  to a stochastic process [5]:
Theorem 3.2 (Kano's Consistency). An elliptic distribution is consistent if and only if its density generator function,  $g(\cdot)$ , has the following form
$$
g (r) = \int_ {0} ^ {\infty} \left(\frac {s}{2 \pi}\right) ^ {\frac {d}{2}} \exp \left\{- \frac {r s}{2} \right\} p (s) d s \tag {9}
$$
where  $p(s)$  is a strictly positive mixing distribution independent of  $d$  and  $p(s = 0) = 0$ .
3.1 Consistent Multivariate  $Q$ -exponential Distribution
In the above theorem 3.2, if we choose  $p(s) = \delta_{r^{\frac{q}{2} -1}}(s)$ , then we have  $g(r) = r^{\left(\frac{q}{2} -1\right)\frac{d}{2}}\exp \left\{-\frac{r^{\frac{q}{2}}}{2}\right\}$ , which leads to the following consistent multivariate  $q$ -exponential distribution  $\mathbf{q} - \mathrm{ED}_d(\pmb {\mu},\mathbf{C})$ .
Definition 3.1. A multivariate  $q$ -exponential distribution, denoted as  $\mathrm{q - ED}_d(\pmb {\mu},\mathbf{C})$  has the following density
$$
p (\mathbf {u} | \boldsymbol {\mu}, \mathbf {C}, q) = \frac {q}{2} (2 \pi) ^ {- \frac {d}{2}} | \mathbf {C} | ^ {- \frac {1}{2}} \boxed {r ^ {\left(\frac {q}{2} - 1\right) \frac {d}{2}}} \exp \left\{- \frac {r ^ {\frac {q}{2}}}{2} \right\}, \quad r (\mathbf {u}) = (\mathbf {u} - \boldsymbol {\mu}) ^ {\top} \mathbf {C} ^ {- 1} (\mathbf {u} - \boldsymbol {\mu}) \tag {10}
$$
Remark 1. When  $q = 2$ ,  $\mathfrak{q} - \mathrm{ED}_d(\pmb{\mu}, \mathbf{C})$  reduces to MVN  $\mathcal{N}_d(\pmb{\mu}, \mathbf{C})$ . When  $d = 1$ , if we let  $C = 1$ , then we have the density for  $u$  as  $p(u) \propto |u|^{\frac{q}{2} - 1} \exp \left\{-\frac{1}{2}|u|^q\right\}$ , differing from the original un-normalized density  $\pi_q$  in (2) by a term  $|u|^{\frac{q}{2} - 1}$ . This is needed for the consistency of process generalization. Numerically, it has the similar "edge-preserving" property as the Besov prior.
Remark 2. If taken negative logarithm, the density of  $\mathfrak{q} - \mathrm{ED}_d$  in (10) yields a quantity dominated by some weighted  $L_{q}$  norm of  $\mathbf{u} - \pmb{\mu}$ , i.e.  $\frac{1}{2} r^{\frac{q}{2}} = \frac{1}{2}\|\mathbf{u} - \pmb{\mu}\|_{\mathbf{C}}^{q}$ . From the optimization perspective,  $\mathfrak{q} - \mathrm{ED}_d$ , when used as a prior, imposes  $L_{q}$  regularization in obtaining the maximum a posteriori (MAP).
Regardless of the normalizing constant, our proposed multivariate  $q$ -exponential distribution  $\mathbf{q}-\mathrm{ED}_d(\pmb{\mu},\mathbf{C})$  differs from the Gomez's EP distribution  $\mathrm{EP}_d(\pmb{\mu},\mathbf{C},q)$  by a boxed term  $r^{\left(\frac{q}{2} -1\right)\frac{d}{2}}$ . As stated in the following theorem,  $\mathbf{q}\text{-ED}_d$  satisfies the two conditions of Kolmogorov extension theorem thus is ready to generalize to a stochastic process (See the right panel of Figure 2 for the consistency).
Theorem 3.3. The multivariate  $q$ -exponential distribution is both exchangeable and consistent.
Proof. See Appendix A.1.
Like student- $t$  distribution [41] and other elliptic distributions [5], we can show (See Appendix A.5) that  $\mathrm{q - ED}_d$  is represented as a scale mixture of Gaussian distributions for  $0 < q < 2$  [27, 3, 47].
Numerically, thanks to our choice of density generator  $g(r) = r^{\left(\frac{q}{2} -1\right)\frac{d}{2}}\exp \left\{-\frac{r^{\frac{q}{2}}}{2}\right\}$ , one can show that  $R^{q}\sim \chi_{d}^{2}$  (as in Appendix A.4) thus  $R$  in Theorem 2.1 can be sampled as  $q$ -root of a  $\chi_d^2$  random variable, which completes the recipe for generating random vector  $\mathbf{u}\sim \mathbf{q} - \mathrm{ED}_d(0,\mathbf{C})$  based on the stochastic representation (6). This is important for the Bayesian inference as detailed in Section 3.3.1. Note the matrix  $\mathbf{C}$  in the definition (10) characterizes the covariance between the components, as shown in the following proposition.
Proposition 3.1. If  $\mathbf{u} \sim \mathrm{q - ED}_d(\pmb{\mu},\mathbf{C})$ , then we have
$$
\operatorname {E} [ \mathbf {u} ] = \boldsymbol {\mu}, \operatorname {C o v} (\mathbf {u}) = \frac {2 ^ {\frac {2}{q}} \Gamma \left(\frac {d}{2} + \frac {2}{q}\right)}{d \Gamma \left(\frac {d}{2}\right)} \mathbf {C} \dot {\sim} d ^ {\frac {2}{q} - 1} \mathbf {C}, a s d \rightarrow \infty \tag {11}
$$
Proof. See Appendix A.4.
3.2  $Q$ -exponential Process as Probabilistic Definition of Besov Process
To generalize  $\mathbf{u} \sim \mathrm{q - ED}_d(0,\mathbf{C})$  to a stochastic process, we need to scale it to  $\mathbf{u}^{*} = d^{\frac{1}{2} -\frac{1}{q}}\mathbf{u}$  so that its covariance is asymptotically finite. If  $\mathbf{u} \sim \mathrm{q - ED}_d(0,\mathbf{C})$ , then we denote  $\mathbf{u}^{*} \sim \mathrm{q - ED}_{d}^{*}(0,\mathbf{C})$  following a scaled  $q$ -exponential distribution. Let  $\mathcal{C}:L^q\to L^q$  be a kernel operator in the trace class, i.e. having eigen-pairs  $\{\lambda_{\ell},\phi_{\ell}(x)\}_{\ell = 1}^{\infty}$  such that  $\mathcal{C}\phi_{\ell}(x) = \phi_{\ell}(x)\lambda_{\ell}$ ,  $\| \phi_{\ell}\|_{2} = 1$  for all  $\ell \in \mathbb{N}$  and  $\operatorname {tr}(\mathcal{C}) = \sum_{\ell = 1}^{\infty}\lambda_{\ell} < \infty$ . Now we are ready to define the  $q$ -exponential process (Q-EP) with the scaled  $q$ -exponential distribution.
Definition 3.2 (Q-EP). A (centered)  $q$ -exponential process  $u(x)$  with kernel  $\mathcal{C}$  in the trace class,  $\mathbf{q} - \mathcal{E}\mathcal{P}(0,\mathcal{C})$ , is a collection of random variables such that any finite set,  $\mathbf{u} = (u(x_1),\dots u(x_d))$ , follows a scaled multivariate  $q$ -exponential distribution, i.e.  $\mathbf{u} \sim \mathbf{q} - \mathrm{ED}_d^* (0,\mathbf{C})$ .
Note, the process is defined on the  $d^{\star}$ -dimensional space depending on the applications  $(x\in \mathbb{R}^{d^{\star}})$ ; while  $d$  refers to the dimension of discretized process  $(\mathbf{u}\in \mathbb{R}^{d})$ . While Q-EP reduces to GP when  $q = 2$ ,  $q = 1$  is often adopted for imaging analysis as an "edge-preserving" prior. Illustrated in Figure 3 for selected  $q\in (0,2]$ , smaller  $q$  leads to sharper image reconstruction with varying  $q$  interpolating between different regularization effects. Both Besov and Q-EP are valid stochastic processes stemming from the  $q$ -exponential distribution  $\pi_q$ . They are both designed to generalize GP to have sharper regularization (through  $q$ ) but Q-EP has advantages in 1) the capability of specifying correlation structure directly and 2) the tractable prediction formula.
It follows from (1) immediately that the covariance of the Besov process  $u(\cdot)$  at two points  $x, x' \in \mathbb{R}^{d^*}$ :
$$
\operatorname {C o v} (u (x), u \left(x ^ {\prime}\right)) = \sum_ {\ell = 1} ^ {\infty} \gamma_ {\ell} ^ {2} \phi_ {\ell} (x) \otimes \phi_ {\ell} \left(x ^ {\prime}\right) \tag {12}
$$
Figure 3: Image of satellite: MAP estimates by Q-EP with varying  $q$  parameters.
Although the smoothness and correlation strength of Besov process can be configured by proper orthonormal basis  $\{\phi_{\ell}\}$  as in (12), it is less straightforward than the kernel function working for GP. On the other hand, Q-EP has more freedom on the correlation structure through (11) with flexible choices from a large class of kernels including powered exponential, Matérn and periodic where we can directly specify the correlation length.
While Q-EP can be viewed as a probabilistic definition of Besov, the following theorem further establishes their connection in sharing equivalent series representations.
Theorem 3.4 (Karhunen-Loève). If  $u(x) \sim \mathrm{q} - \mathcal{E}\mathcal{P}(0,\mathcal{C})$  with a trace class operator  $\mathcal{C}$  having eigen-pairs  $\{\lambda_{\ell},\phi_{\ell}(x)\}_{\ell = 1}^{\infty}$  such that  $\mathcal{C}\phi_{\ell}(x) = \phi_{\ell}(x)\lambda_{\ell}$ ,  $\| \phi_{\ell}\| _2 = 1$  for all  $\ell \in \mathbb{N}$  and  $\sum_{\ell = 1}^{\infty}\lambda_{\ell} < \infty$ , then we have the following series representation for  $u(x)$ :
$$
u (x) = \sum_ {\ell = 1} ^ {\infty} u _ {\ell} \phi_ {\ell} (x), \quad u _ {\ell} := \int_ {D} u (x) \phi_ {\ell} (x) \stackrel {{i n d}} {{\sim}} \mathrm {q} - \mathrm {E D} ^ {*} \left(0, \lambda_ {\ell}\right) \tag {13}
$$
where  $\operatorname{E}[u_{\ell}] = 0$  and  $\operatorname{Cov}(u_{\ell}, u_{\ell'}) = \lambda_{\ell} \delta_{\ell \ell'}$  with Dirac function  $\delta_{\ell \ell'} = 1$  if  $\ell = \ell'$  and  $0$  otherwise.
Proof. See Appendix A.5.
Remark 3. If we factor  $\sqrt{\lambda_{\ell}}$  out of  $u_{\ell}$ , we have the following expansion for  $Q$ -EP more comparable to (1) for Besov:
$$
u (x) = \sum_ {\ell = 1} ^ {\infty} \sqrt {\lambda_ {\ell}} u _ {\ell} \phi_ {\ell} (x), \quad u _ {\ell} \stackrel {i i d} {\sim} q - E D (0, 1) \propto \pi_ {q} (\cdot) \tag {14}
$$
3.3 Bayesian Modeling with  $Q$ -exponential Process
Now let us consider the generic Bayesian regression model:
$$
y = u (x) + \varepsilon , \quad \varepsilon \sim L (\because 0, \Sigma) \tag {15}
$$
$$
u \sim \mu_ {0} (d u)
$$