Title: Bayesian Learning via Q-Exponential Process
Venue: NeurIPS 2023
Accepted: yes
Keywords: Functional Regularization

Besov Process

$Q$-Exponential Distribution

Elliptic Contour Distribution
Abstract: Regularization is one of the most fundamental topics in optimization, statistics and machine learning. To get sparsity in estimating a parameter $u\in\mathbb{R}^d$, an $\ell_q$ penalty term, $\Vert u\Vert_q$, is usually added to the objective function. What is the probabilistic distribution corresponding to such $\ell_q$ penalty? What is the \emph{correct} stochastic process corresponding to $\Vert u\Vert_q$ when we model functions $u\in L^q$? This is important for statistically modeling high-dimensional objects such as images, with penalty to preserve certainty properties, e.g. edges in the image.
In this work, we generalize the $q$-exponential distribution (with density proportional to) $\exp{(- \frac{1}{2}|u|^q)}$ to a stochastic process named \emph{$Q$-exponential (Q-EP) process} that corresponds to the $L_q$ regularization of functions. The key step is to specify consistent multivariate $q$-exponential distributions by choosing from a large family of elliptic contour distributions. The work is closely related to Besov process which is usually defined in terms of series. Q-EP can be regarded as a definition of Besov process with explicit probabilistic formulation, direct control on the correlation strength, and tractable prediction formula. From the Bayesian perspective, Q-EP provides a flexible prior on functions with sharper penalty ($q<2$) than the commonly used Gaussian process (GP, $q=2$).
We compare GP, Besov and Q-EP in modeling functional data, reconstructing images and solving inverse problems and demonstrate the advantage of our proposed methodology.

Bayesian Learning via Q -Exponential Process

Abstract

Regularization is one of the most fundamental topics in optimization, statistics and machine learning. To get sparsity in estimating a parameter u ∈ R d , an ℓ q penalty term, ∥ u ∥ q , is usually added to the objective function. What is the probabilistic distribution corresponding to such ℓ q penalty? What is the correct stochastic process corresponding to ∥ u ∥ q when we model functions u ∈ L q ? This is important for statistically modeling high-dimensional objects such as images, with penalty to preserve certain properties, e.g. edges in the image. In this work, we generalize the q -exponential distribution (with density proportional to) exp ( -1 2 | u | q ) to a stochastic process named q-exponential (Q-EP) process that corresponds to the Lq regularization of functions. The key step is to specify consistent multivariate q -exponential distributions by choosing from a large family of elliptic contour distributions. The work is closely related to Besov process which is usually defined in terms of series. Q-EP can be regarded as a definition of Besov process with explicit probabilistic formulation, direct control on the correlation strength, and tractable prediction formula. From the Bayesian perspective, QEP provides a flexible prior on functions with sharper penalty ( q < 2) than the commonly used Gaussian process (GP, q = 2). We compare GP, Besov and Q-EP in modeling functional data, reconstructing images and solving inverse problems and demonstrate the advantage of our proposed methodology.

1 INTRODUCTION

Regularization on function spaces is one of the fundamental questions in statistics and machine learning. High-dimensional objects such as images can be viewed as discretized functions defined on 2d or 3d domains. Statistical models for these objects on function spaces demand regularization to induce sparsity, prevent over-fitting, produce meaningful reconstruction, etc. Gaussian process [GP 38, 24] has been widely used as an L 2 penalty (negative log-density as a quadratic form) or a prior on the function space. Despite the flexibility, sometimes random candidate functions drawn from GP are over-smooth for modeling certain objects such as images with sharp edges. To address this issue, researchers have proposed a class of L 1 penalty based priors including Laplace random field [37, 34, 28] and Besov process [31, 15, 25, 16]. They have been extensively applied in spatial modeling [37], signal processing [28], imaging analysis [44, 34] and inverse problems [31, 15]. Figure 1 demonstrates an application of nonparametric regression models on functions endowed with GP, Besov and our proposed q -exponential process (Q-EP) priors respectively to reconstruct a blurry image of a satellite. Q-EP model generates the best reconstruction, indicating its advantage over GP in modeling objects with abrupt changes or sharp contrast such as 'edges" in image.

For these high-dimensional (refer to its discretization) inhomogeneous objects on d ⋆ domains D ⊂ R d ⋆ , particularly 2d images with sharp edges ( d ⋆ = 2), one can model them as a random function u from a

Truth

Observation

Besov

q-Exponential

1.0

0.8

0.6

0.4

0.2

0.0

0.0

0.2

0.4

0.6

0.8

1.0 0.0

0.2

0.4

0.6

0.8

1.0 0.0

0.2

0.4

0.6

0.8

1.0 0.0

0.2

0.4

0.6

0.8

1.00.0

0.2

0.4

0.6

0.8

1.0

Besov process represented by the following series for a given orthonormal basis { φ ℓ } ∞ ℓ = 1 in L 2 ( D ) [31, 15]:

where q ≥ 1 and γ ℓ = κ -1 q ℓ -( s d ⋆ + 1 2 -1 q ) with (inverse) variance κ > 0 and smoothness s > 0. When q = 2 and { φ ℓ } is chosen to be Fourier basis, this reduces to GP [16] but Besov is often used with q = 1 and wavelet basis [31] to provide 'edge-preserving" function candidates suitable for image analysis. Historically, [32] discovered that the total variation prior degenerates to GP prior as the discretization mesh becomes denser and thus loses the edge-preserving properties in high dimensional applications. Therefore, [31] proposed the Besov prior defined as in (1) and proved its discretization-invariant property. Though straightforward, such series definition lacks a direct way to specify the correlation structure as GP does through the covariance function. What is more, once the basis { φ ℓ } is chosen, there is no natural way to make prediction with Besov process.

We propose a novel stochastic process named q-exponential process (Q-EP) to address these issues. We start with the q -exponential distribution π q ( · ) and generalize it to a multivariate distribution (from a family of elliptic contour distributions) that is consistent to marginalization. Such consistency requires the joint distribution and the marginalized one (by any subset of components) to have the same format of density (See Section 3). We then generalize such multivariate q -exponential distribution to the process Q-EP and establish its connection and contrast to the Besov process. Note, if we view the negative log-density of the proposed distribution and process, Q-EP would impose an Lq regularization on the function space, similarly as L 2 regularization given by GP whose negative log-density is a quadratic form of the input variable x (See Remark 2).

Connection to existing works The proposed Q-EP process is related to the studentt process (TP) [41] as alternatives to GP. TP generalizes multivariate t -distribution (MVT) and is derived as a scale mixture of GP. Both TP and Q-EP can be viewed as a special case of the elliptical process [5] which gives the condition on general elliptic distributions that can be generalized to a valid stochastic process. Both papers focus on extending GP to robust models for heavier tail data, while our proposed work innovates a new Bayesian learning method on function spaces through the regularization parameter q (See Figure 3 for its effect on regularization when it varies), as is usually done in the optimization. Both our proposed Q-EP and [5] are inspired by Kano's consistency result [27], however the later focuses on a completely different process named squeezebox process. Our work on Q-EP makes multi-fold contributions to the learning of functional data in statistics and machine learning:

We propose a novel stochastic process Q-EP corresponding to the Lq regularization on function spaces.

For the first time we define/derive Besov process probabilistically as Q-EP with direct ways to configure correlation and to make prediction.

We provide flexible Bayesian inference methods based on the Markov Chain Monte Carlo (MCMC) algorithms using a white-noise representation for Q-EP prior models.

The rest of the paper is organized as follows. Section 2 introduces the q -exponential distribution and its multivariate generalizations. We propose the Q-EP with details in Section 3 and introduce it as a nonparametric prior for modeling functional data. In Section 4 we demonstrate the advantage of Q-EP over GP and Besov in time series modeling, image reconstruction, and Bayesian inverse problems (Appendix C.4). Finally we discuss some future directions in Section 5.

2 THE Q -EXPONENTIAL DISTRIBUTION AND ITS MULTIVARIATE GENERALIZATIONS

Let us start with the q -exponential distribution for a scalar random variable u ∈ R . It is named in [15] and defined with the following density not in an exact form (as a probability density normalized to 1):

This q -exponential distribution (2) is actually a special case of the following exponential power (EP) distribution EP ( µ , σ , q ) with µ = 0, σ = 1:

where Γ denotes the gamma function. Note the parameter q > 0 in (3) controls the tail behavior of the distribution: the smaller q the heavier tail and vice versa. This distribution also includes many commonly used ones such as the normal distribution N ( µ , σ 2 ) for q = 2 and the Laplace distribution L ( µ , b ) with σ = 2 -1 / q b when q = 1.

How can we generalize it to a multivariate distribution and further to a stochastic process? Gomez [23] provided one possibility of a multivariate EP distribution, denoted as EP d ( µ , C , q ) , with the following density:

When q = 2, it reduces to the familiar multivariate normal (MVN) distribution N d ( µ , C ) .

Unfortunately, unlike MVN being the foundation of GP, the Gomez's EP distribution EP d ( µ , C , q ) fails to generalize to a valid stochastic process because it does not satisfy the marginalization consistency as MVN does (See Section 3 for more details). It turns out we need to seek candidates in an even larger family of elliptic (contour) distributions EC d ( µ , C , g ) :

Definition 2.1 (Elliptic distribution) . A multivariate elliptic distribution EC d ( µ , C , g ) has the following density [26]

where kd > 0 is the normalizing constant and g ( · ) , a one-dimensional real-valued function independent of d and kd , is named density generating function [19].

Every elliptic (contour) distributed random vector u ∼ EC d ( µ , C , g ) has a stochastic representation mainly due to Schoenberg [40, 12, 26], as stated in the following theorem.

Theorem 2.1. u ∼ EC d ( µ , C , g ) if and only if

where S ∼ Unif ( S d + 1 ) uniformly distributed on the unit-sphere S d + 1 , L is the Cholesky factor of C such that C = LL T , R ⊥ S and R 2 d = r ( u ) ∼ f ( r ) = π d 2 Γ ( d 2 ) kd r d 2 -1 g ( r ) .

The Gomez's EP distribution EP d ( µ , C , q ) is a special elliptic distribution EC d ( µ , C , g ) with g ( r ) = exp {-1 2 r q 2 } and R q ∼ Γ ( α = d q , β = 1 2 ) [23]. Not all elliptical distributions can be used to create a valid process [5]. In the following, we will carefully choose the density generator g in EC d ( µ , C , g ) to define a consistent multivariate q -exponential distribution generalizable to a process appropriately.

3 THE Q -EXPONENTIAL PROCESS

To generalize EC d ( µ , C , g ) to a valid stochastic process, we need to choose proper g such that the resulting distribution satisfies two conditions of Kolmogorov extension theorem [35]:

Exponential Power

Q-Exponential

20

marginal

10

marginal

10

full-dim

5

full-dim

X1

0

X1

0

-10

-5

-20

-10

15

20

15

10

10

X2

0

X2

5

-10

0

5

20

-10

30

20

5

X3

10

Ex

0

0

-10

-5

-20

20

0

20

20

0

20

20

0

20

-10

0

10

-10

0

10

0

10

X1

X2

X3

X1

X2

X3

Theorem 3.1 (Kolmogorov's Extension) . For all t 1 , · · · , t k ∈ T, k ∈ N let ν t 1 , ··· , t k be probability measures on R dk satisfying

(7) Then there exists a probability space ( Ω , F , P ) and a stochastic process { Xt } on Ω , Xt : Ω → R n such that

for all ti ∈ T, k ∈ N and all Borel sets F i ∈ F . (K1) and (K2) are referred to as exchangeability and consistency conditions respectively.

As pointed out by Kano [27], the elliptic distribution EC d ( µ , C , g ) in the format of Gomez's EP distribution (4) with g ( r ) = exp {-1 2 r q 2 } does not satisfy the consistency condition [also c.f. Proposition 5.1 of 23]. Figure 2 (left panel) also illustrates such inconsistency numerically. However, Kano's consistency theorem [27] suggests a different viable choice of g to make a valid generalization of EC d ( µ , C , g ) to a stochastic process [5]:

Theorem 3.2 (Kano's Consistency) . An elliptic distribution is consistent if and only if its density generator function, g ( · ) , has the following form

where p ( s ) is a strictly positive mixing distribution independent of d and p ( s = 0 ) = 0 .

3.1 Consistent Multivariate Q -exponential Distribution

In the above theorem 3.2, if we choose p ( s ) = δ r q 2 -1 ( s ) , then we have g ( r ) = r ( q 2 -1 ) d 2 exp { -r q 2 2 } , which leads to the following consistent multivariate q-exponential distribution q -ED d ( µ , C ) .

Definition 3.1. A multivariate q-exponential distribution, denoted as q -ED d ( µ , C ) , has the following density

Remark 1. When q = 2 , q -ED d ( µ , C ) reduces to MVN N d ( µ , C ) . When d = 1 , if we let C = 1 , then we have the density for u as p ( u ) ∝ | u | q 2 -1 exp { -1 2 | u | q } , differing from the original un-normalized density π q in (2) by a term | u | q 2 -1 . This is needed for the consistency of process generalization. Numerically, it has the similar 'edge-preserving" property as the Besov prior.

Remark 2. If taken negative logarithm, the density of q -ED d in (10) yields a quantity dominated by some weighted Lq norm of u -µ , i.e. 1 2 r q 2 = 1 2 ∥ u -µ ∥ q C . From the optimization perspective, q -ED d , when used as a prior, imposes Lq regularization in obtaining the maximum a posterior (MAP).

Regardless of the normalizing constant, our proposed multivariate q -exponential distribution q -ED d ( µ , C ) differs from the Gomez's EP distribution EP d ( µ , C , q ) by a boxed term r ( q 2 -1 ) d 2 . As stated in the following theorem, q -ED d satisfies the two conditions of Kolmogorov extension theorem thus is ready to generalize to a stochastic process (See the right panel of Figure 2 for the consistency).

Theorem 3.3. The multivariate q-exponential distribution is both exchangeable and consistent .

Like studentt distribution [41] and other elliptic distributions [5], we can show (See Appendix A.5) that q -ED d is represented as a scale mixture of Gaussian distributions for 0 < q < 2 [27, 3, 47].

Numerically, thanks to our choice of density generator g ( r ) = r ( q 2 -1 ) d 2 exp { -r q 2 2 } , one can show that R q ∼ χ 2 d (as in Appendix A.4) thus R in Theorem 2.1 can be sampled as q -root of a χ 2 d random variable, which completes the recipe for generating random vector u ∼ q -ED d ( 0 , C ) based on the stochastic representation (6). This is important for the Bayesian inference as detailed in Section 3.3.1. Note the matrix C in the definition (10) characterizes the covariance between the components, as shown in the following proposition.

Proposition 3.1. If u ∼ q -ED d ( µ , C ) , then we have

Proof.

3.2 Q -exponential Process as Probabilistic Definition of Besov Process

To generalize u ∼ q -ED d ( 0 , C ) to a stochastic process, we need to scale it to u ∗ = d 1 2 -1 q u so that its covariance is asymptotically finite. If u ∼ q -ED d ( 0 , C ) , then we denote u ∗ ∼ q -ED ∗ d ( 0 , C ) following a scaled q -exponential distribution. Let C : L q → L q be a kernel operator in the trace class, i.e. having eigen-pairs { λ ℓ , φ ℓ ( x ) } ∞ ℓ = 1 such that C φ ℓ ( x ) = φ ℓ ( x ) λ ℓ , ∥ φ ℓ ∥ 2 = 1 for all ℓ ∈ N and tr ( C ) = ∑ ∞ ℓ = 1 λ ℓ < ∞ . Now we are ready to define the q-exponential process (Q-EP) with the scaled q -exponential distribution.

Definition 3.2 (Q-EP) . A (centered) q-exponential process u ( x ) with kernel C in the trace class, q -EP ( 0 , C ) , is a collection of random variables such that any finite set, u = ( u ( x 1 ) , · · · u ( xd )) , follows a scaled multivariate q-exponential distribution, i.e. u ∼ q -ED ∗ d ( 0 , C ) .

Note, the process is defined on the d ⋆ -dimensional space depending on the applications ( x ∈ R d ⋆ ); while d refers to the dimension of discretized process ( u ∈ R d ). While Q-EP reduces to GP when q = 2, q = 1 is often adopted for imaging analysis as an 'edge-preserving" prior. Illustrated in Figure 3 for selected q ∈ ( 0 , 2 ] , smaller q leads to sharper image reconstruction with varying q interpolating between different regularization effects. Both Besov and Q-EP are valid stochastic processes stemming from the q -exponential distribution π q . They are both designed to generalize GP to have sharper regularization (through q ) but Q-EP has advantages in 1) the capability of specifying correlation structure directly and 2) the tractable prediction formula.

It follows from (1) immediately that the covariance of the Besov process u ( · ) at two points x , x ′ ∈ R d ⋆ :

Q-EP(q=0.5)

Q-EP (q=1)

Q-EP (q=1.5)

Gaussian(q=2)

1.0

0.8

0.6

0.4

0.2

0.0

0.0

0.2

0.4

0.6

0.8

1.00.0

0.2

0.4

0.6

0.8

1.00.0

0.2

0.4

0.6

0.8

1.00.0

0.2

0.4

0.6

0.8

1.0

Although the smoothness and correlation strength of Besov process can be configured by proper orthonormal basis { φ ℓ } as in (12), it is less straightforward than the kernel function working for GP. On the other hand, Q-EP has more freedom on the correlation structure through (11) with flexible choices from a large class of kernels including powered exponential, Matérn and periodic where we can directly specify the correlation length.

While Q-EP can be viewed as a probabilistic definition of Besov, the following theorem further establishes their connection in sharing equivalent series representations.

Theorem 3.4 (Karhunen-Loéve) . If u ( x ) ∼ q -EP ( 0 , C ) with a trace class operator C having eigen-pairs { λ ℓ , φ ℓ ( x ) } ∞ ℓ = 1 such that C φ ℓ ( x ) = φ ℓ ( x ) λ ℓ , ∥ φ ℓ ∥ 2 = 1 for all ℓ ∈ N and ∑ ∞ ℓ = 1 λ ℓ < ∞ , then we have the following series representation for u ( x ) :

where E [ u ℓ ] = 0 and Cov ( u ℓ , u ℓ ′ ) = λ ℓ δ ℓℓ ′ with Dirac function δ ℓℓ ′ = 1 if ℓ = ℓ ′ and 0 otherwise.

Remark 3. If we factor √ λ ℓ out of u ℓ , we have the following expansion for Q-EP more comparable to (1) for Besov:

3.3 Bayesian Modeling with Q -exponential Process

Now let us consider the generic Bayesian regression model:

where L ( · ; 0 , Σ ) denotes some likelihood model with zero mean and covariance Σ , and the mean function u can be given a prior either Besov or Q-EP. Because of the definition (1) in terms of expanded series, there is no explicit formula for the posterior prediction using Besov prior. By contrast, a tractable formula exists for the posterior predictive distribution for (15) with Q-EP prior µ 0 = q -EP ( 0 , C ) when the likelihood happens to be L ( · ; 0 , C ) = q -ED ( 0 , C ) , as stated in the following theorem.

Theorem 3.5 (Posterior Prediction) . Given covariates x = { xi } N i = 1 and observations y = { yi } N i = 1 following q -ED in the model (15) with q -EP prior for the same q > 0 , we have the following posterior predictive distribution for u ( x ∗ ) at (a) new point(s) x ∗ :

Remark 4. From Theorem 3.5, we know that Q-EP has same predictive mean as GP. But their predictive covariances differ by a constant ( 2 d ) 2 q Γ ( d 2 + 2 q ) Γ ( d 2 ) (asymptotically 1) based on Proposition 3.1.

When the likelihood L is not Q-EP, e.g. multinomial, such conjugacy is absent. Then we refer to the following sampling method for the posterior inference.

3.3.1 Inference by White-Noise MCMC

Wefollow [13] to consider the pushforward ( T ) of Gaussian white noise ν 0 for non-Gaussian measures µ 0 = T # ν 0. More specifically, we construct a measurable transformation T : z → u that maps standard Gaussian random variables to q -exponential random variables. The transformation based on the stochastic representation (6) is more straightforward than that for Besov based on series expansions proposed by [13].

Recall the stochastic representation (6) of u ∼ q -ED d ( 0 , C ) : u = R L S with R q ∼ χ 2 d and S ∼ Unif ( S d + 1 ) . We can rewrite S d = z ∥ z ∥ 2 , R q d = ∥ z ∥ 2 2 , for z ∼ N d ( 0 , I d ) . Therefore, we have the pushforward mapping ( T ) and its inverse ( T -1 ) as

Figure B.1 illustrates that sampling with the white-noise representation (17) is indistinguishable from sampling by the stochastic representation (6). Then we can apply such white-noise representation to dimension-independent MCMC algorithms including preconditioned Crank-Nicolson (pCN) [14], infinite-dimensional Metropolis adjusted Langevin algorithm ( ∞ -MALA) [10], infinite-dimensional Hamiltonian Monte Carlo ( ∞ -HMC) [7], and infinite-dimensional manifold MALA ( ∞ -mMALA) [8] and HMC ( ∞ -mHMC) [9]. See Algorithm 1 for an implementation on pCN, hence named white-noise pCN (wn-pCN) .

3.3.2 Hyper-parameter Tuning

As in GP, there are hyper-parameters in the covariance function of Q-EP, e.g. variance magnitude ( σ 2 ) and correlation length ( ρ ), that require careful adjustment and fine tuning. If the data process y ( x ) and its mean u ( x ) are both Q-EP with the same q , then we can have the marginal likelihood [38] as another Q-EP (c.f. Theorem 3.5). In general, when there is no such tractability, hyper-parameter tuning by optimizing the marginal likelihood is unavailable. However, we could impose conjugate hyper-priors on some parameters or even marginalize them to facilitate the inference of them (See Appendix A.7 for a proposition on such conditional conjugacy for the variance magnitude σ 2 ).

To tune the correlation length ( ρ ), we could impose a hyper-prior for ρ and sample from p ( ρ | u ) . We then alternate updating u and hyper-parameters in a Gibbs scheme. In general, one could also use Bayesian optimization methods [33, 18, 4] for the hyper-parameter tuning.

4 NUMERICAL EXPERIMENTS

In this section, we compare GP, Besov and Q-EP by modeling time series (temporal), reconstructing images (spatial) from computed tomography and solving a (spatiotemporal) inverse problem (Appendix C.4). These numerical experiments demonstrate that our proposed Q-EP enables faster convergence in obtaining a better maximum a posterior (MAP) estimate. What is more, whitenoise MCMC based inference provides appropriate uncertainty quantification (UQ) (by the posterior standard deviation). More numerical results can be found in the supplementary materials which also contain some demonstration codes. All the computer codes are publicly available at https://github.com/lanzithinking/Q-EXP .

4.1 Time Series Modeling

We first consider two simulated time series, one with step jumps and the other with sharp turnings, whose true trajectories are as follows:

Wegenerate the time series { yi } by adding Gaussian noises to the true trajectories evaluated at N = 200 evenly spaced points { t i } in [ 0 , 2 ] , that is, y ∗ i = u ∗ ( t i ) + ε i , ε i ind ∼ N ( 0 , σ 2 ∗ ( t i )) , i = 1 , · · · , N , ∗ = J , T . Let σ J / ∥ u J ∥ = 0 . 015 f or t i ∈ [ 0 , 2 ] and σ T / ∥ u T ∥ = 0 . 01 i f t i ∈ [ 0 , 1 ] ; 0 . 07 i f t i ∈ ( 1 , 2 ] . In addition, we also consider two real data sets of Tesla and Google stock prices in 2022. See Figures 4 (and Figures C.2) for the true trajectories (blue lines) and realizations (orange dots) respectively.

2022

2022

(a) Time series with sharp turnings (model fitting).

(b) Time series with turnings (prediction).

Besov

q-Exponential

2.5

q-Exponential

2.0

2.0

1.5

1.5

1.0

1.0

0.5

0.5

0.0

A

0.0

0.5

0.5

1.0

0.0

0.5

1.0

1.5

2.0

0.0

0.5

1.0

1.5

2.0

0.0

0.5

1.0

1.5

2.0

0.0

0.5

1.0

1.5

2.0

0.0

0.5

1.0

1.5

2.0

Besov

q-Exponential

q-Exponential

400

400

350

350

300

300

250

250

200

200

150

150

100

100

2022-01-01

2022

-09

2022

1O

2023

2022

2022

-01

2022

2022

-01

2023

01

2022

5-01

2022-

2-09-

2022-11-01

-01

2023-01-01

2022-01-01

2022-05-01

2022-09-

2022-11-

-01

2023-

10-

-01

2022-

10-

01

2022

2022-09-01

(c) Tesla stock prices in 2022 (model fitting).

(d) Tesla stock prices in 2022 (prediction).

We use the above likelihood and test three priors: GP, Besov and Q-EP. For Besov, we choose the Fourier basis φ 0 ( t ) = √ 2 , φ ℓ ( t ) = cos ( π ℓ t ) , ℓ ∈ N (results with other wavelet bases including Haar, Shannon, Meyer and Mexican Hat are worse hence omitted). For both GP and Q-EP, we adopt the Matérn kernel with ν = 1 2 , σ 2 = 1, ρ = 0 . 5 and s = 1: C ( t , t ′ ) = σ 2 2 1 -ν Γ ( ν ) w ν K ν ( w ) , w = √ 2 ν ( ∥ t -t ′ ∥ / ρ ) s . In both Besov and Q-EP, we set q = 1. Figures 4a and 4c (and Figures C.2a and C.2c) compare the MAP estimates (red dashed lines). We can see that Q-EP yields the best estimates closest to the true trajectories in the simulation and the best fit to the Tesla/Google stock prices. We also investigate the negative posterior densities and relative errors, ∥ ˆ u ∗-u ∗∥ / ∥ u ∗∥ , as functions of iterations in Figure C.1. Though incomparable in the absolute values, the negative posterior densities indicate faster convergence in both GP and Q-EP models than in Besov model. The error reducing plots on the right panels of subplots in Figure C.1 indicate that Q-EP prior model can achieve the smallest errors. Table 1 compares them in terms of root mean of squared error (RMSE) and log-likelihood (LL).

Then we consider the prediction problem. In the simulations, the last 1 / 8 portion and every other of the last but 3 / 8 part of the data points are selected for testing. The models with GP and Q-EP priors are trained on the rest of the data, as indicated by short 'ticks" in Figures 4b and 4d (and Figures C.2b and C.2d). For the Tesla/google stocks, we select every other day in the first half year, every 4 days in the 3rd quarter and every 8 days in the last quarter for training and test on the rest. They pose challenges on both interpolation (among observations) and extrapolation (at no-observation region) tasks. As we can see in those figures, uncertainty grows as the data become scarce. Nevertheless, the Q-EP yields smaller errors than GP. Note, such prediction is not immediately available for models with Besov prior.

Truth

Observation

Besov

q-Exponential

1.0

0.8

0.6

0.4

0.2

0.0

0.0

0.2

0.4

0.6

0.8

1.0-n/2-3n/10-n/10

n/10

3n/10

n/20.0

0.2

0.4

0.6

0.8

1.00.0

0.2

0.4

0.6

0.8

1.00.0

0.2

0.4

0.6

0.8

1.0

4.2 Computed Tomography Imaging

Computed tomography (CT) is a medical imaging technique used to obtain detailed internal images of human body. CT scanners use a rotating X-ray tube and a row of detectors to measure X-ray attenuations by different tissues inside the body from different angles. Denote the true imaging as a function u ( x ) on the square unit D =[ 0 , 1 ] 2 taking values as the pixels. The observed data, y , (a.k.a. sinogram) are results of Radon transformation ( A ) of the discretized n × n field u with n θ angles and ns sensors, contaminated by noise ε [6]:

In general n θ ns ≪ d = n 2 so the linear inverse problem is under-determined. Baysian approach could fill useful prior information (e.g. edges) in the sparse data.

We first consider the Shepp-Logan phantom, a standard test image created by Shepp and Logan in [42] to model a human head and to test image reconstruction algorithms. In this simulation, we create the true image u † for a resolution of n 2 = 128 × 128 and project it at n θ = 90 angles with ns = 100 equally spaced sensors. The generated sinogram is then added by noise with signal noise ratio SNR = ∥ Au † ∥ / ∥ ε ∥ = 100. The first two panels of Figure 5 show the truth and the observation.

Note, the computation involving a full sized ( d × d ) kernel matrix C for GP and Q-EP is prohibitive. Therefore, we consider its Mercer's expansion (12) with Fourier basis for a truncation at the first L = 2000 items. Figure 5 shows that while GP and Besov models reconstruct very blurry phantom images, the Q-EP prior model produces MAP estimate of the highest quality. For each of the three models, we also apply wn-pCN to generate 10000 posterior samples (after discarding 5000) and use them to reconstruct u (posterior mean or median) and quantify uncertainty (posterior standard deviation).

Table 2 summarizes the errors relative to MAP ( u ∗ ) and posterior mean ( u ) respectively, ∥ ˆ u -u † ∥ / ∥ u † ∥ (with ˆ u being u ∗ or u ), log-likelihood (LL), and several quality metrics in imaging analysis including the peak signal-to-noise ratio (PSNR) [20], the structured similarity index (SSIM) [46], and the Haar wavelet-based perceptual similarity index (HaarPSI) [39]. Q-EP attains the lowest error and highest quality scores in most cases. In Figure C.3, we compare the uncertainty by these models. It seems that GP has uncertainty filed with more recognizable shape than the other two. However, the posterior standard deviation by GP is much smaller (about 1% of that with Q-EP) compared with the other two. Therefore, this raises a red flag that GP could be over-confident about a less accurate estimate.

Truth

Observation

Besov

q-Exponential

1.0

0.8 -

- 9°0

0.4

0.2

00

0.0

0.2

0.4

0.6

0.8

-4nπ/53n/52n/5-n/5

00.0

0.2

0.4

0.6

0.8

1.0 0.0

0.2

0.4

0.6

0.8

1.0 0.0

0.2

0.4

0.6

0.8

1.0

Truth

Observation

Besov

q-Exponential

1.0

0.8 -

90

0.4

0.2

0.0

0.0

0.2

0.4

0.6

8°0

1.0 I

4n/5

3n/5

-2m/5

0.2

0.4

0.6

0.8

1.0 0.0

0.2

0.4

0.6

0.8

1.0 0.0

0.2

0.4

0.6

0.8

1.0

00.0

Finally, we apply these methods to CT scans of a human cadaver and torso from the Visible Human Project [1]. These images contain n 2 = 512 × 512 pixels and the sinograms are obtained with n θ = 200 angles and ns = 512 sensors. The first two panels of each row in Figure 6 show a highly calibrated CT reconstruction (treated as 'truth") and the observed sinogram. The rest three panels illustrate that both Besov and Q-EP models outperform GP in reconstructions, as verified in the quantitative summaries in Table C.2. Figure C.4 indicates that GP tends to underestimate the uncertainty.

In these CT reconstruction examples, we observe larger discrepancy of performance between Besov and Q-EP in the low-dimensional data-sparse application (Shepp-Logan phantom at resolution n 2 = 128 × 128 with n θ = 90 angles and ns = 100 sensors) compared with the high-dimensional data-intensive applications (two human body CTs at resolution n 2 = 512 × 512 with n θ = 200 angles and ns = 512 sensors). This may be due to the truncation in Mercer's kernel representation (12) and different rates of posterior contraction [21, 22, 2]. We will explore them in another journal paper.

5 CONCLUSION

In this paper, we propose the q -exponential process (Q-EP) as a prior on L q functions with a flexible parameter q > 0 to control the degree of regularization. Usually, q = 1 is adopted to capture abrupt changes or sharp contrast in data such as edges in the image as the Besov prior has recently gained popularity for. Compared with GP, Q-EP can impose sharper regularization through q . Compared with Besov, Q-EP enjoys the explicit formula with more control on the correlation structure as GP. The numerical experiments in time series modeling, image reconstruction and Bayesian inverse problems demonstrate our proposed Q-EP is superior in Bayesian functional data modeling.

In the numerical experiments of current work, we manually grid-search for the optimal hyperparameters. The reported results are not sensitive to some of these hyper-parameters such as the variance magnitude ( σ 2 ) and the correlation length ( ρ ) but may change drastically to others like the regularity parameter ( ν ) and the smoothness parameter ( s ). In future, we will incorporate hyper-priors for some of those parameters and adopt a hierarchical scheme to overcome such shortcoming. We plan to study the properties such as regularity of function draws of Q-EP and the posterior contraction, and compare the contraction rates among GP, Besov and Q-EP priors [21, 22, 2]. Future work will also consider operator based kernels such as graph Laplacian [16, 17, 29].

Acknowledgments and Disclosure of Funding

SL is supported by NSF grant DMS-2134256.

References

The visible human project.

Sergios Agapiou, Masoumeh Dashti, and Tapio Helin. Rates of contraction of posterior distributions based on p-exponential priors. Bernoulli , 27(3):1616 - 1642, 2021.

D. F. Andrews and C. L. Mallows. Scale mixtures of normal distributions. Journal of the Royal Statistical Society. Series B (Methodological) , 36(1):99-102, 1974.

Noor Awad, Neeratyoy Mallik, and Frank Hutter. DEHB: Evolutionary hyberband for scalable, robust and efficient hyperparameter optimization. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence . International Joint Conferences on Artificial Intelligence Organization, aug 2021.

Maria Bånkestad, Jens Sjölund, Jalil Taghia, and Thomas Schön. The elliptical processes: a family of fat-tailed stochastic processes. 03 2020.

Johnathan M. Bardsley. Applications of a nonnegatively constrained iterative method with statistically based stopping rules to ct, pet, and spect imaging. Electron. Trans. Numer. Anal. , 38:34-43, 2011.

A. Beskos, F. J. Pinski, J. M. Sanz-Serna, and A. M. Stuart. Hybrid Monte-Carlo on Hilbert spaces. Stochastic Processes and their Applications , 121:2201-2230, 2011.

Alexandros Beskos. A stable manifold MCMC method for high dimensions. Statistics & Probability Letters , 90:46-52, 2014.

Alexandros Beskos, Mark Girolami, Shiwei Lan, Patrick E. Farrell, and Andrew M. Stuart. Geometric MCMCfor infinite-dimensional inverse problems. Journal of Computational Physics , 335, 2017.

Alexandros Beskos, Gareth Roberts, Andrew Stuart, and Jochen Voss. MCMC methods for diffusion bridges. Stochastics and Dynamics , 8(03):319-350, 2008.

Alessandro Buccini, Mirjeta Pasha, and Lothar Reichel. Linearized krylov subspace bregman iteration with nonnegativity constraint. Numerical Algorithms , 87(3):1177-1200, sep 2020.

Stamatis Cambanis, Steel Huang, and Gordon Simons. On the theory of elliptically contoured distributions. Journal of Multivariate Analysis , 11(3):368-385, 1981.

Victor Chen, Matthew M. Dunlop, Omiros Papaspiliopoulos, and Andrew M. Stuart. Dimension-robust mcmc in bayesian inverse problems. 03 2018.

Simon L Cotter, Gareth O Roberts, AM Stuart, and David White. MCMC methods for functions: modifying old algorithms to make them faster. Statistical Science , 28(3):424-446, 2013.

Masoumeh Dashti, Stephen Harris, and Andrew Stuart. Besov priors for bayesian inverse problems. Inverse Problems and Imaging , 6(2):183-200, may 2012.

Masoumeh Dashti and Andrew M. Stuart. The Bayesian Approach to Inverse Problems , pages 311-428. Springer International Publishing, Cham, 2017.

Matthew M. Dunlop, Dejan Slepˇ cev, Andrew M. Stuart, and Matthew Thorpe. Large data and zero noise limits of graph-based semi-supervised learning algorithms. Applied and Computational Harmonic Analysis , 49(2):655-697, 2020.

Stefan Falkner, Aaron Klein, and Frank Hutter. BOHB: Robust and efficient hyperparameter optimization at scale. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning , volume 80 of Proceedings of Machine Learning Research , pages 1437-1446. PMLR, 10-15 Jul 2018.

K. Fang and Y.T. Zhang. Generalized Multivariate Analysis . Science Press, 1990.

Osama S. Faragallah, Heba El-Hoseny, Walid El-Shafai, Wael Abd El-Rahman, Hala S. El-Sayed, ElSayed M. El-Rabaie, Fathi E. Abd El-Samie, and Gamal G. N. Geweid. A comprehensive survey analysis for present solutions of medical image fusion and future directions. IEEE Access , 9:11358-11371, 2021.

Subhashis Ghosal, Jayanta K. Ghosh, and Aad W. van der Vaart. Convergence rates of posterior distributions. The Annals of Statistics , 28(2), apr 2000.

Subhashis Ghosal and Aad van der Vaart. Fundamentals of nonparametric bayesian inference. 2017.

E. Gómez, M.A. Gomez-Viilegas, and J.M. Marín. A multivariate generalization of the power exponential family of distributions. Communications in Statistics - Theory and Methods , 27(3):589-600, jan 1998.

A. P. Dawid J. M. Bernardo, J. O. Berger and A. F. M. Smith. Regression and classification using gaussian process priors. Bayesian Statistics , 6:475-501, 1998.

Junxiong Jia, Jigen Peng, and Jinghuai Gao. Bayesian approach to inverse problems for functions with a variable-index besov prior. Inverse Problems , 32(8):085006, 2016.

Mark E. Johnson. Multivariate Statistical Simulation , chapter 6 Elliptically Contoured Distributions, pages 106-124. Probability and Statistics. John Wiley & Sons, Ltd, 1987.

Y. Kano. Consistency property of elliptic probability density functions. Journal of Multivariate Analysis , 51(1):139-147, 1994.

Tomasz J. Kozubowski, Krzysztof Podgórski, and Igor Rychlik. Multivariate generalized laplace distribution and related random fields. Journal of Multivariate Analysis , 113:59-72, 2013. Special Issue on Multivariate Distribution Theory in Memory of Samuel Kotz.

Shiwei Lan. Learning temporal evolution of spatial dependence with generalized spatiotemporal gaussian process models. Journal of Machine Learning Research , 23(259):1-53, 2022.

Shiwei Lan, Shuyi Li, and Babak Shahbaba. Scaling up bayesian uncertainty quantification for inverse problems using deep neural networks. SIAM Journal of Uncertainty Quantification , 2022. to appear.

Matti Lassas, Eero Saksman, and Samuli Siltanen. Discretization-invariant bayesian inversion and besov space priors. Inverse Problems and Imaging , 3(1):87-122, 2009.

Matti Lassas and Samuli Siltanen. Can one use total variation prior for edge-preserving Bayesian inversion? Inverse Problems , 20(5):1537, 2004.

Lisha Li, Kevin Jamieson, Giulia DeSalvo, Afshin Rostamizadeh, and Ameet Talwalkar. Hyperband: A novel bandit-based approach to hyperparameter optimization. J. Mach. Learn. Res. , 18(1):6765-6816, jan 2017.

Felix Lucka. Fast markov chain monte carlo sampling for sparse bayesian inference in high-dimensional inverse problems using l1-type priors. Inverse Problems , 28(12):125012, nov 2012.

Bernt Øksendal. Stochastic Differential Equations . Springer Berlin Heidelberg, 2003.

N. Petra and G. Stadler. Model variational inverse problems governed by partial differential equations. Technical report, The Institute for Computational Engineering and Sciences, The University of Texas at Austin., 2011.

Krzysztof Podgórski and Jörg Wegener. Estimation for stochastic models driven by laplace motion. Communications in Statistics - Theory and Methods , 40(18):3281-3302, sep 2011.

Carl Edward Rasmussen and Christopher K. I. Williams. Gaussian Processes for Machine Learning . The MIT Press, 2005.

Rafael Reisenhofer, Sebastian Bosse, Gitta Kutyniok, and Thomas Wiegand. A haar wavelet-based perceptual similarity index for image quality assessment. Signal Processing: Image Communication , 61:33-43, 2018.

I. J. Schoenberg. Metric spaces and completely monotone functions. Annals of Mathematics , 39:811-841, 1938.

Amar Shah, Andrew Wilson, and Zoubin Ghahramani. Student-t Processes as Alternatives to Gaussian Processes. In Samuel Kaski and Jukka Corander, editors, Proceedings of the Seventeenth International Conference on Artificial Intelligence and Statistics , volume 33 of Proceedings of Machine Learning Research , pages 877-885, Reykjavik, Iceland, 22-25 Apr 2014. PMLR.

L. A. Shepp and B. F. Logan. The fourier reconstruction of a head section. IEEE Transactions on Nuclear Science , 21(3):21-43, 1974.

Andrew M Stuart. Inverse problems: a Bayesian perspective. Acta Numerica , 19:451-559, 2010.

Simopekka Vänskä, Matti Lassas, Samuli Siltanen, and Rolf Insitute. Statistical x-ray tomography using empirical besov priors. International Journal of Tomography and Statistics , 11, 06 2009.

Umberto Villa, Noemi Petra, and Omar Ghattas. hIPPYlib: An extensible software framework for large-scale inverse problems governed by PDEs; part i: Deterministic Inversion and Linearized Bayesian Inference. ACM Transactions on Mathematical Software , 47(2):1-34, jun 2021.

Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing , 13(4):600-612, 2004.

MIKE WEST. On scale mixtures of normal distributions. Biometrika , 74(3):646-648, 09 1987.

Supplement Document for 'Bayesian Learning via Q-Exponential Process"

A PROOFS

A.1 Proof of Theorem 3.3

Proof. First we prove the exchangeability of q -ED d ( µ , C ) with general (non-identity) covariance matrix C =[ C ( t i , t j )] d × d for some kernel function C . It actually holds for all elliptic distributions including MVN. Their densities contain the essential quadratic form r ( u ) = u T C -1 u which is invariant under any permutation of coordinates.

Denote u = [ ut 1 , · · · , uti , · · · , ut j , · · · , ut d ] T . Without loss of generality, we only need to show r ( u ) is invariant by switching two coordinates, say, t i ↔ t j . Denote u ′ =[ ut 1 , · · · , ut j , · · · , uti , · · · , ut d ] T . Switching t i and t j leads to a different covariance matrix C ′ obtained by switching both i -th and j -th rows and columns simultaneously in C . If we denote the elementary matrix Eij as derived from switching i -th and j -th rows of the identity matrix I . Then we have

Note Eij is idempotent, i.e. Eij = E -1 i j . Therefore

Next, the consistency directly follows from Kano's consistency Theorem 3.2 with our choice of g ( r ) . The proof is hence completed.

A.2 Theorem of Q-EP as a mixture of Gaussians

Theorem A.1. Suppose u ∼ q -ED d ( 0 , C ) for 0 < q < 2 , then there exist an random variable V > 0 and a standard normal random vector Z ∼ N d ( 0 , I ) independent of each other such that u d = Z / V.

Proof. Based on [3], it suffices to show ( -d dr ) k g ( r ) ≥ 0 for all k ∈ N . Observe that q

pk ( r ( q 2 -1 ) / 2 , r -1 ) exp {-r q 2 2 } where the coefficients of polynomial pk are all non-negative. Then we have

where pk + 1 ( r ( q 2 -1 ) / 2 , r -1 ) being the term in the square bracket has all positive coefficients because the powers ( q 2 -1 ) / 2 and -1 appear as coefficients in d dr pk ( r ( q 2 -1 ) / 2 , r -1 ) and are both negative. The proof is completed by induction.

A.3 Proposition of distribution of r ( u )

The following proposition determines the distribution of R = √ r ( u ) as q -root of a gamma (also chi -squared) distribution thus gives a complete recipe for generating random vector u ∼ q -ED d ( 0 , C ) based on the stochastic representation (6).

Proposition A.1. If u ∼ q -ED d ( 0 , C ) , then we have

Proof. With out chosen g ( r ) , the density of r becomes

A change of variable r → r q 2 yields the density of R q = r q 2 that can be recognized as the density of χ 2 d .

On the other hand, since v : = R q ∼ Γ ( α = d 2 , β = 1 2 ) , we have:

where we use Γ ( x + α ) · ∼ Γ ( x ) x α as x → ∞ with x = d 2 and α = k q when d → ∞ .

A.4 Proof of Proposition 3.1

Proof. By Theorem 2.6.4 in [19] for q -ED d ( µ , C ) = EC d ( µ , C , g ) with our chosen g , we know E [ u ] = µ and Cov ( u ) = ( E [ R 2 ] / rank ( C )) C . It follows by letting k = 2 in Proposition A.1 and using the similar asymptotic analysis.

A.5 Proof of Theorem 3.4

Proof. Note we can approximate φ ℓ ( x ) ∈ L 2 ( D ) with simple functions ˜ φ ℓ ( x ) = ∑ d i = 1 ki χ Di ( x ) where Di 's are measurable subsets of D and χ Di ( x ) = 1 if x ∈ Di and 0 otherwise. By the linear combination property of elliptic distributions [c.f. Theorem 2.6.3 in 19], ˜ u ℓ = ∫ D u ( x ) ˜ φ ℓ ( x ) dx ∼ q -ED ( 0 , c ) with scaling u ∗ = d 1 2 -1 q u in Definition 3.2. We have α d = Γ ( d 2 + 2 q ) Γ ( d 2 ) ( 2 d ) 2 q → 1 as d → ∞ . Taking the limit d → ∞ , we have u ℓ = ∫ D u ( x ) φ ℓ ( x ) dx ∼ q -ED ( 0 , c ) . In general, by the similar argument we have

c = α -1 d E [ ˜ u 2 ℓ ] to be determined. Note α d = 2 2 q Γ ( d 2 + 2 q ) d Γ ( d 2 ) d 1 -2 q comes from Proposition 3.1 and the

Thus it completes the proof.

A.6 Proof of Theorem 3.5

Before proving Theorem 3.5, we first prove the following lemma based on the conditional of elliptic distribution [12, 19].

for i = 1 , 2 and d 1 + d 2 = d, then we have the following conditional distribution

Proof. This directly follows from [Corollary 5 of Theorem 5 in 12] or [Corollary 3 of Theorem 2.6.6 in 19] for q -ED d ( µ , C ) = EC d ( µ , C , g ) with our chosen g .

Now we prove the Theorem 3.5.

Proof. By the linear combination property of the elliptic distributions [26, 19], we have y ∼ q -ED ( 0 , C + Σ ) . Then based on the consistency, we have the joint distribution

Therefore, the conclusion follows from Lemma A.1.

A.7 Proposition of Conditional Conjugacy for Variance Magnitude ( σ 2 )

Proposition A.2. If we assume a proper inverse-gamma prior for the variance magnitude such that u | σ 2 ∼ q -ED d ( µ , C = σ 2 C 0 ) , and σ q ∼ Γ -1 ( α , β ) , then we have

Proof. Denote r 0 =( u -µ ) T C -1 0 ( u -µ ) . We can compute the joint density of u and σ 2

By identifying the parameters for σ q we recognize that σ q | u is another inverse-gamma with parameters α ′ and β ′ as given.

B ALGORITHM

Algorithm 1 White-noise Preconditioed Crank-Nicolson (wn-pCN) for Q-EP Prior Models

1: Fix β ∈ ( 0 , 1 ] . Choose initial state z ( 0 ) ∈ R d .

2: for k = 0 , · · · , K -1 do

3: Propose ˆ z ( k ) =( 1 -β 2 ) 1 2 z ( k ) + β z ′ , z ′ ∼ N ( 0 , I ) .

4: Set z ( k + 1 ) = ˆ z ( k ) with acceptance probability

5: or else set z ( k + 1 ) = z ( k ) .

6: end for

C ADDITIONAL EXPERIMENTAL RESULTS

In this section, we present some additional numerical experimental results that cannot be included in the main text due to the page limit.

First, we numerically verify the equivalence between the stochastic representation (6) and the whitenoise representation (17) of q -ED d random variable in Figure B.1. More specifically, we generate 10000 samples using each of these two representations and illustrate in Figure B.1 that the two samples yield empirical marginal distributions (1d and 2d) close enough to each other.

C.1 Time Series Modeling

For modeling the simulated time series and stock prices, we include the optimization trace of negative (log)-posterior densities and relative errors for the two simulations and two stocks prices in Figure C.1. As commented in the main text, these plots show that Q-EP model can converge faster to lower errors compared with GP and Besov models.

Next, we compare MAP estimates by GP, Besov and Q-EP models in Figure C.2a for simulated time series with step jumps and in Figure C.2c for the Google stock prices in 2022. We also investigate the prediction results by GP and Q-EP in these two examples in Figures C.2b and C.2d. Table C.1 summarizes the RMSE of estimated stock prices by the three models and its standard deviation for repeating the experiments 10 times independently.

10

stoch-rep

wn-rep

5

10

-10

-5

5

X4

(a) Time series with step jumps.

(b) Time series with sharp turnings.

(c) Tesla stock prices in 2022.

(d) Google stock prices in 2022.

posterior

3 × 102

4 × 102

q-Exponential

Besov

relative error

negative posterior

Besov

q-Exponential

relative error

2 × 10-1

3 × 101

Besov

q-Exponential

2 ×102

101.

negative

9 × 102

8×10-2

101

102

7 × 102

Besov

102

0

100

iteration

200

300

400

6 ×102

0

100

iteration

200

q-Exponential

300

400

0

100

iteration

200

300

400

6 × 10-2

0

100

iteration

200

300

400

Besov

100

Besov

Besov

100

Besov

105

q-Exponential

q-Exponential

q-Exponential

q-Exponential

relative error

relative error

104

101

104

10-1

103

102

103

0

10

20

30

iteration

40

50

60

70

0

10

20

30

iteration

40

50

60

70

0

50

iteration

100

150

200

0

50

iteration

100

150

200

C.2 Computed Tomography Imaging

In the problem of reconstructing human head and torso CT images, Table C.2 compares GP, Besov and Q-EP models in terms of relative error (RLE), log-likelihood (LL), and imaging quality metrics including PSNR, SSIM and HarrPSI. In most cases, Q-EP outperforms, or achieves comparable scores with the other two methods.

Lastly, Figures C.3 and C.4 show that the posterior standard deviations estimated by wn-pCN using GP model could be misleading because the seemingly more recognizable shape deludes the fact that they are about two orders of magnitude smaller in value compared with the other two models. This implies that GP might underestimate the uncertainty present in the observed sinograms in the CT imaging analysis.

2022

(a) Time series with step jumps (model fitting).

(b) Time series with jumps (prediction).

Besov

q-Exponential

3.0

q-Exponential

2.0

2.5

2.0

1.5

1.5

1.0

1.0

0.5

0.5

0.0

0.0

0.5

0.0

0.5

1.0

1.5

2.0

0.0

0.5

1.0

1.5

2.0

0.0

0.5

1.0

1.5

2.0

0.0

0.5

1.0

1.5

2.0

0.0

0.5

1.0

1.5

2.0

Besov

q-Exponential

q-Exponential

140

140

120

120

100

100

80

-09

-01

01

LO

TO-

-01

-05

5-01

2-09-

-01

10-

-01

-01

10

2022-01-01

2022

2023

2022

2022

2022

2022

2022

2022

2022

2022

2022-

2023-01-01

2022-01-01

2022-05-01

2022-09-

2022-11-

2023-01

2022-01

2022-09-01

2022-11-01

(c) Google stock prices in 2022 (model fitting).

(d) Google stock prices in 2022 (prediction).

Besov

q-Exponential

1.0

0.6

0.0200

0.8

0.8

0.0175

0.5

0.0

1

50

0.4

0.6

0.6

O.

0.4

O.

00

0.3

0.4

O.

75

0.2

0.2

O.

50

0.1

0.2

O.

2

5

0.0

0.0

0.2

0.4

0.6

0.8

1.0

0.0000

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.0

0.2

0.4

0.6

0.8

1.0

0.0

Besov

q-Exponential

1.0

20.0

20.0

80

0.20

17.5

17.5

15.0

15.0

0.6

0.15

12.5

12.5

0.4

0.10

10.0

10.0

7.5

7.5

0.2

0.05

5.0

5.0

2.5

2.5

0.0

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

Besov

q-Exponential

1.0

0.175

17.5

17.5

80

0.150

15.0

15.0

0.125

12.5

12.5

0.6

0.100

10.0

10.0

0.4

0.075

7.5

7.5

0.050

5.0

5.0

0.2

0.025

2.5

2.5

0.0

0.0

0.2

0.4

0.6

0.8

1.0

0.000

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.0

0.2

0.4

0.6

0.8

1.0

0.0

C.3 Noisy/Blurry Image Reconstruction

Next we consider reconstructing a (128 × 128 pixels) image of satellite shown on the leftmost of Figure 1 from a blurred observation next to it. The image itself can be viewed as a function u ( x ) on the square unit D =[ 0 , 1 ] 2 taking values as the pixels. When evaluating u ( x ) on the discretized domain, u ( x ) becomes a matrix of size 128 × 128, which can further be vectorized to u ∈ R d with d = 128 2 . The true image, denoted as u † , is blurred by applying a motion blur point spread function [PSF 11] and adding 5% Gaussian noise. The actual observation, y ( x ) , can be written as in the following linear model:

where A ∈ R J × d is the blur motion PSF with J = d and σε / ∥ Au ∥ = 5%. Note, the blurring effect in the observed image (the second from left of Figure 1) is mainly due to the PSF operator A , not the small Gaussian noise.

Besov q-Exponential

150

200

250

iteration

0.6

Besov

error

q-Exponential

0.4

ativee

0.3

0.2

0

50

100

150

200

250

300

350

400

iteration

We compare the reconstructions by MAP estimate in Figure 1. The output by GP is blurry and close to the observed image, which means that GP does not 'de-noise" much. The result by Besov is much better than GP due to the L 1 regularization but it is still not sharp enough. We can see that the Q-EP prior model produces the reconstruction of the highest quality. Figure 3 demonstrates the effect of q > 0: the smaller q , the more regularization and hence sharper reconstruction. We also compare their negative posterior densities and relative errors, ∥ ˆ u -u † ∥ / ∥ u † ∥ , in Figure C.5. The Q-EP prior model yields the smallest error among all the three models.

300

350

400

posterior

106

105

negative p

50

100

1.0

Time: 0.0 s

Time: 1.0 s

0.5

1.0

Time: 1.0 s

Time: 2.0 s

0.8

0.8

0.20

0.6

0.6

0.4

0.15

0.4

0.4

0.2

0.2

0.10

0.3

0.0

0.0

0.05

Time: 2.0 s

Time: 4.0 s

Time: 3.0 s

Time: 4.0 s

1.0

1.0

0.2

0.00

0.8

0.8

0.6

0.6

0.05

0.1

0.4

0.4

0.10

0.2

0.2

0.0

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.0

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

0.15

Truth

Besov

q-Exponential

1.0

0.5

0.8

0.4

0.6

0.3

0.4

0.2

0.2

0.1

0.0

0.0

0.0

0.2

0.4

0.6

0.8

1.00.0

0.2

0.4

0.6

0.8

1.00.0

0.2

0.4

0.6

0.8

1.0 0.0

0.2

0.4

0.6

0.8

1.0

C.4 Advection-Diffusion Inverse Problem

Finally, we consider a Bayesian inverse problem governed by a time-dependent advection-diffusion equation [36, 30] that can be applied to heat transfer, pollution tracing, etc. The inverse problem involves inferring an unknown initial condition u 0 ∈ L 2 ( Ω ) from spatiotemporal point measurements { y ( x i , t j ) } as

The forward mapping G : u 0 → O u maps the initial condition u 0 to pointwise spatiotemporal observations of the concentration field u ( x , t ) through the solution of the following advection-diffusion equation [36, 45]:

where Ω ⊂ [ 0 , 1 ] 2 is a bounded domain shown in Figure C.6a, κ = 10 -3 is the diffusion coefficient, and T > 0 is the final time. The velocity field v is computed by solving the following steady-state Navier-Stokes equation with the side walls driving the flow [36]. Here, p is the pressure, and Re is the Reynolds number, which is set to 100 in this example. The Dirichlet boundary data g ∈ R 2 is given by g = e 2 =( 0 , 1 ) on the left wall, g = -e 2 on the right wall, and g = 0 everywhere else.

To generate data, we set the true value of parameter u 0 in (C.4) as u † 0 = 0 . 5 ∧ exp {-100 [( x -0 . 35 ) 2 + ( y -0 . 7 ) 2 ] } , illustrated in the top left panel of Figure C.6a, which also shows a few snapshots of the solutions u ( x , t ) at other time points on a regular grid mesh of size 61 × 61. Spatiotemporal observations { y ( x i , t j ) } I , J i = 1 , j = 1 are collected at I = 80 selected locations { x i } I i = 1 around the boundary of two inner boxes (See Figure C.6a and also Figure C.6b ) across J = 16 time points { t j } J j = 1 evenly

forwardprediction(x=0.249,y=0.350)

0.15

0.15

0.20

0.10

0.10

0.15

0.05

0.05

0.00

0.10

0.00

-0.05

0.05

-0.05

0.10

Truth

0.10

Truth

0.00

Truth

0.15

0.15

-0.05

-0.20

q-Exponential

q-Exponential

q-Exponential

-0.20

0

1

2

3

4

5

0

1

2

3

4

5

0

1

2

3

4

5

t

t

t

distributed between 1 and 4 seconds (thus denoted as O u ) with noise variance Σ = σ 2 η I 1280 where ση = 0 . 5max O u , i.e. y ( x i , t j ) = G ( u † 0 ) + η i j = u ( x i , t j ) + η i j .

To solve the inverse problem of finding the initial condition u 0 in the Bayesian framework [43, 16], we impose u 0 with GP, Besov and Q-EP priors respectively and seek the posterior p ( u 0 | y ) . For GP and QEP, we adopt a covariance kernel, C =( δ I -γ ∆ ) -2 , defined through the Laplace operator ∆ , where δ governs the variance of the prior and γ / δ controls the correlation length [16, 30]. We set γ = 1 and δ = 8 in this example. For Besov, we adopt 2d Fourier basis of the format φ i j = cos ( π ix 1 ) cos ( π jx 2 ) and truncate the series (1) for the first L = 1000 terms.

We apply wn-pCN to this challenging nonlinear inverse problem with high dimensionality (3413) of spatially discretized u at each time t . Figure C.7 compares the posterior mean estimates of u 0 given by these three models. Because the truth (leftmost) has clear edge at its cutoff by 0 . 5, Q-EP is more appropriate than GP and it indeed generates better estimate closer to the truth. Figure C.8 plots the prediction of forward mapping at a few selective locations on the left side of lower inner box by G ( x , t ∗ ) = 1 S ∑ S s = 1 G ( u ( s ) )( x , t ∗ ) with u ( s ) ∼ p ( u 0 | y ) . Compared with GP, Q-EP predicts the solution path closer to the truth G ( u † 0 )( x , t ∗ ) where the observations see more dynamical changes. More importantly, Q-EP provides proper UQ with credible bands wide enough to include the true trajectories. On the other hand, the posterior estimates by GP come with much narrower error bands that miss the truth. Again, we observe GP prior model being overconfident about less accurate estimates.

Peer reviews:
comments: SUMMARY:
This paper proposes a generalization of Besov processes to higgher dimensions that satisfies the stochastic process constraints that previous works could not satisfy. This is achieved by finding the right radius density function so that the corresponding elliptic distribution satisfies Kolmogorov's extension theorem. They then demonstrate how one may do Bayesian modeling and inference with this Q-EP process and provide experiments demonstrating their results.

STRENGTHS:
* Very interesting stochastic process family that I can see having a lot of use cases
* Compelling experimental results showing the efficacy of their method

WEAKNESSES:
* The experimental results use pretty small images. How well do these methods scale up to larger images and high-dimensional domains more generally?

QUESTIONS:
* How should one choose q? Should q=1 always be the default starting point?

LIMITATIONS:
The authors have adequately addressed the limitations.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nMotivated by the correspondence between Gaussian Process priors and ridge regularization for non-parametric regression problems, the authors in this paper develop a stochastic process prior, namely the $\\textbf{Q-exponential (Q-EP) process}$, which can correspond to $\\ell_q$-regularization. Specifically, by starting from multivariate  $q$-exponential distributions, the authors verify a Kolmogorov Consistency criterion to eventually develop the aforementioned process. Subsequently, the authors further provide justifications how the (Q-EP) process process allows a more flexible modeling of covariance kernels compared to parallel Besov processes also designed for similar tasks and derive a posterior predictive formula. Lastly, the benefits of the prior are demonstrated through numerical examples pertaining to problems in functional data analysis, image reconstruction, and solving inverse problems.\n\nSTRENGTHS:\nOffers a detailed probabilistic description and construction of a stochastic process prior for non-parametric Bayesian regression which (i) corresponds to $\\ell_q$-regularization; (ii) is developed from easy to understand first principles; and (iii) allows easy computation of posterior predictive distributions.\n\nWEAKNESSES:\nTheoretically, it was not discussed in which type of non-parametric problems and function spaces one might get better rates of posterior contraction using the developed process type priors compared to Gaussian process priors. Most of the theory is basic probabilistic calculations and quite stratighforward.\n\nQUESTIONS:\nThere is substantive literature on \"minimax optimal\" posterior contraction over standard function spaces (e.g. Sobolev, Besov etc.) results for suitable designed Gaussian process priors  -- can the authors discuss whether there are some set ups of non-parametric regression problems where some version of the priors developed here offers optimal posterior contraction over some function spaces?\n\nLIMITATIONS:\nNone noted.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nAs a generalization of GP, it is important to construct a stochastic process (prior) that can express various degrees of smoothness. As such process, the Besov processes have been proposed but they are defined in the form of a series expansion, and the corresponding probability distribution is not given in an easy-to-handle form.\nIn this work, the q-exponential process is proposed, which addresses the drawback of the conventional Besov process. A probability distribution called the q-exponential distribution (q-ED) that facilitates sampling from the Besov process is proposed. It is extended to multivariate in a consistent manner in the sense of marginalization. q-ED is expressed as an elliptic distribution, which also satisfies the exchange rate, thus satisfying the conditions of Kolmogorov's extension theorem, and the existence of the corresponding stochastic process is guaranteed. The KL expansion shows that the q-exponential process (q-EP) corresponding to the q-ED can be defined and has a series representation almost equivalent to the Besov process.\nThe q-EP has the strong advantage that the posterior predictive distribution can be constructed by MCMC, and Bayesian regression can be performed.\n\nSTRENGTHS:\nIt is an important contribution from the viewpoint of statistical modeling to define an explicit probability distribution as q-ED and to provide a sampling method for Besov processes described in series representation. q-ED is firstly defined as a one-dimensional probability distribution and then is extended to a multivariate distribution paying attention to consistency. The method of extending to a stochastic distribution and the method of performing MCMC inference (posterior distribution calculation) based on stochastic representation is standard and reasonable in the context of SDE simulation (e.g., a scaled mixture of normals). \nThe proposed q-EP will have wide applicability and some successful application results are presented in the paper.\n\nWEAKNESSES:\nMinor issue:\nIt is claimed that \"we have less control on the correlation strength once the orthonormal basis is chosen\". It is true but when we are free to choose the orthonormal basis, I feel the conventional Besov process has equivalent or even better freedom for specifying the correlation structure to the proposed q-EP. I acknowledge that specifying the correlation structure via the covariance function is handy, but in terms of the degree of freedom, it is fair to consider the case we can freely choose the basis.\n\nIt is claimed in line 218 that \"codes will be publicly available at to-be-released\", hence currently I cannot evaluate the reproducibility of the experimental results.\n\nVery minor presentation issues:\n- Eq.(5) and elsewhere, p(u) should be treated as the function of u, not r. I mean, for example in Eq.(5), \np(u) = k_d |C|^{-1/2} g(r(u)).\n\n- F_i in Theorem 3.1. should be defined.\n- \\mathbb{R}^k in Eq.(7) should be \\mathbb{R}^{n}\n\n- References should be properly described. For example, ref [1] does not have a link to the project, and ref [12] lacks bibliographic information.\n\nQUESTIONS:\nQ1: In my understanding, the proposed q-EP is essentially equivalent to the Besov process, and the major difference is its explicit control of the covariance structure. If this is the case, what is the main reason for the performance gap between Besov and q-EP in experiments, particularly for image reconstruction? If not, please reveal the difference between Besov proc. and q-EP. In particular, does q-EP includes Besov proc as a special case?\n\nQ2: Related to the problem mentioned in 6.Weakness, I'm not sure the experimental setting in subsection 4.1 is fair. For the Besov process, the Fourier basis is chosen and fixed. It is plausible but the inferior performance of the Besov process could be contributed to the bad choice of the basis. Please validate this experimental setting in more detail.\n\nQ3: I don't understand the meaning of the second sentence in \"Introduction\". How the \"High-dimensional objects\" can be viewed as \"evaluation of proper functions\", and what is the \"proper function\"?\n\nQ4: What is the difference between \"d^{\\star}\" in line 34 and \"d\" in other places?\n\nLIMITATIONS:\nI'm concerned that the scale mixture expression is not possible for all q. \nSee, e.g., \nM. West, ``On scale mixtures of normal distributions'', Biometrika (1987).\nI found that the range 0 < q < 2 is specified in the Appendix, but should be explicitly stated in the body of the paper to make the applicability of the methodology clear.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-21"}

comments: SUMMARY:
This paper proposed a new random process prior that corresponds to estimating parameters with $\ell_q$ penalty. The process, named Q-EP, can be used to provide a shaper penalty than the standard Gaussian process. Empirical experiments show the practical use case for Q-EP.

STRENGTHS:
The derivation of Q-EP looks solid to me. I am not sure about the novelty of the work. 

The experimental results look convincing, though I do not know if there is a standard benchmark.

WEAKNESSES:
It is hard for people unfamiliar with the field to understand the paper. In particular, the authors do not give a fair amount of text to explain the background. It is hard to relate the abstract math in the introduction with the examples provided (e.g. Figure 1). There is no clear statement claiming the connection between Q-EP with the $L_p$ regularization. The Bayesian model is not introduced until page 6. In sum, I suggest a major rewriting of the paper, potentially with a different organization.

QUESTIONS:
What is your insight when comparing the Bayesian models with various denoising works in deep learning? I believe they are very different in many aspects.

LIMITATIONS:
I do not see any potential negative societal impact of this paper.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-27

comments: SUMMARY:
The authors propose the 'q-exponential process', a stochastic process interpretation of Lq function regularization, which can induce sparsity in the solutions to optimization problems and can be used as a functional prior for Bayesian applications in time series regression, image reconstruction, and other applications.

STRENGTHS:
The proposed q-exponential process has several important benefits:
- enforcing sparsity or more sharpness in function space
- control of correlation structure, similarly to GP
- flexibility in choice of kernel
- conjugacy for posterior prediction with appropriate likelihoods, in contrast to the Besov case

WEAKNESSES:
The paper could provide more background on the problem setting and Besov processes, as the current introduction is difficult to follow.

QUESTIONS:
It may be interesting to further unpack Remark 3 regarding the similar mean & covariance for Q-EP and GP, for example showing how this affects predictions for Q-EP and GP in a tractable toy example.

LIMITATIONS:
The authors note the need for grid search over hyperparameters such as regularity parameter and smoothness parameter.    The method is tested only for q=1 and it would be interesting to explore other values.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-27