Title: Solving and Learning Partial Differential Equations with Variational Q-Exponential Processes
Venue: NeurIPS 2025
Accepted: yes
Keywords: Probabilistic PDE Solvers

Bayesian Inverse Problems

Data Inhomogeneity

Modeling Derivatives

Uncertainty Quantification
Abstract: Solving and learning partial differential equations (PDEs) lies at the core of physics-informed machine learning. Traditional numerical methods, such as finite difference and finite element approaches, are rooted in domain-specific techniques and often lack scalability. Recent advances have introduced neural networks and Gaussian processes (GPs) as flexible tools for automating PDE solving and incorporating physical knowledge into learning frameworks. While GPs offer tractable predictive distributions and a principled probabilistic foundation, they may be suboptimal in capturing complex behaviors such as sharp transitions or non-smooth dynamics. To address this limitation, we propose the use of the Q-exponential process (Q-EP), a recently developed generalization of GPs designed to better handle data with abrupt changes and to more accurately model derivative information. We advocate for Q-EP as a superior alternative to GPs in solving PDEs and associated inverse problems. Leveraging sparse variational inference, our method enables principled uncertainty quantification -- a capability not naturally afforded by neural network-based approaches. Through a series of experiments, including the Eikonal equation, Burgers’ equation, and an inverse Darcy flow problem, we demonstrate that the variational Q-EP method consistently yields more accurate solutions while providing meaningful uncertainty estimates.

Solving and Learning Partial Differential Equations with Variational Q-Exponential Processes

Abstract

Solving and learning partial differential equations (PDEs) lies at the core of physicsinformed machine learning. Traditional numerical methods, such as finite difference and finite element approaches, are rooted in domain-specific techniques and often lack scalability. Recent advances have introduced neural networks and Gaussian processes (GPs) as flexible tools for automating PDE solving and incorporating physical knowledge into learning frameworks. While GPs offer tractable predictive distributions and a principled probabilistic foundation, they may be suboptimal in capturing complex behaviors such as sharp transitions or non-smooth dynamics. To address this limitation, we propose the use of the q -exponential process (Q-EP), a recently developed generalization of GPs designed to better handle data with abrupt changes and to more accurately model derivative information. We advocate for Q-EP as a superior alternative to GPs in solving PDEs and associated inverse problems. Leveraging sparse variational inference, our method enables principled uncertainty quantification - a capability not naturally afforded by neural network-based approaches. Through a series of experiments, including the Eikonal equation, Burgers' equation, and an inverse Darcy flow problem, we demonstrate that the variational Q-EP method consistently yields more accurate solutions while providing meaningful uncertainty estimates.

Keywords: Probabilistic PDE Solvers, Bayesian Inverse Problems, Data Inhomogeneity, Modeling Derivatives, Uncertainty Quantification

1 Introduction

It is of fundamental importance in science and technology to solve mathematical models represented as a system of differential equations and to learn such a complex system by identifying crucial physical quantities with estimated uncertainty (a.k.a. inverse problems). Over centuries, theoretic foundations and computational methods have been developed for solving and learning partial differential equations (PDEs), which is facilitated by the development of modern computers. Traditional numerical algorithms such as the finite element method remain demanding for both domain knowledge and computing resources. Recently, there has been increasing interest and effort to efficiently automate this process using machine learning techniques.

The surge of physics-informed machine learning is driven by two main thrusts: neural network-based algorithms and Gaussian process (GP)-based probabilistic solvers. The former works are represented by physics-informed neural networks [PINN 37, 49], the deep Ritz method [9], the deep Galerkin method [44], and operator learning [20] methods including the Fourier neural operator [FNO 25], deep operator networks [DeepONet 26] and the neural inverse operator [NIO 28]. See [19] for a

review of recent advances. The basic idea is to parametrize the solution with a neural network and to minimize certain loss with respect to network parameters to obtain the solution. Despite empirical successes, these neural network-type approaches typically require large samples but lack convergence guarantee or uncertainty quantification (UQ). On the other hand, GP has been introduced to solve and learn ordinary differential equations (ODEs) [45, 39, 4, 43, 15] and PDEs [32, 33, 35, 38] with theoretic guarantee [34] and UQ [16]. More recent development on GP for solving and learning PDEs includes [6, 27, 14, 13, 29, 3]. Different from neural network approaches, these probabilistic solvers model the solution as a GP conditioned on PDE constraints and identify the solution as the maximum a posteriori (MAP).

Due to the tractability of conditional and predictive densities, GP has been widely adopted in machine learning and scientific computing [40]. However, as an L 2 regularization, GP tends to be over-smooth for modeling certain objects with abrupt changes or sharp contrast. For example, it is known in imaging analysis that GP may not detect or preserve edges very well in an image [23, 7]. On the other hand, researchers [42, 48] notice that total variation regularization penalizes the L 1 norm of derivatives and yields edge-preserving reconstructions. However, the total variation prior degenerates to GP prior with increasingly finer discretization mesh [23] and hence loses its edge-preserving feature. Therefore, [22] propose the Besov prior as an L q regularization and prove its discretizationinvariant property. [24] further develop the q -exponential process (Q-EP) as a probabilistic definition of the Besov process with tractable posterior prediction and demonstrate it as a superior generalization of GP (with q = 2 ) in modeling inhomogeneous data with sharp transitions.

In this paper, we discover that Q-EP (with q = 1 ) is better in modeling derivative information than GP, and hence presents as a preferable candidate for solving PDEs. Heuristically, this is attributed to Q-EP's enhanced ability to model inhomogeneous objects with sharp variations, resulting in better regularization of large derivatives. Theoretically, this can be explained by a faster posterior convergence rate in Bayesian modeling with Q-EP priors. Unlike optimization-based approaches [6, 27], we adopt sparse variational inference [46, 47] for Q-EP [30, 5] to solve and learn PDEs, allowing natural UQ. An emerging challenge is that in addition to mapping the Q-EP mean function by the nonlinear PDE dynamics, one also needs to propagate the whole variational distribution through, which no longer renders a Q-EP. We solve this difficulty by linearizing the complicated PDE mapping. We also extend the resulting variational Q-EP solver for inverse problems.

Connection to the literature Our work is motivated by [6] which optimizes the log-posterior for MAP as the PDE solution. Our proposed method replaces GP with a more general Q-EP and adopts variational Bayes for UQ. We investigate Q-EPs in solving various forward and inverse PDEs for a spectrum of q 's with q = 2 corresponding to GP. As a probabilistic solver, Q-EP may not be best compared with neural network-based approaches. However, we still include PINN [37] and Bayesian PINN [B-PINN 49] as baselines in our comparison. We emphasize that our algorithms rely only on limited data, e.g. boundary values or interior observations, while providing meaningful UQ. Our work is also related to the recently proposed physics-informed state-space GP [13], which however focuses on time-dependent PDEs. It adopts a variational spatiotemporal state-space GP and can be regarded as a related special case of ours for q = 2 . Our work on solving and learning PDEs makes multiple contributions to the field of physics-informed machine learning:

It is a novel probabilistic PDE solver based on Q-EP with superior capability of modeling data inhomogeneity and derivative information.

It theoretically justifies the preference of Q-EP over GP in solving and learning PDEs.

It provides efficient UQ for solving forward and inverse PDE problems.

The remainder of the paper is organized as follows. Section 2 reviews Q-EP as a flexible prior in Bayesian models for inhomogeneous data and introduces an extension to incorporate derivative information. Section 3 explains the details of applying Q-EP to solve the forward and inverse problems of PDEs. We follow [6] to model the solution as MAP of Q-EP but highlight the challenges of variational inference including distribution propagation and variational lower bound. In Section 4, we justify the preference of Q-EP for q = 1 over GP ( q = 2 ) in solving PDEs. In Section 5, we demonstrate the numerical advantages, particularly faster convergence, of Q-EP compared with alternatives using forward problems involving Eikonal equation and Burgers' equation and inverse problems of identifying permeability in the Darcy flow. Section 6 concludes with a discussion of limitations and future improvements.

2 Bayesian Modeling with Q-Exponential Process

2.1 Q -Exponential Process

The univariate q -exponential distribution [7] has density π q ( u ) ∝ exp( -1 2 | u | q ) , whose logarithm yields an L q regularization term. [24] generalize the univariate q -exponential random variable to a multivariate random vector, based on which a stochastic process can be defined. Suppose a function u ( x ) is observed at N locations, x 1 , · · · , x N ∈ Ω ⊂ R d . [24] define the multivariate q -exponential distribution for u := ( u ( x 1 ) , · · · , u ( x N )) , as a member of the family of elliptic distributions [18].

Definition 1. A multivariate random vector u ∈ R N follows the q -exponential distribution, denoted as u ∼ q -ED N ( µ , C ) , if it has the following density:

Remark 1. The negative log density of q -ED in (1) yields a quantity dominated by some weighted L q norm of u -µ , i.e. 1 2 r q 2 = 1 2 ∥ u -µ ∥ q C . From the optimization perspective, q -ED , when used as a prior, imposes L q regularization in obtaining the maximum a posteriori (MAP).

Li et al. [24] prove that the above multivariate q -exponential distribution satisfies the conditions of Kolmogorov's extension theorem [31] and thus can be generalized to a stochastic process. For this purpose, we scale u ∼ q -ED N ( 0 , C ) by a factor N 1 2 -1 q so that the scaled q -exponential random variable u ∗ := N 1 2 -1 q u ∼ q -ED ∗ N ( 0 , C ) has covariance asymptotically finite [Proposition 3.1 of 24]. With a covariance (symmetric and positive-definite) kernel C : Ω × Ω → R , we define the following q -exponential process (Q-EP) based on the scaled q -exponential distribution.

Definition 2. A (centered) q -exponential process u ( x ) with a kernel C , q -EP (0 , C ) , is a collection of random variables such that any finite set, u = ( u ( x 1 ) , · · · u ( x N )) , follows a scaled multivariate q -exponential distribution q -ED ∗ ( 0 , C ) , where C = [ C ( x i , x j )] N × N .

Remark 2. When q = 2 , q -ED N ( µ , C ) reduces to N N ( µ , C ) and q -EP (0 , C ) becomes GP (0 , C ) . When q ∈ (0 , 2) , q -EP (0 , C ) lends flexibility to modeling functional data with more regularization than GP. In practice, q = 1 is often adopted for faster posterior convergence [1, 21] and the capability of preserving inhomogeneous features (rough functional data, edges in image, etc).

The covariance kernel C is associated with a Hilbert-Schmidt (HS) integral operator T C : L 2 (Ω) → L 2 (Ω) , u ( · ) ↦→ ∫ Ω C ( · , x ′ ) u ( x ′ ) µ ( d x ′ ) which has eigen-pairs { λ ℓ , ϕ ℓ ( · ) } ∞ ℓ =1 such that for ∀ ℓ ∈ N , T C ϕ ℓ ( x ) = ϕ ℓ ( x ) λ ℓ and ∥ ϕ ℓ ∥ 2 = 1 . Assume T C is trace-class, i.e. tr( T C ) := ∑ ∞ ℓ =1 λ ℓ < ∞ . Theorem 3.4 of [24] presents a series representation of Q-EP similar to GP and the Besov process [8].

Theorem 2.1 (Karhunen-Loéve) . If u ( · ) ∼ q -EP (0 , C ) with a trace-class HS operator T C having eigen-pairs { λ ℓ , ϕ ℓ ( · ) } ∞ ℓ =1 , then we have the following series representation for u ( x ) :

where E[ u ℓ ] = 0 and Cov( u ℓ , u ℓ ′ ) = λ ℓ δ ℓℓ ′ with Dirac function δ ℓℓ ′ = 1 if ℓ = ℓ ′ and 0 otherwise. Moreover, we have E[ ∥ u ( · ) ∥ 2 2 ] = ∑ ∞ ℓ =1 E[ u 2 ℓ ] = tr( T C ) < ∞ .

2.2 Bayesian Regression with Q-EP Priors

Given X N × d = { x n } N n =1 and y N × 1 = { y n } N n =1 , we consider the Bayesian regression model:

Li et al. [24, Theorem 3.5] show that the posterior (predictive) distribution is analytically tractable when both the prior and the likelihood are q -exponential.

Theorem 2.2. For the regression model (3) , the posterior distribution of u ( x ∗ ) at x ∗ is

3.0

True values

3.0

True x-derivatives

3.0

True y-derivatives

True values

True x-derivatives

True y-derivatives

2.5

2.5

2.5

2.0

1.5

2.0

1.5

2.0

1.5

2

2

2

1.0

1.0

1.0

0

0

0

0.5

0.5

0.5

0.0

0.0

0.0

2

-2

-2

0.5

0.5

0.5

4

1.0

-2

1

0

1

Z

1.0

1

1

2

1.0

Z-

1

0

1

-4

2

0

Z

4

Z-

0

2

4

-4

2

0

2

4

3.0

Predicted values (q=1.0)

3.0

Predicted x-derivatives (q=1.0)

3.0

Predicted y-derivatives (q=1.0)

Predicted values (q=1.0)

Predicted x-derivatives (q=1.0)

Predicted y-derivatives (q=1.0)

2.5

2.5

2.5

4

4

4

2.0

2.0

2.0

2

2

2

1.5

1.5

1.5

1.0

1.0

1.0

0

0

0.5

0.5

0.5

0.0

0.0

0.0

-2

2

2

0.5

0.5

0.5

4

4

4

1.0

=2

-1

0

1

-1.0

-1

0

1

2

1.0

2

1

0

1

2

0

2

t

2

0

4

-2

0

2

4

3.0

Predicted values(q=2.0)

3.0

Predicted x-derivatives (q=2.0)

3.0

Predicted y-derivatives (q=2.0)

Predicted values (q=2.0)

Predicted x-derivatives (q=2.0)

Predicted y-derivatives (q=2.0)

2.5

2.5

2.5

4

4

2.0

1.5

2.0

1.5

2.0

1.5

2

1.0

1.0

1.0

0

0

Q

0.5

0.5

0.5

0.0

0.0

0.0

2

2

0.5

1.0

2

-1

0

0.5

1.0

2

0

0.5

1.0

Z-

4

2.3 Modeling with Derivative Information

Let u ∼ q -EP (0 , C ) . Denote the function and its derivatives by ˜ u = ( u, ∂ ∂ x u, · · · , ∂ k ∂ x k u ) up to order k . Because linear operation preserves elliptic distributions [18, 10], ˜ u ∼ q -EP (0 , ˜ C ) is also a Q-EP if C in Definition 2 is differentiable up to order k , where the augmented kernel, ˜ C , has a structure illustrated in Table A.1. For example, the (1 , 2) block of ˜ C is interpreted as Cov( u ( x ) , ∂ ∂ x ′ u ( x ′ )) = ∂ ∂ x ′ C ( x , x ′ ) . For solving the second-order PDEs in Section 5, we could adopt Matérn kernel ( matern52 for ν = 5 / 2 ), C ( x , x ′ ) = σ 2 (1 + √ 5 r + 5 3 r 2 ) exp( - √ 5 r ) , r = √ ∑ d i =1 ( x i -x ′ i ) 2 /ρ 2 i , for which the process is twice differentiable in the mean-square sense.

To prepare for solving PDEs, we define ∥ · ∥ s,q for u based on (2) with a smoothness parameter s > 0 and an integrability parameter q ≥ 1 [22, 7]: ∥ u ( · ) ∥ s,q = (∑ ∞ ℓ =1 ℓ τ q ( s ) q | u ℓ | q ) 1 q , τ q ( s ) = s d + 1 2 -1 q . Consider the Banach space B s,q (Ω) := { u : Ω → R | ∥ u ( · ) ∥ s,q < ∞} . If q = 2 and { ϕ ℓ } ∞ ℓ =1 form the Fourier basis, then B s, 2 (Ω) reduces to the Sobolev space H s (Ω) . For u to be regular enough, we make the assumption on ˜ C so that ˜ u ∈ L q (Ω) almost surely by the following proposition [21].

Assumption 1. Suppose λ = { λ ℓ } ∞ ℓ =1 are eigenvalues of HS operator T ˜ C for the kernel ˜ C . We assume λ ∈ ℓ q 2 , i.e. ∥ λ ∥ q 2 q 2 = ∑ ∞ ℓ =1 λ q 2 ℓ < ∞ .

Proposition 2.1. If ˜ u ( · ) ∼ q -EP (0 , ˜ C ) with a trace-class HS operator T ˜ C satisfying Assumption 1, then ˜ u ( · ) ∈ L q P ( R ∞ , L q (Ω)) := { ˜ u : Ω × R ∞ → R | E ( ∥ ˜ u ∥ q q ) < ∞} and E [ ∥ ˜ u ( · ) ∥ q q ] = ∥ λ ∥ q 2 q 2 < ∞ .

Let ˜ U := ˜ u ( X ) N × (1+ kd ) = [ u ( X ) , ∂ ∂ x u ( X ) , · · · , ∂ k ∂ x k u ( X )] . For brevity, we denote D = 1 + kd . Then Y N × D and E N × D are the corresponding observations and errors respectively in the model (3). We apply this model to Rosenbrock ( f ( x ) = ∑ d i =1 [100( x i +1 -x 2 i ) 2 +(1 -x i ) 2 ] ) and Rastrigin ( f ( x ) = 10 d + ∑ d i =1 [ x 2 i -10 cos(2 πx i )] ) test functions with function values and their derivatives observed on a 20 × 20 grid, i.e. N = 400 , d = 2 , k = 1 . Figure 1 contrasts Q-EP ( q = 1 . 0 ) and GP ( q = 2 . 0 ) in predicting function and derivative values on the 50 × 50 grid. Q-EP outperforms GP in yielding a more accurate recovery, especially in the more challenging example of Rastrigin function.

Heuristically, the superiority of Q-EP in modeling derivatives over GP comes from its improved ability to handle inhomogeneous data with sharp variation. From the perspective of the L q norm of the gradient, the L 1 norm imposes stronger regularization than the L 2 norm on large values in ∇ x f ,

resulting in a better model for prediction. In Section 4 we will give a more rigorous justification. In the following, we take advantage of Q-EP's capability of modeling derivative information and apply it to solve PDEs.

3 Solving Partial Differential Equations with Q-EP

3.1 Bayesian Solver

Consider the following general PDE defined on a bounded domain Ω ⊂ R d :

where D : B s,q (Ω) → L q (Ω) is a differential operator and B : B s,q ( ∂ Ω) → L q ( ∂ Ω) is a boundary operator with data f ∈ L q (Ω) and g ∈ L q ( ∂ Ω) . Here we assume a sufficiently large smoothness index s > 0 such that the PDE (4) is well-defined pointwise and has a unique strong solution [6]. Let Ω = Ω ∪ ∂ Ω . For convenience of exposition, we denote the joint operator as P = ( D , B ) : B s,q (Ω) → L q (Ω) , and the right-hand side function as h = ( f, g ) ∈ L q (Ω) .

A set of collocation points X = { x n } N n =1 consists of N d interior points X d = { x 1 , · · · , x N d ∈ Ω } and N b boundary points X b = { x N d +1 , · · · , x N ∈ ∂ Ω } , i.e. X = X d ∪ X b , and N = N d + N b . Regarding the evaluation of P ( u ) on X , we make the following assumption so that we can properly define the likelihood model.

Assumption 2. There exists a differentiable function P : R D → R such that P ( u )( x ) = P (˜ u ( x )) . And further there is a constant C > 0 such that ∥∇ P ∥ ≤ C .

Then P ( u )( X ) becomes a nonlinear function of ˜ u ( X ) , denoted as P (˜ u ( X )) = P ( u )( X ) . Let h = h ( X ) . The probabilistic solver seeks to obtain ˜ U = ˜ u ( X ) based on observations ( P (˜ u ( X )) , h ) .

Even if we model ˜ u ( X ) ∼ q -ED(˜ u , S ) with ˜ u , S to be specified in (8) in Section 3.2, the nonlinear mapping P would not render P (˜ u ( X )) another q -ED random variable. Therefore, to properly define the likelihood model, we propose the following distribution propagation by linearizing P :

where the Taylor expansion of P is about ˜ u 0 , which can be chosen as ˜ u n -1 from the previous training epoch or simply ˜ u , and δ > 0 is a small nugget to ensure positive definiteness of Γ .

Let Y N × 1 := P (˜ u ( X )) . The Q-EP solver aims to minimize the discrepancy E = Y -h . The potential (negative log-likelihood) function, Φ : L q (Ω) × R N → R , can then be defined:

where φ ( r ; Γ , N ) := -1 2 log | Γ | + N 2 ( q 2 -1 ) log r -1 2 r q 2 . Under Assumption 2, Φ is Lipschitz continuous in ˜ u , which is used in the convergence theorem in Section 4.

Proposition 3.1. Suppose that the PDE mapping P satisfies Assumption 2. Let q ∈ (0 , 2] . Then for every r > 0 , there exists L = L ( r ) > 0 such that for every Y ∈ R N and for all ˜ u 1 , ˜ u 2 ∈ L q (Ω) with max {∥ ˜ u 1 ∥ q , ∥ ˜ u 2 ∥ q } < r , | Φ(˜ u 1 ; Y ) -Φ(˜ u 2 ; Y ) | ≤ L ∥ ˜ u 1 -˜ u 2 ∥ q .

Therefore, the Bayesian model for the solution u to (4) can be summarized as

Our goal is to infer the posterior p ( ˜ U | Y ) ∝ p ( Y | ˜ U ) p ( ˜ U ) . Note that because the extended function ˜ u ( X ) enters the likelihood model in a nonlinear way, Theorem 2.2 does not apply. In the following, we solve the inference problem using variational Bayes.

3.2 Variational Inference

We approximate the posterior p ( ˜ U | Y ) with some variational distribution q ( ˜ U ) using the variational Bayes method, which aims to minimize the Kullback-Leibler divergence KL( q ( ˜ U ) ∥ p ( ˜ U | Y )) . Because log p ( Y ) = KL( q ( ˜ U ) ∥ p ( ˜ U | Y )) + L ( q ( ˜ U )) , it reduces to maximizing the lower bound L ( q ( ˜ U )) . The sparse variational approximation [46, 47] is adopted by introducing inducing points ˜ X ∈ R M × d with their function values ˜ V = ˜ u ( ˜ X ) ∈ R M × D .

With the variational distribution for inducing values q ( ˜ V ) ∼ q -ED MD ( µ , Σ ) , the marginal variational distribution q ( ˜ U ) can be obtained as [17, 30]

The final evidence lower bound (ELBO) L ∗ ( q ( ˜ U )) is (Refer to Section A.1 for more details.)

The variational solution q ( ˜ U ) can be obtained by maximizing the ELBO (9) with respect to the variational parameters ( µ , Σ , ˜ X ) and hyper-parameters in the kernel C . By introducing the M inducing points, the overall computational complexity is reduced from O ( N 3 ) to O ( NM 2 ) [47].

3.3 Bayesian Inverse Problems

The above Bayesian framework can be readily extended to solve inverse problems. The following adaptation enables us to obtain both forward and inverse PDE solutions simultaneously.

Suppose that the PDE (4) contains a quantity of interest, a ( x ) , which could appear in the differential equation D or as part of the boundary condition B . The task of Bayesian inverse problems is to find a true solution, a † , with proper UQ based on observations. Suppose a is differentiable enough and we denote ˜ a = ( a, ∂ ∂ x a, · · · , ∂ k ′ ∂ x k ′ a ) to the order k ′ ≤ k . Now the joint operator P applies to both u and a , which produces a nonlinear function of ˜ u ( X ) , ˜ a ( X ) when evaluated on X , denoted as P (˜ u ( X ) , ˜ a ( X )) = P ( u, a )( X ) .

In addition, there is an observation operator O such that observations, O ( u )( X ) = u ( X o ) , are obtained on some set of N o observation points, X o ⊂ Ω with | X o | = N o . This can be achieved by solving (4) with true a † in simulations or simply modeling measurement data as noisy realization of (4) in real-world applications. Let ˜ N = N + N o . We supplement the joint equation operator P with the observation operator O to form an augmented operator ˜ P = ( P , O ) . Therefore, we have ˜ Y ˜ N × 1 = ˜ P ( u, a )( X ) = ˜ P (˜ u ( X ) , ˜ a ( X )) = [ P (˜ u ( X ) , ˜ a ( X )) T , O (˜ u ( X )) T ] T . Similarly, we augment the right-hand side data h with observed u ( X o ) to make ˜ h ˜ N × 1 = [ h T , u ( X o ) T ] T .

If we model ˜ a ( X ) using q (˜ a ( X )) ∼ q -ED(˜ a , S a ) with variational mean and covariance ˜ a , S a respectively, then q (˜ u ( X )) q (˜ a ( X )) propagates through the PDE dynamics similarly as in (5). Finally, we summarize the Bayesian inverse model for a ( x ) as follows:

where ˜ a 0 can be similarly chosen as ˜ a n -1 from the previous training epoch or simply ˜ a . The variational Bayes procedure in Section 3.2 can be modified accordingly to obtain the variational solution of ˜ a ( X ) | ˜ Y . Meanwhile, we obtain the variational solution of ˜ u ( X ) | ˜ Y as a byproduct.

4 Convergence Theorem

In this section, we study the posterior contraction of the Bayesian model (7) in the infinite data limit. Similar theory can be developed for the model (10). We focus on q ∈ [1 , 2] and leave the technically more challenging case q ∈ (0 , 1) to future study. For brevity, we denote u = ˜ u , C = ˜ C , and n = N .

Consider the separable Banach space X = ( L q (Ω) , ∥ · ∥ q ) ⊃ ( B s,q (Ω) , ∥ · ∥ s,q ) for s > d ( 2 q -1 2 ) , and Y = R n . Define the concentration function of Q-EP measure Π at u = u † as

Let P ( n ) u be the measure of the observations Y ( n ) on ( Y , B , µ 0 ) having density p u and corresponding potential function Φ( u ; · ) with respect to the Lebesgue measure µ 0 , i.e. d P ( n ) u dµ 0 ( Y ) = p u ( Y ) ∝ exp( -Φ( u ; Y )) . Define the Hellinger distance as d 2 n,H ( u, u ′ ) = ∫ ( √ p u - √ p u ′ ) 2 dµ 0 . We have the following posterior contraction theorem.

Theorem 4.1 (Posterior Contraction) . Let u ∼ q -EP (0 , C ) with C satisfying Assumption 1 in Θ := L q (Ω) and P ( n ) u is the measure of Y ( n ) parameterized by u with PDE (4) satisfying Assumption 2. If the true value u † ∈ Θ is in the support of u , and ε n satisfies the rate equation φ u † ( ε n ) ≤ nε 2 n with ε n ≥ n -1 2 , then there exists Θ n ⊂ Θ such that Π n ( u ∈ Θ n : d n,H ( u, u † ) ≥ M n ε n | Y ( n ) ) → 0 in P ( n ) u † -probability for every M n →∞ .

Denote a ∧ b = min { a, b } , a ∨ b = max { a, b } , and x + = x ∨ 0 . By solving the inequality φ u † ( ε n ) ≤ nε 2 n for the minimal ε n , we obtain the posterior contraction rate as follows.

Theorem 4.2 (Contraction Rate) . Let u ∼ q -EP (0 , C ) with C satisfying Assumption 1 in Θ := L q (Ω) . The rest of the settings are the same as in Theorem 4.1. If the true value u † ∈ B s † ,q † (Ω) with s † > s ′ + ( d q † -d q ) + , s ′ = d q -d 2 , and q † , q ∈ [1 , 2] , then we have the rate of the posterior contraction as

Remark 3. If we set the smoothness parameter s = s † + d q -( d q † -d q ) + , and allow the integrability parameter q ≤ q † , then the contraction rate ε n is maximized as ε † n = n -1 2+ d s † -s ′ > n -1 2 . Note that this optimal rate is achieved regardless of the value of modeling regularization parameter q as long as q ≤ q † . This implies that when modeling inhomogeneous data and derivative information, under-smoothing (with smaller regularization parameter q ) is preferred to over-smoothing. When the true integrability q † is at least L 1 , setting q = 1 guarantees the fastest convergence of the posterior. In the following section, we will demonstrate such optimal choice with various numerical examples.

5 Numerical Experiments

In this section, we test variational Q-EP for solving and learning PDEs. Using the Eikonal equation,

Burgers' equation, a nonlinear elliptic equation (Section C.3), and an inverse problem involving the Darcy flow, we investigate the proposed method for a variety of different q s. We include PINN [37] and B-PINN [49] as baselines and the variational GP as a special case with q = 2 and mainly compare them using the relative error of the estimated solution ˆ u with reference to the true solution u † : RLEp = ∥ u † -ˆ u ∥ p ∥ u † ∥ p for p = 1 , 2 , ∞ . Q-EP with q = 1 outperforms PINN and variational GP in all cases and attains the best results in most comparisons. The numerical evidence also supports that Q-EP converges the fastest with the optimal choice of q = 1 .

1.0

FD Solution

0.40

Solution(q=0.8)

0.40

Uncertainty (q=0.8)

0.0001110

Solution(q=1.0)

0.40

Uncertainty (q=1.0)

0.0001140

0.4

0.2

0.8

0.6

0.35

0.15

0.30

0.25

0.20

0.10

0.05

0.4

0.8

0.2

0.6

0.35

0.30

0.25

0.20

0.15

0.05

0.10

0.8

0.6

0.4

0.2

0.0001095

0.0001080

0.0001035

0.0001020

0.0001065

0.0001050

0.6

0.4

0.8

0.2

0.35

0.30

0.20

0.15

0.25

0.10

0.05

0.8

0.6

0.4

0.2

0.0001125

0.0001110

0.0001080

0.0001050

0.0001095

0.0001065

0.0001035

0.0001020

0.0

0.0

0.2

0.4

0.6

0.8

1.0

0.00

0.2

0.4

0.6

0.8

0.00

0.2

0.4

0.6

0.8

0.0001005

0.2

0.4

0.6

0.8

0.00

0.2

0.4

0.6

0.8

0.0001005

Solution (PINN)

0.40

Solution(q=1.5)

0.40

Uncertainty (q=1.5)

0.000145

Solution(q=2.0)

0.40

Uncertainty(q=2.0)

0.00066

0.8

0.35

0.30

0.8

0.35

0.30

8'0

0.000140

0.000135

0.8

0.35

0.30

0.8

0.00060

0.00054

0.4

0.6

0.25

0.20

0.15

0.4

0.6

0.25

0.20

0.15

0.10

0.4

0.6

0.000130

0.000125

0.000120

0.000115

0.6

0.4

0.20

0.25

0.15

0.10

0.6

0.4

0.00048

0.00042

0.00036

0.2

0.10

0.05

0.2

0.05

0.00

0.2

0.000110

0.000105

0.2

0.05

0.00

0.2

0.00030

0.00024

0.2

0.4

0.6

8'0

0.00

0.2

0.4

0.6

8'0

0.05

0.2

0.4

0.6

8'0

0.000100

0.2

0.4

0.6

0.8

0.05

0.2

0.4

0.6

8'0

0.00018

In most experiments, we choose the interior collocation points on a 24 × 24 mesh grid and the corresponding 100 boundary collocation points unless otherwise stated. For the sparse variational inference, M = 256 inducing points are randomly initialized and learned by optimizing the ELBO (9). The kernel C of Q-EP/GP is chosen to be matern52 with the hyperparameters, e.g. the correlation strength, automatically tuned in the python package GPyTorch [11] implemented based on PyTorch . PINN is configured to have neural network parameters (weights and biases) of similar size to the variational Q-EP model. The computer codes are publicly available at https://github.com/ lanzithinking/Diff_QEP .

5.1 Eikonal Equation

First, we consider the following regularized Eikonal equation on Ω = [0 , 1] 2 also considered in [6]:

where f ≡ 1 and ε = 0 . 1 . Based on the set-up in Section 3.1, we define the nonlinear function D ( u, d 1 u, d 2 u, d 2 1 u, d 2 2 u ) = ( d 1 u ) 2 + ( d 2 u ) 2 -ε ( d 2 1 u + d 2 2 u ) . Then the observations are Y =

obtain N d = 24 2 interior and N b = 100 boundary collocation points. Then we apply the variational Bayes in Section 3.2 to solve (12) with M = 256 inducing points. Though not required for convergence, we train each algorithm for 5000 iterations for fair comparison (PINN and B-PINN need much more training epochs than Q-EP solvers to converge).

Figure 2 compares solutions and uncertainty estimates generated by a variety of Q-EP solvers for q = 0 . 8 , 1 . 0 , 1 . 5 and 2 . 0 (GP) respectively. We solve the equation using a highly-resolved finite difference method with the Cole-Hopf transformation [6] and use it as the true solution for comparison (upper left). We notice that the solution by PINN (lower left) is much worse and UQ is not available. All Q-EP solvers ( q < 2 ) produce better solutions than GP ( q = 2 ) which does not precisely characterize the pyramid feature of the true solution. Only two solvers with q = 0 . 8 and 1 . 0 yield solutions that correctly match the range of true solution. GP also manifests higher uncertainty in its generated solution. Table C.1 further verifies the superior accuracy of Q-EP solvers compared to GP and PINN (B-PINN) in terms of multiple error metrics including MAE, MSE and RLE by repeating the experiments for 10 times with different random seeds. In this example, Q-EP solver with q = 1 . 0 attains the result comparable to the most accurate solution. Note that with similar size of collocation points, our best results (4.43e-2 in L 2 error and 1.03e-2 in L ∞ error) are comparable to those reported (1.64e-2 in L 2 error and 7.76e-2 in L ∞ error) in [6] from which UQ is absent.

5.2 Burgers' Equation

Next, we test our Q-EP solvers on the Burgers' equation [6] with ν = 0 . 1 :

1.0

Iteration10(q=1.0)

0.90

1.0

Iteration1o0(q=1.0)

1.00

1.0

Iteration1000(q=1.0)

1.00

Relative L Error

0.75

0.75

0.75

0.8

0.60

0.8

0.50

0.8

0.50

100

0.6

0.45

0.6

0.25

0.6

0.25

0.4

0.2

0.30

0.15

0.00

0.4

0.2

0.00

0.25

0.50

0.4

0.2

0.00

0.25

0.50

0.75

10-1

1.5

1.0

0.8

PINN

2

0.5

00

0.5

0.15

0.5

0.0

0.5

0.75

0.5

00

0.5

1.00

100

101

102

103

1.0

Iteration10(q=2.0)

0.75

1.0

Iteration100(q=2.0)

0.75

1.0

Iteration1000(q=2.0)

1.00

Relative L2 Error

0.60

0.60

0.75

0.8

0.45

0.8

0.45

0.8

0.50

100

0.6

0.30

0.6

0.30

0.6

0.25

0.4

0.2

0.5

00

0.5

0.15

0.00

0.15

0.30

0.45

0.4

0.2

0.5

0.0

0.5

0.15

0.00

0.15

0.30

0.45

0.4

0.2

0.5

0.0

0.5

0.00

0.25

0.50

1.00

0.75

10-1

100

1.5

1.0

0.8

PINN

2

101

102

103

0.8

0.6

0.4

0.2

1.0

iteration 10(PINN)

0.20

0.25

0.15

0.10

0.05

0.00

0.05

0.15

0.10

80

0.6

1.0

0.4

0.2

iteration100(PINN)

0.50

0.25

0.75

0.00

0.75

0.25

0.50

0.8

0.6

0.4

0.2

1.0

iteration 100o (PINN)

0.25

0.75

0.50

0.00

1.00

0.25

0.50

0.75

10-1

100

1.5

1.0

0.8

PINN

2

Relative LError

0.5

0.0

0.5

0.20

0.5

0.0

0.5

1.00

0.5

0.0

0.5

1.00

100

101

102

103

We use the same experiment setup as above. Figure C.1 compares the solutions by a highly-resolved finite difference method (upper left, treated as true solution for comparison purpose), PINN (lower left), Q-EP (right three in upper row: q = 1 . 0 ), and GP (right three in lower row: q = 2 . 0 ). Q-EP is about one order of magnitude more accurate than PINN and GP, which can be verified from the pointwise error plots and Table 1. The plots of posterior standard deviation on the right column also indicate meaningful uncertainty in the middle area around x = 0 where the shock is difficult to resolve. Q-EP still achieves one order of magnitude lower uncertainty compared with GP.

Table 1 compares the accuracy of solutions in terms of MAE, MSE and RLE, for which Q-EP with q = 1 . 0 attains the most accurate one, verifying its best performance. The error for Q-EP models does not decrease monotonically with q > 0 , but reaches the lowest at q = 1 . 0 . To further investigate the convergence, we plot solutions by Q-EP (top row: q = 1 . 0 ), GP (middle row: q = 2 . 0 ), and PINN (bottom row) in Figure 3 at 10 (first column), 100 (second column), and 1000 (third column) iterations respectively. We can tell that Q-EP with q = 1 . 0 converges the fastest to the best estimate, which is confirmed by the error-reducing plots on the right column. Note that in this example, our algorithm does not need random perturbation of mesh grid as required by [6].

TrueInverseSolution

2.2

InverseSolution (PiNN)

5.50

PDE Solution (q=1.0)

0.032

InverseSolution (q=1.0)

2.8

Uncertainty of Inverse Solution (q=1.0)

0.008

0.8

0.4

0.6

0.2

2.0

1.8

1.4

1.2

1.0

0.8

1.6

0.8

0.6

0.4

0.2

5.55

5.60

5.65

5.70

5.75

5.80

5.85

0.8

0.6

0.4

0.2

?

其

0.028

0.004

0.024

0.020

0.016

0.012

0.008

0.000

0.8

0.6

0.4

0.2

2.4

2.0

1.6

1.2

8'0

0.4

0.8

0.6

0.4

0.2

0.005

0.007

0.002

0.006

0.004

0.003

0.2

0.4

0.6

0.8

0.6

0.2

0.4

9'0

8'0

5.90

0.2

0.4

0.6

0.8

0.004

0.2

0.4

0.6

0.8

0.0

0.2

0.4

0.6

0.8

0.001

1.0

FEMSolution

0.040

PDESolution (PINN)

0.028

0.09

Inverse Solution (q=2.0)

0

Uncertainty of Inverse Solution (q=2.0)

0.0105

80

0.035

0.030

80

0.024

0.020

0.8

X

0.06

0.03

0.8

2

1

0.8

0.0090

0.0075

0.6

0.025

0.6

0.016

0.6

0.00

0.6

3

0.6

0.0060

0.4

0.020

0.015

0.4

0.012

0.4

高

0.03

-0.06

0.4

4

0.4

0.0045

0.2

0.010

0.005

0.2

0.008

0.004

0.2

X

0.09

0.12

0.2

5

6

0.2

0.0030

0.0015

00

0000

0.000

0.15

-7

0.0000

0.0

0.2

0.4

0.6

8'0

1.0

0.2

0.4

9°0

0.8

0.2

0.4

0.6

0.8

0.2

0.4

0.6

0.8

0.2

0.4

90

0.8

5.3 Inverse Darcy Flow

Now we consider an inverse problem that involves the following Darcy flow:

where the true coefficient such that exp( a † ( x )) = exp(sin(2 πx 1 )+sin(2 πx 2 ))+exp( -sin(2 πx 1 ) -sin(2 πx 2 )) [6] is plotted in the upper left panel of Figure 4. We generate data by solving (14) with a † on a mesh 80 × 80 to obtain u † using the finite element method, illustrated in the lower left panel in Figure 4. On a (coarser) mesh N d = 20 × 20 used for inference, we randomly select N o = 100 points X o in Ω and obtain observations as u † ( X o ) + ε with noise ε ∼ N (0 , γ 2 I N o ) for γ = 10 -3 .

We solve the inverse problem of finding a in (14) given these observations. With N d = 400 interior and N b = 84 boundary collocation points and M = 256 inducing points, we train Q-EP solvers for 2000 iterations. Figure 4 illustrates the forward PDE solution u (third column), the inverse solution a (forth column), and the uncertainty of a (rightmost column). Compared with the true a † , Q-EP ( q = 1 . 0 ) recovers a more faithfully than GP ( q = 2 . 0 ) and PINN (upper in the second column). Meanwhile, Q-EP also generates the solution u much closer to that by the finite element method, as shown in the lower left panel. Higher uncertainty is observed by Q-EP ( q = 1 . 0 ) around the corners with less data (Figure C.3), reflecting the configuration of observations. If run for longer (e.g. 5000) iterations, PINN may improve its forward solution, but still gets a poor inverse solution (Figure C.4).

Table C.5 compares the relative errors (RLEs) of forward and inverse solutions. Q-EP with q = 1 . 0 also achieves the best or comparable solutions. Here we emphasize that all the results are based on only N o = 100 observations from one solution, as opposed to the thousands of PDE solutions typically used in operator learning algorithms. Figure C.5 compares the inverse solutions of Q-EP with those of GP in increasingly finer meshes. If we view training in finer mesh as a process to see more data, Q-EP ( q = 1 . 0 ) has already converged fast to better estimates in coarser mesh, leaving less room to improve compared to GP ( q = 2 . 0 ).

6 Conclusion

In this paper, we propose variational Q-EP to solve and learn PDEs. We advocate Q-EP with q = 1 . 0 over GP ( q = 2 . 0 ) for modeling derivative information and hence a better probabilistic solver for PDEs. The fastest convergence at q = 1 . 0 is theoretically justified and empirically verified using two nonlinear forward PDE problems and an inverse Darcy flow problem.

One of the limitations might be the variational inference adopted in this paper. The highly nonlinear nature of some PDEs imposes challenges on the quality of variational approximation to the resulting posterior, which in turn may undermine both the solution accuracy and the associated UQ. A possible remedy could be more flexible inference methods, such as normalizing flow [41, 36].

References

Sergios Agapiou, Masoumeh Dashti, and Tapio Helin. Rates of contraction of posterior distributions based on p-exponential priors. Bernoulli , 27(3):1616 - 1642, 2021.

Frank Aurzada. On the lower tail probabilities of some random sequences in lp. Journal of Theoretical Probability , 20(4):843-858, Dec 2007.

Ricardo Baptista, Edoardo Calvello, Matthieu Darcy, Houman Owhadi, Andrew M. Stuart, and Xianjin Yang. Solving roughly forced nonlinear pdes via misspecified kernel methods and neural networks. arXiv preprint arXiv:2501.17110 , 01 2025.

Ben Calderhead, Mark Girolami, and Neil Lawrence. Accelerating bayesian inference over nonlinear differential equations with gaussian processes. In D. Koller, D. Schuurmans, Y . Bengio, and L. Bottou, editors, Advances in Neural Information Processing Systems , volume 21. Curran Associates, Inc., 2008.

Zhi Chang, Chukwudi Obite, Shuang Zhou, and Shiwei Lan. Deep q-exponential processes. In Proceedings of the 7th Symposium on Advances in Approximate Bayesian Inference . AABI, 10 2025.

Yifan Chen, Bamdad Hosseini, Houman Owhadi, and Andrew M. Stuart. Solving and learning nonlinear pdes with gaussian processes. Journal of Computational Physics , 447:110668, 2021.

Masoumeh Dashti, Stephen Harris, and Andrew Stuart. Besov priors for bayesian inverse problems. Inverse Problems and Imaging , 6(2):183-200, may 2012.

Masoumeh Dashti and Andrew M. Stuart. The Bayesian Approach to Inverse Problems. In Roger Ghanem, David Higdon, and Houman Owhadi, editors, Handbook of Uncertainty Quantification , pages 311-428. Springer International Publishing, Cham, 2017.

Weinan E and Bing Yu. The deep ritz method: A deep learning-based numerical algorithm for solving variational problems. Communications in Mathematics and Statistics , 6(1):1-12, Mar 2018.

K. Fang and Y.T. Zhang. Generalized Multivariate Analysis . Science Press, 1990.

Jacob Gardner, Geoff Pleiss, Kilian Q Weinberger, David Bindel, and Andrew G Wilson. Gpytorch: Blackbox matrix-matrix gaussian process inference with gpu acceleration. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 31. Curran Associates, Inc., 2018.

S. Ghosal and A.W. van der Vaart. Convergence rates of posterior distributions for non-i.i.d. observations. Annals of Statistics , 35(1):192-223, 2007. MR2332274.

Oliver Hamelijnck, Arno Solin, and Theodoros Damoulas. Physics-informed variational statespace gaussian processes. In The Thirty-eighth Annual Conference on Neural Information Processing Systems , 2024.

Marc Harkonen, Markus Lange-Hegermann, and Bogdan Raita. Gaussian process priors for systems of linear partial differential equations with constant coefficients. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning , volume 202 of Proceedings of Machine Learning Research , pages 12587-12615. PMLR, 23-29 Jul 2023.

Markus Heinonen, Cagatay Yildiz, Henrik Mannerström, Jukka Intosalmi, and Harri Lähdesmäki. Learning unknown ODE models with Gaussian processes. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning , volume 80 of Proceedings of Machine Learning Research , pages 1959-1968. PMLR, 10-15 Jul 2018.

Philipp Hennig, Michael A. Osborne, and Mark Girolami. Probabilistic numerics and uncertainty in computations. Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences , 471(2179):20150142, July 2015.

James Hensman, Alexander Matthews, and Zoubin Ghahramani. Scalable Variational Gaussian Process Classification. In Guy Lebanon and S. V. N. Vishwanathan, editors, Proceedings of the Eighteenth International Conference on Artificial Intelligence and Statistics , volume 38 of Proceedings of Machine Learning Research , pages 351-360, San Diego, California, USA, 09-12 May 2015. PMLR.

Mark E. Johnson. Multivariate Statistical Simulation , chapter 6 Elliptically Contoured Distributions, pages 106-124. Probability and Statistics. John Wiley & Sons, Ltd, 1987.

George Em Karniadakis, Ioannis G. Kevrekidis, Lu Lu, Paris Perdikaris, Sifan Wang, and Liu Yang. Physics-informed machine learning. Nature Reviews Physics , 3(6):422-440, May 2021.

Nikola Kovachki, Zongyi Li, Burigede Liu, Kamyar Azizzadenesheli, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Neural operator: Learning maps between function spaces with applications to pdes. Journal of Machine Learning Research , 24(89):1-97, 2023.

Shiwei Lan, Mirjeta Pasha, Shuyi Li, and Weining Shen. Spatiotemporal besov priors for bayesian inverse problems. arXiv preprint arXiv:2306.16378 , 06 2023.

Matti Lassas, Eero Saksman, and Samuli Siltanen. Discretization-invariant bayesian inversion and besov space priors. Inverse Problems and Imaging , 3(1):87-122, 2009.

Matti Lassas and Samuli Siltanen. Can one use total variation prior for edge-preserving Bayesian inversion? Inverse Problems , 20(5):1537, 2004.

Shuyi Li, Michael O' Connor, and Shiwei Lan. Bayesian learning via q-exponential process. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Proceedings of the 37th Conference on Neural Information Processing Systems , volume 36, pages 72867-72887. Curran Associates, Inc., 2023.

Zongyi Li, Nikola Borislavov Kovachki, Kamyar Azizzadenesheli, Burigede liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Fourier neural operator for parametric partial differential equations. In International Conference on Learning Representations , 2021.

Lu Lu, Pengzhan Jin, Guofei Pang, Zhongqiang Zhang, and George Em Karniadakis. Learning nonlinear operators via deeponet based on the universal approximation theorem of operators. Nature Machine Intelligence , 3(3):218-229, March 2021.

Rui Meng and Xianjin Yang. Sparse gaussian processes for solving nonlinear pdes. Journal of Computational Physics , 490:112340, 2023.

Roberto Molinaro, Yunan Yang, Björn Engquist, and Siddhartha Mishra. Neural inverse operators for solving PDE inverse problems. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning , volume 202 of Proceedings of Machine Learning Research , pages 25105-25139. PMLR, 23-29 Jul 2023.

Carlos Mora, Amin Yousefpour, Shirin Hosseinmardi, and Ramin Bostanabad. A gaussian process framework for solving forward and inverse problems involving nonlinear partial differential equations. Computational Mechanics , 75(4):1213-1239, 2025.

Chukwudi Paul Obite, Zhi Chang, Keyan Wu, and Shiwei Lan. Bayesian regularization of latent representation. In The Thirteenth International Conference on Learning Representations , 2025.

Bernt Øksendal. Stochastic Differential Equations . Springer Berlin Heidelberg, 2003.

Houman Owhadi. Bayesian numerical homogenization. Multiscale Modeling and Simulation , 13(3):812-828, January 2015.

Houman Owhadi. Multigrid with rough coefficients and multiresolution operator decomposition from hierarchical information games. SIAM Review , 59(1):99-149, January 2017.

Houman Owhadi and Clint Scovel. Operator-Adapted Wavelets, Fast Solvers, and Numerical Homogenization: From a Game Theoretic Approach to Numerical Approximation and Algorithm Design . Cambridge Monographs on Applied and Computational Mathematics. Cambridge University Press, 2019.

Houman Owhadi and Lei Zhang. Gamblets for opening the complexity-bottleneck of implicit schemes for hyperbolic and parabolic odes/pdes with rough coefficients. Journal of Computational Physics , 347:99-128, 2017.

George Papamakarios, Eric Nalisnick, Danilo Jimenez Rezende, Shakir Mohamed, and Balaji Lakshminarayanan. Normalizing flows for probabilistic modeling and inference. Journal of Machine Learning Research , 22(57):1-64, 2021.

M. Raissi, P. Perdikaris, and G.E. Karniadakis. Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational Physics , 378:686-707, 2019.

Maziar Raissi, Paris Perdikaris, and George Em Karniadakis. Numerical gaussian processes for time-dependent and nonlinear partial differential equations. SIAM Journal on Scientific Computing , 40(1):A172-A198, January 2018.

J. O. Ramsay, G. Hooker, D. Campbell, and J. Cao. Parameter estimation for differential equations: a generalized smoothing approach. Journal of the Royal Statistical Society: Series B (Statistical Methodology) , 69(5):741-796, 2007.

Carl Edward Rasmussen and Christopher K. I. Williams. Gaussian Processes for Machine Learning . The MIT Press, 2005.

Danilo Rezende and Shakir Mohamed. Variational inference with normalizing flows. In Francis Bach and David Blei, editors, Proceedings of the 32nd International Conference on Machine Learning , volume 37 of Proceedings of Machine Learning Research , pages 1530-1538, Lille, France, 07-09 Jul 2015. PMLR.

Leonid I. Rudin, Stanley Osher, and Emad Fatemi. Nonlinear total variation based noise removal algorithms. Physica D: Nonlinear Phenomena , 60(1):259-268, 1992.

Michael Schober, David Duvenaud, and Philipp Hennig. Probabilistic ode solvers with rungekutta means. In Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems , volume 27. Curran Associates, Inc., 2014.

Justin Sirignano and Konstantinos Spiliopoulos. Dgm: A deep learning algorithm for solving partial differential equations. Journal of Computational Physics , 375:1339-1364, 2018.

John Skilling. Bayesian solution of ordinary differential equations. In C. Ray Smith, Gary J. Erickson, and Paul O. Neudorfer, editors, Maximum Entropy and Bayesian Methods: Seattle, 1991 , pages 23-37. Springer Netherlands, Dordrecht, 1992.

Michalis Titsias. Variational learning of inducing variables in sparse gaussian processes. In David van Dyk and Max Welling, editors, Proceedings of the Twelth International Conference on Artificial Intelligence and Statistics , volume 5 of Proceedings of Machine Learning Research , pages 567-574, Hilton Clearwater Beach Resort, Clearwater Beach, Florida USA, 16-18 Apr 2009. PMLR.

Michalis Titsias and Neil D. Lawrence. Bayesian gaussian process latent variable model. In Yee Whye Teh and Mike Titterington, editors, Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics , volume 9 of Proceedings of Machine Learning Research , pages 844-851, Chia Laguna Resort, Sardinia, Italy, 13-15 May 2010. PMLR.

C.R. Vogel and M.E. Oman. Fast, robust total variation-based reconstruction of noisy, blurred images. IEEE Transactions on Image Processing , 7(6):813-824, 1998.

Liu Yang, Xuhui Meng, and George Em Karniadakis. B-pinns: Bayesian physics-informed neural networks for forward and inverse pde problems with noisy data. Journal of Computational Physics , 425:109913, 2021.

Technical Appendices for 'Solving and Learning Partial Differential Equations with Variational Q-Exponential Process"

A Calculations

A.1 Variational Lower Bound

Because log p ( Y ) = KL( q ( ˜ U ) ∥ p ( ˜ U | Y )) + L ( q ( ˜ U )) , it reduces to maximizing the lower bound L ( q ( ˜ U )) . The sparse variational approximation [46, 47] is adopted by introducing inducing points ˜ X ∈ R M × d with their function values ˜ V = ˜ u ( ˜ X ) ∈ R M × D . Hence the joint distribution of Y and ˜ U can be augmented by including ˜ V :

where we have vec( ˜ V ) | ˜ X ∼ q -ED MD ( 0 , ˜ C MM ) and the conditional distribution

Now we approximate the joint posterior p ( ˜ U , ˜ V | Y ) with the following variational distribution

where the covariance Σ is of size MD × MD and can be chosen as a (block)-diagonal matrix for convenience. A standard variational Bayes procedure yields the variational bound:

where the marginal variational distribution q ( ˜ U ) can be obtained as [17, 30]

Denote by φ ( r ; C , N ) := -1 2 log | C | + N 2 ( q 2 -1 ) log r -1 2 r q 2 which is convex for q ∈ (0 , 2] . Let φ 0 ( r ) = φ ( r ; Γ , N ) and r ( Y ) = ( Y -h ) T Γ -1 ( Y -h ) be a quadratic form of random variable Y . Then log p ( Y | ˜ U ) = φ 0 ( r ( Y )) . Therefore, by Jensen's inequality, we can bound from below as

Now we compute the K-L divergence KL ˜ V := KL( q ( ˜ V ) ∥ p ( ˜ V )) :

Denote by r ( ˜ V ) = vec( ˜ V -µ ) T Σ -1 vec( ˜ V -µ ) . Then log q ( ˜ V ) = φ ( r ( ˜ V ); Σ , MD ) . From [Proposition A.1. of 24] we know that r q 2 ∼ χ 2 ( MD ) . Therefore

Let φ 1 ( r ) := φ ( r ; ˜ C MM , MD ) . Then by Jensen's inequality

Therefore, the final evidence lower bound (ELBO) L ∗ ( q ( ˜ U )) is

B Proofs

Notations : ≲ means 'less than or approximately equal to"; a n ≲ b n implies a n ≤ Cb n for some constant C > 0 . ≍ means 'asymptotically equal to"; a n ≍ b n implies lim n →∞ a n b n = c for some constant c .

Proof of Proposition 2.1. Note r (˜ u ℓ ) q 2 = λ -q 2 ℓ | ˜ u ℓ | q ∼ χ 2 (1) for all ℓ ∈ N by Proposition A.1. of [24]. Denote χ 2 ℓ := λ -q 2 ℓ | ˜ u ℓ | q iid ∼ χ 2 (1) . Hence ∥ ˜ u ∥ q q = ∑ ∞ ℓ =1 λ q 2 ℓ χ 2 ℓ becomes an infinite mixture of chi-squared random variables whose density is analytically intractable. Yet we have

if Assumption 1 holds. Thus it completes the proof.

Proof of Proposition 3.1. Based on (5) and (6), the potential Φ(˜ u ; Y ) = -φ ( r ; Γ , N ) where φ ( r ; Γ , N ) := -1 2 log | Γ | + N 2 ( q 2 -1 ) log r -1 2 r q 2 is convex in r if q ∈ (0 , 2] , and r = r (˜ u ) with

Since convex function is Lipschitz continuous over compact domain, it suffices to prove that r (˜ u ) is bounded (both bounds achievable) and Lipschitz.

Note that r (˜ u ) ≤ δ -1 ∥ A ˜ u -b ∥ 2 2 ≤ δ -1 ( ∥ A ∥∥ ˜ u ∥ + ∥ b ∥ ) 2 where ˜ u represents a PDE solution in the compact domain Ω ⊂ R d and is therefore bounded by some M > 0 . By Assumption 2, ∥ A ∥ ≤ C . Therefore, 0 ≤ r (˜ u ) ≤ δ -1 ( CM + ∥ b ∥ ) 2 .

On the other hand, we have the gradient of the quadratic form r (˜ u ) bounded as

Lastly, there exists L 1 , L 2 > 0 such that

assuming the collocation points are uniformly sampled. Therefore, it completes the proof.

According to Theorem 3.1 and Lemma 5.14 of [1], the following general contraction conditions hold for Q-EP and will be used in the proof of posterior contraction Theorem 4.1.

Theorem B.1. Let µ be a q -EP (0 , C ) measure satisfying Assumption 1 in the separable Banach space ( L q (Ω) , ∥ · ∥ q ) , where q ∈ [1 , 2] . Let u ∼ µ and the true parameter u † ∈ L q (Ω) . Assume ε n > 0 such that φ u † ( ε n ) ≤ nε 2 n , where nε 2 n ≳ 1 . Then for any C > 1 , there exists a measurable set B n ⊂ L q (Ω) and a constant R > 0 depending on C and q , such that

where N (4 ε n , B n , ∥ · ∥ q ) is the minimal number of ∥ · ∥ q -balls of radius 4 ε n to cover B n .

We need the following lemma to bound the Hellinger distance, Kullback-Leibler (K-L) divergence, and K-L variation to complete the proof of Theorem 4.1 [21].

Lemma B.1. Suppose the potential function Φ (6) satisfies Lipschitz continuity in u as in Proposition 3.1. Then we have

d H ( p u , p u ′ ) ≲ ∥ u -u ′ ∥ q .

K ( p u , p u ′ ) ≲ ∥ u -u ′ ∥ q .

V ( p u , p u ′ ) ≲ ∥ u -u ′ ∥ 2 q .

Proof. First, we consider K-L divergence:

by Proposition 3.1. Similarly, we have for K-L variation:

Lastly, we bound the Hellinger distance:

where the inequality holds for ∥ u -u ′ ∥ 2 q small enough.

Proof of Theorem 4.1. Based on [Theorem 1 of 12], it suffices to verify the following two conditions (the entropy condition (2.4), and the prior mass condition (2.5)) for some universal constants η, K > 0 and sufficiently large k ∈ N ,

where the left side of (18) is logarithm of the minimal number of d n,H -balls of radius ξε/ 2 needed to cover a ball of radius ε around the true value u † ; B n ( u † , ε n ) = { u ∈ Θ : 1 n K ( u † , u ) ≤ ε 2 , 1 n V ( u † , u ) ≤ ε 2 } with K ( u † , u ) = K ( p u † , p u ) and V ( u † , u ) = V ( p u † , p u ) .

Since u ( · ) ∈ L q (Ω) satisfy conditions for Theorem B.1, there exists B n ⊂ L q (Ω) such that (15)(17) holds. Now we set Θ n = B n . For ∀ u, u ′ ∈ Θ n such that ∥ u ( · ) -u ′ ( · ) ∥ q ≤ ε n , we have d n,H ( u, u ′ ) ≲ ∥ u -u ′ ∥ q ≤ ε n by Lemma B.1. Therefore by (15) we have the following global entropy bound holds

which is stronger than the local entropy condition (18).

Now by Lemma B.1 and (17), we have

Then the prior mass condition (19) is satisfied because the numerator is bounded by 1. The proof is hence completed.

The following lemma studies the small ball probability in the concentration function (11) [21].

Lemma B.2 (Small ball probability) . Let Π be a q -EP (0 , C ) prior on B s ′ ,q (Ω) with s ′ < s -d q . Then as ε → 0 , we have

Proof. We can compute

where P is the probability measure on the infinite product space ( L q (Ω)) ∞ . From the proof of Proposition 2.1 we know ∥ u ∥ q q = ∑ ∞ ℓ =1 λ q 2 ℓ χ 2 ℓ is an infinite mixture of χ 2 (1) random variables, so the condition of [Theorem 4.2 of 2] is trivially met and we have

The second lemma gives an upper bound of the first term of the concentration function (11) [21].

Lemma B.3 (Decentering function) . Assume u † ∈ B s † ,q † (Ω) for some s † > s ′ and q † ∈ [1 , 2] . Then as ε → 0 , we have the following bounds

(i) If q † ≥ q , we require s † > s ′ :

(ii) If q † < q , we require s † > s ′ -d q + d q † :

Proof. We identify u † ∈ B s † ,q † with { u † ℓ } ∞ ℓ =1 ∈ ℓ q † ,τ q † ( s † ) . Then we follow [1] to approximate u † with h 1: L = { u † ℓ } ∞ ℓ =1 where u † ℓ ≡ 0 for all ℓ > L . Note h 1: L ∈ ℓ q,τ q ( s ) for any finite L ∈ N . Identifying h 1: L with h ∈ B s,q (Ω) , we could get

Therefore, to have ∥ h -u † ∥ s ′ ,q ≤ ε we let

On the other hand, the infimum is less than ∥ h ∥ q s,q with above h , which can be bounded as follows. If q † = q ,

If q † > q , by similar argument using Hölder inequality,

If q † < q , by similar argument,

Substituting L in (20) to the above equations yields the conclusion.

Proof of Theorem 4.2. By Lemmas B.2 and B.3, we have the following bounds for the concentration function (11) as ε → 0 , if q † ≥ q ,

For s ≤ s † , the bound is dominated by ε -1 s -s ′ d -1 q . For the last case, we need to determine a balancing point of s for the two terms by setting their powers equal. The calculation shows that if s ≤ s † + d q , the bound is still dominated by ε -1 s -s ′ d -1 q , but otherwise is dominated by ε -s -s † s † -s ′ q . Therefore, we have

We need to determine minimal ε n such that φ u † ( ε n ) ≤ nε 2 n . Hence for q † ≥ q ,

Now if q † < q , by similar argument we have the concentration function (11) as ε → 0

Thus the contraction rate for q † < q becomes

Rewriting the equations into one yields the conclusion.

C More Numerical Results

C.1 Eikonal Equation

To verify the systematic superiority of Q-EP with q = 1 . 0 than GP ( q = 2 . 0 ), we extend our comparison to more kernels including the radius basis function (rbf), C ( x , x ′ ) = σ 2 exp( -0 . 5 r 2 ) , and rational quadratic (rq), C ( x , x ′ ) = σ 2 (1 + 1 2 α r 2 ) -α , α > 0 , for r = √ ∑ d i =1 ( x i -x ′ i ) 2 /ρ 2 i in Table C.2. Within each type of kernel, Q-EP with q = 1 . 0 consistently outperforms GP ( q = 2 . 0 ). This indicates that the advantage of Q-EP over GP is independent of the choice of kernels (as long as they are compared with the same kernel).

C.2 Burgers' Equation

Define the nonlinear function D ( u, d 1 u, d 2 u, d 2 1 u ) = d 2 u + ud 1 u -νd 2 1 u . The observations Y can then be expressed as P (˜ u ( X )) = [ D ( u ( X d ) , ∂ ∂x u ( X d ) , ∂ ∂t u ( X d ) , ∂ 2 ∂x 2 u ( X d )) u ( X b ) ] , and h = [ 0 N d g ] , where g is a vector of size N b whose elements are -sin( πx ) or 0 depending on the order of the corresponding elements in X b . We use the same experiment setup as above.

In Table C.3, we also observe a consistently better performance of Q-EP ( q = 1 . 0 ) compared to GP ( q = 2 . 0 ) in solving Burgers' equation (13) using various kernels.

One can possibly find a GP with fine-tuned kernel, e.g. rational quadratic (rq), to have better (Q-EP with rbf in Table C.2) or matching (Q-EP with Matern52 in Table C.3) results. However, as a

1.0

FDSolution

1.00

1.0

Solution(q=1.0)

1.00

1.0

Error (q=1.0)

0.032

1.0

Uncertainty(q=1.0)

0.000210

0.75

0.75

0.024

0.000195

0.8

0.6

0.50

0.25

0.00

0.8

0.6

0.50

0.25

0.00

0.8

0.6

0.000

0.016

0.008

0.6

0.8

0.000165

0.000180

0.4

0.25

0.50

0.4

0.25

0.50

0.4

-0.016

-0.008

0.024

0.4

0.000150

0.000135

0.2

0.75

0.2

0.75

0.2

0.032

0.2

0.000120

0.5

00

0.5

-1.00

0.5

0.0

0.5

1.00

0.5

0.0

0.5

0.040

0.5

00

0.5

0.000105

1.0

Solution(PINN)

1.00

1.0

Solution(q=2.0)

1.00

1.0

Error(q=2.0)

0.100

1.0

Uncertainty(q=2.0)

0.0036

0.75

0.75

0.075

0.0032

0.8

0.50

0.8

0.50

0.8-

0.050

0.8

0.0028

0.6

0.25

0.00

0.6

X

0.25

0.00

0.6

0.025

0.000

0.6

0.0020

0.0024

0.0016

0.4

0.25

0.4

0.25

0.4

0.025 0.4

0.0012

0.50

0.50

0.050

0.0008

0.2

0.75

0.2

0.75

0.2

0.075

0.2

0.0004

0.5

0.0

0.5

1.00

0.5

0.0

0.5

1.00

0.5

00

0.5

-0.100

0.5

0.0

0.5

0.0000

principled regularization over function spaces, Q-EP facilitates effective modeling of derivatives and differential equations, thereby alleviating the struggle of GP on meticulous kernel engineering.

C.3 Nonlinear Elliptic Equation

Now we consider a nonlinear elliptic equation with Dirichlet boundary condition on Ω = [0 , 1] 2 :

where we choose τ ( u ) = u 3 , g ( x ) ≡ 0 , and f such that the following true solution u † satisfies equation (21) [6]

We follow the variational procedure in Section 3.1 and use the same experiment setup in Section 5. Figure C.2 compares the solutions by a highly-resolved finite difference method (upper left, treated as true solution), PINN (lower left), Q-EP (right three in upper row: q = 1 . 0 ), and GP (right three in lower row: q = 2 . 0 ). Compared to the true solution, Q-EP with q = 1 . 0 yields an estimate more accurate than PINN and GP. Based on Table C.4, the best solution is obtained by Q-EP with q = 0 . 5 . Since the true solution is highly fluctuating over the domain Ω , imposing more challenges on the boundary, where the largest pointwise errors occur. Adding more boundary points might help with Q-EP solvers.

C.4 Inverse Darcy Flow

Finally, we consider a more realistic Darcy flow data used by Fourier Neural Operator (FNO) and Physics-Informed Neural Operator (PINO) available in NVIDIA PhysicsNeMo. This dataset contains thousands of permeability-solution pairs that reflect realistic porous media. Since our method is

TrueSolution

6.0

Solution(q=1.0)

6.0

Error (q=1.0)

2.4

Uncertainty (q=1.0)

0.0027

0.6

0.8

4.5

1.5

3.0

0.6

0.8

4.5

1.5

3.0

0.8-

0.6

0.8

1.6

2.0

1.2

0.6

0.8

0.0015

0.0018

0.0024

0.0021

0.4

0.2

0.0

3.0

1.5

0.4

0.2

0.0

3.0

1.5

0.4

0.2

0.4

0.0

0.4

0.4

0.2

0.0009

0.0006

0.0003

0.0012

0.2

0.4

0.6

0.8

4.5

0.2

0.4

0.6

0.8

4.5

0.2

0.4

0.6

0.8

-0.8

0.2

0.4

0.6

0.8

0.0000

Solution(PINN)

Solution(q=2.0)

Error(q=2.0)

Uncertainty(q=2.0)

10.0

8

8

0.0016

0.8

7.5

0.8

6

0.8

6

0.8

0.0014

5.0

4

4

0.0012

0.6

2.5

0.6

2

0.6

2

0.6

0.0010

0.0

0

0.0008

0.4

2.5

0.4

2

0.4

0

0.4

0.0006

5.0

4

2

0.0004

0.2

7.5

0.2

-6

0.2

4

0.2

0.0002

0.2

0.4

0.6

0.8

-10.0

0.2

0.4

0.6

0.8

8-

0.2

0.4

0.6

0.8

-6

0.2

0.4

0.6

0.8

0.0000

not to train a surrogate model, we take only one pair and impose the data on 200 × 200 mesh to obtain 2000 randomly sampled observations. We then train Q-EP solvers on 60 × 60 mesh (with collocation points taken from the grid) and with 512 inducing points. This problem has much larger scale ( ∼ 20 times larger) than all the previous examples. As illustrated in Table C.6, Q-EP with q = 1 still achieves remarkable advantage compared with GP ( q = 2 ) and PINN. Note, Q-EP with q = 1 has comparable accuracy with q = 0 . 5 in forward solution. With only one training pair in the NIVIDA FNO-Darcy data, it is understandably more challenging to obtain the inverse solution, yet for which Q-EP with q = 1 attains the best accuracy. In both solutions, GP ( q = 2 ) is much worse by most metrics.

TrueInverseSolution

2.2

PDESolution(q=1.0)

0.036

InverseSolution(q=1.o)

2.4

Uncertainty of Inverse Solution (q=1.0)

0.018

0.8

2.0

1.8

0.8

0.024

0.030

0.8

2.0

1.6

0.8

0.016

0.014

0.6

0.4

0.2

1.6

1.4

1.2

0.8

1.0

0.4

0.6

0.2

XX

X

0.018

0.012

0.006

0.000

0.006

0.012

0.6

0.4

0.2

0.8

0.4

0.0

1.2

0.6

0.4

0.2

0.012

0.010

0.008

0.006

0.004

0.2

0.4

0.6

8'0

0.6

0.2

0.4

0.6

0.8

-0.018

0.2

0.4

0.6

0.8

0.4

0.2

0.4

0.6

0.8

0.002

1.0

FEMSolution

0.040

PDESolution(q=2.0)

0.2

Inverse Solution (q=2.0)

0.5

Uncertainty of Inverse Solution (q=2.0)

0.0032

0.8

0.6

0.4

0.2

0.0

0.0

0.2

0.4

0.6

0.8

1.0

0.035

0.030

0.005

0.025

0.020

0.015

0.010

0.000

0.4

0.8

0.6

0.2

XXXX

XXXX

真

其

其

XXXXXX

其

XX

XX

H

0.4

XX

XX

X

?

?

X

X

Xx

XX

X

X

0.6

xx

XX

xx

其

XX

X

XXXXX

xX

其

XXXX

XXxx

0.8

XX

XX

0.1

0.0

0.2

0.5

0.1

0.3

0.4

0.6

0.2-

0.8

0.4

0.6

0.4

0.6

0.8

0.0

-0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

0.2

0.8

0.6

0.4

0.4

0.6

0.8

0.0028

0.0020

0.0012

0.0008

0.0004

0.0000

0.0024

0.0016

0.2

0.2

0.2

PDE Solution (PINN)

0.040

Inverse Solution (PINN)

4.5

Relative LError

Relative L2Error

Relative LError

0.8

0.035

0.030

0.8

5.0

5.5

0.6

0.025

0.6

6.0

6.5

0.4

0.020

0.015

0.4

7.0

7.5

100

100

100

0.2

0.010

0.005

0.2

8.0

8.5

PDE solution

Inverse solution

Inverse solution

PDE solution

PDE solution

Inverse solution

0.2

0.4

0.6

0.8

0.000

0.2

0.4

0.6

0.8

9.0

100

101

102

103

100

101

102

103

100

101

102

103

0.6

0.4

0.2

0.8

Inverse Solution (q=1.0) 16x16

0.2

0.4

0.6

0.8

2.0

1.6

0.8

0.4

0.0

2.4

1.2

-0.4

8°0

0.6

0.4

0.2

Inverse Solution (q=1.0) 20x20

0.2

0.4

0.6

0.8

80

0.4

0.0

2.4

2.0

1.6

1.2

2.8

0.4

0.6

0.4

0.2

0.8

Inverse Solution (q=1.0) 24x24

0.2

0.4

0.6

0.8

2.8

2.4

2.0

1.6

1.2

0.8

0.4

0.0

0.6

0.8

0.4

0.2

Inverse Solution (q=1.0) 29x29

0.2

0.4

0.6

0.8

2.4

2.1

1.8

0.9

0.6

0.3

1.5

1.2

0.0

2.7

10°

16×16

工

玉

20×20

Relative LError

24x24

壬士十士

205

1.0

0.8

29×29

0.8

Inverse Solution (q=2.0)16x16

0

1

0.8

Inverse Solution (q=2.0)20x20

1.8

1.2

0.6

0.8

Inverse Solution (q=2.0) 24x24

1.8

2.4

3.0

0.8

Inverse Solution (q=2.0) 29x29

3.0

2.5

2.0

Relative L Error

0.4

0.6

E-

4

5

2

0.4

0.6

0.0

0.6

1.2

0.6

0.4

1.2

0.6

0.0

0.6

0.4

1.5

1.0

0.5

10°

0.2

0.2

0.4

0.6

0.8

6

7

-8

0.2

0.2

0.4

0.6

8'0

1.8

2.4

3.0

0.2

0.2

0.4

0.6

0.8

0.6

1.2

1.8

0.2

0.2

0.4

0.6

0.8

0.0

0.5

1.0

16x16

正

20×20

24x24

29×29

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Refer to the last section Conclusion where the limitations and future directions are discussed.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory assumptions and proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: Please refer to Assumption s and the theorem statements for the assumptions and the Appendix for the full proof.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental result reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: All the details of the experiments are disclosed in the section Numerical Experiments as well as the Technical Appendices . Sample codes are included in the supplementary materials and all the codes will be released upon acceptance.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: Sample codes are included in the supplementary materials and all the codes will be released upon acceptance. Data are simulated and details on how to generate them are included in the section Numerical Experiments and the Technical Appendices .

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines () for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines () for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental setting/details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: All the details of the experiments are disclosed in the section Numerical Experiments and the Technical Appendices .

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment statistical significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: Please refer to relevant figures for error bars.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments compute resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Details of experiments are disclosed at the beginning of the section Numerical Experiments . One can also refer to relevant tables for the information.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code of ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics ?

Answer: [Yes]

Justification: The research conducted in the paper conforms, in every respect, with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: The paper discusses the scientific contributions at the end of the section Introduction .

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The authors cite the original paper that produced the code package or dataset and respect the license and terms of use.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and research with human subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional review board (IRB) approvals or equivalent for research with human subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

16. Declaration of LLM usage

Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.

Answer: [NA]

Justification: The core method development in this research does not involve LLMs as any important, original, or non-standard components.

Guidelines:

The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.

Please refer to our LLM policy () for what should or should not be described.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes q exponential distributions as an alternative to GPs for solving PDEs with uncertainty quantification. This can be seen as a generalization (q=2 is Gaussian). Specifically for derivative information (which is very relevant for PDEs), the method is claimed as more accurate. The method is well presented within existing literature, and modelling PDEs with uncertainty is a relevant problem for the community. The method is validated on some simple toy problems.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- Extensive exposition on the methods.\n- Learning stochastic PDEs from data is a relevant problem, and not trivial.\n- The authors focus on more accurate gradients, and show improvements, both theoretically and empirically.\n\nWeaknesses:\n- The method is validated on a limited number of toy problems.\n- Writing quality is a bit on the low side. This makes the paper a bit harder to read. Some concrete (friendly) suggestions:\nline 99: The negative log density of q-ED (eq. 1) yields ..\nline 102: Li et al. [24] prove ..\nline 126: Li et al. [24, Theorem 3.5] show that ..\nline 131: remove the sentence \"We often receive ..\"\n\nQUESTIONS:\nQ1: The q-exponential distribution seems similar to the generalized Gaussian distribution $\\propto \\exp(-\\frac{1}{2}(|x-\\mu|/\\alpha)^\\beta)$, can you please comment on the differences in the context of your paper?\n\nQ2: What is matern52? (line 136)\n\nQ3: Apart from replacing the Gaussian with the q-exponential, in what ways does your work differ from Hamelijnck, Solin et al.?\n\nMinor questions and remarks:\n- line 56: preferable\n- line 61: An emerging challenge\n- lines 85-88. Very long sentence that is not totally understandable\n- Fig. 1: labels are too small. Add labels indicating the rows (q=1, q=2, ..). Same for Fig. 2\n- (very minor remark) take a look at your references: **B**esov, **G**aussian, \"amp;\", ..\n- line 508: satisfies\n\nLIMITATIONS:\nIn the checklist the authors refer to the scientific contributions at the end of the introduction. I could not find a discussion on societal impacts, however, this should be trivial for this kind of theoretical work.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper extends recent approaches in solving and learning PDEs based on Gaussian processes to a new class of processes called Q-exponential process. The proposed contribution is the more flexible properties for this new process, especially for PDEs that have stiff solutions, which may be impacted by smoothness assumptions coming from standard GPs.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: This paper is based on a novel idea of using q-exponential processes in the setting of modelling PDEs. This can lead to interesting applications, especially in the settings author mentioned, e.g., PDEs with erratic solutions.\n\nWeaknesses: I list the weaknesses below - please see questions part for further inquiries:\n\n(1) The paper is somewhat an incremental improvement over the previous approaches. The methodology is rather standard, replacing GP with q-exponential process, but deriving a standard variational inference procedure. Therefore, despite this is an interesting extension, I find the whole framework's novelty a bit limited.\n\n(2) Experimental results are insufficient and not very conclusive in my opinion - to justify the whole framework introduced in this paper.\n\n(3) There are some theoretical results - but zero intuition about how to interpret them. Please see my questions.\n\nQUESTIONS:\n1) Theory: In Section 4, there are theoretical results justifying this approach. However, while they look impressive, the message of this section is unclear in the main text. How do these theorems relate to the results on Gaussian processes, if there are any corresponding results? In terms of handling solutions that are less smooth, is there any noticeable improvement over GPs in theory?\n\n2) UQ: The main claim against a PINN based approach is that there is UQ available from GP and q-exponential based approaches. Can authors elaborate why such tightly concentrated uncertainty would be useful? Is there any realistic example where the UQ is meaningful, e.g,. depending on the observation locations (e.g. in a sparser area)? Can authors construct such an example?\n\n3) I'm not entirely convinced that by tuning and changing GP kernels, similar results you have obtained with the q-exponential process cannot be obtained. You have fixed the kernel. Can you elaborate and provide additional results to argue whether the improvements you observe remain with different (suitable) kernels?\n\n4) The comparisons are really thin. Many recent approaches like DeepONets are not compared to - there are similar probabilistic approaches, see e.g. *Vadeboncoeur, Arnaud, et al. \"Fully probabilistic deep models for forward and inverse problems in parametric PDEs.\" Journal of Computational Physics 491 (2023): 112369* and references therein.\n\nI think the experimental section needs much more work before this paper is ready for NeurIPS or a similar venue.\n\nLIMITATIONS:\nYes, very briefly. It could be expanded.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses the limitations of neural network and Gaussian Process (GP)-based methods in solving PDEs, particularly for systems with sharp transitions. It introduces the Q-exponential process (Q-EP), a generalisation of GPs that offers more expressive and flexible function estimators in settings involving non-smooth dynamics and derivative information. Using sparse variational inference, the proposed method achieves scalable and accurate PDE solutions with principled uncertainty quantification. Toy experiments on the Eikonal equation, Burgers’ equation, and inverse Darcy flow show that Q-EP outperforms GPs and neural networks in both accuracy and uncertainty estimation.\n\nSTRENGTHS AND WEAKNESSES:\nI quite like the idea of using the Q-exponential process, which can be seen as a generalised version of GPs for solving PDEs, offering a flexible choice of the exponent $q$. It also makes sense to employ sparse variational inference with inducing points to reduce algorithmic complexity when solving the functions. Overall, the proposed method is well-justified and theoretically sound. However, I have some comments and potential limitations below that I hope the authors can address.\n- However, a natural limitation arises in selecting the optimal value of $q$ for different settings. I’m curious about the performance when $q>2$, and is $q=1$ always optimal, as suggested by the experiments? Please clarify this point.\n- Another limitation I notice is the absence of real-world datasets in the experiments. I suggest the authors incorporate experiments on real datasets -such as physics-informed datasets, time series, or, ideally, image datasets. I would highly appreciate this addition, and it would positively influence my evaluation further. \n- Could the authors mention any potential applications of the Q-exponential process beyond solving PDEs and inverse problems? For example, might it be applicable in modelling processes within diffusion models? I’m curious whether such an extension is feasible.\n\nOverall, I lean toward accepting this work, but only weakly, due to two main limitations: the unclear guidance on selecting the exponent $𝑞$, and the lack of experiments on real-world datasets. I will adjust my score based on the authors’ responses to these concerns.\n\n**[After Rebuttal]**: My concerns have been addressed, and I am happy to raise my confidence score in advocating for the acceptance of this paper (from 2 to 4).\n\nQUESTIONS:\nPlease refer to the Weaknesses section for Questions.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

comments: SUMMARY:
The authors propose to use q-exponential process to solve and learn PDEs as they are designed to better handle data with abrupt changes and to more accurately model derivative information.

STRENGTHS AND WEAKNESSES:
Strengths:

1. The idea of using Q-EP is quite novel
2. They have great theoretical justification
3. Uncertainty quantification is provided
4. They demonstrate their approach in a range of tasks and show superior performance

Weaknesses:

for highly nonlinear PDEs, the variational approximation to the true posterior distribution might be poor, which could undermine the final solution

QUESTIONS:
1. how do we select the optimal q in real problems? it seems q=1 is nearly always the best and the convergence is faster in theory. do you suggest using q=1 for all tasks

2. it seems that the uncertainty provided by q=1 appears more structurally informative than that of the GP solver q=2. can you elaborate on the reason for this?

3. i feel that it may be a bit misleading to say q-ep is designed to handle abrupt changes in data. does it only refer to q<1? can you comment on that? also, is it possible to adjust the kernel in GP to address the same issue rather than use q-ep? 

4. q-ep does not have such trctable conditional and predictive distributions as GP. how do you overcome that?

5. how is q-ep compared with Besov process

LIMITATIONS:
no
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-23