Title: Physics-Informed Variational State-Space Gaussian Processes
Venue: NeurIPS 2024
Accepted: yes
Keywords: gaussian processes

variational approximations

state space gaussian processes

physics informed gaussian processes
Abstract: Differential equations are important mechanistic models that are integral to many scientific and engineering applications. With the abundance of available data there has been a growing interest in data-driven physics-informed models. Gaussian processes (GPs) are particularly suited to this task as they can model complex, non-linear phenomena whilst incorporating prior knowledge and quantifying uncertainty. Current approaches have found some success but are limited as they either achieve poor computational scalings or focus only on the temporal setting. This work addresses these issues by introducing a variational spatio-temporal state-space GP that handles linear and non-linear physical constraints while achieving efficient linear-in-time computation costs. We demonstrate our methods in a range of synthetic and real-world settings and outperform the current state-of-the-art in both predictive and computational performance.

Physics-Informed Variational State-Space

Abstract

Differential equations are important mechanistic models that are integral to many scientific and engineering applications. With the abundance of available data there has been a growing interest in data-driven physics-informed models. Gaussian processes (GPs) are particularly suited to this task as they can model complex, nonlinear phenomena whilst incorporating prior knowledge and quantifying uncertainty. Current approaches have found some success but are limited as they either achieve poor computational scalings or focus only on the temporal setting. This work addresses these issues by introducing a variational spatio-temporal state-space GP that handles linear and non-linear physical constraints while achieving efficient linear-in-time computation costs. We demonstrate our methods in a range of synthetic and real-world settings and outperform the current state-of-the-art in both predictive and computational performance.

1 Introduction

Physical modelling is integral in modern science and engineering with applications from climate modelling [62] to options pricing [6]. Here, the key formalism to inject mechanistic physical knowledge are differential equations (DEs), which given initial and/or boundary values, are typically solved numerically [8]. In contrast machine learning is data-driven, and aims to learn latent functions from observations. However the increasing availability of data has spurred interest in combining these traditional mechanistic models with data-driven methods through physics-informed machine learning. These hybrids approaches aim to improve predictive accuracy, computational efficiency by leveraging both physical inductive biases with observations [30, 44].

A principled way to incorporate prior physical knowledge is through Gaussian processes (GPs). GPs are stochastic processes

Naïve vs. Physics-Informed State-Space GP (PHYSS-GP)

∂

f

∂

t

-N

θ

f

= 0

∂

f

∂

t

-N

θ

f

= 0

∂

f

∂

t

-N

θ

f

= 0

∂

f

∂

t

-N

θ

f

= 0

∂

f

∂

t

-N

θ

f

= 0

vs.

Input,

x

Time,

t

and are a data-centric approach that facilitates the quantification of uncertainty. Recently AUTOIP was proposed in order to integrate non-linear physics into GPs [40], where solutions to ordinary and partial differential equations (ODEs, PDEs) are observed at a finite set of collocation points. This is an extension of the probabilistic meshless method (PMM, [12]) to the variational setting such that non linear equations can be incorporated. Similarly, [4] introduced HELMHOLTZ-GP, that constructs GP priors that adhere to curl and divergence-free constraints. Such properties are required for the

successful modelling of electromagnetic fields [61] and ocean currents through the Helmholtz decomposition [4]. These approaches enable the incorporation of physics but incur a cubic computational complexity from needlessly computing full covariance matrices, as illustrated in Fig. 1. For ODEs (time-series setting), extended Kalman smoothers incorporate non-linear physics (EKS) [65, 34] and recover popular ODE solvers whilst achieving linear-in-time complexity through state-space GPs [58, 25].

In this work we propose a unified physics informed state-space GP (PHYSS-GP) that is a probabilistic models where mechanistic/physics knowledge is incorporated as an inductive bias. We can handle both linear and non-linear PDEs and ODEs whilst maintaining linear-in-time computational efficiency. We additionally derive a state-space variational inference algorithm that further reduces the computational cost in the spatial dimension. We recover EKS, PMM, and HELMHOLTZ-GP as special cases, and outperform AUTOIP in terms of computational efficiency and predictive performance. In summary:

We derive a state-space GP that can handle spatio-temporal derivatives with a computational complexity that is linear in the temporal dimension.

With this we derive a unifying state-space variational inference framework that allows the incorporation of both linear and non-linear PDEs whilst achieving a linear-in-time complexity and recovering state-of-the-art methods such as EKS, PMM and HELMHOLTZ-GP.

We further explore three approximations, namely a structured variational posterior, spatial sparsity, and spatial minibatching, that reduce the cubic spatial computational costs to linear.

We showcase our methods on a variety of synthetic and real-world experiments and outperform the current state-of-the-art methods AUTOIP and HELMHOLTZ-GP both in terms computational and predictive performance.

Code to reproduce experiments is available at https://github.com/ohamelijnck/physs_gp .

2 Background on Gaussian Processes

Gaussian processes A GP is a distribution on an infinite collection of random variables such that any finite subset is jointly Gaussian [50]. Given observations X ∈ R N × F and y ∈ R N then

is a joint model where p ( f | θ ) is a zero mean GP prior with kernel K ( · , · ) , f ( X ) ∼ p ( f ( X ) | 0 , K ( X , X )) , and θ are (hyper) parameters. We are primarily concerned with the spatiotemporal setting where we observe N t temporal and N s spatial observations x t , s ∈ R , y t , s ∈ R on a spatio-temporal grid. Under a Gaussian likelihood, all quantities for inference and training are available analytically and, naïvely, carry a dominant computational cost of O (( N t N s ) 3 ) . For time series data, an efficient way to construct a GP over f (and its time derivatives) is through the state-space representation of GPs. Given a Markov kernel, the temporal GP prior can be written as the solution of a discretised linear time-invariant stochastic differential equation (LTI-SDE), which at time k is

where A is a transition matrix, q k is Gaussian noise, H is an observation matrix, and ¯ f is a d -dimensional vector of temporal derivatives ¯ f = [ f ( · ) , ∂ f ( · ) ∂ x , ∂ 2 f ( · ) ∂ x 2 , · · · ] ⊤ . With appropriately designed states, matrices and densities, SDEs of this form represent a large class of GP models, and Kalman smoothing enables inference in O ( N t d 3 ) , see [56]. In the spatio-temporal setting, when the kernel matrix decomposes as a Kronecker product K = K t ⊗ K s , then with a Markov time kernel, a state space form is admitted. This takes a particularly convenient form where the state is ¯ f t = [ ¯ f (( X s ) 1 , t ) , · · · , ¯ f (( X s ) Ns , t )] ⊤ , and inference requires O ( N t ( N s d ) 3 ) , see [60].

Derivative Gaussian processes One main appeal of GPs is that they are closed under linear operators. Let D [ · ] = D t D s [ · ] be linear functional that computes D = d t d s space-time derivatives with D t [ · ] = [ · , ∂ · ∂ t , ∂ 2 · ∂ t 2 , · · · ] and D s [ · ] = [ · , ∂ · ∂ s , ∂ 2 · ∂ s 2 , · · · ] , then at a finite set of index points, the joint prior between f and its time and spatial derivatives is

where ¯ f ( X ) = D f ( X ) and D ∗ is the adjoint of D , meaning it operates on the second argument of the kernel [54]. When jointly modelling a single time and space derivative ( d t = d s = 1 ) the latent functions are ¯ f = [ f , ∂ f ∂ s , ∂ f ∂ t , ∂ 2 f ∂ t ∂ s ] ⊤ and the kernel is

This is a multi-output prior whose samples are paths of f with its corresponding derivatives. This prior is commonly known as a derivative GP and has found applications in monotonic GPs [51], input-dependent noise [41, 67] and explicitly modelling derivatives [59, 17, 43]. State-space GPs can be employed in the temporal setting since the underlying state computes f ( x ) with its corresponding time derivatives. In Sec. 3.1, we extend this to the spatio-temporal setting.

3 Physics-Informed State-Space Gaussian Processes (PHYSS-GP)

Wenowpropose a flexible generative model for incorporating information from both data observations and (non-linear) physical mechanics. We consider general non-linear evolution equations of the form

with appropriate boundary conditions, where f : R F → R is the latent quantity of interest and N θ is a non-linear differential operator [49]. We assume that g : R P · D → R is measurable, and is well-defined such that there are sensible solutions to the differential equation [25]. We wish to place a GP prior over f and update our beliefs after 'observing' that it should follow the solution of the differential equation. In general this is intractable and can only be handled approximately. By viewing Eqn. (4) as a loss function that measures the residual between ∂ f ∂ t and the operator N θ f then the right hand side ( 0 ) are virtual observations. The PDE can now be observed at a finite set of locations known as collocation points. This is a soft constraint ( i.e. f is not guaranteed to follow the differential equation), but it can handle non-linear and linear mechanisms. However, there are special cases, namely curl and divergence-free constraints, that can be solved exactly. This follows from properties of vectors fields, where f defines a potential function where linear combinations of its partial derivatives define vector fields that enforce these properties. To handle both of these situations we propose the following generative model

where ¯ f q are derivative GPs (see Eqn. (3)) that are linearly mixed by W ∈ R (P D ) × (Q D ) , and Y ( O ) , 0 ( C ) ∈ R N × P are observations and collocation points over the P outputs and Y ( B ) ∈ R N × (P D ) are boundary values over the derivatives of each output. The observation matrices H O , H B simply select the relevant parts of F n . For further details on notation see App. A. In many case we want to observe the solution of the differential equation exactly, however in some cases it may be required to add observation noise ϵ C to the collocation points, whether for numerical reasons or to model inexact mechanics. This is a flexible generative model where different assumptions and approximations will lead to various physics informed methods such as AUTOIP, EKS, PMM, and HELMHOLTZ-GP that we will develop state space algorithms for. Additionally it is possible to learn missing physics by parameterising unknown terms in Eqn. (4) through the GP priors in Eqn. (6) (see App. B.2).

Example 3.1 (EKS Prior and PMM) . We recover EKS style generative models (see Hennig et al. [25]) when the mixing weight is identity W = I , and ϵ C , ϵ B → 0 , and the non-linear transform g is linearised. Let the prior be Markov p ( ¯ f ) = ∏ N t k p ( ¯ f k | ¯ f k -1 ) with marginals p ( ¯ f k ) = N ( ¯ f k | m -k , P -k ) . By taking a first-order Taylor linearisation g ( ¯ f k ) ≃ g ( m -k ) + ∂ g ( m -k ) ∂ m -k δ ¯ f k with

δ ¯ f k ∼ N ( 0 , P -k ) the joint is

This is now a form that can directly be implemented into an extended Kalman smoothing algorithm [63]. When Q > 1 the state ¯ f is constructed by stacking the individual states of each latent [56]. With linear ODEs EKS coincides with PMM.

Example 3.2 (HELMHOLTZ-GP and Curl and Divergence-Free Vector Fields in 2 D) . Let v = [ v t , v s 1 , v s 2 ] denote a 3 D-vector field, then curl indicates the tendency of a vector field to rotate and divergence at a specific point indicates the tendency of the field to spread out. Curl and divergence-free fields follow

where ∇ = [ ∂ ∂ t , ∂ ∂ s 1 , ∂ ∂ s 2 ] . Two basic properties of vector fields state that the divergence of a curl field and the curl of a derivative field are zero [3]. Let [ f 1 , f 2 ] be scalar potential functions then

define curl and divergence-free fields. In 2 D this simplifies to using the grad and rot operators over v = [ v s 1 , v s 2 ] (see [4]). Placing GP priors over f q we incorporate this into Eqn. (6) by defining

HELMHOLTZ-GP is defined as the sum of GP priors over 2 D curl and divergence-free fields [4].

3.1 A Spatio-Temporal State-Space Prior

The generative model in Eqn. (6) contains two complications: i) it includes potential non-lineararities, and ii) the independent priors are defined over latent functions with their partial derivatives which substantially increases the computational complexity. We wish to tackle both issues through statespace algorithms that are linear-in-time. We begin by deriving a state-space model that observes derivatives across space and time (see App. A.3 for the simpler time-series setting). In Sec. 3.2 we further derive a state-space variational lower bound that will enable computational speeds up in the spatial dimension.

First, we show how Kronecker structure in the kernel allows us to rewrite the model as the solution to an LTI-SDE. From the definition of D , the separable covariance matrix has a repetitive structure that can be represented through a Kronecker product. The gram matrix is

To find a Kronecker form of the gram matrix over X , we will exploit the fact that X is on a spatiotemporal grid and that the kernel is separable. Due to the separable structure a derivative over either the spatio (or temporal) dimension only affects the corresponding kernel, and so when considering X , the gram matrix is still Kronecker structured:

The full prior over (a permuted) X is now given as

This is the form of a spatio-temporal Gaussian process with derivative kernels that can be immediately cast into a state-space form as in Eqn. (2) where H = I , as we want to observe the whole state, not just f . The marginal likelihood and the GP posterior can now be computed using standard Kalman filtering and smoothing algorithms with a computational time of O ( N t · ( N s · d s · d ) 3 ) . Inference in PHYSS-GP now follows Ex. 3.1 by recognising that the filtering state consists of the spatial points with there spatio-temporal derivatives. The EKS prior in Ex. 3.1 can now be simply extended to the PDE setting by placing colocation points on a spatio-temporal grid [35].

3.2 A State-Space Variational Lower Bound (PHYSS-VGP and PHYSS-EKS)

We now derive a variational lower bound for PHYSS-GP that maintains the computational benefits of state-space GPs. This acts as an alternative way of handling the non-linearity of g in Eqn. (6), and will also enable the reduction of the cubic spatial computation complexity in Sec. 4. We start by focusing on the single latent function setting ( Q = 1 ) and collect all terms that relate to observations in Eqn. (6) with p ( Y | ¯ f ) = ∏ N n p ( y ( O ) n | H O F n ) p ( 0 ( C ) n | g ( F n )) p ( y ( B ) n | H B F n ) . VI frames inference as the minimisation of the Kullback-Leibler divergence between the true posterior and an approximate posterior, which leads the optimisation of the ELBO [28]:

where we define the approximate posterior q ( f | ξ ) ≜ = N( f | m , S ) as a free-form Gaussian with ξ = ( m , S ) and m ∈ R DN × 1 , S ∈ R DN × DN . The aim is to represent the approximate posterior as a state-space GP posterior, which will enable efficient computation of the whole evidence lower bound (ELBO). We will achieve this through the use of natural gradients. The natural gradient preconditions the standard gradient with the inverse Fisher matrix, meaning the information geometry of the parameter space is taken into account, leading to faster convergence and superior performance [2, 31, 27]. For Gaussian approximate posteriors the natural gradient has a simple form [26]

where λ = ( S -1 m , 1 / 2 S -1 ) and µ = ( m , mm ⊤ + S ) are the natural and expectation parameterisations. This is known as conjugate variational inference (CVI) as ˜ λ represent the natural parameters for the conjugate prior η [31, 10, 20, 72]. For now, we will assume that the likelihood is conjugate to ensure that [ λ k ] 2 is p.s.d , this will be relaxed in Sec. 5. The derivative of the ELL is

where the expectation is under q ( ¯ f (t , s) ) , a D dimensional Gaussian over the spatio-temporal derivatives at location x t , s . Within the sum, the only elements of [ µ ] 2 whose gradient will propagate through the expectation are the D × D elements corresponding to these locations. These points are unique and so ∂ ELL ∂ [ µ ] 2 has some (permutated) block-diagonal structure, hence Eqn. (14) can be written as

where ˜ Y t is D -dimensional. The natural gradient update, i.e. q ( ¯ f t ) in moment parameterisation, can now be computed using Kalman smoothing in O ( N t · ( N s · d s · d ) 3 ) . Collecting ˜ Y = vec ( [ ˜ Y t ] ) , ˜ V = blkdiag ( [ ˜ V t ] ) , then the ELBO can also be computed efficiently by substituting this form of q ( ¯ f t ) in

where the first two terms only depend on q ( D f t ) and the final term is simply a by-product of running the Kalman filter, leading to a dominant computational complexity of O ( N · ( N s · d s · d ) 3 ) . This cost is linear in the datapoints ( N ) because the expected log likelihood above decomposes across all spatio-temporal locations. In summary we have shown that natural gradient is equivalent updating a block-diagonal likelihood that decomposes across time; hence the approximate posterior is computable via Kalman smoothing algorithms. Extending to multiple latent functions ( Q > 1 ) we define a full Gaussian approximate posterior that captures all correlations between the latent functions q ( ¯ f 1 , · · · , ¯ f Q ) ≜ = N ( ¯ f 1 , · · · , ¯ f Q | m , S ) where m ∈ R ( N × Q ) × 1 , S ∈ R ( N × Q ) × ( N × Q ) . All the observation models in Eqn. (6) decompose across data points, hence Eqn. (16) is still block-diagonal and decomposes across time, except now each component is of dimension Q × N t as it encodes the correlations of spatial points and their spatio-temporal derivatives across the latent functions. We denote this model as PHYSS-VGP and PHYSS-EKS when using a EKS prior (see Ex. 3.1).

Theorem 3.1. Let the approximate posterior be (full) Gaussian q ( ¯ f 1 , · · · , ¯ f Q ) ≜ = N ( ¯ f 1 , · · · , ¯ f Q | m , S ) where m ∈ R ( N × Q ) × 1 , S ∈ R ( N × Q ) × ( N × Q ) . When g is linear a single natural gradient step with β = 1 recovers the optimal solution p ( ¯ f 1 , · · · , ¯ f Q | Y ) .

We prove this in App. A.5.4. This result not only demonstrates the optimality of our proposed inference scheme in the linear Gaussian setting, but confirms that we recover batch models like PMM and HELMHOLTZ-GP, as well as EKS (see Ex. 3.1).

4 Reducing the Spatial Computational Complexity

We now propose three approaches that reduce the cubic computational complexity in the number of spatial derivatives and locations. The first augments the process with inducing points that alleviate cubic costs associated with N s . The second is a structured variational approximation that defines the approximate posterior only over the temporal prior and alleviates cubic costs associated with d s . Finally, we introduce spatial mini-batching that alleviates linear N s costs. When used in conjunction, the dominant computation cost is O ( N t · d s · ( M s · d t ) 3 ) . These approximations are not only useful for the state-space setting and can readily be applied to reduce the computational complexity for batch variational models (such as AUTOIP). See App. B.1 for more details.

Spatio-Temporal Inducing Points (PHYSS-SVGP) In this first approximation, denoted by PHYSSSVGP, we augment the full prior p ( ¯ f ) with inducing points. By defining these inducing points on a spatio-temporal grid, we will show that we can still exploit Markov conjugate operations through natural gradients. Let ¯ u = D u ∈ R M × D be inducing points at locations Z ∈ R M × F . From the standard SVGP formulation [27], the ELBO is

where q ( ¯ f , ¯ u ) ≜ = p ( ¯ f | ¯ u ) q (¯ u ) . By defining the inducing points on a spatio-temporal grid at temporal locations X t ∈ R N t and spatial Z s ∈ R M s × ( F -1) then the marginal p ( ¯ f | ¯ u ) is Gaussian with mean

and variance given in Eqn. (41). This Kronecker structure allows us to again 'decouple' space and time, leading to natural gradient updates with block size M s × D , reducing the computational complexity to O ( N ( M s · d s · d ) 3 ) . For full details, see App. A.5.1.

Structured Variational Inference (PHYSS-SVGPH ) This second approximation, denoted as PHYSSSVGPH, defines the inducing points only over the temporal derivatives. This is a useful approximation as it can drastically reduce the size of the filter state, making it more computationally and memory efficient. We begin by defining the joint prior as

where p ( F | D t f ) is a Gaussian conditional with mean

We then define a structured variational posterior

Substituting this into the ELBO we see that all the terms with the prior spatial derivatives cancel

Again, the marginal q ( D t f ) maintains Kronecker structure, enabling Markov conjugate operations, leading to a computational cost of O ( N · d s · ( N s · d ) 3 ) , see App. A.5.2. These variational approximations can simply be applied to non-state-space variational approximation, see App. B.1.

Spatial Mini-Batching A standard approach for handling big data is through mini-batching where the ELL is approximated using only a data subsample [27]. Directly appling mini-batching would be of little computation benefit because computation of the ELBO requires running a Kalman smoother that iterates through all time points. Instead, we mini-batch by subsampling B s spatial points

where i is uniformly sampled. We used in conjunction with PHYSS-SVGP and PHYSS-SVGPH, this results in dominant costs of O ( N t ( M s · d s · d ) 3 ) and O ( N t · d s · ( M s · d ) 3 ) when B s ≪ N s .

5 Handling the PSD Constraint

As discussed in Sec. 3.2 when the differential equation is non-linear, the model is no longer conjugate and the resulting natural gradients are not guaranteed to result in p.s.d updates. This issue has received some attention in the literature [53, 64, 39], but these approaches do not maintain an efficient conjugate representation. One distinction is [72], which uses the Gauss-Newton approximation to maintain conjugate operations. We now extend this to support spatial inducing points and non-linear transformations. Due to space we focus on PHYSS-SVGP, but see App. A.5.3 for further details. The troublesome term for the natural gradient update in Eqn. (14) is the Jacobian of the ELL w.r.t. to the second expectation parameter; which is not guaranteed to be p.s.d unless the ELL is log convex [39]. Focusing at a single location n = ( t, s ) :

we apply the Bonnet's and Price's theorem [38] to bring the differential inside the expectation and make a Gauss-Newton [19] approximation ensuring that the Jacobian is p.s.d

and g n,p = g (¯ u n ) (Eqn. (4)) and µ n is the mean of p ( ¯ f n | ¯ u t ) (Eqn. (19)). When using spatial mini-batching Eqn. (21) is also subsampled.

6 Related Work

From the optimality of natural gradients, in the conjugate setting, we exactly recover batch GP based models such as [68, 29, 4]. Our inference scheme also applies to models that do not require derivative information i.e. in d t = d s = 1 . As a special case, we recover [20], but we have extended the inference scheme to support spatial mini-batching, allowing big spatial datasets to be used. The linear weighting matrix can be used to define a linear model of coregionalisation and its variants [7, 77, 42, 66] and through appropriately designed functionals also non-linear variants [73].

In Álvarez et al. [76] GP priors over the solution of differential equations are obtained through a stochastic forcing term but they only consider situations where the Greens function is available. In [22, 23, 57, 33], efficient state-space algorithms are derived but are limited to the temporal setting only. Similarly, Heinonen et al. [24], learn a 'free-form ODE'. In the spatio-temporal setting Krämer et al. [35] and Duffin et al. [14] (which builds [18]) derive extended Kalman filter algorithms. Additionally there are approaches to constraining GPs by linear differential equations [37, 1, 5]. More generally than [4] in [21] GP priors over the solutions to linear PDEs with constant coefficients are derived.

Beyond GP based models, physics informed neural networks (PINNs) incorporate physics by constructing a loss function between the network and the differential equation at a finite set of collocation points [48]. This amounts to a highly complex optimisation problem [36] bringing difficulties for training [70, 71] and uncertainty quantification (UQ) [16]. Current approaches to quantifying uncertainty in PINNs are based on dropout [75] and conformal predictions [47]. In recent years UQ and deep learning has received much attention however is limited by its computational cost [45].

-

3

-

2

-

1

0

1

2

3

'time'

-

2

0

2

'space'

Scalar Potential (

f

)

-

3

-

2

-

1

0

1

2

3

'time'

Magnetic Field Strengh (

H

-field:

-∇

f

)

-

0

.

02

0

.

00

0

.

02

7 Experiments

We now examine the performance of our PHYSS-GP methods on multiple synthetic and real-world datasets. We compare against a batch GP (no physical knowledge) and current state-of-art methods AUTOIP and HELMHOLTZ-GP. We provide more details on all experiments in App. B.

Non-linear Damped Pendulum In this first synthetic example, we consider learning the non-linear dynamics of a damped swinging pendulum. This is described by a second-order differential equation

where b > 0 . The first term is a non-linear forcing term, and the third is the damping term. With b = 0 . 2 we simulate a solution using Euler's method [9] and generate 20 points in t ∈ [0 , 6] for training and 200 in t ∈ [6 , 30] for testing, with additive Gaussian noise of variance 0 . 01 .

We are interested in i) the effect of the number of collocation points, ii) the effect of the optimisation algorithm. To answer these questions, we compare against AUTOIP on [10 , 100 , 500 , 1000] collocation points, with and without whitening. Results are tabulated in Table 1. As expected, the predictive RMSE of all models decreases as the number of collocation points increases. Due to the cubic complexity of AUTOIP, the total time significantly increases as the number of collocation points increases. For example, when using 1000 collocation points, AUTOIP is ≈ 39 times slower than PHYSS-GP. Interestingly, the un-whitened case performs poorly, possibly due to the nonlinearity of the differential equation making optimisation difficult. This indicates that either whitening or natural gradients are required to handle the non-linearity arising due to the differential equation.

Curl-free Magnetic Field Strength In this experiment, we consider modelling the magnetic field strength of a dipole H ( r ) = -∇ ψ ( r ) , where ψ ( r ) = m · r / | r | 3 is a scalar potential function [11]. Labelling the input dimensions as 'time', 'space' and ' z ', we let m = [0 , 1 , 0] and generate observations from a spatio-temporal grid with N t = 50 , and N s = [5 , 10 , 20] , at z = 1 . H ( r ) is a curl-free field and so we compare the curl free part of HELMHOLTZ-GP against PHYSS-GP and its variants. HELMHOLTZ-GP and PHYSS-GP are equivalent models (as this is the conjugate setting,

-

1

0

1

Mean

PHYSS-EKS

AUTOIP

PHYSS-SVGPH

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

-

1

0

1

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

Diffusion-Reaction System Consider a diffusion-reaction system given by an Allen-Cahn equation

where x ∈ [ -1 , 1] , t ∈ [0 , 1] , u (0 , x ) = x 2 cos( π x ) , u ( t, -1) = u ( t, 1) and ∂ u ∂ x ( t, -1) = ∂ u ∂ x ( t, 1) . Following [40], we use the solution provided by [49] and sample 256 training examples from t ∈ [0 , 0 . 28] . We compare PHYSS-EKS (where g is linerized in the EKS prior), PHYSS-SVGP and PHYSS-SVGPH against AUTOIP. Following [40], we use a learning rate of 0 . 001 for Adam. For AUTOIP, we place 100 collocation points across the whole input domain on a regular grid. For both PHYSS-SVGP, and PHYSS-SVGPH we require more collocation points in the temporal dimension and place them on a regular grid of size 20 × 10 . For PHYSS-EKS we use an integrated Wiener kernel (IWP) on time [56] and place 100 × 40 collocation points. We are unable to place more collocation for AUTOIP due to computational limits. Results are presented in Fig. 3 and Table 3. PHYSS-EKS requires only a single epoch and can better handle the sharp boundaries. Our method PHYSS-SVGPH is over twice as fast as AUTOIP whilst achieving similar predictive RMSE.

Ocean Currents We now model oceanic currents in the Gulf of Mexico. We follow [4] and use the dataset provided by D'Asaro et al. [15] that has information from over 1 , 000 buoys. We focus on the region in long. [ -90 , -84 . 5] , lat. [26 , 30] on 2016-02-25, by computing hourly averages. This results in N = 42 , 243 observations, and we construct a test-train split on 0 . 1 per cent of the data. It is infeasible to run HELMHOLTZ-GP due to data size (in Berlinghieri et al. [4], observations from only 19 buoys are used with N = 55 ). However, we run PHYSS-SVGPH with 50 spatial inducing

-

9

.

9

-

9

.

8

-

9

.

7

-

9

.

6

-

9

.

5

-

9

.

4

Eastings

×

10

6

3

.

0

3

.

2

3

.

4

Northings

×

10

6

2016-02-25 00:00:00

-

9

.

9

-

9

.

8

-

9

.

7

-

9

.

6

-

9

.

5

-

9

.

4

Eastings

×

10

6

2016-02-25 20:00:00

points and a spatial mini-batch size of 10 , and plot results in Fig. 4. Our predictions are in excellent agreement with the test data, achieving an RMSE of 0 . 14 , NLPD of -0 . 52 , CRPS of 0 . 078 , and an average run-time of 1 . 86( s ) per epoch.

8 Conclusion

We introduced a physics-informed state-space GP that integrates observational data with physical knowledge. Within the variational inference framework, we derived a computationally efficient algorithm that uses Kalman smoothing to achieve linear-in-time costs. To gain further computational speed-ups, we proposed three approximations with inducing points, spatial mini batching and structured variational posteriors. When used in conjunction, they allow us to handle large-scale spatiotemporal problems. The bottleneck is always the state size, where nearest neighbours GPs [13, 74] could be explored. For highly non-linear problems, future directions could explore deep approaches [52] or more flexible kernel families [69]. One limitation is the use of the collocation method which is only enforcing the differential equation point wise, whilst future work could look at the more general methods of weighted residuals [46].

Acknowledgments and Disclosure of Funding

OH acknowledges funding from The Alan Turing Institute PhD fellowship programme and the UKRI Turing AI Fellowship (EP/V02678X/1). AS acknowledges support from the Research Council of Finland (339730). TD acknowledges support from UKRI Turing AI Acceleration Fellowship (EP/V02678X/1) and a Turing Impact Award from the Alan Turing Institute. The authors acknowledges the University of Warwick Research Technology Platform (aquifer) for assistance in the research described in this paper. For the purpose of open access, the authors have applied a Creative Commons Attribution (CC-BY) license to any Author Accepted Manuscript version arising from this submission.

References

C. G. Albert. Gaussian processes for data fulfilling linear differential equations. In Proceedings , volume 33, page 5. MDPI, 2019.

S.-i. Amari. Natural gradient works efficiently in learning. Neural Computation , 10(2):251-276, 1998.

G. B. Arfken, H. J. Weber, and F. E. Harris. Mathematical Methods for Physicists: A Comprehensive Guide . Academic Press, 2011.

R. Berlinghieri, B. L. Trippe, D. R. Burt, R. Giordano, K. Srinivasan, T. Özgökmen, J. Xia, and T. Broderick. Gaussian processes at the Helm (holtz): A more fluid model for ocean currents. arXiv preprint arXiv:2302.10364 , 2023.

A. Besginow and M. Lange-Hegermann. Constraining Gaussian processes to systems of linear ordinary differential equations. In Advances in Neural Information Processing Systems 35 (NeurIPS) , pages 29386-29399. Curran Associates, Inc., 2022.

F. Black and M. Scholes. The pricing of options and corporate liabilities. Journal of Political Economy , 81(3):637-654, 1973.

E. V. Bonilla, K. Chai, and C. Williams. Multi-task Gaussian process prediction. In Advances in Neural Information Processing Systems 20 (NIPS) . Curran Associates, Inc., 2008.

D. Borthwick. Introduction to Partial Differential Equations . Universitext. Springer International Publishing, 2017.

J. Butcher. Numerical Methods for Ordinary Differential Equations . Wiley, 2016.

P. E. Chang, W. J. Wilkinson, M. E. Khan, and A. Solin. Fast variational learning in state-space Gaussian process models. In 30th IEEE International Workshop on Machine Learning for Signal Processing (MLSP) , pages 1-6. IEEE, 2020.

T. Chow. Introduction to Electromagnetic Theory: A Modern Perspective . Jones and Bartlett Publishers, 2006.

J. Cockayne, C. Oates, T. Sullivan, and M. Girolami. Probabilistic numerical methods for PDE-constrained Bayesian inverse problems. AIP Conference Proceedings , 1853(1), 06 2017.

A. Datta, S. Banerjee, A. O. Finley, and A. E. Gelfand. Hierarchical nearest-neighbor Gaussian process models for large geostatistical datasets. Journal of the American Statistical Association , 111(514):800-812, 2016.

C. Duffin, E. Cripps, T. Stemler, and M. Girolami. Low-rank statistical finite elements for scalable model-data synthesis. Journal of Computational Physics , 463:111261, 2022.

E. D'Asaro, C. Guigand, A. Haza, H. Huntley, G. Novelli, T. Özgökmen, and E. Ryan. Lagrangian submesoscale experiment (LASER) surface drifters, interpolated to 15-minute intervals, 2017. URL https://data.gulfresearchinitiative.org/data/R4.x265.237:0001 .

C. Edwards. Neural networks learn to speed up simulations. Communications of the ACM , 65: 27-29, 04 2022.

D. Eriksson, K. Dong, E. Lee, D. Bindel, and A. G. Wilson. Scaling Gaussian process regression with derivatives. In Advances in Neural Information Processing Systems 31 (NeurIPS) . Curran Associates, Inc., 2018.

M. Girolami, E. Febrianto, G. Yin, and F. Cirak. The statistical finite element method (statFEM) for coherent synthesis of observation data and model predictions. Computer Methods in Applied Mechanics and Engineering , 375:113533, 2021.

G. H. Golub and V. Pereyra. The differentiation of pseudo-inverses and nonlinear least squares problems whose variables separate. SIAM Journal on Numerical Analysis , 10(2):413-432, 1973.

O. Hamelijnck, W. J. Wilkinson, N. A. Loppi, A. Solin, and T. Damoulas. Spatio-temporal variational Gaussian processes. In Advances in Neural Information Processing Systems (NeurIPS) . Curran Associates, Inc., 2021.

M. Harkonen, M. Lange-Hegermann, and B. Raita. Gaussian process priors for systems of linear partial differential equations with constant coefficients. In International Conference on Machine Learning , pages 12587-12615. PMLR, 2023.

J. Hartikainen and S. Särkkä. Sequential inference for latent force models. In Proceedings of the Twenty-Seventh Conference on Uncertainty in Artificial Intelligence , UAI'11, page 311-318, Arlington, Virginia, USA, 2011. AUAI Press.

J. Hartikainen, M. Seppänen, and S. Särkkä. State-space inference for non-linear latent force models with application to satellite orbit prediction. In Proceedings of the 29th International Coference on International Conference on Machine Learning , ICML'12, page 723-730. Omnipress, 2012.

M. Heinonen, Çagatay Yildiz, H. Mannerström, J. Intosalmi, and H. Lähdesmäki. Learning unknown ODE models with Gaussian processes. In International Conference on Machine Learning (ICML) , 2018.

P. Hennig, M. Osborne, and H. Kersting. Probabilistic Numerics: Computation as Machine Learning . Cambridge University Press, 2022.

J. Hensman, M. Rattray, and N. D. Lawrence. Fast variational inference in the conjugate exponential family. In Advances in Neural Information Processing Systems (NIPS) , pages 2888-2896. Curran Associates Inc., 2012.

J. Hensman, N. Fusi, and N. D. Lawrence. Gaussian processes for big data. In Proceedings of the Twenty-Ninth Conference on Uncertainty in Artificial Intelligence (UAI) , pages 282-290. AUAI Press, 2013.

M. D. Hoffman, D. M. Blei, C. Wang, and J. Paisley. Stochastic variational inference. Journal of Machine Learning Research , 14, 2013.

C. Jidling, N. Wahlström, A. Wills, and T. B. Schön. Linearly constrained Gaussian processes. In Advances in Neural Information Processing Systems 30 (NeurIPS) . Curran Associates, Inc., 2017.

G. E. Karniadakis, I. G. Kevrekidis, L. Lu, P. Perdikaris, S. Wang, and L. Yang. Physicsinformed machine learning. Nature Reviews Physics , 3(6):422-440, 2021.

M. E. Khan and W. Lin. Conjugate-computation variational inference: Converting variational inference in non-conjugate models to inferences in conjugate models. In Proceedings of the International Conference on Artificial Intelligence and Statistics (AISTATS) , 2017.

D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXv preprint arXiv:1412.6980 , 2014.

N. Krämer and P. Hennig. Linear-time probabilistic solution of boundary value problems. In Advances in Neural Information Processing Systems 34 (NeurIPS) , pages 11160-11171. Curran Associates, Inc., 2021.

N. Krämer and P. Hennig. Stable implementation of probabilistic ODE solvers. Journal of Machine Learning Research , 25(111):1-29, 2024.

N. Krämer, J. Schmidt, and P. Hennig. Probabilistic numerical method of lines for timedependent partial differential equations. In Proceedings of the 25th International Conference on Artificial Intelligence and Statistics , volume 151 of Proceedings of Machine Learning Research , pages 625-639. PMLR, 2022.

A. S. Krishnapriyan, A. Gholami, S. Zhe, R. Kirby, and M. W. Mahoney. Characterizing possible failure modes in physics-informed neural networks. Advances in Neural Information Processing Systems 34 (NeurIPS) , 2021.

M. Lange-Hegermann. Algorithmic linearly constrained Gaussian processes. In Advances in Neural Information Processing Systems 31 (NeurIPS) , pages 2137-2148. Curran Associates, Inc., 2018.

W. Lin, M. E. Khan, and M. Schmidt. Stein's lemma for the reparameterization trick with exponential family mixtures. arXiv preprint arXiv:1910.13398 , 2019.

W. Lin, M. Schmidt, and M. E. Khan. Handling the positive-definite constraint in the Bayesian learning rule. In Proceedings of the 37th International Conference on Machine Learning , Proceedings of Machine Learning Research. PMLR, 2020.

D. Long, Z. Wang, A. Krishnapriyan, R. Kirby, S. Zhe, and M. Mahoney. AutoIP: A united framework to integrate physics into Gaussian processes. In Proceedings of the 39th International Conference on Machine Learning , volume 162 of Proceedings of Machine Learning Research . PMLR, 17-23 Jul 2022.

A. Mchutchon and C. Rasmussen. Gaussian process training with input noise. In Advances in Neural Information Processing Systems 25 (NeurIPS) . Curran Associates, Inc., 2011.

P. Moreno-Muñoz, A. Artés, and M. Álvarez. Heterogeneous multi-output Gaussian process prediction. In Advances in Neural Information Processing Systems 31 (NeurIPS) . Curran Associates, Inc., 2018.

M. Padidar, X. Zhu, L. Huang, J. R. Gardner, and D. S. Bindel. Scaling Gaussian processes with derivative information using variational inference. In Advances in Neural Information Processing Systems (NeurIPS) . Curran Associates, Inc., 2021.

I. Pan, L. R. Mason, and O. K. Matar. Data-centric engineering: integrating simulation, machine learning and statistics. challenges and opportunities. Chemical Engineering Science , 249: 117271, 2022.

T. Papamarkou, M. Skoularidou, K. Palla, L. Aitchison, J. Arbel, D. Dunson, M. Filippone, V. Fortuin, P. Hennig, J. M. Hernández-Lobato, A. Hubin, A. Immer, T. Karaletsos, M. E. Khan, A. Kristiadi, Y. Li, S. Mandt, C. Nemeth, M. A. Osborne, T. G. J. Rudner, D. Rügamer, Y. W. Teh, M. Welling, A. G. Wilson, and R. Zhang. Position: Bayesian deep learning is needed in the age of large-scale AI. In Proceedings of the 41st International Conference on Machine Learning , volume 235 of Proceedings of Machine Learning Research , pages 39556-39586. PMLR, 2024.

M. Pförtner, I. Steinwart, P. Hennig, and J. Wenger. Physics-informed Gaussian process regression generalizes linear PDE solvers. arXiv preprint arXiv:2212.12474 , 2022.

L. Podina, M. T. Rad, and M. Kohandel. Conformalized physics-informed neural networks. In ICLR 2024 Workshop on AI4Differential Equations in Science , 2024. URL https:// openreview.net/forum?id=ZoFWS818qG .

M. Raissi and G. E. Karniadakis. Hidden physics models: Machine learning of nonlinear partial differential equations. Journal of Computational Physics , 2017.

M. Raissi, P. Perdikaris, and G. Karniadakis. Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational Physics , 378:686-707, 2019.

C. Rasmussen and C. Williams. Gaussian Processes for Machine Learning . MIT Press, Cambridge, MA, USA, 2006.

J. Riihimäki and A. Vehtari. Gaussian processes with monotonicity information. In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics (AISTATS) , volume 9 of Proceedings of Machine Learning Research . PMLR, 2010.

H. Salimbeni and M. Deisenroth. Doubly stochastic variational inference for deep Gaussian processes. In Advances in Neural Information Processing Systems 30 (NeurIPS) , pages 45884599. Curran Associates, Inc., 2017.

H. Salimbeni, S. Eleftheriadis, and J. Hensman. Natural gradients in practice: Non-conjugate variational inference in gaussian process models. In Proceedings of the International Conference on Artificial Intelligence and Statistics (AISTATS) , 2018.

S. Särkkä. Linear operators and stochastic partial differential equations in gaussian process regression. In Proceedings of the 21st International Conference on Artificial Neural Networks Volume Part II , ICANN'11, page 151-158. Springer-Verlag, 2011.

S. Särkkä and A. F. García-Fernández. Temporal parallelization of bayesian smoothers. IEEE Transactions on Automatic Control , 2021.

S. Särkkä and A. Solin. Applied Stochastic Differential Equations . Cambridge University Press, 2019.

J. Schmidt, N. Krämer, and P. Hennig. A probabilistic state space model for joint inference from differential equations and data. In Advances in Neural Information Processing Systems 34 (NeurIPS) , pages 12374-12385. Curran Associates, Inc., 2021.

M. Schober, D. K. Duvenaud, and P. Hennig. Probabilistic ODE solvers with Runge-Kutta means. In Advances in Neural Information Processing Systems 27 (NeurIPS) . Curran Associates, Inc., 2014.

E. Solak, R. Murray-smith, W. Leithead, D. Leith, and C. Rasmussen. Derivative observations in Gaussian process models of dynamic systems. In Advances in Neural Information Processing Systems 15 (NIPS) . MIT Press, 2002.

A. Solin. Stochastic Differential Equation Methods for Spatio-Temporal Gaussian Process Regression . Doctoral thesis, Aalto University, 2016.

A. Solin, M. Kok, N. Wahlström, T. B. Schön, and S. Särkkä. Modeling and interpolation of the ambient magnetic field by Gaussian processes. Transactions on Robotics , 34(4):1112-1127, 2018.

T. Stocker. Introduction to Climate Modelling . Springer Science & Business Media, 2011.

S. Särkkä. Bayesian Filtering and Smoothing . Institute of Mathematical Statistics Textbooks. Cambridge University Press, 2013.

M.-N. Tran, D. H. Nguyen, and D. Nguyen. Variational Bayes on manifolds. Statistics and Computing , 31:1-17, 2021.

F. Tronarp, H. Kersting, S. Särkkä, and P. Hennig. Probabilistic solutions to ordinary differential equations as nonlinear Bayesian filtering: a new perspective. Statistics and Computing , 29: 1297-1315, 2018.

J. Vanhatalo, M. Hartmann, and L. Veneranta. Additive Multivariate Gaussian Processes for Joint Species Distribution Modeling with Heterogeneous Data. Bayesian Analysis , 15:415 447, 2020.

C. Villacampa-Calvo, B. Zaldívar, E. C. Garrido-Merchán, and D. Hernández-Lobato. Multiclass Gaussian process classification with noisy inputs. Journal of Machine Learning Research , 22(1), 2021.

N. Wahlström, M. Kok, T. B. Schön, and F. Gustafsson. Modeling magnetic fields using Gaussian processes. In IEEE International Conference on Acoustics, Speech and Signal Processing , pages 3522-3526, 2013.

K. Wang, O. Hamelijnck, T. Damoulas, and M. Steel. Non-separable non-stationary random fields. In Proceedings of the 37th International Conference on Machine Learning (ICML) , volume 119 of Proceedings of Machine Learning Research , pages 9887-9897. PMLR, 2020.

S. Wang, X. Yu, and P. Perdikaris. When and why PINNs fail to train: A neural tangent kernel perspective. Journal of Computational Physics , 449:110768, 2022.

S. Wang, S. Sankaran, and P. Perdikaris. Respecting causality for training physics-informed neural networks. Computer Methods in Applied Mechanics and Engineering , 421:116813, 2024.

W. J. Wilkinson, S. Särkkä, and A. Solin. Bayes-Newton methods for approximate Bayesian inference with PSD guarantees. Journal of Machine Learning Research , 24(83):1-50, 2023.

A. G. Wilson, D. A. Knowles, and Z. Ghahramani. Gaussian process regression networks. In Proceedings of the 29th International Coference on International Conference on Machine Learning , page 1139-1146, 2012.

L. Wu, G. Pleiss, and J. P. Cunningham. Variational nearest neighbor Gaussian process. In Proceedings of the 39th International Conference on Machine Learning (ICML) , volume 162 of Proceedings of Machine Learning Research , pages 24114-24130. PMLR, 2022.

D. Zhang, L. Lu, L. Guo, and G. E. Karniadakis. Quantifying total uncertainty in physicsinformed neural networks for solving forward and inverse stochastic problems. Journal of Computational Physics , 397:108850, 2019.

M. Álvarez, D. Luengo, and N. D. Lawrence. Latent force models. In Proceedings of the Twelth International Conference on Artificial Intelligence and Statistics (AISTATS) , volume 5 of Proceedings of Machine Learning Research , pages 9-16. PMLR, 2009.

M. A. Álvarez, L. Rosasco, and N. D. Lawrence. Kernels for vector-valued functions: A review. Foundations and Trends in Machine Learning , 4(3):195-266, 2012.

Appendices

A Variational Approximation Derivation

A.1 Overview of Notation

A.2 Layout of Vectors and Matrices

We use a numerator layout for derivatives. Let Q denote the number of independent latent functions and D the number of derivatives computed, and let f q,d denote the latent GP for d 'th derivative of the q 'th latent function. We will need to keep track of the permutation of our data w.r.t. to space, time, and latent functions. Inspired by 'row-major' and 'column-major' layouts, we will use the following terminology that describes the ordering of the data across latent functions and time and space:

latent-data: F = F ld = [ F 1 ( X ) , · · · , F Q ( X )] with F q ( X ) = [ f q, 1 ( X ) , · · · , f q,D ( X )] which is ordered by stacking each of the latent functions on top of each other.

data-latent: F dl = [ F 1 ( X n ) , · · · , F Q ( X n )] N n which is ordered by stacking the latent functions evaluated at each data point across all data points.

time-space: [ f ( X (st) t )] N t t which is ordered by stacking each of the input points at each time point on top of each other. This is only applicable when there is a single latent function.

latent-time-space: [ F 1 ( X (st) 1 ) , · · · , F 1 ( X (st) N t ) , · · · , F Q ( X (st) 1 ) , · · · , F Q ( X (st) N t )]

The default order will be latent-data (and latent-time-space for spatio-temporal problems). Since all of these are just simple permutations of each other, there exists a permutation matrix that permutes between any two of the layouts above. We use the function π to denote a function that performs this permutation such that:

A.3 Timeseries Setting - Single Latent Function

Let X ∈ R N × D , Y ∈ R N × P be input-output observations across P outputs, where N = N t . For now, we only consider the case where Q = 1 . We assume that f has a state-space representation, and we denote its state with its D time derivatives as F ( X ) = [ f ( X ) , ∂ f ( X ) ∂ X , ∂ 2 f ( X ) ∂ X 2 , · · · ] in latent-data format. The vector F ( X ) is of dimension ( N × D ) . We also use the notation F n = F ( X n ) , which is a D -dimensional vector of the derivatives at location X n . The joint model is

At this point, we place no particular restriction on the form of the likelihood, aside from decomposing across N . The prior p ( F ) is a multivariate GP of dimension N × D

Let q ( F ) be a free-form multivariate Gaussian of the same dimension as p ( F ) then the corresponding ELBO is:

and the marginal q ( F n ) is a D -dimensional Gaussian corresponding to the n 'th observation. The natural gradients are

where ˜ λ = [ [ ˜ λ ] 1 , [ ˜ λ ] 2 ] ⊤ and [ ˜ λ ] 1 is an ( N × D ) vector and [ ˜ λ ] 2 an ( N × D ) × ( N × D ) matrix. Eqn. (29) is a sum of natural parameters, and so is the conjugate Bayesian update. Naively computing this would yield no computation speed up as the computation cost would be cubic O ( N 3 ) . However, the natural parameters of the likelihood ( ˜ λ ) are guaranteed to be block diagonal, one block per data point (if ˜ λ 0 is initialised as so). This immediately implies that Eqn. (29) can be computed using efficient Kalman filter and smoothing algorithms. The structure of ˜ λ depends on the gradient of the expected log-likelihood ∂ ELL ∂ µ . Expanding this out

where each component ˜ µ n is a ( N × D ) × ( N × D ) matrix that only has D × D non-zero entries; as these are the only elements that directly affect F n . Collecting all these submatrices into a block diagonal matrix, we have a matrix in data-latent format, however, ∂ ELL ∂ µ is in latent-data, and so all we need to do is permute by P :

Converting from natural to moment paramerisation the surrogate update is:

where ˜ Y n is a D -dimensional vector, and ˜ V n is a D × D matrix, and efficient Kalman filtering and smoothing algorithms can be used to compute the surrogate model update. Substituting q ( F ) back into the ELBO it further simplifies:

each term can be computed efficiently as the by-product of the Kalman filtering and smoothing algorithm used to compute q ( F ) .

A.4 Timeseries Setting - Multiple Latent Functions

We now generalise the previous section to handle multiple independent latent functions, i.e. Q > 0 . The model prior now has the form

where p ( F q ) is a prior over f q, 1 and its D partial deriatives. We consider two approaches: a mean-field approximate posterior and a full Gaussian.

The first approach defined mean-field approximate posterior q ( F ) ≜ = ∏ Q q q ( F q ) where each q ( F q ) is a free-form Gaussian of dimension ( N × D ) . The natural gradient updates are now simply applied to each component q ( F q ) separately, and we essentially follow the update set out in App. A.3.

The second approach is a full-Gaussian approximate posterior where q ( F ) is a (Q × D × N ) -dimensional free-form Gaussian. In this case the ELL is

where q ( F n ) is of dimension (Q × D ) . This implies that the gradient of the ELL ∂ ELL ∂ [ µ ] 2 = ∑ N n ˜ µ n where ˜ µ n now has (Q × D ) × (Q × D ) non-zero entries. Switching to moment parameterisation

where ˜ Y n is of dimension (Q × D ) and ˜ V n is (Q × D ) × (Q × D ) . We can still use state-space algorithms by simply stacking the states corresponding to each F q [56].

A.5 Spatio-temporal Data - Single Latent Function

We now turn to the spatio-temporal setting where X , Y are spatio-temporal observations on a spatiotemporal grid ordered in time-space format. We now derive the conjugate variational algorithm for PHYSS-SVGP and PHYSS-SVGPH. The algorithms for PHYSS-GP and PHYSS-VGPH are recovered as special cases when Z = X s .

A.5.1 Spatial Derivative Inducing Points

We follow the standard sparse variational GP procedure and augment that prior with inducing points U = D u at locations Z . We require that the inducing points are defined on a spatial-temporal grid at the same temporal points as the data X , such that Z = [ Z t ] N t t . This is required to ensure Kronecker structure between the inducing points and the data. The joint model is

where and the conditional mean and covariance are given by

and

where K ⊗ X , Z = [ K D t ( X t , X t ) ⊗ K D s ( X s , Z s ) ] which simplifies to

Due to the Kronecker structure, the marginal at time t only depends on the inducing points in that time slice so we can still get a CVI-style update that can be computed using a state-space model. To see why we again look at the Jacobian of the ELL: ∂ ELL ∂ [ µ ] 2 = ∑ N n ˜ µ n where ˜ µ n now has ( D × M ) × ( D × M ) non-zero entries, which corresponding to needed all M spatial inducing points with there derivatives to predict at a single time point. This is similar to the time series setting, except we have now predicted in space to compute marginals of q ( F ) . To be complete, we write that the marginal q ( U ) is

where ˜ Y t and F t are vectors of dimension ( D × M ) , and ˜ V t is a matrix of dimension ( D × M ) × ( D × M ) . The marginals q ( U t ) , and the corresponding marginal likelihood p ( ˜ Y | ˜ V ) can be computed by running a Kalman filter and smoother in O ( N t · ( M s · d s · d ) 3 ) . The marginal q ( F ) = N ( F | µ F , Σ F ) where

and

A.5.2 Structured Approximate Posterior With Spatial Inducing Points

We now derive the algorithm for the case of the structured approximate posterior with spatial inducing points. The key is to define the free-form approximate posterior over the inducing points and their temporal derivatives and then use the model conditional to compute the spatial derivatives. The model is

Each term is where

and

where ˜ K ⊗ X , Z s = K D t ( X t , X t ) ⊗ ˜ K D s ( X s , Z s s ) and

The approximate posterior is defined as

where q ( D t u is a free-form Gaussian of dimension ( Nd × N t × M s ) . The rest of the derivation simply follows App. A.5.1 by simpling substituting ˜ K D s ( X s , Z s s ) into the corresponding conditionals. The final result is that the approximate posterior decomposes as

where ˜ Y t and F t are vectors of dimension ( d t × M s ) , and ˜ V t is a matrix of dimension ( d t × M s ) × ( d t × M s ) . The marginals q ( U t ) , and the corresponding marginal likelihood p ( ˜ Y | ˜ V ) can be computed by running a Kalman filter and smoother in O ( N t · ( M s · d ) 3 ) , which compared to App. A.5.1 is not cubic in the number of spatial derivatives.

A.5.3 Gauss-Newton Natural Gradient Approximation

We now provide the full derivation of the Gauss-Newton approximation of the natural gradient used to ensure p.s.d updates. We will make use of the following identities, known as the Bonnet and Price theorems (see, [38]),

which describes how to bring derivatives inside expectations. To ease notations, we work with a more general description of the model presented in the main paper, where we have multiple independent latent functions and use T p to denote likelihood-specific functions which, for example, can be used to represent DE or as the identity of standard Gaussian likelihoods. The model is

where the shapes are u q ∈ R M , f q ∈ R N , T p : R Q → R P , Y ∈ R N × P , and Y n,p ∈ R . The variational approximation is

where U = [ u 1 , · · · , u Q ] , m ∈ R QM × 1 and S ∈ R QM × QM . Let F = [ f 1 , . . . , f Q ] . The expected log-likelihood of the variational approximation is

where k = t ( n ) is the time period associated with data X n . Here U k are the spatial inducing points at time t ( n ) and hence U k ∈ R QM s . We need to compute

where the shapes are with

and P k is a permutation matrix that permutes from data-latent to latent-data format. In implementation, we do not need to perform this permutation as we only require the blocks ∂ ELL ∂ S k but write it here for completeness.

A.5.4 Optimality of Natural Gradients In Linear Models

Theorem A.1. Consider a linear multi-task model of the form

then under a full Gaussian variational approximate posterior

where m ∈ R ( N × Q ) × 1 , S ∈ R ( N × Q ) × ( N × Q ) then the natural gradient update with a learning rate of 1 recovers the optimal solution p ( ˜ f | Y ) .

Proof. To prove this we first derive the natural parameters of the posterior p ( ˜ f | Y ) . We then derive the closed form expression of the natural parameter update and show that they recover that of the posterior. Let

where ˜ K X , X = blkdiag ( [ K q ] Q q =1 ) and ˜ W = W ⊗ I then the posterior p ( F ( X ) | Y ) is Gaussian

The covariance matrix can be simplified by invoking Woodbury's identity twice

and the mean can be expressed as

Now we can immediately read off the posterior p ( ˜ f | Y ) as p ( F ( X ) | Y ) = p ( ˜ W ˜ f | Y ) is simply a transformed version

whose natural parameters are

We now derive the closed form expression of the natural gradient update with a learning rate of 1 , and show that it recovers λ ˜ f | Y . The expected log likelihood (ELL) is

The required derivatives are

where the last follows because Φ is symmetric and

The natural gradient is now given as

The natural gradient update with a learning rate of 1 is

where λ p ( ˜ f ) = [ 0 , -1 2 K -1 ] ⊤ are the natural parameters of the prior p ( ˜ f ) , hence after the update the natural parameters are

which recover those of p ( f | Y ) , and hence we recover the optimal posterior.

˜

B Further Experimental Details and Results

EKS methods were run on CPUs. State-space methods running on GPU used the parallel form of the Kalman smoother (see [55, 20]).

B.1 An extension of AUTOIP

If one drops the requirement for state-space representations then the approximations proposed in Sec. 4 directly define approximations to the variational GP defined by Eqn. (13), and hence directly extend AUTOIP. For example on the non-linear damped pendulum in Sec. 7 we run this extension of AUTOIP with whitening and 50 inducing points for C = 1000 and achieve an RMSE of 0 . 06 ± 0 . 001 and running time of 158 . 16 ± 0 . 34 , clearly improving the running time against AUTOIP. However the benefit of our methods is that PHYSS-GP remains linear in temporal dimensions which is vital for applications that are highly structured in time [20].

B.2 Modelling Unknown Physics

Modelling of missing physics can be handled by parameterising unknown terms with GPs. For example take a simple non-linear pendulum

Now consider that the the sin( θ ) is unknown and we would like to learn it. If we define the our differential equation in Eqn. (4) as

where both f 1 ( · ) , f 2 ( · ) are latent GPs that we wish to learn. We now construct 300 observations for training from the solution of Eqn. (75) across the range [0 , 30] and 1000 for testing. We run PHYSS-GP and compare the similarity of the learnt latent GP f 2 ( · ) to the true function at the test locations and achieve an RMSE of 0 . 068 indicating we have recovered the latent force/unknown physics well.

B.3 Monotonic Timeseries

This first example showcases the effectiveness of PHYSS-GP in learning monotonic functions. Monotonicity information is expressed by regularising the first derivative to be positive at a set of collocation points [51]:

where Φ( · ) is a Gaussian cumulative distribution function, and v = 1 e -1 is a tuning parameter that controls the steepness of the step function. We plot predictive distributions of (batch) GP and PHYSS-GP in Fig. 5. The GP fits data and does not learn a monotonic function. However, using 300 collocation points, PHYSS-GP is able to include the additional information and learn a monotonic function whilst running 1 . 5 times faster.

B.4 Non-linear Damped Pendulum

All models were run using an Nvidia Titan RTX GPU and an Intel Core i5 CPU. All were optimised for 1000 epochs using Adam [32] with a learning rate of 0 . 01 . Both the GP and AUTOIP had an RBF kernel (following Long et al. [40]) and PHYSS-GP used a Matérn- 7 /2; all with a lengthscale of 1 . 0 . The observation noise was initialised to 0 . 01 and the collocation 0 . 001 . Both were fixed for the first 40% of training and then released. Predictive distribution of PHYSS-GP and AUTOIP are plotted in Fig. 6.

B.5 Curl-free Magnetic Field Strength

All models were run using an Nvidia Titan RTX GPU and an Intel Core i5 CPU. All models are run for 5000 epochs using Adam with a learning rate of 0 . 01 , and use a Matérn- 3 /2 kernel on time,

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

0

25

50

75

100

= 10

-

1

0

1

2

-

1

0

1

2

PHYSS-GP- C

= 1000

-

1

0

1

2

AUTOIP- C

= 10

0

5

10

15

20

25

30

-

1

0

1

2

AUTOIP- C

= 1000

PHYSS-GP- C

with ARD RBF kernels on the spatial dimensions, with a lengthscale of 0 . 1 across all. The Gaussian likelihood is initialised with a variance of 0 . 01 and held for 40% of training. All our methods used a natural gradient learning rate of 1 . 0 as this is the conjugate setting.

B.6 Diffusion-Reaction System

We use data provided by [49] under an MIT license. All models were run using an Nvidia Titan RTX GPU and an Intel Core i5 CPU. Our method PHYSS-SVGP and PHYSS-SVGPH use a Matérn 72 kernel on time and an RBF of space, both initialised with a lengthscale of 0 . 1 . We place the collocation points on a regular grid of size 20 × 10 and use M s = 20 spatial inducing points. We pretrain for 100 iterations using a natural gradient learning rate of 0 . 01 and after use a learning rate 0 . 1 for the remaining 19000 iterations. AUTOIP uses a RBF kernel on both time and space with a lengthscale of 0 . 1 . We place the collocation points on a regular grid of size 10 × 10 . All models use Adam with a learning rate 0 . 001 and train for a total of 20000 iterations.

B.7 Ocean Currents

Our method PHYSS-SVGPH was run using an Nvidia Titan RTX GPU and an Intel Core i5 CPU. We ran for 10000 iterations, using Adam with a learning rate of 0 . 01 . For natural gradients with used a learning rate of 0 . 1 . We used a Matérn- 3 /2 kernel on time and RBF kernels on both spatial dimensions with lengthscales [24 . 0 , 1 . 0 , 1 . 0] . We used 100 spatial inducing points and a spatial mini-batch size of 10 .

NeurIPS Paper Checklist

The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.

Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:

You should answer [Yes] , [No] , or [NA] .

[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.

Please provide a short (1-2 sentence) justification right after your answer (even for NA).

The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include it (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.

The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation. While "[Yes] " is generally preferable to "[No] ", it is perfectly acceptable to answer "[No] " provided a proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] " or "[NA] " is not grounds for rejection. While the questions are phrased in a binary way, we acknowledge that the true answer is often more nuanced, so please just use your best judgment and write a justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.

IMPORTANT, please:

Delete this instruction block, but keep the section heading 'NeurIPS paper checklist" ,

Keep the checklist subsection headings, questions/answers and guidelines below.

Do not modify the questions and only use the provided macros for your answers .

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We have both theoretically (Theorem 3.1), and empirically demonstrated our claims (Sec. 7). We provide computational complexities for all proposed models showing a linear in the temporal dimension performance.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: A discussion of two main limitations (collocation method, and spatial scaling) and future work is provided in the conclusion.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: The assumptions of Theorem 3.1 are provided in Sec. 3 with full proof (correct to the best of our knowledge) provided in the appendix App. A.5.4.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Experimental details, sufficient to reproduce the results, are provided both in the main paper Sec. 7 and the appendix App. B. Code reproducing all methods and results is provided in the supplementary material.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: Code for data downloading and processing for train test splits, as well as code reproducing all methods and results is provided in the supplementary material.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: All required details are provided in App. B.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: We followed established published results for experimental setups, which did not include error bars.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Specific details on the CPUs and GPUs used for experiments are provided in App. B.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We have reviewed the NeurIPS Code of Ethics and we conform with this in every respect.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We discuss the benefits of physics informed machine learning in our introduction. We do not envisage any negative societal implications.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: All existing methods and datasets used are open and cited throughout. Any licenses are explicitly stated in the appendix.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: We release our code under CC-BY 4.0.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a method for training spatio-temporal Gaussian processes which incorporate physics constraints including satisfying the governing equations at a number of collocation points and satisfying curl / divergence free constraints. Their approach scales both linearly in time and space by leveraging a number of approximations.\n\nSTRENGTHS:\n- While many of the individual components (such as incorporating curl / divergence free constraints and satisfying governing equations at a number of collocation points) have been done before, I believe the combination of ideas here is novel. \n- Showing how to achieve linear complexity in the space and time dimension is valuable.\n- Showing how your approach recovers some prior works as a special case from a more general perspective is also valuable.\n- The bevy of test cases demonstrating the efficacy of your approach on both synthetic and real-world problems convincingly demonstrate the advantages offered by your approach.\n\nWEAKNESSES:\n- The problem statement is poorly motivated in my perspective. It would be helpful to be more specific about what you hope to achieve with hybrid modeling. For example, is your goal to improve the predictive accuracy of mechanistic models by incorporating data? What computational efficiency do you hope to improve? i.e. do you want to reduce the amount of data needed to train physics surrogates? Do you argue your approach is more efficient than solving the mechanistic model using traditional methods? etc. While I understand space is limited, being more specific about the problem statement will also help guide more targeted numerical studies in future works.\n- I think you need slightly more of a discussion on virtual observations (see questions below). When virtual observations are competing with actual data it would be helpful to discuss in detail how you think about $\\epsilon_C$ and choosing the number of collocation points.\n- Since you state that one of the goals of your approach is to be useful in uncertainty quantification, the numerical studies would have been strengthened by comparing more than just RMSE. For example, comparing the continuous ranked probability score (CRPS) would have given an indication of how well your approach is estimating uncertainty.\n- Minor:\n  - L94 space cases -> special cases?\n\nQUESTIONS:\n- As I understand it, you can always choose to satisfy the differential equations at enough collocation points such that you overwhelm the data. For example, say I have very limited data so I choose the number of collocation points to be much greater than the number of data points. Clearly, if I use too many collocation points, my model will ignore the data and just follow the differential equations. How do you think about this perspective in the context of uncertainty quantification?\n- As a follow up to this previous point, what are the implications of assuming that $0_n^{(C)} = g(F_n) + \\epsilon_C$? Is it correct to think about this as implicitly assuming that the PDE is stochastic? \n- Are there situations where you believe the Gaussian process assumption could be limiting?\n\nLIMITATIONS:\nI think the authors have done a good job of identifying some potential limitations. I think the paper would greatly benefit from a more in depth discussion on using collocation points to enforce governing equation constraints and the limitations this perspective brings.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a formulation for state-space GP which aims to capture the behaviour of PDE-based systems. The paper suggests multiple techniques to speed up the GP inference, including through decomposed kernel, variational methods on natural parameterisation of distribution, use of inducing points\n\nSTRENGTHS:\nThe paper provides good motivation and use cases for physics-informed GPs, and the need for scaling the inference process. The paper also provides adequate results and demonstrations of the GP for prediction and uncertainty quantifications, which well motivates the benefits of GPs over other solver types.\n\nWEAKNESSES:\n- The paper may be a bit hard to follow if one is not familiar with GPs already. For example, it may be difficult to immediately see how the PDE or the BV are incorporated into the probability distribution during the inference stage.\n- It may also be arguable that some parts of the paper can be seen as just application of standard sparse GP techniques and variational methods into the context of physics-informed GPs.\n\nQUESTIONS:\n- Are there any intuition as to why the decomposition of the kernel is done between the spatial and the temporal variables? Is it for computational reasons or can it be linked to properties of physical phenomena?\n- Does (or can) the method perform in the case that the full form of the PDE is unknown? Or is the paper mostly focused on cases of just generating the PDE solution? \n- In the formulation given by Eq 6, the PDE and BC residuals are assumed to arise from a Gaussian distribution. How can they be related to the physical realisation of the PDE, or how can they be interpreted properly as a probability distribution? How realistic is this assumption? \n- Are there any additional assumptions in the form of g(f) should take for this method to work?\n\nLIMITATIONS:\nLimitations are adequately mentioned.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a novel approach for solving partial differential equations (PDEs) using a Gaussian process prior. It leverages established methods for approximate inference and provides a cohesive framework that unifies existing results in probabilistic differential equation solving.\n\nSTRENGTHS:\nThe paper introduces a novel approach for solving partial differential equations (PDEs) using a Gaussian process prior. It leverages established methods for approximate inference and provides a cohesive framework that unifies existing results in probabilistic differential equation solving.\n\nWEAKNESSES:\nThe paper’s presentation is challenging to follow. Despite my familiarity with continuous-time filtering, smoothing, Gaussian process (GP) literature, and continuous-time stochastic optimal control (where some of these partial differential equations (PDEs) appear), I struggle to determine the paper’s solidity. Numerous mathematical inaccuracies and inconsistencies hinder comprehension. For instance, the abrupt switch between notation for space variables (e.g., $s$ and $t$) and the combined space-time variable ($x$) lacks clear definition. While I recognize that machine learning sometimes necessitates complex notation, in this case, it impedes understanding.\n\nThe main model section exacerbates the issue. Equations 5 and 6 remain elusive; the dimensions of $Q$ and $P$ are unspecified. The definition of $F_n$ confuses me, as it prevents a sensible matrix product computation. Additionally, the collocation point definition remains unclear. Even in Example 3.1, the meaning of $g_k$ remains unknown to me. Consequently, assessing the paper’s solidity proves challenging.\nIn my view, this paper requires substantial revision before acceptance. Although I appreciate the effort invested, from a reviewer’s standpoint, the current presentation poses significant hurdles.\n\nQUESTIONS:\n- In the background section please define N and F. And the relations between al the other variables N_t, N_s x_{t,s}, y_{t,s}, d, etc. I find this very confusing – is there also a clash in notation?\n-\tIn Eq (3) can you elaborate if $\\mathbf f$ is multidimensional and how does it relate to $f$. This notation is again used in line 78.\n-\tIn line 66 is this definition correct? Somehow I have a hard time coming up with the correct dimensions.\n-\tIn line 70 $\\bar{\\mathbf f} $ is a time depdent function. But in line 78 it is a space time depdent function. Could you elaborate on this?\n-\tIn Line 87 is the non-linear differential operator only dependent on space derivatives, or can the also be higher order time derivatives?\n-\tEq (5): $\\bar f_q(\\mathbf X_n)$ should be a vector of dimensionality D, but in line 98 W is of dimensionality $PD \\times QD$. Can you elaborate on this.\n-\tIn eq(7) can you elaborate what $g(\\mathbf F_n)$ is? Is it an element-wise operation for each element in $\\mathbf F_n$?\n-\tIn line 265-266 can you please elaborate what $z$ is?\n\nLIMITATIONS:\n-", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 1, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors present a physics-informed Gaussian process based approach to learn the solution of ODE and PDE systems. In particular, they address the challenge of the cubic computational complexity with respect to the number of spatial observations. It is shown that multiple state-of-the-art approaches can be recovered as special cases of the presented approach, i.e., it is a unifying framework. With additional approximation techniques, the cubic spatial computational cost is reduced to linear. In multiple simulations, the proposed method achieves similar or better results compared to SOTA methods but with a significant reduced computational time.\n\nSTRENGTHS:\n- The proposed method demonstrates a significant improvement of the computational complexity with respect to the number of spatial observations compared to SOTA methods. \n- The beauty and originality of the proposed model is its unifying property, including existing approaches such as HelmholzGP and AUTOIP.\n- The techniques and ideas in the paper are clearly presented. I really enjoyed reading the paper.\n\nWEAKNESSES:\n- There are some typos such as line 103 “develop sate space algorithms for”, line 167 “complexity of O(N  (N_s” (missing t), and eq (17) Nt instead of N_t. Furthermore, I think that line 257  “RMSE of all models increases as the number of collocation points increases” make no sense as the RMSE is decreasing. Please check the paper carefully.\n- In the experimental section, some phenomena are rarely discussed. For instance, the RMSE of PHYSS-GP is increasing with 1000 collocation points.\n\nQUESTIONS:\n- I assume that there exist PINN based approaches with some form of uncertainty quantification. It would be interesting to compare the GP-based method to NN-based approaches (not necessarily for this paper but in general). Do you have any thoughts on that?\n- Is it possible to use the same approximations in AUTOIP to reduce the cubic computational cost? If so, is there still a benefit of the proposed method?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe submission discusses conditioning spatiotemporal Gaussian processes using differential equation constraints and observational data.\nSpecifically, the temporal component is handled by a Markovian prior (to achieve linear complexity), and the spatial component is dealt with by variational methods.\nAs such, the proposed algorithm extends the work by Hamelijnck et al. [18] to differential-equation constraints (and to handle spatial mini-batching); or, from the opposite perspective, it implements a version of a probabilistic numerical solver for spatiotemporal PDEs with variational methods (in space).\nThe resulting algorithm is evaluated on a damped pendulum ODE, a curl-free magnetic strength field, a one-dimensional reaction-diffusion PDE, and an ocean-current problem.\n\nSTRENGTHS:\nAll components are technically sound, and the submission fits into recent literature on dynamical systems, Gaussian processes, and physics-informed machine learning.\nOverall, I think this is a nice paper, and despite a few minor weaknesses (see \"Weaknesses\"), I recommend acceptance.\n\nWEAKNESSES:\nI identify two weaknesses:\n\n1. The proposed algorithm's scientific novelty as a combination of known techniques is limited. Hamelijnck et al. essentially cover spatiotemporal variational inference with a Markovian prior in the temporal dimension. Conditioning Gaussian processes on differential equation constraints via collocation has become standard practice in recent years. Berlingheri et al. developed the curl- and divergence-free spatial priors. The combination of these techniques is novel, but the increment to the existing literature is relatively small.\n2. The clarity of the submission's relation to existing approaches would improve by discussing more articles related to probabilistic numerical methods. Currently, the manuscript cites the PMM by Cockayne et al., the ODE (initial value problem) solvers by Schober et al., Krämer et al., and Tronarp et al., and the GP-PDE paper by Pförtner et al. (all references are in the submission). Beyond those papers, Krämer et al. (below) describe solving time-dependent, nonlinear PDEs with collocation and Markovian priors; Schmidt et al. (below) combine nonlinear collocation constraints with observational data (for ODEs); and Krämer and Hennig (below) solve boundary value problems with collocation. There are other related papers, but I believe the former articles should be discussed in the manuscript. Further, the literature on statistical finite element methods is related and should be acknowledged: see Duffin et al. (below). \n\n\nThe first weakness is more significant than the second one. I expect that the second weakness will be relatively straightforward to resolve. \nIn any case, I believe the strengths outweigh the weaknesses and lean towards recommending acceptance.\n\n\n**References:**\n\nSchmidt, Jonathan, Nicholas Krämer, and Philipp Hennig. \"A probabilistic state space model for joint inference from differential equations and data.\" Advances in Neural Information Processing Systems 34 (2021): 12374-12385.\n\nKrämer, Nicholas, Jonathan Schmidt, and Philipp Hennig. \"Probabilistic numerical method of lines for time-dependent partial differential equations.\" International Conference on Artificial Intelligence and Statistics. PMLR, 2022.\n\nKrämer, Nicholas, and Philipp Hennig. \"Linear-time probabilistic solution of boundary value problems.\" Advances in Neural Information Processing Systems 34 (2021): 11160-11171.\n\nDuffin, Connor, et al. \"Statistical finite elements for misspecified models.\" Proceedings of the National Academy of Sciences 118.2 (2021): e2015006118.\n\nQUESTIONS:\nNone\n\nLIMITATIONS:\nDiscussed.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}