\section{Introduction}
\label{sec:intro}

Many areas of environmental science seek to predict a space-time variable of interest from observations at scattered points in the space-time field of study. Among modern techniques proposing efficient methods for estimation and prediction in a spatio-temporal framework, there is a clear distinction between two possible ways of constructing and treating spatio-temporal models \citep{wikle2010}: either one follows the traditional geostatistical paradigm, using joint space-time covariance functions (see for example \citet{cressie99}, \citet{gneiting2002}, \citet{stein2005}), or one uses dynamical models, by combining time series and spatial statistics (see for example \citet{wikle99}, \citet{sigrist2012}).

While the theoretical aspects of spatio-temporal geostatistics show good progress \citep{cressie2011}, implementations lack behind. The geostatistical paradigm can be computationally expensive for large spatio-temporal datasets, due to the factorization of dense $(N_SN_T,N_SN_T)$ covariance matrices, which is of the order of $\mathcal{O}((N_SN_T)^3)$, $N_S$ and $N_T$ being the number of points in space and time respectively. \citet{banerjee2014} called this issue the ``big $n$ problem''. Moreover, it is hard to define complex space-time covariance functions. For this reason, separable space-time covariance functions have often been applied to spatio-temporal models in order to take advantage of their computational convenience, even when they are not realistic in describing the processes due to the impossibility of allowing space-time interaction in the covariance. Recent studies have focused on constructing non-separable models, which are physically more realistic, albeit computationally more expensive. Non-separable space-time covariance models can be constructed from Fourier transforms of permissible spectral densities, mixtures of separable models, and partial differential equations (PDEs) representing physical laws \citep{chen2021,lindgren2022}. They can be fully symmetric or asymmetric, stationary or non-stationary, univariate or multivariate, and in the Euclidean space or on the sphere. See \citet{porcu202130} for a recent comprehensive review.

In this paper, we follow the dynamic approach that makes use of physical laws and study models which are defined through Stochastic Partial Differential Equations (SPDEs). The SPDE approach relies on the representation of a continuously indexed Gaussian Field (GF) as a discretely indexed random process, i.e. a Gaussian Markov Random Field (GMRF, see \citet{rue2005}). Passing from a GF to a GRMF, the covariance function and the dense covariance matrix are substituted respectively by a neighborhood structure and a sparse precision matrix. The advantage of using GMRFs is that the use of sparse precision matrices implies computationally efficient numerical methods, especially for matrix factorization. The link between GF and GMRFs in the purely spatial case has been pioneered by \citet{lindgren2011}, who proposed to construct a GMRF representation of the spatial Mat\'ern field on a triangulated lattice of the domain through the discretization of a diffusion SPDE with a Finite Element Method (FEM). We refer to \citet{bakka2018tuto} for a simple explanation of FEM applied to the spatial SPDE or to Section \ref{sec:discretization} for a detailed generalization to spatio-temporal SPDE.

In the spatial framework, major mathematical and algorithmic advances in the SPDE approach have been made \citep{pereira2019thesis, pereira2019, fulgstad2015}, making it possible to efficiently process very large datasets, even in the presence of non-stationarities and varying local anisotropy. The development of SPDE-based approaches to Gaussian processes has led to several practical solutions, among which we find the R package for approximate Bayesian inference R-INLA \citep{lindgren2015} that uses SPDEs to sample from spatial and separable spatio-temporal models.

When generalizing to the spatio-temporal framework, a direct space-time formulation of the SPDE approach was first suggested in \citet{lindgren2011}, without any precise detail on estimation and prediction methods. The SPDE approach was coupled with the Bayesian framework by \citet{cameletti2011} to provide a separable space-time model. Non-separable spatio-temporal models have been elaborated in \citet{krainski2019} and \citet{bakka2020diffusion} as a spatio-temporal generalization of the diffusion-Mat\'ern model of \citet{lindgren2011}.

In all the approaches overviewed above, the space-time processes are symmetrical in the sense that the spatio-temporal covariance does not change when the sign of the space and/or time lag changes. However, atmospheric and geophysical processes are often asymmetric due to transport effects, such as air and water flows. \citet{vergara2022general} defined new spatio-temporal models incorporating the physical processes linked to the studied phenomena (advection, diffusion, etc.). Problems relating to the estimation of the parameters and the conditioning to the observed data remained however open. \citet{sigrist2015} built non-symmetrical and non-separable space-time Gaussian as a solution to an advection-diffusion SPDE with computationally efficient algorithms for statistical estimation using fast Fourier transforms and Kalman filters. The applicability of this approach remains difficult however, especially in a non-stationary context or with scattered data, as it relies on the Fourier transform of the data. \citet{liu2020} extended the approach to spatially-varying advection-diffusion and non-zero mean source-sink, leading to a space-time covariance which is non-stationary in space.

In this wok, we propose an alternative approach for dealing with spatio-temporal SPDEs that include both a diffusion and an advection terms. In contrast to \citet{sigrist2015}, we make use of the sparse formulation of the spatio-temporal field which is the approximate solution of the SPDE obtained by a combination of FEM and finite differences. This sparse formulation allows to get fast algorithms for parameter estimation and spatio-temporal prediction. We also treat the case of an advection-dominated SPDE, by introducing the Streamline Diffusion stabilization term in the SPDE. We apply our method to a practical dataset of solar irradiance, where a transport effect due to wind is clearly present.

The paper is organized as follows: Section \ref{sec:spatiotemp_spde} first presents the background of the spatio-temporal SPDE approach, then defines the spatio-temporal advection-diffusion model developed in the paper and its discretization. Moreover, the stabilization of advection-dominated SPDEs is introduced. Section \ref{sec:estimation} explores fast and scalable estimation methods, kriging formula for prediction and conditional simulations. Section \ref{sec:application} presents an application of the proposed spatio-temporal SPDE approach to a solar radiation dataset. Section \ref{sec:discussion} discusses the advantages and the limitations of the approach and opens the way to further works on the subject.

\section{The spatio-temporal advection-diffusion SPDE and its discretization}
\label{sec:spatiotemp_spde}

\subsection{Background}
\label{sec:background_spde}

The spatial SPDE-based approach is a formalism introduced by \citet{lindgren2011} (from an idea of \citet{whittle54,whittle63}), that considers the solution $X(\mathbf{s})$, $\mathbf{s} \in \Omega\subseteq\mathbb{R}^d$, $d$ being the space dimension, of the SPDE
\begin{align}
(\kappa^2 - \Delta)^{\alpha/2} X(\mathbf{s}) = \tau W(\mathbf{\cdot}),
\label{eq:spatial_spde}
\end{align}
where $\Delta = \sum_{i=1}^d \frac{\partial^2}{\partial s_i^2}$ is the Laplacian operator and $W(\cdot)$ is a standard spatial Gaussian white noise defined as a (zero-mean) Generalized GRF with the property that the covariance measure of $W({\cdot})$ over two subregions $A, B \subseteq \Omega$ is equal to the area measure of their intersection, $\lvert A \cap B \rvert_\Omega$. In principle, $W(\cdot)$ has no element-wise definition, but for a sake of clarity, we will make an abuse of notation and write $W(\sbold)$.

\citet{whittle54,whittle63} showed that $X(\mathbf{s})$, solution to Equation \eqref{eq:spatial_spde}, is a spatial GF with Mat\'ern covariance
\begin{equation}
 C(h)=\frac{\sigma^2}{2^{\nu-1}\Gamma(\nu)}\left(\frac{h}{a}\right)^\nu \mathcal{K}_\nu\left(\frac{h}{a}\right),
\label{eq:matern_spatial}
\end{equation}
having regularity parameter $\nu=\alpha-d/2>0$, scale parameter $a=\kappa^{-1}$ and variance $\sigma^2=\tau^{2}(4\pi)^{-d/2}\Gamma(\nu) \Gamma(\nu+d/2)^{-1}\kappa^{-2\nu}$, and where $h=\norm{\mathbf{s}-\mathbf{s}'}$ is the distance between two spatial locations and $\mathcal{K}_\nu$ is the modified $2^\text{nd}$ order Bessel function. In particular, when $\nu=1/2$ we get the exponential covariance function and when $\nu\rightarrow +\infty$, after proper renormalization, we get the Gaussian covariance function.

In \citet{lindgren2011}, the smoothness parameter $\nu$ considered in the Mat\'ern covariance function corresponds to integer values of $\alpha$, hence $\nu = 0, 1, 2, \dots$ in $\bbR^2$. When non-integer values of $\nu$ are introduced in the modeling, the SPDE is said to be fractional. A review of results and applications of the fractional SPDE approach is available in \citet{bolin2019, roques2022}, but this case will not be treated further in this work.

When generalizing to spatio-temporal processes $X(\sbold,t)$, we consider the general framework proposed in  \citet{vergara2022general} for extending the SPDE approach to a wide class of linear spatio-temporal SPDEs. Let us denote $\xibold \in \bbR^2$ a spatial frequency and $\omega \in \bbR$ a temporal frequency. The space-time white noise with unit variance, denoted $W(\sbold,t)$, is characterized by its spectral measure $d\mu_W(\xibold, \omega) = (2\pi)^{-(d+1)/2} d\xibold d\omega$.
New spatio-temporal models were obtained from known PDEs describing physical processes, such as diffusion, advection, and oscillations with stochastic forcing terms. In particular, Theorem 1 in \citet{vergara2022general} provides necessary and sufficient conditions to the existence and unicity of stationary solutions to the following general SPDE
\begin{equation}
    \left[\frac{\partial^\beta}{\partial t^\beta} + \mathcal{L}_g\right] X(\sbold,t) = W_S(\sbold) \otimes W_T(t).
    \label{eq:evol_eq}
\end{equation}
In \eqref{eq:evol_eq}, the spatial operator $\mathcal{L}_g$ is defined using the spatial Fourier transform on $\bbR^2$ (denoted $\mathcal{F}_S$):
$$\mathcal{L}_g(\cdot)=\mathcal{F}_S^{-1}(g\mathcal{F}_S(\cdot)),$$
where  $g : \bbR^2 \to \bbC$ is a sufficiently regular and Hermitian-symmetric function called the \textit{symbol function} of the operator $\mathcal{L}_g$. The temporal operator $\frac{\partial^\beta}{\partial t^\beta}$ is
$$\frac{\partial^\beta}{\partial t^\beta}(\cdot)=\mathcal{F}_T^{-1}((i\omega)^\beta\mathcal{F}_T(\cdot)),$$
where $\mathcal{F}_T$ is the temporal Fourier transform on $\mathbb{R}$ and where we have used the symbol function over $\bbR$
$$\omega \mapsto (i\omega)^\beta = \lvert \omega\rvert^\beta e^{i\sgn(\omega)\beta\pi/2}.$$
The spatio-temporal symbol function of the operator involved in \eqref{eq:evol_eq} is thus
\begin{equation*}
    (\xibold, \omega) \mapsto (i\omega)^\beta + g(\xibold) = \lvert\omega\rvert^\beta \cos\left(\frac{\beta \pi}{2}\right) + g_R(\xibold) + i\left(\sgn(\omega)\lvert\omega\rvert^\beta \sin\left(\frac{\beta \pi}{2}\right) + g_I(\xibold)\right)
\end{equation*}
where $g_R$ and $g_I$ are the real and imaginary part of the spatial symbol function $g(\xibold)$. Theorem 1 and Proposition 3 in \citet{vergara2022general} state that \eqref{eq:evol_eq} admits a unique stationary solution if and only if $g_R$ is such that $\lvert g_R\lvert$ is inferiorly bounded by the inverse of a strictly positive polynomial and $g_R\cos\left(\frac{\beta \pi}{2}\right)\geq 0$.

\subsection{The spatio-temporal advection-diffusion SPDE}
\label{sec:model}
The advection-diffusion equation is a Partial Differential Equation (PDE) that describes physical phenomena where particles, energy, or other physical quantities evolve inside a physical system due to two processes: diffusion and advection. Advection represents the mass transport due to the average velocity of all molecules, and diffusion represents the mass transport due to the instantaneously varying velocity of individual molecules, compared to the average velocity of the fluid as a whole.

We here introduce the non-separable space-time models (with $\sbold \in \bbR^2$), derived from advection-diffusion SPDEs, i.e., advection-diffusion PDEs where a stochastic noise is added as a forcing term. The model is similar to the one presented in \citet{lindgren2011}, \citet{sigrist2015}, \citet{liu2020} and \citet{vergara2022general} and reads
\begin{equation}
    \left[\frac{\partial}{\partial t} + \frac{1}{c}(\kappa^2  - \nabla \cdot \Hbold\nabla)^{\alpha} + \frac{1}{c}\gammabold \cdot \nabla \right] X(\sbold,t) = \frac{\tau}{\sqrt{c}} Z(\sbold,t),
    \label{eq:adv_diff}
\end{equation}
where
\begin{itemize}
    \item the operator $\gammabold \cdot \nabla$ models the \textit{advection}, where $\gammabold=(\gamma_x,\gamma_y) \in \bbR^2$ is a velocity vector;
    \item the operator $\nabla \cdot \Hbold\nabla$ is a \textit{diffusion} term that can incorporate \textit{anisotropy} in the matrix $\Hbold$. When the field is isotropic, i.e., $\Hbold = \lambda\Ibold$, this term reduces to the Laplacian operator $\lambda\Delta$;
    \item $\kappa^2 >0$ accounts for \textit{damping};
    \item $c$ is a time-scale parameter;
    \item $\alpha$ is either equal to 1 in presence of diffusion or equal to 0 when there is no diffusion behavior;
    \item $\tau$ is a standard deviation factor;
    \item $Z$ is a stochastic forcing term.
\end{itemize}

Since Equation \eqref{eq:adv_diff} explicitly models phenomena such as transport and diffusion, the parameters can be given a physical interpretation if desired, but the use of the SPDE is not restricted to situations where it is a priori known that phenomena such as transport and diffusion occur \citep{sigrist2015}. The stochastic forcing term $Z(\sbold,t)$ is assumed separable with
\begin{align*}
Z(\mathbf{s},t) = W_T(t) \otimes  Z_S(\mathbf{s}),
\end{align*}
where $Z_S$ is a spatial Generalized Random Field and $W_T$ is a temporal white noise. $Z_S$ is often chosen to be a spatial white noise, denoted $W_S$ in this case. In order to ensure a given regularity for $Z$, $Z_S$ can alternatively chosen to be a \textit{colored noise}, such as for example the solution to the spatial Wittle-Matérn SPDE \citep{lindgren2011}
\begin{equation}
(\kappa^2 - \nabla \cdot \Hbold\nabla)^{\alpha_S/2} Z_S(\mathbf{s}) = W_S(\mathbf{s}).
\label{eq:col_noise}
\end{equation}
The spatial dependence of $Z_S$ ensures a desired regularity of the solutions, and is physically interpreted as a smoothing mechanism for introducing new mass into the system.

\begin{proposition}
\label{prop:spatial_trace_sep}
Let the coefficients of the SPDE \eqref{eq:adv_diff} be such that $\alpha=0$ and $\gammabold=\0bold$; the spatial operator applied to the spatio-temporal field $X(\sbold,t)$ is then the constant value $c^{-1}$. Let $Z(\sbold,t)$ be a spatio-temporal noise colored in space, with $Z_S(\sbold)$ satisfying \eqref{eq:col_noise}. If $\alpha_S>1$, the solution of the SPDE is a separable spatio-temporal field whose covariance is the product of an exponential temporal covariance (with scale parameter equal to $c$) and a Mat\'ern spatial covariance \eqref{eq:matern_spatial} with scale parameter equal to $a = \kappa^{-1}$, regularity parameter $\nu=\alpha_S-1$ and marginal variance equal to
$$\sigma^2 = \frac{\tau^{2}\Gamma(1/2)\Gamma(\alpha_S -1)}{\Gamma(1)\Gamma(\alpha_S)8\pi^{3/2}\kappa^ {2(\alpha_S-1)}\lvert\Hbold\rvert^{1/2}}.$$
\end{proposition}

In order to get an identifiable model, we will always set $\Hbold[1,1]=1$. Proposition \ref{prop:spatial_trace_sep} is a particular case of Corollary 3.3 of \citet{bakka2020diffusion}, hence the proof is not reported here.

When $\alpha=1$ and $\gammabold$ is non-null, $X(\sbold,t)$ is a non-separable spatio-temporal field.
The advection-diffusion equation \eqref{eq:adv_diff} is a particular first order evolution model as in Equation \eqref{eq:evol_eq} with $\beta=1$. Its spatial symbol function
$$g(\xibold) = \frac{1}{c}(\kappa^2  - \xibold^\top \Hbold \xibold + i\gammabold^\top\xibold),$$
verifies the necessary and sufficient condition for existence and uniqueness of a stationary solution recalled at the end of Section \ref{sec:background_spde}. The spatial behavior of this model is described by the spatial SPDE \citep{vergara2022general}
\begin{equation*}
\frac{\sqrt{2}}{c}(\kappa^2  - \nabla \cdot \Hbold\nabla)^{1/2} X_S(\sbold) = \frac{\tau}{\sqrt{c}}Z_S(\sbold).
\end{equation*}

\begin{proposition}
\label{prop:spatial_trace}
Let $Z(\sbold,t)$ be a spatio-temporal noise colored in space (such that $Z_S(\sbold)$ satisfies \eqref{eq:col_noise}), and let define $\alpha_{tot}=\alpha + \alpha_S$. If $\alpha_{tot}>1$, the spatial trace of the solution $X(\sbold,t)$ of the SPDE \eqref{eq:adv_diff} is a Mat\'ern field with covariance \eqref{eq:matern_spatial} with $a=\kappa^{-1}$, $\nu = \alpha_{tot}-1$ and marginal variance $\sigma^2$ equal to
\begin{equation}
	\sigma^2 = \frac{\tau^{2}\Gamma(1/2)\Gamma(\alpha_{tot}-1)}{\Gamma(1)\Gamma(\alpha_{tot})8\pi^{3/2} \kappa^{2(\alpha_{tot}-1)} \lvert\Hbold\rvert^{1/2}}.
	\label{eq:variance}
\end{equation}
\end{proposition}

A similar proposition, along with its proof, can be found in Proposition 3.1 in \citet{bakka2020diffusion}.

\subsection{Discretization}
\label{sec:discretization}

The advection-diffusion SPDE \eqref{eq:adv_diff} can be discretized in time and space using a Finite Difference Method (FDM) and a Finite Element Method (FEM), respectively. Since implicit solvers are usually less sensitive to numerical instability than explicit solvers, we choose the implicit Euler scheme for the temporal discretization. This choice prevents errors in the convergence of the algorithm. The FEM method for the spatial discretization is the Continous Galerkin method with Neumann Boundary Conditions. This choice is justified in detail in \citet{lindgren2011}.

First, the Implicit Euler scheme is applied to the temporal derivative in order to obtain an approximation of the temporal behavior defined at each time step of the discretization. Let us consider a triangulation $\cal T$ of the spatial domain $\Omega \subset \mathbb{R}^2$ with $N_S$ vertices $\{\sbold_1,\dots,\sbold_{N_S}\} \subset \mathbb{R}^2$ and define $h$ the size of the triangulation. Let then consider the purely spatial SPDE defined at each discretization time step. By writing its stochastic weak formulation, we construct a finite element representation $X_h$ of the solution of the spatial SPDE as a linear combination of piecewise linear basis functions $\{\psi_i\}_{i=1}^{N_S}$ equal to $1$ at the node $\sbold_i$ and $0$ at all the other nodes, and Gaussian distributed weights $\{x_i\}_{i=1}^{N_S}$, such that
$$X_h = \sum_{i=1}^{N_S} x_i\psi_i.$$
The weights determine the values of the field at the vertices, while the values in the interior of the triangles are determined by linear interpolation. The continuous Galerkin finite dimensional solution is obtained by finding the distribution for the representation weights that fulfills the stochastic weak SPDE formulation for the specific set of test functions equal to the basis functions, $\{\psi_i\}_{i=1}^{N_S}$.

Given the spatial white noise $W_S(\mathbf{s})$, for any set of test functions $\{\psi_i\}_{i=1}^{N_S}$, the integrals $$\int_\Omega \psi_i(\sbold) W_S(\mathbf{s})d\sbold \quad i=1,\dots,N_S$$
are jointly Gaussian, with expectation and covariance measures given by
\begin{eqnarray*}
	& \mathbb{E}\left(\int_\Omega \psi_i(\sbold) W_S(\mathbf{s})d\sbold\right)= 0,\\
	& \text{Cov}\left(\int_\Omega \psi_i(\sbold) W_S(\mathbf{s})d\sbold, \int_\Omega \psi_j(\sbold) W_S(\mathbf{s})d\sbold\right) = \int_\Omega \psi_i(\sbold)\psi_j(\sbold)d\sbold.
\end{eqnarray*}

\begin{proposition}
\label{prop:discr_adv_diff}
Let $X(\sbold,t)$ be the spatio-temporal process of Equation \eqref{eq:adv_diff} with $\alpha=1$ and $\Hbold=\Ibold$. Let $Z(\sbold,t)$ be a spatio-temporal white noise $W(\sbold,t)$. Let $\mathcal{T}$ be a triangulation of the spatial domain $\Omega$ of the process $X(\sbold,t)$. Let $\{\psi_i\}_{i=1}^{N_S}$ be the piecewise linear basis functions defined over $\mathcal{T}$. Finally, let $\Mbold$ (mass matrix), $\Gbold$ (stiffness matrix), $\Bbold$ (advection matrix) and $\Kbold$ be the $(N_S, N_S)$ matrices, with entries
\begin{align*}
    M_{ij}&= \int_\Omega \psi_i(\sbold) \psi_j(\sbold)d\sbold, \nonumber \\
    G_{ij}&= \int_\Omega  \nabla\psi_i(\sbold)\cdot \nabla\psi_j(\sbold)d\sbold, \\
    B_{ij}&= \int_\Omega  \gammabold\cdot\nabla\psi_i(\sbold)\psi_j(\sbold)d\sbold, \\
    (K_{\kappa^2})_{ij} &= \kappa^2 M_{ij} + G_{ij}.
\end{align*}
Then, the continuous Galerkin finite element solution vector $\xbold = \{x_i\}_{i=1}^{N_S}$, defined on the vertices of the triangulation $\mathcal{T}$ satisfies at each discretization time step $(t+dt)$
\begin{equation}
\left(\Mbold+\frac{dt}{c}(\Kbold+\Bbold)\right)\mathbf{x}_{t+dt}=\Mbold\mathbf{x}_{t}+\frac{\tau\sqrt{dt}}{\sqrt{c}} \Mbold^{1/2}\mathbf{z}_{t},
\label{eq:implicit_scheme}
\end{equation}
where $\mathbf{z}_t$ is a standard independent Gaussian vector, $\mathbf{z}_t \sim \mathcal{N}(\0bold,\Ibold)$, and $\Mbold^{1/2}$ is any matrix such that $\Mbold^{1/2}\Mbold^{1/2} = \Mbold$.
When the noise on the right-hand side is colored in space, the discretization reads
\begin{equation*}
	\left(\Mbold+\frac{dt}{c}(\Kbold+\Bbold)\right)\mathbf{x}_{t+dt}=\Mbold\mathbf{x}_{t}+\frac{\tau\sqrt{dt}}{\sqrt{c}} \Lbold_S^\top\mathbf{z}_{t},
\end{equation*}
where $\Lbold_S$ is the Cholesky decomposition of the discretized spatial precision matrix $\Qbold_{S}$ of the solution of the spatial SPDE $(\kappa -\nabla \cdot \Hbold \nabla)^{\alpha_S/2}X_S(\sbold) = W(\sbold)$, obtained with the Continuous Galerkin FEM \citep{lindgren2011}.
\end{proposition}

\begin{proof}
The proof is available in Appendix \ref{sec:A1}.
\end{proof}

We remark that the elements of the matrices $\Mbold$, $\Gbold$ and $\Bbold$ are non-zero only for pairs of basis functions which share common triangles. This implies that the matrix $(\Mbold+\frac{dt}{c}(\Kbold+\Bbold))$ is sparse and that Equation \eqref{eq:implicit_scheme} can be solved by Cholesky decomposition in an efficient  way.

\subsection{Stabilization of advection-dominated SPDE}
\label{sec:adv_domination}
When the advection term is too important with respect to the diffusion term, we obtain an advection-dominated SPDE. The advection-dominated flow is defined with respect to the P\'eclet number $\text{Pe}^h = \frac{\norm{\gammabold} h}{2\lambda}$, where $\lambda$ is the coefficient of the isotropic Laplacian operator: when $\text{Pe}^h > 1$, we are in presence of an advection-dominated flow. When the advection dominates the diffusion, it is well known that the stability of the Finite Element method with the continuous Galerkin method is unsatisfactory (see, for example, \citet{mekuria2016} or \citet{quarteroni2008}, Chapter 5). This is due to the fact that the advection term is non-symmetric, and causes the condition number of the matrix $\left[\Mbold + \frac{dt}{c}(\Kbold +\Bbold)\right]$ to increase, which leads to oscillations in the solution and may induce instability. A possible solution could be to refine the triangulation until the advection no longer dominates on the element-level. However, in many cases this is not a feasible solution because it would  be too much computer demanding. Several stabilization terms can be introduced, some of which are more accurate than others. A detailed explanation of the stabilization approach is reported in Appendix \ref{sec:A2}.

In our case, in the trade-off of order of accuracy and computational complexity, we opt for the Streamline Diffusion stabilization term, that stabilizes the extra advection by introducing an artificial diffusion term along the advection direction.

\begin{proposition}
\label{prop:discr_adv_diff_stab}
Let consider the same hypotheses as in Proposition \ref{prop:discr_adv_diff}. The introduction of the Streamline Diffusion stabilization implies that the discretization defined in Equation \eqref{eq:implicit_scheme} becomes
\begin{eqnarray}
\left(\Mbold+\frac{dt}{c}(\Kbold+\Bbold+\Sbold)\right) \mathbf{x}_{t+dt}=\Mbold\mathbf{x}_{t} + \frac{\tilde\tau\sqrt{dt}}{\sqrt{c}}\Mbold^{1/2} \mathbf{z}_{t},
\label{eq:implicit_scheme_stab}
\end{eqnarray}
where $\Sbold = [S_{ij}]_{i,j=1}^{N_S}$ is the matrix of the Streamline Diffusion stabilization operator $\mathcal{S}$, such that $S_{ij} = \mathcal{S}(\psi_i,\psi_j)=h\lvert\gammabold\rvert^{-1}\int_\Omega (\gammabold\cdot \nabla \psi_i )(\gammabold\cdot \nabla \psi_j) d\sbold$, and $\tilde\tau= \tau \left( \lvert\Hbold + h\gammabold\lvert\gammabold\rvert^{-1}\gammabold^\top \rvert \right)^{-1/4} \left( \lvert\Hbold\rvert \right)^{1/4}$ (see, for example, \citet{fulgstad2015}).
If the noise on the right-hand side of Equation \eqref{eq:adv_diff} is colored in space, the discretization reads
\begin{equation*}
\left(\Mbold+\frac{dt}{c}(\Kbold+\Bbold+\Sbold)\right) \mathbf{x}_{t+dt}=\Mbold\mathbf{x}_{t} + \frac{\tilde\tau\sqrt{dt}}{\sqrt{c}} \Lbold_S^\top \mathbf{z}_{t},
\end{equation*}
where $\Lbold_S$ is the Cholesky decomposition of the discretized spatial precision matrix $\Qbold_S$ of the solution of the spatial SPDE $(\kappa -\nabla \cdot \Hbold \nabla)^{\alpha_S/2}X_S(\sbold) = W(\sbold)$, obtained with the Continuous Galerkin FEM \citep{lindgren2011}. With this parametrization, the marginal variance of the spatial trace of the field is the same as the one defined in Equation \eqref{eq:variance} with $\tilde{\tau}$ instead of $\tau$.
\end{proposition}

The Streamline Diffusion operator $\mathcal{S}$ can be considered from another point of view as a perturbation of the original SPDE \citep{bank1990}. In fact, by making the classical hypothesis of Neumann boundary condition on $\Omega$ and by using the Green's first identity, we have that
$$\int_\Omega(\gammabold\cdot \nabla x )(\gammabold\cdot \nabla v) d\sbold = -\int_\Omega\nabla \cdot (\gammabold\gammabold^\top) \nabla x vd\sbold.$$
This means that the original SPDE \eqref{eq:adv_diff} can be rewritten with an additional diffusion term as
\begin{equation}
    \left[\frac{\partial}{\partial t} + \frac{1}{c} \left(\kappa^2 - \nabla \cdot \left(\Hbold + h\lvert\gammabold\rvert^{-1}\gammabold\gammabold^\top\right) \nabla + \gammabold \cdot \nabla \right)\right] X(\sbold,t) = \frac{\tilde\tau}{\sqrt{c}}Z(\sbold,t).
    \label{eq:diff_adv_sl_spde}
\end{equation}
The term $(h\lvert\gammabold\rvert^{-1}\gammabold\gammabold^\top)$ acts as an anisotropic “diffusion” tensor that is added to the anisotropy (or identity) matrix $\Hbold$ of the original diffusion. This extra diffusion stabilizes the advection directed along the direction $\gammabold$.

\subsection{Spatio-temporal Gaussian Markov Random Field approximation}
\label{sec:prec_mat}

\begin{proposition}
In presence of an advection-dominated flow and a spatio-temporal white noise on the right-hand side of Equation \eqref{eq:adv_diff}, the discretized vector $\xbold_{t+dt}$ on the mesh $\cal T$ at each time step can be found as the solution of the following equation:
\begin{align}
\mathbf{x}_0 &\sim \mathcal{N}(\0bold,\Sigmabold_0), \nonumber\\
\mathbf{x}_{t+dt}&=\Dbold\mathbf{x}_{t} + \Ebold \mathbf{z}_{t},
\label{eq:timestep}
\end{align}
where
\begin{eqnarray}
\Dbold & = & (\Mbold + \frac{dt}{c}(\Kbold + \Bbold + \Sbold))^{-1}\Mbold, \nonumber \\
\Ebold & = & \frac{\tilde\tau \sqrt{dt}}{ \sqrt{c}}(\Mbold + \frac{dt}{c}(\Kbold + \Bbold + \Sbold))^{-1}\Mbold^{1/2},
\label{eq:matrices_timestep}
\end{eqnarray}
and $\mathbf{z}_{t}\sim\mathcal{N}(\0bold,\Ibold)$ is independent of $\mathbf{x}_0,\dots,\mathbf{x}_t$. In presence of a spatio-temporal noise colored in space on the right-hand side of Equation \eqref{eq:adv_diff}, the matrix $\Ebold$ reads
$$\Ebold = \frac{\tilde\tau \sqrt{dt}}{ \sqrt{c}}(\Mbold + \frac{dt}{c}(\Kbold + \Bbold + \Sbold))^{-1}\Lbold_S^\top,$$
with $\Lbold_S$ defined in Proposition \ref{prop:discr_adv_diff_stab}.
\end{proposition}

\begin{proof}
Starting from Equation \eqref{eq:implicit_scheme_stab}, which represents the numerical scheme for the advection-diffusion spatio-temporal SPDE with stabilization, it is straightforward to obtain \eqref{eq:timestep}.
\end{proof}

When the SPDE is not advection-dominated (and hence no stabilization term is needed), Equation \eqref{eq:matrices_timestep} is replaced by the similar equation where the matrix $\Sbold$ is deleted and where $\tilde\tau$ is replaced by $\tau$.

The covariance matrix $\Sigmabold_0$ can be taken to be equal to any admissible positive definite matrix. The speed of convergence to the stationary solution of the equation will depend on the proximity of the covariance of the spatial trace of $X(\sbold,t)$ to $\Sigmabold_0$. When the noise is colored in space, we can set $\Sigmabold_0$ to be equal to the Mat\'ern covariance of the spatial trace defined in Proposition \ref{prop:spatial_trace}.

To make the precision matrix sparse to obtain GMRF models, $\Mbold$ should be replaced by the diagonal matrix $\tilde{\Mbold}$, where $\tilde{\Mbold}_{ii} = \langle \psi_i,1 \rangle$. This technique is called mass lumping and is common practice in FEM. From now on, we always use the lumped matrix $\tilde{\Mbold}$, but for ease of reading, it will still be denoted  $\Mbold$.

\begin{proposition}\label{prop:global_prec}
Let us denote $\xbold_{0:t}=[\mathbf{x}_0,\dots,\mathbf{x}_{t}]^\top$ the vector containing all spatial solutions until time step $t$,
the global precision matrix $\Qbold$ of the vector $\xbold_{0:t}$ reads
\begin{equation}
\Qbold = \begin{pmatrix}
\Sigmabold_0^{-1}+\Dbold^\top\Fbold^{-1}\Dbold   & -\Dbold^\top\Fbold^{-1}   & 0 & \dots & 0 \\
-\Fbold^{-1}\Dbold & \Fbold^{-1}+\Dbold^\top\Fbold^{-1}\Dbold & -\Dbold^\top\Fbold^{-1}  & \ddots & \vdots \\
\vdots & \ddots & \ddots & \ddots & 0 \\
\vdots & \ddots & -\Fbold^{-1}\Dbold & \Fbold^{-1}+\Dbold^\top\Fbold^{-1}\Dbold & -\Dbold^\top\Fbold^{-1} \\
0 & \dots  & 0 & -\Fbold^{-1}\Dbold & \Fbold^{-1}
\end{pmatrix},
\label{eq:prec_matrix}
\end{equation}
where $\Fbold=\Ebold \Ebold^\top$.
\end{proposition}

\begin{proof}
The proof is available in Appendix \ref{sec:A3}.
\end{proof}

\section{Estimation, prediction and simulation}
\label{sec:estimation}

This section explores the techniques for estimation and prediction of spatio-temporal processes following the SPDE approach described in Section \ref{sec:spatiotemp_spde}. We will consider the SPDE \eqref{eq:adv_diff} with $\alpha=1$, $\Hbold = \Ibold$ (isotropic diffusion) and colored noise in space with $\alpha_S=2$. Similar computations can be generalized to other values of $\alpha_S$ such that $\alpha_S/2$ is integer or to anisotropic diffusion.

We consider $n$ spatio-temporal data $\zbold$ scattered in the spatio-temporal domain $\Omega \times [0,T]$, discretized in space with a triangulation $\cal T$ with $N_S$ nodes and discretized in time by means of $N_T$ regular time steps. We shall denote this space-time discretization  ${\cal T}' = {\cal T} \times \{ 1/T,\dots,N_T/T\}$. We assume a statistical model with fixed and random effects. The fixed effect is a regression on a set of covariates and the random effect is modeled as the FEM discretization of a random field described by the SPDE \eqref{eq:adv_diff} with the addition of random noise:
\begin{equation}
  \zbold=\mathbf{\boldsymbol{\eta}}\bbold + \mathbf{A}^\top \xbold + \sigma_0 \varepsilonbold,
  \label{eq:stat_model}
\end{equation}
where $\bbold$ is the vector of $q$ fixed effects and $\boldsymbol{\eta}$ is a $(n,q)$ matrix of covariates with $[\boldsymbol{\eta}]_{jk} = \eta_k(\sbold_j,t_j)$, $j=1\dots,n$ and $k=1,\dots,q$. The matrix $\Abold$ is the $(N_SN_T,n)$ projection matrix between the points in ${\cal T}'$ and the data, and $\varepsilonbold$ is a standard Gaussian random vector with independent components.

\subsection{Estimation of the parameters}
\label{sec:estimation_scattered}

The parameters of the SPDE are estimated using Maximum Likelihood (ML).  We collect the parameters of the SPDE in the vector $\thetabold^\top= (\kappa, \gamma_x, \gamma_y, c, \tau)$, while the parameters of the statistical model are collected in $\psibold^\top = (\thetabold^\top,\bbold^\top,\sigma_0)$.
Following \eqref{eq:stat_model}, $\zbold$ is a Gaussian vector with expectation $\boldsymbol{\eta}\bbold$ and covariance matrix
$$\Sigmabold_{\zbold}=\Abold^\top \Qbold^{-1}(\thetabold)\Abold+\sigma_0^2 \Ibold_{(n,n)},$$
where $\Qbold(\thetabold)$ is a precision matrix of size $(N_SN_T, N_SN_T)$ depending on the parameters $\thetabold$. For ease of notation, we use $\Qbold$ instead of $\Qbold(\thetabold)$. The log-likelihood is equal to
\begin{equation}
\mathcal{L}(\psibold)=-\frac{n}{2}\log(2\pi)-\frac{1}{2}\log \lvert\Sigmabold_{\zbold}(\psibold)\rvert -\frac{1}{2}(\zbold-\boldsymbol{\eta}\bbold)^\top\Sigmabold_{\zbold}^{-1}(\psibold)(\zbold-\boldsymbol{\eta}\bbold).
\label{eq:loglike_latent}
\end{equation}

We use  the Broyden, Fletcher, Goldfarb, and Shanno (BFGS) optimization algorithm, that makes use of the second-order derivative of the objective function. The gradients of the log-likelihood function \eqref{eq:loglike_latent} with respect to the different parameters included in $\psibold$ are approximately computed with a Finite Difference Method.

We now propose an easier formulation of each term of the profile log-likelihood \eqref{eq:loglike_latent} to speed up the computations. Concerning the term $\log\lvert\Sigmabold_{\zbold}\rvert$ we derive the following property:

\begin{proposition}
In the framework outlined above, we have
\begin{equation}
    \log\lvert\Sigmabold_{\zbold}\rvert=n\log\sigma_0^2 -\log\lvert\Qbold\rvert+\log\lvert\Qbold+\sigma_0^{-2}\Abold\Abold^\top\rvert.
    \label{eq:logdet_sigmaz}
\end{equation}
\end{proposition}

\begin{proof}
To compute $\log\lvert\Sigmabold_{\zbold}\rvert$, let us consider the augmented matrix
\begin{equation}
\Sigmabold_c=\begin{pmatrix} \Qbold^{-1} & \Qbold^{-1}\Abold\\ \Abold^\top \Qbold^{-1} & \Sigmabold_{\zbold}\end{pmatrix}.
\label{eq:augmented_mat}
\end{equation}
By using block formulas, we have
$$\log\lvert\Sigmabold_c\rvert=-\log\lvert\Qbold\rvert+n\log\sigma_0^2,$$
and
$$\log\lvert\Sigmabold_c\rvert=\log\lvert\Sigmabold_{\zbold}\rvert+\log\lvert\Qbold^{-1}-\Qbold^{-1}\Abold\Sigmabold_{\zbold}^{-1}\Abold^\top \Qbold^{-1}\rvert=\log\lvert\Sigmabold_{\zbold}\rvert-\log\lvert\Qbold+\sigma_0^{-2}\Abold\Abold^\top\rvert,$$
where the last equality is a consequence of the Woodbury identity.
This leads to the result.
\end{proof}

\begin{proposition}
The term $\log\lvert\Qbold\rvert$ of Equation \eqref{eq:logdet_sigmaz} can be computed with the computationally cheap formula
\begin{equation}
    \log\lvert \Qbold\rvert = \log\lvert \Sigmabold_0^{-1}\rvert+(N_T-1)\log\lvert \Fbold^{-1}\rvert,
    \label{eq:logdet_Q}
\end{equation}
where $N_T$ is the number of time steps and  $\Fbold^{-1}=\frac{c}{\tilde\tau^2 dt}(\Mbold+\frac{dt}{c}(\Kbold+\Bbold+\Sbold))^\top \Mbold^{-1} (\Mbold+\frac{dt}{c}(\Kbold+\Bbold+\Sbold))$. $\Mbold^{-1}$ is replaced by $\Qbold_S$ when the noise is colored in space. $\lvert \Fbold^{-1}\rvert$ is the determinant of an $(N_S, N_S)$ sparse, symmetric and positive definite matrix. The computation of its determinant can be obtained by Cholesky decomposition of $\Fbold^{-1}$.
\end{proposition}

\begin{proof}
Following \citet{powell2011}, let $\Nbold_N = [\Nbold_ij]_{i,j=1}^N$ be an $(nN, nN)$ matrix, which is partitioned into $N$ blocks, each of size $(n, n)$. Then the determinant of $\Nbold_N$ is
$$\lvert \Nbold_N\rvert = \prod_{k=1}^N \lvert \alpha_{kk}^{(N-k)}\rvert,$$
where the $\alpha^{(k)}$ are defined by
\begin{align*}
\alpha_{ij}^{(0)} &= \Nbold_{ij} \\
\alpha_{ij}^{(k+1)} &= \alpha_{ij}^{(k)} - \alpha_{i,N-k}^{(k)} (\alpha_{N-k,N-k}^{(k)})^{-1} \alpha_{N-k,j}^{(k)},\quad k \geq 1.
\end{align*}
$\Qbold$ is a block-matrix organized as $\Nbold_N$. Hence, the formula for $\lvert \Qbold\rvert$ is
\begin{equation}
\lvert \Qbold\rvert= \lvert \Sigmabold_0^{-1}\rvert\lvert \Fbold^{-1}\rvert^{N-1}.
\label{eq:det_Q}
\end{equation}
Applying the logarithm, we obtain Equation \eqref{eq:logdet_Q}.
\end{proof}

The term $\log\lvert\Qbold+\sigma_0^{-2}\Abold\Abold^\top\rvert$ must be analyzed in detail. The term $\sigma_0^{-2}\Abold\Abold^\top$ is an $(N_SN_T,N_SN_T)$ diagonal block matrix, whose $(N_S,N_S)$ blocks are sparse. The computation of $\log\lvert\Qbold+\sigma_0^{-2}\Abold\Abold^\top\rvert$ is not straightforward as in the case of $\log\lvert\Qbold\rvert$, since there is no way of reducing the computation to purely spatial matrices. Depending on the size $N_SN_T$, we can either apply a Cholesky decomposition of the $(N_SN_T, N_SN_T)$ matrix $(\Qbold+\sigma_0^{-2}\Abold\Abold^\top)$ or a matrix-free approach that is detailed below.

When the number of mesh points prevents us from computing the Cholesky decomposition of $(\Qbold+\sigma_0^{-2}\Abold\Abold^\top)$, we can approximate $\log\lvert \Qbold+\sigma_0^{-2}\Abold\Abold^\top\rvert$ by expressing it as $\text{tr}[\log(\Qbold+\sigma_0^{-2}\Abold\Abold^\top)]$,  approximating the log function with a Chebyschev polynomial and using the Hutchinson's estimator \citep{hutchinson1990} to obtain a stochastic estimate of the trace of the matrix $[\log(\Qbold+\sigma_0^{-2}\Abold\Abold^\top)]$. The method is detailed in Algorithm 5 of \citet{pereira2022hal}.

Concerning the computation of the quadratic term of the log-likelihood, we note that using the Woodbury formula again, we can work with the more convenient expression
$$\Sigmabold_{\zbold}^{-1}=\sigma_0^{-2}\Ibold_{(n,n)}-\sigma^{-4}\Abold^\top(\Qbold+\sigma_0^{-2}\Abold\Abold^\top)^{-1}\Abold$$
that leads to the formula
\begin{eqnarray*}
(\zbold-\boldsymbol{\eta}\bbold)^\top \Sigmabold_{\zbold}^{-1}(\psibold) (\zbold & - & \boldsymbol{\eta}\bbold) = \sigma_0^{-2}(\zbold-\boldsymbol{\eta}\bbold)^\top \Ibold_{(n,n)}(\zbold-\boldsymbol{\eta}\bbold) + \\
& - & \sigma_0^{-4}(\zbold-\boldsymbol{\eta}\bbold)^\top \Abold^\top(\Qbold+\sigma_0^{-2}\Abold\Abold^\top)^{-1}\Abold(\zbold-\boldsymbol{\eta}\bbold),
\end{eqnarray*}
which is obtained by first computing the second term either by Cholesky decomposition or with the Conjugate Gradient method. This latter method solves $\Nbold \vbold=\wbold$ with respect to $\vbold$ and computes $\vbold_{sol} = \wbold^\top \vbold$, with $\Nbold = (\Qbold+\sigma_0^{-2}\Abold\Abold^\top)$ and $\wbold=\Abold(\zbold-\boldsymbol{\eta}\bbold)$. In this case, it is useful to find a good preconditioner for the matrix $(\Qbold+\sigma_0^{-2}\Abold\Abold^\top)$ to ensure fast convergence of the conjugate gradient method. A temporal block Gauss-Seidel preconditioner is a good choice in this case. A detailed explanation of the Conjugate Gradient method and the Gauss-Seidel preconditioner is available in Appendix \ref{sec:A4}.

In Table \ref{tab:ml_spatio_temp} we report some results of estimation of the parameters $\thetabold^\top = (\kappa, \gamma_x, \gamma_y, c, \tau)$ for a spatio-temporal model simulated with the SPDE \eqref{eq:adv_diff}. We set $\Hbold=\Ibold$, $\alpha=1$, $\alpha_S=2$ and the other parameters of the spatio-temporal discretization to the following: $N_S=900$, $dx=dy=1$, $dt=1$, $N_T = 10$ and $n_S=500$ observations in the spatial domain at the same locations for the $N_T$ time-steps (hence $n=5000$). Since the size of both the dataset and the spatio-temporal mesh is quite small, we report only the estimations computed with the Cholesky decomposition.

The initial values for the estimation have been set in the following way: first, a Matérn variogram with regularity parameter fixed at $\nu = \alpha +\alpha_S - 1=2$ is fitted to one (or more) spatial trace of data in order to set the value of $\kappa$ equal to $\kappa = \sqrt{12\nu}/r_M$, where $r_M$ is the practical range (the distance at which the correlation is approximately equal to 0.05), and the value of $\sigma^2$ equal to the sill. Then an exponential variogram is fitted to the temporal trace and $c$ is set to be equal to $c = r_E/3$, where $r_E$ is the practical range of the exponential variogram. In other words, we initialize the parameters of the SPDE model to the values obtained through the method of moments applied assuming the hypothesis of Proposition \ref{prop:spatial_trace}. Finally $\gammabold$ is set to $\gammabold=\0bold$, i.e., to the purely diffusive case.

\begin{table}[hbt]
\begin{adjustbox}{width=\columnwidth,center}
\begin{tabular}{cccccccccccc}
\toprule
$\kappa$ & $\hat{\kappa}$ & $\gamma_x$ & $\hat{\gamma}_x$ & $\gamma_y$ & $\hat{\gamma}_y$ & $c$ & $\hat{c}$ & $\tau$ & $\hat{\tau}$ & average time $(s)$\\
\midrule
	0.2 & 0.203 (0.051) & -2 & -2.043 (0.262) & 3 & 2.979 (0.351) & 1 & 1.037 (0.093) & 1 & 1.036 (0.092) & 120 \\
	\midrule
	0.33 & 0.328 (0.059) & -1 & -1.008 (0.134) & 1 & 1.018 (0.143) & 0.5 & 0.546 (0.041) & 1.2 & 1.221 (0.037) & 124\\
\botrule
\end{tabular}
\end{adjustbox}
\caption{Mean (and standard deviation) of ML estimates $\hat{\thetabold}^\top = (\hat{\kappa}, \hat{\gamma}_x, \hat{\gamma}_y, \hat{c}, \hat{\tau})$ over 10 simulations for two different subsets of advection-diffusion model's parameters}

\label{tab:ml_spatio_temp}
\end{table}

\subsection{Kriging}
\label{sec:kriging}

From the vector $\zbold$ of data at the observed locations defined in Equation \eqref{eq:stat_model}, we aim to predict of the spatio-temporal vector $\xbold$ on the entire spatial mesh during the time window $\left[0,T\right]$, i.e. on ${\cal T}'$. The prediction is made through kriging, which computes the conditional expectation of $\xbold$ knowing $\zbold$ as
\begin{equation}
\xbold^\star = \mathbb{E}(\xbold\lvert \zbold) = (\Qbold+\sigma_0^{-2}\Abold\Abold^\top)^{-1}\Abold \zbold.
    \label{eq:kriging}
\end{equation}
This formula comes from the definition of the augmented matrix $\Sigmabold_c$ of equation \eqref{eq:augmented_mat}. The computation of \eqref{eq:kriging} requires the inversion of the $(N_SN_T, N_SN_T)$ matrix $(\Qbold+\sigma_0^{-2}\Abold\Abold^\top)^{-1}$, that can be achieved by Cholesky decomposition or Conjugate Gradient method as explained in Section \ref{sec:estimation_scattered}. The conditional variance, also called kriging variance, is
$$
\text{Var}(\xbold \lvert \zbold) = (\Qbold+\sigma_0^{-2}\Abold\Abold^\top)^{-1}.
$$

The extrapolation of the diagonal of an inverse matrix is not straightforward when only the Cholesky decomposition of the matrix is available. Some methods exist to as the Takahashi method described in \citet{takahashi73} and \citet{erisman75}. Another way of computing the kriging variance is through conditional simulations, as will be detailed in Section \ref{sec:conditional_simu}.

If we want to predict the $(N_S)$ vector $\xbold_{T+1}$ at time $(T+1)$ on the mesh, we can define $\xbold_{T+1}$ using a time discretization step as defined in Equation \eqref{eq:timestep}
$$\xbold_{T+1} = \Dbold\xbold_{T} + \Ebold \varepsilonbold_{T+1},$$
where $\xbold_T$ is the last subset of size $(N_S)$ of vector $\xbold$ related to time $T$ and $\varepsilonbold_{T+1}$ is a $(N_S)$ standardized Gaussian vector.
If we want to link $\xbold_{T+1}$ to $\xbold$ (the vector containing all time steps), we must replace the $(N_S,N_S)$ matrix $\Dbold$ with a sparse $(N_S,N_SN_T)$ matrix $\Dbold_{ST}$ with $N_T$ blocks, all of whom are null except the last $(N_S,N_S)$ block, which is equal to $\Dbold$. The same procedure is done for $\Ebold$. Hence, we obtain
$$\xbold_{T+1} = \Dbold_{ST}\xbold + \Ebold_{ST} \varepsilonbold_{ST},$$
where $\varepsilonbold_{ST}$ is a $(N_SN_T)$ standardized Gaussian vector.
The covariance between $\xbold_{T+1}$ and $\zbold$ (the $(n)$ observation vector) is then given by $\Dbold_{ST}\Qbold^{-1}\Abold$.
If we consider the augmented covariance matrix $\Sigmabold_{c+1}$ of the vectors $\xbold$, $\xbold_{T+1}$ and $\zbold$, we have

\begin{equation*}
\Sigmabold_{c+1}=\begin{pmatrix} \Qbold^{-1} & \Qbold^{-1}\Dbold_{ST}^\top & \Qbold^{-1}\Abold\\
\Dbold_{ST} \Qbold^{-1} & \Dbold_{ST} \Qbold^{-1}\Dbold_{ST}^\top + \Ebold_{ST}\Ebold_{ST}^\top & \Dbold_{ST} \Qbold^{-1}\Abold\\
\Abold^\top \Qbold^{-1} &  \Abold^\top \Qbold^{-1} \Dbold_{ST}^\top & \Sigmabold_{\zbold}\end{pmatrix}.
\end{equation*}

Finally, we obtain the formula for kriging $\xbold_{T+1}^\star$ at $(T+1)$ as
\begin{equation}
\ybold_{T+1}^\star = \mathbb{E}(\ybold_{T+1}\lvert \zbold) = \Dbold_{ST}(\Qbold+\sigma_0^{-2}\Abold\Abold^\top)^{-1}\Abold \zbold.
    \label{eq:kriging_t1}
\end{equation}
This formula means that we can obtain the kriging $\xbold_{T+1}^\star$ at time $(T+1)$ just by applying one Implicit Euler step to the kriging result $\xbold_{T+1}^\star$ at time $T$. The same procedure can be used to predict at time $(T+2)$ and so on.

\subsection{Conditional simulations}
\label{sec:conditional_simu}

To realize a conditional simulation, we use the conditional kriging paradigm detailed below. First we realize a non-conditional simulation $\xbold_{NC}$ on the spatio-temporal grid ${\cal T}'$.
At the observation points $(\sbold, t) \in \mathcal{I}$, we compute the vector $\zbold_\mathcal{I}$
$$\zbold_\mathcal{I}=\Abold^\top\xbold_{NC}+\varepsilonbold.$$
We define the residuals as $ \rbold_\mathcal{I} = \zbold_\mathcal{I} - \xbold_{NC}(\mathcal{I})$, where the last term represents $\xbold_{NC}$ considered at the observation locations only. Then we apply kriging on the residuals and obtain the values $\rbold^\star$ over the entire spatio-temporal grid $\mathcal{T}'$; we here use the method explained in Section \ref{sec:kriging}. Finally, the conditional simulation is obtained as
$$\zbold_{C}(\mathcal{I}) = \Abold^\top\xbold_{NC} + \Abold^\top\rbold^\star.$$

\section{Application to a solar radiation dataset}
\label{sec:application}

The approach detailed in the previous sections is now applied to a solar radiation dataset for which experts agree on the presence of advection due to Western prevailing winds transporting the clouds from one side of the domain to the other.
The HOPE campaign \citep{macke2017} recorded  Global Horizontal Irradiance (GHI) (or SSI, Surface Solar Irradiance) over a $10000 \times 16000 \ m^2$ region in West Germany near the city of J\"ulich from the 2nd of April, to the second of July, 2013. The sensors were located at 99 stations located as pictured in Figure \ref{fig:locations} and GHI was recorded every 15 seconds. A detailed description of the campaign can be found in \citet{macke2017}.

\begin{figure}
\centering
\includegraphics[width=0.45\textwidth]{mymap_large.pdf}
\includegraphics[width=0.4\textwidth]{mymap.pdf}
\caption{Stations over the spatial domain}
\label{fig:locations}
\end{figure}

The dataset was cleaned for outlying values and non-operating sensors, and the temporal resolution was reduced from 15 seconds to 1 minute. Figure \ref{fig:4_stations} (left panel) shows  GHI as a function of time (in minute, during a full day -- the 28th of May 2013) at 4 different stations. These stations, represented in color in Figure \ref{fig:locations},  are located at the border of the domain, far from each other. The GHI starts close to 0, increases after sunrise, peaks at midday and tends to 0 at sunset. The maximal theoretical amount of irradiance reaching the sensor follows an ideal concave curve. The divergence between the measured irradiance and the optimal curve can be slight or important, depending on the presence of clouds. One can see on this example that the evolution among the 4 stations is similar, with variations accounting for spatio-temporal variations of the clouds.

\begin{figure}
\centering
\includegraphics[width=0.45\textwidth]{GHI_d_40.png}
\includegraphics[width=0.45\textwidth]{CS_index_d_40.png}
\caption{GHI $G$ and Clear Sky Index $K_c$ for 4 different stations on the 28th of May 2013}
\label{fig:4_stations}
\end{figure}

A first preprocessing was made in order to stationarize the phenomenon. \citet{oumbe2014} showed that the solar irradiance  at ground level, GHI (denoted $G$ for short from now on), computed by a radiative transfer model can be approximated by the product of the irradiance under clear atmosphere (called Clear Sky GHI, or $G_c$) and a modification factor due to cloud properties and ground albedo only (Clear Sky Index, or $K_c$, \citet{beyer96}):
\begin{equation}
    G \simeq G_c K_c.
\label{eq:ghi_general}
\end{equation}
The error made in using this approximation depends mostly on the solar zenith angle, the ground albedo and the cloud optical depth. In most cases, the maximum errors (95th percentile) on global and direct surface irradiances are less than 15 $Wm^{-2}$ and less than 2- to 5 \% in relative value, as recommended by the World Meteorological Organization for high-quality measurements of the solar irradiance \citep{oumbe2014}. Practically, it means that a model for fast calculation of surface solar irradiance may be separated into two distinct and independent models: i) a deterministic model for $G$, under clear-sky conditions, as computed according to \citet{gschwind2019}, considered as known in this study; ii) a model for $K_c$ which accounts for cloud influence on the downwelling radiation and is expected to change in time and space. $K_c$ is modeled as a random spatio-temporal process and will the subject of our analysis. Figure  \ref{fig:4_stations} (right panel) shows the variable $K_c$ at the same 4 stations and on the same day of left panel. In general, $K_c$ lies between 0 and 1, but in rare occasions, values above 1 can be observed. This phenomenon is called \textit{overshooting} \citep{schade2007}.

\subsection{Estimation and prediction}
\label{sec:results}

As an example of application of the estimation and prediction methods described in Section \ref{sec:estimation}, we extract a time window of 16 minutes around 4 p.m. the 28th of May 2013 and we consider observations every minute at the 73 stations with well recorded values. We compute the spatial prediction by kriging on a fine grid over the $T=16$ time steps and the spatio-temporal prediction at the two following minutes: $(T+1)$ and $(T+2)$. The estimation of the SPDE parameters is done according to the method described in Section \ref{sec:estimation_scattered}. Kriging equations are computed as presented in Section \ref{sec:kriging} for the 16 time steps on a grid of 900 points. Results are shown in Figure \ref{fig:krig} for the last five minutes (from $t=12$ to $t=16$). To assess the accuracy of the predictions in the direction of the prevailing winds, we apply the kriging procedure to a restricted dataset where a subset of observations in the South-Eastern region has been removed. The kriging formula are then computed with a subset of observations. The two step-ahead predictions are made by applying the Implicit Euler discretization scheme to the kriging vector at time $T$ as presented at the end of Section \ref{sec:kriging}. Prediction maps at time $T$, $(T+1)$ and $(T+2)$ are plotted in the upper panel of Figure \ref{fig:error}, along with the correct values with black outline. In the lower panel, we plot the signed error $(\text{Real} \ K_c - \text{Pred} \ K_c)$.

\begin{figure}
\centering
\includegraphics[width=0.4\textwidth]{hist_d_40_t_985_ns_16.png}
\caption{Histogram of GHI over 20 time steps}
\label{fig:hist}
\end{figure}

\begin{figure}
\centering
\includegraphics[width=0.99\textwidth]{krig_partial_d_40_t_985_ns_16.png}
\caption{Kriging of $K_c$ over the last 5 time steps. The points with black outline are the real values of $K_c$}
\label{fig:krig}
\end{figure}

\begin{figure}
\centering
\includegraphics[width=0.99\textwidth]{krig_partial_t012_d_40_t_985_ns_16.png}
\includegraphics[width=0.99\textwidth]{error_partial_t012_d_40_t_985_ns_16.png}
\caption{High: Predicted kriging of $K_c$ at $T$, $(T+1)$ and $(T+2)$. The points with black outline are the real values of $K_c$. Low: signed error $(\text{Real} \ K_c - \text{Pred} \ K_c)$}
\label{fig:error}
\end{figure}

\subsection{Conditional simulations}

We  present here the results of 100 conditional simulations for time $T$, $(T+1)$ and $(T+2)$. We consider a subset of stations that are located along the direction of the advection direction, i.e. the stations represented in red in the first panel of Figure \ref{fig:cond_simu}. Then we represent the real $K_c$, the mean of the predicted $K_c$ over 10 simulations, along with a $\pm 2\sigma$ envelope, where $\sigma^2$ is the kriging variance among the simulations. The results for time steps $T$, $(T+1)$ and $(T+2)$ are shown in the last three panels of Figure \ref{fig:cond_simu}. We remark that the kriging variance is very small at time $T$ and that the predicted values almost lie on the exact values; at time $(T+1)$ and $(T+2)$ the variance is larger, but all the exact values lie in the $\pm 2\sigma$ envelope.

\begin{figure}
\centering
\includegraphics[width=0.4\textwidth]{adv_dir_d_40_t_985_ns_16.png}\\
\includegraphics[width=0.32\textwidth]{cond_simu_t_d_40_t_985_ns_16.png}
\includegraphics[width=0.32\textwidth]{cond_simu_t1_d_40_t_985_ns_16.png}
\includegraphics[width=0.32\textwidth]{cond_simu_t2_d_40_t_985_ns_16.png}
\caption{Up: Stations with estimated advection direction. Down: Real $K_c$, mean of predicted $K_c$ and $\pm2\sigma$ envelope at time $T$, $(T+1)$ and $(T+2)$}
\label{fig:cond_simu}
\end{figure}

\section{Discussion}
\label{sec:discussion}
The spatio-temporal SPDE approach based on  advection-diffusion equations proposed in this work combines elements of physics, numerical analysis and statistics. It can be seen as a first step toward \textit{physics informed geostatistics}, which introduces physical dynamics into the  statistical model, accounting for possible hidden structures governing the evolution of the spatio-temporal phenomenon.
The different terms of the SPDE (advection, diffusion) directly influence the spatio-temporal dependencies of the process, by controlling its variability in space and time. Compared to spatio-temporal models built on covariance functions such as the Gneiting class \citep{gneiting2002}, we gain in interpretability since the parameters of the model can be directly linked to the physical coefficients of SPDEs.

We showed that it is possible to build an accurate space-time approximations of the process driven by the SPDE using a combination of FEM in space and implicit Euler scheme in time. It leads to sparse structured linear systems. We obtained
promising results for the estimation and for the prediction of processes both in terms of precision and speed. When the size of the data set is moderate, direct matrix implementation is possible. We showed how matrix-free methods can be implemented in order to obtain scalable computations even for very large datasets.

Further work would be necessary to better assess the prediction accuracy and the computational complexity.
Applications to larger and more complex datasets should be considered and comparison to models expressing the advection in a Lagrangian framework \citep{ailliot2011space,salvana2021} should be performed.

One of the main advantages of the SPDE formulation is that it is easy to generalize to non-stationary settings. Non-stationary fields can be defined by letting the parameters ($\kappa(\sbold,t)$, $\vbold(\sbold,t)$) be space-time-dependent. This generalization implies only minimal changes to the method used in the stationary case concerning the simulation, but needs more work for estimation and prediction, since the maximum likelihood approach becomes much more expensive. We can also incorporate models of spatially varying anisotropy by modifying the general operator $\nabla \cdot \Hbold(\sbold,t)\nabla X(\sbold,t)$ with a non-stationary anisotropic matrix $\Hbold(\sbold,t)$. The introduction of non-stationarities could allow to better describe phenomena where local variations are clearly present. The generalization of the approaches by \citet{fulgstad2015} and \citet{pereira2019thesis} should be investigated and generalized to the spatio-temporal framework.

Another interesting consequence of defining the models through local stochastic partial differential equations is that the SPDEs still make sense when $\bbR^d$ is replaced by a space that is only locally flat. We can define non-stationary Gaussian fields on manifolds, and still obtain a GMRF representation. Important improvements were obtained in the spatial case \citep{pereira2022hal}. The generalization to space-time processes could be explored further.

Possible generalization to spatio-temporal SPDEs with a fractional exponent in the diffusion term could also be considered. A development of the methods proposed by \citet{bolin2019} and \citet{vabishchevich2015} should be explored.

\backmatter

\bmhead{Acknowledgments}

We are grateful to O.I.E. center of Mines Paris -- ARMINES, especially to Yves-Marie Saint-Drenan, Philippe Blanc and Hadrien Verbois, for providing the data and for inspiring discussions about renewable resources evaluation. In addition, we thank the Mines Paris / INRAE chair ``Geolearning'' for the constant support.

\bibliographystyle{apalike}