SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Fiona Lippert
f.lippert@uva.nl
E. Emiel van Loon
e.e.vanloon@uva.nl
Bart Kranstauber
b.kranstauber@uva.nl
Probabilistic inference in high-dimensional state-space models is computationally challenging. For many spatiotemporal systems, however, prior knowledge about the dependency structure of state variables is available. We leverage this structure to develop a computationally efficient approach to state estimation and learning in graph-structured state-space models with (partially) unknown dynamics and limited historical data. Building on recent methods that combine ideas from deep learning with principled inference in Gaussian Markov random fields (GMRF), we reformulate graph-structured state-space models as Deep GMRFs defined by simple spatial and temporal graph layers. This results in a flexible spatiotemporal prior that can be learned efficiently from a single time sequence via variational inference. Under linear Gaussian assumptions, we retain a closed-form posterior, which can be sampled efficiently using the conjugate gradient method, scaling favourably compared to classical Kalman filter based approaches.
Consider the problem of monitoring air pollution, the leading environmental risk factor for mortality worldwide [11, 42]. In the past years, sensor networks have been installed across major metropolitan areas, measuring the concentration of pollutants across time and space [27, 38]. These measurements are, however, prone to noise or might be missing completely due to hardware failures or limited sensor coverage. To identify sources of pollution or travel routes with limited exposure, it is essential to recover pollution levels and provide principled uncertainty estimates for unobserved times and locations. Similar problems occur also in the geosciences, ecology, neuroscience, epidemiology, or transportation systems, where animal movements, the spread of diseases, or traffic load need to be estimated from imperfect data to facilitate scientific discovery and decision-making. From a probabilistic inference perspective, all these problems amount to estimating the posterior distribution over the latent states of a spatiotemporal system given partial and noisy multivariate time series data.
To make inference in these systems feasible, it is common to assume a state-space model, where the latent states evolve according to a Markov chain. Then, the classical Kalman filter (KF) [29] or its variants [47, 37] can be used for efficient inference, scaling linearly with the time series length. However, the complexity with respect to the state dimension remains cubic due to matrix-matrix multiplications and inversions. Thus, as the spatial coverage of available measurements and thereby the dimensionality of the state variables increase, KF-based approaches quickly become computationally prohibitive. This is especially problematic in cases where the underlying dynamics are (partially) unknown and need to be learned from data, requiring repeated inference during the
optimization loop [21, 40, 9]. If, additionally, the access to historical training data is limited, it is essential to incorporate prior knowledge for learning and inference to remain both data-efficient and computationally feasible [56].
For spatiotemporal systems, prior knowledge is often available in the form of graphs, representing the dependency structure of state variables. For air pollution, this could be the city road network and knowledge about wind directions which influence the spread of particulate matter. Incorporating this knowledge into the initial state distribution and the transition model of a state-space model results in a Dynamic Bayesian network (DBN) [12, 39, 56], an interpretable and data-efficient graphical model for multivariate time series data. If the joint space-time graph is sparse and acyclic, belief propagation methods allow for fast and principled inference of marginal posteriors in DBNs, scaling linear with the number of edges [45, 39]. However, when the underlying dynamics are complex, requiring denser graph structures and flexible error terms with a loopy spatial structure, the scalability and convergence of these methods is no longer guaranteed.
Combining the favorable statistical properties of classical graphical models with the flexibility and scalability of deep learning offers promising solutions in this regard [57]. Deep Gaussian Markov random fields (DGMRF), for example, integrate principled inference in Gaussian Markov random fields (GMRF) with convolutional [50] or graph layers [43] to define a new flexible family of GMRFs that can be learned efficiently by maximizing a variational lower bound, while facilitating principled Bayesian inference that remains computationally feasible even for high-dimensional complex models with dense dependency structures. While in principle DGMRFs can be applied to spatiotemporal systems by treating each time step independently, they are by design limited to spatial structures.
In this paper we extend the DGMRF framework to spatiotemporal systems with graph-structured states, transitions and noise terms. To this end, we formulate a dynamical prior based on a DBN that relates state variables over adjacent time steps, and DGMRFs that capture the spatial structure of unmodeled exogenous influences
Figure 1: ST-DGMRF overview. We reconstruct the latent states of a graph-structured dynamical system from partial and noisy observations (orange arrow with question mark). Temporal and spatial layers transform the state  $\mathbf{x}$  to a standard Gaussian, which implicitly defines a space-time GMRF prior.
and errors in the transition model. The key to our approach is the insight that, using locally linear Gaussian transition models, the precision matrix of the joint space-time process preserves the structure of transition and noise precision matrices. This is in contrast to the marginal distributions used in Kalman filter based approaches for which both precision and covariance matrices become dense over time. A convenient factorization of the joint precision matrix then leads to a spatiotemporal DGMRF formulation that is equivalent to the generative state-space formulation, but inherits the favorable properties of DMGRFs for learning and inference.
The simplest approach to scaling Kalman filtering and smoothing to high-dimensional state variables is to assume a decomposition into subprocesses, that can be reconstructed independently [30]. For spatiotemporal processes, however, this assumption can be detrimental. Sample-based approaches, like the Ensemble KF [17, 18, 26, 31], instead approximate state distributions with Monte Carlo estimates, balancing computational complexity and accuracy. This is widely used in the geosciences [53, 55] where accurate models of the underlying process are available. If the dynamics are (partially) unknown, the parameters are typically estimated jointly with the system states via state augmentation [2]. However, this becomes problematic if the transition model is only a poor approximation and complex error covariances need to be estimated [15, 52]. Alternatively, expectation maximization
(EM) is used to estimate parameters iteratively [21], which requires repeated ensemble simulations and thus quickly becomes computationally demanding.
In machine learning, neural network-based approaches map high-dimensional data to a latent space that is either low-dimensional [19] or factorizes conveniently [7], such that standard KF inference remains feasible. To learn suitable maps, however, these methods require sufficiently large training data sets. Moreover, the projection to an abstract latent space hinders the incorporation of structural or functional prior knowledge. Alternatively, variational inference allows for fast inference in (deep) state space models [20, 3, 28, 34, 54] by approximating the true posterior with a computationally convenient variational distribution. While this approach is highly flexible and scalable, it is known to severely underestimate uncertainties [5], which can be detrimental for science and decision-making.
Gaussian processes (GP) are another widely used model class for spatiotemporal inference [35]. In contrast to KF-based approaches, GPs encode prior knowledge into their covariance kernel. To increase the expressivity beyond standard kernels and overcome computational limitations, GPs have been combined with deep learning [14, 13]. However, scaling GPs to large-scale spatiotemporal processes with graph structure remains challenging. Scalable approaches for temporal data involve reformulating GPs as state-space models and applying KF-based recursions [24, 49, 10, 1, 58, 23] which, as discussed before, remains computationally prohibitive for large state spaces. Extensions to graph-structured domains [8, 41] rely on the relation between GPs and stochastic differential equations (SDE) [36, 51], which makes it cumbersome to design new kernels or to incorporate prior knowledge that cannot be formulated as an SDE.
1.2 Our contributions
In contrast to most previous approaches, we focus on settings where the dynamics are (partially) unknown and need to be learned from data, while historical training data is limited or even unavailable. Our contributions can be summarized as follows:
1. We propose ST-DGMRF, which extends the DGMRF framework to spatiotemporal systems, by reformulating graph-structured state-space models as joint space-time GMRFs that are implicitly defined by simple spatial and temporal graph layers.
2. We show that the multi-layer space-time formulation facilitates efficient learning and inference, scaling linearly w.r.t. the number of layers, time series length, and state dimensionality.
3. In experiments on synthetic and real world data, we demonstrate that our approach provides accurate state and uncertainty estimates, comparing favorably to other scalable approaches relying on ensembles or simplifications of the dependency structure.
Consider a single sequence of high-dimensional measurements  $\mathbf{y}_{0:K} = (\mathbf{y}_0,\dots ,\mathbf{y}_K)$  taken at consecutive time points  $t_0,\ldots ,t_K$ . Each  $\mathbf{y}_k\in \mathbb{R}^{M_k}$  represents partial and noisy observations (e.g. from spatially distributed sensors) generated according to a linear Gaussian noise model
$$
\mathbf {y} _ {k} = \mathbf {H} _ {k} \mathbf {x} _ {k} + \boldsymbol {\xi} _ {k}, \quad \boldsymbol {\xi} _ {k} \sim \mathcal {N} (\mathbf {0}, \mathbf {R} _ {k}) \tag {1}
$$
from the latent state  $\mathbf{x}_k\in \mathbb{R}^N$  of an underlying dynamical process of dimensionality  $N\geq M_{k}$  (e.g. air pollution along  $N$  roads within an urban area). It is common to assume either  $\mathbf{R}_k = \mathrm{diag}(\sigma_{k,1}^2,\dots ,\sigma_{k,M_k}^2)$ , representing independent sensors with varying noise levels, or  $\mathbf{R}_k = \sigma^2\mathbf{I}_{M_k}$ , representing a fixed noise level across all sensors and time points [9]. Further, the observation matrices  $\mathbf{H}_k\in \mathbb{R}^{M_k\times N}$  are typically very sparse with  $\mathcal{O}(M_k)$  number of non-zero entries.
Assuming that all  $\mathbf{H}_k$  are known, we aim at reconstructing the latent system states  $\mathbf{x}_{0:K} = (\mathbf{x}_0,\dots ,\mathbf{x}_K)\in \mathbb{R}^{(K + 1)\times N}$  from data  $\mathbf{y}_{0:K}$ . Since observations are partial and noisy,  $\mathbf{x}_{0:K}$  cannot be recovered without ambiguity. It is therefore essential to treat the problem probabilistically and incorporate available domain knowledge into the prior distribution  $p(\mathbf{x}_{0:K}\mid \boldsymbol {\theta})$ . The task then amounts to (i) specifying a suitable prior that captures the spatiotemporal structure of  $\mathbf{x}_{0:T}$ , (ii) estimating unknown parameters, and (iii) computing the posterior distribution  $p(\mathbf{x}_{0:K}\mid \mathbf{y}_{0:K},\hat{\boldsymbol{\theta}})$ .
Notation We use  $\mathbf{x} = \mathrm{vec}(\mathbf{x}_{0:K})\in \mathbb{R}^{(K + 1)N}$  and  $\mathbf{y} = \mathrm{vec}(\mathbf{y}_{0:K})\in \mathbb{R}^M$  with  $M = \sum_{k = 0}^{K}M_{k}$  to denote vectorized states and measurements. Similarly, we use  $\mathbf{H} = \mathrm{diag}(\mathbf{H}_0,\dots ,\mathbf{H}_K)\in$ $\mathbb{R}^{M\times (K + 1)N}$  and  $\mathbf{R} = \mathrm{diag}(\mathbf{R}_0,\ldots ,\mathbf{R}_K)\in \mathbb{R}^{M\times M}$  to denote the joint spatiotemporal observation and covariance matrix.
In the following sections, we briefly introduce two graphical models, linear dynamical systems and Gaussian Markov random fields, which form the temporal and spatial backbone of our method. We then introduce the deep GMRF framework on which we build in Section 3 to formulate a flexible multi-layer spatiotemporal prior.
2.2 Linear dynamical system
A discrete-time linear dynamical system (LDS) is a state-space model that defines the prior over states  $\mathbf{x}_{0:K}$  in terms of linear Gaussian transition models
$$
\mathbf {x} _ {k} = \mathbf {F} _ {k} \mathbf {x} _ {k - 1} + \mathbf {c} _ {k} + \boldsymbol {\epsilon} _ {k}, \quad \boldsymbol {\epsilon} _ {k} \sim \mathcal {N} (\mathbf {0}, \mathbf {Q} _ {k} ^ {- 1}) \quad \forall k \in \{1, \dots , K \} \tag {2}
$$
and an initial distribution  $\mathbf{x}_0\sim \mathcal{N}(\pmb {\mu}_0,\mathbf{Q}_0^{-1})$  . The classical Kalman filter [29] and its variants [47, 37] use recursive algorithms that exploit the temporal independence structure of the prior and its conjugacy to the observation model (see Eq. (1)) to compute marginal posterior distributions  $p(\mathbf{x}_k\mid \mathbf{y}_{0:k})$  in  $\mathcal{O}(KN^3)$  . The same formulas can be used within the EM-algorithm to learn unknown parameters  $\pmb{\mu}_0,\mathbf{Q}_0,\mathbf{F}_k,\mathbf{c}_k,\mathbf{Q}_k^{-1}$  in  $\mathcal{O}(JKN^3)$  for  $J$  iterations. While this scales favorably for long time series, it remains computationally prohibitive for high-dimensional systems with  $N\gg K$
2.3 Gaussian Markov random fields
A multivariate Gaussian  $\mathcal{N}(\pmb{\mu},\pmb{\Omega}^{-1})$  forms a GMRF [48] with respect to an undirected graph  $\mathcal{G} = (\mathcal{V},\mathcal{E})$  if the edges  $\mathcal{E}$  correspond to the non-zero entries of the precision matrix  $\pmb{\Omega}$ , i.e.  $\pmb{\Omega}_{ij} \neq 0 \Leftrightarrow (i,j) \in \mathcal{E}$ . Given a linear Gaussian observation model  $\mathbf{y} \sim \mathcal{N}(\mathbf{Hx},\mathbf{R})$ , the posterior can be written in closed form  $p(\mathbf{x} \mid \mathbf{y},\hat{\pmb{\theta}}) = \mathcal{N}(\pmb{\mu}^{+},(\pmb{\Omega}^{+})^{-1})$  with precision matrix  $\pmb{\Omega}^{+} = \pmb{\Omega} + \mathbf{H}^{T}\mathbf{R}^{-1}\mathbf{H}$  and mean  $\pmb{\mu}^{+} = (\pmb{\Omega}^{+})^{-1}(\pmb{\Omega}\pmb{\mu} + \mathbf{H}^{T}\mathbf{R}^{-1}\mathbf{y})$ .
However, naively computing  $\mu^{+}$  by matrix inversion requires  $\mathcal{O}(K^3 N^3)$  computations and  $\mathcal{O}(K^2 N^2)$  memory, which is infeasible for high-dimensional systems. Instead, the conjugate gradient method [6] can be used to iteratively solve the sparse linear system  $\Omega^{+}\pmb{\mu}^{+} = \Omega \pmb{\mu} + \mathbf{H}^{T}\mathbf{R}^{-1}\mathbf{y}$  for  $\pmb{\mu}^{+}$ . The same approach allows to generate samples  $\hat{\mathbf{x}}\sim \mathcal{N}(\pmb{\mu}^{+},(\Omega^{+})^{-1})$  and obtain Monte Carlo estimates of marginal variances [44]. In practice, however, it remains difficult to design suitable precision matrices that are expressive but sparse enough to remain computationally feasible.
2.4 Deep Gaussian Markov random fields
A DGMRF [50] is defined by an affine transformation
$$
\mathbf {z} = \mathbf {g} _ {\theta} (\mathbf {x}) = \mathbf {G} _ {\theta} \mathbf {x} + \mathbf {b} _ {\theta} \quad \text {w i t h} \quad \mathbf {z} \sim \mathcal {N} (\mathbf {0}, \mathbf {I}), \tag {3}
$$
where  $\mathbf{g}_{\theta} = \mathbf{g}_{\theta}^{(L)} \circ \dots \circ \mathbf{g}_{\theta}^{(1)}$  is a composition of  $L$  simple linear layers with convenient computational properties. This implicitly defines a GMRF  $\mathbf{x} \sim \mathcal{N}(\boldsymbol{\mu}, \boldsymbol{\Omega}^{-1})$  with  $\boldsymbol{\mu} = -\mathbf{G}_{\theta}^{-1}\mathbf{b}_{\theta}$  and  $\boldsymbol{\Omega} = \mathbf{G}_{\theta}^{T}\mathbf{G}_{\theta}$ . The multi-layer construction facilitates fast and statistically sound posterior inference using the conjugate gradient method, as well as fast parameter learning using a variational approximation, even when the resulting precision matrix  $\boldsymbol{\Omega}$  becomes dense.
While originally, [50] defined their layers only for lattice-structured data, [43] generalized this to arbitrary graphs. Central to their approach is that each layer  $\mathbf{h}^{(l)} = \mathbf{G}_{\theta}^{(l)}\mathbf{h}^{(l - 1)} + \mathbf{b}_{\theta}^{(l)}$  is defined on a sparse base graph  $\bar{\mathcal{G}}$  with adjacency matrix  $\mathbf{A}$  and degree matrix  $\mathbf{D}$  such that
$$
\mathbf {G} ^ {(l)} = \alpha_ {l} \mathbf {D} ^ {\gamma_ {l}} + \beta_ {l} \mathbf {D} ^ {\gamma_ {l} - 1} \mathbf {A} \quad \text {a n d} \quad \mathbf {b} _ {\theta} ^ {(l)} = b _ {l} \mathbf {1} \tag {4}
$$
with parameters  $\theta_{l} = (\alpha_{l},\beta_{l},\gamma_{l},b_{l})$ . This construction allows for fast log-determinant computations during learning, and GPU-accelerated computation of  $\mathbf{g}_{\theta}(\mathbf{x})$  using existing software for graph neural networks. Note that for  $L$  layers defined on  $\bar{\mathcal{G}}$ , the sparsity pattern of  $\Omega$  corresponds to the sparsity pattern of  $(\mathbf{A} + \mathbf{D})^{2L}$ . And thus, the resulting model defines a GMRF w.r.t the  $2L$ -hop graph of  $\bar{\mathcal{G}}$ .
3 Spatiotemporal DGMRFs
To extend the DGMRF framework to spatiotemporal systems, we first formulate a dynamical prior in terms of a GMRF that encodes prior knowledge in the form of spatial and temporal independence assumptions. We then parameterize this prior using simple spatial and temporal layers, which results in a flexible model architecture that facilitates efficient learning and principled Bayesian inference in high-dimensional dynamical systems, scaling favorably compared to Kalman filter based methods.
3.1 Graph-structured dynamical prior
Consider a dynamical system for which the state evolution is well described by Eq. (2). We say that this process is graph-structured if each dimension  $d_{i}$  of the system state  $\mathbf{x}_k$  can be associated with a node  $i\in \mathcal{V}$  in a multigraph  $\mathcal{G} = (\mathcal{V},\mathcal{E}_{\mathrm{spatial}},\mathcal{E}_{\mathrm{temporal}})$ , where the set of undirected spatial edges  $\mathcal{E}_{\mathrm{spatial}}\subseteq \mathcal{V}\times \mathcal{V}$  defines the sparsity pattern of noise precision (inverse covariance) matrices
$$
\left(\mathbf {Q} _ {k}\right) _ {i j} \neq 0 \quad \Longleftrightarrow \quad \left(\mathbf {Q} _ {k}\right) _ {j i} \neq 0 \quad \Longleftrightarrow \quad (j, i) \in \mathcal {E} _ {\text {s p a t i a l}} \quad \forall k \in \{0, \dots , K \}, \tag {5}
$$
and the set of directed temporal edges  $\mathcal{E}_{\mathrm{temporal}} \subseteq \mathcal{V} \times \mathcal{V}$  defines the sparsity pattern of the state transition matrices
$$
\left(\mathbf {F} _ {k}\right) _ {i j} \neq 0 \quad \Longleftrightarrow \quad (j, i) \in \mathcal {E} _ {\text {t e m p o r a l}} \quad \forall k \in \{1, \dots , K \}. \tag {6}
$$
This defines a DBN over  $\mathbf{x}_{0:K}$  encoding conditional independencies of the form
$$
x _ {k, i} \perp \perp \mathbf {x} _ {k - 1, \mathcal {V} \backslash n _ {\text {t e m p o r a l}} (i)} \mid \mathbf {x} _ {k - 1, n _ {\text {t e m p o r a l}} (i)} \quad \text {a n d} \quad x _ {k, i} \perp \perp \mathbf {x} _ {l, \mathcal {V}} \mid \mathbf {x} _ {k - 1, n _ {\text {t e m p o r a l}} (i)} \forall l <   k - 1, \tag {7}
$$
where  $n_{\mathrm{temporal}}(i)$  denotes the set of neighbors  $j$  of  $i$  for which  $(j,i) \in \mathcal{E}_{\mathrm{temporal}}$ . Intuitively,  $\mathcal{E}_{\mathrm{temporal}}$  represent causal effects over time, while  $\mathcal{E}_{\mathrm{spatial}}$  represent the structure of random effects that are not captured by the transition model. To define the graph structure, prior knowledge about, for example, the physical system structure or underlying causal mechanisms can be exploited.
3.1.1 Joint distribution
The graph-structured dynamical prior induces a multivariate Gaussian prior  $\mathcal{N}(\pmb{\mu},\pmb{\Omega}^{-1})$  on  $\mathbf{x}$  with sparse precision matrix  $\Omega \in \mathbb{R}^{(K + 1)N\times (K + 1)N}$ . Importantly,  $\Omega$  can be shown to factorize as  $\mathbf{F}^T\mathbf{Q}\mathbf{F}$  with block diagonal matrix  $\mathbf{Q}$  and unit lower block bidiagonal matrix  $\mathbf{F}$  defined as
$$
\mathbf {Q} = \operatorname {d i a g} \left(\mathbf {Q} _ {0}, \mathbf {Q} _ {1}, \dots , \mathbf {Q} _ {K}\right), \quad \mathbf {F} := \left[ \begin{array}{c c c c} \mathbf {I} & & & \\ - \mathbf {F} _ {1} & \mathbf {I} & & \\ & \dots & \dots & \\ & & - \mathbf {F} _ {K} & \mathbf {I} \end{array} \right], \tag {8}
$$
where empty positions represent zero-blocks. Further, while the mean  $\pmb{\mu} \in \mathbb{R}^{(K + 1)N}$  needs to be computed iteratively as  $\pmb{\mu}_k = \mathbf{F}_k\pmb{\mu}_{k - 1} + \mathbf{c}_k$ , the information vector  $\pmb{\eta} = \pmb{\Omega}\pmb{\mu}$  can be expressed compactly as
$$
\boldsymbol {\eta} = \mathbf {F} ^ {T} \mathbf {Q} \mathbf {F} \boldsymbol {\mu} = \mathbf {F} ^ {T} \mathbf {Q} \mathbf {c} \tag {9}
$$
with  $\mathbf{c} = [\pmb{\mu}_0, \mathbf{c}_1, \dots, \mathbf{c}_K] \in \mathbb{R}^{(K + 1)N}$ . See Appendix A.1 for detailed derivations.
3.1.2 DGMRF formulation
We now reformulate  $\mathbf{x} \sim \mathcal{N}(\boldsymbol{\mu}, \boldsymbol{\Omega}^{-1})$  as a DGMRF. Importantly, in contrast to [50, 43], the graph-structured dynamical prior allows us to impose additional structure, reflecting the spatiotemporal nature of the underlying system. In particular, since  $\mathbf{Q}_0, \mathbf{Q}_1, \ldots, \mathbf{Q}_K$  are symmetric positive definite, we can factorize  $\mathbf{Q} = \mathbf{S}^T\mathbf{S}$  with symmetric block-diagonal matrix  $\mathbf{S}$ , and thus  $\boldsymbol{\Omega} = \mathbf{F}^T\mathbf{S}^T\mathbf{S}\mathbf{F}$ . Together with Eq. (9), this results in a GMRF defined by
$$
\mathbf {G} _ {\theta} := \mathbf {S F} \quad \text {a n d} \quad \mathbf {b} _ {\theta} := - \mathbf {G} _ {\theta} \boldsymbol {\mu} = - \mathbf {S c} \tag {10}
$$
Finally, we separate  $\mathbf{g}_{\theta}$  into a temporal map  $\mathbf{f}_{\theta}:\mathbb{R}^{(K + 1)N}\to \mathbb{R}^{(K + 1)N}$  and a spatial map  $\mathbf{s}_{\theta}:\mathbb{R}^{(K + 1)N}\to \mathbb{R}^{(K + 1)N}$  defined as
$$
\mathbf {f} _ {\theta} (\mathbf {x}) := \mathbf {F x} + \mathbf {b} _ {f} = \mathbf {h} \quad \text {a n d} \quad \mathbf {s} _ {\theta} (\mathbf {h}) := \mathbf {S h} + \mathbf {b} _ {s} = \mathbf {z}. \tag {11}
$$
Note that this results in an overall bias  $\mathbf{b}_{\theta} = \mathbf{S}\mathbf{b}_{f} + \mathbf{b}_{s}$  and thus  $\mathbf{c} = -(\mathbf{b}_{f} + \mathbf{S}^{-1}\mathbf{b}_{s})$ , which allows for modelling long-range spatial dependencies in  $\pmb{\mu}_0$  and  $\mathbf{c}_1,\dots ,\mathbf{c}_K$ . The combined transformation  $\mathbf{z} = (\mathbf{s}_{\theta}\circ \mathbf{f}_{\theta})(\mathbf{x})$  essentially defines a standard two-layer DGMRF, which describes a graph-structured dynamical system if the parameters  $\pmb{\theta} = (\mathbf{b}_s,\mathbf{b}_f,\mathbf{F}_1,\dots ,\mathbf{F}_K,\mathbf{Q}_0,\dots ,\mathbf{Q}_K)$  are subject to sparsity constraints (5) and (6).
3.2 Parameterization
Learning the unknown parameters  $\theta$  directly from a single sequence  $\mathbf{y}_{0:K}$  will result in a highly overparameterized model that is unsuitable for the problem of interest. In addition, careful parameterization of  $\mathbf{g}_{\theta}$  can greatly reduce the computational complexity of the transformation and the associated log-determinant computations required during learning [50, 43]. To achieve a good trade-off between data-efficiency, expressivity and scalability, we define both  $\mathbf{s}_{\theta}$  and  $\mathbf{f}_{\theta}$  in terms of simple layers with few parameters and convenient computational properties.
3.2.1 Spatial layer(s)