SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Jonathan Schmidt
Tübingen AI Center, University of Tübingen  
jonathan.schmidt@uni-tuebingen.de
Tübingen AI Center, University of Tübingen philipp.hennig@uni-tuebingen.de
Jörg Nick
University of Tübingen nick@na.uni-tuebingen.de
Lund University filip.tronarp@matstat.lu.se
Inference and simulation in the context of high-dimensional dynamical systems remain computationally challenging problems. Some form of dimensionality reduction is required to make the problem tractable in general. In this paper, we propose a novel approximate Gaussian filtering and smoothing method which propagates low-rank approximations of the covariance matrices. This is accomplished by projecting the Lyapunov equations associated with the prediction step to a manifold of low-rank matrices, which are then solved by a recently developed, numerically stable, dynamical low-rank integrator. Meanwhile, the update steps are made tractable by noting that the covariance update only transforms the column space of the covariance matrix, which is low-rank by construction. The algorithm differentiates itself from existing ensemble-based approaches in that the low-rank approximations of the covariance matrices are deterministic, rather than stochastic. Crucially, this enables the method to reproduce the exact Kalman filter as the low-rank dimension approaches the true dimensionality of the problem. Our method reduces computational complexity from cubic (for the Kalman filter) to quadratic in the state-space size in the worst-case, and can achieve linear complexity if the state-space model satisfies certain criteria. Through a set of experiments in classical data-assimilation and spatio-temporal regression, we show that the proposed method consistently outperforms the ensemble-based methods in terms of error in the mean and covariance with respect to the exact Kalman filter. This comes at no additional cost in terms of asymptotic computational complexity.
Spatio-temporal dynamical systems have always played an important role in the applied sciences, such as climate science, numerical weather prediction, or geophysics. In precisely these settings, one is often faced with massive amounts of data to be processed. At the same time the interactions are of such complex nature that it is imperative to include a notion of uncertainty in the model and in its outputs. Both these requirements have a reputation as being computationally expensive. Hence, sensible approximations are indispensable at a certain scale.
Concretely, we consider state-space models of the form
$$
\mathrm {d} x (t) = A x (t) \mathrm {d} t + B \mathrm {d} w (t), \tag {1a}
$$
$$
\left. \mathrm {y} \left(t _ {l}\right) \mid \mathrm {x} \left(t _ {l}\right) \sim \mathcal {N} \left(\mathrm {C x} \left(t _ {l}\right), \mathrm {R}\right), \quad l = 1, \dots , N \right. \tag {1b}
$$
ours:  $r = 3$
ours:  $r = 7$
Figure 1: Rank- $r$  approximations to the true KF covariance (left) for increasing  $r$ . The considered problem's true rank is  $r^* = 7$  by construction, the state-dimension is  $n = 1000$ . The respective low-rank factors are shown above and left of their outer product. For  $r \geq r^*$ , the KF estimate is recovered. For  $r > r^*$ , the excess columns of the low-rank factor collapse (rightmost plot).
$^{7}$  ours:  $r = 11$
ours:  $r = 11$
where  $\mathbf{x} \in \mathbb{R}^n$  is the latent state,  $\mathbf{y} \in \mathbb{R}^m$  is the measurement process, and  $\mathbf{w}$  is a standard Wiener process in  $\mathbb{R}^q$ . Equation (1a) defines the prior dynamics model via a linear time-invariant (LTI) stochastic differential equation (SDE) with drift matrix  $\mathrm{A} \in \mathbb{R}^{n \times n}$  and dispersion matrix  $\mathrm{B} \in \mathbb{R}^{n \times q}$ . The linear Gaussian observation model Eq. (1b) is specified via the measurement matrix  $\mathrm{C} \in \mathbb{R}^{m \times n}$  and measurement-noise covariance  $\mathrm{R} \in \mathbb{R}^{m \times m}$ , where possible time-dependency has been omitted from the notation. While the probabilistic state-estimation problem may in principle be solved in closed form using the Kalman filter (KF) and smoother [25, 34], we will consider the case where the state dimension  $n$  and measurement dimension  $m$  are very large, making the covariance recursion computationally prohibitive in practice. Such settings include spatio-temporal Gaussian process regression [35, 37, 45, 7, 20], and data assimilation applications such as meteorology and oceanography [17, 11], numerical weather forecasting [21], geoscience [6], inverse problems [9, 41], and brain imaging [16]. While the Gaussian process (GP) literature offers many ways of tractable, approximate inference, approaches based on approximations of the underlying kernel function are not amenable to the present problem setting. Without a kernel to approximate, the problem is often solved by ensemble methods, which stochastically propagate low-rank approximations to the covariances in the filtering recursion. This stochasticity introduces unfavorable properties in these methods. To avert this shortcoming, we propose a novel low-rank filtering recursion, which is fully deterministic.
Contributions This text develops a method for efficient, approximate Gaussian filtering, including smoothing and marginal likelihood computation. The method employs the standard predict/correct formulation of the filtering problem, and can in principle be thought of as a square-root implementation of a Kalman filter, in that it represents all covariance matrices as their left square-root factors [19]. However, instead of square matrix square roots, the covariances are approximated by  $(n \times r)$ -dimensional low-rank factors, where  $r \ll n$ . We propose an approximate low-rank prediction-correction loop, together with the computation of the backwards Markov process representing the smoothing posterior, all while preserving this low-rank structure throughout time.
Crucial to this work are (i) the use of dynamic-low-rank approximation (DLRA) [27, 29, 8] to solve Lyapunov equations on the manifold of rank- $r$  matrices in the prediction step and (ii) an efficient correction scheme that updates only the low-dimensional column space of the predicted covariance factor, which introduces no additional approximation error. The computational complexity of the proposed filter is asymptotically equal to those of the widely-used ensemble Kalman filters.
The method offers a deterministic alternative to existing ensemble methods. In particular, it recovers the exact Kalman filter estimate both in the full-rank limit  $r = n$  and if the problem is in fact of rank  $r < n$ . The latter is demonstrated in Fig. 1, in which a problem with  $n = 1000$  state dimensions and a true rank of 7 is approximated with  $r = 3, 7, 11$ . As soon as  $r$  exceeds the true rank of the problem, the residual subspace dimensions collapse. The method will be detailed in Section 3 and evaluated on the basis of various experiments in Section 4. All source code is publicly available on GitHub. $^{1,2}$
<sup>1</sup>https://github.com/schmidtjonathan/RRKF.jl
$^{2}$ https://github.com/schmidtjonathan/RRKF_experiments/
Related work The state estimation problem defined by Eq. (1) amounts to linear Gaussian state estimation and can be computed in closed form by Kalman filtering and smoothing. A modern overview can be found in Särkkä and Solin [36]. While the exact filtering and smoothing recursions scale linearly with the number of time points  $N$ , they scale cubically in the state dimension  $n$  and the measurement dimension  $m$ , making their use computationally infeasible in many high-dimensional problems. A widely-used approximate filtering technique used to circumvent this computational burden are ensemble Kalman filters (EnKF) [11-13, 26, 32, 6]. The ensemble is a set of  $r$  randomly sampled state vectors of dimension  $n$ , which parametrize the mean and covariance via sample statistics. It is processed successively in a prediction-correction loop similar to the KF. The mean-centered ensemble can be regarded as a low-rank square-root factor of the sample covariance matrix. In Section 4 we compare our method to the standard version of the EnKF and the ensemble transform Kalman filter [4], which is categorized as an ensemble-square-root filter [43].
In so far as it aims for computationally tractable Gaussian process regression, the proposed method is related to approaches using the Nystrom method [48], inducing point methods [38, 44, 22], random Fourier features [31], and doubly-sparse variational Gaussian processes [1].
The literature offers more contributions related to the present work under the broader context of low-rank and dimensionality-reduction methods. Loiseau et al. [28], Netto and Mili [30], Vijayshankar et al. [47] approach low-rank modelling of dynamical systems based on measured data, whereas Farrell and Ioannou [15] are concerned with low-rank approximations of a given model based on classical theory on linear time-invariant systems. Cressie et al. [10] develop a tractable hierarchical Bayesian model to approximate high-dimensional spatio-temporal dynamics in a reduced latent space and condition on measurements by transporting the low-dimensional diffusive process to the full space by a projection that is assumed to be known. In those cases, once a low rank model is obtained for the phenomena under study, state estimation becomes computationally tractable, due to the reduced size of the resulting model. These works build upon the assumption that the state of the system evolves approximately in a low-dimensional manifold, whereas our method works under the assumption that the error in the state estimate evolves approximately in a low-dimensional manifold.
Throughout this work, for a function  $x(t)$  and a grid  $(t_1, \dots, t_N)$  we use the abbreviated notation  $x(t_l) = x_l$  and  $(x(t_a), \dots, x(t_b)) = x_{a:b}$ . The task of interest is computing the filtering densities  $p(x_l \mid y_{1:l})$  and smoothing densities  $p(x_l \mid y_{1:N})$  for  $l = 1, \dots, N$ . LTI SDEs of the form in Eq. (1a) admit an equivalent discrete formulation in terms of linear Gaussian transition densities of the form
$$
\mathrm {x} _ {l} \mid \mathrm {x} _ {l - 1} \sim \mathcal {N} \left(\Phi_ {l} \mathrm {x} _ {l - 1}, \mathrm {Q} _ {l}\right). \tag {2}
$$
The transition matrix  $\Phi(t) \in \mathbb{R}^{n \times n}$  and process-noise covariance matrix  $\mathrm{Q}(t) \in \mathbb{R}^{n \times n}$  solve the matrix differential equations  $\dot{\Phi}(t) = \mathrm{A}\Phi(t)$  and
$$
\dot {\mathrm {Q}} (t) = \mathrm {A Q} (t) + \mathrm {Q} (t) \mathrm {A} ^ {*} + \mathrm {B B} ^ {*}, \tag {3}
$$
in the interval  $[t_{l-1}, t_l)$  with initial conditions  $\Phi(t_{l-1}) = I$  and  $Q(t_{l-1}) = 0$ , respectively [36]. Equation (3) is known as a Lyapunov equation, and will be of special importance in this work.
2.1 Square-root filtering
For numerically stable filtering and smoothing, all covariance matrices involved can be represented by matrix square roots (e.g. Cholesky factors). This is known as square-root filtering/smoothing [19]. Let  $(\mathbf{M}_1\quad \mathbf{M}_2)$  denote a wide block matrix in  $\mathbb{R}^{d_1\times (d_2 + d_3)}$ , where  $\mathbf{M}_1\in \mathbb{R}^{d_1\times d_2}$  and  $\mathbf{M}_2\in \mathbb{R}^{d_1\times d_3}$  are some matrices. Given the filtering covariance at time  $t_{l - 1}$  as  $\Sigma_{l - 1} = \Sigma_{l - 1}^{1 / 2}\Sigma_{l - 1}^{* / 2}$  a square-root factorization of the predicted covariance  $\Pi_l = \Pi_l^{1 / 2}\Pi_l^{* / 2}$  is given by
$$
\Pi_ {l} = \Phi_ {l} \Sigma_ {l - 1} \Phi_ {l} ^ {*} + Q _ {l} = \left( \begin{array}{c c} \Phi_ {l} \Sigma_ {l - 1} ^ {1 / 2} & Q _ {l} ^ {1 / 2} \end{array} \right) \left( \begin{array}{c c} \Phi_ {l} \Sigma_ {l - 1} ^ {1 / 2} & Q _ {l} ^ {1 / 2} \end{array} \right) ^ {*}, \tag {4}
$$
and, hence,  $\left(\Phi_l\sum_{l = 1}^{1 / 2}\mathrm{Q}_l^{1 / 2}\right)$  is a matrix square-root of  $\Pi_l$ . Similarly, the correction step can be carried out entirely on square-root factors. For more details, see Grewal and Andrews [19].
2.2 Dynamical low-rank approximation
Consider an initial value problem with a matrix-valued flow field  $F:\mathbb{R}\times \mathbb{R}^{u\times v}\to \mathbb{R}^{u\times v}$
$$
\dot {\mathrm {M}} (t) = F (t, \mathrm {M} (t)), \quad \mathrm {M} (t _ {0}) = \mathrm {M} _ {0}, \tag {5}
$$
where  $u$  and  $v$  are potentially very large. Dynamic low-rank approximation (DLRA) methods [27, 8] efficiently compute low-rank factorizations  $\mathrm{Y}(t) = \mathrm{U}(t)\mathrm{D}(t)\mathrm{V}^{*}(t)\approx \mathrm{M}(t)$  of Eq. (5) by solving
$$
\dot {\mathrm {Y}} (t) = \mathcal {P} _ {r} [ \mathrm {Y} (t) ] \circ F (t, \mathrm {Y} (t)), \quad \mathrm {Y} (t _ {0}) = \mathrm {U} _ {0} \mathrm {D} _ {0} \mathrm {V} _ {0} ^ {*}, \tag {6}
$$
instead. The matrices  $\mathrm{U}_0\in \mathbb{R}^{u\times r}$ ,  $\mathrm{D}_0\in \mathbb{R}^{r\times r}$ , and  $\mathrm{V}_0\in \mathbb{R}^{v\times r}$  are an initial low-rank factorization of  $\mathrm{Y}(t_0)\approx \mathrm{M}(t_0)$ .  $\mathcal{P}_r[\mathrm{Y}(t)]$  denotes the projection operator onto the tangent space at  $\mathrm{Y}(t)$ , where  $\mathrm{Y}(t)$  lies in the manifold of rank-  $r$  matrices. In the present work, we leverage this technique to obtain a low-rank factor  $\mathrm{Q}^{1 / 2}$  of the process-noise covariance at the next prediction location. Thus, we avoid much of the computational cost in the prediction step, otherwise caused by solving the full Lyapunov equation for Q in Eq. (3), for example using matrix-fraction decomposition [40, 3]. Concretely, let  $F(\mathrm{Q}) = \mathrm{AQ} + \mathrm{QA}^* +\mathrm{BB}^*$ . Then, an approximate low-rank process-noise covariance matrix associated with the prediction step is obtained by integrating
$$
\dot {\mathrm {Q}} (t) = \mathcal {P} _ {r} [ \mathrm {Q} (t) ] \circ F (\mathrm {Q} (t)), \quad \mathrm {Q} (t _ {l - 1}) = \mathrm {U} _ {0} \mathrm {D} _ {0} ^ {2} \mathrm {U} _ {0} ^ {*} = 0 \tag {7}
$$
from  $t_{l-1}$  to  $t_l$ . At the start of the filtering recursion  $t_0$ , an initial low-rank factorization is constructed with a random orthogonal matrix  $\mathrm{U}_0$  and  $\mathrm{D}_0 = 0$ . For all subsequent steps the propagated orthogonal basis  $\mathrm{U}(t_l)$  can be reused. In this work, all mentions of dynamic low-rank integration always refer to the recently developed, numerically stable basis update & Galerkin (BUG) integrator [8], whose error bounds are independent of small singular values. More details are given in Appendix C.
3 Rank-reduced Kalman filtering
In this section, a method for approximate inference in Eq. (1) is developed. The idea is to approximate the full filtering/smoothing covariance matrices of the latent state by an eigendecomposition truncated at the  $r$ -th largest eigenvalue. The main challenge is to thereby attain linear computational scaling in  $n$  and  $m$  under appropriate assumptions on the state-space model. Before proceeding with the filtering and smoothing recursions, it is instructive to examine inference in a static low-rank model.
3.1 Efficient inference in static models of low rank
Consider the following latent variable model
$$
\mathrm {x} \sim \mathcal {N} (\mu , \Pi^ {1 / 2} \Pi^ {* / 2}), \quad \mathrm {y} \sim \mathcal {N} (\mathrm {C x}, \mathrm {R}), \tag {8}
$$
where  $\Pi^{1/2} \in \mathbb{R}^{n \times r}$  is of full column rank,  $r \leq n$ , and  $\mathbf{R} \in \mathbb{R}^{m \times m}$ . Representing the latent state  $\mathbf{x}$  as
$$
\mathrm {x} = \mu + \Pi^ {1 / 2} \mathrm {z}, \quad \mathrm {z} \sim \mathcal {N} (0, \mathrm {I} _ {r \times r}), \tag {9}
$$
reduces the problem to inference in the following model
$$
z \sim \mathcal {N} \left(0, I _ {r \times r}\right), \quad y \mid z \sim \mathcal {N} \left(\mathrm {C} \mu + \mathrm {C} \Pi^ {1 / 2} z, R\right). \tag {10}
$$
An efficient inference scheme is given by the following proposition, which is proved in Appendix A.1.
Proposition 1. Let  $\mathbf{z}$  and  $\mathbf{y}$  be two random variables governed by Eq. (10). Assume  $r \leq m$  and consider the following singular value decomposition  $(\mathrm{R}^{-1/2}\mathrm{C}\Pi^{1/2})^* = \mathrm{UDV}^*$ , where  $\mathrm{U}, \mathrm{D} \in \mathbb{R}^{r \times r}$  and  $\mathrm{V} \in \mathbb{R}^{m \times r}$ . Then, defining the whitened residual  $\mathrm{e} = \mathrm{R}^{-1/2}(\mathrm{y} - \mathrm{C}\mu)$ , we get
$$
\mathrm {y} \sim \mathcal {N} \left(\mathrm {C} \mu , \mathrm {R} ^ {1 / 2} \left(\mathrm {V D} ^ {2} \mathrm {V} ^ {*} + \mathrm {I}\right) \mathrm {R} ^ {* / 2}\right), \tag {11a}
$$
$$
z \mid y \sim \mathcal {N} \left(\mathrm {U} \left(\mathrm {I} + \mathrm {D} ^ {2}\right) ^ {- 1} \mathrm {D V} ^ {*} \mathrm {e}, \mathrm {U} \left(\mathrm {I} + \mathrm {D} ^ {2}\right) ^ {- 1} \mathrm {U} ^ {*}\right). \tag {11b}
$$
Furthermore, let  $|\cdot|$  denote the matrix determinant. The marginal log-likelihood of  $y$  is given by
$$
\begin{array}{l} \log \mathcal {N} (\mathrm {y}; \mathrm {C} \mu , \mathrm {R} ^ {1 / 2} \left(\mathrm {V D} ^ {2} \mathrm {V} ^ {*} + \mathrm {I}\right) \mathrm {R} ^ {* / 2}) = - \frac {m}{2} \log 2 \pi - \log | \mathrm {R} ^ {1 / 2} | - \frac {1}{2} \sum_ {k = 1} ^ {r} \log \left(1 + \mathrm {D} _ {k k} ^ {2}\right) \tag {12} \\ - \frac {1}{2} \| e \| ^ {2} + \frac {1}{2} e ^ {*} V D (D ^ {2} + I) ^ {- 1} D V ^ {*} e. \\ \end{array}
$$
The below corollary follows from the deterministic relationship between  $\mathbf{z}$  and  $\mathbf{x}$  given by Eq. (9).
Corollary 1. Let  $y$  and  $x$  be two random variables governed by the model Eq. (8) and  $r \leq m$ . Then
$$
\mathrm {x} \mid \mathrm {y} \sim \mathcal {N} (\mu + \operatorname {K e}, \Sigma), \tag {13}
$$
$$
\mathrm {K} = \Pi^ {1 / 2} \mathrm {U} (\mathrm {I} + \mathrm {D} ^ {2}) ^ {- 1} \mathrm {D V} ^ {*}, \quad \Sigma = \Pi^ {1 / 2} \mathrm {U} (\mathrm {I} + \mathrm {D} ^ {2}) ^ {- 1} \mathrm {U} ^ {*} \Pi^ {* / 2}. \tag {14}
$$
Moreover, a square-root of  $\Sigma$  is readily obtained by
$$
\Sigma^ {1 / 2} = \Pi^ {1 / 2} \mathrm {U} (\mathrm {I} + \mathrm {D} ^ {2}) ^ {- 1 / 2}. \tag {15}
$$
3.2 The rank-reduced filtering recursion
In this section, a low-rank prediction-correction recursion is developed for the purpose of obtaining the filtering densities and the logarithm of the marginal likelihood.
The prediction equations Suppose the filtering distribution at time  $t_{l-1}$  is given by
$$
p \left(\mathrm {x} _ {l - 1} \mid \mathrm {y} _ {1: l - 1}\right) = \mathcal {N} \left(\mathrm {x} _ {l - 1}; \mu_ {l - 1}, \Sigma_ {l - 1} ^ {1 / 2} \Sigma_ {l - 1} ^ {* / 2}\right), \tag {16}
$$
where  $\Sigma_{l-1}^{1/2} \in \mathbb{R}^{n \times r}$ . We begin by detailing how to compute a low-rank factor  $\Pi_l^{1/2}$  of the predicted covariance. First compute the truncated singular value decomposition (SVD)
$$
\left( \begin{array}{l l} \Phi_ {l} \Sigma_ {l - 1} ^ {1 / 2} & Q _ {l} ^ {1 / 2} \end{array} \right) \approx \tilde {U} _ {l} \tilde {D} _ {l} \tilde {V} _ {l} ^ {*} \tag {17}
$$
of the square-root factor (cf. Section 2.1), with  $\tilde{\mathrm{U}}_l\in \mathbb{R}^{n\times r}$ ,  $\tilde{\mathrm{D}}_l\in \mathbb{R}^{r\times r}$ , and  $\tilde{\mathrm{V}}_l\in \mathbb{R}^{r\times r}$ . This can be done in an optimal way by computing the full SVD and truncating it at the  $r$ -th largest singular value in  $\mathcal{O}(nr^2)$  [18]. The process-noise covariance factor  $\mathrm{Q}_l^{1 / 2}\in \mathbb{R}^{n\times r}$  in Eq. (17) is computed using DLRA as described in Section 2.2. The rank-reduced predicted moments at time  $t_l$  are thus given as
$$
\mu_ {l} ^ {-} = \Phi_ {l} \mu_ {l - 1} \quad \in \mathbb {R} ^ {n}, \tag {18}
$$
$$
\Pi_ {l} ^ {1 / 2} = \tilde {\mathrm {U}} _ {l} \tilde {\mathrm {D}} _ {l} \quad \in \mathbb {R} ^ {n \times r}. \tag {19}
$$
The update equations The low-rank update equations follow from Proposition 1 and Corollary 1:
$$
\mu_ {l} = \mu_ {l} ^ {-} + \mathrm {K} _ {l} \mathrm {e} _ {l} \quad \in \mathbb {R} ^ {n}, \tag {20}
$$
$$
\Sigma_ {l} ^ {1 / 2} = \Pi_ {l} ^ {1 / 2} \mathrm {U} _ {l} \left(\mathrm {I} + \mathrm {D} _ {l} ^ {2}\right) ^ {- 1 / 2} \quad \in \mathbb {R} ^ {n \times r}, \tag {21}
$$
with the whitened residual  $\mathrm{e}_l = \mathrm{R}^{-1 / 2}(\mathrm{y}_l - \mathrm{C}\mu_l^-)$  and
$$
\left(\mathrm {R} ^ {- 1 / 2} \mathrm {C I I} _ {l} ^ {1 / 2}\right) ^ {*} = \mathrm {U} _ {l} \mathrm {D} _ {l} \mathrm {V} _ {l} ^ {*}, \quad \mathrm {K} _ {l} = \Pi_ {l} ^ {1 / 2} \mathrm {U} _ {l} \left(\mathrm {I} + \mathrm {D} _ {l} ^ {2}\right) ^ {- 1} \mathrm {D} _ {l} \mathrm {V} _ {l} ^ {*}, \tag {22}
$$
for  $\mathrm{U}_l, \mathrm{D}_l \in \mathbb{R}^{r \times r}$  and  $\mathrm{V}_l \in \mathbb{R}^{m \times r}$ . The marginal predictive log-likelihood of  $\mathrm{y}_l$  is given by
$$
\begin{array}{l} \log p \left(\mathrm {y} _ {l} \mid \mathrm {y} _ {1: l - 1}\right) = - \frac {m}{2} \log 2 \pi - \log \left| \mathrm {R} ^ {1 / 2} \right| - \frac {1}{2} \sum_ {k = 1} ^ {r} \log \left(\left(\mathrm {D} _ {l}\right) _ {k k} ^ {2} + 1\right) \tag {23} \\ - \frac {\left\| \mathbf {e} _ {l} \right\| ^ {2}}{2} + \frac {1}{2} \mathrm {e} _ {l} ^ {*} \mathrm {V} _ {l} \mathrm {D} _ {l} (\mathrm {D} _ {l} ^ {2} + \mathrm {I}) ^ {- 1} \mathrm {D} _ {l} \mathrm {V} _ {l} ^ {*} \mathrm {e} _ {l}. \\ \end{array}
$$
Corollary 1—and by extension, the above filtering recursion—is only valid when  $r \leq m$ , which is exactly the intended usecase for the proposed method. For settings in which  $m < r \leq n$  the correction step follows the square-root correction of a Kalman filter and is detailed in Appendix B.
3.3 Time complexity of the rank-reduced filtering recursion
This section analyzes the computational complexity of the proposed method. In the worst case, the method scales quadratically in the state dimension  $n$  and measurement dimension  $m$ . Under favorable conditions, which are met in many real-world applications, the rank-reduced Kalman filter obtains a computational complexity of  $O(nr^2)$  as stated by Proposition 2, which is proven in Appendix A.2. We begin by specifying a set of assumptions for Proposition 2.
Assumption 1. The maps  $\mathrm{x} \mapsto \mathrm{Ax}, \mathrm{x} \mapsto \Phi \mathrm{x}$ , and  $\mathrm{x} \mapsto \mathrm{BB}^* \mathrm{x}$  can be evaluated in  $\mathcal{O}(n)$ .
Assumption 1 is fulfilled naturally in many dynamical systems, in which local operators introduce sparsity into the dynamics, as, e.g., in finite-difference approximations of spatial differential operators. Another example is Kronecker structure in the system matrices arising in spatio-temporal GP regression by assuming the covariance function is separated into a product over the respective spatial and temporal components [35, 39, 20, 42].
Assumption 2. The map  $\mathrm{x}\mapsto \mathrm{Cx}$  can be evaluated in  $\mathcal{O}(m)$
Assumption 3. The map  $\mathrm{x} \mapsto \mathrm{R}^{-1/2}\mathrm{x}$  and the log-determinant  $\log |\mathrm{R}^{1/2}|$  can be evaluated in  $\mathcal{O}(m)$ .
Throughout this work, we refer to the situation in which Assumption 1 or 2 do not apply as the "worst case". Assumption 3 is taken for granted as the measurement-noise covariance  $\mathbf{R}$  is often a diagonal matrix, which implies that sensor errors are uncorrelated. This is not only realistic but also commonly imposed in modelling. The above assumptions allow the following proposition.
Proposition 2. Given Assumptions 1 to 3, the proposed method approximates the filtering densities and the marginal likelihood at a cost of  $\mathcal{O}(nr^2 + mr^2 + r^3)$ .
The below corollary follows from the book-keeping in the proof of Proposition 2 in Appendix A.2.
Corollary 2. When Assumption 1 or Assumption 2 are not satisfied, the worst-case complexity of the proposed low-rank filtering recursion is  $\mathcal{O}(n^2 r + nm + m^2 r)$ .
Overall, the proposed filtering recursions achieve the same asymptotic complexity as the existing ensemble methods under the same set of assumptions. However, our method is entirely deterministic.
3.4 The rank-reduced smoothing recursion