Title: The Rank-Reduced Kalman Filter: Approximate Dynamical-Low-Rank Filtering In High Dimensions
Venue: NeurIPS 2023
Accepted: yes
Keywords: Gaussian

filtering

smoothing

bayesian

state-space models

dynamic-low-rank

high-dimensional

spatio-temporal

Gaussian processes

regression

low rank

state estimation
Abstract: Inference and simulation in the context of high-dimensional dynamical systems remain computationally challenging problems.
Some form of dimensionality reduction is required to make the problem tractable in general.
In this paper, we propose a novel approximate Gaussian filtering and smoothing method
which propagates low-rank approximations of the covariance matrices.
This is accomplished by projecting the Lyapunov equations associated with the prediction step to a manifold of low-rank matrices,
which are then solved by a recently developed, numerically stable, dynamical low-rank integrator.
Meanwhile, the update steps are made tractable by noting that the covariance update only transforms the column space of the covariance matrix, which is low-rank by construction.
The algorithm differentiates itself from existing ensemble-based approaches in that
the low-rank approximations of the covariance matrices are deterministic, rather than stochastic.
Crucially, this enables the method to reproduce the exact Kalman filter as the low-rank dimension approaches the true dimensionality of the problem.
Our method reduces computational complexity from cubic (for the Kalman filter) to quadratic in the state-space size in the worst-case, and can achieve linear complexity if the state-space model satisfies certain criteria.
Through a set of experiments in classical data-assimilation and spatio-temporal regression, we show that the proposed method consistently outperforms the ensemble-based methods in terms of error in the mean and covariance with respect to the exact Kalman filter. This comes at no additional cost in terms of asymptotic computational complexity.

The Rank-Reduced Kalman Filter: Approximate Dynamical-Low-Rank Filtering In High Dimensions

Abstract

Inference and simulation in the context of high-dimensional dynamical systems remain computationally challenging problems. Some form of dimensionality reduction is required to make the problem tractable in general. In this paper, we propose a novel approximate Gaussian filtering and smoothing method which propagates low-rank approximations of the covariance matrices. This is accomplished by projecting the Lyapunov equations associated with the prediction step to a manifold of low-rank matrices, which are then solved by a recently developed, numerically stable, dynamical low-rank integrator. Meanwhile, the update steps are made tractable by noting that the covariance update only transforms the column space of the covariance matrix, which is low-rank by construction. The algorithm differentiates itself from existing ensemble-based approaches in that the low-rank approximations of the covariance matrices are deterministic, rather than stochastic. Crucially, this enables the method to reproduce the exact Kalman filter as the lowrank dimension approaches the true dimensionality of the problem. Our method reduces computational complexity from cubic (for the Kalman filter) to quadratic in the state-space size in the worst-case, and can achieve linear complexity if the state-space model satisfies certain criteria. Through a set of experiments in classical data-assimilation and spatio-temporal regression, we show that the proposed method consistently outperforms the ensemble-based methods in terms of error in the mean and covariance with respect to the exact Kalman filter. This comes at no additional cost in terms of asymptotic computational complexity.

1 Introduction

Spatio-temporal dynamical systems have always played an important role in the applied sciences, such as climate science, numerical weather prediction, or geophysics. In precisely these settings, one is often faced with massive amounts of data to be processed. At the same time the interactions are of such complex nature that it is imperative to include a notion of uncertainty in the model and in its outputs. Both these requirements have a reputation as being computationally expensive. Hence, sensible approximations are indispensable at a certain scale.

Concretely, we consider state-space models of the form

7

Kalman filter

ours :

r

=3

ours :

r

=7

ours :

r

=11

0

where x ∈ R n is the latent state, y ∈ R m is the measurement process, and w is a standard Wiener process in R q . Equation (1a) defines the prior dynamics model via a linear time-invariant (LTI) stochastic differential equation (SDE) with drift matrix A ∈ R n × n and dispersion matrix B ∈ R n × q . The linear Gaussian observation model Eq. (1b) is specified via the measurement matrix C ∈ R m × n and measurement-noise covariance R ∈ R m × m , where possible time-dependency has been omitted from the notation. While the probabilistic state-estimation problem may in principle be solved in closed form using the Kalman filter (KF) and smoother [25, 34], we will consider the case where the state dimension n and measurement dimension m are very large, making the covariance recursion computationally prohibitive in practice. Such settings include spatio-temporal Gaussian process regression [35, 37, 45, 7, 20], and data assimilation applications such as meteorology and oceanography [17, 11], numerical weather forecasting [21], geoscience [6], inverse problems [9, 41], and brain imaging [16]. While the Gaussian process (GP) literature offers many ways of tractable, approximate inference, approaches based on approximations of the underlying kernel function are not amenable to the present problem setting. Without a kernel to approximate, the problem is often solved by ensemble methods, which stochastically propagate low-rank approximations to the covariances in the filtering recursion. This stochasticity introduces unfavorable properties in these methods. To avert this shortcoming, we propose a novel low-rank filtering recursion, which is fully deterministic.

Contributions This text develops a method for efficient, approximate Gaussian filtering, including smoothing and marginal likelihood computation. The method employs the standard predict/correct formulation of the filtering problem, and can in principle be thought of as a square-root implementation of a Kalman filter, in that it represents all covariance matrices as their left square-root factors [19]. However, instead of square matrix square roots, the covariances are approximated by ( n × r ) -dimensional low-rank factors, where r ≪ n . We propose an approximate low-rank predictioncorrection loop, together with the computation of the backwards Markov process representing the smoothing posterior, all while preserving this low-rank structure throughout time.

Crucial to this work are (i) the use of dynamic-low-rank approximation (DLRA) [27, 29, 8] to solve Lyapunov equations on the manifold of rankr matrices in the prediction step and (ii) an efficient correction scheme that updates only the low-dimensional column space of the predicted covariance factor, which introduces no additional approximation error. The computational complexity of the proposed filter is asymptotically equal to those of the widely-used ensemble Kalman filters.

The method offers a deterministic alternative to existing ensemble methods. In particular, it recovers the exact Kalman filter estimate both in the full-rank limit r = n and if the problem is in fact of rank r < n . The latter is demonstrated in Fig. 1, in which a problem with n = 1000 state dimensions and a true rank of 7 is approximated with r = 3 , 7 , 11 . As soon as r exceeds the true rank of the problem, the residual subspace dimensions collapse. The method will be detailed in Section 3 and evaluated on the basis of various experiments in Section 4. All source code is publicly available on GitHub. 1,2

Related work The state estimation problem defined by Eq. (1) amounts to linear Gaussian state estimation and can be computed in closed form by Kalman filtering and smoothing. A modern overview can be found in Särkkä and Solin [36]. While the exact filtering and smoothing recursions scale linearly with the number of time points N , they scale cubically in the state dimension n and the measurement dimension m , making their use computationally infeasible in many high-dimensional problems. A widely-used approximate filtering technique used to circumvent this computational burden are ensemble Kalman filters (EnKF) [11-13, 26, 32, 6]. The ensemble is a set of r randomly sampled state vectors of dimension n , which parametrize the mean and covariance via sample statistics. It is processed successively in a prediction-correction loop similar to the KF. The meancentered ensemble can be regarded as a low-rank square-root factor of the sample covariance matrix. In Section 4 we compare our method to the standard version of the EnKF and the ensemble transform Kalman filter [4], which is categorized as an ensemble-square-root filter [43].

In so far as it aims for computationally tractable Gaussian process regression, the proposed method is related to approaches using the Nyström method [48], inducing point methods [38, 44, 22], random Fourier features [31], and doubly-sparse variational Gaussian processes [1].

The literature offers more contributions related to the present work under the broader context of lowrank and dimensionality-reduction methods. Loiseau et al. [28], Netto and Mili [30], Vijayshankar et al. [47] approach low-rank modelling of dynamical systems based on measured data, whereas Farrell and Ioannou [15] are concerned with low-rank approximations of a given model based on classical theory on linear time-invariant systems. Cressie et al. [10] develop a tractable hierarchical Bayesian model to approximate high-dimensional spatio-temporal dynamics in a reduced latent space and condition on measurements by transporting the low-dimensional diffusive process to the full space by a projection that is assumed to be known. In those cases, once a low rank model is obtained for the phenomena under study, state estimation becomes computationally tractable, due to the reduced size of the resulting model. These works build upon the assumption that the state of the system evolves approximately in a low-dimensional manifold, whereas our method works under the assumption that the error in the state estimate evolves approximately in a low-dimensional manifold.

2 Background

Throughout this work, for a function x ( t ) and a grid ( t 1 , .., t N ) we use the abbreviated notation x ( t l ) = x l and ( x ( t a ) , ...x ( t b )) = x a : b . The task of interest is computing the filtering densities p ( x l | y 1: l ) and smoothing densities p ( x l | y 1: N ) for l = 1 , . . . , N . LTI SDEs of the form in Eq. (1a) admit an equivalent discrete formulation in terms of linear Gaussian transition densities of the form

The transition matrix Φ( t ) ∈ R n × n and process-noise covariance matrix Q( t ) ∈ R n × n solve the matrix differential equations ˙ Φ( t ) = AΦ( t ) and

in the interval [ t l -1 , t l ) with initial conditions Φ( t l -1 ) = I and Q( t l -1 ) = 0 , respectively [36]. Equation (3) is known as a Lyapunov equation , and will be of special importance in this work.

2.1 Square-root filtering

For numerically stable filtering and smoothing, all covariance matrices involved can be represented by matrix square roots (e.g. Cholesky factors). This is known as square-root filtering/smoothing [19]. Let (M 1 M 2 ) denote a wide block matrix in R d 1 × ( d 2 + d 3 ) , where M 1 ∈ R d 1 × d 2 and M 2 ∈ R d 1 × d 3 are some matrices. Given the filtering covariance at time t l -1 as Σ l -1 = Σ 1 / 2 l -1 Σ ∗ / 2 l -1 a square-root factorization of the predicted covariance Π l = Π 1 / 2 l Π ∗ / 2 l is given by

and, hence, (Φ l Σ 1 / 2 l -1 Q 1 / 2 l ) is a matrix square-root of Π l . Similarly, the correction step can be carried out entirely on square-root factors. For more details, see Grewal and Andrews [19].

2.2 Dynamical low-rank approximation

Consider an initial value problem with a matrix-valued flow field F : R × R u × v → R u × v

where u and v are potentially very large. Dynamic low-rank approximation (DLRA) methods [27, 8] efficiently compute low-rank factorizations Y( t ) = U( t )D( t )V ∗ ( t ) ≈ M( t ) of Eq. (5) by solving

instead. The matrices U 0 ∈ R u × r , D 0 ∈ R r × r , and V 0 ∈ R v × r are an initial low-rank factorization of Y( t 0 ) ≈ M( t 0 ) . P r [Y( t )] denotes the projection operator onto the tangent space at Y( t ) , where Y( t ) lies in the manifold of rankr matrices. In the present work, we leverage this technique to obtain a low-rank factor Q 1 / 2 of the process-noise covariance at the next prediction location. Thus, we avoid much of the computational cost in the prediction step, otherwise caused by solving the full Lyapunov equation for Q in Eq. (3), for example using matrix-fraction decomposition [40, 3]. Concretely, let F (Q) = AQ + QA ∗ +BB ∗ . Then, an approximate low-rank process-noise covariance matrix associated with the prediction step is obtained by integrating

from t l -1 to t l . At the start of the filtering recursion t 0 , an initial low-rank factorization is constructed with a random orthogonal matrix U 0 and D 0 = 0 . For all subsequent steps the propagated orthogonal basis U( t l ) can be reused. In this work, all mentions of dynamic low-rank integration always refer to the recently developed, numerically stable basis update & Galerkin (BUG) integrator [8], whose error bounds are independent of small singular values. More details are given in Appendix C.

3 Rank-reduced Kalman filtering

In this section, a method for approximate inference in Eq. (1) is developed. The idea is to approximate the full filtering/smoothing covariance matrices of the latent state by an eigendecomposition truncated at the r -th largest eigenvalue. The main challenge is to thereby attain linear computational scaling in n and m under appropriate assumptions on the state-space model. Before proceeding with the filtering and smoothing recursions, it is instructive to examine inference in a static low-rank model.

3.1 Efficient inference in static models of low rank

Consider the following latent variable model

where Π 1 / 2 ∈ R n × r is of full column rank, r ≤ n , and R ∈ R m × m . Representing the latent state x as

reduces the problem to inference in the following model

An efficient inference scheme is given by the following proposition, which is proved in Appendix A.1. Proposition 1. Let z and y be two random variables governed by Eq. (10) . Assume r ≤ m and consider the following singular value decomposition (R -1 / 2 CΠ 1 / 2 ) ∗ = UDV ∗ , where U , D ∈ R r × r and V ∈ R m × r . Then, defining the whitened residual e = R -1 / 2 (y -C µ ) , we get

Furthermore, let | · | denote the matrix determinant. The marginal log-likelihood of y is given by

where

Moreover, a square-root of Σ is readily obtained by

3.2 The rank-reduced filtering recursion

In this section, a low-rank prediction-correction recursion is developed for the purpose of obtaining the filtering densities and the logarithm of the marginal likelihood.

The prediction equations Suppose the filtering distribution at time t l -1 is given by

where Σ 1 / 2 l -1 ∈ R n × r . We begin by detailing how to compute a low-rank factor Π 1 / 2 l of the predicted covariance. First compute the truncated singular value decomposition (SVD)

of the square-root factor (cf. Section 2.1), with ˜ U l ∈ R n × r , ˜ D l ∈ R r × r , and ˜ V l ∈ R r × r . This can be done in an optimal way by computing the full SVD and truncating it at the r -th largest singular value in O ( nr 2 ) [18]. The process-noise covariance factor Q 1 / 2 l ∈ R n × r in Eq. (17) is computed using DLRA as described in Section 2.2. The rank-reduced predicted moments at time t l are thus given as

The update equations The low-rank update equations follow from Proposition 1 and Corollary 1:

with the whitened residual e l = R -1 / 2 (y l -C µ -l ) and

for U l , D l ∈ R r × r and V l ∈ R m × r . The marginal predictive log-likelihood of y l is given by

Corollary 1-and by extension, the above filtering recursion-is only valid when r ≤ m , which is exactly the intended usecase for the proposed method. For settings in which m < r ≤ n the correction step follows the square-root correction of a Kalman filter and is detailed in Appendix B.

3.3 Time complexity of the rank-reduced filtering recursion

This section analyzes the computational complexity of the proposed method. In the worst case, the method scales quadratically in the state dimension n and measurement dimension m . Under favorable conditions, which are met in many real-world applications, the rank-reduced Kalman filter obtains a computational complexity of O ( nr 2 ) as stated by Proposition 2, which is proven in Appendix A.2. We begin by specifying a set of assumptions for Proposition 2.

. Then

The below corollary follows from the deterministic relationship between z and x given by Eq. (9). Corollary 1. Let y and x be two random variables governed by the model Eq. (8) and r ≤ m

Assumption 1. The maps x ↦→ Ax , x ↦→ Φx , and x ↦→ BB ∗ x can be evaluated in O ( n ) .

Assumption 1 is fulfilled naturally in many dynamical systems, in which local operators introduce sparsity into the dynamics, as, e.g., in finite-difference approximations of spatial differential operators. Another example is Kronecker structure in the system matrices arising in spatio-temporal GP regression by assuming the covariance function is separated into a product over the respective spatial and temporal components [35, 39, 20, 42].

Assumption 2. The map x ↦→ Cx can be evaluated in O ( m ) .

Assumption 3. The map x ↦→ R -1 / 2 x and the log-determinant log | R 1 / 2 | can be evaluated in O ( m ) .

Throughout this work, we refer to the situation in which Assumption 1 or 2 do not apply as the 'worst case'. Assumption 3 is taken for granted as the measurement-noise covariance R is often a diagonal matrix, which implies that sensor errors are uncorrelated. This is not only realistic but also commonly imposed in modelling. The above assumptions allow the following proposition.

Proposition 2. Given Assumptions 1 to 3, the proposed method approximates the filtering densities and the marginal likelihood at a cost of O ( nr 2 + mr 2 + r 3 ) .

The below corollary follows from the book-keeping in the proof of Proposition 2 in Appendix A.2.

Corollary 2. When Assumption 1 or Assumption 2 are not satisfied, the worst-case complexity of the proposed low-rank filtering recursion is O ( n 2 r + nm + m 2 r ) .

Overall, the proposed filtering recursions achieve the same asymptotic complexity as the existing ensemble methods under the same set of assumptions. However, our method is entirely deterministic.

3.4 The rank-reduced smoothing recursion

It remains to obtain a recursion for the smoothing densities, which can be shown to be computationally tractable given that the filtering recursion is tractable. The smoothing densities are denoted by

It can be shown that the posterior process has a backward Markov representation [5], hence the smoothing marginals may be obtained by

where we call b l,l +1 the backwards kernel . Consequently, the problem consists of approximating b l,l +1 such that the marginalization Eq. (25) may be implemented in a computationally frugal manner.

Approximating the backwards kernel For the linear Gaussian case, the backward kernel [46]

is parametrized by the smoothing gain G l ∈ R n × n , the shift vector v l ∈ R n , and the covariance of the backwards kernel P l ∈ R n × n . Let ( · ) + denote the Moore-Penrose pseudoinverse. Then

In the low-rank setting, the backwards kernel is efficiently approximated given the results from above. The approximate smoothing gain is a product of a tall, a small quadratic, and a wide matrix:

During filtering, Γ l is saved alongside the low-rank factors of the prediction and filtering covariances. We proceed to compute a low-rank representation of the backwards-transition covariance. Consider the following singular value decomposition, truncated at the r -th largest singular value,

2

51

100

RMSE

0

1

2

A.

vs. KF mean

2

51

100

Frobenius distance

0.0

0.5

1.0

B.

vs. KF cov.

low-rank dimension

10

77

144

RMSE

0

5

10

vs. KF mean

10

77

144

Frobenius distance

0.0

0.5

1.0

vs. KF cov.

low-rank dimension

RRKF

(ours)

EnKF

ETKF

(a) Linear advection dynamics (true rank = 51).

(b)

B.

London air-quality regression.

where ̂ U l ∈ R n × r , ̂ D l ∈ R r × r diagonal, ̂ V l ∈ R r × r , and G l as given in Eq. (28). Then a rankr factor of P l is given as

As for filtering, in the worst case, the cost for low-rank smoothing scales quadratically with the state dimension n . The following proposition, which is proved in Appendix A.3, states that this can be further reduced to linear complexity in n

, given the assumptions on the dynamics model are satisfied. Proposition 3. Given Assumption 1, computing the smoothing density p (x l | y 1: N ) costs O ( nr 2 + r 3 ) .

Finally, it is useful to mention that realizations of the backwards process are posterior samples.

4 Experiments

This section evaluates the proposed method in different experimental settings. The chosen measure of quality is the distance of the approximate low-rank moments to the exact KF. We measure the mean deviations with the root-mean-squared error (RMSE) and the covariance deviations with the time-averaged relative Frobenius distances. The presented rank-reduced Kalman filter (RRKF) is compared to the ensemble Kalman filter (EnKF) and the ensemble transform Kalman filter (ETKF). All EnKF and ETKF results are given in sample statistics over 20 runs. Section 4.1 shows that for a truly low-rank system, our method recovers the KF estimate up to numerical error as soon as r exceeds the true rank of the problem. Section 4.2 tests the method on a spatio-temporal GP regression problem with real data. After Section 4.3 evaluates the approximation quality for increasingly lowrank systems in a controlled experimental environment, Section 4.4 verifies the stated asymptotic cost of the method. Finally, a large-scale spatio-temporal regression problem is solved in Section 4.5. In all experiments, we compute the stationary mean and a low-rank factorization of the stationary covariance matrix of the prior and condition the stationary moments on the first measurement in the respective time-series dataset. More details on the experimental setups are given in Appendix D.

4.1 Linear advection model

The proposed algorithm is evaluated in a standard data-assimilation setup in which linear-advection dynamics with periodic boundary conditions are assimilated to a set of simulated data. The setup follows the description in Sakov and Oke [33] and the experimental details are additionally detailed in Appendix D. This problem has rank 51 by construction. Figure 2a shows how the exact KF estimate is recovered by our method for r = 51 , while the ensemble methods converge according to a Monte-Carlo rate. This is particularly clear for the covariance estimate (Fig. 2a, B. ).

4.2 London air-quality regression

This experiment uses hourly data from the London air-quality network [23] between January 2019 and April 2019, which amounts to measurements at 72 spatial locations and 2159 points in time. The

A.

RMSE

0.0

0.5

1.0

ℓ

x

=0.01

low-rank dim.

1

111

221

331

441

distance

0.0

0.5

1.0

0.0

0.5

1.0

ℓ

x

=0.1

low-rank dim.

1

111

221

331

441

0.0

0.5

1.0

0.0

0.5

1.0

ℓ

x

=0.25

low-rank dim.

1

111

221

331

441

0.0

0.5

1.0

0.0

0.5

1.0

ℓ

x

=1.0

low-rank dim.

1

111

221

331

441

0.0

0.5

1.0

RRKF (ours)

EnKF

ETKF

cumulative spectrum

A.

state dimension n

10

2

10

3

10

4

10

5

wall time [s]

10

-2

10

-1

10

0

10

1

10

2

Assumptions 1 to 3

state dimension n

10

2

10

3

10

4

wall time [s]

10

-2

10

-1

10

0

10

1

10

2

Assumptions 2 & 3

RRKF

(ours)

EnKF

ETKF

𝓞

(

n

)

𝓞

(

n

2

)

𝓞

(

n

3

)

𝓞

(

n

)

𝓞

(

n

2

)

𝓞

(

n

3

)

B.

data used, together with its processing (except the train-test split), is the same as in the corresponding experiment by Hamelijnck et al. [20]. The model is a spatio-temporal Matérn- 3 /2 process with prior hyperparameters that maximize the marginal log likelihood. Figure 2b shows that-compared to the ensemble methods-our algorithm is consistently closer to the KF estimate and recovers it at r = n .

4.3 Spatio-temporal Matérn process with varying spatial lengthscale

Consider a spatio-temporal Gaussian process x( t ) ∼ GP (0 , k t ⊗ k x ) with covariance structure that is separable in time and space. Let both k t and k x be of the Matérn family with characteristic spatial and temporal lengthscales ℓ t , ℓ x and output scales σ 2 t , σ 2 x . Spatio-temporal Matérn processes can be translated to the formulation in Eq. (1a), as detailed, for instance, by Solin [39]. This setting allows for varying the 'low-rankness' by the choice of ℓ x : the larger ℓ x , the more information about a spatial point is prescribed by its surroundings. As ℓ x approaches zero, the spatial kernel matrix gets more and more diagonal, encoding spatially independent states, and causing a barely decaying spectrum. The spectrum for large ℓ x decays rapidly, making low-rank approximations more accurate for small r .

In the experiment, we consider a spatial domain [0 , 2] × [0 , 2] ⊂ R 2 , which is subsampled at a uniformly-spaced grid with ∆ x = 0 . 1 ( n = 21 × 21 = 441 ). Noisy observations of the full state trajectory, are drawn from a realization of the prior process. Figure 3 demonstrates that, for increasing spatial lengthscales, all methods converge faster to the true KF estimate. Crucially, however, our method is consistently closer to the optimal estimate and recovers it for r = n .

0

25

50

0

25

50

-6

-3

0

3

6

30 Jan 2023

Data

Mean

Z-score

0

20

40

0

20

40

-6

-3

0

3

6

19 Feb 2023

0

20

40

60

80

0

20

40

60

80

-6

-3

0

3

6

04 Mar 2023

4.4 Runtime

To demonstrate the asymptotic computational complexities given in Section 3.3, we investigate the runtimes on two different settings: The first fulfills both Assumptions 1 and 2, whereas the second only fulfills Assumption 2, which means that the cost of prediction is quadratic in the state dimension. Both problems are solved on a temporal grid of size N = 100 with the measurement dimension fixed at m = 100 and the low-rank dimension fixed at r = 5 .

The problem from Section 4.1 serves as the best-case setting. The noise-free linear-advection dynamics amount to multiplication of the state with a circulant matrix, which is implemented efficiently using fast Fourier transform. The measurement operator amounts to array indexing. As subplot A. in Fig. 4 confirms, the computational cost grows linearly with the state dimension. As a worst-case example we employ a spatio-temporal Matérn process with a dense spatial kernel matrix (cf. Section 4.3). In this case, we expect the computation time to scale quadratically with the state dimension because BB ∗ is dense. This is verified by the right plot in Fig. 4 ( B. ).

4.5 Large-scale spatio-temporal GP regression on rainfall data

As a final experiment, a spatio-temporal GP regression problem is solved on a large gridded analysis data set measuring rainfall in Australia. The data is provided by the Australian Water Availability Project (AWAP) [2, 24]. The number of spatial data points is n = 30911 , which rules out the cubic-inn Kalman filter. The smoothing posterior is computed at N = 40 time points, from 25 January, 2023 through 5 March, 2023. The total number of observations is thus n · N = 1236440 . The spatio-temporal prior Matérn model is selected by maximizing the marginal log-likelihood of a lower-resolution data set. A set of 6160 spatial points is excluded from the data during filtering and smoothing in order to evaluate spatial interpolation performance. Thus, the measurement dimension is m = n -6160 = 24751 . Figure 5 shows the low-rank approximation to the GP posterior with r = 1000 low-rank dimensions. The non-observed spatial locations are framed in the horizontal-'S' shape. At measurement locations, the data is described well by the model, while a slight smoothingout effect can be observed due to the high-frequency features of the model being truncated. At the evaluation points the interpolation aligns with the expectations regarding the simple Matérn model.

The Z-score map shows ξ l -y l λ l , where ξ l and λ l denote the smoothing mean and marginal standard deviation and y l the data point, at time t l . This highlights badly calibrated uncertainty estimates, wherever the model under-/overestimates the data and is divided by a small standard deviation, in blue/red. The Z-score distribution receives further attention in Appendix E. All in all, the RRKF achieves high-quality approximate estimates while compressing the state by a factor of 30 911 1000 ≈ 30 .

5 Limitations

The main limitation of the proposed approximate probabilistic inference scheme is that the truncation to r dimensions cuts away covariance information, which is not accounted for. As a result, confidence grows as r shrinks, while estimates should arguably get more uncertain instead with increased compression. In existing low-rank filters, this issue is often counteracted manually by inflating the covariance matrices, on a per-application basis [6, Section 4.4] and calls for a more principled treatment. Finding a way to keep track of that residual uncertainty information is beyond the scope of this paper. It remains unclear how to preserve the stated asymptotic complexities while doing so.

Further, all results herein are stated for linear dynamics and observation models. Extensions to non-linear models could include linearization of the according transitions, or cubature methods [34].

6 Conclusion

By building upon well-established knowledge about optimal compression, we have proposed an algorithm providing a principled way to balance approximation accuracy against computational cost in high-dimensional state estimation. It combines simple (truncated) singular value decompositions with recent numerical low-rank integrators of large matrix differential equations. We have offered both theoretical and empirical arguments for why it is desirable to use a deterministic algorithm when approximating large-scale probabilistic state-estimation problems.

Acknowledgments and Disclosure of Funding

The authors gratefully acknowledge financial support by the German Federal Ministry of Education and Research (BMBF) through Project ADIMEM (FKZ 01IS18052B), and financial support by the European Research Council through ERC StG Action 757275 / PANAMA; the DFG Cluster of Excellence 'Machine Learning - New Perspectives for Science', EXC 2064/1, project number 390727645; the German Federal Ministry of Education and Research (BMBF) through the Tübingen AI Center (FKZ: 01IS18039A); and funds from the Ministry of Science, Research and Arts of the State of Baden-Württemberg. Jörg Nick is funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) - Project-ID 258734477 - SFB 1173. Filip Tronarp was partially supported by the Wallenberg AI, Autonomous Systems and Software Program (WASP) funded by the Knut and Alice Wallenberg Foundation. The authors thank the International Max Planck Research School for Intelligent Systems (IMPRS-IS) for supporting Jonathan Schmidt. The authors also thank Nathanael Bosch and Christian Lubich for many valuable discussions and for helpful feedback.

References

Adam, V., Eleftheriadis, S., Artemev, A., Durrande, N., and Hensman, J. (2020). Doubly sparse variational Gaussian processes. In Chiappa, S. and Calandra, R., editors, Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics , volume 108 of Proceedings of Machine Learning Research , pages 2874-2884. PMLR.

Australian Data Archive for Meteorology (ADAM) (2023). Australian Water Availability Project. http://www.bom.gov.au/jsp/awap/index.jsp . [Online; accessed May 2023].

Axelsson, P. and Gustafsson, F. (2015). Discrete-time solutions to the continuous-time differential Lyapunov equation with applications to Kalman filtering. IEEE Transactions on Automatic Control , 60(3):632-643.

Bishop, C. H., Etherton, B. J., and Majumdar, S. J. (2001). Adaptive sampling with the ensemble transform Kalman filter. part i: Theoretical aspects. Monthly weather review , 129(3):420-436.

Cappé, O., Moulines, E., and Ryden, T. (2005). Inference in Hidden Markov Models . Springer Series in Statistics. Springer.

Carrassi, A., Bocquet, M., Bertino, L., and Evensen, G. (2018). Data assimilation in the geosciences: An overview of methods, issues, and perspectives. Wiley Interdisciplinary Reviews: Climate Change , 9(5).

Carron, A., Todescato, M., Carli, R., Schenato, L., and Pillonetto, G. (2016). Machine learning meets Kalman filtering. In 2016 IEEE 55th conference on decision and control (CDC) , pages 4594-4599. IEEE.

Ceruti, G. and Lubich, C. (2022). An unconventional robust integrator for dynamical low-rank approximation. BIT. Numerical Mathematics , 62(1):23-44.

Chada, N. K., Stuart, A. M., and Tong, X. T. (2020). Tikhonov regularization within ensemble Kalman inversion. SIAM Journal on Numerical Analysis , 58(2):1263-1294.

Cressie, N., Shi, T., and Kang, E. L. (2010). Fixed Rank Filtering for Spatio-Temporal Data. Journal of Computational and Graphical Statistics , 19(3):724-745.

Evensen, G. (1994). Sequential data assimilation with a nonlinear quasi-geostrophic model using Monte Carlo methods to forecast error statistics. Journal of Geophysical Research: Oceans , 99(C5):10143-10162.

Evensen, G. (2009a). Data assimilation: the ensemble Kalman filter , volume 2. Springer.

Evensen, G. (2009b). The ensemble Kalman filter for combined state and parameter estimation. IEEE Control Systems Magazine , 29(3):83-104.

Evensen, G. (2021). Formulating the history matching problem with consistent error statistics. Computational Geosciences , 25(3).

Farrell, B. F. and Ioannou, P. J. (2001). State estimation using a reduced-order kalman filter. Journal of the Atmospheric Sciences , 58(23):3666-3680.

Galka, A., Yamashita, O., Ozaki, T., Biscay, R., and Valdés-Sosa, P. (2004). A solution to the dynamical inverse problem of EEG generation using spatiotemporal Kalman filtering. NeuroImage , 23(2):435-453.

Ghil, M. and Malanotte-Rizzoli, P. (1991). Data assimilation in meteorology and oceanography. In Advances in geophysics , volume 33, pages 141-266. Elsevier.

Golub, G. H. and Van Loan, C. F. (2013). Matrix computations . JHU press.

Grewal, M. and Andrews, A. (2011). Kalman Filtering: Theory and Practice Using MATLAB . Wiley.

Hamelijnck, O., Wilkinson, W., Loppi, N., Solin, A., and Damoulas, T. (2021). Spatio-temporal variational Gaussian processes. Advances in Neural Information Processing Systems , 34:2362123633.

Hamill, T. M. and Snyder, C. (2000). A hybrid ensemble Kalman filter-3D variational analysis scheme. Monthly Weather Review , 128(8):2905-2919.

Hensman, J., Fusi, N., and Lawrence, N. D. (2013). Gaussian processes for big data. In Proceedings of the 29th Conference on Uncertainty in Artificial Intelligence (UAI) , pages 282-290.

Imperial College London (2020). Londonair - London air quality network (LAQN). https: //www.londonair.org.uk .

Jones, D., Wang, W., and Fawcett, R. (2009). High-quality spatial climate data-sets for Australia. Australian Meteorological and Oceanographic Journal , 58(04):233-248.

Kalman, R. E. (1960). A new approach to linear filtering and prediction problems. Transactions of the ASME-Journal of Basic Engineering , 82(Series D):35-45.

Katzfuss, M., Stroud, J. R., and Wikle, C. K. (2016). Understanding the ensemble Kalman filter. The American Statistician , 70(4):350-357.

Koch, O. and Lubich, C. (2007). Dynamical low-rank approximation. SIAM Journal on Matrix Analysis and Applications , 29(2):434-454.

Loiseau, J.-C., Noack, B. R., and Brunton, S. L. (2018). Sparse reduced-order modelling: sensor-based dynamics to full-state estimation. Journal of Fluid Mechanics , 844:459-490.

Lubich, C. and Oseledets, I. V. (2014). A projector-splitting integrator for dynamical low-rank approximation. BIT Numerical Mathematics , 54(1):171-188.

Netto, M. and Mili, L. (2018). A robust data-driven koopman kalman filter for power systems dynamic state estimation. IEEE Transactions on Power Systems , 33(6):7228-7237.

Rahimi, A. and Recht, B. (2007). Random features for large-scale kernel machines. In Advances in Neural Information Processing Systems (NeurIPS) .

Roth, M., Hendeby, G., Fritsche, C., and Gustafsson, F. (2017). The ensemble Kalman filter: a signal processing perspective. EURASIP Journal on Advances in Signal Processing , 2017(1):1-16.

Sakov, P. and Oke, P. R. (2008). A deterministic formulation of the ensemble Kalman filter: An alternative to ensemble square root filters. Tellus A: Dynamic Meteorology and Oceanography , 60(2):361.

Särkkä, S. (2013). Bayesian Filtering and Smoothing . Cambridge University Press.

Särkkä, S. and Hartikainen, J. (2012). Infinite-dimensional Kalman filtering approach to spatiotemporal Gaussian process regression. In Artificial Intelligence and Statistics , pages 993-1001. PMLR.

Särkkä, S. and Solin, A. (2019). Applied Stochastic Differential Equations . Cambridge University Press.

Särkkä, S., Solin, A., and Hartikainen, J. (2013). Spatiotemporal learning via infinitedimensional Bayesian filtering and smoothing: A look at Gaussian process regression through Kalman filtering. IEEE Signal Processing Magazine , 30(4):51-61.

Snelson, E. and Ghahramani, Z. (2005). Sparse Gaussian processes using pseudo-inputs. Advances in Neural Information Processing Systems (NeurIPS) , 18:1257-1264.

Solin, A. (2016). Stochastic Differential Equation Methods for Spatio-Temporal Gaussian Process Regression . Doctoral thesis, Aalto University, School of Science.

Stengel, R. (1994). Optimal Control and Estimation . Dover Publications.

Stuart, A. M. (2010). Inverse problems: a Bayesian perspective. Acta numerica , 19:451-559.

Tebbutt, W., Solin, A., and Turner, R. E. (2021). Combining pseudo-point and state space approximations for sum-separable Gaussian processes. In de Campos, C. and Maathuis, M. H., editors, Proceedings of the Thirty-Seventh Conference on Uncertainty in Artificial Intelligence , volume 161 of Proceedings of Machine Learning Research , pages 1607-1617. PMLR.

Tippett, M. K., Anderson, J. L., Bishop, C. H., Hamill, T. M., and Whitaker, J. S. (2003). Ensemble square root filters. Monthly weather review , 131(7):1485-1490.

Titsias, M. (2009). Variational learning of inducing variables in sparse Gaussian processes. In Proceedings of the International Conference on Artificial Intelligence and Statistics (AISTATS) , pages 567-574. PMLR.

Todescato, M., Carron, A., Carli, R., Pillonetto, G., and Schenato, L. (2020). Efficient spatiotemporal Gaussian regression via Kalman filtering. Automatica , 118:109032.

Tronarp, F., Bosch, N., and Hennig, P. (2022). Fenrir: Physics-enhanced regression for initial value problems. In Chaudhuri, K., Jegelka, S., Song, L., Szepesvari, C., Niu, G., and Sabato, S., editors, Proceedings of the 39th International Conference on Machine Learning , volume 162 of Proceedings of Machine Learning Research , pages 21776-21794. PMLR.

Vijayshankar, S., Nabi, S., Chakrabarty, A., Grover, P., and Benosman, M. (2020). Dynamic mode decomposition and robust estimation: Case study of a 2d turbulent boussinesq flow. In 2020 American Control Conference (ACC) , pages 2351-2356. IEEE.

Williams, C. and Seeger, M. (2001). Using the Nyström method to speed up kernel machines. In Advances in Neural Information Processing Systems (NeurIPS) .

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper develops a method for performing approximate Kalman filtering and smoothing using a low-rank approximation.  Fully low-rank predict, update and smoothing equations are derived.  Under certain reasonable assumptions, these equations reduce the cubic complexity to linear complexity (and quadratic if the assumptions are violated), while being “approximately exact” compared to Monte Carlo-based ensemble methods.  The method is evaluated on a range of problems and appears to perform favourably.  Sensible limitations and future opportunities are highlighted.\n\nSTRENGTHS:\nThis paper addresses a critical issue – the super-linear (cubic) complexity of exact Kalman filtering/smoothing (KF/S).  This makes stable and closed-form estimation intractable for anything other than small problems.  The proposed solution is appealing: introduce a low-rank factorization that, through some linear algebra and recent solvers, is linear (or quadratic) in the rank.  While low-rank factorizations are common, I have not seen it applied to KF/S, a domain where it is very naturally appealing.  \n\nThe detail in which the authors walk through the derivation is excellent, and appears to be mathematically sound (although I am not super familiar with this application domain).  The purely theoretical contributions of the paper are not huge; but are well-judged, interesting, apt for the problem at hand (modulo Limitations), and dovetail nicely with recent literature.  \n\nThe experimental evaluation touches on a number of interesting examples that are high dimensional, and for which the proposed method seems to excel.  The final application, although a little off-the-wall, really does a good job in highlighting how high-dimensional this method can go.  Other experiments do a good job of comparing asymptotic to non-asymptotics in a range of domains.  \n\nI think there is definitely an audience at NeurIPS that will enjoy and really benefit from this work.  I also believe this work will engender follow-up work which will further build this direction.  The paper is on the whole well-written (modulo Weakness B) and is nicely visually presented.  Figures are very well prepared, with clearly legible fonts and clear colours.\n\nWEAKNESSES:\nThe paper, on the whole, is very good, but I do think there are some areas for improvement.\n- A:  The experimental evaluation is, on the whole, good.  Figure 1 is excellent in visually depicting how covariance matrices can be low rank.  WIth that in mind, I wonder if there is a simple, low-dimensional model that the authors can present that can be easily visualised.  I don’t have a great “feel” for any of the examples presented.  The results for this experiment certainly don’t need to be SoTA (since existing methods will probably also perform well).  Some visualisations (PCA projections of state, low-rank factors etc) can then be visualised to highlight the differences between methods and build reader intuition.  This example could be as simple as a synthetic LDS with dimension ~10.  Note:  I wouldn’t expect to see this during the rebuttal period, but I would _strongly_ encourage the authors to consider including it in a camera ready.\n- B:  [I am aware this is a very general and somewhat hard-to-address comment, so please take these more as suggestions]  The paper itself is very notationally dense, to the point of obfuscating the simplicity and applicability of the method.  While the authors are to be commended for their attention to detail, I think parts of the paper can be thinned out, with more technical content being dropped to the supplement.  This will result in a paper that is more immediately digestible, and will have a larger reach and impact.  Some examples:\n  -  I think Sec 3.1 is great for building reader intuition, but this intuition is then lost a little bit by having to wade through long math equations, where the details aren’t actually super important.  \n  - I  wonder if some of the smoothing equations can be cut to the supplement, with just the final forms retained and note directing the interested reader to the supplement.  This newly cleared space can then be used for including higher-level sketch proofs, intuition-building worked examples or text, algorithm blocks, diagrams, experiment visualisations etc. \n  - Including a banner figure (and maybe even an algorithm block) explaining the whole method would be nice, just to visually complement the math.  \n  - Maybe consider including a table outlining the positives and negatives of each approach under particular operating conditions?  Then the benefits are stated clearly and concisely outside of the body text.  \n  - **The key takeaway** is that you have already solved the hard equations and derived an elegant solution!  Therefore, I would encourage you to strip out as much complexity from the main text as possible, and add extra emphasis to the end result – even something as cartoonish as a callout box with the key equations (akin to how Sarkka has a full equation block per model/inference methodology). \n\n## Minor weaknesses:\n- It would be nice if the DLRA/BUG integrator/solver were introduced in the main text.  You highlight it as a key component, but there is not really much discussion of its implementation, assumptions, limitations, complexity etc.  \n- I don't quite understand the $(\\Phi \\Sigma \\quad Q)$ notation in (16) (specifically the white space between the matrices).  Please explicitly define it somewhere. \n\nIf these changes (and the limitations)  were made/commented on, then I would consider upgrading my review.\n\nQUESTIONS:\nAlso see weaknesses and limitations.\n\nIs operating in the continuous time domain strictly necessary?  My understanding is that a lot of complexity (at least notationally) goes away when you consider discrete time/equally spaced observations.  I think all of your experiments used equally spaced observations as well.  It could be simpler if you present the material in the discrete time domain, and then have a paragraph outlining that this also extends to the continuous time/SDE domain.\n\nLIMITATIONS:\nA limitation is that the authors do not consider bounding the error introduced by the low-rank approximation.  For instance, the LML in (12) is for the approximate model.  There is no comment on the approximation gap between the approximate model and the full-rank model.  I am not fully sure on this, but I think it is possible to construct a bound based on the truncated parts of the spectrum of the matrix being truncated.  If this isn’t possible, some qualitative discussion would help reinforce this point.  If you were able to construct a bound – any bound! – on the error, then that would put a nice bow on the approximation, as opposed to relying on evaluating the error empirically.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes an approximation method for Kalman filtering of intractable, high-dimensional Gauss-Markov settings with linear observations. It adopts existing DLRA methods to compute a discrete-time representation of the sampled process and suggests a recursive filtering and smoothing methods in a low-rank formulation.\n\nSTRENGTHS:\nTo the best of my knowledge, the proposed filtering scheme is novel and is shown to outperform existing EnKFs under the same configurations. As opposed to EnKF approximation methods, this method is deterministic and capable of accurately restoring the optimal filter given the right parameterization. \n\nDerivation of the recursive filter seems correct, and filtering is done in a square-root fashion, which is stable and efficient. The recursion step is shown to be efficient (although under very strong assumptions). The method additionally suggests an efficient calculation for the observation likelihood.\n\nWEAKNESSES:\nThe method assumes the low rank $r$, which is a drawback of many low-rank approximation methods. Even if it is beyond the scope of the paper, there should be a short discussion about choosing this parameter or mention this as a limitation.\n\nIn Figs. 2, 3 the RRKF is compared with two other methods, both are Monte-Carlo, and are not expected to restore the exact filter. A righteous comparison should include some deterministic algorithms e.g. [10], [r1] below. \n\n[r1] MLA\tFarrell, Brian F., and Petros J. Ioannou. \"State estimation using a reduced-order Kalman filter.\" Journal of the Atmospheric Sciences 58.23 (2001): 3666-3680\n\nQUESTIONS:\n1. The recursive methods assumes knowing $\\Sigma_{l-1}$ at each step, but don't mention how to initialize $\\Sigma_{1}$ at $t_1$. Is it just $Q_{t_1}$?\n\n2. The estimations on Rainfall in Australia regression (Fig.5) don't seem very accurate, especially at spatial points where data deviates from the mean value. Authors claim that \"the RRKF achieves high-quality approximate estimates\", but I'm not convinced. Can they support this claim quantitively?\n\nMinor comments:\n\nEqn. (4) and (16) are somewhat unclear. It would be helpful to clearly state matrices dimensions.\n\nIn l.146 it is stated that truncated SVD complexity is $O(nr^2)$, which seems to be right. But can you explain/reference this claim?\n\nLIMITATIONS:\nThe method assumes the low rank $r$. This limitation is not discussed.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-02"}

comments: SUMMARY:
The paper presents a deterministic low-rank approximation of the Kalman filter for estimation in high-dimensional linear Gaussian setting. This setting is important for many problems that arise in meteorology and solved by Ensemble Kalman Filter (EnKF). The paper presents formula for low rank approximation of the covariance for both prediction and correction steps and demonstrates the proposed approach in different scenarios in comparison with EnKF.

STRENGTHS:
- paper is well-written, has clear message
- it considers an important problem in meteorology
- the proposed approach is motivated well 
- comprehensive numerical experiments are given

WEAKNESSES:
- the main weakness is the fact that in most high-dimensional data assimilation problems, one has has only access to a simulator for the forward dynamic update. It is not clear how the proposed approach perform low rank approximation in that case. In EnKF, one simulates each member of the ensemble according to the dynamic model
- the numerical code for reproducing the results are not given
- the relevance to NeurIPS audience is weak.

QUESTIONS:
1- What is a reference for stochasticity in EnKF introduces unfavorable properties. It will be good to be more concrete here, and explain what are exactly those unfavorable properties, since it is a central motivation for your approach. 

2- The computational time plot in figure 4 is a bit surprising. How is that low rank approximation has smaller computational time compared to EnKF?  

3- Can you produce a figure that demonstrates error as a function of raw computational time? 

4- What is the reference for O(nr^2) complexity of truncated SVD?

LIMITATIONS:
yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-08

comments: SUMMARY:
The paper proposes a deterministic low rank approximation to the Kalman filter in high dimensional settings where the computational complexity of the full Kalman filter is very expensive. They compare to the ensemble Kalman filter and ensemble transform Kalman filter.

STRENGTHS:
The paper is well written.

WEAKNESSES:
See questions. The biggest issue is that the comparisons are not against “fair” competing methods (any low rank Kalman filter). They also do not mention anything about the large swaths of low rank variants of Kalman filters.

QUESTIONS:
The scenario for this problem is linear dynamic systems? Ensemble style Kalman filters are often used in non-linear dynamic systems and have the greatest advantage there; including computational ones, but that is compared to particle filters etc. . The authors cite using ensemble Kalman filters for computational cheapness, but the vast majority of those papers are from not very good journals and their accuracy is suspect...

In contrast the real data experiments are for atmospheric problems that generally have non-linear dynamics? 

How are you computing the low rank dimension of the ensemble Kalman filter, which is not a low rank method (figure 2)? Neither the ensemble Kalman filter nor the ensemble transform Kalman filter are explicitly attempting to estimate a low rank approximation. It would be more appropriate to compare against a variant of the Kalman filter that is a low rank approximation (or a sparse one at least). There are tons of variants from just a quick google search.

LIMITATIONS:
Yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-25

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nFiltering (estimation) and smoothing of large-dimensional state-space models are computationally challenging. Exact Kalman filtering, at least insofar as LTI SDEs are considered, is a mature solution for such tasks but the cubic complexity of such methods makes it computationally intractable. This paper proposes an approximate Gaussian filtering and smoothing method that propagates low-rank approximations of the covariance matrices. To enable this, i) Lyapunov equations are projected into a manifold of lower rank in the prediction step combined with ii) square root filtering, which together offer numerically stable and tractable solution. Four examples are provided, with increasing complexity, and in one case runtime analysis is also carried out to support the claims. Two measures are used RMSE and covariance deviation to assess the method’s performance. Corollary 1 provides a “Kalman-like” result (an important contribution of the paper) while Section 3.2 discusses the algorithm of the method for filtering (3.4 for smoothing) and section 3.3 provides the time complexity of the method.\n\nSTRENGTHS:\nThe proposed method differs from existing ensemble-based approaches in that the low-rank approximations of the covariance matrices are deterministic, rather than stochastic. This is important because it allows the method to reproduce the exact Kalman filter as the low-rank dimension approaches the true dimensionality of the problem.\n\nThe time complexity analysis is well-written, and authors clearly demonstrate their method reduces computational complexity from cubic (for the Kalman filter) to quadratic in the state-space size at most (and linear in some cases—refer to proposition 3 and assumptions therein).\n\nWEAKNESSES:\nMy major issue with this paper is there is a large body of work on estimation of high- and infinite-dimensional dynamical systems and this paper does not position itself in such context. There are numerous works on using sparse identification (Sparse reduced-order modeling: Sensor-based dynamics to full-state estimation), Koopman-based (A Robust Data-Driven Koopman Kalman Filter for Power Systems Dynamic State Estimation), balanced-truncation ROMs and Kalman (State Estimation Using a Reduced-Order Kalman Filter), DMD-based ROM and Kalman (Dynamic mode decomposition and robust estimation: Case study of a 2d turbulent boussinesq flow), etc. In above-mentioned works, usually the high-dimensional governing equations is first projected in some appropriate basis (low-rank), and then an appropriate filter is deigned. Once estimation is done, the reconstruction is projected back into the physical domain. This also enables the application of the method to i) nonlinear dynamical system, ii) unknown systems. Such approaches benefit from lower computational cost, similar to the proposed method in the paper. In a sense authors acknowledge, more or less, such line of work in the last paragraph of Section 1. However, when they write “In contrast to our work, their method assumes that the dynamics unfold entirely in a lower-dimensional space, and conditioning on measurements happens by transporting the low-dimensional diffusive process to the full space by a projection that is also assumed to be known” there is no quantitative or qualitative comparison to support their claim. Let’s say for example 1 considered in this paper, authors first project the PDE into low rank ODE, and then design the standard Kalman filter in such a manifold, and once estimation is propagated in time, the low dimension is converted back into actual space. What are the advantages and disadvantages? Please note I am not simply asking for more comparison (in addition to, say, EnKF) but more a conceptual question of what the real advantage of the proposed method is to state of the art for ROM-based estimation. I’m sure authors are aware of works like “Krylov subspace methods for solving large Lyapunov equations” where large scale Lyapunov equations are solved with efficient algorithms. I hope authors clarify their novelties in the rebuttal.\n\nQUESTIONS:\nIn section 1, Please be more specific about such unfavorable properties (line 45). Overall, the issue of GP is not discussed clearly in this section and elsewhere in the manuscript.\n\nAny insights if such approach can be applied to nonlinear systems? What if the dynamics is unknown (A and B)?\n\nI'd suggest moving section 4.4 as either last section or right after 4.1. Currently the flow of the results section is hard to follow: first three examples are given to demonstrate the accuracy of the model with the given two measures, then a discussion of runtime and finally again another result for rainfall data.\n\nIt would be helpful to include a discussion on the impact of snapshots noise (measurement noise) on the results.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-27"}