SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Shahriar Talebi $^{1,2}$  Amirhossein Taghvaei $^{1}$  Mehran Mesbahi $^{1}$
<sup>1</sup>University of Washington, Seattle, WA, 98105
$^{2}$ Harvard University, Cambridge, MA, 02138
talebi@seas.harvard.edu amirtag@uw.edu mesbahi@uw.edu
This paper examines learning the optimal filtering policy, known as the Kalman gain, for a linear system with unknown noise covariance matrices using noisy output data. The learning problem is formulated as a stochastic policy optimization problem, aiming to minimize the output prediction error. This formulation provides a direct bridge between data-driven optimal control and, its dual, optimal filtering. Our contributions are twofold. Firstly, we conduct a thorough convergence analysis of the stochastic gradient descent algorithm, adopted for the filtering problem, accounting for biased gradients and stability constraints. Secondly, we carefully leverage a combination of tools from linear system theory and high-dimensional statistics to derive bias-variance error bounds that scale logarithmically with problem dimension, and, in contrast to subspace methods, the length of output trajectories only affects the bias term.
The duality of control and estimation plays a crucial role in system theory, linking two distinct synthesis problems [1, 2, 3, 4, 5, 6]. This duality is an effective bridge between two distinct disciplines, facilitating development of theoretical and computational techniques in one domain and then adopting them for use in the other. For example, the stability proof of the Kalman filter relies on the stabilizing characteristic of the optimal feedback gain in the dual Linear Quadratic Regulator (LQR) optimal control problem [7, Ch. 9]. In this paper, we leverage this duality to learn optimal filtering policies using recent advances in data-driven algorithms for optimal control.
We consider the estimation problem for a system with a known linear dynamic and observation model, but unknown process and measurement noise covariances. Our objective is to learn the optimal steady-state Kalman gain using a training dataset comprising independent realizations of the observation signal. This problem has a rich history in system theory, often explored within the context of adaptive Kalman filtering [8, 9, 10, 11, 12, 13]. A comprehensive summary of four solution approaches to this problem can be found in the classical reference [9]. These approaches include Bayesian inference [14, 15, 16], Maximum likelihood [17, 18], covariance matching [12], and innovation correlation methods [8, 10]. While Bayesian and maximum likelihood approaches are computationally intensive, and covariance matching introduces biases in practice, the innovation correlation-based approaches have gained popularity and have been the subject of recent research [19, 20, 21]. For an excellent survey on this topic, refer to the article [22]. However, it is important to note that these approaches often lack non-asymptotic guarantees and heavily depend on statistical assumptions about the underlying model.
In the realm of optimal control, significant progress has been made in the development of data-driven synthesis methods. Notably, recent advances have focused on the adoption of first-order methods for state-feedback LQR problems [23, 24]. The direct optimization of policies from a gradient-dominant
perspective has first proven in [25] to be remarkably effective with global convergence despite non-convex optimization landscape. It has been demonstrated that despite the non-convex nature of the cost function, when expressed directly in terms of the policy, first-order methods exhibit global convergence to the optimal policy. Building upon this line of work, the use of first-order methods for policy optimization has been explored in variants of the LQR problem. These include Output-feedback Linear Quadratic Regulators (OLQR) [26], model-free setup [27], risk-constrained setup [28], Linear Quadratic Gaussian (LQG) [29], and most recently, Riemannian constrained LQR [30]. These investigations have expanded the scope of data-driven optimal control, demonstrating the versatility and applicability of first-order methods for a wide range of synthesis problems.
The objective of this paper is to provide fresh insights into the classical estimation problem by leveraging the duality between control and estimation and incorporating recent advances in data-driven optimal control. Specifically, building on the fundamental connection between the optimal mean-squared error estimation problem and the LQR problem (Prop. 1), we reformulate determining the optimal Kalman gain as a problem of synthesizing an optimal policy for the adjoint system, under conditions that differ from those explored in the existing literature (see (10) and Remark 3). Upon utilizing this relationship, we propose a Stochastic Gradient Descent (SGD) algorithm for learning the optimal Kalman gain, accompanied by novel non-asymptotic error guarantees in presence of biased gradient and stability constraint. Our approach opens up promising avenues for addressing the estimation problem with robust and efficient data-driven techniques. The following is an informal statement of our main results (combination of Thm. 1 and Thm. 2), and missing proofs appear in the supplementary materials.
Theorem 3 (Informal). Suppose the system is observable and both dynamic and measurement noise are bounded. Then, with high probability, direct policy updates using stochastic gradient descent with small stepsize converges linearly and globally (from any initial stabilizing policy) to the optimal steady-state Kalman gain.
More recently, the problem of learning the Kalman gain has been considered from a system identification perspective, for completely unknown linear systems [31, 32, 33, 34]. In [32] and [33], subspace system identification methods are used to obtain error bounds for learning the Markov parameters of the model over a time horizon and establish logarithmic regret guarantee for output prediction error. Due to the inherent difficulty of learning a completely unknown stochastic system from partial observations, subspace methods assume marginal stability of the unknown system, and lead to sub-optimal sample complexity bounds that grow with the number of Markov parameters, instead of the number of unknowns [35, pp. 14]. Alternatively, [34] considers minimizing the output prediction error and introduces a model-free policy gradient approach, under the same stability assumptions, that achieves sublinear convergence rate. This paper provides a middle ground between completely known and completely unknown systems, for a learning scenario that not only has relevant practical implications, but also utilizes the duality relationship to LQR to establish linear convergence rates even for unstable systems as long as they are observable. See the Appendix A for the discussion of additional related works [36, 37, 38].
Herein, first we propose the model setup in detail and discuss the Kalman filter as the estimation strategy. Consider the discrete-time filtering problem given by the stochastic difference equations,
$$
x (t + 1) = A x (t) + \xi (t), \quad \text {a n d} \quad y (t) = H x (t) + \omega (t), \tag {1}
$$
where  $x(t) \in \mathbb{R}^n$  is the state of the system,  $y(t) \in \mathbb{R}^m$  is the observation signal, and  $\{\xi(t)\}_{t \in \mathbb{Z}}$  and  $\{\omega(t)\}_{t \in \mathbb{Z}}$  are the uncorrelated zero-mean random vectors, that represent the process and measurement noise respectively, with the following covariances,
$$
\mathbb {E} \left[ \xi (t) \xi (t) ^ {\intercal} \right] = Q \in \mathbb {R} ^ {n \times n}, \quad \mathbb {E} \left[ \omega (t) \omega (t) ^ {\intercal} \right] = R \in \mathbb {R} ^ {m \times m},
$$
for some positive (semi-)definite matrices  $Q, R \succeq 0$ . Let  $m_0$  and  $P_0 \succeq 0$  denote the mean and covariance of the initial condition  $x_0$ .
In the filtering setup, the state  $x(t)$  is hidden, and the objective is to estimate it given the history of the observation signal  $\mathcal{Y}(t) = \{y(0), y(1), \ldots, y(t - 1)\}$ . The best linear mean-squared error (MSE)
estimate of  $x(t)$  is defined according to
$$
\hat {x} (t) = \underset {\hat {x} \in \mathcal {L} (\mathcal {Y} (t))} {\operatorname {a r g m i n}} \mathbb {E} \left[ \| x (t) - \hat {x} \| ^ {2} \right] \tag {2}
$$
where  $\mathcal{L}(\mathcal{Y}(t))$  denotes the space of all linear functions of the history of the observation signal  $\mathcal{V}(t)$ . If the model parameters  $(A, H, Q, R)$  are known, the optimal MSE estimate  $\hat{x}(t)$  can be recursively computed by the Kalman filter algorithm [2]:
$$
\hat {x} (t + 1) = A \hat {x} (t) + L (t) (y (t) - H \hat {x} (t)), \quad \hat {x} (0) = m _ {0}, \tag {3a}
$$
$$
P (t + 1) = A P (t) A ^ {\intercal} + Q - A P (t) H ^ {\intercal} S (t) ^ {- 1} H P (T) A ^ {\intercal}, \quad P (0) = P _ {0}, \tag {3b}
$$
where  $S(t) = HP(t)H^{\intercal} + R$ ,  $L(t) \coloneqq AP(t)H^{\intercal}S(t)^{-1}$  is the Kalman gain, and  $P(t) \coloneqq \mathbb{E}[(x(t) - \hat{x}(t))(x(t) - \hat{x}(t))^{\intercal}]$  is the error covariance matrix.
Assumption 1. The pair  $(A, H)$  is detectable, and the pair  $(A, Q^{\frac{1}{2}})$  is stabilizable, where  $Q^{\frac{1}{2}}$  is the unique positive semidefinite square root of  $Q$ .
Under this assumption, the error covariance  $P(t)$  converges to a steady-state value  $P_{\infty}$ , resulting in a unique steady-state Kalman gain  $L_{\infty} = AP_{\infty}H^{\intercal}(HP_{\infty}H^{\intercal} + R)^{-1}$  [39, 40]. It is common to evaluate the steady-state Kalman gain  $L_{\infty}$  offline and use it, instead of  $L(t)$ , to update the estimate in real-time. Furthermore, we note that the assumption of uncorrelated random vectors is sufficient to establish that Kalman filter provides the best linear estimate of the states given the observations for minimizing the MSE criterion [2, Theorem 2].
2.1 Learning problem
Now, we describe our learning setup: 1) The system matrices  $A$  and  $H$  are known, but the process and the measurement noise covariances,  $Q$  and  $R$ , are not available. 2) We have access to an oracle that generates independent realizations of the observation signal for given length  $T$ :  $\{y(t)\}_{t=0}^{T}$ . However, ground-truth measurements of the state  $x(t)$  is not available.
Remark 1. Our proposed learning setup arises in various important engineering applications where merely approximate or reduced-order linear models are available due to difficulty in analytically capturing the effect of complex dynamics or disturbances, hence represented by noise with unknown covariance matrices. Additionally, the system identification procedure often occurs through the application of physical principles and collection of data from experiments in a controlled environment (e.g., in a wind tunnel). However, identifying the noise covariance matrices strongly depends on the operating environment which might be significantly different than the experimental setup. Therefore, it is common engineering practice to use the learned system matrices and tune the Kalman gain to improve the estimation error. We refer to [41] for the application of this procedure for gust load alleviation in wings and [42] for estimation in chemical reactor models. We also emphasize that this learning setup has a rich history in adaptive filtering with numerous references with a recent survey on this topic [22]. As part of our future research, we will carry-out a robustness analysis, similar to its LQR dual counterpart [43, 44], to study the effect of the error in system matrices on the learning performance.
Inspired by the structure of the Kalman filter, our goal is to learn the steady-state Kalman gain  $L_{\infty}$  from the data described in the learning setup:
Given: independent random realizations of  $\{y(0),\ldots ,y(T)\}$  with the parameters  $A,H$
Learn: steady-state Kalman gain  $L_{\infty}$
For that, we formulate the learning problem as a stochastic optimization described next.
2.2 Stochastic optimization formulation
Define  $\hat{x}_L(T)$  to be the estimate given by the Kalman filter at time  $T$  realized by the constant gain  $L$ . Rolling out the update law (3a) for  $t = 0$  to  $t = T - 1$ , and replacing  $L(t)$  with  $L$ , leads to the following expression for the estimate  $\hat{x}_L(T)$  as a function of  $L$ ,
$$
\hat {x} _ {L} (T) = A _ {L} ^ {T} m _ {0} + \sum_ {t = 0} ^ {T - 1} A _ {L} ^ {T - t - 1} L y (t), \tag {4}
$$
where  $A_{L} \coloneqq A - LH$ . Note that evaluating this estimate does not require knowledge of  $Q$  or  $R$ . However, it is not possible to directly aim to learn the gain  $L$  by minimizing the MSE (2) because the ground-truth measurement of the state  $x(T)$  is not available. Instead, we propose to minimize the MSE in predicting the observation  $y(T)$  as a surrogate objective function:
$$
\min  _ {L} J _ {T} ^ {\mathrm {e s t}} (L) := \mathbb {E} \left[ \| y (T) - \hat {y} _ {L} (T) \| ^ {2} \right] \tag {5}
$$
where  $\hat{y}_L(T) \coloneqq H\hat{x}_L(T)$ . Note that while the objective function involves finite time horizon  $T$ , our goal is to learn the steady-state Kalman gain  $L_{\infty}$ .
The justification for using the surrogate objective function in (5) instead of MSE error (2) lies in the detectability assumption 1. Detectability implies that all unobservable states are stable; in other words, their impact on the output signal vanishes quickly—depending on their stability time constant.
Numerically, the optimization problem (5) falls into the category of stochastic optimization and can be solved by algorithms such as Stochastic Gradient Descent (SGD). Such an algorithm would need access to independent realizations of the observation signal which are available. Theoretically however, it is not yet clear if this optimization problem is well-posed and admits a unique minimizer. This is the subject of following section where certain properties of the objective function, such as its gradient dominance and smoothness, are established. These theoretical results are then used to analyze first-order optimization algorithms and provide stability guarantees of the estimation policy iterates. The results are based on the duality relationship between estimation and control that is presented next.
3 Estimation-Control Duality Relationship
The stochastic optimization problem (5) is related to an LQR problem through the application of the classical duality relationship between estimation and control [45, Ch.7.5]. In order to do so, we introduce the adjoint system (dual to (1)) according to:
$$
z (t) = A ^ {\intercal} z (t + 1) - H ^ {\intercal} u (t + 1), \quad z (T) = a \tag {6}
$$
where  $z(t) \in \mathbb{R}^n$  is the adjoint state and  $\mathcal{U}(T) \coloneqq \{u(1),\dots ,u(T)\} \in \mathbb{R}^{mT}$  are the control variables (dual to the observation signal  $\mathcal{V}(T)$ ). The adjoint state is initialized at  $z(T) = a \in \mathbb{R}^n$  and simulated backward in time starting with  $t = T - 1$ . We introduce an LQR cost for the adjoint system:
$$
J _ {T} ^ {\mathrm {L Q R}} (a, \mathcal {U} _ {T}) := z ^ {\intercal} (0) P _ {0} z (0) + \sum_ {t = 1} ^ {T} \left[ z ^ {\intercal} (t) Q z (t) + u ^ {\intercal} (t) R u (t) \right], \tag {7}
$$
and formalize a relationship between linear estimation policies for the system (1) and linear control policies for the adjoint system (6). A linear estimation policy takes the observation history  $\mathcal{V}_T \in \mathbb{R}^{mT}$  and outputs an estimate  $\hat{x}_{\mathcal{L}}(T) \coloneqq \mathcal{L}(\mathcal{V}_T)$  where  $\mathcal{L} : \mathbb{R}^{mT} \to \mathbb{R}^n$  is a linear map. The adjoint of this linear map, denoted by  $\mathcal{L}^\dagger : \mathbb{R}^n \to \mathbb{R}^{mT}$ , is used to define a control policy for the adjoint system (6) which takes the initial condition  $a \in \mathbb{R}^n$  and outputs the control signal  $\mathcal{U}_{\mathcal{L}^\dagger} = \mathcal{L}^\dagger(a)$ ; i.e.,
$$
\{y (0), \dots , y (T - 1) \} \xrightarrow {\mathcal {L}} \hat {x} _ {\mathcal {L}} (T) \qquad \text {a n d} \qquad \{u (1), \dots , u (T) \} \xleftarrow {\mathcal {L} ^ {\dagger}} a.
$$
The duality relationship between optimal MSE estimation and LQR control is summarized in the following proposition. The proof is presented in the supplementary material.
Proposition 1. Consider the estimation problem for the system (1) and the LQR problem (7) subject to the adjoint dynamics (6). For any linear estimation policy  $\hat{x}_{\mathcal{L}}(T) = \mathcal{L}(\mathcal{Y}_T)$ , and for any  $a \in \mathbb{R}^n$ , we have the identity
$$
\mathbb {E} \left[ | a ^ {\top} x (T) - a ^ {\top} \hat {x} _ {\mathcal {L}} (T) | ^ {2} \right] = J _ {T} ^ {L Q R} (a, \mathcal {U} _ {\mathcal {L} ^ {\dagger}} (T)), \tag {8}
$$
where  $\mathcal{U}_{\mathcal{L}^{\dagger}}(T) = \mathcal{L}^{\dagger}(a)$ . In particular, for a Kalman filter with constant gain  $L$ , the output prediction error (5)
$$
\mathbb {E} \left[ \| y (T) - \hat {y} _ {L} (T) \| ^ {2} \right] = \sum_ {i = 1} ^ {m} J _ {T} ^ {L Q R} \left(H _ {i}, \mathcal {U} _ {L ^ {\intercal}} (T)\right) + \operatorname {t r} [ R ], \tag {9}
$$
where  $\mathcal{U}_{L^{\intercal}}(T) = \{L^{\intercal}z(1), L^{\intercal}z(2), \ldots, L^{\intercal}z(T)\}$ , i.e., the feedback control policy with constant gain  $L^{\intercal}$ , and  $H_{i}^{\intercal} \in \mathbb{R}^{n}$  is the  $i$ -th row of the  $m \times n$  matrix  $H$  for  $i = 1, \ldots, m$ .
The expectation in (5) is taken over all the random variables; consisting of the initial state  $x_0$ , dynamic noise  $\xi(t)$ , and measurement noise  $\omega(t)$  for  $t = 0, \dots, T$ . A conditional expectation is not necessary as the estimate is constrained to be measurable with respect to the history of observation.
Remark 2. The duality is also true in the continuous-time setting where the estimation problem is related to a continuous-time LQR. Recent extensions to the nonlinear setting appear in [46] with a comprehensive study in [47]. This duality is distinct from the maximum likelihood approach which involves an optimal control problem over the original dynamics instead of the adjoint system [5].
3.1 Duality in steady-state regime
Using the duality relationship (9), the MSE in prediction (5) is expressed as:
$$
J _ {T} ^ {\mathrm {e s t}} (L) = \operatorname {t r} \left[ X _ {T} (L) H ^ {\intercal} H \right] + \operatorname {t r} \left[ R \right],
$$
where  $X_{T}(L) \coloneqq A_{L}^{T}P_{0}(A_{L}^{\intercal})^{T} + \sum_{t = 0}^{T - 1}A_{L}^{t}(Q + LRL^{\intercal})(A_{L}^{\intercal})^{t}$ . Define the set of Schur stabilizing gains
$$
\mathcal {S} := \left\{L \in \mathbb {R} ^ {n \times m}: \rho \left(A _ {L}\right) <   1 \right\}.
$$
For any  $L \in S$ , in the steady-state, the mean-squared prediction error assumes the form,
$$
\lim  _ {T \rightarrow \infty} J _ {T} ^ {\mathrm {e s t}} (L) = \operatorname {t r} \left[ X _ {\infty} (L) H ^ {\intercal} H \right] + \operatorname {t r} [ R ],
$$
where  $X_{\infty}(L) \coloneqq \lim_{T \to \infty} X_T(L)$  and coincides with the unique solution  $X$  of the discrete Lyapunov equation  $X = A_L X A_L^\top + Q + LRL^\top$  (existence of unique solution follows from  $\rho(A_L) < 1$ ). Given the steady-state limit, we formally analyze the following constrained optimization problem:
$$
\min  _ {L \in S} \leftarrow J (L) := \operatorname {t r} \left[ X _ {(L)} H ^ {\intercal} H \right], \quad \text {s . t .} \quad X _ {(L)} = A _ {L} X _ {(L)} A _ {L} ^ {\intercal} + Q + L R L ^ {\intercal}. \tag {10}
$$
Remark 3. Note that the latter problem is technically the dual of the optimal LQR problem as formulated in [23] by relating  $A \leftrightarrow A^{\intercal}$ ,  $-H \leftrightarrow B^{\intercal}$ ,  $L \leftrightarrow K^{\intercal}$ , and  $H^{\intercal}H \leftrightarrow \Sigma$ . However, the main difference here is that the product  $H^{\intercal}H$  may not be positive definite, for example, due to rank deficiency in  $H$  specially when  $m < n$  (whereas  $\underline{\lambda}(\Sigma) > 0$  appears in all of the bounds in [25, 23]). Thus, in general, the cost function  $J(L)$  is not necessarily coercive in  $L$ , which can drastically effect the optimization landscape. For the same reason, in contrast to the LQR case [25, 23], the gradient dominant property of  $J(L)$  is not clear in the filtering setup. In the next section, we show that such issues can be avoided as long as the pair  $(A, H)$  is observable. Also, the learning problem posed here is distinct from its LQR counterpart (see Table 1).
3.2 Optimization landscape
The first result is concerned with the behaviour of the objective function at the boundary of the optimization domain. It is known [48] that the set of Schur stabilizing gains  $S$  is regular open, contractible, and unbounded when  $m \geq 2$  and the boundary  $\partial S$  coincides with the set  $\{L \in \mathbb{R}^{n \times m} : \rho(A - LH) = 1\}$ . For simplicity of presentation, we consider a slightly stronger assumption:
Assumption 2. The pair  $(A,H)$  is observable, and the noise covariances  $Q\succ 0$  and  $R\succ 0$
Lemma 1. The function  $J(\cdot): S \to \mathbb{R}$  is real-analytic and coercive with compact sublevel sets; i.e.,
$$
L \rightarrow \partial S o r \| L \| \rightarrow \infty e a c h i m p l i e s J (L) \rightarrow \infty ,
$$
and  $S_{\alpha} \coloneqq \{L \in \mathbb{R}^{n \times m} : J(L) \leq \alpha\}$  is compact and contained in  $S$  for any finite  $\alpha > 0$ .
The next result establishes the gradient dominance property of the objective function. While this result is known in the LQR setting ([25, 23]), the extension to the estimation setup is not trivial as  $H^{\mathsf{T}}H$ , which takes the role of the covariance matrix of the initial state in LQR, may not be positive definite (instead, we only assume  $(A,H)$  is observable). This, apparently minor issue, hinders establishing the gradient dominated property globally. However, we recover this property on every sublevel sets of  $J$  which is sufficient for the subsequent convergence analysis.
Lemma 2. Consider the constrained optimization problem (10). Then,
- The explicit formula for the gradient of  $J$  is:  $\nabla J(L) = 2Y_{(L)}\left(-LR + A_L X_{(L)}H^\intercal\right)$ , where  $Y_{(L)} = Y$  is the unique solution of  $Y = A_L^\intercal Y A_L + H^\intercal H$ .
- The global minimizer  $L^{*} = \arg \min_{L\in \mathcal{S}}J(L)$  satisfies  $L^{*} = AX^{*}H^{\intercal}\left(R + HX^{*}H^{\intercal}\right)^{-1}$ , with  $X^{*}$  being the unique solution of  $X^{*} = A_{L^{*}}X^{*}A_{L^{*}}^{\intercal} + Q + L^{*}R(L^{*})^{\intercal}$ .
- The function  $J(\cdot): S_{\alpha} \to \mathbb{R}$ , for any non-empty sublevel set  $S_{\alpha}$  for some  $\alpha > 0$ , satisfies the following inequalities; for all  $L, L' \in S_{\alpha}$ :
$$
c _ {1} [ J (L) - J \left(L ^ {*}\right) ] + c _ {2} \| L - L ^ {*} \| _ {F} ^ {2} \leq \langle \nabla J (L), \nabla J (L) \rangle , \tag {11a}
$$
$$
c _ {3} \| L - L ^ {*} \| _ {F} ^ {2} \leq J (L) - J \left(L ^ {*}\right), \tag {11b}
$$
$$
\left\| \nabla J (L) - \nabla J \left(L ^ {\prime}\right) \right\| _ {F} \leq \ell \left\| L - L ^ {\prime} \right\| _ {F}, \tag {11c}
$$
for positive constants  $c_{1}, c_{2}, c_{3}$  and  $\ell$  that are only a function of  $\alpha$  and independent of  $L$ .
Note that the expression for the gradient is consistent with Proposition 3.8 in [23] after applying the duality relationship explained in Remark 3.
Remark 4. The proposition above implies that  $J(\cdot)$  has the Polyak-Łojasiewicz (PL) property (aka gradient dominance) on every  $S_{\alpha}$ ; i.e., for any  $L \in S_{\alpha}$  we have  $J(L) - J(L^{*}) \leq \frac{1}{c_{1}(\alpha)} \langle \nabla J(L), \nabla J(L) \rangle$ . The inequality (11a) is more general as it characterizes the dominance gap in terms of the iterate error from the optimality. This is useful in obtaining the iterate convergence results in the next section. Also, the Lipschitz bound resembles its "dual" counterpart in [23, Lemma 7.9], however, it is not implied as a simple consequence of duality because  $H^{\mathsf{T}}H$  may not be positive definite.
4 SGD for Learning the Kalman Gain