Title: Data-driven Optimal Filtering for Linear Systems with Unknown Noise Covariances
Venue: NeurIPS 2023
Accepted: yes
Keywords: Optimal filtering

data-driven control

stochastic optimization

learning
Abstract: This paper examines learning the optimal filtering policy, known as the Kalman gain, for a linear system with unknown noise covariance matrices using noisy output data. The learning problem is formulated as a stochastic policy optimiza- tion problem, aiming to minimize the output prediction error. This formulation provides a direct bridge between data-driven optimal control and, its dual, op- timal filtering. Our contributions are twofold. Firstly, we conduct a thorough convergence analysis of the stochastic gradient descent algorithm, adopted for the filtering problem, accounting for biased gradients and stability constraints. Secondly, we carefully leverage a combination of tools from linear system theory and high-dimensional statistics to derive bias-variance error bounds that scale logarithmically with problem dimension, and, in contrast to subspace methods, the length of output trajectories only affects the bias term.

Data-driven Optimal Filtering for Linear Systems with Unknown Noise Covariances

Abstract

This paper examines learning the optimal filtering policy, known as the Kalman gain, for a linear system with unknown noise covariance matrices using noisy output data. The learning problem is formulated as a stochastic policy optimization problem, aiming to minimize the output prediction error. This formulation provides a direct bridge between data-driven optimal control and, its dual, optimal filtering. Our contributions are twofold. Firstly, we conduct a thorough convergence analysis of the stochastic gradient descent algorithm, adopted for the filtering problem, accounting for biased gradients and stability constraints. Secondly, we carefully leverage a combination of tools from linear system theory and high-dimensional statistics to derive bias-variance error bounds that scale logarithmically with problem dimension, and, in contrast to subspace methods, the length of output trajectories only affects the bias term.

1 Introduction

The duality of control and estimation plays a crucial role in system theory, linking two distinct synthesis problems [1, 2, 3, 4, 5, 6]. This duality is an effective bridge between two distinct disciplines, facilitating development of theoretical and computational techniques in one domain and then adopting them for use in the other. For example, the stability proof of the Kalman filter relies on the stabilizing characteristic of the optimal feedback gain in the dual Linear Quadratic Regulator (LQR) optimal control problem [7, Ch. 9]. In this paper, we leverage this duality to learn optimal filtering policies using recent advances in data-driven algorithms for optimal control.

We consider the estimation problem for a system with a known linear dynamic and observation model, but unknown process and measurement noise covariances. Our objective is to learn the optimal steadystate Kalman gain using a training dataset comprising independent realizations of the observation signal. This problem has a rich history in system theory, often explored within the context of adaptive Kalman filtering [8, 9, 10, 11, 12, 13]. A comprehensive summary of four solution approaches to this problem can be found in the classical reference [9]. These approaches include Bayesian inference [14, 15, 16], Maximum likelihood [17, 18], covariance matching [12], and innovation correlation methods [8, 10]. While Bayesian and maximum likelihood approaches are computationally intensive, and covariance matching introduces biases in practice, the innovation correlation-based approaches have gained popularity and have been the subject of recent research [19, 20, 21]. For an excellent survey on this topic, refer to the article [22]. However, it is important to note that these approaches often lack non-asymptotic guarantees and heavily depend on statistical assumptions about the underlying model.

In the realm of optimal control, significant progress has been made in the development of data-driven synthesis methods. Notably, recent advances have focused on the adoption of first-order methods for state-feedback LQR problems [23, 24]. The direct optimization of policies from a gradient-dominant

perspective has first proven in [25] to be remarkably effective with global convergence despite non-convex optimization landscape. It has been demonstrated that despite the non-convex nature of the cost function, when expressed directly in terms of the policy, first-order methods exhibit global convergence to the optimal policy. Building upon this line of work, the use of first-order methods for policy optimization has been explored in variants of the LQR problem. These include Output-feedback Linear Quadratic Regulators (OLQR) [26], model-free setup [27], risk-constrained setup [28], Linear Quadratic Gaussian (LQG) [29], and most recently, Riemannian constrained LQR [30]. These investigations have expanded the scope of data-driven optimal control, demonstrating the versatility and applicability of first-order methods for a wide range of synthesis problems.

The objective of this paper is to provide fresh insights into the classical estimation problem by leveraging the duality between control and estimation and incorporating recent advances in datadriven optimal control. Specifically, building on the fundamental connection between the optimal mean-squared error estimation problem and the LQR problem (Prop. 1), we reformulate determining the optimal Kalman gain as a problem of synthesizing an optimal policy for the adjoint system, under conditions that differ from those explored in the existing literature (see (10) and Remark 3). Upon utilizing this relationship, we propose a Stochastic Gradient Descent (SGD) algorithm for learning the optimal Kalman gain, accompanied by novel non-asymptotic error guarantees in presence of biased gradient and stability constraint. Our approach opens up promising avenues for addressing the estimation problem with robust and efficient data-driven techniques. The following is an informal statement of our main results (combination of Thm. 1 and Thm. 2), and missing proofs appear in the supplementary materials.

Theorem 3 (Informal) . Suppose the system is observable and both dynamic and measurement noise are bounded. Then, with high probability, direct policy updates using stochastic gradient descent with small stepsize converges linearly and globally (from any initial stabilizing policy) to the optimal steady-state Kalman gain.

More recently, the problem of learning the Kalman gain has been considered from a system identification perspective, for completely unknown linear systems [31, 32, 33, 34]. In [32] and [33], subspace system identification methods are used to obtain error bounds for learning the Markov parameters of the model over a time horizon and establish logarithmic regret guarantee for output prediction error. Due to the inherent difficulty of learning a completely unknown stochastic system from partial observations, subspace methods assume marginal stability of the unknown system , and lead to sub-optimal sample complexity bounds that grow with the number of Markov parameters, instead of the number of unknowns [35, pp. 14]. Alternatively, [34] considers minimizing the output prediction error and introduces a model-free policy gradient approach, under the same stability assumptions, that achieves sublinear convergence rate . This paper provides a middle ground between completely known and completely unknown systems, for a learning scenario that not only has relevant practical implications, but also utilizes the duality relationship to LQR to establish linear convergence rates even for unstable systems as long as they are observable . See the Appendix A for the discussion of additional related works [36, 37, 38].

2 Background and Problem Formulation

Herein, first we propose the model setup in detail and discuss the Kalman filter as the estimation strategy. Consider the discrete-time filtering problem given by the stochastic difference equations,

where x ( t ) ∈ R n is the state of the system, y ( t ) ∈ R m is the observation signal, and { ξ ( t ) } t ∈ Z and { ω ( t ) } t ∈ Z are the uncorrelated zero-mean random vectors, that represent the process and measurement noise respectively, with the following covariances,

for some positive (semi-)definite matrices Q,R ⪰ 0 . Let m 0 and P 0 ⪰ 0 denote the mean and covariance of the initial condition x 0 .

In the filtering setup, the state x ( t ) is hidden, and the objective is to estimate it given the history of the observation signal Y ( t ) = { y (0) , y (1) , . . . , y ( t -1) } . The best linear mean-squared error (MSE)

estimate of x ( t ) is defined according to

where L ( Y ( t )) denotes the space of all linear functions of the history of the observation signal Y ( t ) . If the model parameters ( A,H,Q,R ) are known, the optimal MSE estimate ˆ x ( t ) can be recursively computed by the Kalman filter algorithm [2]:

where S ( t ) = HP ( t ) H ⊺ + R , L ( t ) := AP ( t ) H ⊺ S ( t ) -1 is the Kalman gain, and P ( t ) := E [( x ( t ) -ˆ x ( t ))( x ( t ) -ˆ x ( t )) ⊺ ] is the error covariance matrix.

Assumption 1. The pair ( A,H ) is detectable, and the pair ( A,Q 1 2 ) is stabilizable, where Q 1 2 is the unique positive semidefinite square root of Q .

Under this assumption, the error covariance P ( t ) converges to a steady-state value P ∞ , resulting in a unique steady-state Kalman gain L ∞ = AP ∞ H ⊺ ( HP ∞ H ⊺ + R ) -1 [39, 40]. It is common to evaluate the steady-state Kalman gain L ∞ offline and use it, instead of L ( t ) , to update the estimate in real-time. Furthermore, we note that the assumption of uncorrelated random vectors is sufficient to establish that Kalman filter provides the best linear estimate of the states given the observations for minimizing the MSE criterion [2, Theorem 2].

2.1 Learning problem

Now, we describe our learning setup: 1) The system matrices A and H are known, but the process and the measurement noise covariances, Q and R , are not available. 2) We have access to an oracle that generates independent realizations of the observation signal for given length T : { y ( t ) } T t =0 . However, ground-truth measurements of the state x ( t ) is not available.

Remark 1 . Our proposed learning setup arises in various important engineering applications where merely approximate or reduced-order linear models are available due to difficulty in analytically capturing the effect of complex dynamics or disturbances, hence represented by noise with unknown covariance matrices. Additionally, the system identification procedure often occurs through the application of physical principles and collection of data from experiments in a controlled environment (e.g., in a wind tunnel). However, identifying the noise covariance matrices strongly depends on the operating environment which might be significantly different than the experimental setup. Therefore, it is common engineering practice to use the learned system matrices and tune the Kalman gain to improve the estimation error. We refer to [41] for the application of this procedure for gust load alleviation in wings and [42] for estimation in chemical reactor models. We also emphasize that this learning setup has a rich history in adaptive filtering with numerous references with a recent survey on this topic [22]. As part of our future research, we will carry-out a robustness analysis, similar to its LQR dual counterpart [43, 44], to study the effect of the error in system matrices on the learning performance.

Inspired by the structure of the Kalman filter, our goal is to learn the steady-state Kalman gain L ∞ from the data described in the learning setup:

Given: Learn: steady-state Kalman gain L ∞

independent random realizations of { y (0) , . . . , y ( T ) } with the parameters A,H

For that, we formulate the learning problem as a stochastic optimization described next.

2.2 Stochastic optimization formulation

Define ˆ x L ( T ) to be the estimate given by the Kalman filter at time T realized by the constant gain L . Rolling out the update law (3a) for t = 0 to t = T -1 , and replacing L ( t ) with L , leads to the following expression for the estimate ˆ x L ( T ) as a function of L ,

where A L := A -LH . Note that evaluating this estimate does not require knowledge of Q or R . However, it is not possible to directly aim to learn the gain L by minimizing the MSE (2) because the ground-truth measurement of the state x ( T ) is not available. Instead, we propose to minimize the MSE in predicting the observation y ( T ) as a surrogate objective function: 1

where ˆ y L ( T ) := H ˆ x L ( T ) . Note that while the objective function involves finite time horizon T , our goal is to learn the steady-state Kalman gain L ∞ .

The justification for using the surrogate objective function in (5) instead of MSE error (2) lies in the detectability assumption 1. Detectability implies that all unobservable states are stable; in other words, their impact on the output signal vanishes quickly-depending on their stability time constant.

Numerically, the optimization problem (5) falls into the category of stochastic optimization and can be solved by algorithms such as Stochastic Gradient Descent (SGD). Such an algorithm would need access to independent realizations of the observation signal which are available. Theoretically however, it is not yet clear if this optimization problem is well-posed and admits a unique minimizer. This is the subject of following section where certain properties of the objective function, such as its gradient dominance and smoothness, are established. These theoretical results are then used to analyze first-order optimization algorithms and provide stability guarantees of the estimation policy iterates. The results are based on the duality relationship between estimation and control that is presented next.

3 Estimation-Control Duality Relationship

The stochastic optimization problem (5) is related to an LQR problem through the application of the classical duality relationship between estimation and control [45, Ch.7.5]. In order to do so, we introduce the adjoint system (dual to (1)) according to:

where z ( t ) ∈ R n is the adjoint state and U ( T ) := { u (1) , . . . , u ( T ) } ∈ R mT are the control variables (dual to the observation signal Y ( T ) ). The adjoint state is initialized at z ( T ) = a ∈ R n and simulated backward in time starting with t = T -1 . We introduce an LQR cost for the adjoint system:

and formalize a relationship between linear estimation policies for the system (1) and linear control policies for the adjoint system (6). A linear estimation policy takes the observation history Y T ∈ R mT and outputs an estimate ˆ x L ( T ) := L ( Y T ) where L : R mT → R n is a linear map. The adjoint of this linear map, denoted by L † : R n → R mT , is used to define a control policy for the adjoint system (6) which takes the initial condition a ∈ R n and outputs the control signal U L † = L † ( a ) ; i.e.,

The duality relationship between optimal MSE estimation and LQR control is summarized in the following proposition. The proof is presented in the supplementary material.

Proposition 1. Consider the estimation problem for the system (1) and the LQR problem (7) subject to the adjoint dynamics (6) . For any linear estimation policy ˆ x L ( T ) = L ( Y T ) , and for any a ∈ R n , we have the identity

where U L † ( T ) = L † ( a ) . In particular, for a Kalman filter with constant gain L , the output prediction error (5)

where U L ⊺ ( T ) = { L ⊺ z (1) , L ⊺ z (2) , . . . , L ⊺ z ( T ) } , i.e., the feedback control policy with constant gain L ⊺ , and H ⊺ i ∈ R n is the i -th row of the m × n matrix H for i = 1 , . . . , m .

Remark 2 . The duality is also true in the continuous-time setting where the estimation problem is related to a continuous-time LQR. Recent extensions to the nonlinear setting appears in [46] with a comprehensive study in [47]. This duality is distinct from the maximum likelihood approach which involves an optimal control problem over the original dynamics instead of the adjoint system [5].

3.1 Duality in steady-state regime

Using the duality relationship (9), the MSE in prediction (5) is expressed as:

where X T ( L ) := A T L P 0 ( A ⊺ L ) T + ∑ T -1 t =0 A t L ( Q + LRL ⊺ )( A ⊺ L ) t . Define the set of Schur stabilizing gains

For any L ∈ S , in the steady-state, the mean-squared prediction error assumes the form,

where X ∞ ( L ) := lim T →∞ X T ( L ) and coincides with the unique solution X of the discrete Lyapunov equation X = A L XA ⊺ L + Q + LRL ⊺ (existence of unique solution follows from ρ ( A L ) < 1 ). Given the steady-state limit, we formally analyze the following constrained optimization problem:

Remark 3 . Note that the latter problem is technically the dual of the optimal LQR problem as formulated in [23] by relating A ↔ A ⊺ , -H ↔ B ⊺ , L ↔ K ⊺ , and H ⊺ H ↔ Σ . However, the main difference here is that the product H ⊺ H may not be positive definite, for example, due to rank deficiency in H specially when m<n (whereas λ (Σ) > 0 appears in all of the bounds in [25, 23]). Thus, in general, the cost function J ( L ) is not necessarily coercive in L , which can drastically effect the optimization landscape. For the same reason, in contrast to the LQR case [25, 23], the gradient dominant property of J ( L ) is not clear in the filtering setup. In the next section, we show that such issues can be avoided as long as the pair ( A,H ) is observable. Also, the learning problem posed here is distinct from its LQR counterpart (see Table 1).

3.2 Optimization landscape

The first result is concerned with the behaviour of the objective function at the boundary of the optimization domain. It is known [48] that the set of Schur stabilizing gains S is regular open, contractible, and unbounded when m ≥ 2 and the boundary ∂ S coincides with the set { L ∈ R n × m : ρ ( A -LH ) = 1 } . For simplicity of presentation, we consider a slightly stronger assumption:

Assumption 2. The pair ( A,H ) is observable, and the noise covariances Q ≻ 0 and R ≻ 0 .

Lemma 1. The function J ( . ) : S → R is real-analytic and coercive with compact sublevel sets; i.e.,

and S α := { L ∈ R n × m : J ( L ) ≤ α } is compact and contained in S for any finite α > 0 .

The next result establishes the gradient dominance property of the objective function. While this result is known in the LQR setting ([25, 23]), the extension to the estimation setup is not trivial as H ⊺ H , which takes the role of the covariance matrix of the initial state in LQR, may not be positive definite (instead, we only assume ( A,H ) is observable). This, apparently minor issue, hinders establishing the gradient dominated property globally. However, we recover this property on every sublevel sets of J which is sufficient for the subsequent convergence analysis.

Lemma 2. Consider the constrained optimization problem (10) .Then,

The explicit formula for the gradient of J is: ∇ J ( L ) = 2 Y ( L ) ( -LR + A L X ( L ) H ⊺ ) , where Y ( L ) = Y is the unique solution of Y = A ⊺ L Y A L + H ⊺ H.

The global minimizer L ∗ = arg min L ∈S J ( L ) satisfies L ∗ = AX ∗ H ⊺ ( R + HX ∗ H ⊺ ) -1 , with X ∗ being the unique solution of X ∗ = A L ∗ X ∗ A ⊺ L ∗ + Q + L ∗ R ( L ∗ ) ⊺ .

The function J ( . ) : S α → R , for any non-empty sublevel set S α for some α > 0 , satisfies the following inequalities; for all L, L ′ ∈ S α :

for positive constants c 1 , c 2 , c 3 and ℓ that are only a function of α and independent of L .

Note that the expression for the gradient is consistent with Proposition 3.8 in [23] after applying the duality relationship explained in Remark 3.

Remark 4 . The proposition above implies that J ( . ) has the Polyak-Łojasiewicz (PL) property (aka gradient dominance) on every S α ; i.e., for any L ∈ S α we have J ( L ) -J ( L ∗ ) ≤ 1 c 1 ( α ) ⟨∇ J ( L ) , ∇ J ( L ) ⟩ . The inequality (11a) is more general as it characterizes the dominance gap in terms of the iterate error from the optimality. This is useful in obtaining the iterate convergence results in the next section. Also, the Lipschitz bound resembles its 'dual' counterpart in [23, Lemma 7.9], however, it is not implied as a simple consequence of duality because H ⊺ H may not be positive definite.

4 SGD for Learning the Kalman Gain

In order to emphasize on the estimation time horizon T for various measurement sequences, we use Y T := { y ( t ) } T t =0 to denote the measurement time-span. Note that, any choice of L ∈ S corresponds to a filtering strategy that outputs the following prediction,

We denote the squared-norm of the estimation error for this filtering strategy as,

where e T ( L ) := y ( T ) -ˆ y L ( T ) . We also define the truncated objective function as

where the expectation is taken over all possible random measurement sequences, and note that, at the steady-state limit, we obtain lim T →∞ J T ( L ) = J ( L ) .

The SGD algorithm aims to solve this optimization problem by replacing the gradient, in the Gradient Descent (GD) update, with an unbiased estimate of the gradient in terms of samples from the measurement sequence. In particular, with access to an oracle that produces independent realization of the measurement sequence, say M random independent measurements sequences {Y i T } i M =1 , the gradient can be approximated as follows: denote the approximated cost value

then the approximate gradient with batch-size M is ∇ ̂ J T ( L ) = 1 M ∑ M i =1 ∇ L ε ( L, Y i T ) . This forms an unbiased estimate of the gradient of the 'truncated objective', i.e., E [ ∇ ̂ J T ( L ) ] = ∇ J T ( L ) . Next, for implementation purposes, we compute the gradient estimate explicitly in terms of the measurement sequence and the filtering policy L .

Lemma 3. Given L ∈ S and a sequence of measurements Y = { y ( t ) } T t =0 , we have,

Finally, using this approximate gradient, the so-called SGD update proceeds as,

for k ∈ Z , where η k > 0 is the step-size. Numerical results of the application of this algorithm appears in Appendix G.

Remark 5 . Computing this approximate gradient only requires the knowledge of the system matrices A and H , and does not require the noise covariance information Q and R . Simulation results for the SGD algorithm are provided in the supplementary material.

Although the convergence of the SGD algorithm is expected to follow similar to the GD algorithm under the gradient dominance condition and Lipschitz property, the analysis becomes complicated due to the possibility of the iterated gain L k leaving the sub-level sets. It is expected that a convergence guarantee would hold under high-probability due to concentration of the gradient estimate around the true gradient. The complete analysis in this direction is provided in the subsequent sections.

We first provide sample complexity and convergence guarantees for SGD with a biased estimation of gradient for locally Lipschitz objective functions and in presence of stability constraint S . Subsequently, we study the stochastic problem of estimating the gradient for the estimation problem. Distinct features of our approach as compared with similar formulations in the literature are highlighted in Table 1.

We now provide a road map to navigate the technical results that concludes with Theorem 3: Section 4.1 is concerned with the convergence analysis of the SGD algorithm under an assumption for the biased gradient oracle, summarized in Theorem 1 which concludes the linear convergence of the iterates for sufficiently small stepsize. Section 4.2 is concerned with the bias-variance error analysis of the gradient estimate, summarized in Theorem 2 providing the sufficient values for the batch-size and trajectory length that guarantees the desired bound on the gradient oracle required in Theorem 1. Finally, combining the results from Theorem 1 and Theorem 2 concludes our main result in Theorem 3.

4.1 SGD with biased gradient and stability constraint

First, we characterize the 'robustness' of a policy at which we aim to estimate the gradient. This is formalized in the following lemma which is a consequence of [30, Lemma IV.1].

Lemma 4. Consider any L ∈ S and let Z be the unique solution of Z = A L ZA ⊺ L +Λ for any Λ ≻ 0 . Then, L +∆ ∈ S for any ∆ ∈ R n × m satisfying 0 ≤ ∥ ∆ ∥ F ≤ λ (Λ) /[ 2 λ ( Z ) ∥ H ∥ ] .

Second, we provide a uniform lowerbound for the stepsize of gradient descent for an approximated direction 'close' to the true gradient direction.

Lemma 5 (Uniform Lower Bound on Stepsize) . Let L 0 ∈ S α for some α ≥ α ∗ := J ( L ∗ ) , and choose any finite β ≥ α . Consider any direction E such that ∥ E -∇ J ( L 0 ) ∥ F ≤ γ ∥∇ J ( L 0 ) ∥ F for some γ ∈ [0 , 1] , then we have J ( L 0 -ηE ) ≤ β for any η satisfying:

Remark 6 . Note that for the case of exact gradient direction, i.e. when E = ∇ J ( L 0 ) , we have γ = 0 and choosing β = α implies the known uniform bound of η ≤ 1 ℓ ( β ) for feasible stepsizes as expected. Also, by this choice of β , this guarantees that the next iterate remains in sublevel set S α . This lemma generalizes this uniform bound for general directions and (potentially) larger sublevel set.

The next proposition provides a decay guarantee for one iteration of gradient descent with an approximate direction which will be used later for convergence of SGD with a biased gradient estimate.

̸

Proposition 2 (Linear Decay in Cost Value) . Suppose L 0 ∈ S α for some α > 0 and a direction E = 0 is given such that ∥ E -∇ J ( L ) ∥ F ≤ γ ∥∇ J ( L ) ∥ F for some γ < 1 . Let ¯ η := (1 -γ ) / ( γ +1) 2 ℓ ( α ) . Then, L 1 := L 0 -¯ ηE remains in the same sublevel set, i.e., L 1 ∈ S α . Furthermore, we obtain the following linear decay of the cost value:

Next, we guarantee that SGD algorithm with this biased estimation of gradient obtains a linear convergence rate outside a small set C τ around optimality defined as

for some τ ∈ (0 , 1) and arbitrarily small s 0 > 0 . First, we assume access to the following oracle that provides a biased estimation of the true gradient.

Assumption 3. Suppose, for some α > 0 , we have access to a biased estimate of the gradient ∇ ̂ J ( L ) such that, there exists constants s, s 0 > 0 implying ∥∇ ̂ J ( L ) -∇ J ( L ) ∥ F ≤ s ∥∇ J ( L ) ∥ F + s 0 for all L ∈ S α \ C τ .

Theorem 1 (Convergence) . Suppose Assumption 3 holds with small enough s and s 0 such that s ≤ γ/ 2 and S α \ C γ/ 2 is non-empty for some γ ∈ (0 , 1) . Then, SGD algorithm starting from any L 0 ∈ S α \ C γ/ 2 with fixed stepsize ¯ η := (1 -γ ) ( γ +1) 2 ℓ ( α ) generates a sequence of policies { L k } that are stable (i.e. each L k ∈ S α ) and cost values decay linearly before entering C γ/ 2 ; i.e.,

for each k ≥ 0 unless L j ∈ C γ/ 2 for some j ≤ k .

Remark 7 . By combining Theorem 1 and the PL property of the cost in (11a), we immediately obtain a sample complexity for our algorithm; e.g., choose γ = 1 / 2 , then in order to guarantee an ε error on the cost J ( L k ) -J ( L ∗ ) ≤ ε , it suffices to have a small enough bias term s 0 ≤ √ c 1 ( α ) ε 4 and variance coefficient s ≤ 1 4 for the oracle, and run the SGD algorithm for k > ln( ε α ) / ln(1 -c 1 ( α ) 18 ℓ ( α ) )) steps.

4.2 Observation model for the estimation problem

Herein, first we show that the estimation error and its differential can be characterized as a 'simple norm' of the concatenated noise (Proposition 3). This norm is induced by a metric that encapsulates the system dynamics which is explained below. Before proceeding to the results of this section, we assume that both the process and measurement noise are bounded:

Assumption 4. Assume that (almost surely) ∥ x 0 ∥ , ∥ ξ ( t ) ∥ ≤ κ ξ and ∥ ω ( t ) ∥ ≤ κ ω for all t . Also, for simplicity, suppose the initial state has zero mean, i.e., m 0 = 0 n .

For two vectors ⃗ v, ⃗ w ∈ R ( T +1) n , we define

where A L := ( A 0 L A 1 L . . . A T L ) . Also, define M L [ E ] := ( M 0 [ E ] M 1 [ E ] · · · M T [ E ]) with M 0 [ E ] = 0 , M 1 [ E ] = EH and M i +1 [ E ] = ∑ i k =0 A i -k L EHA k L for i = 1 , · · · , T -1 .

Proposition 3. The estimation error ε ( L, Y T ) takes the following form

where ⃗ η L := ξ ⃗ -( I ⊗ L ) ⃗ ω with ξ ⃗ ⊺ = ( ξ ( T -1) ⊺ . . . ξ (0) ⊺ x (0) ⊺ ) and ⃗ ω ⊺ = ( ω ( T -1) ⊺ . . . ω (0) ⊺ 0 ⊺ m ) . Furthermore, its differential acts on small enough E ∈ R n × m as,

where X L := ⃗ η L ⃗ η ⊺ L and N L [ E ] := M L [ E ] ⊺ H ⊺ H A L + A ⊺ L H ⊺ H M L [ E ] .

Now, we want to bound the error in the estimated gradient ∇ ̂ J T ( L ) by considering the concentration error (on length T trajectories) and truncation error separately as follows:

recalling that J T ( L ) = E [ ε ( L, Y T )] by definition.

Next, we aim to provide the analysis of concentration error on trajectories of length T with probability bounds. However, for any pair of real ( T × T ) -matrices M and N , by Cauchy Schwartz inequality we obtain that | tr [ MN ] | ≤ ∥ M ∥ F ∥ N ∥ F ≤ √ T ∥ M ∥∥ N ∥ F . This bound becomes loose (in terms of dimension T ) as the condition number of N increases 2 .

Nonetheless, we are able to provide concentration error bounds that 'scale well with respect to the length T ' that hinges upon the following idea: from von Neumann Trace Inequality [49, Theorem 8.7.6] one obtains,

where ∥ N ∥ ∗ := tr [ √ N ⊺ N ] = ∑ i σ i ( N ) is the nuclear norm with σ i ( N ) denoting the i -th largest singular value of N . Additionally, the same inequality holds for non-square matrices of appropriate dimension which is tight in terms of dimension.

Proposition 4 (Concentration Error Bound) . Consider M independent length T trajectories {Y i T } i M =1 and suppose Assumption 4 holds. Let ∇ ̂ J T ( L ) := 1 M ∑ M i =1 ∇ ε ( L, Y i T ) , then for any s > 0 ,

where ν L := 4 κ 2 L C 3 L ∥ H ∥ 2 ∥ H ∥ ∗ / [1 -√ ρ ( A L )] 3 with κ L = κ ξ + ∥ L ∥ κ ω .

We can also show how the truncation error decays linearly as T grows, with constants that are independent of the system dimension n :

Proposition 5 (Truncation Error Bound) . Under Assumption 4, the truncation error is bounded:

where ¯ γ L := 10 κ 4 L C 6 L ∥ H ∥ 2 ∥ H ∥ ∗ / [1 -ρ ( A L )] 2 .

Remark 8 . Notice how the trajectory length T determines the bias in the estimated gradient. However, the concentration error bound is independent of T and only depends on the noise bounds proportionate to κ 4 ξ , κ 4 ω and the stability margin of A L proportionate to C 6 L / (1 -√ ρ ( A L )) 6 .

Finally, by combining the truncation bound in Proposition 5 with concentration bounds in Proposition 4 we can provide probabilistic bounds on the 'estimated cost' ̂ J T ( L ) and the 'estimated gradient' ∇ ̂ J T ( L ) . Its precise statement is deferred to the supplementary materials (Theorem 5).

4.3 Sample complexity of SGD for Kalman Gain

Note that the open-loop system may be unstable. Often in learning literature, it is assumed that the closed-loop system can be contractible ( i.e., the spectral norm ∥ A L ∥ < 1 ) which is quite convenient for analysis, however, it is not a reasonable system theoretic assumption. Herein, we emphasize that we only require the close-loop system to be Schur stable, meaning that ρ ( A L ) < 1 ; yet, it is very well possible that the system is not contractible. Handling systems that are merely stable requires more involved system theoretic tools that are established in the following lemma.

Lemma 6 (Uniform Bounds for Stable Systems) . Suppose L ∈ S , then there exit a constants C L > 0 such that

Furthermore, consider S α for some α > 0 , then there exist constants D α > 0 , C α > 0 and ρ α ∈ (0 , 1) such that ∥ L ∥ ≤ D α , C L ≤ C α , and ρ ( A L ) ≤ ρ α for all L ∈ S α .

The following result provides sample complexity bounds for this stochastic oracle to provide a biased estimation of the gradient that satisfies our oracle model of SGD analysis in Assumption 3.

Theorem 2. Under the premise of Proposition 4, consider S α for some α > 0 and choose any s, s 0 > 0 and τ ∈ (0 , 1) . Suppose the trajectory length T ≥ ln ( ¯ γ α √ min( n, m ) /s 0 )/ ln ( 1 / √ ρ α ) and the batch size M ≥ 4 ν 2 α min( n, m ) ln(2 n/δ ) / ( s s 0 ) 2 , where ¯ γ α := 10( κ ξ + D α κ ω ) 4 C 6 α ∥ H ∥ 2 ∥ H ∥ ∗ and ν α := 5 C 3 α ∥ H ∥ 2 ∥ H ∥ ∗ ( κ ξ + D α κ ω ) 2 / [1 - √ ρ α ] 3 . Then, with probability no less than 1 -δ , Assumption 3 holds.

Proof. First, note that for any L ∈ S α by Lemma 6 we have that ¯ γ L ≤ ¯ γ α and ν L ≤ ν α . Then, note that the lower bound on T implies that ¯ γ α √ min( n, m ) √ ρ α T +1 ≤ s 0 . The claim then follows by applying Theorem 5 and noting that for any L ̸∈ C τ the gradient is lowerbound as ∥∇ J ( L ) ∥ > s 0 /τ .

Theorem 3. Consider the linear system (1) under Assumptions 2 and 4. Suppose the SGD algorithm is implemented with initial stabilizing gain L 0 , the step-size ¯ η := 2 9 ℓ ( J ( L 0 )) , for k iterations, a batch-size of M , and data-length T . Then, ∀ ε > 0 and with probability larger than 1 -δ , J ( L k ) -J ( L ∗ ) ≤ ε if

Proof of Theorem 3. Recall that according to Remark 7, in order to obtain ε error on the cost value we require k ≥ O (ln(1 /ε )) of SGD algorithm using gradient estimates with small enough bias term s 0 ≤ √ c 1 ( α ) ε 4 and variance coefficient s < 1 4 . Now, we can guarantee Assumption 3 holds for such s 0 and s (with probability at least 1 -δ ) by invoking Theorem 2. In fact, it suffices to ensure that the length of data trajectories T and the batch size M are large enough; specifically, T ≥ O (ln(1 /s 0 )) = O (ln(1 /ε )) and by using union bound for bounding the failure probability M ≥ O (ln(1 /δ ) ln( k ) / s 2 0 ) = O (ln(1 /δ ) ln(ln(1 /ε )) / ε ) .

5 Conclusions, Broader Impact, and Limitations

In this work, we considered the problem of learning the optimal (steady-state) Kalman gain for linear systems with unknown process and measurement noise covariances. Our approach builds on the duality between optimal control and estimation, resulting in a direct stochastic Policy Optimization (PO) algorithm for learning the optimal filter gain. We also provided convergence guarantees and finite sample complexity with bias and variance error bounds that scale well with problem parameters. In particular, the variance is independent of the length of data trajectories and scales logarithmically with problem dimension, and bias term decreases exponentially with the length.

This work contributes a generic optimization algorithm and introduces a filtering strategy for estimating dynamical system states. While theoretical, it raises privacy concerns similar to the model-based Kalman filter. Limitations include the need for prior knowledge of system parameters; nonetheless, parameter uncertainties can be treated practically as process and measurement noise. Finally, sample complexities depend on the stability margin 1 -√ ρ ( A L ) , inherent to the system generating the data.

Finally, a direction for future research is to study how to adapt the proposed algorithm and error analysis for the setting when a single long trajectory is available as opposed to several independent trajectories of finite length. Another interesting direction is to carry out a robustness analysis, similar to its LQR dual counterpart, to study the effect of the error in system parameters on the policy learning accuracy. Of course, the ultimate research goal is to use the recently introduced duality in nonlinear filtering [47] as a bridge to bring tools from learning to nonlinear filtering.

Acknowledgments and Disclosure of Funding

The research of A. Taghvaei has been supported by NSF grant EPCN-2318977; S. Talebi and M. Mesbahi have been supported by NSF grant ECCS-2149470 and AFOSR grant FA9550-20-1-0053. The authors are also grateful for constructive feedback from the reviewers.

References

R. E. Kalman, 'On the general theory of control systems,' in Proceedings First International Conference on Automatic Control, Moscow, USSR , pp. 481-492, 1960.

R. E. Kalman, 'A new approach to linear filtering and prediction problems,' ASME. Journal of Basic Engineering , vol. 82, pp. 35-45, 03 1960.

J. Pearson, 'On the duality between estimation and control,' SIAM Journal on Control , vol. 4, no. 4, pp. 594-600, 1966.

R. Mortensen, 'Maximum-likelihood recursive nonlinear filtering,' Journal of Optimization Theory and Applications , vol. 2, no. 6, pp. 386-394, 1968.

A. Bensoussan, Estimation and control of dynamical systems , vol. 48. Springer, 2018.

W. H. Fleming and S. K. Mitter, 'Optimal control and nonlinear filtering for nondegenerate diffusion processes,' Stochastics: An International Journal of Probability and Stochastic Processes , vol. 8, no. 1, pp. 63-77, 1982.

J. Xiong, An Introduction to Stochastic Filtering Theory , vol. 18. OUP Oxford, 2008.

R. Mehra, 'On the identification of variances and adaptive Kalman filtering,' IEEE Transactions on Automatic Control , vol. 15, no. 2, pp. 175-184, 1970.

R. Mehra, 'Approaches to adaptive filtering,' IEEE Transactions on Automatic Control , vol. 17, no. 5, pp. 693-698, 1972.

B. Carew and P. Belanger, 'Identification of optimum filter steady-state gain for systems with unknown noise covariances,' IEEE Transactions on Automatic Control , vol. 18, no. 6, pp. 582-587, 1973.

P. R. Belanger, 'Estimation of noise covariance matrices for a linear time-varying stochastic process,' Automatica , vol. 10, no. 3, pp. 267-275, 1974.

K. Myers and B. Tapley, 'Adaptive sequential estimation with unknown noise statistics,' IEEE Transactions on Automatic Control , vol. 21, no. 4, pp. 520-523, 1976.

K. Tajima, 'Estimation of steady-state Kalman filter gain,' IEEE Transactions on Automatic Control , vol. 23, no. 5, pp. 944-945, 1978.

D. Magill, 'Optimal adaptive estimation of sampled stochastic processes,' IEEE Transactions on Automatic Control , vol. 10, no. 4, pp. 434-439, 1965.

C. G. Hilborn and D. G. Lainiotis, 'Optimal estimation in the presence of unknown parameters,' IEEE Transactions on Systems Science and Cybernetics , vol. 5, no. 1, pp. 38-43, 1969.

P. Matisko and V. Havlena, 'Noise covariances estimation for Kalman filter tuning,' IFAC Proceedings Volumes , vol. 43, no. 10, pp. 31-36, 2010.

R. Kashyap, 'Maximum likelihood identification of stochastic linear systems,' IEEE Transactions on Automatic Control , vol. 15, no. 1, pp. 25-34, 1970.

R. H. Shumway and D. S. Stoffer, 'An approach to time series smoothing and forecasting using the EM algorithm,' Journal of Time Series Analysis , vol. 3, no. 4, pp. 253-264, 1982.

B. J. Odelson, M. R. Rajamani, and J. B. Rawlings, 'A new autocovariance least-squares method for estimating noise covariances,' Automatica , vol. 42, no. 2, pp. 303-308, 2006.

B. M. Åkesson, J. B. Jørgensen, N. K. Poulsen, and S. B. Jørgensen, 'A generalized autocovariance least-squares method for Kalman filter tuning,' Journal of Process Control , vol. 18, no. 7-8, pp. 769-779, 2008.

J. Duník, M. ˆ Simandl, and O. Straka, 'Methods for estimating state and measurement noise covariance matrices: Aspects and comparison,' IFAC Proceedings Volumes , vol. 42, no. 10, pp. 372-377, 2009.

L. Zhang, D. Sidoti, A. Bienkowski, K. R. Pattipati, Y. Bar-Shalom, and D. L. Kleinman, 'On the identification of noise covariances and adaptive Kalman filtering: A new look at a 50 year-old problem,' IEEE Access , vol. 8, pp. 59362-59388, 2020.

J. Bu, A. Mesbahi, M. Fazel, and M. Mesbahi, 'LQR through the lens of first order methods: Discrete-time case,' arXiv preprint arXiv:1907.08921 , 2019.

J. Bu, A. Mesbahi, and M. Mesbahi, 'Policy gradient-based algorithms for continuous-time linear quadratic control,' arXiv preprint arXiv:2006.09178 , 2020.

M. Fazel, R. Ge, S. Kakade, and M. Mesbahi, 'Global convergence of policy gradient methods for the linear quadratic regulator,' in Proceedings of the 35th International Conference on Machine Learning , vol. 80, pp. 1467-1476, PMLR, 2018.

I. Fatkhullin and B. Polyak, 'Optimizing static linear feedback: Gradient method,' SIAM Journal on Control and Optimization , vol. 59, no. 5, pp. 3887-3911, 2021.

H. Mohammadi, M. Soltanolkotabi, and M. R. Jovanovic, 'On the linear convergence of random search for discrete-time LQR,' IEEE Control Systems Letters , vol. 5, no. 3, pp. 989-994, 2021.

F. Zhao, K. You, and T. Ba¸ sar, 'Global convergence of policy gradient primal-dual methods for risk-constrained LQRs,' arXiv preprint arXiv:2104.04901 , 2021.

Y. Tang, Y. Zheng, and N. Li, 'Analysis of the optimization landscape of linear quadratic gaussian (LQG) control,' in Proceedings of the 3rd Conference on Learning for Dynamics and Control , vol. 144, pp. 599-610, PMLR, June 2021.

S. Talebi and M. Mesbahi, 'Policy optimization over submanifolds for constrained feedback synthesis,' IEEE Transactions on Automatic Control (to appear), arXiv preprint arXiv:2201.11157 , 2022.

S. Lale, K. Azizzadenesheli, B. Hassibi, and A. Anandkumar, 'Logarithmic regret bound in partially observable linear dynamical systems,' Advances in Neural Information Processing Systems , vol. 33, pp. 20876-20888, 2020.

A. Tsiamis and G. J. Pappas, 'Finite sample analysis of stochastic system identification,' in 2019 IEEE 58th Conference on Decision and Control (CDC) , pp. 3648-3654, IEEE, 2019.

A. Tsiamis and G. J. Pappas, 'Online learning of the Kalman filter with logarithmic regret,' IEEE Transactions on Automatic Control , vol. 68, no. 5, pp. 2774-2789, 2023.

J. Umenberger, M. Simchowitz, J. Perdomo, K. Zhang, and R. Tedrake, 'Globally convergent policy search for output estimation,' in Advances in Neural Information Processing Systems (S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, eds.), vol. 35, pp. 2277822790, Curran Associates, Inc., 2022.

A. Tsiamis, I. Ziemann, N. Matni, and G. J. Pappas, 'Statistical learning theory for control: A finite sample perspective,' arXiv preprint arXiv:2209.05423 , 2022.

Y. Zheng, L. Furieri, M. Kamgarpour, and N. Li, 'Sample complexity of linear quadratic gaussian (LQG) control for output feedback systems,' in Proceedings of the 3rd Conference on Learning for Dynamics and Control (A. Jadbabaie, J. Lygeros, G. J. Pappas, P. A.;Parrilo, B. Recht, C. J. Tomlin, and M. N. Zeilinger, eds.), vol. 144 of Proceedings of Machine Learning Research , pp. 559-570, PMLR, 07 - 08 June 2021.

X. Zhang, B. Hu, and T. Ba¸ sar, 'Learning the Kalman filter with fine-grained sample complexity,' in 2023 American Control Conference (ACC) , pp. 4549-4554, 2023.

W. Liu, J. Sun, G. Wang, F. Bullo, and J. Chen, 'Learning robust data-based LQG controllers from noisy data,' arXiv preprint arXiv:2305.01417 , 2023.

H. Kwakernaak and R. Sivan, Linear Optimal Control Systems , vol. 1072. Wiley-interscience, 1969.

F. Lewis, Optimal Estimation with an Introduction to Stochastic Control Theory . New York, Wiley-Interscience, 1986.

K. A. Hinson, K. A. Morgansen, and E. Livne, 'Autocovariance least squares noise covariance estimation for a gust load alleviation test-bed,' in AIAA SCITECH 2022 Forum , p. 0617, 2022.

B. J. Odelson, A. Lutz, and J. B. Rawlings, 'The autocovariance least-squares method for estimating covariances: Application to model-based control of chemical reactors,' IEEE Transactions on Control Systems Technology , vol. 14, no. 3, pp. 532-540, 2006.

M. Safonov and W. Wang, 'Singular value properties of LQ regulators,' IEEE Transactions on Automatic Control , vol. 37, no. 8, pp. 1210-1211, 1992.

C. Chen and A. Holohan, 'Stability robustness of linear quadratic regulators,' International Journal of Robust and Nonlinear Control , vol. 26, no. 9, pp. 1817-1824, 2016.

K. J. Åström, Introduction to Stochastic Control Theory . Courier Corporation, 2012.

J.-W. Kim, P. G. Mehta, and S. P. Meyn, 'What is the lagrangian for nonlinear filtering?,' in 2019 IEEE 58th Conference on Decision and Control (CDC) , pp. 1607-1614, IEEE, 2019.

J. W. Kim, 'Duality for nonlinear filtering,' arXiv preprint arXiv:2207.07709 , 2022.

J. Bu, A. Mesbahi, and M. Mesbahi, 'On topological and metrical properties of stabilizing feedback gains: The MIMO case,' arXiv preprint arXiv:1904.02737 , 2019.

R. A. Horn and C. R. Johnson, Matrix Analysis . Cambridge University Press, 2012.

H. Mohammadi, A. Zare, M. Soltanolkotabi, and M. R. Jovanovi´ c, 'Convergence and sample complexity of gradient methods for the model-free linear-quadratic regulator problem,' IEEE Transactions on Automatic Control , vol. 67, no. 5, pp. 2435-2450, 2021.

A. Beck, First-Order Methods in Optimization . Philadelphia, PA: Society for Industrial and Applied Mathematics, 2017.

A. Cohen, A. Hasidim, T. Koren, N. Lazic, Y. Mansour, and K. Talwar, 'Online linear quadratic control,' in Proceedings of the 35th International Conference on Machine Learning (J. Dy and A. Krause, eds.), vol. 80 of Proceedings of Machine Learning Research , pp. 1029-1038, PMLR, 10-15 Jul 2018.

N. J. Higham, Functions of Matrices: Theory and Computation . SIAM, 2008.

J. A. Tropp, 'An introduction to matrix concentration inequalities,' Foundations and Trends® in Machine Learning , vol. 8, no. 1-2, pp. 1-230, 2015.

S. Talebi, A. Taghvaei, and M. Mesbahi, 'SGD for Filtering,' Oct. 2023. Available on GitHub at https://github.com/shahriarta/sgd4filtering .

Contents

Appendix A Discussion of additional related works

After the initial submission of this manuscript, we encountered the following related works, and we will now briefly discuss their connections and differences in relation to our work: [36] establishes an end-to-end sample complexity bound on learning a robust LQG controller (which is different than our filtering problem). As the system parameters are also unknown, this work only considers the open-loop stable systems. Additionally, it establishes a nice trade-off between optimality and robustness characterized by a suboptimality performance bound as a result of the robust LQG design. While our filtering design is based on the knowledge of system parameters, we do not consider the open-loop stability assumption; and the robust synthesis is included as one of the main future directions of this work. Furthermore, the complexity bounds in [36] depends on the length of trajectory, whereas ours does not. Also, their suboptimality bound scale as O (1 / √ N ) in the number of trajectories, whereas ours scales as O (1 /N ) . Next, [37] similarly considers the problem of learning the steady-state Kalman gain but in a different setup: The model is assumed to be completely unknown. However, the algorithm requires access to a simulator that provides noisy measurement of the MSE E [ ∥ X ( t ) -ˆ X ( t ) ∥ 2 ] which requires generation of ground-truth state trajectories X ( t ) (see Assumption 3.2 therein). The proposed approaches are different: zeroth-order global optimization (theirs) vs first-order stochastic gradient descent (ours). The analysis reports a O ( ε -2 ) sample complexity on optimal policy error which aligns with our result. However, it is difficult to provide more detailed comparison as explicit dependence of the error terms on problem dimension is not provided. Finally, [38] considers the problem of simultaneously learning the optimal Kalman filter and linear feedback control policy in the LQG setup. Their approach involves solving Semidefinite Programmings (SDP) using input-state-output trajectories. Their result, for the the case when trajectories involve noise, relies on the assumption that the magnitude of the noise can be made arbitrary small. This is in contrast to our setup where we only assume a bound on the noise level and do not require access to state trajectories.

Appendix B Summary of Assumptions

Herein, we provide a summary of all assumptions considered in our work and their relation to our results.

Note that if the given pair ( A,H ) of system parameters is not detectable, then the estimation problem is not well-posed, simply because there may not exist any stabilizing policy L 0 . Therefore, we consider the minimum required assumption for well-posedness of the problem as follows:

Assumption 1. The pair ( A,H ) is detectable, and the pair ( A,Q 1 2 ) is stabilizable, where Q 1 2 is the unique positive semidefinite square root of Q .

It is indeed possible to work with the weaker conditions compared with Assumption 1 and obtain similar results presented here-by incorporating more system theoretic tools. However, in order to improve the clarity of the learning problem with less system theoretic technicalities, we work with the following stronger assumption in lieu of Assumption 1.

Assumption 2. The pair ( A,H ) is observable, and the noise covariances Q ≻ 0 and R ≻ 0 .

It is clear that Assumption 2 implies Assumption 1.

Next, in order to provide an independent analysis of the SGD algorithm for locally Lipschitz functions in presence of gradient bias, we consider the following assumption on the gradient oracle.

Assumption 3. Suppose, for some α > 0 , we have access to a biased estimate of the gradient ∇ ̂ J ( L ) such that, there exists constants s, s 0 > 0 implying ∥∇ ̂ J ( L ) -∇ J ( L ) ∥ F ≤ s ∥∇ J ( L ) ∥ F + s 0 for all L ∈ S α \ C τ .

Next, in order to utilize the complexity analysis of SGD algorithm provided in Theorem 1, we need to build an oracle that satisfies Assumption 3. This is oracle is built in Section 4.2, building on the following assumptions on the process and observation noise.

Assumption 4. Assume that (almost surely) ∥ x 0 ∥ , ∥ ξ ( t ) ∥ ≤ κ ξ and ∥ ω ( t ) ∥ ≤ κ ω for all t . Also, for simplicity, suppose the initial state has zero mean, i.e., m 0 = 0 n .

Then, based on this assumption, Theorem 2 provides lowerbounds on the trajectory length and the batch-size in order for the oracle to satisfy Assumption 3. Finally, the combined version of our results is presented in Theorem 3.

Appendix C Proof of the duality relationship: Proposition 1

By pairing the original state dynamics (1) and its dual (6):

Summing this relationship from t = 0 to t = T -1 yields,

Upon subtracting the estimate a ⊺ ˆ x L ( T ) , using the adjoint relationship

and z ( T ) = a , lead to

Squaring both sides and taking the expectation concludes the duality result in (8).

Consider the adjoint system (6) with the linear feedback law u ( t ) = L ⊺ z ( t ) . Then,

Therefore, as a function of a , u ( t ) = L ⊺ ( A ⊺ L ) T -t a . These relationships are used to identify the control policy

This control policy corresponds to an estimation policy by the adjoint relationship (14):

This relationship holds for all a ∈ R n . Therefore,

which coincides with the Kalman filter estimate with constant gain L given by the formula (4) (with m 0 = 0 ). Therefore, the adjoint relationship (14) relates the control policy with constant gain L ⊺ to the Kalman filter with constant gain L . The result (9) follows from the identity

and the application of the first result (9) with a = H i .

Appendix D Proofs for the results for the analysis of the optimization landscape

D.1 Preliminary lemma

The following lemmas are a direct consequence of duality and useful for our subsequent analysis. Lemma 7. The set of Schur stabilizing gains S is regular open, contractible, and unbounded when m ≥ 2 and the boundary ∂ S coincides with the set { L ∈ R n × m : ρ ( A -LH ) = 1 } . Furthermore, J ( . ) is real analytic on S .

Proof. Consider the duality described in Remark 3. The proof then follows identical to [23, Lemmas 3.5 and 3.6] by noting that the spectrum of a matrix is identical to the spectrum of its transpose.

Next, we present the proof of Lemma 1 that provides sufficient conditions to recover the coercive property of J ( . ) which resembles Lemma 3.7 in [23] (but extended for the time-varying parameters). The proof is removed on the account of space and appears in the extended version of this work. Remark 9 . This approach recovers the claimed coercivity also in the control setting with weaker assumptions. In particular, using this result, one can replace the positive definite condition on the covariance of the initial condition in [23], i.e., Σ ≻ 0 , with just the controllability of ( A, Σ 1 / 2 ) .

D.2 Proof of Lemma 1

Proof. Consider any L ∈ S and note that the right eigenvectors of A and A L that are annihilated by H are identical. Thus, by Popov-Belevitch-Hautus (PBH) test, observability of ( A,H ) is equivalent to observability of ( A L , H ) . Therefore, there exists a positive integer n 0 ≤ n such that

is full-rank, implying that H ⊺ n 0 ( L ) H n 0 ( L ) is positive definite. Now, recall that for any such stabilizing gain L , we compute

where we used the cyclic property of trace and the inequality follows because for any PSD matrices P 1 , P 2 ⪰ 0 we have

Also, X n 0 ( L ) is well defined because A L is Schur stable if and only if ( A L ) n 0 is. Moreover, X n 0 ( L ) coincides with the unique solution to the following Lyapunov equation

Next, as R ⪰ 0 ,

where the last inequality follows by the fact that

with ∥ · ∥ op denoting the operator norm induced by 2-norm. Now, by Lemma 7 and continuity of the spectral radius, as L k → ∂ S we observe that ρ ( A L k ) → 1 . But then, the obtained lowerbound implies that J ( L k ) →∞ . On the other hand, as Q ≻ 0 , R ≻ 0 are both time-independent, by using a similar technique we also provide the following lowerbound

where ∥ · ∥ F denotes the Frobenius norm. Therefore, by equivalency of norms on finite dimensional spaces, ∥ L k ∥ → ∞ implies that J ( L k ) →∞ which concludes that J ( . ) is coercive on S . Finally, note that for any L ̸∈ S , by (17) we can argue that J ( L ) = ∞ , therefore the sublevel sets S α ⊂ S whenever α is finite. The compactness of S α is then a direct consequence of the coercive property and continuity of J ( . ) (Lemma 7).

D.3 Derivation of the gradient formula in Lemma 2

Next, we aim to compute the gradient of J in a more general format. We do the derivation for time-varying R and Q with specialization to time-invariant setting at the end. For any admissible ∆ , we have

where the ⋆ is hiding the following term

Therefore, by linearity and cyclic permutation property of trace, we get that

Finally, by considering the Euclidean metric on real matrices induced by the inner product ⟨ Q,P ⟩ = tr [ Q ⊺ P ] , we obtain the gradient of J as follows

whenever the series are convergent! And, by switching the order of the sums it simplifies to

For the case of time-independent Q and R , this reduces to

where Y ( L ) = Y is the unique solution of

D.4 Proof of existence of global minimizer in Lemma 2

The domain S is non-empty whenever ( A,H ) is observable. Thus, by continuity of L → J ( L ) , there exists some finite α > 0 such that the sublevel set S α is non-empty and compact. Therefore, the minimizer is an interior point and thus must satisfy the first-order optimality condition ∇ J ( L ∗ ) = 0 . Therefore, by coercivity, it is stabilizing and unique which satisfies

with X ∗ being the unique solution of

As expected, the global minimizer L ∗ is equal to the steady-state Kalman gain, but explicitly dependent on the noise covariances Q and R .

D.5 Proof of gradient dominance property in Lemma 2

Note that X = X ( L ) satisfies

Then, by combining (18) and (19), and some algebraic manipulation, we recover part of the gradient information, i.e., ( -LR + A L XH ⊺ ) , in the gap of cost matrices by arriving at the following identity

where the upperbound is valid for any choice of a > 0 . Now, as R ≻ 0 , we choose a = λ ( R ) / 2 . As X ⪰ 0 , it further upperbounds

Now, let ˜ X and ̂ X be, respectively, the unique solution of the following Lyapunov equations

Then by comparison, we conclude that

Recall that by the fact in (16),

Let Y ∗ ≻ 0 be the unique solution of

then, by cyclic permutation property

where the last equality follows by the obtained formula for the gradient ∇ J ( L ) . Similarly, we obtain that

Notice that the mapping L → Y ( L ) is continuous on S ⊃ S α , and also by observability of ( A,H ) , Y ( L ) ≻ 0 for any L ∈ S . To see this, let H n 0 ( L ) ≻ 0 be as defined in Lemma 1. Then,

Now, by Lemma 1, S α is compact and therefore we claim that the following infimum is attained with some positive value κ α :

Finally, the first claimed inequality follows by combining the inequalities (21), (22) and (23), with the following choice of

For the second claimed inequality, one arrive at the following identity by similar computation to (20):

where the second equality follows because Y ( L ) ≻ 0 and thus

Recall that then by the equality in (20) and cyclic property of trace we obtain

Therefore, for any L ∈ S α , we have

and thus, we complete the proof of first part by the following choice of

D.6 Proof of Lipschitz property in Lemma 2

Next, we provide the proof of locally Lipschitz property. Notice that the mappings L → X ( L ) , L → Y ( L ) and L → A L are all real-analytic on the open set S ⊃ S α , and thus so is the mapping L →∇ J ( L ) = 2 Y ( L ) [ -LR + A L X ( L ) H ⊺ ] . Also, by Lemma 1, S α is compact and therefore the mapping L → ∇ J ( L ) is ℓ -Lipschitz continuous on S α for some ℓ = ℓ ( α ) > 0 . In the rest of the proof, we attempt to characterize ℓ ( α ) in terms of the problem parameters. By direct computation we obtain

Therefore,

where

On the other hand, by direct computation we obtain

where

Now, consider the mapping L → Z ( L ) where Z ( L ) = Z is the unique solution of the following Lyapunov equation:

which is well-defined and continuous on S ⊃ S α . Therefore, by comparison, we claim that

where

By a similar computation to that of (20), we obtain that

where

Therefore, by comparison, we claim that

Finally, by compactness of S α , we claim that the following supremums are attained and thus, are achieved with some finite positive values:

Then, the claim follows by combining the bound in (25) with (26) and (27), and the following choice of

D.7 Gradient Flow (GF)

For completeness, in the next two sections, we anlayze first-order methods in order to solve the minimization problem (10), although they are not part of the main paper. In this section, we consider a policy update according to the the GF dynamics:

We summarize the convergence result in the following proposition which is a direct consequence of Lemma 2.

Proposition 6. Consider any sublevel set S α for some α > 0 . Then, for any initial policy L 0 ∈ S α , the GF updates converges to optimality at a linear rate of exp( -c 1 ( α )) (in both the function value and the policy iterate). In particular, we have and

Proof. Consider a Lyapunov candidate function V ( L ) := J ( L ) -J ( L ∗ ) . Under the GF dynamics

Therefore, L s ∈ S α for all s > 0 . But then, by Lemma 2, we can also show that

By recalling that c 1 ( α ) > 0 is a positive constant independent of L , we conclude the following exponential stability of the GF:

for any L 0 ∈ S α which, in turn, guarantees convergence of J ( L s ) → J ( L ∗ ) at the linear rate of exp( -c 1 ( α )) . Finally, the linear convergence of the policy iterates follows directly from the second bound in Lemma 2:

The proof concludes by noting that V ( L 0 ) ≤ α -J ( L ∗ ) for any such initial value L 0 ∈ S α .

D.8 Gradient Descent (GD)

Here, we consider the GD policy update:

for k ∈ Z and a positive stepsize η k . Given the convergence result for the GF, establishing convergence for GD relies on carefully choosing the stepsize η k , and bounding the rate of change of ∇ J ( L ) -at least on each sublevel set. This is achieved by the Lipschitz bound for ∇ J ( L ) on any sublevel set.

In what follows, we establish linear convergence of the GD update. Our convergence result only depends on the value of α for the initial sublevel set S α that contains L 0 . Note that our proof technique is distinct from those in [23] and [50]; nonetheless, it involves a similar argument using the gradient dominance property of J .

Theorem 4. Consider any sublevel set S α for some α > 0 . Then, for any initial policy L 0 ∈ S α , the GD updates with any fixed stepsize η k = η ∈ (0 , 1 /ℓ ( α )] converges to optimality at a linear rate of 1 -ηc 1 ( α ) / 2 (in both the function value and the policy iterate). In particular, we have

and with c 1 ( α ) and c 3 ( α ) as defined in Lemma 2.

̸

Proof. First, we argue that the GD update with such a step size does not leave the initial sublevel set S α for any initial L 0 ∈ S α . In this direction, consider L ( η ) = L 0 -η ∇ J ( L 0 ) for η ≥ 0 where L 0 = L ∗ . Then, by compactness of S α and continuity of the mapping η → J ( L ( η )) on S ⊃ S α , the following supremum is attained with a positive value ¯ η 0 :

where positivity of ¯ η 0 is a direct consequence of the strict decay of J ( L ( η )) for sufficiently small η as ∇ J ( L 0 ) = 0 . This implies that L ( η ) ∈ S α ⊂ S for all η ∈ [0 , ¯ η 0 ] and J ( L (¯ η 0 )) = α . Next, by the Fundamental Theorem of Calculus and smoothness of J ( · ) (Lemma 7), for any η ∈ [0 , ¯ η 0 ] we have that,

̸

where ∥ · ∥ F denotes the Frobenius norm, the first inequality is a consequence of Cauchy-Schwartz, and the second one is due to (11c) and the fact that L ( ηs ) remains in S α for all s ∈ [0 , 1] . 3 By the definition of L ( η ) , it now follows that,

This implies J ( L ( η )) ≤ J ( L 0 ) for all η ≤ 2 /ℓ ( α ) , and thus concluding that ¯ η 0 ≥ 2 /ℓ ( α ) . This justifies that L ( η ) ∈ S α for all η ∈ [0 , 2 /ℓ ( α )] . Next, if we consider the GD update with any

fixed stepsize η ∈ (0 , 1 /ℓ ( α )] and apply the bound in (28) and the gradient dominance property in Lemma 2, we obtain

which by subtracting J ( L ∗ ) results in

as ηc 1 ( ℓ ( α ) η/ 2 -1) ≤ -ηc 1 / 2 for all η ∈ (0 , 1 /ℓ ( α )] . By induction, and the fact that both c 1 ( α ) and the choice of η only depends on the value of α , we conclude the convergence in the function value at a linear rate of 1 -( ηc 1 / 2) and the constant coefficient of α -J ( L ∗ ) ≥ J ( L 0 ) -J ( L ∗ ) . To complete the proof, the linear convergence of the policy iterates follows directly from the second bound in Lemma 2.

Appendix E Proofs for the analysis of the constrained SGD algorithm

E.1 Proof of lemma 3: Derivation of stochastic gradient formula

Proof. For small enough ∆ ∈ R n × m ,

The difference

with the following terms that are linear in ∆ :

Therefore, combining the two identities, the definition of gradient under the inner product ⟨ A,B ⟩ := tr [ AB ⊺ ] , and ignoring the higher order terms in ∆ yields,

which by linearity and cyclic permutation property of trace reduces to:

This holds for all admissible ∆ , concluding the formula for the gradient.

E.2 Proof of Lemma 4: Robustness of the policy with respect to perturbation

Proof. Recall the stability certificate s K as proposed in [30, Lemma IV.1] for a choice of constant mapping Q : K → Λ ≻ 0 and dual problem parameters as discussed in Remark 3. Then, we arrive at s K = λ (Λ) / (2 λ ( Z ) ∥ H ⊺ ∆ ⊺ ∥ ) , for which ρ ( A ⊺ + H ⊺ ( L ⊺ + η ∆ ⊺ )) < 1 for any η ∈ [0 , s K ] . But, the spectrum of a square matrix and its transpose are identical, thus L + η ∆ ∈ S for any such η . The claim then follows by noting that the operator norm of a matrix and its transpose are identical, and the resulting lowerbound as follows s K ≥ λ (Λ) / (2 λ ( Z ) ∥ H ∥∥ ∆ ∥ F ) .

E.3 Proof of Lemma 5: Uniform lower-bound on stepsize

̸

Proof. Without loss of generality, suppose L 0 = L ∗ and E = 0 , and let L ( η ) := L 0 -ηE . By compactness of S β and continuity of the mapping η → J ( L ( η )) on S ⊃ S β , the following supremum is attained by η β :

̸

Note that η β is strictly positive for β > α because L 0 ∈ S α ⊂ S β and J ( · ) is coercive (Lemma 1) and its domain is open (Lemma 4). This implies that L ( η ) ∈ S β ⊂ S for all η ∈ [0 , η β ] and J ( L ( η β )) = β .

Next, we want to show that η β is uniformly lower bounded with high probability. By the Fundamental Theorem of Calculus, for any η ∈ [0 , η β ] we have

where the first inequality is a consequence of Cauchy-Schwartz, and the second one is due to (11c) and the fact that L ( ηs ) remains in S β for all s ∈ [0 , 1] by definition of η β . Note that the assumption implies ∥ E ∥ F ≤ ( γ +1) ∥∇ J ( L 0 ) ∥ F . Thus, (30) implies that

Therefore, for η to be a feasible point in the supremum in (29), it suffices to satisfy:

or equivalently,

But then it suffices to have

Finally, note that by Lemma 2 and (11c) we have

Using this uniform bound of gradient on sublevel set S α and noting that γ ∈ [0 , 1] , we can obtain the sufficient condition for η to be feasible. This completes the proof.

E.4 Proof of Proposition 2: Linear decay in cost value

̸

Proof. Suppose L 0 = L ∗ and let L ( η ) := L 0 -ηE . Note that E may not be necessarily in the direction of decay in J ( L ) , however, we can argue the following:

Choose β = α in Lemma 5 and note that η β as defined in (29) will be lower bounded as η β ≥ ¯ η 0 , and thus ¯ η 0 is feasible. Recall that L ( η ) ∈ S β ⊂ S for all η ∈ [0 , η β ] . Also, for any η ∈ [0 , ¯ η 0 ] and γ ∈ [0 , 1) , from (31) we obtain that:

where, as γ < 1 , the last inequality follows by (11a) for any η ≤ min { 2¯ η 0 , η β } . By the choice of ¯ η 0 , then we obtain that

This implies that

E.5 Proof of Theorem 1: Convergence of SGD algorithm

Before proving this result, we first discuss that the claim of Theorem 1 is sufficient for establishing the complexity result of Theorem 3; i.e., it suffices to guarantee convergence from every initial (stabilizing) policy to a neighborhood of optimality where norm of the gradient is controlled.

As shown in Lemma 2 (and discussed in Remark 4), the cost maintains the PL property on each sublevel set. In particular (11a) implies that on each S α , ∥∇ J ( L ) ∥ characterizes the optimality gap J ( L ) -J ( L ∗ ) by:

for some constant c 1 ( α ) . This implies that if we have arrived at a candidate policy L k for which the gradient is small, then the optimality gap should be small (involving the constant c 1 ( α ) that is independent of L k ). This is the reason that in Theorem 1, it suffices to argue about the generated sequence L k to have a linear decay unless entering a neighborhood of L ∗ containing policies with small enough gradients (denoted by C τ ). In particular, if for some j < k , we arrive at some policy L j ∈ C τ , then by (11a) we can conclude that:

which is directly controlled by the bias term s 0 . This is the bound used also in the proof of Theorem 3.

Finally, recall that every (stabilizing) initial policy L 0 amounts to a finite value of J ( L 0 ) and thus lies in some sublevel set S α . So, starting from such L 0 , Theorem 1 guarantees linear decay of the optimality gap till the trajectory enters that small neighborhood. Finally, note that the radius of this neighborhood C τ is characterized by the bias term s 0 which itself is exponentially decaying to zero in the trajectory length T .

Next, we provide the proof of this result that is essentially an induction argument using Proposition 2.

Proof of Theorem 1. The first step of the proof is to show that the assumption of Proposition 2 is satisfied for E = ∇ ̂ J ( L ) for all L ∈ S α \ C γ/ 2 . This is true because

where the first inequality follows from Assumption 3, the second inequality follows from L ̸∈ C γ/ 2 (i.e. s 0 ∥∇ J ( L ) ∥ ≥ γ/ 2 ), and the last step follows from the assumption s ≤ γ/ 2 .

The rest of the proof relies on repeated application of Proposition 2. In particular, starting from L 0 ∈ S α \ C γ/ 2 , the application of Proposition 2 implies that L 1 := L 0 -¯ η ∇ ̂ J ( L 0 ) remains in the same sublevel set, i.e., L 1 ∈ S α , and we obtain the following linear decay of the cost value:

Now, if L 1 ∈ C γ/ 2 then we stop; otherwise L 1 ∈ S α \ C γ/ 2 and we can repeat the above process to arrive at L 2 := L 1 -¯ η ∇ ̂ J ( L 1 ) ∈ S α , with a guaranteed linear decay

Combining the last two linear decays yields

Repeating the process generates a sequence of policies L 0 , L 1 , L 2 ... with a combined linear decay of

unless at some iteration j , we arrive at a policy L j such that L j ∈ C γ/ 2 . This completes the proof.

Appendix F Proofs of the result for observation model and sample complexity

F.1 Preliminary lemmas and their proofs

First, we provide the proof for the complete version of Lemma 6: 4

Lemma 6' (Uniform Bounds for Stable Systems) . Suppose L ∈ S , then there exit a constants C L > 0 such that

whenever ρ ( A L ) > 0 , and otherwise √ ρ ( A L ) is replaced with any arbitrarily small r ∈ (0 , 1) . Additionally,

Furthermore, consider S α for some α > 0 , then there exist constants D α > 0 , C α > 0 and ρ α ∈ (0 , 1) such that ∥ L ∥ ≤ D α , C L ≤ C α and ρ ( A L ) ≤ ρ α , ∀ L ∈ S α .

Proof of Lemma 6. Recall the Cauchy Integral formula for matrix functions [53, Theorem 1.12]: for any matrix M ∈ C n × n ,

whenever f is real analytic on and inside a closed contour Γ that encloses spectrum of M . Note that L ∈ S implying that ρ ( A L ) < 1 . Now, fix some r ∈ ( ρ ( A L ) , 1) and define Γ( θ ) = re iθ with θ ranging on [0 , 2 π ] . Therefore, Cauchy Integral formula applies to f ( z ) = z k for any positive integer k and the contour Γ defined above. So, for matrix A L , we obtain

implying that

Finally, the first claim follows by choosing r = √ ρ ( A L ) (whenever ρ ( A L ) > 0 , otherwise r ∈ (0 , 1) can be chosen arbitrarily small) and defining

which is attained and bounded.

Next, by applying the first claim, we have

implying the second bound. For the third claim, note that for i = 1 , 2 , · · · we obtain

But, then by recalling that M 0 [ E ] = 0 and ∥ M 1 [ E ] ∥ = ∥ EH ∥ we have

where we used the following convergent sum for any ρ ∈ (0 , 1) :

This implies the third bound.

The final claim follows directly from compactness of sublevel set S α (Lemma 1) and continuity of the mappings ( L, θ ) ↦→ ( ρ ( A L ) , θ ) ↦→∥ ( √ ρ ( A L ) e iθ I -A L ) -1 ∥ on S α × [0 , 2 π ] whenever ρ ( A L ) > 0 (and otherwise considering the mapping ( L, θ ) ↦→∥ ( re iθ I -A L ) -1 ∥ for arbitrarily small and fixed r ∈ (0 , 1) ).

As mentioned in Section 4.2, a key idea behind these error bounds that scale well with respect to the length T is the following consequence of von Neumann Trace Inequality [49, Theorem 8.7.6]:

with ∥ · ∥ ∗ denoting the nuclear norm. Additionally, as a direct consequence of Courant-Fischer Theorem, one can also show that nuclear norm is sub-multiplicative. More precisely,

Next, we require the following lemma to bound these errors.

Lemma 8. For any L ∈ S α , we have

Proof of Lemma 8. For the first claim, note that A ⊺ L H ⊺ H A L is positive semi-definite, so

where the last inequality follows by Lemma 6. Next, we have

where the second inequality follows by (33) and the third one by the following convergent sum for any ρ ∈ (0 , 1) :

Also, by the properties of nuclear norm

where the last inequality follows by Lemma 6. Finally, notice that

and thus combining the last three bounds implies the second claim. This completes the proof.

The next tool we will be using is the following famous bound on random matrices which is a variant of Bernstein inequality:

Lemma 9 (Matrix Bernstein Inequality [54, Corollary 6.2.1]) . Let Z be a d 1 × d 2 random matrices such that E [ Z ] = ¯ Z and ∥ Z ∥ ≤ K almost surely. Consider M independent copy of Z as Z 1 , · · · , Z M , then for every t ≥ 0 , we have

where σ 2 = max {∥ E [ ZZ ⊺ ] ∥ , ∥ E [ Z ⊺ Z ] ∥} is the per-sample second moment. This bound can be expressed as the mixture of sub-gaussian and sub-exponential tail as ( d 1 + d 2 ) exp { -c min { t 2 σ 2 , t 2 K } } for some c .

We are now well-equipped to provide the main proofs.

F.2 Proof of Proposition 3: The Observation model

Proof. Recall that

where H ⊺ i is the i -th row of H . Also, by duality, if z ( t ) = ( A ⊺ L ) T -t H i is the adjoint dynamics' closed-loop trajectory with control signal u ( t ) = L ⊺ z ( t ) then

But ⃗ u = ( I ⊗ L ⊺ ) ⃗ z i and then ⃗ z i = A ⊺ L H i . Therefore,

Then, by using the fact that ∑ m i =1 H i H ⊺ i = H ⊺ H , we obtain that

Thus, we can rewrite the estimation error as

Next, we can compute that for small enough E

where

This implies that

On the other hand,

Therefore, the second claim follows by the chain rule.

F.3 Proof of Proposition 4: Concentration bounds

We provide the proof for a detailed version of Proposition 4:

Proposition 4' (Concentration independent of length T ) . Consider length T trajectories {Y i [ t 0 ,t 0 + T ] } i M =1 and let ̂ J T ( L ) := 1 M ∑ M i =1 ε ( L, Y i T ) . Then, under Assumption 4, for any s > 0

where κ L = κ ξ + ∥ L ∥ κ ω and

Proof of Proposition 4. Note that E [ ⃗ ξ ⃗ ξ ⊺ ] = ( I ⊗ Q 0 0 P 0 ) =: Q , E [ ⃗ ω⃗ ω ⊺ ] = ( I ⊗ R 0 0 0 m ) =:

R , and E [ ⃗ ξ ⃗ ω ⊺ ] = 0 . Assume m 0 = 0 and recall that ⟨∇ ε ( L, Y T ) , E ⟩ = d ε ( · , Y T ) ∣ ∣ L ( E ) thus, using Proposition 3, we can rewrite the J T ( L ) and its gradient as

where E [ X L ] = Q +( I ⊗ L ) R ( I ⊗ L ⊺ ) . Therefore, by definition of ̂ J T ( L ) we obtain

with Z L = 1 M ∑ M i =1 X L ( Y i ) which can be expanded as

where

and E [ Z L ] = Q +( I ⊗ L ) R ( I ⊗ L ⊺ ) . Therefore,

Thus, by cyclic permutation property of trace and (12) we obtain

Next, we consider the symmetric random matrix A L ( Z L -E [ Z L ]) A ⊺ L . Note that ∥ ξ ( t ) -Lω ( t ) ∥ ≤ κ L almost surely and thus

It then follows that

Therefore, by Lemma 9 we obtain that

Substituting t with t/ ∥ H ⊺ H ∥ ∗ together with (34) implies the first claim.

Similarly, we can compute that

and thus

Thus, by cyclic permutation property of trace and (12) we obtain that

where S L ( E, Y ) is the symmetric part of the following random matrix

Next, we provide the following almost sure bounds for each term: first,

where the last equality follows by Lemma 6; second, similarly

and finally

where the last inequality follows by Lemma 6. Now, by combining the last three bounds we can claim that almost surely

This also implies that

Therefore, by Lemma 9 we obtain that

Thus, by substituting t with t ∥ E ∥ / ∥ H ⊺ H ∥ ∗ and applying this bound to (35) we obtain that

Finally, choosing E = ∇ ̂ J T ( L ) -∇ J T ( L ) proves the second claim.

F.4 Proof of the Proposition 5: Truncation error bound

We provide the proof for a detailed version of Proposition 5:

Proposition 5' (Truncation Error Bound) . Suppose m 0 = 0 , then under Assumption 4 we have

and where

Proof of Proposition 5. For the purpose of this proof, we denote the same matrices by A L,T and M L,T [ E ] in order to emphasize on length T . Recall that

where E [ X L ] = Q +( I ⊗ L ) R ( I ⊗ L ⊺ ) , which implies

On the other hand, and thus

Therefore, by the properties of trace and Lemma 6 we obtain that

where the last line follows by Lemma 8. So,

This, together with Assumption 4 imply the first claim.

Next, for simplicity we adopt the notation A L, ∞ to interpret the limit as t →∞ , then similar to the proof of Proposition 4 we can compute that

Therefore, using (12) and Lemma 6 we have the following bound

which by Lemma 8 is bounded as follows

Finally, choosing E = ∇ J ( L ) -∇ J T ( L ) together with Assumption 4 implies

Finally, the second claim follows by the following simple facts:

as max t ≥ 0 tρ t = 2 e ln 1 /ρ ≤ 1 ln 1 /ρ ≤ 1 1 -ρ for any ρ ∈ (0 , 1) . This completes the proof.

F.5 Complete version of Theorem 2: Sample complexity bounds for the stochastic oracle

The following is a detailed version of Theorem 2:

Theorem 2'. Suppose m 0 = 0 n and Assumption 4 holds for a data-set {Y i T } i M =1 . Define ∇ ̂ J T ( L ) := 1 M ∑ M i =1 ∇ ε ( L, Y i T ) , where ∇ L ε ( L, Y ) is obtained in Lemma 3. Consider S α for some α > 0 and any s, s 0 > 0 and τ ∈ (0 , 1) . Suppose the trajectory length

and the batch size

where

with ρ α , C α and D α defined in Lemma 6. Then, with probability no less than 1 -δ , Assumption 3 holds.

F.6 Additional concentration bound results

Combining the truncation bound in Proposition 5 with concentration bounds in Proposition 4 we can provide probabilistic bounds on the 'estimated cost' ̂ J T ( L ) and the 'estimated gradient' ∇ ̂ J T ( L ) . The result involves the bound for the Frobenius norm of the error with probabilities independent of T .

Theorem 2, can be viewed as a simplified application of Theorem 5 to characterize the required minimum trajectory length and minimum batch so that the approximate gradient satisfies Assumption 3, with a specific s and s 0 .

Theorem 5. Suppose Assumption 4 holds. For any s > 0 and L ∈ S α , if

then with probability no less than 1 -δ ,

with ν L and ¯ γ L defined in Proposition 4 and Proposition 5, respectively.

Proof of Theorem 5. Recall that for any L ∈ S α for some α > 0 we have

Thus, by Proposition 4 with s replaced by s ∥∇ J ( L ) ∥ F / √ min( n, m ) and applying Proposition 5 to the second term, we obtain that with probability at least 1 -δ :

where

Noticing ∥∇ ̂ J T ( L ) -∇ J ( L ) ∥ F ≤ √ min( n, m ) ∥∇ ̂ J T ( L ) -∇ J ( L ) ∥ and rearranging terms will complete the proof.

One can also provide the analogous concentration error bounds where the probabilities are independent of the system dimension n .

Proposition 7 (Concentration independent of system dimension n ) . Under the same hypothesis, we have

and

where

Proof of Proposition 7. Similar to the previous proof, we have

and thus, by (12) we obtain

Next, we consider the symmetric random matrix ( Z L -E [ Z L ]) and recall that ∥ ξ ( t ) -Lω ( t ) ∥ ≤ κ L almost surely; thus

It then follows that

Therefore, by Lemma 9 we obtain that

Substituting t with t/ ∥A ⊺ L H ⊺ H A L ∥ ∗ together with (36) implies the first claim because by Lemma 8 κ 2 L T 2 ∥A ⊺ L H ⊺ H A L ∥ ∗ ≤ ¯ µ L T .

Again, similar to (35) in the previous proof, by (12) we obtain that

where S L ( E, Y ) is the symmetric part of the following random matrix

So, we claim that almost surely

and thus

Therefore, by Lemma 9 we obtain that

Substituting t with t/ ∥A ⊺ L H ⊺ H A L ∥ ∗ implies that

because by Lemma 8 we have κ 2 L T 2 ∥A ⊺ L H ⊺ H A L ∥ ∗ ≤ ¯ µ L T .

Next, by substituting t with t/ ∥N L [ E ] ∥ ∗ in (37) we have

because Lemma 8 implies that κ 2 L T 2 ∥N L [ E ] ∥ ∗ ≤ ¯ ν L T ∥ E ∥ . Thus, by combining the last two inequalities and using the union bound for (38) we obtain that

Finally, substituting t with t ∥ E ∥ and choosing E = ∇ ̂ J T ( L ) -∇ J T ( L ) proves the second claim.

Remark 10 . We obtained a better bound for truncation of the gradient as

which has been simplified for clarity of the presentation.

Appendix G Numerical Results

Herein, we showcase the application of the developed theory for improving the estimation policy for an LTI system. Specifically, we consider an undamped mass-spring system with known parameters ( A,H ) with n = 2 and m = 1 . In the hindsight, we consider a variance of 0 . 1 for each state dynamic noise, a state covariance of 0 . 05 and a variance of 0 . 1 for the observation noise. Assuming a trajectory of length T at every iteration, the approximate gradient is obtained as in Lemma 3, only requiring an output data sequence collected from the system in (1). Then, the progress of policy updates using the SGD algorithm for different values of trajectory length T and batch size M are depicted in Figure 1 where each figure shows average progress over 50 rounds of simulation. The figure demonstrates a linear convergence outside of a neighborhood of global optimum that depends on the bias term in the approximate gradient (due to truncated data trajectories). The rate then drops when the policy iterates enter into this neighborhood which is expected as every update only relies on a biased gradient -in contrast to the linear convergence established for deterministic GD (to the exact optimum) using the true gradient.

Specifically, recall that our convergence guarantee to a small neighborhood around the optimal value is due to the finite-length of the data trajectories. The region can be made arbitrary small by choosing larger trajectory length. In particular, to achieve ε error, we only require the length T ≥ O (ln(1 /ε )) -see Theorem 3. Also, Fig 1(d) is illustrating that optimality gap at the final iteration (i.e. the radius of the small neighborhood around optimality) which is decaying linearly as a function of trajectory length T ≤ 50 -until the variance error dominates beyond T = 50 . It is clear that, increasing the batch size M will allow further decrease of this optimality gap beyond T = 50 .

The code for regenerating these results is available online at this GitHub repository [55].

Appendix H Nomenclature

Constant Values

α Scalar value of the cost

¯ γ α Uniform constant for gradient estimate-see Theorem 2

¯ γ L Constant for bounding bias of gradient estimate-see Proposition 5

δ Failure probability

η Step-size of SGD algorithm

γ Positive constant smaller than one-see Proposition 2

κ ω Scalar upperbound on 2-norm of measurement noise

κ ξ Scalar upperbound on 2-norm of dynamic noise

0

250

500

750

1000

1250

1500

1750

2000

iteration

10

4

10

3

10

2

10

1

10 0

J

(

Lk

)

J

(

L

)

J

(

L

)

M

=10

M

=20

M

=30

M

=40

M

=80

(a)

10 1

10 2

M

10

6

10

5

10

4

10

3

J

(

LK

)

J

(

L

)

J

(

L

)

(b)

0

250

500

750

1000

1250

1500

1750

2000

iteration

10

3

10

2

10

1

10 0

J

(

Lk

)

J

(

L

)

J

(

L

)

T

=10

T

=20

T

=30

T

=40

T

=50

(c)

20

40

60

80

100

T

10

7

10

6

10

5

10

4

10

3

10

2

10

1

J

(

LK

)

J

(

L

)

J

(

L

)

(d)

κ L Combined Scalar upperbound on 2-norm of noise

ν α Uniform constant for gradient estimate-see Theorem 2

ν L Combined matrix variance statistics of gradient estimate-see Proposition 4

k Iteration step of SGD algorithm

M Batch-size of SGD algorithm

m Dimension of observation vector

n Dimension of state vector

s Variance coefficient term of oracle model-see Assumption 3

s 0 Bias term of oracle model-see Assumption 3

T Length of output data trajectory

t Time step of the system

Other symbols

∥ · ∥ 2-norm of a matrix

∥ · ∥ F Frobenius-norm of a matrix

L Linear operator describing the filtering policy that maps measurements to state estimation

L † Adjoint of the operator L

X T Cost matrix on length T trajectory

Y Auxiliary cost matrix

y Output (measurement) vector

z Adjoint state vector

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper builds on the duality between estimation and control, in order to develop an online data-driven method for MSE-optimal filtering of linear systems with linear observations. Process and observation noise covariances are considered unknown, but stochastic states are assumed to be bounded.  The paper proposes using SGD in the space of steady-state  stabilizing gains, claims asymptotic convergence to the optimal gain, and provides an asymptotic probabilistic bound on the deviation from optimal error.\n\nSTRENGTHS:\nAlthough the use of online optimization for controlling linear-quadratic settings is not new (e.g. [r1]), exploiting the duality between control and filtering problems is novel and interesting in this context. The concentration and error bounds are not trivial and useful, as concentration bound is non-asymptotic in the series length T.\nProof of SGD convergence and error bounds are novel as well, even if derived under very strong assumptions.\n\n[r1] Cohen A. et al., \"Online Linear Quadratic Control\", 2018\n\nWEAKNESSES:\nOverall, I believe the authors did their best that the paper will be well-organized and clear (as it can be for a rather technical manuscript). The explanations and given outlines before each section are indeed helpful.  However, it is still very hard to follow the assumptions and constant definitions, and some of the conclusions. Writing becomes very laconic at some crucial points (e.g. Thm. 2, Remark 7).\n\nParticularly, Thm.2 and it's proof are not clear hence it is hard to get convinced in their soundness.  Furthermore, In the introduction it is stated that convergence is guaranteed from every initial policy, but according to Theorem 2 the policy cannot enter (or start in) a class of policies where gradient is smaller than some constant. I didn't find any discussion about when trajectories enter this region. My impression is that this is a good paper, and I might be missing something. I will be willing to raise my score when given a more detailed proof and this clarification.\n\nQUESTIONS:\nI ask for a more detailed proof for Thm. 2 (see above).\n\nIn addition, a summary of all assumptions, results and notations can be very useful.\n\n\n\nMinor typos:\n\nl. 249 't' should be replaced by $\\gamma$.\n\nl. 276 I think i should go between 1 and T-1.\n\nl. 309 quite.\n\nLIMITATIONS:\nThe online method uses a surrogate loss using the observations, this is a reasonable choice due to lack of ground-truth states and the observability, which is a strong assumption. The paper should discuss the implications of using this loss with more general (i.e. non observable) systems.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis submission examines the learning of the Kalman filter gain for linear systems with unknown covariance matrices using noisy output data. Similar to learning the linear quadratic regulators for unknown linear systems, the learning problem here is posed as a stochastic policy optimization problem which minimizes the expected output prediction error. Bridging together the learning of kalman gain and the optimal control gain, the paper provides an interesting convergence analysis of stochastic gradient descent for the policy optimization problem, and bias-variance error bounds of the learning problem by employing a set of tools from linear systems and stochastic geometry.\n\nSTRENGTHS:\n+) The learning of the Kalman gain as a stochastic policy optimization problem which is amendable to results and studies of learning linear quadratic regulators in the literature;\n+) the dual of learning the Kalman gain and the optimal control gain in the data-driven setting;\n+) biased gradients and stability constraints are strategically handled when learning the Kalman gain, along with the bias-variance error bounds.\n\nWEAKNESSES:\n-) The setup is not very practical as the system matrices are assumed perfectlt known but only the covariance matrices are unknown; even in e.g., the stated application aircraft wign dynamics where only approximate models are known, one would assume imperfect knowledge of both the system matrices and covariance matrices; or identifying them via data-driven method first. \n-) For the learning problem described in lines 106-108, it is unclear whether the problem is well-posed in the sense that given these information (data), wheher one is able to learn the steady -state Kalman gain L_\\infinity, or at least how large the horizon T shall be such that one will be able to have a unique L_\\infty?\n-) In (1), are uncorrelated random vectors enough for deriving the optimal Kalman filter without mutually independent random noise vectors? Also in the formulation of (5), what are random quantities and do we have any condition expectations or not?\n\nQUESTIONS:\nIt would be great if the results can be compared with more related works on learning the Kalman filter under different settings; Zheng, Y. et al. Sample complexity of linear quadratic gaussian (LQG) control for output feedback systems. In Learning for dynamics and control (pp. 559-570). PMLR. Zhang, X. et al. Learning the Kalman filter with fine-grained sample complexity. arXiv preprint arXiv:2301.12624. In the behavioral theory literature, results are available for learning the Kalman filter for unknown linear systems by using data, although seemingly in addition to the noisy output data and also the pure state data, in e.g., Liu, W. et al. Learning Robust Data-based LQG Controllers from Noisy Data. arXiv preprint arXiv:2305.01417.\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on learning the optimal filtering policy (Kalman gain) in a linear system with known system matrices and unknown covariance matrices of noise. The paper considers a proxy objective to avoid learning with hidden variables and characterizes a dual form of the objective, optimized subsequently using SGD. Convergence analysis and finite-time error bounds are provided.\n\nSTRENGTHS:\noriginality: studies a traditional Kalman filtering problem with a different setting and objective. The duality theory and the proposed method of optimizing the dual objective seem original.\n\nquality: provides detailed theoretical results and analysis.\n\nclarity: clearly formulate the setting. The whole picture from the background of Kalman filtering, problem setting, duality theory, to SGD convegence analysis is well organized and easy to follow.\n\nWEAKNESSES:\n1. The paper lacks enough empirical experiments to support their idea. The only two figures appear in the supplementary materials, but the linear convergence outside a region near optimal, along with reasons why performances of the $M=20$ and $T=200$ case is different from other settings, is not presented clearly in this paper. It is suggested that:\n - the figures can be plot in the log scale to illustrate the linear convergence; also it is better to illustrate in the plot the scale of the \"no-linear-rate\" small region;\n - additional studies to demonstrate why medium hyperparameters $M=20$ and $T=200$, yield worst/best convergence rate among other hyperparameters (otherwise it is likely to think that the proposed SGD approach is not so robust to hyperparameters);\n - maybe comparison with some existing methods on real data is preferred.\n\n2. The organization of this paper can be improved. For example,\n - numerical results can be put in main paper.\n - from my view, it is more natural to first characterize the bias in estimated gradient (Sec.4.2) and then provide convergence guarantees under such biased gradient (Sec.4.1). Otherwise, it may lead to confusion on the need of convergence analysis under biased estimates (e.g. the appeared $E$ in Lemma 5 & Proposition 2).\n\n3. Some typos:\n - Eq.(3b) $P(T)$ should be $P(t).\n\nQUESTIONS:\nNone\n\nLIMITATIONS:\nThe main limitation of the proposed approach is that the biased gradient causes the convergence rate to be linear only outside a small region near the true optimal value. It may not seem so preferable if the region is large, thus not ensuring exact convergence to optimal value.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the learning of optimal steady-state Kalman filter gain for a linear dynamical system with known system matrices but unknown process and measurement noise covariances. In particular, the learning process involves minimizing the prediction error of the observed output using a dataset of independently realized observation sequences. Leveraging the duality between LQR control and Kalman filtering, the authors reformulate this learning problem as synthesizing an optimal control policy for an adjoint system and propose a stochastic gradient descent (SGD) algorithm algorithm to solve it. The authors also provide sample complexity and non-asymptotic error guarantees by conducting a convergence analysis of SGD accounting for biased gradients and stability constraints. In particular, they show that the bias-variance error bounds scale logarithmically in system dimension and that the variance term doe snot change with the horizon.\n\nSTRENGTHS:\nI think this paper is well-written. The text is easy to read and and the  technical details are mostly neatly presented. The analysis also makes sense although I didn't verify in detail. The results are fairly general and significant as the problem of unknown noise covariances in Kalman filtering should be practically relevant as well.\n\nWEAKNESSES:\nI don't consider these as weaknesses necessarily but here are a few things I would recommend the authors to consider:\n1. The results of remark 7 can be discussed earlier maybe within the informal theorem 1.\n2. You might find it useful to adopt $(\\kappa,\\gamma)-$stability definition by Cohen et al. 2018 in your bounds involving $\\sqrt{\\rho(A_L)}$, especially in Lemma 6.\n3.it wasn't clear to me in the first place why singularity of $H^TH$ requires a significantly different treatment than LQR case. maybe you can clarify this earlier in the text. \n4. It would be helpful to discuss possible future directions.\n\nQUESTIONS:\nI have a few questions to the authors out of curiosity.  How essential is it to assume bounded noise for propositions 4 and 5? In figure 1a, could you explain why we see that M=10 case seems like the best performing , even better than M=30?\n\nLIMITATIONS:\nThe authors already addressed some of the limitations such as the requirement for perfect knowledge of system matrices. I would also consider the bounded noise setting one of the limitations that might be improved in the future works.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-17"}