SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In order to emphasize on the estimation time horizon  $T$  for various measurement sequences, we use  $\mathcal{V}_T\coloneqq \{y(t)\}_{t = 0}^T$  to denote the measurement time-span. Note that, any choice of  $L\in S$  corresponds to a filtering strategy that outputs the following prediction,
$$
\hat {y} _ {L} (T) = H A _ {L} ^ {T} m _ {0} + \sum_ {t = 0} ^ {T - 1} H A _ {L} ^ {T - t - 1} L y (t).
$$
We denote the squared-norm of the estimation error for this filtering strategy as,
$$
\varepsilon (L, \mathcal {Y} _ {T}) := \| e _ {T} (L) \| ^ {2},
$$
where  $e_T(L) \coloneqq y(T) - \hat{y}_L(T)$ . We also define the truncated objective function as
$$
J _ {T} (L) := \mathbb {E} \left[ \varepsilon (L, \mathcal {Y} _ {T}) \right],
$$
where the expectation is taken over all possible random measurement sequences, and note that, at the steady-state limit, we obtain  $\lim_{T\to \infty}J_T(L) = J(L)$ .
The SGD algorithm aims to solve this optimization problem by replacing the gradient, in the Gradient Descent (GD) update, with an unbiased estimate of the gradient in terms of samples from the measurement sequence. In particular, with access to an oracle that produces independent realization of the measurement sequence, say  $M$  random independent measurements sequences  $\{\mathcal{Y}_T^i\}_{i=1}^M$ , the gradient can be approximated as follows: denote the approximated cost value
$$
\widehat {J} _ {T} (L) := \frac {1}{M} \sum_ {i = 1} ^ {M} \varepsilon (L, \mathcal {Y} _ {T} ^ {i}),
$$
then the approximate gradient with batch-size  $M$  is  $\nabla \widehat{J}_T(L) = \frac{1}{M} \sum_{i=1}^{M} \nabla_{L} \varepsilon(L, \mathcal{Y}_T^i)$ . This forms an unbiased estimate of the gradient of the "truncated objective", i.e.,  $\mathbb{E}\left[\nabla \widehat{J}_T(L)\right] = \nabla J_T(L)$ . Next, for implementation purposes, we compute the gradient estimate explicitly in terms of the measurement sequence and the filtering policy  $L$ .
Lemma 3. Given  $L \in S$  and a sequence of measurements  $\mathcal{Y} = \{y(t)\}_{t=0}^{T}$ , we have,
$$
\begin{array}{l} \nabla_ {L} \varepsilon (L, \mathcal {Y}) = - 2 H ^ {\intercal} e _ {T} (L) y (T - 1) ^ {\intercal} \\ + 2 \sum_ {t = 1} ^ {T - 1} - (A _ {L} ^ {\intercal}) ^ {t} H ^ {\intercal} e _ {T} (L) y (T - t - 1) ^ {\intercal} + \sum_ {k = 1} ^ {t} (A _ {L} ^ {\intercal}) ^ {t - k} H ^ {\intercal} e _ {T} (L) y (T - t - 1) ^ {\intercal} L ^ {\intercal} (A _ {L} ^ {\intercal}) ^ {k - 1} H ^ {\intercal}. \\ \end{array}
$$
Finally, using this approximate gradient, the so-called SGD update proceeds as,
$$
L _ {k + 1} = L _ {k} - \eta_ {k} \nabla_ {L} \widehat {J} _ {T} (L),
$$
for  $k \in \mathbb{Z}$ , where  $\eta_k > 0$  is the step-size. Numerical results of the application of this algorithm appear in Appendix G.
Table 1: Differences between SGD algorithms for optimal LQR and optimal estimation problems
Problem	Parameters	Constraints	Gradient Oracle
cost value	Q and R	A and H	stability S	model	biased
LQR [25]	known	known	unknown	yes	E [J(L+rΔ)Δ] Δ ~ U(Smn)	yes
Estimation (this work)	unknown	unknown	known	yes	E [∇ε(L,γ)] γ ~output data	yes
Vanila SGD	*	*	*	no	E [∇ε(L,γ)] γ ~ data dist.	no
Remark 5. Computing this approximate gradient only requires the knowledge of the system matrices  $A$  and  $H$ , and does not require the noise covariance information  $Q$  and  $R$ . Simulation results for the SGD algorithm are provided in the supplementary material.
Although the convergence of the SGD algorithm is expected to follow similar to the GD algorithm under the gradient dominance condition and Lipschitz property, the analysis becomes complicated due to the possibility of the iterated gain  $L_{k}$  leaving the sub-level sets. It is expected that a convergence guarantee would hold under high-probability due to concentration of the gradient estimate around the true gradient. The complete analysis in this direction is provided in the subsequent sections.
We first provide sample complexity and convergence guarantees for SGD with a biased estimation of gradient for locally Lipschitz objective functions and in presence of stability constraint  $S$ . Subsequently, we study the stochastic problem of estimating the gradient for the estimation problem. Distinct features of our approach as compared with similar formulations in the literature are highlighted in Table 1.
We now provide a road map to navigate the technical results that concludes with Theorem 3: Section 4.1 is concerned with the convergence analysis of the SGD algorithm under an assumption for the biased gradient oracle, summarized in Theorem 1 which concludes the linear convergence of the iterates for sufficiently small stepsize. Section 4.2 is concerned with the bias-variance error analysis of the gradient estimate, summarized in Theorem 2 providing the sufficient values for the batch-size and trajectory length that guarantees the desired bound on the gradient oracle required in Theorem 1. Finally, combining the results from Theorem 1 and Theorem 2 concludes our main result in Theorem 3.
4.1 SGD with biased gradient and stability constraint
First, we characterize the "robustness" of a policy at which we aim to estimate the gradient. This is formalized in the following lemma which is a consequence of [30, Lemma IV.1].
Lemma 4. Consider any  $L \in S$  and let  $Z$  be the unique solution of  $Z = A_{L}ZA_{L}^{\intercal} + \Lambda$  for any  $\Lambda \succ 0$ . Then,  $L + \Delta \in S$  for any  $\Delta \in \mathbb{R}^{n \times m}$  satisfying  $0 \leq \| \Delta \|_F \leq \underline{\lambda}(\Lambda) / [2\overline{\lambda}(Z)\|H\|$ \n\n
Second, we provide a uniform lowerbound for the stepsize of gradient descent for an approximated direction "close" to the true gradient direction.
Lemma 5 (Uniform Lower Bound on Stepsize). Let  $L_0 \in S_\alpha$  for some  $\alpha \geq \alpha^* \coloneqq J(L^*)$ , and choose any finite  $\beta \geq \alpha$ . Consider any direction  $E$  such that  $\| E - \nabla J(L_0) \|_F \leq \gamma \| \nabla J(L_0) \|_F$  for some  $\gamma \in [0,1]$ , then we have  $J(L_0 - \eta E) \leq \beta$  for any  $\eta$  satisfying:
$$
0 \leq \eta \leq \frac {1 - \gamma}{(\gamma + 1) ^ {2}} \cdot \frac {1}{\ell (\beta)} + \frac {c _ {3} (\alpha)}{\ell (\alpha) [ \alpha - \alpha^ {*} ]} \sqrt {\frac {\beta - \alpha}{2 \ell (\beta)}}.
$$
Remark 6. Note that for the case of exact gradient direction, i.e. when  $E = \nabla J(L_0)$ , we have  $\gamma = 0$  and choosing  $\beta = \alpha$  implies the known uniform bound of  $\eta \leq \frac{1}{\ell(\beta)}$  for feasible step sizes as expected. Also, by this choice of  $\beta$ , this guarantees that the next iterate remains in sublevel set  $S_{\alpha}$ . This lemma generalizes this uniform bound for general directions and (potentially) larger sublevel set.
The next proposition provides a decay guarantee for one iteration of gradient descent with an approximate direction which will be used later for convergence of SGD with a biased gradient estimate.
Proposition 2 (Linear Decay in Cost Value). Suppose  $L_0 \in S_\alpha$  for some  $\alpha > 0$  and a direction  $E \neq 0$  is given such that  $\| E - \nabla J(L) \|_F \leq \gamma \| \nabla J(L) \|_F$  for some  $\gamma < 1$ . Let  $\bar{\eta} \coloneqq (1 - \gamma) / (\gamma + 1)^2 \ell(\alpha)$ . Then,  $L_1 \coloneqq L_0 - \bar{\eta} E$  remains in the same sublevel set, i.e.,  $L_1 \in S_\alpha$ . Furthermore, we obtain the following linear decay of the cost value:
$$
J \left(L _ {1}\right) - J \left(L ^ {*}\right) \leq \left[ 1 - c _ {1} (\alpha) \bar {\eta} (1 - \gamma) / 2 \right] \left[ J \left(L _ {0}\right) - J \left(L ^ {*}\right) \right].
$$
Next, we guarantee that SGD algorithm with this biased estimation of gradient obtains a linear convergence rate outside a small set  $\mathcal{C}_{\tau}$  around optimality defined as
$$
\mathcal {C} _ {\tau} := \{L \in \mathcal {S} \mid \| \nabla J (L) \| _ {F} \leq s _ {0} / \tau \},
$$
for some  $\tau \in (0,1)$  and arbitrarily small  $s_0 > 0$ . First, we assume access to the following oracle that provides a biased estimation of the true gradient.
Assumption 3. Suppose, for some  $\alpha >0$ , we have access to a biased estimate of the gradient  $\nabla \widehat{J} (L)$  such that, there exists constants  $s,s_0 > 0$  implying  $\| \nabla \widehat{J} (L) - \nabla J(L)\| _F\leq s\| \nabla J(L)\| _F + s_0$  for all  $L\in S_{\alpha}\setminus \mathcal{C}_{\tau}$ .
Theorem 1 (Convergence). Suppose Assumption 3 holds with small enough  $s$  and  $s_0$  such that  $s \leq \gamma / 2$  and  $S_{\alpha} \setminus \mathcal{C}_{\gamma / 2}$  is non-empty for some  $\gamma \in (0, 1)$ . Then, SGD algorithm starting from any  $L_0 \in S_{\alpha} \setminus \mathcal{C}_{\gamma / 2}$  with fixed stepsize  $\bar{\eta} := \frac{(1 - \gamma)}{(\gamma + 1)^2 \ell(\alpha)}$  generates a sequence of policies  $\{L_k\}$  that are stable (i.e. each  $L_k \in S_{\alpha}$ ) and cost values decay linearly before entering  $C_{\gamma / 2}$ ; i.e.,
$$
J (L _ {k}) - J (L ^ {*}) \leq \left[ 1 - c _ {1} (\alpha) \bar {\eta} (1 - \gamma) / 2 \right] ^ {k} \left[ J (L _ {0}) - J (L ^ {*}) \right],
$$
for each  $k \geq 0$  unless  $L_{j} \in \mathcal{C}_{\gamma /2}$  for some  $j \leq k$ .
Remark 7. By combining Theorem 1 and the PL property of the cost in (11a), we immediately obtain a sample complexity for our algorithm; e.g., choose  $\gamma = 1/2$ , then in order to guarantee an  $\varepsilon$  error on the cost  $J(L_k) - J(L^*) \leq \varepsilon$ , it suffices to have a small enough bias term  $s_0 \leq \frac{\sqrt{c_1(\alpha)\varepsilon}}{4}$  and variance coefficient  $s \leq \frac{1}{4}$  for the oracle, and run the SGD algorithm for  $k > \ln \left(\frac{\varepsilon}{\alpha}\right) / \ln \left(1 - \frac{c_1(\alpha)}{18\ell(\alpha)}\right)$  steps.
4.2 Observation model for the estimation problem
Herein, first we show that the estimation error and its differential can be characterized as a "simple norm" of the concatenated noise (Proposition 3). This norm is induced by a metric that encapsulates the system dynamics which is explained below. Before proceeding to the results of this section, we assume that both the process and measurement noise are bounded:
Assumption 4. Assume that (almost surely)  $\| x_0\|, \| \xi(t)\| \leq \kappa_\xi$  and  $\| \omega(t)\| \leq \kappa_\omega$  for all  $t$ . Also, for simplicity, suppose the initial state has zero mean, i.e.,  $m_0 = 0_n$ .
For two vectors  $\vec{v},\vec{w}\in \mathbb{R}^{(T + 1)n}$  , we define
$$
\langle \vec {v}, \vec {w} \rangle_ {\mathcal {A} _ {L}} := \operatorname {t r} \left[ \vec {v} \vec {w} ^ {\intercal} \mathcal {A} _ {L} ^ {\intercal} H ^ {\intercal} H \mathcal {A} _ {L} \right],
$$
where  $\mathcal{A}_L\coloneqq \left(A_L^0 A_L^1\dots A_L^T\right)$  . Also, define  $\mathcal{M}_L[E]\coloneqq (M_0[E]M_1[E]\dots M_T[E])$  with  $M_0[E] = 0$ $M_{1}[E] = EH$  and  $M_{i + 1}[E] = \sum_{k = 0}^{i}A_{L}^{i - k}EHA_{L}^{k}$  for  $i = 1,\dots ,T - 1$
Proposition 3. The estimation error  $\varepsilon(L, \mathcal{Y}_T)$  takes the following form
$$
\varepsilon (L, \mathcal {Y} _ {T}) = \| \vec {\eta} _ {L} \| _ {\mathcal {A} _ {L}} ^ {2},
$$
where  $\vec{\eta}_L\coloneqq \vec{\xi} -(I\otimes L)\vec{\omega}$  with  $\vec{\xi}^{\mathsf{T}} = (\xi (T - 1)^{\mathsf{T}}\ldots \xi (0)^{\mathsf{T}}x(0)^{\mathsf{T}})$  and  $\vec{\omega}^{\mathsf{T}} =$ $(\omega (T - 1)^{\intercal}\dots \omega (0)^{\intercal}0_{m}^{\intercal})$  . Furthermore, its differential acts on small enough  $E\in \mathbb{R}^{n\times m}$  as,
$$
\left. \mathrm {d} \varepsilon (\cdot , \mathcal {Y} _ {T}) \right| _ {L} (E) = - 2 \left\langle \vec {\eta} _ {L}, (I \otimes E) \vec {\omega} \right\rangle_ {\mathcal {A} _ {L}} + \operatorname {t r} \left[ \mathcal {X} _ {L} \mathcal {N} _ {L} [ E ] \right],
$$
where  $\mathcal{X}_L\coloneqq \vec{\eta}_L\vec{\eta}_L^\intercal$  and  $\mathcal{N}_L[E]\coloneqq \mathcal{M}_L[E]^{\intercal}H^{\intercal}H\mathcal{A}_L + \mathcal{A}_L^\intercal H^{\intercal}H\mathcal{M}_L[E]$
Now, we want to bound the error in the estimated gradient  $\nabla \hat{J}_T(L)$  by considering the concentration error (on length  $T$  trajectories) and truncation error separately as follows:
$$
\| \nabla \widehat {J} _ {T} (L) - \nabla J (L) \| \leq \| \nabla \widehat {J} _ {T} (L) - \nabla J _ {T} (L) \| + \| \nabla J _ {T} (L) - \nabla J (L) \|,
$$
recalling that  $J_{T}(L) = \mathbb{E}\left[\varepsilon (L,\mathcal{Y}_{T})\right]$  by definition.
Next, we aim to provide the analysis of concentration error on trajectories of length  $T$  with probability bounds. However, for any pair of real  $(T \times T)$ -matrices  $M$  and  $N$ , by Cauchy Schwartz inequality we obtain that  $|\operatorname{tr}[MN]| \leq \|M\|_F \|N\|_F \leq \sqrt{T} \|M\| \|N\|_F$ . This bound becomes loose (in terms of dimension  $T$ ) as the condition number of  $N$  increases².
Nonetheless, we are able to provide concentration error bounds that "scale well with respect to the length  $T$ " that hinges upon the following idea: from von Neumann Trace Inequality [49, Theorem 8.7.6] one obtains,
$$
\left| \operatorname {t r} [ M N ] \right| \leq \sum_ {i = 1} ^ {T} \sigma_ {i} (M) \sigma_ {i} (N) \leq \| M \| \| N \| _ {*}, \tag {12}
$$
where  $\| N\|_{*} \coloneqq \operatorname{tr}\left[\sqrt{N^{\intercal}N}\right] = \sum_{i}\sigma_{i}(N)$  is the nuclear norm with  $\sigma_{i}(N)$  denoting the  $i$ -th largest singular value of  $N$ . Additionally, the same inequality holds for non-square matrices of appropriate dimension which is tight in terms of dimension.
Proposition 4 (Concentration Error Bound). Consider  $M$  independent length  $T$  trajectories  $\{\mathcal{Y}_T^i\}_{i=1}^M$  and suppose Assumption 4 holds. Let  $\nabla \widehat{J}_T(L) \coloneqq \frac{1}{M} \sum_{i=1}^{M} \nabla \varepsilon(L, \mathcal{Y}_T^i)$ , then for any  $s > 0$ ,
$$
\mathbb {P} \left[ \| \nabla \widehat {J} _ {T} (L) - \nabla J _ {T} (L) \| \geq s \right] \leq 2 n \exp \left[ \frac {- M s ^ {2} / 2}{\nu_ {L} ^ {2} + 2 \nu_ {L} s / 3} \right],
$$
where  $\nu_{L} := 4\kappa_{L}^{2}C_{L}^{3}\|H\|^{2}\|H\|_{*}/[1-\sqrt{\rho(A_{L})}]^{3}$  with  $\kappa_{L} = \kappa_{\xi} + \|L\|\kappa_{\omega}$ .
We can also show how the truncation error decays linearly as  $T$  grows, with constants that are independent of the system dimension  $n$ :
Proposition 5 (Truncation Error Bound). Under Assumption 4, the truncation error is bounded:
$$
\| \nabla J (L) - \nabla J _ {T} (L) \| \leq \bar {\gamma} _ {L} \sqrt {\rho (A _ {L})} ^ {T + 1},
$$
where  $\bar{\gamma}_{L}:= 10\kappa_{L}^{4}C_{L}^{6}\| H\|^{2}\| H\|_{*} / [1 - \rho (A_{L})]^{2}$
Remark 8. Notice how the trajectory length  $T$  determines the bias in the estimated gradient. However, the concentration error bound is independent of  $T$  and only depends on the noise bounds proportionate to  $\kappa_{\xi}^{4}$ ,  $\kappa_{\omega}^{4}$  and the stability margin of  $A_{L}$  proportionate to  $C_L^6 / (1 - \sqrt{\rho(A_L)})^6$ .
Finally, by combining the truncation bound in Proposition 5 with concentration bounds in Proposition 4 we can provide probabilistic bounds on the "estimated cost"  $\widehat{J}_T(L)$  and the "estimated gradient"  $\nabla \widehat{J}_T(L)$ . Its precise statement is deferred to the supplementary materials (Theorem 5).
4.3 Sample complexity of SGD for Kalman Gain
Note that the open-loop system may be unstable. Often in learning literature, it is assumed that the closed-loop system can be contractible (i.e., the spectral norm  $\| A_L \| < 1$ ) which is quite convenient for analysis, however, it is not a reasonable system theoretic assumption. Herein, we emphasize that we only require the close-loop system to be Schur stable, meaning that  $\rho(A_L) < 1$ ; yet, it is very well possible that the system is not contractible. Handling systems that are merely stable requires more involved system theoretic tools that are established in the following lemma.
Lemma 6 (Uniform Bounds for Stable Systems). Suppose  $L \in S$ , then there exist constants  $C_L > 0$  such that
$$
\left\| A _ {L} ^ {k} \right\| \leq C _ {L} \sqrt {\rho \left(A _ {L}\right)} ^ {k + 1}, \quad \forall k \geq 0.
$$
Furthermore, consider  $S_{\alpha}$  for some  $\alpha > 0$ , then there exist constants  $D_{\alpha} > 0$ ,  $C_{\alpha} > 0$  and  $\rho_{\alpha} \in (0,1)$  such that  $\|L\| \leq D_{\alpha}$ ,  $C_{L} \leq C_{\alpha}$ , and  $\rho(A_{L}) \leq \rho_{\alpha}$  for all  $L \in S_{\alpha}$ .
2The reason is that the first equality is sharp whenever  $M$  is in the "direction" of  $N$ , while the second inequality is sharp whenever condition number of  $M$  is close to one.
The following result provides sample complexity bounds for this stochastic oracle to provide a biased estimation of the gradient that satisfies our oracle model of SGD analysis in Assumption 3.
Theorem 2. Under the premise of Proposition 4, consider  $S_{\alpha}$  for some  $\alpha > 0$  and choose any  $s, s_0 > 0$  and  $\tau \in (0,1)$ . Suppose the trajectory length  $T \geq \ln \left( \frac{\bar{\gamma}_{\alpha} \sqrt{\min(n,m)}}{s_0} \right) / \ln \left( \frac{1}{\sqrt{\rho_{\alpha}}} \right)$  and the batch size  $M \geq 4\nu_{\alpha}^{2} \min(n,m) \ln(2n/\delta) / (s s_0)^2$ , where  $\bar{\gamma}_{\alpha} := 10(\kappa_{\xi} + D_{\alpha} \kappa_{\omega})^4 C_{\alpha}^6 \|H\|^2 \|H\|_*$  and  $\nu_{\alpha} := 5C_{\alpha}^3 \|H\|^2 \|H\|_*(\kappa_{\xi} + D_{\alpha} \kappa_{\omega})^2 / [1 - \sqrt{\rho_{\alpha}}]^3$ . Then, with probability no less than  $1 - \delta$ , Assumption 3 holds.
Proof. First, note that for any  $L \in S_{\alpha}$  by Lemma 6 we have that  $\bar{\gamma}_{L} \leq \bar{\gamma}_{\alpha}$  and  $\nu_{L} \leq \nu_{\alpha}$ . Then, note that the lower bound on  $T$  implies that  $\bar{\gamma}_{\alpha} \sqrt{\min(n,m)} \sqrt{\rho_{\alpha}}^{T+1} \leq s_{0}$ . The claim then follows by applying Theorem 5 and noting that for any  $L \notin \mathcal{C}_{\tau}$  the gradient is lowerbound as  $\| \nabla J(L) \| > s_{0} / \tau$ .
Theorem 3. Consider the linear system (1) under Assumptions 2 and 4. Suppose the SGD algorithm is implemented with initial stabilizing gain  $L_0$ , the step-size  $\bar{\eta} \coloneqq \frac{2}{9\ell(J(L_0))}$ , for  $k$  iterations, a batch-size of  $M$ , and data-length  $T$ . Then,  $\forall \varepsilon > 0$  and with probability larger than  $1 - \delta$ ,  $J(L_k) - J(L^*) \leq \varepsilon$  if
$$
T \geq O \left(\ln \left(\frac {1}{\varepsilon}\right)\right), \quad M \geq O \left(\frac {1}{\varepsilon} \ln \left(\frac {1}{\delta}\right) \ln \left(\ln \left(\frac {1}{\varepsilon}\right)\right)\right), \quad a n d \quad k \geq O \left(\ln \left(\frac {1}{\varepsilon}\right)\right). \tag {13}
$$
Proof of Theorem 3. Recall that according to Remark 7, in order to obtain  $\varepsilon$  error on the cost value we require  $k \geq O(\ln(1/\varepsilon))$  of SGD algorithm using gradient estimates with small enough bias term  $s_0 \leq \frac{\sqrt{c_1(\alpha)\varepsilon}}{4}$  and variance coefficient  $s < \frac{1}{4}$ . Now, we can guarantee Assumption 3 holds for such  $s_0$  and  $s$  (with probability at least  $1 - \delta$ ) by invoking Theorem 2. In fact, it suffices to ensure that the length of data trajectories  $T$  and the batch size  $M$  are large enough; specifically,  $T \geq O(\ln(1/s_0)) = O(\ln(1/\varepsilon))$  and by using union bound for bounding the failure probability  $M \geq O(\ln(1/\delta)\ln(k)/s_0^2) = O(\ln(1/\delta)\ln(\ln(1/\varepsilon))/\varepsilon$ .
5 Conclusions, Broader Impact, and Limitations
In this work, we considered the problem of learning the optimal (steady-state) Kalman gain for linear systems with unknown process and measurement noise covariances. Our approach builds on the duality between optimal control and estimation, resulting in a direct stochastic Policy Optimization (PO) algorithm for learning the optimal filter gain. We also provided convergence guarantees and finite sample complexity with bias and variance error bounds that scale well with problem parameters. In particular, the variance is independent of the length of data trajectories and scales logarithmically with problem dimension, and bias term decreases exponentially with the length.
This work contributes a generic optimization algorithm and introduces a filtering strategy for estimating dynamical system states. While theoretical, it raises privacy concerns similar to the model-based Kalman filter. Limitations include the need for prior knowledge of system parameters; nonetheless, parameter uncertainties can be treated practically as process and measurement noise. Finally, sample complexities depend on the stability margin  $1 - \sqrt{\rho(A_L)}$ , inherent to the system generating the data.
Finally, a direction for future research is to study how to adapt the proposed algorithm and error analysis for the setting when a single long trajectory is available as opposed to several independent trajectories of finite length. Another interesting direction is to carry out a robustness analysis, similar to its LQR dual counterpart, to study the effect of the error in system parameters on the policy learning accuracy. Of course, the ultimate research goal is to use the recently introduced duality in nonlinear filtering [47] as a bridge to bring tools from learning to nonlinear filtering.
The research of A. Taghvaei has been supported by NSF grant EPCN-2318977; S. Talebi and M. Mesbahi have been supported by NSF grant ECCS-2149470 and AFOSR grant FA9550-20-1-0053. The authors are also grateful for constructive feedback from the reviewers.
[45] K. J. Åström, Introduction to Stochastic Control Theory. Courier Corporation, 2012.
[46] J.-W. Kim, P. G. Mehta, and S. P. Meyn, “What is the lagrangian for nonlinear filtering?” in 2019 IEEE 58th Conference on Decision and Control (CDC), pp. 1607–1614, IEEE, 2019.
[47] J. W. Kim, “Duality for nonlinear filtering,” arXiv preprint arXiv:2207.07709, 2022.
[48] J. Bu, A. Mesbahi, and M. Mesbahi, “On topological and metrical properties of stabilizing feedback gains: The MIMO case,” arXiv preprint arXiv:1904.02737, 2019.
[49] R. A. Horn and C. R. Johnson, Matrix Analysis. Cambridge University Press, 2012.
[50] H. Mohammadi, A. Zare, M. Soltanolkotabi, and M. R. Jovanovic, “Convergence and sample complexity of gradient methods for the model-free linear-quadratic regulator problem,” IEEE Transactions on Automatic Control, vol. 67, no. 5, pp. 2435–2450, 2021.
[51] A. Beck, First-Order Methods in Optimization. Philadelphia, PA: Society for Industrial and Applied Mathematics, 2017.
[52] A. Cohen, A. Hasidim, T. Koren, N. Lazic, Y. Mansour, and K. Talwar, "Online linear quadratic control," in Proceedings of the 35th International Conference on Machine Learning (J. Dy and A. Krause, eds.), vol. 80 of Proceedings of Machine Learning Research, pp. 1029-1038, PMLR, 10-15 Jul 2018.
[53] N. J. Higham, Functions of Matrices: Theory and Computation. SIAM, 2008.
[54] J. A. Tropp, “An introduction to matrix concentration inequalities,” Foundations and Trends® in Machine Learning, vol. 8, no. 1-2, pp. 1-230, 2015.
[55] S. Talebi, A. Taghvaei, and M. Mesbahi, "SGD for Filtering," Oct. 2023. Available on GitHub at https://github.com/shahriarta/sgd4filtering.
1 Introduction 1
2 Background and Problem Formulation 2
2.1 Learning problem 3
2.2 Stochastic optimization formulation 3
3 Estimation-Control Duality Relationship 4
3.1 Duality in steady-state regime 5
3.2 Optimization landscape 5
4 SGD for Learning the Kalman Gain 6
4.1 SGD with biased gradient and stability constraint 7
4.2 Observation model for the estimation problem 8
4.3 Sample complexity of SGD for Kalman Gain 9
5 Conclusions, Broader Impact, and Limitations 10
A Discussion of additional related works 15
B Summary of Assumptions 15
C Proof of the duality relationship: Proposition 1 16
D Proofs for the results for the analysis of the optimization landscape 17
D.1 Preliminary lemma 17
D.2 Proof of Lemma 1 17
D.3 Derivation of the gradient formula in Lemma 2 18
D.4 Proof of existence of global minimizer in Lemma 2 19
D.5 Proof of gradient dominance property in Lemma 2. 19
D.6 Proof of Lipschitz property in Lemma 2 21
D.7 Gradient Flow (GF) 22
D.8 Gradient Descent (GD) 23
E Proofs for the analysis of the constrained SGD algorithm 24