SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \mathbb {P} \left[ \| \frac {1}{M} \sum_ {i = 1} ^ {M} S _ {L} \left(E, \mathcal {Y} _ {T} ^ {i}\right) - \mathbb {E} \left[ S _ {L} \left(E, \mathcal {Y} _ {T} ^ {i}\right) \right] \| \geq t \right] \\ \leq 2 n \exp \left[ \frac {- M \| H ^ {\intercal} H \| _ {*} ^ {2} t ^ {2} / 2}{\nu_ {L} ^ {2} \| E \| ^ {2} + 2 \nu_ {L} \| H ^ {\intercal} H \| _ {*} \| E \| t / 3} \right] \\ \end{array}
$$
Thus, by substituting  $t$  with  $t\| E\| /\| H^{\top}H\|_{*}$  and applying this bound to (35) we obtain that
$$
\mathbb {P} \left[ | \langle \nabla \widehat {J} _ {T} (L) - \nabla J _ {T} (L), E \rangle | \leq t \| E \| \right] \geq 1 - 2 n \exp \left[ \frac {- M t ^ {2} / 2}{\nu_ {L} ^ {2} + 2 \nu_ {L} t / 3} \right]
$$
Finally, choosing  $E = \nabla \widehat{J}_T(L) - \nabla J_T(L)$  proves the second claim.
F.4 Proof of the Proposition 5: Truncation error bound
We provide the proof for a detailed version of Proposition 5:
Proposition 5' (Truncation Error Bound). Suppose  $m_0 = 0$ , then under Assumption 4 we have
$$
\left| J (L) - J _ {T} (L) \right| \leq \bar {\xi} _ {L} \frac {\rho \left(A _ {L}\right) ^ {T + 1}}{1 - \rho \left(A _ {L}\right)},
$$
$$
\left\| \nabla J (L) - \nabla J _ {T} (L) \right\| \leq \bar {\gamma} _ {L} \frac {\sqrt {\rho (A _ {L})} ^ {T + 1}}{\left[ 1 - \rho (A _ {L}) \right] ^ {2}}
$$
$$
\begin{array}{l} \bar {\xi} _ {L} := \left[ \kappa_ {\xi} ^ {2} + \left(\kappa_ {\xi} ^ {2} + \kappa_ {\omega} ^ {2} \| L \| ^ {2}\right) C _ {L} ^ {2} \right] \| H ^ {\intercal} H \| _ {*} C _ {L} ^ {2}, \\ \bar {\gamma} _ {L} := 2 \left[ \kappa_ {\xi} ^ {2} + C _ {L} ^ {2} \left(\kappa_ {\xi} ^ {2} + \kappa_ {\omega} ^ {2} \| L \| ^ {2}\right) \right] C _ {L} ^ {2} \| H \| \| H ^ {\intercal} H \| _ {*} \\ + 2 \kappa_ {\omega} ^ {2} (\kappa_ {\xi} ^ {2} + \kappa_ {\omega} ^ {2} \| L \| ^ {2}) \| L \| \| H \| \| H ^ {\intercal} H \| _ {*} \left(C _ {L} + 2 C _ {L} ^ {3} \rho (A _ {L}) ^ {3 / 2}\right) C _ {L} ^ {3} \sqrt {\rho (A _ {L})} ^ {T + 1}. \\ \end{array}
$$
Proof of Proposition 5. For the purpose of this proof, we denote the same matrices by  $\mathcal{A}_{L,T}$  and  $\mathcal{M}_{L,T}[E]$  in order to emphasize on length  $T$ . Recall that
$$
J _ {T} (L) = \mathbb {E} [ \varepsilon (L, \mathcal {Y}) ] = \operatorname {t r} \left[ \mathbb {E} [ \mathcal {X} _ {L} ] \mathcal {A} _ {L, T} ^ {\intercal} H ^ {\intercal} H \mathcal {A} _ {L, T} \right],
$$
where  $\mathbb{E}\left[\mathcal{X}_L\right] = \mathcal{Q} + (I\otimes L)\mathcal{R}(I\otimes L^{\intercal})$  , which implies
$$
J _ {T} (L) = \operatorname {t r} \left[ [ I \otimes (Q + L R L ^ {\intercal}) ] \mathcal {A} _ {L, T - 1} ^ {\intercal} H ^ {\intercal} H \mathcal {A} _ {L, T - 1} \right] + \operatorname {t r} \left[ P _ {0} (A _ {L} ^ {\intercal}) ^ {T} H ^ {\intercal} H A _ {L} ^ {T} \right],
$$
$$
J (L) = \lim  _ {t \rightarrow \infty} \operatorname {t r} \left[ [ I \otimes (Q + L R L ^ {\intercal}) ] \mathcal {A} _ {L, t} ^ {\intercal} H ^ {\intercal} H \mathcal {A} _ {L, t} \right],
$$
$$
\begin{array}{l} J (L) - J _ {T} (L) = - \operatorname {t r} \left[ P _ {0} \left(A _ {L} ^ {\intercal}\right) ^ {T} H ^ {\intercal} H A _ {L} ^ {T} \right] \\ + \lim  _ {t \rightarrow \infty} \operatorname {t r} \left[ [ I \otimes (Q + L R L ^ {\intercal}) ] [ I \otimes (A _ {L} ^ {\intercal}) ^ {T} ] \mathcal {A} _ {L, t} ^ {\intercal} H ^ {\intercal} H \mathcal {A} _ {L, t} [ I \otimes A _ {L} ^ {T} ] \right] \\ = - \operatorname {t r} \left[ A _ {L} ^ {T} P _ {0} \left(A _ {L} ^ {\mathbf {T}}\right) ^ {T} H ^ {\mathbf {T}} H \right] \\ + \lim  _ {t \rightarrow \infty} \operatorname {t r} \left[ [ I \otimes A _ {L} ^ {T} (Q + L R L ^ {\intercal}) (A _ {L} ^ {\intercal}) ^ {T} ] \mathcal {A} _ {L, t} ^ {\intercal} H ^ {\intercal} H \mathcal {A} _ {L, t} \right]. \\ \end{array}
$$
Therefore, by the properties of trace and Lemma 6 we obtain that
$$
\begin{array}{l} | J (L) - J _ {T} (L) | \leq \| P _ {0} \| \| A _ {L} ^ {T} \| ^ {2} \operatorname {t r} [ H ^ {\intercal} H ] \\ + \| Q + L R L ^ {\intercal} \| \| (A _ {L}) ^ {T} \| ^ {2} \lim _ {t \to \infty} \operatorname {t r} \left[ \mathcal {A} _ {L, t} ^ {\intercal} H ^ {\intercal} H \mathcal {A} _ {L, t} \right] \\ \leq \| P _ {0} \| C _ {L} ^ {2} \rho (A _ {L}) ^ {T + 1} \| H ^ {\intercal} H \| _ {*} \\ + \left(\| Q \| + \| R \| \| L \| ^ {2}\right) C _ {L} ^ {2} \rho \left(A _ {L}\right) ^ {T + 1} \frac {C _ {L} ^ {2} \| H ^ {\intercal} H \| _ {*}}{1 - \rho \left(A _ {L}\right)}, \\ \end{array}
$$
where the last line follows by Lemma 8. So,
$$
| J (L) - J _ {T} (L) | \leq \left[ \| P _ {0} \| + \frac {(\| Q \| + \| R \| \| L \| ^ {2}) C _ {L} ^ {2}}{1 - \rho (A _ {L})} \right] \| H ^ {\intercal} H \| _ {*} C _ {L} ^ {2} \rho (A _ {L}) ^ {T + 1}
$$
This, together with Assumption 4 imply the first claim.
Next, for simplicity we adopt the notation  $\mathcal{A}_{L,\infty}$  to interpret the limit as  $t\to \infty$  , then similar to the proof of Proposition 4 we can compute that
$$
\begin{array}{l} \langle \nabla J (L) - \nabla J _ {T} (L), E \rangle = - 2 \mathrm {t r} \left[ M _ {T} [ E ] P _ {0} (A _ {L} ^ {\intercal}) ^ {T} H ^ {\intercal} H \right] \\ + \operatorname {t r} \left[ \left[ I \otimes A _ {L} ^ {T} (Q + L R L ^ {\intercal}) \left(A _ {L} ^ {\intercal}\right) ^ {T} \right] \mathcal {N} _ {L, \infty} [ E ] \right] \\ + 2 \operatorname {t r} \left[ [ I \otimes M _ {T} [ E ] (Q + L R L ^ {\intercal}) (A _ {L} ^ {\intercal}) ^ {T} ] \mathcal {A} _ {L, \infty} ^ {\intercal} H ^ {\intercal} H \mathcal {A} _ {L, \infty} \right] \\ + 2 \operatorname {t r} \left[ \left[ I \otimes A _ {L} ^ {T} E R L ^ {\intercal} \left(A _ {L} ^ {\intercal}\right) ^ {T} \right] \mathcal {A} _ {L, \infty} ^ {\intercal} H ^ {\intercal} H \mathcal {A} _ {L, \infty} \right]. \\ \end{array}
$$
Therefore, using (12) and Lemma 6 we have the following bound
$$
\begin{array}{l} \left| \left\langle \nabla J (L) - \nabla J _ {T} (L), E \right\rangle \right| \leq 2 \| E H \| C _ {L} ^ {2} (T + 1) \rho \left(A _ {L}\right) ^ {T + 1} \| P _ {0} \| \| H ^ {\intercal} H \| _ {*} \\ + \| Q + L R L ^ {\intercal} \| C _ {L} ^ {2} \rho (A _ {L}) ^ {T + 1} \| \mathcal {N} _ {L, \infty} [ E ] \| _ {*} \\ + 2 \| Q + L R L ^ {\intercal} \| \| E H \| C _ {L} ^ {2} (T + 1) \rho \left(A _ {L}\right) ^ {T + 1} \| \mathcal {A} _ {L, \infty} ^ {\intercal} H ^ {\intercal} H \mathcal {A} _ {L, \infty} \| _ {*} \\ + 2 \| E R L ^ {\intercal} \| C _ {L} ^ {2} \rho (A _ {L}) ^ {T + 1} \| A _ {L, \infty} ^ {\intercal} H ^ {\intercal} H A _ {L, \infty} \| _ {*} \\ \end{array}
$$
which by Lemma 8 is bounded as follows
$$
\begin{array}{l} \left| \left\langle \nabla J (L) - \nabla J _ {T} (L), E / \| E \| \right\rangle \right| \leq 2 \| P _ {0} \| \| H \| \| H ^ {\intercal} H \| _ {*} C _ {L} ^ {2} (T + 1) \rho \left(A _ {L}\right) ^ {T + 1} \\ + \| Q + L R L ^ {\intercal} \| \| H \| \| H ^ {\intercal} H \| _ {*} \frac {\left[ 2 C _ {L} ^ {3} + 4 C _ {L} ^ {5} \rho (A _ {L}) ^ {3 / 2} \right] \rho (A _ {L}) ^ {T + 1}}{\left[ 1 - \rho (A _ {L}) \right] ^ {2}} \\ + 2 \| Q + L R L ^ {\intercal} \| \| H \| \| H ^ {\intercal} H \| _ {*} \frac {C _ {L} ^ {4} (T + 1) \rho \left(A _ {L}\right) ^ {T + 1}}{1 - \rho \left(A _ {L}\right)} \\ + 2 \| R \| \| L \| \| H ^ {\intercal} H \| _ {*} \frac {C _ {L} ^ {4} \rho (A _ {L}) ^ {T + 1}}{1 - \rho (A _ {L})}. \\ \end{array}
$$
Finally, choosing  $E = \nabla J(L) - \nabla J_T(L)$  together with Assumption 4 implies
$$
\begin{array}{l} \| \nabla J (L) - \nabla J _ {T} (L) \| \leq 2 \left[ \frac {\kappa_ {\xi} ^ {2} + C _ {L} ^ {2} \left(\kappa_ {\xi} ^ {2} + \kappa_ {\omega} ^ {2} \| L \| ^ {2}\right)}{1 - \rho \left(A _ {L}\right)} \right] C _ {L} ^ {2} \| H \| \| H ^ {\intercal} H \| _ {*} (T + 1) \rho \left(A _ {L}\right) ^ {T + 1} \\ + 2 \left[ \frac {\kappa_ {\omega} ^ {2} \left(\kappa_ {\xi} ^ {2} + \kappa_ {\omega} ^ {2} \| L \| ^ {2}\right) \| L \| \| H \| \left(C _ {L} + 2 C _ {L} ^ {3} \rho \left(A _ {L}\right) ^ {3 / 2}\right)}{1 - \rho \left(A _ {L}\right)} \right] \| H ^ {\intercal} H \| _ {*} \frac {C _ {L} ^ {3} \rho \left(A _ {L}\right) ^ {T + 1}}{1 - \rho \left(A _ {L}\right)}. \\ \end{array}
$$
Finally, the second claim follows by the following simple facts:
$$
(T + 1) \rho (A _ {L}) ^ {T + 1} \leq \frac {\sqrt {\rho (A _ {L})} ^ {T + 1}}{1 - \rho (A _ {L})}, \quad \forall T > 0,
$$
as  $\max_{t\geq 0}t\rho^t = \frac{2}{e\ln 1 / \rho}\leq \frac{1}{\ln 1 / \rho}\leq \frac{1}{1 - \rho}$  for any  $\rho \in (0,1)$ . This completes the proof.
F.5 Complete version of Theorem 2: Sample complexity bounds for the stochastic oracle
The following is a detailed version of Theorem 2:
Theorem 2'. Suppose  $m_0 = 0_n$  and Assumption 4 holds for a data-set  $\{\mathcal{Y}_T^i\}_{i=1}^M$ . Define  $\nabla \widehat{J}_T(L) := \frac{1}{M} \sum_{i=1}^{M} \nabla \varepsilon(L, \mathcal{Y}_T^i)$ , where  $\nabla_L \varepsilon(L, \mathcal{Y})$  is obtained in Lemma 3. Consider  $S_\alpha$  for some  $\alpha > 0$  and any  $s, s_0 > 0$  and  $\tau \in (0,1)$ . Suppose the trajectory length
$$
T \geq \ln \left(\frac {\bar {\gamma} _ {\alpha} \sqrt {\operatorname* {m i n} (n , m)}}{s _ {0}}\right) / \ln \left(\frac {1}{\sqrt {\rho_ {\alpha}}}\right)
$$
and the batch size
$$
M \geq \left[ 2 \left(\frac {\nu_ {\alpha} \sqrt {\min (n , m)}}{s s _ {0} / \tau}\right) ^ {2} + \frac {4}{3} \left(\frac {\nu_ {\alpha} \sqrt {\min (n , m)}}{s s _ {0} / \tau}\right) \right] \ln (2 n / \delta),
$$
$$
\begin{array}{l} \bar {\gamma} _ {\alpha} := 2 \left[ \kappa_ {\xi} ^ {2} + C _ {\alpha} ^ {2} \left(\kappa_ {\xi} ^ {2} + \kappa_ {\omega} ^ {2} D _ {\alpha} ^ {2}\right) \right] C _ {\alpha} ^ {2} \| H \| \| H ^ {\intercal} H \| _ {*} \\ + 2 \kappa_ {\omega} ^ {2} (\kappa_ {\xi} ^ {2} + \kappa_ {\omega} ^ {2} D _ {\alpha} ^ {2}) D _ {\alpha} \| H \| \| H ^ {\intercal} H \| _ {*} \left(C _ {\alpha} + 2 C _ {\alpha} ^ {3} \rho_ {\alpha} ^ {3 / 2}\right) C _ {\alpha} ^ {3} \sqrt {\rho_ {\alpha}} ^ {T + 1}, \\ \end{array}
$$
$$
\nu_ {\alpha} := \frac {2 (\kappa_ {\xi} + D _ {\alpha} \kappa_ {\omega}) \kappa_ {\omega} C _ {\alpha} ^ {2} + [ C _ {\alpha} + 2 C _ {\alpha} ^ {3} \rho_ {\alpha} ^ {3 / 2} ] \| H \| (\kappa_ {\xi} + D _ {\alpha} \kappa_ {\omega}) ^ {2}}{[ 1 - \sqrt {\rho_ {\alpha}} ] ^ {3} / \| H ^ {\intercal} H \| _ {*}},
$$
with  $\rho_{\alpha}$ ,  $C_{\alpha}$  and  $D_{\alpha}$  defined in Lemma 6. Then, with probability no less than  $1 - \delta$ , Assumption 3 holds.
F.6 Additional concentration bound results
Combining the truncation bound in Proposition 5 with concentration bounds in Proposition 4 we can provide probabilistic bounds on the "estimated cost"  $\widehat{J}_T(L)$  and the "estimated gradient"  $\nabla \widehat{J}_T(L)$ . The result involves the bound for the Frobenius norm of the error with probabilities independent of  $T$ .
Theorem 2, can be viewed as a simplified application of Theorem 5 to characterize the required minimum trajectory length and minimum batch so that the approximate gradient satisfies Assumption 3, with a specific  $s$  and  $s_0$ .
Theorem 5. Suppose Assumption 4 holds. For any  $s > 0$  and  $L \in S_{\alpha}$ , if
$$
M \geq \left[ 2 \left[ \frac {\nu_ {L} \sqrt {\min (n , m)}}{s \| \nabla J (L) \| _ {F}} \right] ^ {2} + \frac {4}{3} \left[ \frac {\nu_ {L} \sqrt {\min (n , m)}}{s \| \nabla J (L) \| _ {F}} \right] \right] \ln (2 n / \delta),
$$
then with probability no less than  $1 - \delta$
$$
\| \nabla \widehat {J} _ {T} (L) - \nabla J (L) \| _ {F} \leq s \| \nabla J (L) \| _ {F} + \bar {\gamma} _ {L} \sqrt {\min  (n , m)} \sqrt {\rho (A _ {L})} ^ {T + 1},
$$
with  $\nu_{L}$  and  $\bar{\gamma}_{L}$  defined in Proposition 4 and Proposition 5, respectively.
Proof of Theorem 5. Recall that for any  $L \in S_{\alpha}$  for some  $\alpha > 0$  we have
$$
\| \nabla \widehat {J} _ {T} (L) - \nabla J (L) \| \leq \| \nabla \widehat {J} _ {T} (L) - \nabla J _ {T} (L) \| + \| \nabla J _ {T} (L) - \nabla J (L) \|.
$$
Thus, by Proposition 4 with  $s$  replaced by  $s\| \nabla J(L)\| _F / \sqrt{\min(n,m)}$  and applying Proposition 5 to the second term, we obtain that with probability at least  $1 - \delta$ :
$$
\| \nabla \widehat {J} _ {T} (L) - \nabla J (L) \| \leq \frac {s \| \nabla J (L) \| _ {F}}{\sqrt {\min (n , m)}} + \bar {\gamma} _ {L} \frac {\sqrt {\rho (A _ {L})} ^ {T + 1}}{[ 1 - \rho (A _ {L}) ] ^ {2}},
$$
$$
\delta \geq 2 n \exp \left[ \frac {- M s ^ {2} / 2}{\left[ \frac {\nu_ {L} \sqrt {\min  (n , m)}}{\| \nabla J (L) \| _ {F}} \right] ^ {2} + 2 \left[ \frac {\nu_ {L} \sqrt {\min  (n , m)}}{\| \nabla J (L) \| _ {F}} \right] s / 3} \right].
$$
Noticing  $\| \nabla \widehat{J}_T(L) - \nabla J(L)\| _F\leq \sqrt{\min(n,m)}\| \nabla \widehat{J}_T(L) - \nabla J(L)\|$  and rearranging terms will complete the proof.
One can also provide the analogous concentration error bounds where the probabilities are independent of the system dimension  $n$ .
Proposition 7 (Concentration independent of system dimension  $n$ ). Under the same hypothesis, we have
$$
\mathbb {P} \left[ | \widehat {J} _ {T} (L) - J _ {T} (L) | \leq s \right] \geq 1 - 2 T \exp \left[ \frac {- M s ^ {2} / 2}{\bar {\mu} _ {L} ^ {2} T ^ {2} + 2 \bar {\mu} _ {L} T s / 3} \right],
$$
$$
\begin{array}{l} \mathbb {P} \left[ \| \nabla \widehat {J} _ {T} (L) - \nabla J _ {T} (L) \| \leq s \right] \geq 1 - 2 T \exp \left[ \frac {- M s ^ {2} / 2}{\bar {\nu} _ {L} ^ {2} T ^ {2} + 2 \bar {\nu} _ {L} T s / 3} \right] \\ - 2 T \exp \left[ \frac {- M s ^ {2} / 2}{\kappa_ {\omega} ^ {2} \bar {\mu} _ {L} ^ {2} T ^ {2} + 2 \kappa_ {\omega} \bar {\mu} _ {L} T s / 3} \right] \\ \end{array}
$$
$$
\begin{array}{l} \bar {\mu} _ {L} := \frac {C _ {L} ^ {2} \| H ^ {\intercal} H \| _ {*}}{1 - \rho (A _ {L})} \kappa_ {L} ^ {2} \\ \bar {\nu} _ {L} := \frac {\left[ 2 C _ {L} + 4 C _ {L} ^ {3} \rho (A _ {L}) ^ {3 / 2} \right] \| H \| \| H ^ {\intercal} H \| _ {*} ^ {*}}{\left[ 1 - \rho (A _ {L}) \right] ^ {2}} \kappa_ {L} ^ {2}. \\ \end{array}
$$
Proof of Proposition 7. Similar to the previous proof, we have
$$
\widehat {J} _ {T} (L) - J _ {T} (L) = \mathrm {t r} \left[ (\mathcal {Z} _ {L} - \mathbb {E} [ \mathcal {Z} _ {L} ]) \mathcal {A} _ {L} ^ {\intercal} H ^ {\intercal} H \mathcal {A} _ {L} \right],
$$
and thus, by (12) we obtain
$$
\left| \widehat {J} _ {T} (L) - J _ {T} (L) \right| \leq \| \left(\mathcal {Z} _ {L} - \mathbb {E} \left[ \mathcal {Z} _ {L} \right]\right) \| \| \mathcal {A} _ {L} ^ {\intercal} H ^ {\intercal} H \mathcal {A} _ {L} \| _ {*}. \tag {36}
$$
Next, we consider the symmetric random matrix  $(\mathcal{Z}_L - \mathbb{E}[\mathcal{Z}_L])$  and recall that  $\| \xi (t) - L\omega (t)\| \leq \kappa_L$  almost surely; thus
$$
\| \mathcal {X} _ {L} \| = \| \vec {\eta} _ {L} \| ^ {2} \leq \kappa_ {L} ^ {2} T ^ {2}.
$$
$$
\left\| \mathbb {E} \left[ \mathcal {X} _ {L} ^ {2} \right] \right\| \leq \mathbb {E} \left[ \| \mathcal {X} _ {L} \| ^ {2} \right] \leq \kappa_ {L} ^ {4} T ^ {4}.
$$
$$
\mathbb {P} \left[ \| \left(\mathcal {Z} _ {L} - \mathbb {E} \left[ \mathcal {Z} _ {L} \right]\right) \| \geq t \right] \leq 2 T \exp \left[ \frac {- M t ^ {2} / 2}{\kappa_ {L} ^ {4} T ^ {4} + 2 \kappa_ {L} ^ {2} T ^ {2} t / 3} \right] \tag {37}
$$
Substituting  $t$  with  $t / \| \mathcal{A}_L^\top H^\top H\mathcal{A}_L\|_*$  together with (36) implies the first claim because by Lemma 8  $\kappa_L^2 T^2\| \mathcal{A}_L^\top H^\top H\mathcal{A}_L\|_*\leq \bar{\mu}_L T.$
Again, similar to (35) in the previous proof, by (12) we obtain that
$$
\begin{array}{l} | \langle \nabla \widehat {J} _ {T} (L) - \nabla J _ {T} (L), E \rangle | \leq \| \frac {1}{M} \sum_ {i = 1} ^ {M} S _ {L} (E, \mathcal {Y} _ {T} ^ {i}) - \mathbb {E} \left[ S _ {L} (E, \mathcal {Y} _ {T} ^ {i}) \right] \| \| \mathcal {A} _ {L} ^ {\intercal} H ^ {\intercal} H \mathcal {A} \| _ {*} \\ + \| \frac {1}{M} \sum_ {i = 1} ^ {M} \mathcal {X} _ {L} \left(\mathcal {Y} ^ {i}\right) - \mathbb {E} \left[ \mathcal {X} _ {L} \left(\mathcal {Y} ^ {i}\right) \right] \| \left\| \mathcal {N} _ {L} [ E ] \right\| _ {*} \tag {38} \\ \end{array}
$$
$$
- 2 \vec {\xi} \vec {\omega} ^ {\intercal} (I \otimes E ^ {\intercal}) + 2 (I \otimes L) \vec {\omega} \vec {\omega} ^ {\intercal} (I \otimes E ^ {\intercal}).
$$
So, we claim that almost surely
$$
\begin{array}{l} \| S _ {L} (E, \mathcal {Y}) \| \leq 2 \| (I \otimes E) \vec {\omega} \| (\| \vec {\xi} \| + \| (I \otimes L) \vec {\omega} \|) \\ \leq 2 \| E \| \kappa_ {\omega} T (\kappa_ {\xi} T + \| L \| \kappa_ {\omega} T) \\ = \kappa_ {L} \kappa_ {\omega} T ^ {2} \| E \|, \\ \end{array}
$$
$$
\left\| \mathbb {E} \left[ S _ {L} (E, \mathcal {Y}) ^ {2} \right] \right\| \leq \mathbb {E} \left[ \| S _ {L} (E, \mathcal {Y}) \| ^ {2} \right] \leq \kappa_ {L} ^ {2} \kappa_ {\omega} ^ {2} T ^ {4} \| E \| ^ {2}.
$$
$$
\mathbb {P} \left[ \| \frac {1}{M} \sum_ {i = 1} ^ {M} S _ {L} (E, \mathcal {Y} _ {T} ^ {i}) - \mathbb {E} \left[ S _ {L} (E, \mathcal {Y} _ {T} ^ {i}) \right] \| \geq t \right] \leq 2 T \exp \left[ \frac {- M t ^ {2} / 2}{\kappa_ {\omega} ^ {2} \kappa_ {L} ^ {2} T ^ {4} \| E \| ^ {2} + 2 \kappa_ {L} \kappa_ {\omega} T ^ {2} \| E \| t / 3} \right]
$$
Substituting  $t$  with  $t / \| \mathcal{A}_L^\top H^\top H\mathcal{A}_L\|_*$  implies that
$$
\begin{array}{l} \mathbb {P} \left[ \| \frac {1}{M} \sum_ {i = 1} ^ {M} S _ {L} (E, \mathcal {Y} _ {T} ^ {i}) - \mathbb {E} \left[ S _ {L} (E, \mathcal {Y} _ {T} ^ {i}) \right] \| \left\| \mathcal {A} _ {L} ^ {\intercal} H ^ {\intercal} H \mathcal {A} _ {L} \right\| _ {*} \geq t \right] \\ \leq 2 T \exp \left[ \frac {- M t ^ {2} / 2}{\kappa_ {\omega} ^ {2} \bar {\mu} _ {L} ^ {2} T ^ {2} \| E \| ^ {2} + 2 \kappa_ {\omega} \bar {\mu} _ {L} T \| E \| t / 3} \right] \\ \end{array}
$$
because by Lemma 8 we have  $\kappa_L^2 T^2\| \mathcal{A}_L^\intercal H^\intercal H\mathcal{A}_L\|_*\leq \bar{\mu}_L T$
Next, by substituting  $t$  with  $t / \|\mathcal{N}_L[E]\|_*$  in (37) we have
$$
\mathbb {P} \left[ \| \mathcal {Z} _ {L} - \mathbb {E} \left[ \mathcal {Z} _ {L} \right] \| \| \mathcal {N} _ {L} [ E ] \| _ {*} \geq t \right] \leq 2 T \exp \left[ \frac {- M t ^ {2} / 2}{\bar {\nu} _ {L} ^ {2} T ^ {2} \| E \| ^ {2} + 2 \bar {\nu} _ {L} T \| E \| t / 3} \right]
$$
because Lemma 8 implies that  $\kappa_L^2 T^2\| \mathcal{N}_L[E]\|_* \leq \bar{\nu}_L T\| E\|$ . Thus, by combining the last two inequalities and using the union bound for (38) we obtain that
$$
\begin{array}{l} \mathbb {P} \left[ | \langle \nabla \widehat {J} _ {T} (L) - \nabla J _ {T} (L), E \rangle | \geq t \right] \leq 2 T \exp \left[ \frac {- M t ^ {2} / 2}{\bar {\nu} _ {L} ^ {2} T ^ {2} \| E \| ^ {2} + 2 \bar {\nu} _ {L} T \| E \| t / 3} \right] \\ + 2 T \exp \left[ \frac {- M t ^ {2} / 2}{\kappa_ {\omega} ^ {2} \bar {\mu} _ {L} ^ {2} T ^ {2} \| E \| ^ {2} + 2 \kappa_ {\omega} \bar {\mu} _ {L} T \| E \| t / 3} \right] \\ \end{array}
$$
Finally, substituting  $t$  with  $t\| E\|$  and choosing  $E = \nabla \widehat{J}_T(L) - \nabla J_T(L)$  proves the second claim.
Remark 10. We obtained a better bound for truncation of the gradient as
$$
\| \nabla J (L) - \nabla J _ {T} (L) \| \leq \bar {\gamma} _ {1} (L) \frac {\sqrt {\rho (A _ {L})} ^ {T + 1}}{\left[ 1 - \rho (A _ {L}) \right] \ln \left(1 / \rho (A _ {L})\right)} + \bar {\gamma} _ {2} (L) \frac {\rho (A _ {L}) ^ {T + 1}}{\left[ 1 - \rho (A _ {L}) \right] ^ {2}},
$$
which has been simplified for clarity of the presentation.
Appendix G Numerical Results
Herein, we showcase the application of the developed theory for improving the estimation policy for an LTI system. Specifically, we consider an undamped mass-spring system with known parameters  $(A,H)$  with  $n = 2$  and  $m = 1$ . In the hindsight, we consider a variance of 0.1 for each state dynamic noise, a state covariance of 0.05 and a variance of 0.1 for the observation noise. Assuming a trajectory of length  $T$  at every iteration, the approximate gradient is obtained as in Lemma 3, only requiring an output data sequence collected from the system in (1). Then, the progress of policy updates using the SGD algorithm for different values of trajectory length  $T$  and batch size  $M$  are depicted in Figure 1 where each figure shows average progress over 50 rounds of simulation. The figure demonstrates a linear convergence outside of a neighborhood of global optimum that depends on the bias term in the approximate gradient (due to truncated data trajectories). The rate then drops when the policy iterates enter into this neighborhood which is expected as every update only relies on a biased gradient—in contrast to the linear convergence established for deterministic GD (to the exact optimum) using the true gradient.
Specifically, recall that our convergence guarantee to a small neighborhood around the optimal value is due to the finite-length of the data trajectories. The region can be made arbitrary small by choosing larger trajectory length. In particular, to achieve  $\varepsilon$  error, we only require the length  $T \geq O(\ln(1 / \varepsilon))$ —see Theorem 3. Also, Fig 1(d) is illustrating that optimality gap at the final iteration (i.e. the radius of the small neighborhood around optimality) which is decaying linearly as a function of trajectory length  $T \leq 50$ —until the variance error dominates beyond  $T = 50$ . It is clear that, increasing the batch size  $M$  will allow further decrease of this optimality gap beyond  $T = 50$ .
The code for regenerating these results is available online at this GitHub repository [55].
Appendix H Nomenclature
Constant Values
$\alpha$  Scalar value of the cost
$\bar{\gamma}_{\alpha}$  Uniform constant for gradient estimate—see Theorem 2
$\bar{\gamma}_{L}$  Constant for bounding bias of gradient estimate—see Proposition 5
$\delta$  Failure probability
$\eta$  Step-size of SGD algorithm
$\gamma$  Positive constant smaller than one—see Proposition 2
$\kappa_{\omega}$  Scalar upperbound on 2-norm of measurement noise
$\kappa_{\xi}$  Scalar upperbound on 2-norm of dynamic noise
(d)
Figure 1: Simulation result of the SGD algorithm to learn the steady-state Kalman gain for the mass-spring example. (a) The optimality gap as a function of iterations  $k$  for different value of batch-size  $M$  averaged over 50 simulations; (b) The optimality gap at final iteration as a function of batch-size  $M$  for all 50 simulations; (c) The optimality gap as a function of iterations  $k$  for different value trajectory length  $T$ . The figure depicts the linear decay of the optimality gap, with respect to the iteration  $k$ , before the iterate enters the small neighborhood of optimality where the direction of the oracle gradient is not informative anymore. The neighborhood shrinks as  $T$  increases; (d) The optimality gap at final iteration as a function of trajectory length  $T$ . The gap is decaying linearly as a function of trajectory length  $T$  until the variance error dominates beyond  $T = 50$ .
$\kappa_{L}$  Combined Scalar upperbound on 2-norm of noise
$\nu_{\alpha}$  Uniform constant for gradient estimate—see Theorem 2
$\nu_{L}$  Combined matrix variance statistics of gradient estimate—see Proposition 4
$k$  Iteration step of SGD algorithm
$M$  Batch-size of SGD algorithm
$m$  Dimension of observation vector
$n$  Dimension of state vector
$s$  Variance coefficient term of oracle model—see Assumption 3
$s_0$  Bias term of oracle model—see Assumption 3
$T$  Length of output data trajectory
Time step of the system
$\| \cdot \|$  2-norm of a matrix
$\| \cdot \| _F$  Frobenius-norm of a matrix
$\mathcal{L}$  Linear operator describing the filtering policy that maps measurements to state estimation
$\mathcal{L}^{\dagger}$  Adjoint of the operator  $\mathcal{L}$
$\rho (\cdot)$  Spectral radius of a matrix
$\rho_{\alpha}$  Uniform upperbound of spectral radius of closed-loop system over  $S_{\alpha}$  -see Lemma 6
$c_{1}, c_{2}, c_{3}$  Uniform positive constants related to the cost function on  $S_{\alpha}$ -see Lemma 2
$C_{\alpha}$  Uniform upperbound of constant coefficient over  $S_{\alpha}$  -see Lemma 6
$C_L$  Constant coefficient of upperbound in Lemma 6
$D_{\alpha}$  Uniform upperbound of 2-norm of policy  $L$  over  $S_{\alpha}$ -see Lemma 6
Problem Parameters
System Quantities