SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} X - X ^ {*} - A _ {L} (X - X ^ {*}) A _ {L} ^ {\intercal} \\ = \left(L ^ {*} R - A _ {L ^ {*}} X ^ {*} H ^ {\intercal}\right) \left(L - L ^ {*}\right) ^ {\intercal} + \left(L - L ^ {*}\right) \left(R L ^ {* ^ {\intercal}} - H X ^ {*} A _ {L ^ {*}} ^ {\intercal}\right) \\ + (L - L ^ {*}) R (L - L ^ {*}) ^ {\intercal} + (L - L ^ {*}) H X ^ {*} H ^ {\intercal} (L - L ^ {*}) ^ {\intercal} \\ = (L - L ^ {*}) \left(R + H X ^ {*} H ^ {\intercal}\right) \left(L - L ^ {*}\right) ^ {\intercal} \\ \end{array}
$$
where the second equality follows because  $Y_{(L)} \succ 0$  and thus
$$
L ^ {*} R - A _ {L ^ {*}} X ^ {*} H ^ {\intercal} = - Y _ {(L)} ^ {- 1} \nabla J (L ^ {*}) = 0.
$$
$$
J (L) - J \left(L ^ {*}\right) = \operatorname {t r} \left[ \left(X - X ^ {*}\right) H ^ {\intercal} H \right],
$$
then by the equality in (20) and cyclic property of trace we obtain
$$
J (L) - J \left(L ^ {*}\right) = \operatorname {t r} \left[ Z Y _ {(L)} \right],
$$
$$
\begin{array}{l} Z := (L - L ^ {*}) (R + H X ^ {*} H ^ {\intercal}) (L - L ^ {*}) ^ {\intercal} \\ \succeq \underline {{\lambda}} (R) (L - L ^ {*}) (L - L ^ {*}) ^ {\intercal}. \\ \end{array}
$$
Therefore, for any  $L\in S_{\alpha}$  , we have
$$
J (L) - J \left(L ^ {*}\right) \geq \underline {{\lambda}} \left(Y _ {(L)}\right) \operatorname {t r} [ Z ] \geq \underline {{\lambda}} (R) \kappa_ {\alpha} \| L - L ^ {*} \| _ {F} ^ {2},
$$
and thus, we complete the proof of first part by the following choice of
$$
c _ {3} (\alpha) = \underline {{\lambda}} (R) \kappa_ {\alpha}.
$$
D.6 Proof of Lipschitz property in Lemma 2
Next, we provide the proof of locally Lipschitz property. Notice that the mappings  $L \to X_{(L)}$ ,  $L \to Y_{(L)}$  and  $L \to A_L$  are all real-analytic on the open set  $S \supset S_\alpha$ , and thus so is the mapping  $L \to \nabla J(L) = 2Y_{(L)}\left[-LR + A_LX_{(L)}H^\top\right]$ . Also, by Lemma 1,  $S_\alpha$  is compact and therefore the mapping  $L \to \nabla J(L)$  is  $\ell$ -Lipschitz continuous on  $S_\alpha$  for some  $\ell = \ell(\alpha) > 0$ . In the rest of the proof, we attempt to characterize  $\ell(\alpha)$  in terms of the problem parameters. By direct computation we obtain
$$
\begin{array}{l} \nabla J (L _ {1}) - \nabla J (L _ {2}) = \left(2 Y _ {(L _ {1})} - 2 Y _ {(L _ {2})}\right) \left[ - L _ {1} R + A _ {L _ {1}} X _ {(L _ {1})} H ^ {\intercal} \right] \\ + 2 Y _ {(L _ {2})} \left(\left[ - L _ {1} R + A _ {L _ {1}} X _ {(L _ {1})} H ^ {\mathsf {T}} \right] - \left[ - L _ {2} R + A _ {L _ {2}} X _ {(L _ {2})} H ^ {\mathsf {T}} \right]\right) \\ = 2 \left(Y _ {\left(L _ {1}\right)} - Y _ {\left(L _ {2}\right)}\right) \left[ - L _ {1} \left(R + H X _ {\left(L _ {1}\right)} H ^ {\intercal}\right) + A X _ {\left(L _ {1}\right)} H ^ {\intercal} \right] \\ + 2 Y _ {(L _ {2})} \left[ \left(L _ {2} - L _ {1}\right) \left(R + H X _ {(L _ {1})} H ^ {\intercal}\right) + A _ {L _ {2}} \left(X _ {(L _ {1})} - X _ {(L _ {2})}\right) H ^ {\intercal} \right]. \\ \end{array}
$$
$$
\left\| \nabla J (L _ {1}) - \nabla J (L _ {2}) \right\| _ {F} ^ {2} \leq \ell_ {1} ^ {2} \left\| Y _ {(L _ {1})} - Y _ {(L _ {2})} \right\| _ {F} ^ {2} + \ell_ {2} ^ {2} \left\| L _ {1} - L _ {2} \right\| _ {F} ^ {2} + \ell_ {3} ^ {2} \left\| X _ {(L _ {1})} - X _ {(L _ {2})} \right\| _ {F} ^ {2} \tag {25}
$$
$$
\begin{array}{l} \ell_ {1} = \ell_ {1} (L _ {1}) := 2 \| - L _ {1} \left(R + H X _ {\left(L _ {1}\right)} H ^ {\intercal}\right) + A X _ {\left(L _ {1}\right)} H ^ {\intercal} \| _ {\text {o p}}, \\ \ell_ {2} = \ell_ {2} (L _ {1}, L _ {2}) := 2 \| Y _ {(L _ {2})} \| _ {\mathrm {o p}} \| R + H X _ {(L _ {1})} H ^ {\mathsf {T}} \| _ {\mathrm {o p}}, \\ \ell_ {3} = \ell_ {3} (L _ {2}) := 2 \| Y _ {(L _ {2})} \| _ {\mathrm {o p}} \| A _ {L _ {2}} \| _ {\mathrm {o p}} \| H ^ {\intercal} \| _ {\mathrm {o p}}. \\ \end{array}
$$
On the other hand, by direct computation we obtain
$$
\begin{array}{l} Y _ {(L _ {1})} - Y _ {(L _ {2})} - A _ {L _ {1}} ^ {\top} \left(Y _ {(L _ {1})} - Y _ {(L _ {2})}\right) A _ {L _ {1}} \\ = \left(L _ {2} - L _ {1}\right) ^ {\intercal} H ^ {\intercal} Y _ {\left(L _ {2}\right)} A _ {L _ {2}} + A _ {L _ {2}} ^ {\intercal} Y _ {\left(L _ {2}\right)} H \left(L _ {2} - L _ {1}\right) \tag {26} \\ + \left(L _ {1} - L _ {2}\right) ^ {\intercal} H ^ {\intercal} Y _ {(L _ {2})} H \left(L _ {1} - L _ {2}\right) \\ \preceq \| L _ {1} - L _ {2} \| _ {F} \ell_ {4} I \\ \end{array}
$$
$$
\ell_ {4} = \ell_ {4} \left(L _ {1}, L _ {2}\right) := 2 \| H ^ {\intercal} Y _ {\left(L _ {2}\right)} A _ {L _ {2}} \| _ {\mathrm {o p}} + \| H ^ {\intercal} Y _ {\left(L _ {2}\right)} H \left(L _ {1} - L _ {2}\right) \| _ {\mathrm {o p}}.
$$
Now, consider the mapping  $L \to Z_{(L)}$  where  $Z_{(L)} = Z$  is the unique solution of the following Lyapunov equation:
$$
Z = A _ {L} ^ {\intercal} Z A _ {L} + I,
$$
which is well-defined and continuous on  $S \supset S_{\alpha}$ . Therefore, by comparison, we claim that
$$
\| Y _ {(L _ {1})} - Y _ {(L _ {2})} \| _ {F} \preceq \| L _ {1} - L _ {2} \| _ {F} \ell_ {4} \| Z _ {(L _ {1})} \| _ {F}.
$$
By a similar computation to that of (20), we obtain that
$$
\begin{array}{l} X _ {(L _ {1})} - X _ {(L _ {2})} - A _ {L _ {2}} \left(X _ {(L _ {1})} - X _ {(L _ {2})}\right) A _ {L _ {2}} ^ {\intercal} \\ = \left(L _ {1} R - A _ {L _ {1}} X _ {\left(L _ {1}\right)} H ^ {\intercal}\right) \left(L _ {1} - L _ {2}\right) ^ {\intercal} \\ + \left(L _ {1} - L _ {2}\right) \left(R L _ {1} ^ {\intercal} - H X _ {\left(L _ {1}\right)} A _ {L _ {1}} ^ {\intercal}\right) \tag {27} \\ - \left(L _ {1} - L _ {2}\right) R \left(L _ {1} - L _ {2}\right) ^ {\intercal} \\ - \left(L _ {1} - L _ {2}\right) H X _ {\left(L _ {1}\right)} H ^ {\intercal} \left(L _ {1} - L _ {2}\right) ^ {\intercal} \\ \preceq \| L _ {1} - L _ {2} \| _ {F} \ell_ {5} (Q + L _ {2} R L _ {2} ^ {\intercal}) \\ \end{array}
$$
$$
\ell_ {5} = \ell_ {5} (L _ {1}) := 2 \| - L _ {1} R + A _ {L _ {1}} X _ {(L _ {1})} H ^ {\mathsf {T}} \| _ {\mathrm {o p}} / \underline {{\lambda}} (Q).
$$
Therefore, by comparison, we claim that
$$
\left\| X _ {(L _ {1})} - X _ {(L _ {2})} \right\| _ {F} \preceq \left\| L _ {1} - L _ {2} \right\| _ {F} \ell_ {5} \left\| X _ {(L _ {2})} \right\| _ {F}.
$$
Finally, by compactness of  $S_{\alpha}$ , we claim that the following supremums are attained and thus, are achieved with some finite positive values:
$$
\bar {\ell} _ {1} (\alpha) := \sup  _ {L _ {1}, L _ {2} \in \mathcal {S} _ {\alpha}} \ell_ {1} (L _ {1}) \ell_ {4} (L _ {1}, L _ {2}) \| Z _ {(L _ {1})} \| _ {F},
$$
$$
\bar {\ell} _ {2} (\alpha) := \sup  _ {L _ {1}, L _ {2} \in \mathcal {S} _ {\alpha}} \ell_ {2} (L _ {1}, L _ {2}),
$$
$$
\bar {\ell} _ {3} (\alpha) := \sup  _ {L _ {1}, L _ {2} \in \mathcal {S} _ {\alpha}} \ell_ {3} (L _ {2}) \ell_ {5} (L _ {1}) \| X _ {(L _ {2})} \| _ {F}.
$$
Then, the claim follows by combining the bound in (25) with (26) and (27), and the following choice of
$$
\ell (\alpha) := \sqrt {\bar {\ell} _ {1} ^ {2} (\alpha) + \bar {\ell} _ {2} ^ {2} (\alpha) + \bar {\ell} _ {3} ^ {2} (\alpha)}.
$$
D.7 Gradient Flow (GF)
For completeness, in the next two sections, we analyze first-order methods in order to solve the minimization problem (10), although they are not part of the main paper. In this section, we consider a policy update according to the GF dynamics:
$$
[ \mathrm {G F} ] \quad \dot {L} _ {s} = - \nabla J (L _ {s}).
$$
We summarize the convergence result in the following proposition which is a direct consequence of Lemma 2.
Proposition 6. Consider any sublevel set  $S_{\alpha}$  for some  $\alpha > 0$ . Then, for any initial policy  $L_0 \in S_{\alpha}$ , the GF updates converges to optimality at a linear rate of  $\exp(-c_1(\alpha))$  (in both the function value and the policy iterate). In particular, we have
$$
J \left(L _ {s}\right) - J \left(L ^ {*}\right) \leq (\alpha - J \left(L ^ {*}\right)) \exp \left(- c _ {1} (\alpha) s\right),
$$
$$
\| L _ {s} - L ^ {*} \| _ {F} ^ {2} \leq \frac {\alpha - J (L ^ {*})}{c _ {3} (\alpha)} \exp (- c _ {1} (\alpha) s).
$$
Proof. Consider a Lyapunov candidate function  $V(L) \coloneqq J(L) - J(L^{*})$ . Under the GF dynamics
$$
\dot {V} \left(L _ {s}\right) = - \langle \nabla J (L _ {s}), \nabla J (L _ {s}) \rangle \leq 0.
$$
Therefore,  $L_{s}\in S_{\alpha}$  for all  $s > 0$  . But then, by Lemma 2, we can also show that
$$
\dot {V} (L _ {s}) \leq - c _ {1} (\alpha) V (L _ {s}) - c _ {2} (\alpha) \| L _ {s} - L ^ {*} \| _ {F} ^ {2}, \quad \text {f o r} s > 0.
$$
By recalling that  $c_{1}(\alpha) > 0$  is a positive constant independent of  $L$ , we conclude the following exponential stability of the GF:
$$
V (L _ {s}) \leq V (L _ {0}) \exp (- c _ {1} (\alpha) s),
$$
for any  $L_0 \in S_\alpha$  which, in turn, guarantees convergence of  $J(L_s) \to J(L^*)$  at the linear rate of  $\exp(-c_1(\alpha))$ . Finally, the linear convergence of the policy iterates follows directly from the second bound in Lemma 2:
$$
\| L _ {s} - L ^ {*} \| _ {F} ^ {2} \leq \frac {1}{c _ {3} (\alpha)} V (L _ {s}) \leq \frac {V (L _ {0})}{c _ {3} (\alpha)} \exp (- c _ {1} (\alpha) s).
$$
The proof concludes by noting that  $V(L_0) \leq \alpha - J(L^*)$  for any such initial value  $L_0 \in S_\alpha$ .
D.8 Gradient Descent (GD)
Here, we consider the GD policy update:
$$
\left[ \begin{array}{c} \end{array} \right] L _ {k + 1} = L _ {k} - \eta_ {k} \nabla J (L _ {k}),
$$
for  $k \in \mathbb{Z}$  and a positive stepsize  $\eta_k$ . Given the convergence result for the GF, establishing convergence for GD relies on carefully choosing the stepsize  $\eta_k$ , and bounding the rate of change of  $\nabla J(L)$ —at least on each sublevel set. This is achieved by the Lipschitz bound for  $\nabla J(L)$  on any sublevel set.
In what follows, we establish linear convergence of the GD update. Our convergence result only depends on the value of  $\alpha$  for the initial sublevel set  $S_{\alpha}$  that contains  $L_0$ . Note that our proof technique is distinct from those in [23] and [50]; nonetheless, it involves a similar argument using the gradient dominance property of  $J$ .
Theorem 4. Consider any sublevel set  $S_{\alpha}$  for some  $\alpha > 0$ . Then, for any initial policy  $L_0 \in S_{\alpha}$ , the  $GD$  updates with any fixed stepsize  $\eta_k = \eta \in (0,1/\ell(\alpha)]$  converges to optimality at a linear rate of  $1 - \eta c_1(\alpha)/2$  (in both the function value and the policy iterate). In particular, we have
$$
J \left(L _ {k}\right) - J \left(L ^ {*}\right) \leq [ \alpha - J \left(L ^ {*}\right) ] \left(1 - \eta c _ {1} (\alpha) / 2\right) ^ {k},
$$
$$
\| L _ {k} - L ^ {*} \| _ {F} ^ {2} \leq \left[ \frac {\alpha - J (L ^ {*})}{c _ {3} (\alpha)} \right] (1 - \eta c _ {1} (\alpha) / 2) ^ {k},
$$
with  $c_{1}(\alpha)$  and  $c_{3}(\alpha)$  as defined in Lemma 2.
Proof. First, we argue that the GD update with such a step size does not leave the initial sublevel set  $S_{\alpha}$  for any initial  $L_0 \in S_{\alpha}$ . In this direction, consider  $L(\eta) = L_0 - \eta \nabla J(L_0)$  for  $\eta \geq 0$  where  $L_0 \neq L^*$ . Then, by compactness of  $S_{\alpha}$  and continuity of the mapping  $\eta \to J(L(\eta))$  on  $S \supset S_{\alpha}$ , the following supremum is attained with a positive value  $\bar{\eta}_0$ :
$$
\bar {\eta} _ {0} := \sup  \left\{\eta : J (L (\zeta)) \leq \alpha , \forall \zeta \in [ 0, \eta ] \right\},
$$
where positivity of  $\bar{\eta}_0$  is a direct consequence of the strict decay of  $J(L(\eta))$  for sufficiently small  $\eta$  as  $\nabla J(L_0)\neq 0$ . This implies that  $L(\eta)\in S_{\alpha}\subset S$  for all  $\eta \in [0,\bar{\eta}_0]$  and  $J(L(\bar{\eta}_0)) = \alpha$ . Next, by the Fundamental Theorem of Calculus and smoothness of  $J(\cdot)$  (Lemma 7), for any  $\eta \in [0,\bar{\eta}_0]$  we have that,
$$
\begin{array}{l} J (L (\eta)) - J (L _ {0}) - \langle \nabla J (L _ {0}), L (\eta) - L _ {0} \rangle = \int_ {0} ^ {1} \langle \nabla J (L (\eta s)) - \nabla J (L _ {0}), L (\eta) - L _ {0} \rangle d s \\ \leq \| L (\eta) - L _ {0} \| _ {F} \int_ {0} ^ {1} \| \nabla J (L (\eta s)) - \nabla J (L _ {0}) \| _ {F} d s \\ \leq \ell (\alpha) \| L (\eta) - L _ {0} \| _ {F} \int_ {0} ^ {1} \| L (\eta s) - L _ {0} \| _ {F} d s \\ = \frac {1}{2} \ell (\alpha) \eta \| L (\eta) - L _ {0} \| _ {F} \| \nabla J (L _ {0}) \| _ {F}, \\ \end{array}
$$
where  $\| \cdot \| _F$  denotes the Frobenius norm, the first inequality is a consequence of Cauchy-Schwartz, and the second one is due to (11c) and the fact that  $L(\eta s)$  remains in  $S_{\alpha}$  for all  $s\in [0,1]$ . By the definition of  $L(\eta)$ , it now follows that,
$$
J (L (\eta)) - J (L _ {0}) \leq \eta \| \nabla J (L _ {0}) \| _ {F} ^ {2} \left(\frac {\ell (\alpha) \eta}{2} - 1\right). \tag {28}
$$
This implies  $J(L(\eta)) \leq J(L_0)$  for all  $\eta \leq 2 / \ell(\alpha)$ , and thus concluding that  $\bar{\eta}_0 \geq 2 / \ell(\alpha)$ . This justifies that  $L(\eta) \in S_{\alpha}$  for all  $\eta \in [0, 2 / \ell(\alpha)]$ . Next, if we consider the GD update with any
Note that a direct application of Descent Lemma [51, Lemma 5.7] may not be justified as one has to argue about the uniform bound for the Hessian of  $J$  over the non-convex set  $\mathcal{S}_{\alpha}$  where  $J$  is  $\ell (\alpha)$ -Lipschitz only on  $\mathcal{S}_{\alpha}$ . Also see the proof of [50, Theorem 2].
fixed stepsize  $\eta \in (0,1 / \ell (\alpha)]$  and apply the bound in (28) and the gradient dominance property in Lemma 2, we obtain
$$
J (L _ {1}) - J (L _ {0}) \leq \eta c _ {1} (\frac {\ell (\alpha) \eta}{2} - 1) [ J (L _ {0}) - J (L ^ {*}) ],
$$
which by subtracting  $J(L^{*})$  results in
$$
J (L _ {1}) - J (L ^ {*}) \leq \left(1 - \frac {\eta c _ {1}}{2}\right) [ J (L _ {0}) - J (L ^ {*}) ],
$$
as  $\eta c_{1}(\ell (\alpha)\eta /2 - 1)\leq -\eta c_{1} / 2$  for all  $\eta \in (0,1 / \ell (\alpha)]$ . By induction, and the fact that both  $c_{1}(\alpha)$  and the choice of  $\eta$  only depends on the value of  $\alpha$ , we conclude the convergence in the function value at a linear rate of  $1 - (\eta c_{1} / 2)$  and the constant coefficient of  $\alpha -J(L^{*})\geq J(L_0) - J(L^*)$ . To complete the proof, the linear convergence of the policy iterates follows directly from the second bound in Lemma 2.
Appendix E Proofs for the analysis of the constrained SGD algorithm
E.1 Proof of lemma 3: Derivation of stochastic gradient formula
Proof. For small enough  $\Delta \in \mathbb{R}^{n\times m}$
$$
\begin{array}{l} \varepsilon (L + \Delta , \mathcal {Y}) - \varepsilon (L, \mathcal {Y}) = \| e _ {T} (L + \Delta) \| ^ {2} - \| e _ {T} (L) \| ^ {2} \\ = 2 \operatorname {t r} \left[ \left(e _ {T} (L + \Delta) - e _ {T} (L)\right) e _ {T} ^ {\intercal} (L) \right] + o (\| \Delta \|)). \\ \end{array}
$$
The difference
$$
e _ {T} (L + \Delta) - e _ {T} (L) = E _ {1} (\Delta) + E _ {2} (\Delta) + o (\| \Delta \|),
$$
with the following terms that are linear in  $\Delta$ :
$$
E _ {1} (\Delta) := - \sum_ {t = 0} ^ {T - 1} H \left(A _ {L}\right) ^ {t} \Delta y (T - t - 1),
$$
$$
E _ {2} (\Delta) := \sum_ {t = 1} ^ {T - 1} \sum_ {k = 1} ^ {t} H (A _ {L}) ^ {t - k} \Delta H (A _ {L}) ^ {k - 1} L y (T - t - 1).
$$
Therefore, combining the two identities, the definition of gradient under the inner product  $\langle A,B\rangle \coloneqq$  tr  $[AB^{\intercal}]$ , and ignoring the higher order terms in  $\Delta$  yields,
$$
\langle \nabla_ {L} \varepsilon (L, y), \Delta \rangle = 2 \operatorname {t r} \left[ \left(E _ {1} (\Delta) + E _ {2} (\Delta)\right) e _ {T} ^ {\intercal} (L) \right],
$$
which by linearity and cyclic permutation property of trace reduces to:
$$
\begin{array}{l} \left\langle \nabla_ {L} \varepsilon (L, y), \Delta \right\rangle = - 2 \operatorname {t r} \left[ \Delta \left(\sum_ {t = 0} ^ {T - 1} y (T - t - 1) e _ {T} ^ {\mathbf {T}} (L) H \left(A _ {L}\right) ^ {t}\right) \right] \\ + 2 \operatorname {t r} \left[ \Delta \left(\sum_ {t = 1} ^ {T - 1} \sum_ {k = 1} ^ {t} H \left(A _ {L}\right) ^ {k - 1} L y (T - t - 1) e _ {T} ^ {\mathsf {T}} (L) H \left(A _ {L}\right) ^ {t - k}\right) \right]. \\ \end{array}
$$
This holds for all admissible  $\Delta$ , concluding the formula for the gradient.
E.2 Proof of Lemma 4: Robustness of the policy with respect to perturbation
Proof. Recall the stability certificate  $s_K$  as proposed in [30, Lemma IV.1] for a choice of constant mapping  $\mathcal{Q}: K \to \Lambda \succ 0$  and dual problem parameters as discussed in Remark 3. Then, we arrive at  $s_K = \underline{\lambda}(\Lambda) / (2\overline{\lambda}(Z)\|H^{\mathsf{T}}\Delta^{\mathsf{T}}\|)$ , for which  $\rho(A^{\mathsf{T}} + H^{\mathsf{T}}(L^{\mathsf{T}} + \eta\Delta^{\mathsf{T}})) < 1$  for any  $\eta \in [0,s_K]$ . But, the spectrum of a square matrix and its transpose are identical, thus  $L + \eta\Delta \in S$  for any such  $\eta$ . The claim then follows by noting that the operator norm of a matrix and its transpose are identical, and the resulting lowerbound as follows  $s_K \geq \underline{\lambda}(\Lambda) / (2\overline{\lambda}(Z)\|H\|\|\Delta\|_F)$ .
E.3 Proof of Lemma 5: Uniform lower-bound on stepsize
Proof. Without loss of generality, suppose  $L_0 \neq L^*$  and  $E \neq 0$ , and let  $L(\eta) \coloneqq L_0 - \eta E$ . By compactness of  $S_{\beta}$  and continuity of the mapping  $\eta \to J(L(\eta))$  on  $S \supset S_{\beta}$ , the following supremum is attained by  $\eta_{\beta}$ :
$$
\eta_ {\beta} := \sup  \left\{\eta : J (L (\zeta)) \leq \beta , \forall \zeta \in [ 0, \eta ] \right\}. \tag {29}
$$
Note that  $\eta_{\beta}$  is strictly positive for  $\beta >\alpha$  because  $L_0\in S_\alpha \subset S_\beta$  and  $J(\cdot)$  is coercive (Lemma 1) and its domain is open (Lemma 4). This implies that  $L(\eta)\in S_{\beta}\subset S$  for all  $\eta \in [0,\eta_{\beta}]$  and  $J(L(\eta_{\beta})) = \beta$
Next, we want to show that  $\eta_{\beta}$  is uniformly lower bounded with high probability. By the Fundamental Theorem of Calculus, for any  $\eta \in [0,\eta_{\beta}]$  we have
$$
\begin{array}{l} J (L (\eta)) - J (L _ {0}) - \langle \nabla J (L _ {0}), L (\eta) - L _ {0} \rangle = \int_ {0} ^ {1} \langle \nabla J (L (\eta s)) - \nabla J (L _ {0}), L (\eta) - L _ {0} \rangle d s \\ \leq \| L (\eta) - L _ {0} \| _ {F} \int_ {0} ^ {1} \| \nabla J (L (\eta s)) - \nabla J (L _ {0}) \| _ {F} d s \\ \leq \ell (\beta) \| L (\eta) - L _ {0} \| _ {F} \int_ {0} ^ {1} \| L (\eta s) - L _ {0} \| _ {F} d s \\ = \frac {1}{2} \ell (\beta) \eta^ {2} \| E \| _ {F} ^ {2}, \tag {30} \\ \end{array}
$$
where the first inequality is a consequence of Cauchy-Schwartz, and the second one is due to (11c) and the fact that  $L(\eta s)$  remains in  $\mathcal{S}_{\beta}$  for all  $s\in [0,1]$  by definition of  $\eta_{\beta}$ . Note that the assumption implies  $\| E\| _F\leq (\gamma +1)\| \nabla J(L_0)\| _F$ . Thus, (30) implies that
$$
\begin{array}{l} J (L (\eta)) \leq J (L _ {0}) - \eta \langle \nabla J (L _ {0}), E \rangle + \frac {1}{2} \ell (\beta) \eta^ {2} \| E \| _ {F} ^ {2} \\ \leq J \left(L _ {0}\right) - \eta \left\langle \nabla J \left(L _ {0}\right), E - \nabla J \left(L _ {0}\right) \right\rangle - \eta \| \nabla J \left(L _ {0}\right) \| _ {F} ^ {2} + \frac {1}{2} \ell (\beta) \eta^ {2} \| E \| _ {F} ^ {2} \tag {31} \\ \leq J (L _ {0}) + \| \nabla J (L _ {0}) \| _ {F} ^ {2} \left[ \frac {1}{2} (\gamma + 1) ^ {2} \ell (\beta) \eta^ {2} + (\gamma - 1) \eta \right]. \\ \end{array}
$$
Therefore, for  $\eta$  to be a feasible point in the supremum in (29), it suffices to satisfy:
$$
\frac {1}{2} \| \nabla J (L _ {0}) \| _ {F} ^ {2} \left[ (\gamma + 1) ^ {2} \ell (\beta) \eta^ {2} + 2 (\gamma - 1) \eta \right] \leq \beta - \alpha ,
$$
$$
\left[ (\gamma + 1) \ell (\beta) \eta + \frac {\gamma - 1}{\gamma + 1} \right] ^ {2} \leq \left(\frac {\gamma - 1}{\gamma + 1}\right) ^ {2} + \frac {2 \ell (\beta) [ \beta - \alpha ]}{\| \nabla J (L _ {0}) \| _ {F} ^ {2}}.
$$
But then it suffices to have
$$
(\gamma + 1) \ell (\beta) \eta + \frac {\gamma - 1}{\gamma + 1} \leq \frac {\sqrt {2 \ell (\beta) [ \beta - \alpha ]}}{\| \nabla J (L _ {0}) \| _ {F}}.
$$
Finally, note that by Lemma 2 and (11c) we have
$$
\left\| \nabla J \left(L _ {0}\right) \right\| _ {F} \leq \frac {\ell (\alpha)}{c _ {3} (\alpha)} \left[ J \left(L _ {0}\right) - J \left(L ^ {*}\right) \right] \leq \frac {\ell (\alpha)}{c _ {3} (\alpha)} \left[ \alpha - \alpha^ {*} \right]. \tag {32}
$$
Using this uniform bound of gradient on sublevel set  $S_{\alpha}$  and noting that  $\gamma \in [0,1]$ , we can obtain the sufficient condition for  $\eta$  to be feasible. This completes the proof.
E.4 Proof of Proposition 2: Linear decay in cost value
Proof. Suppose  $L_0 \neq L^*$  and let  $L(\eta) \coloneqq L_0 - \eta E$ . Note that  $E$  may not be necessarily in the direction of decay in  $J(L)$ , however, we can argue the following:
Choose  $\beta = \alpha$  in Lemma 5 and note that  $\eta_{\beta}$  as defined in (29) will be lower bounded as  $\eta_{\beta} \geq \bar{\eta}_0$ , and thus  $\bar{\eta}_0$  is feasible. Recall that  $L(\eta) \in S_{\beta} \subset S$  for all  $\eta \in [0, \eta_{\beta}]$ . Also, for any  $\eta \in [0, \bar{\eta}_0]$  and  $\gamma \in [0, 1)$ , from (31) we obtain that:
$$
\begin{array}{l} J (L (\eta)) - J (L _ {0}) \leq \| \nabla J (L _ {0}) \| _ {F} ^ {2} \left[ \frac {1}{2} (\gamma + 1) ^ {2} \ell (\alpha) \eta^ {2} + (\gamma - 1) \eta \right] \\ \leq c _ {1} (\alpha) \big [ J (L _ {0}) - J (L ^ {*}) \big ] \left[ \frac {1}{2} (\gamma + 1) ^ {2} \ell (\alpha) \eta^ {2} + (\gamma - 1) \eta \right] \\ \end{array}
$$
where, as  $\gamma < 1$ , the last inequality follows by (11a) for any  $\eta \leq \min \{2\bar{\eta}_0,\eta_\beta \}$ . By the choice of  $\bar{\eta}_0$ , then we obtain that
$$
J (L (\bar {\eta} _ {0})) - J (L _ {0}) \leq - c _ {1} (\alpha) \left[ \frac {(\gamma - 1) ^ {2}}{2 (\gamma + 1) ^ {2} \ell (\alpha)} \right] [ J (L _ {0}) - J (L ^ {*}) ]
$$
$$
J (L (\bar {\eta} _ {0})) - J (L ^ {*}) \leq \left(1 - c _ {1} (\alpha) \left[ \frac {(\gamma - 1) ^ {2}}{2 (\gamma + 1) ^ {2} \ell (\alpha)} \right]\right) [ J (L _ {0}) - J (L ^ {*}) ].
$$
E.5 Proof of Theorem 1: Convergence of SGD algorithm
Before proving this result, we first discuss that the claim of Theorem 1 is sufficient for establishing the complexity result of Theorem 3; i.e., it suffices to guarantee convergence from every initial (stabilizing) policy to a neighborhood of optimality where norm of the gradient is controlled.
As shown in Lemma 2 (and discussed in Remark 4), the cost maintains the PL property on each sublevel set. In particular (11a) implies that on each  $S_{\alpha}$ ,  $\| \nabla J(L) \|$  characterizes the optimality gap  $J(L) - J(L^{*})$  by:
$$
c _ {1} (\alpha) [ J (L) - J (L ^ {*}) ] \leq \| \nabla J (L) \| ^ {2}
$$
for some constant  $c_{1}(\alpha)$ . This implies that if we have arrived at a candidate policy  $L_{k}$  for which the gradient is small, then the optimality gap should be small (involving the constant  $c_{1}(\alpha)$  that is independent of  $L_{k}$ ). This is the reason that in Theorem 1, it suffices to argue about the generated sequence  $L_{k}$  to have a linear decay unless entering a neighborhood of  $L^{*}$  containing policies with small enough gradients (denoted by  $\mathcal{C}_{\tau}$ ). In particular, if for some  $j < k$ , we arrive at some policy  $L_{j} \in \mathcal{C}_{\tau}$ , then by (11a) we can conclude that:
$$
J (L _ {j}) - J (L ^ {*}) \leq \frac {1}{c _ {1} (\alpha)} \| \nabla J (L _ {j}) \| _ {F} ^ {2} \leq \frac {s _ {0} ^ {2}}{c _ {1} (\alpha) \tau^ {2}},
$$
which is directly controlled by the bias term  $s_0$ . This is the bound used also in the proof of Theorem 3.
Finally, recall that every (stabilizing) initial policy  $L_0$  amounts to a finite value of  $J(L_0)$  and thus lies in some sublevel set  $S_{\alpha}$ . So, starting from such  $L_0$ , Theorem 1 guarantees linear decay of the optimality gap till the trajectory enters that small neighborhood. Finally, note that the radius of this neighborhood  $C_{\tau}$  is characterized by the bias term  $s_0$  which itself is exponentially decaying to zero in the trajectory length  $T$ .
Next, we provide the proof of this result that is essentially an induction argument using Proposition 2.