SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\operatorname {d i s t} \left(w ^ {\tau}, S\right) \leq \operatorname {d i s t} (w, S) + \tau \| \nabla \ell (w, z) \| \leq 2 r.
$$
Therefore the gradient  $\nabla \ell (\cdot ,z)$  is  $\beta$ -Lipschitz on the entire line segment  $\{w_{\tau}:0\leq \tau \leq 1 / \beta \}$ . The descent lemma therefore guarantees  $\| \nabla \ell (w,z)\| ^2\leq 2\beta \ell (w,z)$ . Therefore upon taking expectations we obtain the second moment bound:  $\mathbb{E}\| \nabla \ell (w,z)\| ^2\leq 2\beta \mathcal{L}(w)$ . Next, we compute
$$
\begin{array}{l} \mathbb {E} \operatorname {d i s t} ^ {2} \left(w ^ {+}, S\right) \leq \mathbb {E} \| w ^ {+} - \bar {w} \| ^ {2} = \mathbb {E} \| (w - \bar {w}) - \eta \nabla \mathcal {L} (w, z) \| ^ {2} \\ = \| w - \bar {w} \| ^ {2} - 2 \eta \langle \nabla \mathcal {L} (w), w - \bar {w} \rangle + \eta^ {2} \mathbb {E} \| \nabla \mathcal {L} (w, z) \| ^ {2} \\ \leq \operatorname {d i s t} ^ {2} (w, S) - 2 \eta \theta \cdot \mathcal {L} (w) + 2 \eta^ {2} \beta \mathcal {L} (w) (A.1) \\ = \operatorname {d i s t} ^ {2} (w, S) - 2 \eta (\theta - \beta \eta) \mathcal {L} (w) \\ \leq (1 - \alpha \eta (\theta - \eta \beta)) \operatorname {d i s t} ^ {2} (w, S), (A.2) \\ \end{array}
$$
where (A.1) follows from (2.2) while (A.2) follows from (2.1). The proof is complete.
A.4 Proof of Theorem 2.4
Define the stopping time  $\tau = \inf \{t\geq 1:w_t\notin S_r\}$  and set  $U_{t} = \mathrm{dist}^{2}(w_{t},S)$ . Note that we may equivalently write  $\tau = \inf \{t\geq 1:\bar{U}_t > r^2\}$ . Now, multiplying (2.3) through by  $1_{\tau >t}$  yields the estimate
$$
\mathbb {E} \left[ U _ {t + 1} 1 _ {\tau > t} \mid w _ {1: t} \right] \leq \left(1 - \alpha \eta (\theta - \beta \eta)\right) U _ {t} 1 _ {\tau > t}. \tag {A.3}
$$
An application of Theorem B.2 with  $q = (1 - \alpha \eta (\theta -\beta \eta))$  completes the proof.
A.5 Proof of Theorem 2.5
We begin with the following simple lemma that bounds the second moment of the gradient estimator.
Lemma A.3. For any point  $w \in B_r(w_0)$ , we have  $\mathbb{E}\| \ell (w,z)\| ^2\leq 2\beta \cdot \mathcal{L}(w)$ .
Proof. Let  $\bar{w}_0\in \mathrm{proj}_S(w_0)$  be arbitrary. Observe that Lipschitz continuity of  $\nabla \ell (\cdot ,z)$  on  $B_{3r}(w_0)$  and interpolation ensure
$$
\| \nabla \ell (w, z) \| = \| \nabla \ell (w, z) - \nabla \ell (\bar {w} _ {0}, z) \| \leq \beta \cdot \| w - \bar {w} _ {0} \| \leq 2 \beta r.
$$
$$
\left\| w ^ {\tau} - w _ {0} \right\| \leq \left\| w - w _ {0} \right\| + \tau \| \nabla \ell (w, z) \| \leq 3 r.
$$
Therefore the gradient  $\nabla \ell (\cdot ,z)$  is  $\beta$  -Lipschitz on the entire line segment  $\{w_{\tau}:0\leq \tau \leq 1 / \beta \}$ . The descent lemma therefore guarantees  $\| \nabla \ell (w,z)\| ^2\leq 2\beta \ell (w,z)$ . Taking the expectation of both sides completes the proof.
Next we prove the following lemma that simultaneously estimates (1) one step progress of the iterates towards  $S$  and (2) how far the iterates move away from the center  $w_{0}$ .
Lemma A.4. Fix a point  $w \in B_{r/3}(w_0)$  and choose  $\bar{w}_0 \in \mathrm{proj}_S(w_0) \cap B_{r/3}(w_0)$ . Assume  $\eta \leq \frac{\theta}{\beta}$  and define  $w^+ = w - \eta \nabla \ell(w, z)$  where  $z \sim \mathcal{P}$ . Then the following estimates hold:
$$
\mathbb {E} \left[ \operatorname {d i s t} ^ {2} \left(w ^ {+}, S\right) \right] \leq (1 - \eta (\theta - \beta \eta) \alpha) \operatorname {d i s t} ^ {2} (w, S) \tag {A.4}
$$
$$
\mathbb {E} \left[ \left\| w ^ {+} - \bar {w} _ {0} \right\| ^ {2} \right] \leq \| w - \bar {w} _ {0} \| ^ {2} + 2 \eta \rho \cdot \operatorname {d i s t} (w, S) \tag {A.5}
$$
Proof. Fix any point  $v \in S$  and observe that
$$
\| w ^ {+} - v \| ^ {2} = \| w - v \| ^ {2} - 2 \eta \langle \nabla \ell (w, z), w - v \rangle + \eta^ {2} \| \nabla \ell (w, z) \| ^ {2}.
$$
Taking the expectation with respect to  $z$  and using Lemma A.3, we deduce
$$
\mathbb {E} \| w ^ {+} - v \| ^ {2} \leq \| w - v \| ^ {2} - 2 \eta \langle \nabla \mathcal {L} (w), w - v \rangle + 2 \beta \eta^ {2} \mathcal {L} (w). \tag {A.6}
$$
We will use this estimate multiple times for different vectors  $v$ .
Choose any point  $\bar{w} \in \mathrm{proj}_S(w)$  satisfying the aiming condition (2.2). Define  $U := \mathrm{dist}^2(w, S)$  and  $U_+ := \mathrm{dist}^2(w_+, S)$ . Then setting  $v := \bar{w}$  in (A.6), we deduce
$$
\mathbb {E} \left[ U _ {+} \right] \leq U - 2 \eta \langle \nabla \mathcal {L} (w), w - \bar {w} \rangle + 2 \beta \eta^ {2} \mathcal {L} (w).
$$
Using the aiming condition (2.2) we therefore deduce
$$
\begin{array}{l} \mathbb {E} \left[ U _ {+} \right] \leq U - 2 \eta \theta \mathcal {L} (w) + 2 \beta \eta^ {2} \mathcal {L} (w) \\ = U - 2 \eta (\theta - \beta \eta) \mathcal {L} (w) \\ \leq (1 - \eta (\theta - \beta \eta) \alpha) U, \\ \end{array}
$$
where the last inequality follows from quadratic growth. This establishes (A.4).
Next, set  $v \coloneqq \bar{w}_0$  in (A.6). Defining  $V = \| w - \bar{w}_0\|^2$ ,  $V_+ = \| w^+ - \bar{w}_0\|^2$ , and using Assumption 2 we therefore deduce
$$
\begin{array}{l} \mathbb {E} \left[ V _ {+} \right] \leq V - 2 \eta (\theta \mathcal {L} (w) - \rho \sqrt {U}) + 2 \beta \eta^ {2} \mathcal {L} (w). \\ = V - 2 \eta (\theta - \eta \beta) \mathcal {L} (w) + 2 \eta \rho \sqrt {U} \\ = V + 2 \eta \rho \sqrt {U} \\ \end{array}
$$
where the last estimate follows from the inequality  $\beta \eta \leq \theta$ . This establishes (A.5).
For each  $t \geq 0$ , let  $1_{t}$  denote the indicator of the event  $E_{t} \coloneqq \{w_{0}, \ldots, w_{t} \in B_{r/3}(w_{0})\}$ . Define the random variables  $U_{t} = \mathrm{dist}^{2}(w_{t}, S)$  and  $V_{t} = \| w_{t} - \bar{w}_{0} \|^2$ . We may now multiply (A.4) by  $1_{E_{t}}$ . Noting that  $1_{E_{t+1}} \leq 1_{E_{t}}$  we may iterate the bound yielding
$$
\mathbb {E} \left[ 1 _ {t} U _ {t} \right] \leq (1 - \eta (\theta - \beta \eta) \alpha) ^ {t} U _ {0} \tag {A.7}
$$
Similarly, multiplying (A.4) by  $1_{E_t}$  and using (A.7), we deduce
$$
\begin{array}{l} \mathbb {E} \left[ 1 _ {t + 1} V _ {t + 1} \right] \leq \mathbb {E} \left[ 1 _ {t} V _ {t} \right] + 2 \eta \rho \mathbb {E} \sqrt {1 _ {t} U _ {t}} \\ \leq \mathbb {E} \left[ 1 _ {t} V _ {t} \right] + 2 \eta \rho (1 - \eta (\theta - \beta \eta) \alpha) ^ {t / 2} \sqrt {U _ {0}} \\ \end{array}
$$
Iterating the recursion gives
$$
\begin{array}{l} \mathbb {E} \left[ 1 _ {t} V _ {t} \right] \leq U _ {0} + \frac {2 \rho \eta \sqrt {U _ {0}}}{1 - \sqrt {\left(1 - \eta (\theta - \beta \eta) \alpha\right)}} \\ \leq U _ {0} + \frac {4 \rho \sqrt {U _ {0}}}{(\theta - \beta \eta) \alpha}. \tag {A.8} \\ \end{array}
$$
We now lower bound the probability of escaping from the ball. Note that within event  $E_{t}$ , we have
$$
V _ {t} \leq \left(\left\| w _ {t} - w _ {0} \right\| + \left\| w _ {0} - \bar {w} _ {0} \right\|\right) ^ {2} \leq \left(1 + \delta_ {1}\right) ^ {2} r ^ {2}.
$$
$$
\begin{array}{l} \mathbb {P} \left(E _ {t} ^ {c}\right) \leq \mathbb {P} \left[ V _ {t} > \left(1 + \delta_ {1}\right) ^ {2} r ^ {2} \right] \\ \leq \mathbb {P} \left[ V _ {t} > \left(1 + \delta_ {1}\right) ^ {2} r ^ {2} \mid E _ {t} \right] \cdot \mathbb {P} (E _ {t}) \\ \leq \frac {\mathbb {E} \left[ 1 _ {E _ {t}} V _ {t} \mid E _ {t} \right] \mathbb {P} \left(E _ {t}\right)}{\left(1 + \delta_ {1}\right) ^ {2} r ^ {2}} (A.9) \\ \leq \frac {\mathbb {E} \left[ 1 _ {E _ {t}} V _ {t} \right]}{\left(1 + \delta_ {1}\right) ^ {2} r ^ {2}} \\ \leq \frac {U _ {0} + \frac {4 \rho \sqrt {U _ {0}}}{(\theta - \beta \eta) \alpha}}{(1 + \delta_ {1}) ^ {2} r ^ {2}} (A.10) \\ \leq \left(1 + \frac {4 \rho}{(\theta - \beta \eta) \alpha r}\right) \delta_ {1}, \\ \end{array}
$$
where (A.9) follows from Markov's inequality and (A.10) follows from (A.8).
Next, we estimate the probability that  $U_{t}$  remains small within the event  $E_{t}$ . To that end, let define the constant  $C_t \coloneqq (1 - \eta (\theta -\beta \eta)\alpha)^t U_0 / \delta_2$ . Then Markov's inequality yields
$$
\mathbb {P} \left(U _ {t} > C _ {t} \mid E _ {t}\right) \leq \frac {\mathbb {E} \left[ U _ {t} \mid E _ {t} \right]}{C _ {t}} \leq \frac {\mathbb {E} \left[ 1 _ {E _ {t}} U _ {t} \right]}{C _ {t} \mathbb {P} \left(E _ {t}\right)} \leq \frac {\delta_ {2}}{\mathbb {P} \left(E _ {t}\right)}.
$$
Finally, we unconditionally bound the probability that  $U_{t}$  remains small:
$$
\begin{array}{l} P \left(U _ {t} \leq C _ {t}\right) \geq P \left(U _ {t} \leq C _ {t} \mid E _ {t}\right) P \left(E _ {t}\right) \\ \geq \left(1 - \frac {\delta_ {2}}{\mathbb {P} (E _ {t})}\right) \mathbb {P} (E _ {t}), \\ = \mathbb {P} (E _ {t}) - \delta_ {2}. \\ \end{array}
$$
A.6 Proof of Theorem 3.1
We first prove the following lemma, which does not require quadratic growth and relies on the Lipschitz continuity of the Jacobian  $\nabla \mathcal{L}$ .
Lemma A.5. Consider a function  $\mathcal{L}(w) = \frac{1}{2}\| F(w)\|^2$  where  $F\colon \mathbf{R}^d\to \mathbf{R}^n$  is  $C^1$ -smooth and the Jacobian  $\nabla F$  is  $L$ -Lipschitz on the ball  $B_{r}(w_{0})$ . Then the following estimates
$$
| \langle \nabla \mathcal {L} (w), u - v \rangle | \leq 8 r ^ {2} L \sqrt {2 \mathcal {L} (w)} \tag {A.11}
$$
$$
\left| \langle \nabla \mathcal {L} (w), w - u \rangle - 2 \mathcal {L} (w) \right| \leq L \sqrt {\mathcal {L} (w) / 2} \cdot \| w - u \| ^ {2}, \tag {A.12}
$$
hold for all  $w, u, v \in B_r(w_0)$  satisfying  $F(u) = F(v) = 0$ .
Proof. Fix any  $w \in B_r(w_0)$  and  $u, v \in B_r(w_0)$  satisfying  $F(u) = F(v) = 0$ . We first prove (A.11). To this end, we compute
$$
\begin{array}{l} | \langle \nabla \mathcal {L} (w), u - v \rangle | = | \langle \nabla F (w) ^ {\top} F (w), u - v \rangle | \\ = | \langle F (w), \nabla F (w) (u - v) \rangle | \\ \leq \sqrt {2 \mathcal {L} (w)} \cdot \| \nabla F (w) (u - v) \|, \\ \end{array}
$$
where the last estimate follows from the Cauchy-Schwarz inequality. Next, the fundamental theorem of calculus and Lipschitz continuity of  $\nabla F$  yields
$$
\begin{array}{l} 0 = F (v) - F (u) = \int_ {0} ^ {1} \nabla F (u + t (v - u)) (v - u) d t \\ = \nabla F (w) (v - u) + E, \\ \end{array}
$$
where  $\| E \| \leq L \| v - u \| (\| u - w \| + \| v - u \|) \leq 8r^2 L$ . Thus we have proved (A.11). In order to see (A.12), we compute
$$
\begin{array}{l} \left| \langle \nabla L (w), w - u \rangle - \| F (w) \| ^ {2} \right| = \left| \langle F (w), \nabla F (w) (w - u) - F (w) \rangle \right| \\ \leq \frac {L}{2} \| F (w) \| \| w - u \| ^ {2}, \\ \end{array}
$$
where the last inequality follows from Cauchy-Schwarz and Lipschitz continuity of the Jacobian  $\nabla F$ . Thus (A.12) holds.
Lemma A.5 quickly yields the following corollary.
Corollary A.6. Consider a function  $\mathcal{L}(w) = \frac{1}{2}\| F(w)\|^2$  where  $F\colon \mathbf{R}^d\to \mathbf{R}^n$  is  $C^1$ -smooth and the Jacobian  $\nabla F$  is  $L$ -Lipschitz on the ball  $B_{r}(w_{0})$ . Suppose moreover that
$$
\mathcal {L} (w) \geq \frac {\alpha}{2} \cdot \operatorname {d i s t} ^ {2} (w, S)
$$
where  $S = \{w : F(w) = 0\}$ . Then the estimates
$$
\langle \nabla \mathcal {L} (w), w - \bar {w} \rangle \geq 2 \mathcal {L} (w) - \frac {L \sqrt {2}}{\alpha} \cdot \mathcal {L} (w) ^ {3 / 2}. \tag {A.13}
$$
$$
\langle \nabla \mathcal {L} (w), w - v \rangle \geq 2 \mathcal {L} (w) - \frac {L \sqrt {2}}{\alpha} \cdot \mathcal {L} (w) ^ {3 / 2} - 8 r ^ {2} L \sqrt {2 \mathcal {L} (w)}, \tag {A.14}
$$
hold for all  $w \in B_r(w_0)$ ,  $v \in B_r(w_0) \cap S$ , and  $\bar{w} \in B_r(w_0) \cap \mathrm{proj}_S(w)$ .
Proof. We will apply Lemma A.5. Setting  $u \coloneqq \bar{w}$  in Lemma A.12 and using quadratic growth yields the estimate:
$$
\begin{array}{l} \langle \nabla \mathcal {L} (w), w - \bar {w} \rangle \geq 2 \mathcal {L} (w) - L \sqrt {\mathcal {L} (w) / 2} \cdot \| w - \bar {w} \| ^ {2} \\ \geq 2 \mathcal {L} (w) - \frac {L}{\alpha} \sqrt {2 \mathcal {L} (w)} \cdot \mathcal {L} (w), \\ \end{array}
$$
which establishes (A.13). Adding this estimate to (A.12) with  $u = \bar{w}$  yields (A.14).
Theorem 3.1 follows immediately from the corollary. Indeed, note that let  $\bar{w} \in \mathrm{proj}_S(w)$  satisfies  $\| w - \bar{w}\| \leq 2r$ . Therefore we may apply Corollary A.6 and use the estimate  $\mathcal{L}(w) \leq \frac{\beta}{2}\| w - \bar{w}\|^2$ .
A.7 Proof of Theorem 3.3
Theorem 3.2 implies that with probability at least  $1 - p - 2\exp (-\frac{ml}{2}) - (1 / m)^{\Theta (\ln m)}$ , the Jacobian of  $\nabla F$  is Lipschitz with constant  $L = \tilde{O}\left(\frac{r^{3l}}{\sqrt{m}}\right)$  on  $B_{r}(w_{0})$  and the PL-condition holds on  $B_{2r}(w_0)$  with parameter  $\lambda_0 / 2$ . Using the assumption  $r = \Omega (\frac{1}{\sqrt{\lambda_0}})$ , we may apply Lemma 2.1 to deduce quadratic growth. Next, we will apply Theorem 3.1 in order to deduce (uniform) aiming with parameters  $\theta = 1$ . To this end, it suffices to ensure  $2rL\sqrt{\beta} /\lambda_0\leq 1$ , which follows from the prerequisite assumption  $m = \Omega (\frac{r^{6l + 2}}{\alpha^2})$ .
To see the last claim, for any  $w \in B_r(w_0)$  we compute the Hessian
$$
\nabla^ {2} \ell_ {i} (w) = \nabla f (w, x _ {i}) \nabla f (w, x _ {i}) ^ {T} + (f (w, x _ {i}) - y _ {i}) \nabla^ {2} f (w, x _ {i}).
$$
$$
\left\| \nabla^ {2} \ell_ {i} (w) \right\| _ {\mathrm {o p}} \leq \left\| \nabla f (w, x _ {i}) \right\| ^ {2} + \left| f (w, x _ {i}) - y _ {i} \right| \cdot \left\| \nabla^ {2} f (w, x _ {i}) \right\| _ {\mathrm {o p}}. \tag {A.15}
$$
Setting  $\bar{w}_0\in \mathrm{proj}_S(w_0)\cap B_r(w_0)$ , observe that
$$
\| \nabla f (w, x _ {i}) \| = \| \nabla f (w, x _ {i}) - \nabla f (\bar {w} _ {0}, x _ {i}) \| \leq 2 r \sup  _ {v \in B _ {r} (w _ {0})} \| \nabla^ {2} f (v, x _ {i}) \| _ {\mathrm {o p}} = \tilde {O} \left(\frac {r ^ {3 L + 1}}{\sqrt {m}}\right),
$$
where the last inequality follows from transition to linearity (3.1). Thus  $f(\cdot, x_i)$  is Lipschitz continuous on the ball  $B_r(w_0)$ . Therefore, we may also bound
$$
\left| f (w, x _ {i}) - y _ {i} \right| = \left| f (w, x _ {i}) - f (\bar {w}, x _ {i}) \right| = \tilde {O} \left(\frac {r ^ {3 L + 2}}{\sqrt {m}}\right)
$$
Returning to (A.15), we thus have  $\| \nabla^2\ell_i(w)\|_{\mathrm{op}} = \tilde{O}\left(\frac{r^{6L + 2}}{m}\right)$ . Taking into account that  $m = \tilde{\Omega}\left(\frac{nr^{6l + 2}}{\lambda_0^2}\right)$  we deduce  $\frac{r^{6L + 2}}{m} = O(1)$  thereby completing the proof.
A.8 Proof of Corollary 3.4
The goal is to apply Theorem 2.5. To this end, we begin by applying Theorem 3.3. We may then be sure that with high probability Assumptions 1 and 2 hold on a ball  $B_r(w_0)$  with  $\alpha = \lambda_0 / 2$ ,  $\theta = 1$ , and  $\rho = \tilde{O}\left(\frac{r^{3l + 2}}{\sqrt{m}}\right)$ . We may then choose  $\eta = \frac{1}{2\beta} = \Theta(1)$ . In order to make sure that  $\rho \leq (\theta - \beta \eta)\alpha r$ , it suffices to be in the regime  $m = \Omega\left(\frac{r^{6l + 2}}{\lambda_0^2}\right)$ . Finally it remains to ensure that  $\mathrm{dist}^2(w_0, S) \leq \delta_1^2 r^2$ . To do so, using quadratic growth, we have  $\mathrm{dist}^2(w_0, S) \leq \frac{4}{\lambda_0} \mathcal{L}(w_0) = O\left(\frac{1}{\lambda_0}\right)$ . Thus it suffices to let  $r = \Omega\left(\frac{1}{\delta_1 \sqrt{\lambda_0}}\right)$ . An application of Theorem 2.5 completes the proof.
A.9 Proof of Lemma 4.2
Define the set  $\mathcal{J} = \{i\in [k]:f(w_i)\leq \epsilon \}$ . Then Hoeffding inequality ensures that the inequality  $|\mathcal{J}|\geq \frac{k}{4}$  holds with probability at least  $1 - \exp (-k / 16)$ . Conditioned on this event, consider any index  $i\in [k]$  satisfying  $\mathcal{L}(w_i) > \frac{\lambda\epsilon}{c_1}$ . Then from (3), we know that with probability at least  $1 - \exp (-c_{2}m)$ , we have
$$
\frac {1}{m} \sum_ {i = 1} ^ {m} \ell \left(w _ {i}, z _ {i}\right) \geq c _ {1} \mathcal {L} \left(w _ {i}\right) > \lambda \epsilon ,
$$
and therefore  $i \notin \mathcal{I}$ . Let us now estimate the probability that  $\mathcal{I}$  is nonempty conditioned on  $|\mathcal{I}| \geq \frac{k}{4}$ . To this end, by Markov's inequality for any  $i \in \mathcal{I}$ , we have
$$
P \left(\frac {1}{m} \sum_ {j = 1} ^ {m} \ell (w, z _ {j}) > \lambda \epsilon\right) \leq P \left(\frac {1}{m} \sum_ {j = 1} ^ {m} \ell (w _ {i}, z _ {j}) \geq \lambda \mathcal {L} (w _ {i})\right) \leq \frac {\mathcal {L} (w _ {i})}{\lambda f (w _ {i})} = \frac {1}{\lambda}.
$$
Consequently, the probability that the set  $\mathcal{I}$  is empty is at most  $\lambda^{-k / 4}$ .
B Auxiliary results on stopping times
Theorem B.1 (Stopping time argument). Let  $\{U_t\}_{t \geq 0}$  be a sequence of nonnegative random variables and define the stopping time  $\tau = \inf \{t \geq 0 : U_t > u\}$  for some constant  $u > 0$ . Suppose that
$$
\mathbb {E} \left[ U _ {t + 1} 1 _ {\tau > t} \mid U _ {1: t} \right] \leq U _ {t} 1 _ {\tau > t} + \zeta_ {t} \quad \forall t \geq 0. \tag {B.1}
$$
Then the estimate  $\mathbb{P}[\tau \leq t] \leq \frac{\mathbb{E}U_0 + \sum_{i=0}^{t-1}\zeta_i}{u}$  holds for all  $t \geq 1$ .
Proof. Observe that by Markov's inequality, we have
$$
\mathbb {P} [ \tau \leq t ] = \mathbb {P} [ U _ {t \wedge \tau} > u ] \leq \frac {\mathbb {E} [ U _ {t \wedge \tau} ]}{u}.
$$
Let us therefore bound the expectation of the stopped random variable  $V_{t} \coloneqq U_{t\wedge \tau}$ . Letting  $\mathbb{E}_t[\cdot ]$  denote the conditional expectation  $\mathbb{E}[\cdot \mid U_{1:t}]$ , we successively compute
$$
\begin{array}{l} \mathbb {E} _ {t} \left[ V _ {t + 1} \right] = \mathbb {E} _ {t} \left[ U _ {t + 1 \wedge \tau} \right] \\ = \mathbb {E} _ {t} \left[ U _ {t + 1 \wedge \tau} 1 _ {\tau > t} \right] + \mathbb {E} _ {t} \left[ U _ {t + 1 \wedge \tau} 1 _ {\tau \leq t} \right] \\ = \mathbb {E} _ {t} \left[ U _ {t + 1} 1 _ {\tau > t} \right] + \mathbb {E} _ {t} \left[ U _ {t \wedge \tau} 1 _ {\tau \leq t} \right] \\ = \mathbb {E} _ {t} \left[ U _ {t + 1} 1 _ {\tau > t} \right] + U _ {t \wedge \tau} 1 _ {\tau \leq t} \\ \leq U _ {t} 1 _ {\tau > t} + U _ {t \wedge \tau} 1 _ {\tau \leq t} + \zeta_ {t} \\ = V _ {t} + \zeta_ {t}. \\ \end{array}
$$
Taking the expectation with respect to  $U_{1:t}$ , applying the tower rule, and iterating the recursion we deduce  $\mathbb{E}[V_t] \leq \mathbb{E}[U_0] + \sum_{i=0}^{t-1} \zeta_i$ , thereby completing the proof.
Theorem B.2 (Stopping time with contractions). Let  $\{U_t\}_{t \geq 0}$  be a sequence of nonnegative random variables and define the stopping time  $\tau = \inf \{t \geq 0 : U_t > u\}$  for some constant  $u > 0$ . Suppose that there exists  $q \in (0,1)$  such that
$$
\mathbb {E} \left[ U _ {t + 1} 1 _ {\tau > t} \mid U _ {1: t} \right] \leq q \cdot U _ {t} 1 _ {\tau > t} \quad \forall t \geq 0. \tag {B.2}
$$
Then as long as  $\mathbb{E}U_0\leq \delta_1u$  , the event  $\{\tau = \infty \}$  occurs with probability at least  $1 - \delta_{1}$  . Moreover, with probability at least  $1 - \delta_{1} - \delta_{2}$  , the estimate  $U_{t}\leq \varepsilon U_{0}$  holds after  $t\geq \frac{1}{1 - q}\log \left(\frac{1}{\delta_2\varepsilon}\right)$  iterations.
Proof. Define the stopping time  $\tau = \inf \{t\geq 1:U_t > u\}$ . An application of Lemma B.1 therefore implies  $\mathbb{P}[\tau < t]\leq \frac{\mathbb{E}U_0}{u}\leq \delta_1$ . Taking the limit as  $t\to \infty$ , we deduce that the event  $\{\tau = \infty\}$  occurs with probability at least  $1 - \delta_{1}$ . Next taking the expectation with respect to  $U_{1:t}$  in (B.2) and applying the tower rule gives
$$
\mathbb {E} \left[ U _ {t + 1} 1 _ {\tau > t} \right] \leq q \cdot \mathbb {E} \left[ U _ {t} 1 _ {\tau > t} \right].
$$
Taking into account that  $1_{\tau > t} \geq 1_{\tau > t + 1}$  we may iterate the recursion thereby yielding
$$
\mathbb {E} \left[ U _ {t} 1 _ {\tau > t} \right] \leq q ^ {t} U _ {0}.
$$
Now, setting  $\varepsilon^{\prime} \coloneqq \varepsilon U_{0}$ , Markov's inequality yields
$$
\mathbb {P} [ U _ {t} 1 _ {\tau > t} \geq \varepsilon^ {\prime} ] \leq \frac {\mathbb {E} [ U _ {t} 1 _ {\tau > t} ]}{\varepsilon^ {\prime}} \leq \frac {q ^ {t} U _ {0}}{\varepsilon^ {\prime}} \leq \delta_ {2}.
$$
Finally observe
$$
\mathbb {P} [ U _ {t} \geq \varepsilon^ {\prime} \mid \tau = \infty ] = \frac {\mathbb {P} (U _ {t} \geq \varepsilon^ {\prime} , \tau = \infty)}{\mathbb {P} [ \tau = \infty ]} \leq \frac {\mathbb {P} (U _ {t} 1 _ {\tau > t} \geq \varepsilon^ {\prime})}{\mathbb {P} [ \tau = \infty ]}.
$$
Therefore we deduce
$$
\begin{array}{l} \mathbb {P} \left[ U _ {t} <   \varepsilon^ {\prime} \right] \geq \mathbb {P} \left[ U _ {t} <   \varepsilon^ {\prime} \mid \tau = \infty \right] \cdot \mathbb {P} [ \tau = \infty ] \\ = (1 - \mathbb {P} [ U _ {t} \geq \varepsilon^ {\prime} \mid \tau = \infty ]) \cdot \mathbb {P} [ \tau = \infty ] \\ \geq \mathbb {P} [ \tau = \infty ] - \mathbb {P} [ U _ {t} \geq \varepsilon^ {\prime} \mid \tau = \infty ] \cdot \mathbb {P} [ \tau = \infty ] \\ \geq \mathbb {P} [ \tau = \infty ] - \mathbb {P} (U _ {t} 1 _ {\tau > t} \geq \varepsilon^ {\prime}) \\ \geq 1 - \delta_ {1} - \delta_ {2}, \\ \end{array}
$$
C A bad example
Lemma C.1. Consider the objective  $\mathcal{L}(x,y) = \frac{1}{2} (y - ax^2)^2$  for any  $a > 0$ . Then  $\mathcal{L}$  is  $C^\infty$  near  $(0,0)$  and the function  $\mathcal{L}$  satisfies the PL inequality (PL) with constant  $\alpha = 1$ . Therefore  $\mathcal{L}$  satisfies the aiming condition on some neighborhood of the origin. However, for any neighborhood  $U$  of the origin, the function  $\mathcal{L}$  is not quasar-convex on  $U$  relative to any point  $(x,ax^2)\in U$ .
Proof. To see the validity of the PL-condition, observe that
$$
\frac {\| \nabla \mathcal {L} (x , y) \|}{2 \sqrt {\mathcal {L} (x , y)}} = \| \nabla \sqrt {\mathcal {L} (x , y)} \| = \frac {1}{\sqrt {2}} \| \nabla | y - a x ^ {2} | \| \geq \frac {1}{\sqrt {2}}.
$$
It follows immediately from Lemma 2.2 that  $\mathcal{L}$  satisfies the aiming condition (2.2) on some neighborhood of the origin. Next we verify the failure of quasar convexity. Consider an arbitrary point  $(x, ax^2)$  on the parabola. For any  $(u, v) \in \mathbf{R}^2$ , we compute
$$
\begin{array}{l} \langle \nabla \mathcal {L} (u, v), (u, v) - (x, a x ^ {2}) \rangle = \langle (- 2 a u, 1), (u - x, v - a x ^ {2}) \rangle \cdot (v - a u ^ {2}) \\ = (- 2 a u (u - x) + v - a x ^ {2}) (v - a u ^ {2}). \\ \end{array}
$$
In particular, setting  $z_{\gamma} = (0,\gamma)$  we obtain
$$
\langle \nabla \mathcal {L} (z _ {\gamma}), z _ {\gamma} - (x, a x ^ {2}) \rangle = \gamma^ {2} - \gamma a x ^ {2}.
$$
Note the right hand side is negative for any  $0 < \gamma < ax^2$ . Thus, letting  $\gamma > 0$  tend to zero, we deduce that  $\mathcal{L}$  is not quasar-convex relative to  $(x, ax^2) \in U$  with  $x \neq 0$ . Let us consider now the setting  $x = 0$ . Then we compute
$$
\langle \nabla \mathcal {L} (u, v), (u, v) - (0, 0) \rangle = (v - 2 a u ^ {2}) (v - a u ^ {2}).
$$
The right side is negative if  $au^2 < v < 2au^2$  and therefore  $\nabla \mathcal{L}$  is not quasar-convex relative to  $(0,0)$  on any neighborhood of  $(0,0)$ .
Figure 4: Verifying aiming condition. Left: MNIST on fully-connected neural network; Right: CIFAR-10 on CNN.
D Numerical verification of aiming condition on neural networks
In principle, the aiming condition is difficult to verify, as it involves  $proj_1(w)$  – the nearest point of  $w$  to the solution set  $S$ , which we have no way of computing. Instead, we replace  $proj_1(w)$  by the last iterate  $\bar{w}$  of an SGD run, and compute an estimate of the aiming condition coefficient by the quotients  $\hat{\theta}_t \triangleq \langle \nabla L(w_t), w_t - \bar{w} \rangle / L(w_t)$  along the iterate path. Note that this quotient is not the same quotient as would appear in the definition of quasar-convexity because  $\bar{w}$  is a random point that depends on the iterate path taken by SGD.