SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$w_0 \in S_{\sqrt{\delta_1} r}$ . Then with probability at least  $1 - \delta_1$ , the SGD iterates  $\{w_t\}_{t \geq 0}$  remain in  $\mathcal{W}$ . Moreover, with probability at least  $1 - \delta_1 - \delta_2$ , the estimate  $\mathrm{dist}^2(w_t, S) \leq \varepsilon \cdot \mathrm{dist}^2(w_0, S)$  holds after
$$
t \geq \frac {1}{\alpha \eta (\theta - \beta \eta)} \log \left(\frac {1}{\delta_ {2} \varepsilon}\right) \quad i t e r a t i o n s.
$$
Thus as long SGD is initialized at a point  $w_0 \in S_{\sqrt{\delta_1} r}$ , with probability at least  $1 - \delta_1 - \delta_2$ , the iterates remain in  $S_r$  and converge at linear rate  $O\left(\frac{1}{\delta_2}\exp (-t\theta^2\alpha /\beta)\right)$ . Note that the dependence on  $\delta_2$  is logarithmic, while the dependence on  $\delta_1$  appears linearly in the initialization requirement  $w_0 \in S_{\sqrt{\delta_1} r}$ . One simple way to remove the dependence on  $\delta_1$  is to simply rerun the algorithm from the same initial point logarithmically many times and return the point with the smallest function value. An alternative strategy that bypasses evaluating function values will be discussed in Section 4.
2.2 SGD under regularity on a ball  $B_{r}(w_{0})$
Next, we describe convergence guarantees for SGD when Assumption 1 holds on a ball  $\mathcal{W} = B_r(w_0)$ . The key complication is the following. While  $w_{t}$  are in the ball, the distance  $\mathrm{dist}^2 (w_t,S)$  shrinks in expectation. However, the iterates may in principle quickly escape the ball  $B_{r}(w_{0})$ , after which point we lose control on their progress. Thus we must lower bound the probability that the iterates  $w_{t}$  remain in the ball. To this end, we will require the following additional assumption.
Assumption 2 (Uniform aiming). The estimate
$$
\langle \nabla \mathcal {L} (w), w - v \rangle \geq \theta \mathcal {L} (w) - \rho \cdot \operatorname {d i s t} (w, S) \tag {2.4}
$$
holds for all  $w \in B_r(w_0)$  and  $v \in B_r(w_0) \cap S$ .
The intuition underlying this assumption is as follows. We would like to replace  $\bar{w}$  in the aiming condition (2.2) by an arbitrary point  $v\in B_r(w_0)\cap S$ , thereby having a condition of the form  $\langle \nabla \mathcal{L}(w),w - v\rangle \geq \theta \cdot \mathcal{L}(w)$ . The difficulty is that this condition may not be true for the main problem we are interested in—training wide neural networks. Instead, it suffices to lower bound the inner product by  $\theta \mathcal{L}(w) - \rho \cdot \mathrm{dist}(w,S)$  where  $\rho$  is a small constant. This weak condition provably holds for wide neural networks, as we will see in the next section. The following is our main result.
Theorem 2.5 (Convergence on a ball). Suppose that Assumptions 1 and 2 hold on a ball  $\mathcal{W} = B_{3r}(w_0)$ . Fix constants  $\delta_1 \in (0, \frac{1}{3})$  and  $\delta_2 \in (0, 1)$ , and assume  $\mathrm{dist}^2(w_0, S) \leq \delta_1^2 r^2$ . Fix a stepsize  $\eta < \frac{\theta}{\beta}$  and suppose  $\rho \leq (\theta - \beta \eta) \alpha r$ . Then with probability at least  $1 - 5\delta_1$ , all the SGD iterates  $\{w_t\}_{t \geq 0}$  remain in  $B_r(w_0)$ . Moreover, with probability at least  $1 - 5\delta_1 - \delta_2$ , the estimate  $\mathrm{dist}^2(w_t, S) \leq \varepsilon \cdot \mathrm{dist}^2(w_0, S)$  holds after
$$
t \geq \frac {1}{\alpha \eta (\theta - \beta \eta)} \log \left(\frac {1}{\varepsilon \delta_ {2}}\right) \quad i t e r a t i o n s.
$$
Thus as long as  $\rho$  is sufficiently small and the initial distance satisfies  $\mathrm{dist}^2 (w_0,S)\leq \delta_1^2 r^2$ , with probability at least  $1 - 5\delta_{1} - \delta_{2}$ , the iterates remain in  $B_{r}(w_{0})$  and converge at a fast linear rate  $O\left(\frac{1}{\delta_2}\exp (-t\theta^2\alpha /\beta)\right)$ . While the dependence on  $\delta_{2}$  is logarithmic, the constant  $\delta_{1}$  linearly impacts the initialization region. Section 4 discusses a way to remove this dependence. As explained in Lemma 2.1, both quadratic growth and the initialization quality holds if  $\mathcal{L}$  is  $\alpha$ -PL* on the ball  $B_{r}(w_{0})$ , and  $r$  is sufficiently big relative to  $1 / \alpha$ .
3 Consequences for nonlinear least squares and wide neural networks
We next discuss the consequences of the results in the previous sections to nonlinear least squares and training of wide neural networks. To this end, we begin by verifying the aiming (2.2) and uniform aiming (2.4) conditions for nonlinear least squares. The key assumption we will make is that the nonlinear map's Jacobian  $\nabla F$  has a small Lipschitz constant in operator norm.
Theorem 3.1. Consider a function  $\mathcal{L}(w) = \frac{1}{2}\| F(w)\|^2$ , where  $F\colon \mathbf{R}^d\to \mathbf{R}^n$  is  $C^1$ -smooth. Suppose that there is a point  $w_0$  satisfying  $\mathrm{dist}(w_0,S)\leq r$  and such that on the ball  $B_{2r}(w_0)$ , the gradient  $\nabla \mathcal{L}$  is  $\beta$ -Lipschitz, the Jacobian  $\nabla F$  is  $L$ -Lipschitz in the operator norm, and the quadratic growth condition (2.1) holds. Then as long as  $L\leq \frac{2\alpha}{r\sqrt{\beta}}$ , the aiming (2.2) and uniform aiming (2.4) conditions hold on  $B_r(w_0)$  with  $\theta = 2 - \frac{rL\sqrt{\beta}}{\alpha}$  and  $\rho = 8r^{2}L\sqrt{\beta}$ .
We next instantiate Theorem 3.1 and Theorem 2.5 for a nonlinear least squares problem arising from fitting a wide neural network. Setting the stage, an  $l$ -layer (feedforward) neural network  $f(w;x)$ , with parameters  $w$ , input  $x$ , and linear output layer is defined as follows:
$$
\alpha^ {(0)} = x,
$$
$$
\alpha^ {(i)} = \sigma \left(\frac {1}{\sqrt {m _ {i - 1}}} W ^ {(i)} \alpha^ {(i - 1)}\right), \forall i = 1, \dots , l - 1
$$
$$
f (w; x) = \frac {1}{\sqrt {m _ {l - 1}}} W ^ {(l)} \alpha^ {(l - 1)}.
$$
Here,  $m_{i}$  is the width (i.e., number of neurons) of  $i$ -th layer,  $\alpha^{(i)} \in \mathbf{R}^{m_i}$  denotes the vector of  $i$ -th hidden layer neurons,  $w := \{W^{(1)}, W^{(2)}, \ldots, W^{(l)}, W^{(l+1)}\}$  denotes the collection of the parameters (or weights)  $W^{(i)} \in \mathbb{R}^{m_i \times m_{i-1}}$  of each layer, and  $\sigma$  is the activation function, e.g., sigmoid, tanh, linear activation. We also denote the width of the neural network as  $m := \min_{i \in [l]} m_i$ , i.e., the minimal width of the hidden layers. The neural network is usually randomly initialized, i.e., each individual parameter is initialized i.i.d. following  $\mathcal{N}(0, 1)$ . Henceforth, we assume that the activation functions  $\sigma$  are twice differentiable,  $L_{\sigma}$ -Lipschitz, and  $\beta_{\sigma}$ -smooth. In what follows, the order notation  $\Omega(\cdot)$  and  $O(\cdot)$  will suppress multiplicative factors of polynomials (up to degree  $l$ ) of the constants  $C$ ,  $L_{\sigma}$  and  $\beta_{\sigma}$ .
Given a dataset  $\mathcal{D} = \{(x_i, y_i)\}_{i=1}^n$ , we fit the neural network by solving the least squares problem
$$
\min  _ {w} \mathcal {L} (w) \triangleq \frac {1}{2} \| F (w) \| ^ {2} \quad \text {w h e r e} \quad \frac {1}{2} \| F (w) \| ^ {2} = \frac {1}{n} \sum_ {i = 1} ^ {n} \left(f \left(w, x _ {i}\right) - y _ {i}\right) ^ {2}.
$$
We assume that all the data inputs  $x_{i}$  are bounded, i.e.,  $\| x_{i}\| \leq C$  for some constant  $C$ .
Our immediate goal is to verify the assumptions of Theorem 3.1, which are quadratic growth and (uniform) aiming. We begin with the former. Quadratic growth is a consequence of the PL-condition. Namely, define the Neural Tangent Kernel  $K(w_0) = \nabla F(w_0)\nabla F(w_0)^\top$  at the random initial point  $w_0\sim N(0,I)$  and let  $\lambda_0$  be the minimal eigenvalue of  $K(w_{0})$ . The value  $\lambda_0$  has been shown to be positive with high probability in [10, 8]. Specifically, it was shown that, under a mild non-degeneracy condition on the data set, the smallest eigenvalue  $\lambda_{\infty}$  of NTK of an infinitely wide neural network is positive (see Theorem 3.1 of [10]). Moreover, if the network width satisfies  $m = \Omega \left(\frac{n^2\cdot 2^{O(l)}}{\lambda_\infty^2}\log \frac{nl}{\epsilon}\right)$ , then with probability at least  $1 - \epsilon$  the estimate  $\lambda_0 > \frac{\lambda_\infty}{2}$  holds [8, Remark E.7]. Of course, this is worst case bound and for our purposes we will only need to ensure that  $\lambda_0$  is positive. It will also be important to know that  $\| F(w_0)\| ^2 = O(1)$ , which indeed occurs with high probability as shown in [18]. To simplify notation, let us lump these two probabilities together and define
$$
p \triangleq \mathbb {P} \left\{\lambda_ {0} > 0, \| F (w _ {0}) \| ^ {2} \leq C \right\}.
$$
Next, we require the following theorem, which shows two fundamental properties on  $B_r(w_0)$  when the width  $m$  is sufficiently large: (1) the function  $w \mapsto f(w, x)$  is nearly linear and (2) the function  $\mathcal{L}$  satisfies the PL condition with parameter  $\lambda_0 / 2$ .
Theorem 3.2 (Transition to linearity [25] and the PL condition [24]). Given any radius  $r > 0$ , with probability  $1 - p - 2\exp \left(-\frac{ml}{2}\right) - \left(1 / m\right)^{\Theta (\ln m)}$  of initialization  $w_{0} \sim N(0,I)$ , it holds:
$$
\left\| \nabla^ {2} f (w, x) \right\| _ {\mathrm {o p}} = \tilde {O} \left(\frac {r ^ {3 l}}{\sqrt {m}}\right) \quad \forall w \in B _ {r} \left(w _ {0}\right), \| x \| \leq C. \tag {3.1}
$$
In the same event, as long as the width of the network satisfies  $m = \tilde{\Omega}\left(\frac{nr^{6l + 2}}{\lambda_0^2}\right)$ , the function  $\mathcal{L}$  is  $PL^{*}$  on  $B_{r}(w_{0})$  with parameter  $\lambda_0 / 2$ .
Note that (3.1) directly implies that the Lipschitz constant of  $\nabla F$  is bounded by  $\tilde{O}\left(\frac{r^{3l}}{\sqrt[m]{m}}\right)$  on  $B_r(w_0)$ , and can therefore be made arbitrarily small. Quadratic growth is now a direct consequence of the  $\mathsf{PL}^*$  condition while (uniform) aiming follows from an application of Theorem 3.1.
Theorem 3.3 (Aiming and quadratic growth condition for wide neural network). With probability at least  $1 - p - 2\exp \left(-\frac{ml}{2}\right) - (1 / m)^{\Theta (\ln m)}$  with respect to the initialization  $w_{0}\sim N(0,I)$ , as long as
$$
m = \tilde {\Omega} \left(\frac {n r ^ {6 l + 2}}{\lambda_ {0} ^ {2}}\right) \quad a n d \quad r = \Omega \left(\frac {1}{\sqrt {\lambda_ {0}}}\right),
$$
the following are true:
1. the quadratic growth condition (2.1) holds on  $B_r(w_0)$  with parameter  $\lambda_0 / 2$  and the intersection  $B_r(w_0) \cap S$  is nonempty,
2. aiming (2.2) and uniform aiming (2.4) conditions hold in  $B_r(w_0)$  with  $\theta = 1$  and  $\rho = \tilde{O}\left(\frac{r^{3l + 2}}{\sqrt{m}}\right)$ ,
3. the gradient of each function  $\ell_i(w) \triangleq (f(w, x_i) - y_i)^2$  is  $\beta$ -Lipschitz on  $B_r(w_0)$  with  $\beta = O(1)$ .
Please see Appendix D for a numerical verification of an estimate of the aiming condition. It remains to deduce convergence guarantees for SGD by applying Theorem 2.5.
Corollary 3.4 (Convergence of SGD for wide neural network). Fix constants  $\delta_1 \in (0, \frac{1}{3})$ ,  $\delta_2 \in (0, 1)$ ,  $\varepsilon > 0$  and  $t \in \mathbb{N}$ . There is a stepsize  $\eta = \Theta(1)$  such that the following is true. With probability at least  $1 - p - \delta_1 - \delta_2 - 2\exp\left(-\frac{ml}{2}\right) - (1/m)^{\Theta(\ln m)}$ , as long as
$$
m = \tilde {\Omega} \left(\frac {n r ^ {6 l + 2}}{\lambda_ {0} ^ {2}}\right) \qquad a n d \qquad r = \Omega \left(\frac {1}{\delta_ {1} \sqrt {\lambda_ {0}}}\right),
$$
all the SGD iterates  $\{w_{t}\}_{t\geq 0}$  remain in  $B_{r}(w_{0})$  and the estimate  $\mathrm{dist}^2 (w_t,S)\leq \varepsilon \cdot \mathrm{dist}^2 (w_0,S)$  holds after  $t\geq \frac{1}{\lambda_0}\log \left(\frac{1}{\varepsilon\delta_2}\right)$  iterations.
Thus, the width requirements for SGD to converge at a fast linear rate are nearly identical to those for gradient descent [24], with the exception being that the requirement  $r = \Omega \left( \frac{1}{\sqrt{\lambda_0}} \right)$  is strengthened to  $r = \Omega \left( \frac{1}{\delta_1 \sqrt{\lambda_0}} \right)$ . That is, the radius  $r$  needs to shrink by the probability of failure.
4 Boosting to high probability
A possible unsatisfying feature of Theorems 2.4 and 2.5 and Corollary 3.4 is that the size of the initialization region shrinks with the probability of failure  $\delta_{1}$ . A natural question is whether this requirement may be dropped. Indeed, we will now see how to boost the probability of success to be independent of  $\delta_{1}$ . A first reasonable idea is to simply rerun SGD a few times from an initialization region corresponding to  $\delta_{1} = 1/2$ . Then by Hoeffding's inequality, after very trials, at least a third of them will be successful. The difficulty is to determine which trial was indeed successful. The fact that the solution set is not a singleton rules out strategies based on the geometric median of means [31, p. 243], [29]. Instead, we may try to estimate the function value at each of the returned points. In a classical setting of stochastic optimization, this is a very bad idea because it amounts to mean estimation, which in turn requires  $O(1/\varepsilon^{2})$  samples. The saving grace in the interpolation regime is that  $\ell(w, \cdot)$  is a nonnegative function of the samples. While estimating the mean of nonnegative random variables still requires  $O(1/\varepsilon^{2})$  samples, detecting that a nonnegative random variable is large requires very few samples! This basic idea is often called the small ball principle and is the basis for establishing generalization bounds with heavy tailed data [28]. With this in mind, we will require the following mild condition, stipulating that the empirical average  $\frac{1}{m} \sum_{i=1}^{m} \ell(w, z_i)$  to be lower bounded by  $\mathcal{L}(w)$  with high probability over the iid samples  $z_i$ .
Assumption 3 (Detecting large values). Suppose that there exist constants  $c_{1} > 0$  and  $c_{2} > 0$  such that for any  $w \in \mathbf{R}^{d}$ , integer  $m \in \mathbb{N}$ , and iid samples  $z_{1}, \ldots, z_{m} \sim \mathcal{P}$ , the estimate holds:
$$
P \left(\frac {1}{m} \sum_ {i = 1} ^ {m} \ell (w, z _ {i}) \geq c _ {1} \mathcal {L} (w)\right) \geq 1 - \exp (- c _ {2} m).
$$
Importantly, this condition does not have anything to do with light tails. A standard sufficient condition for Assumption 3 is a small ball property.
Assumption 4 (Small ball). There exist constants  $\tau >0$  and  $p\in (0,1)$  satisfying
$$
\mathbb {P} \left(\ell (w, z) \geq \tau \cdot \mathcal {L} (w)\right) \geq p \quad \forall w \in \mathbf {R} ^ {d}.
$$
The small ball property simply asserts that  $\ell (w,\cdot)$  should not put too much mess on small values relative to its mean  $\mathcal{L}(w)$ . Bernstein's inequality directly shows that Assumption 4 implies Assumption 3. We summarize this observation in the following theorem.
Lemma 4.1. Assumption 4 implies Assumption 3 with  $c_{1} = \frac{p\tau}{2}$  and  $c_{2} = \frac{p}{4}$ .
A valid bound for the small ball probabilities is furnished by the Paley-Zygmund inequality [33]:
$$
\mathbb {P} \big (\ell (w, z) \geq \tau \mathcal {L} (w) \big) \geq (1 - \tau) ^ {2} \frac {\mathcal {L} (w) ^ {2}}{[ \mathbb {E} \ell (w , z) ^ {2} ]} \quad \forall \tau \in [ 0, 1 ].
$$
Thus if the ratio  $\frac{\mathbb{E}\ell(w,z)^2}{\mathcal{L}(w)}$  is bounded by some  $D > 0$ , then the small ball condition holds with  $p = (1 - \tau)^2 / D$  where  $\tau \in [0,1]$  is arbitrary.
The following lemma shows that under Assumption 3, we may turn any estimation procedure for finding a minimizer of  $\mathcal{L}$  that succeeds with constant probability into one that succeeds with high probability. The procedure simply draws a small batch of samples  $z_{1},\ldots ,z_{m}\sim \mathcal{P}$  and rejects those trial points  $w_{i}$  for which the empirical average  $\frac{1}{m}\sum_{j = 1}^{m}\ell (w_i,z_j)$  is too high.
Lemma 4.2 (Rejection sampling). Let  $w_{1},\ldots ,w_{k}$  be independent random variables satisfying  $\mathbb{P}(\mathcal{L}(w_i)\leq \epsilon)\geq 1 / 2$ . For each  $i = 1,\dots ,k$  draw  $m$  samples  $z_{1},\ldots ,z_{m}\stackrel {i.i.d.}{\sim}\mathcal{P}$ . For any  $\lambda >1$ , define admissible indices  $\mathcal{I} = \left\{i\in [k]:\frac{1}{m}\sum_{j = 1}^{m}\ell (w_{i},z_{j})\leq \lambda \epsilon \right\}$ . Then with probability  $1 - \exp (-\frac{k}{16}) - k\exp (-c_2m) - \lambda^{-k / 4}$ , the set  $\mathcal{I}$  is nonempty and  $\mathcal{L}(w_i)\leq \frac{\lambda\epsilon}{c_1}$  for any  $i\in \mathcal{I}$ .
We may now simply combine SGD with rejection sampling to obtain high probability guarantees. Looking at Lemma 4.2, some thought shows that the overhead for high probability guarantees is dominated by  $c_2^{-1}$ . As we saw from the Paley-Zygmond inequality, we always have  $c_2^{-1} \lesssim D$  where  $D$  upper bounds the ratios  $\frac{\mathbb{E}\ell(w,z)^2}{\mathcal{L}(w)}$ . It remains an interesting open question to investigate the scaling of small ball probabilities for overparametrized neural networks.
Existing results ensuring convergence of SGD under interpolation and the PL condition require the method to use a small stepsize, and therefore converge slowly. In this work we isolated conditions that enable SGD to take a large stepsize and therefore have similar iteration complexity as gradient descent. Consequently, our results align theory better with practice, where large stepsizes are routinely used. Moreover, we argued that these conditions are reasonable because they provably hold when training sufficiently wide feedforward neural networks with a linear output layer.
The work of Dmitriy Drusvyatskiy was supported by the NSF DMS 1651851 and CCF 1740551 awards. The work of Damek Davis is supported by an Alfred P. Sloan research fellowship and NSF DMS award 2047637. Yian Ma is supported by the NSF SCALE MoDL-2134209 and the CCF-2112665 (TILOS) awards, as well as the U.S. Department of Energy, Office of Science, and the Facebook Research award. Mikhail Belkin acknowledges support from National Science Foundation (NSF) and the Simons Foundation for the Collaboration on the Theoretical Foundations of Deep Learning (https://deepfoundations.ai/) through awards DMS-2031883 and #814639 and the TILOS institute (NSF CCF-2112665). This work used the programs (1) XSEDE (Extreme science and engineering discovery environment) which is supported by NSF grant numbers ACI-1548562, and (2) ACCESS (Advanced cyberinfrastructure coordination ecosystem: services & support) which is supported by NSF grants numbers #2138259, #2138286, #2138307, #2137603, and #2138296. Specifically, we used the resources from SDSC Expanse GPU compute nodes, and NCSA Delta system, via allocations TG-CIS220009.
[1] Hedy Attouch, Jérôme Bolte, and Benar Fux Svaiter. "Convergence of descent methods for semi-algebraic and tame problems: proximal algorithms, forward-backward splitting, and regularized Gauss-Seidel methods". In: Mathematical Programming 137.1-2 (2013), pp. 91-129.
A Missing proofs
A.1 Proof of Theorem 2.2
The proof relies on the following elementary lemma.
Lemma A.1 (Aiming for smooth functions). Let  $\mathcal{L}\colon \mathbf{R}^d\to \mathbf{R}$  be a differentiable function. Fix two points  $w,\bar{w}\in \mathbf{R}^{d}$  satisfying  $\nabla \mathcal{L}(\bar{w}) = 0$  and  $\mathcal{L}(\bar{w}) = 0$ . Suppose that the Hessian  $\nabla^2\mathcal{L}$  exists and is  $L$ -Lipschitz continuous on the segment  $[w,\bar{w} ]$ . Then we have
$$
\left| \mathcal {L} (w) + \frac {1}{2} \langle \nabla \mathcal {L} (w), \bar {w} - w \rangle \right| \leq \frac {5 L}{1 2} \| w - \bar {w} \| ^ {3}.
$$
Proof. Define the function  $g(t) = \mathcal{L}(w + t(\bar{w} - w))$ . The theorem is evidently equivalent to
$$
\left| g (0) + \frac {1}{2} g ^ {\prime} (0) - g (1) \right| \leq \frac {5 L}{1 2} \| w - \bar {w} \| ^ {3}.
$$
In order to establish this estimate, we first note that  $g''$  is Lipschitz continuous with constant  $\hat{L} \coloneqq L\| w - \bar{w}\|^3$ , as follows from a quick computation. Taylor's theorem with remainder applied to  $g$  and  $g'$ , respectively, then gives
$$
g (1) = g (0) + g ^ {\prime} (0) + \frac {1}{2} g ^ {\prime \prime} (0) + E _ {1}
$$
$$
0 = g ^ {\prime} (1) = g ^ {\prime} (0) + g ^ {\prime \prime} (0) + E _ {2},
$$
where  $|E_1| \leq \frac{\hat{L}}{6}$  and  $|E_2| \leq \frac{\hat{L}}{2}$ . Combining the two estimates yields
$$
\begin{array}{l} g (1) = g (0) + \frac {1}{2} g ^ {\prime} (0) + \frac {1}{2} g ^ {\prime} (0) + \frac {1}{2} g ^ {\prime \prime} (0) + E _ {1} \\ = g (0) + \frac {1}{2} g ^ {\prime} (0) - \frac {1}{2} \left(g ^ {\prime \prime} (0) + E _ {2}\right) + \frac {1}{2} g ^ {\prime \prime} (0) + E _ {1} \\ = g (0) + \frac {1}{2} g ^ {\prime} (0) + E _ {1} - \frac {E _ {2}}{2}. \\ \end{array}
$$
thereby completing the proof.
Turning to the proof of Theorem 2.2, an application of Lemma A.1 guarantees
$$
\langle \nabla \mathcal {L} (w), w - \bar {w} \rangle \geq 2 \mathcal {L} (w) - \frac {5 L r}{6} \| w - \bar {w} \| ^ {2},
$$
for all  $w \in S_r$  and any  $\bar{w} \in \mathrm{proj}_S(w)$ . If the quadratic growth condition (3) is satisfied on the tube  $S_r$ , then we have the upper bound  $\frac{5Lr}{6}\| w - \bar{w}\|^2 \leq \frac{5Lr}{3\alpha}\mathcal{L}(w)$ . Theorem 2.2 follows. It remains to prove Lemma A.1.
A.2 Proof of Lemma 2.1
In this section, we verify the classical result that the  $\mathrm{PL}^*$  condition on a ball implies quadratic growth. We begin with the following lemma estimating the distance of a single point to sublevel set of a function; this result is a special instance of the descent principle [6, Lemma 2.5], whose roots can be traced back to [17, Basic Lemma, Chapter 1].
Lemma A.2 (Descent principle). Fix a differentiable function  $\mathcal{L}\colon \mathbf{R}^d\to [0,\infty)$  and a ball  $B_{r}(w_{0})$  and define  $S = \{w:\mathcal{L}(w) = 0\}$ . Suppose that  $\mathcal{L}$  satisfies the  $PL^{*}$  condition on  $B_{r}(w_{0})$  with parameter  $\alpha$ . Then as long as  $\mathcal{L}(w_0) < \frac{1}{2} r^2\alpha$ , the intersection  $S\cap B_r(w_0)$  is nonempty and the estimate holds:
$$
\mathcal {L} (w _ {0}) \geq \frac {\alpha}{2} \mathrm {d i s t} ^ {2} (w _ {0}, S).
$$
Proof. Define the function  $f(w) = \sqrt{\mathcal{L}(w)}$  and observe that for any  $w \in B_r(w_0)$  with  $\mathcal{L}(w) > 0$  we have  $\| \nabla f(w) \|^2 = \frac{\|\nabla \mathcal{L}(w)\|^2}{4\mathcal{L}(w)} \geq \frac{\alpha}{2}$ . Therefore an application of the descent principle [6, Lemma 2.5] implies that the set  $S \cap B_r(w_0)$  is nonempty and the estimate  $\mathrm{dist}(w_0, S) \leq \frac{1}{\sqrt{\alpha / 2}} \cdot f(w_0)$  holds. Squaring both sides completes the proof.
We may now complete the proof of Lemma 2.1 by extending from a single point  $w_0$  to a neighborhood of  $w_0$  as follows. First, Lemma A.2 ensures that  $B_r(w_0)$  intersects  $S$  at some point  $\bar{w}_0$  and the inequality  $\mathcal{L}(w_0) \geq \frac{\alpha}{2} \mathrm{dist}^2(w_0, S)$  holds. Fix a point  $w \in B_r(w_0)$ . Then clearly  $\mathcal{L}$  satisfies the  $\mathsf{PL}^*$  condition on  $B_r(w)$  with parameter  $\alpha$ . Let us now consider two cases:  $\mathcal{L}(w) < \frac{1}{2} r^2\alpha$  and  $\mathcal{L}(w) \geq \frac{1}{2} r^2\alpha$ . In the former case, Lemma A.2 implies the claimed estimate  $\mathcal{L}(w) \geq \frac{\alpha}{2} \cdot \mathrm{dist}^2(w, S)$ . In the remaining case  $\mathcal{L}(w) \geq \frac{1}{2} r^2\alpha$ , we compute
$$
\operatorname {d i s t} _ {S} ^ {2} (w) \leq \| w - \bar {w} _ {0} \| ^ {2} \leq 4 r ^ {2} \leq 8 \mathcal {L} (w) / \alpha .
$$
Rearranging completes the proof of Lemma 2.1.
Fix a point  $w \in S_r$  and let  $\bar{w} \in \mathrm{proj}_S(w)$  be a point satisfying the aiming condition (2.2). Observe that Lipschitz continuity of  $\nabla \ell(\cdot, z)$  and interpolation ensures
$$
\| \nabla \ell (w, z) \| = \| \nabla \ell (w, z) - \nabla \ell (\bar {w}, z) \| \leq \beta \cdot \operatorname {d i s t} (w, S) \leq \beta r.
$$
Therefore for every  $\tau \in [0,1 / \beta ]$ , the point  $w^{\tau}\coloneqq w - \tau \nabla \ell (w,z)$  satisfies