SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\left\| g _ {1} (\theta) \left(g _ {2} (\theta) - g _ {2} \left(\theta^ {\prime}\right)\right) \right\| + \left\| \left(g _ {1} (\theta) - g _ {1} \left(\theta^ {\prime}\right)\right) g _ {2} \left(\theta^ {\prime}\right) \right\|.
$$
Applying submultiplicativity of the matrix norm and the bounds provided by the  $b_{i}$  and  $c_{i}$  gives
$$
\left\| g _ {1} g _ {2} (\theta) - g _ {1} g _ {2} \left(\theta^ {\prime}\right) \right\| \leq \left(b _ {1} c _ {2} + b _ {2} c _ {1}\right) \| \theta - \theta^ {\prime} \|.
$$
Now suppose we have the result for  $n = k$ . Writing  $\prod_{i=1}^{k+1} g_i$  as  $g_1 \prod_{i=2}^{k+1} g_i$  and applying the above argument, the induction hypothesis tells us that  $\prod_{i=1}^{k+1} g_i$  is indeed bounded by  $\prod_{i=1}^{k+1} b_i$  and Lipschitz with Lipschitz constant  $\sum_{i=1}^{k+1} c_i \left( \prod_{j \neq i} b_j \right)$ . The result follows.
Proof of Proposition 4.3. By Proposition 4.2, it suffices to show that for each  $1 \leq l \leq L$ , the function
$$
\vec {\theta} \mapsto \prod_ {j = l + 1} ^ {L} J f _ {j} \left(\theta_ {j}, f _ {<   j} (\vec {\theta}, X)\right) D f _ {l} \left(\theta_ {l}, f _ {<   l} (\vec {\theta}, X)\right) \tag {24}
$$
is bounded and Lipschitz on  $S$ . To show this, we must first prove that each map  $\vec{\theta} \mapsto f_{<j}(\vec{\theta}, X)$  is bounded and Lipschitz on  $S$ . This we prove by induction.
By hypothesis,  $\vec{\theta} \mapsto f_1(\vec{\theta}, X) = f_1(\theta_1, X)$  is bounded and Lipschitz on  $S$ . Suppose now that for  $j > 1$ , one has  $\vec{\theta} \mapsto f_{<j}(\vec{\theta}, X)$  bounded and Lipschitz on  $S$ . Then the range of  $S \ni \theta \mapsto f_{<j}(\vec{\theta}, X)$  is a bounded subset of  $\mathbb{R}^{d_j \times N}$ . By hypothesis on  $f_j$ , it then follows that  $\theta \mapsto f_{<j+1}(\vec{\theta}, X) = f_j(\theta_j, f_{<j}(\vec{\theta}, X))$  is bounded and Lipschitz on  $S$ .
The hypothesis on the  $Jf_{j}$  and  $Df_{j}$  now implies that the maps  $\vec{\theta} \mapsto Jf_{j}\big(\theta_{j}, f_{< j}(\vec{\theta}, X)\big), l + 1 \leq j \leq L$ , and  $\vec{\theta} \mapsto Df_{l}\big(\theta_{l}, f_{< l}(\vec{\theta}, X)\big)$  are all bounded and Lipschitz on  $S$ . In particular, as a product of bounded and Lipschitz functions, the map given in Equation (24) is also bounded and Lipschitz on  $S$ . Therefore  $DF$  is bounded and Lipschitz on  $S$ .
Proof of Corollary 4.4. By hypothesis,  $F(S)$  is a bounded subset of  $\mathbb{R}^{d_L\times N}$ . Continuity of  $\gamma$  then implies that  $\gamma (F(S))$  is a bounded subset of  $\mathbb{R}$ , so that  $F(S)$  is contained in a sublevel set of  $\gamma$ . The result now follows from the hypotheses.
To prove Theorem 4.5 it will be convenient to recall some tensor calculus. If  $f: \mathbb{R}^{n_1 \times n_2} \to \mathbb{R}^{m_1 \times m_2}$  is a matrix-valued, differentiable function of a matrix-valued variable, its derivative  $Df$  can be regarded as a map  $\mathbb{R}^{n_1 \times n_2} \to \mathbb{R}^{m_1 \times m_2 \times n_1 \times n_2}$  whose components are given by
$$
D f _ {j _ {1}, j _ {2}} ^ {i _ {1}, i _ {2}} (X) = \frac {\partial f _ {i _ {2}} ^ {i _ {1}}}{\partial x _ {j _ {2}} ^ {j _ {1}}} (X), \qquad X \in \mathbb {R} ^ {n _ {1} \times n _ {2}}
$$
where  $1 \leq i_{\alpha} \leq m_{\alpha}$  and  $1 \leq j_{\alpha} \leq n_{\alpha}$  are the indices,  $\alpha = 1,2$ . It is easily deduced from the chain rule of ordinary calculus that if  $f: \mathbb{R}^{n_1 \times n_2} \to \mathbb{R}^{m_1 \times m_2}$  and  $g: \mathbb{R}^{m_1 \times m_2} \to \mathbb{R}^{l_1 \times l_2}$  are differentiable, then  $g \circ f$  is differentiable with derivative  $(Dg \circ f) \cdot Df: \mathbb{R}^{n_1 \times n_2} \to \mathbb{R}^{l_1 \times l_2 \times n_1 \times n_2}$ , where here  $\cdot$  denotes contraction over the  $m_1 \times m_2$  indices. The following lemmata then follow from routine calculation.
Lemma B.2. Let  $bn: \mathbb{R}^{d \times N} \to \mathbb{R}^{d \times N}$  be an  $\epsilon$ -batchnorm layer. Then one can write  $bn = v \circ m$ , where  $v, m: \mathbb{R}^{d \times N} \to \mathbb{R}^{d \times N}$  are given respectively by
$$
v (Y) = \left(N \epsilon + \| Y \| _ {r o w} ^ {2}\right) ^ {- \frac {1}{2}} \sqrt {N} Y, \tag {25}
$$
$$
m (X) = X - \frac {1}{N} X 1 _ {N \times N}. \tag {26}
$$
$$
\frac {\partial v _ {j} ^ {i}}{\partial y _ {l} ^ {k}} = \delta_ {k} ^ {i} \sqrt {N} (N \epsilon + \| y ^ {i} \| ^ {2}) ^ {- \frac {1}{2}} \left(\delta_ {l} ^ {j} - (N \epsilon + \| y ^ {i} \| ^ {2}) ^ {- 1} y _ {l} ^ {i} y _ {j} ^ {i}\right) \tag {27}
$$
$$
\begin{array}{l} \frac {\partial^ {2} v _ {j} ^ {i}}{\partial y _ {n} ^ {m} \partial y _ {l} ^ {k}} = \delta_ {k} ^ {i} \delta_ {m} ^ {i} \sqrt {N} (N \epsilon + \| y ^ {i} \| ^ {2}) ^ {- \frac {3}{2}} \times \\ \times \left(3 (N \epsilon + \| y ^ {i} \| ^ {2}) ^ {- 1} y _ {n} ^ {i} y _ {l} ^ {i} y _ {j} ^ {i} \right. \\ \left. - \left(\delta_ {l} ^ {j} y _ {n} ^ {i} + \delta_ {n} ^ {l} y _ {j} ^ {i} + \delta_ {n} ^ {j} y _ {l} ^ {i}\right)\right), \tag {28} \\ \end{array}
$$
$$
\frac {\partial m _ {j} ^ {i}}{\partial x _ {l} ^ {k}} = \delta_ {k} ^ {i} \left(\delta_ {l} ^ {j} - N ^ {- 1}\right). \tag {29}
$$
and all second derivatives of  $m$  being zero.
Lemma B.3. Let  $wn: \mathbb{R}^{d_1 \times d_0} \to \mathbb{R}^{d_1 \times d_0}$  be an  $\epsilon$ -weight normalised parameterisation (Example A.1). Then one has
$$
\frac {\partial w n _ {j} ^ {i}}{\partial w _ {l} ^ {k}} = \delta_ {k} ^ {i} \left(\epsilon + \| w ^ {i} \| ^ {2}\right) ^ {- \frac {1}{2}} \left(\delta_ {l} ^ {j} - \left(\epsilon + \| w ^ {i} \| ^ {2}\right) ^ {- 1} w _ {l} ^ {i} w _ {j} ^ {i}\right) \tag {30}
$$
$$
\begin{array}{l} \frac {\partial w n _ {j} ^ {i}}{\partial w _ {n} ^ {m} \partial w _ {k} ^ {k}} = \delta_ {k} ^ {i} \delta_ {m} ^ {i} (\epsilon + \| w ^ {i} \| ^ {2}) ^ {- \frac {3}{2}} \times \\ \times \left(3 (\epsilon + \| w ^ {i} \| ^ {2}) ^ {- 1} w _ {n} ^ {i} w _ {l} ^ {i} w _ {j} ^ {i} \right. \\ \left. - \left(\delta_ {l} ^ {j} w _ {n} ^ {i} + \delta_ {n} ^ {l} w _ {j} ^ {i} + \delta_ {n} ^ {j} w _ {l} ^ {i}\right)\right). \tag {31} \\ \end{array}
$$
Similarly, if  $en: \mathbb{R}^{d_1 \times d_0} \to \mathbb{R}^{d_1 \times d_0}$  is an  $\epsilon$ -entry-normalised parameterisation, then
$$
\frac {\partial e n _ {j} ^ {i}}{\partial w _ {l} ^ {k}} = \delta_ {k} ^ {i} \delta_ {l} ^ {j} \epsilon \left(\epsilon + \left(w _ {j} ^ {i}\right) ^ {2}\right) ^ {- \frac {3}{2}} \tag {32}
$$
$$
\frac {\partial e n _ {j} ^ {i}}{\partial w _ {m} ^ {n} \partial w _ {l} ^ {k}} = - \delta_ {n} ^ {i} \delta_ {m} ^ {j} \delta_ {k} ^ {i} \delta_ {l} ^ {j} 3 \epsilon \left(\epsilon + \left(w _ {j} ^ {i}\right) ^ {2}\right) ^ {- \frac {3}{2}} w _ {j} ^ {i} \tag {33}
$$
Proof of Theorem 4.5. (1) follows from continuity of the nonlinearity and its derivative, implying boundedness of both over bounded sets in  $\mathbb{R}^{d\times N}$ .
(2) and (3) follow from a similar argument to the following argument for batch norm, which we give following Lemma B.2. Specifically, for the composite  $f \coloneqq \mathrm{bn} \circ \mathrm{aff} : \mathbb{R}^{d_1 \times d_0} \times \mathbb{R}^{d_0 \times N} \to \mathbb{R}^{d_1 \times N}$  defined by an  $\epsilon$ -BN layer and an affine layer, we will prove that over any set  $B \subset \mathbb{R}^{d_0 \times N}$  consisting of matrices  $X$  whose covariance matrix is nondegenerate, one has  $f, Df$  and  $Jf$  all globally bounded and Lipschitz. Indeed,  $v$  (Equation (25)) is clearly globally bounded, while  $Dv$  (Equation (27)) is globally bounded, decaying like  $\| Y\|_{row}^{-1}$  out to infinity, and  $D^2 v$  (Equation (28)) is globally bounded, decaying like  $\| Y\|_{row}^{-2}$  out to infinity. Consequently,
$$
\mathrm {b n} \circ \operatorname {a f f} = v \circ (m \circ \operatorname {a f f}),
$$
$$
D (\mathrm {b n} \circ \text {a f f}) = (J v \circ m \circ \text {a f f}) \cdot (J m \circ \text {a f f}) \cdot D \text {a f f},
$$
$$
J (\mathrm {b n} \circ \mathrm {a f f}) = (J v \circ m \circ \mathrm {a f f}) \cdot (J m \circ \mathrm {a f f}) \cdot J \mathrm {a f f},
$$
and similarly the derivatives of  $D(\mathrm{bn} \circ \mathrm{aff})$  and  $J(\mathrm{bn} \circ \mathrm{aff})$  are all globally bounded over  $\mathbb{R}^{d_1 \times d_0} \times B$ . The hypothesis that  $B$  consists of matrices with nondegenerate covariance matrix is needed here because while  $Jv \circ m \circ \mathrm{aff}$  decays like  $\| A(X - \mathbb{E}[X]) \|_{row}^{-1}$  out to infinity, the row-norm  $\| (A(X - \mathbb{E}[X]))^i) \|^2 = (A^i(X - \mathbb{E}[X])(X - \mathbb{E}[X])^T (A^i)^T) = \| A^i \|_{\operatorname{Cov}(X)}^2$  can only be guaranteed to increase with  $A$  if  $\operatorname{Cov}(X)$  is nondegenerate. Thus, for instance, without the nondegeneracy hypothesis on  $\operatorname{Cov}(X)$ ,  $A \mapsto J(\mathrm{bn} \circ \mathrm{aff})(A, X)$  grows unbounded like  $J\mathrm{aff}(A, X) = A \otimes \mathrm{Id}_N$  in any direction of degeneracy of  $\operatorname{Cov}(X)$ . Nonetheless, with the nondegenerate covariance assumption on elements of  $B$ ,  $\mathrm{bn} \circ \mathrm{aff}$  satisfies the hypotheses of Proposition 4.3 over  $\mathbb{R}^{d_1 \times d_0} \times B$ .
(2) and (3) now follow from essentially the same boundedness arguments as for batch norm, using Lemma B.3 in the place of Lemma B.2. However, since the row norms in this case are always defined by the usual Euclidean inner product on row-vectors, as opposed to the possibly degenerate inner product coming from the covariance matrix of the input vectors, one does not require any hypotheses aside from boundedness on the set  $B$ . Thus entry- and weight-normalised affine layers satisfy the hypotheses of Proposition 4.3.
Finally, (5) follows from the above arguments. More specifically, if  $g: \mathbb{R}^p \times \mathbb{R}^{d \times N} \to \mathbb{R}^{d \times N}$  is any composite of layers of the above form, then  $g$  satisfies the hypotheses of Proposition 4.3. Consequently, so too does the residual block  $f(\theta, X) := X + g(\theta, X)$ , for which  $Jf(\theta, X) = \mathrm{Id}_d \otimes \mathrm{Id}_N + Jg(\theta, X)$  and  $Df(\theta, X) = Dg(\theta, X)$ .
Proof of Proposition 4.6. In the notation of Proposition 4.2, the product  $DF(\vec{\theta})DF(\vec{\theta})^T$  is the sum of the positive-semidefinite matrices  $D_{\theta_l}F(\vec{\theta})D_{\theta_l}F(\vec{\theta})^T$ . Therefore  $\lambda (DF(\vec{\theta}))\geq \sum_{l}\lambda (D_{\theta_l}F(\vec{\theta}))$ . The result now follows from the inequality  $\lambda (AB)\geq \lambda (A)\lambda (B)$  applied inductively using Equation (11). Note that  $\lambda (AB)\geq \lambda (A)\lambda (B)$  is either trivial if one or both of  $A$  and  $B$  have more rows than columns (in which case the right hand side is zero), and follows from the well-known inequality  $\sigma (AB)\geq \sigma (A)\sigma (B)$  for the smallest singular values if both  $A$  and  $B$  have at least as many columns as rows.
Theorem 4.7 follows from the following two lemmata.
Lemma B.4. Let  $g: \mathbb{R}^p \times \mathbb{R}^{d_0 \times N} \to \mathbb{R}^{d_1 \times N}$  be a layer for which there exists  $\delta > 0$  such that  $\| Jg(\theta, X) \|_2 < (1 - \delta)$  for all  $\theta$  and  $X$ . Let  $I: \mathbb{R}^{d_0 \times N} \to \mathbb{R}^{d_1 \times N}$  be a linear map whose singular values are all equal to 1. Then the residual block  $f(\theta, X) := IX + g(\theta, X)$  has  $\sigma(Jf(\theta, X)) > \delta$  for all  $\theta$  and  $X$ .
$$
J f (\theta , X) = I \otimes \operatorname {I d} _ {N} + J g (\theta , X). \tag {34}
$$
The result then follows from Weyl's inequality: all singular values of  $I \otimes \mathrm{Id}_N$  are equal to 1, so that
$$
\sigma (J f (\theta , X)) \geq 1 - \| J g (\theta , X) \| _ {2} > \delta
$$
for all  $\theta$  and  $X$ .
Lemma B.5. Let  $P:\mathbb{R}^p\to \mathbb{R}^{d_1\times d_0}$  be a parameterisation. Then
$$
\sigma \left(D a f f _ {P} (w, X)\right) \geq \sigma (X) \sigma \left(D P (w)\right) \tag {35}
$$
for all  $w\in \mathbb{R}^p$  and  $X\in \mathbb{R}^{d_0\times N}$
Proof. Follows from Equation (21) and the inequality  $\sigma(AB) \geq \sigma(A)\sigma(B)$ .
Proof of Theorem 4.7. Hypothesis 1 in Theorem 4.7 says that the residual branches of the  $f_{l}, l \geq 2$ , satisfy the hypotheses of Lemma B.4, so that  $\sigma(Jf_{l}(\theta_{l}, f_{<l}(\vec{\theta}, X))) > 0$  for all  $l \geq 2$ . By the assumption that  $d_{l-1} \geq d_{l}$ , this means that  $\lambda(Jf_{l}(\theta_{l}, f_{<l}(\vec{\theta}, X))) = \sigma(Jf_{l}(\theta_{l}, f_{<l}(\vec{\theta}, X)))^{2} > 0$ . On the other hand, hypothesis 2 together with Lemma B.5 implies that  $\lambda(Df_{1}(\theta_{1}, X)) \geq \sigma(Df_{1}(\theta_{1}, X))^{2} > 0$ . The result now follows from Proposition 4.6.
Proof of Theorem 5.1. By Theorem 4.5, all layers satisfy the Hypotheses of Proposition 4.3 and so by Corollary 4.4, the associated loss function is globally Lipschitz, with Lipschitz constant some  $\beta >0$ . Take  $\eta >0$  to be any number smaller than  $2\beta^{-1}$ ; thus the loss can be guaranteed to be decreasing with every gradient descent step.
We now show that the network satisfies the hypotheses of Theorem 4.7. The dimension constraints in item (1) are encoded directly into the definition of the network, while the operator-norm of each of the residual branches, as products of  $P(w)$  and  $D\Phi$  matrices, are globally bounded by 1 by our hypotheses on these factors. For item (2), our data matrix is full-rank since it consists of linearly independent data, while by definition we have  $p_1 = d_1d_0$  with  $Df_{1} = D\mathrm{aff}_{\mathrm{en}}$  being everywhere full-rank since  $\epsilon$ -entry-normalisation is a diffeomorphism onto its image for any  $\epsilon > 0$ . Its hypotheses being satisfied by our weight-normalised residual network, Theorem 4.7 implies the parameter-function map  $F$  associated to  $\{f_l\}_{l = 1}^L$  and  $X$  satisfies  $\lambda (DF(\vec{\theta})) = \sigma (DF(\vec{\theta}))^2 > 0$  for all parameters  $\vec{\theta}$ . There are now two cases to consider.
In the first case, the gradient descent trajectory never leaves some ball of finite radius in  $\mathbb{R}^{d_1\times d_0}$ , the parameter space for the first layer. In any such ball, recalling that the first layer's parameterisation is entry-normalisation (Example A.1), the smallest singular value
$$
\sigma (D \operatorname {e n} (w)) = \min  _ {1 \leq i \leq d _ {1}} \left. \begin{array}{l} 1 \leq j \leq d _ {0} \\ w ^ {i} \end{array} \right. \frac {\epsilon}{\left(\epsilon + \left(w _ {j} ^ {i}\right) ^ {2}\right) ^ {\frac {3}{2}}} \tag {36}
$$
of  $D\mathrm{en}(w)$  is uniformly lower bounded by some positive constant. Thus by Lemmas B.5 and B.4<sup>8</sup>, the smallest singular value of  $DF$  is also uniformly lower bounded by a positive constant in any such ball. It follows from Theorem 2.4 that the loss satisfies the PL-inequality over such a ball, so that gradient descent converges in this case at a linear rate to a global minimum.
The second and only other case that must be considered is when for each  $R > 0$ , there is some time  $T$  for which the weight norm  $\| w_t\|$  of the parameters in the first layer is greater than  $R$  for all  $t\geq T$ . That is, the parameter trajectory in the first layer is unbounded in time. In this case, inspection of Equation (36) reveals that the smallest singular value of  $DF$  cannot be uniformly bounded below by a positive constant over all of parameter space. Theorem 2.4 then says that there is merely a sequence  $(\mu_t)_{t\in \mathbb{N}}$ , with  $\mu_t$  proportional to  $\sigma (D\mathrm{en}(w_t))$ , for which
$$
\ell_ {t} - \ell^ {*} \leq \prod_ {i = 0} ^ {t} (1 - \mu_ {i} \alpha) (\ell_ {0} - \ell^ {*}), \tag {37}
$$
where  $\alpha = \eta (1 - 2\beta^{-1}\eta) > 0$ . To guarantee convergence in this case, therefore, it suffices to show that  $\prod_{t=0}^{\infty}(1 - \mu_t\alpha) = 0$ ; equivalently, it suffices to show that the infinite series
$$
\sum_ {t = 0} ^ {\infty} \log (1 - \mu_ {t} \alpha) \tag {38}
$$
diverges.
The terms of the series (38) form a sequence of negative numbers which converges to zero. Hence, for the series (38) to diverge, it is necessary that  $\mu_t$  decrease sufficiently slowly with time. By the integral test, therefore, it suffices to find an integrable function  $m:[t_0,\infty)\to \mathbb{R}_{\geq 0}$  such that  $\mu_t\geq m(t)$  for each integer  $t\geq t_0$ , for which the integral  $\int_{t_0}^{\infty}\log (1 - m(t)\alpha)dt$  diverges.
We construct  $m$  by considering the worst possible case: where each gradient descent step is in exactly the same direction going out to  $\infty$ , thereby decreasing  $\sigma(D\mathrm{en}(w))$  at the fastest possible rate. By applying an orthogonal-affine transformation to  $\mathbb{R}^{d_1d_0}$ , we can assume without loss of generality that the algorithm is initialised at, and consistently steps in the direction of, the first canonical basis vector  $e_1$  in  $\mathbb{R}^{d_1d_0}$ . Specifically, letting  $\vec{\theta}$  be the vector of parameters for all layers following the first and  $w \in \mathbb{R}^{d_1d_0}$  the first layer parameters, for  $r \in \mathbb{R}_{\geq 1}$  we may assume that
$$
\nabla_ {w} \ell (\vec {\theta}, r e _ {1}) = \partial_ {w _ {1} ^ {1}} \ell (\vec {\theta}, r e _ {1}) e _ {1}, \tag {39}
$$
See the supplementary material.
with  $\partial_{w_1^1}\ell (\vec{\theta},re_1)\geq 0$  for all  $(\vec{\theta},r)$ . Let  $\gamma$  denote the convex function defined by the cost  $c$  (cf. Equation (6)), and let  $A(\vec{\theta},re_1)$  denote the  $d_{1}d_{0}$ -dimensional row vector
$$
D \gamma \left(F (\vec {\theta}, r e _ {1})\right) \prod_ {l = 2} ^ {L} J f _ {l} \left(\theta_ {l}, f _ {<   l} \left(\left(\vec {\theta}, r e _ {1}\right), X\right)\right) \left(\mathrm {I d} _ {d _ {1}} \otimes X ^ {T}\right). \tag {40}
$$
Then, in this worst possible case, the single nonzero partial derivative defining the loss gradient with respect to  $w$  at the point  $(\vec{\theta}, re_1)$  is given by
$$
\partial_ {w _ {1} ^ {1}} \ell (\vec {\theta}, r e _ {1})) = A (\vec {\theta}, r e _ {1}) _ {1} \frac {\epsilon}{\left(\epsilon + r ^ {2}\right) ^ {\frac {3}{2}}}, \tag {41}
$$
where  $A(\vec{\theta},re_1)_1$  denotes the first component of the row vector  $A(\vec{\theta},re_1)$  (cf. Equation (21)). By Theorem 4.5, however, the magnitude of  $A(\vec{\theta},re_1)$  can be globally upper bounded by some constant  $C$ . Thus
$$
\partial_ {w _ {1} ^ {1}} \ell (\vec {\theta}, r e _ {1}) \leq \frac {C \epsilon}{(\epsilon + r ^ {2}) ^ {\frac {3}{2}}} \leq \frac {C \epsilon}{r ^ {3}} \tag {42}
$$
for all  $\vec{\theta}$  and  $r\geq 1$
Let us therefore consider the Euler method, with step size  $\eta$ , applied over  $\mathbb{R}_{\geq 1}$ , starting from  $r_0 = 1$  with respect to the vector field  $V(r) = C\epsilon r^{-3}$ . Labelling the iterates  $(r_t)_{t \in \mathbb{N}}$ , we claim that there exist constants  $\gamma_1, \gamma_2$  and  $\gamma_3$  such that  $0 < r_t \leq \gamma_1 + \gamma_2 (t + \gamma_3)^{\frac{1}{4}}$  for all  $t \in \mathbb{N}$ . Indeed, observe that the solution to the flow equation  $\dot{r}(t) = C\epsilon r(t)^{-3}$  is  $r(t) = (4C\epsilon t + 1)^{\frac{1}{4}}$ , so the claim follows if we can show that there exists a constant  $B$  such that  $|r_t - r(t)| < B$  for all integer  $t \geq 0$ . However this follows from Theorem 10.6 of [17].
Now, since for each  $t$ ,  $r_t$  is an upper bound for the magnitude of the parameter vector  $w_t \in \mathbb{R}^{d_1 d_0}$ , we see from Equation (36) that the smallest singular value  $\sigma(D\mathrm{en}(w_t))$  admits the lower bound
$$
\sigma \left(D \operatorname {e n} \left(w _ {t}\right)\right) \geq \frac {\epsilon}{\left(\epsilon + \left(\gamma_ {1} + \gamma_ {2} \left(t + \gamma_ {3}\right) ^ {\frac {1}{4}}\right) ^ {2}\right) ^ {\frac {3}{2}}} \tag {43}
$$
for all  $t \in \mathbb{N}$ . Clearly,  $(\epsilon + (\gamma_1 + \gamma_2 (t + \gamma_3)^{\frac{1}{4}})^2)^{\frac{3}{2}} = O(t^{\frac{3}{4}})$ . Hence there exist  $t_0 > 0$  and  $\Gamma > 0$  such that
$$
\mu_ {t} \geq \frac {\Gamma}{t ^ {\frac {3}{4}}} \tag {44}
$$
for all integer  $t \geq t_0$ . Then, setting  $m(t) \coloneqq \Gamma t^{-\frac{3}{4}}$ , the integral
$$
\int_ {t _ {0}} ^ {\infty} \log (1 - m (t) \alpha) d t = \int_ {t _ {0}} ^ {\infty} \log \left(\frac {t ^ {\frac {3}{4}} - \Gamma \alpha}{t ^ {\frac {3}{4}}}\right) d t \tag {45}
$$
diverges. It follows that gradient descent converges as  $t \to \infty$  to a global minimum.
For all our experiments, the data was standardised channel-wise using the channel-wise mean and standard deviation over the training set.
On CIFAR10/100, our models were trained using SGD with a batch size of 128 and random crop/horizontal flip data augmentation. We ran 10 trials over each of the learning rates 0.2, 0.1, 0.05 and 0.02. The only exception to this is for CIFAR10 with a learning rate of 0.2, where training diverged 6 out of 10 times on the original network, so we plotted only those 4 trials where training did not diverge. Mean and standard deviation test accuracies, as well as averaged-over-final epoch loss values, are given in Tables 1 and 2.
The plots at the optimal learning rate, 0.1 for CIFAR10 and 0.05 for CIFAR100, are in Figure 3, while we provide the plots for the other learning rates in Figures 4 and 5.
<sup>9</sup> minimally modifying https://github.com/kuangliu/pytorch-cifar
Table 1: ResNet18 on CIFAR10
Learning rate	Original	Modified
Test accuracy	Final loss	Test accuracy	Final loss
0.20	48.14 ± 19.95	1.3883 ± 0.5503	62.71 ± 6.14	0.9495 ± 0.2150
0.10	91.64 ± 0.24	0.0032 ± 0.0002	91.42 ± 0.61	0.0022 ± 0.0002
0.05	91.22 ± 0.25	0.0043 ± 0.0003	91.14 ± 0.25	0.0041 ± 0.0001
0.02	89.50 ± 0.32	0.0112 ± 0.0004	88.94 ± 0.35	0.0125 ± 0.0004
Table 2: ResNet18 on CIFAR100
Learning rate	Original	Modified
Test accuracy	Final loss	Test accuracy	Final loss
0.20	69.67 ± 0.58	0.0150 ± 0.0005	70.07 ± 0.53	0.0115 ± 0.0005
0.10	70.05 ± 0.43	0.0147 ± 0.0004	70.87 ± 0.42	0.0116 ± 0.0003
0.05	70.04 ± 0.48	0.0145 ± 0.0003	70.34 ± 0.56	0.0116 ± 0.0003
0.02	69.75 ± 0.56	0.0145 ± 0.0003	70.13 ± 0.55	0.0116 ± 0.0004
(a)  $lr = 0.2$
(b)  $lr = 0.05$
(c)  $lr = 0.02$
Figure 4: Loss plots for ResNet18 on CIFAR10
(c)  $lr = 0.02$
Figure 5: Loss plots for ResNet18 on CIFAR100
Table 3: ResNet50 on ImageNet
Learning rate	Original	Modified
Test accuracy	Final loss	Test accuracy	Final loss
0.20	49.24 ± 34.75	1.0285 ± 0.0066	-	-
0.10	74.69 ± 0.12	0.9201 ± 0.0014	74.99 ± 0.09	0.8949 ± 0.0019
0.05	74.62 ± 0.12	0.8563 ± 0.0041	74.84 ± 0.03	0.8420 ± 0.0030
0.02	73.87 ± 0.09	0.8690 ± 0.0025	74.00 ± 0.11	0.8441 ± 0.0026
Figure 6: Loss for original and modified ResNet50 at different learning rates on ImageNet. The modified architecture did not converge at the maximum learning rate of 0.2.
On ImageNet, the models were trained using the default PyTorch ImageNet example $^{10}$ , using SGD with weight decay of  $1e - 4$  and momentum of 0.9, batch size of 256, and random crop/horizontal flip data augmentation. See Table 3 for the validation accuracies, and Figure 6 for the plots of the training losses at different learning rates.
$^{10}$ https://github.com/pytorch/examples/tree/main/imagenet