SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Indeed, the summations above are sums of  $d$  i.i.d. non-negative random variables, and hence its mean is  $\Omega(d)$  (as long as  $b \geq -O(1)$ ) and its standard deviation is  $O(\sqrt{d})$ . Now, using these approximations, one can rewrite the GD dynamics on the population loss  $\mathbb{E}[\ell_{\mathrm{logi}}(yf(\pmb{x}; a^{-}, a^{+}, b))]$ .
Using these approximations, the output of the ReLU network (2) can be written as
$$
f (\boldsymbol {x}; a ^ {-}, a ^ {+}, b) \approx d \left(a ^ {-} + a ^ {+}\right) g (b),
$$
which in turn leads to an approximation of the GD dynamics on the population loss  $(a^{-}, a^{+}, b) \mapsto \mathbb{E}[\ell_{\mathrm{logi}}(yf(\pmb{x}; a^{-}, a^{+}, b))]$ :
$$
\begin{array}{l} a _ {t + 1} ^ {\pm} = a _ {t} ^ {\pm} - \eta \mathbb {E} \left[ \ell_ {\log \mathrm {i}} ^ {\prime} \left(y \underbrace {f \left(\boldsymbol {x} ; a _ {t} ^ {-} , a _ {t} ^ {+} , b _ {t}\right)} _ {\approx d \left(a _ {t} ^ {-} + a _ {t} ^ {+}\right) g \left(b _ {t}\right)}\right) \times \underbrace {\sum_ {i = 1} ^ {d} \operatorname {R e L U} \left(\pm \boldsymbol {x} [ i ] + b _ {t}\right)} _ {\approx d g (b _ {t})} \right] \\ \approx a _ {t} ^ {\pm} - \eta \ell_ {\text {s y m}} ^ {\prime} \left(d \left(a _ {t} ^ {-} + a _ {t} ^ {+}\right) g (b _ {t})\right) d g (b _ {t}), \\ \end{array}
$$
$$
\begin{array}{l} b _ {t + 1} = b _ {t} - \eta \mathbb {E} \Big [ \ell_ {\mathrm {l o g i}} ^ {\prime} (y \underbrace {f (\pmb {x} ; a _ {t} ^ {-} , a _ {t} ^ {+} , b _ {t})} _ {\approx d (a _ {t} ^ {-} + a _ {t} ^ {+}) g (b _ {t})}) \times \Big (a _ {t} ^ {-} \underbrace {\sum_ {i = 1} ^ {d} \mathbb {1} \{- \pmb {x} [ i ] + b _ {t} \geq 0 \}} _ {\approx d g ^ {\prime} (b _ {t})} + a _ {t} ^ {+} \underbrace {\sum_ {i = 1} ^ {d} \mathbb {1} \{+ \pmb {x} [ i ] + b _ {t} \geq 0 \}} _ {\approx d g ^ {\prime} (b _ {t})} \Big) \Big ] \\ \approx b _ {t} - \eta \ell_ {\mathrm {s y m}} ^ {\prime} \big (d \left(a _ {t} ^ {-} + a _ {t} ^ {+}\right) g (b _ {t}) \big) d \left(a _ {t} ^ {-} + a _ {t} ^ {+}\right) g ^ {\prime} (b _ {t}), \\ \end{array}
$$
where  $\ell_{\mathrm{sym}}(s) \coloneqq \frac{1}{2} (\log (1 + \exp (-s)) + \log (1 + \exp (+s)))$  is the symmetrized logistic loss. Hence we arrive at the following dynamics on  $a^{\pm}$  and  $b$  that we call the mean model:
$$
\begin{array}{l} a _ {t + 1} ^ {\pm} = a _ {t} ^ {\pm} - \eta \ell_ {\text {s y m}} ^ {\prime} \left(d \left(a _ {t} ^ {-} + a _ {t} ^ {+}\right) g (b _ {t})\right) d g (b _ {t}), \\ b _ {t + 1} = b _ {t} - \eta \ell_ {\mathrm {s y m}} ^ {\prime} \left(d \left(a _ {t} ^ {-} + a _ {t} ^ {+}\right) g (b _ {t})\right) d \left(a _ {t} ^ {+} + a _ {t} ^ {-}\right) g ^ {\prime} (b _ {t}). \\ \end{array}
$$
Now, we can write the above dynamics more compactly in terms of the parameter  $A_{t} \coloneqq d(a_{t}^{-} + a_{t}^{+})$ .
$$
A _ {t + 1} = A _ {t} - 2 d ^ {2} \eta \ell_ {\text {s y m}} ^ {\prime} \left(A _ {t} g \left(b _ {t}\right)\right) g \left(b _ {t}\right),
$$
$$
b _ {t + 1} = b _ {t} - \eta \ell_ {\text {s y m}} ^ {\prime} (A _ {t} g (b _ {t})) A _ {t} g ^ {\prime} (b _ {t}).
$$
E Proofs for the mean model
In this section, we prove the main theorems for the mean model. We first recall the mean model for the reader's convenience.
$$
A _ {t + 1} = A _ {t} - 2 d ^ {2} \eta \ell^ {\prime} \left(A _ {t} g \left(b _ {t}\right)\right) g \left(b _ {t}\right),
$$
$$
b _ {t + 1} = b _ {t} - \eta \ell^ {\prime} \left(A _ {t} g (b _ {t})\right) A _ {t} g ^ {\prime} \left(b _ {t}\right).
$$
E.1 Deferred proofs
In this section, we collect together deferred proofs from Subsection 3.2.
Proof of Lemma 2. By definition,  $g(b) = \int_{-b}^{\infty} (\xi + b) \varphi(\xi) \, \mathrm{d}\xi = \int_{-b}^{\infty} \xi \varphi(\xi) \, \mathrm{d}\xi + b\varPhi(b)$ . Recalling  $\varphi'(\xi) = -\xi \varphi(\xi)$ , the first term equals  $\varphi(b)$ . Moreover,  $g'(b) = -b \varphi(b) + \varPhi(b) + b \varphi(b) = \varPhi(b)$ .
Proof of Lemma 3. Note that  $\partial_t\left(\frac{1}{2} A^2\right) = A\dot{A} = -2d^2\ell'(Ag(b))Ag(b)$  and also that  $\partial_t\kappa(b) = -\ell'(Ag(b))\kappa'(b)Ag'(b) = -\ell'(Ag(b))Ag(b)$  since  $\kappa' = g/g'$ . Hence,  $\partial_t\left(\frac{1}{2} A^2 - 2d^2\kappa(b)\right) = 0$  and the proof is completed.
E.2 Gradient flow regime
Proof of Theorem 3. The following proof is analogous to the proof of Theorem 1. We first list several facts we use in the proof:
(i)  $|g'(b)| = |\varPhi(b)| \leq 1$  for all  $b \in \mathbb{R}$ .
(ii)  $\ell^{\prime}(s) = \frac{1}{2}\frac{\exp(s) - 1}{\exp(s) + 1}$ . Hence,  $|\ell^{\prime}(s)|\leq \frac{1}{2}$  for all  $s\in \mathbb{R}$ , and we have
$$
\frac {\ell^ {\prime} (s)}{s} \geq \frac {1}{8} \times \left\{ \begin{array}{l l} 1, & \text {i f} | s | \leq 2, \\ 2 / | s |, & \text {i f} | s | > 2. \end{array} \right.
$$
(iii)  $\ell''(0) = 1/4$ .
(iv)  $\ell^{\prime \prime \prime}(s) = -\frac{\exp(s)(\exp(s) - 1)}{(\exp(s) + 1)^3}$ . Hence,  $\ell^{\prime \prime \prime}(s) < 0$  for  $s > 0$  and  $\ell^{\prime \prime \prime}(s) > 0$  for  $s < 0$ . In particular,  $|\ell^{\prime}(s)| \leq \frac{1}{4} |s|$  for all  $s \in \mathbb{R}$ .
Throughout the proof, we assume that  $A_0 > 0$  without loss of generality. We prove by induction the following claim: for  $t \geq 0$  and
$$
\gamma := \frac {1}{2 0 0} \min  \left\{\delta , 8 - \delta , \frac {8 - \delta}{A _ {0}} \right\},
$$
it holds that  $|A_{t}| \leq A_{0} \exp(-\gamma t)$ . This clearly holds at initialization.
Suppose that the claim holds up to iteration  $t$ . Using the bounds on  $|g'|$  and  $|\ell'|$ , it follows that
$$
\begin{array}{l} b _ {t + 1} \geq b _ {t} - \left| \ell^ {\prime} \left(A _ {t} g \left(b _ {t}\right)\right) \right| | A _ {t} | g ^ {\prime} (b _ {t}) \geq b _ {t} - \frac {1}{2} \eta | A _ {t} | \\ \geq b _ {t} - \frac {1}{2} \eta A _ {0} \exp (- \gamma t) \geq \dots \geq b _ {0} - \frac {1}{2} \eta A _ {0} \sum_ {s = 0} ^ {t} \exp (- \gamma s) \geq - \frac {\eta A _ {0}}{\gamma}. \\ \end{array}
$$
In particular,  $b_{t} \geq -1$  and  $g(b_{t}) > 0.08$ , since  $\eta \leq \frac{\gamma}{A_{0}}$ . Also, the bound shows that if the claim holds for all  $t$ , then we obtain the desired conclusion.
It remains to establish the inductive claim; assume that it holds up to iteration  $t$ . For the dynamics of  $A$ , by symmetry we may suppose that  $A_{t} > 0$ . From  $\ell^{\prime}(A_{t}g(b_{t}))\leq A_{t}g(b_{t}) / 4$  and  $g(b_{t})\leq g(0) = \frac{1}{\sqrt{2\pi}}$ , it follows that
$$
\begin{array}{l} A _ {t + 1} = A _ {t} - 2 \eta d ^ {2} \ell^ {\prime} (A _ {t} g (b _ {t})) g (b _ {t}) \geq \left(1 - \frac {\eta d ^ {2}}{2} g (b _ {t}) ^ {2}\right) A _ {t} \\ \geq \left(1 - \frac {\eta d ^ {2}}{2} g (0) ^ {2}\right) A _ {t} = - \left(1 - \frac {\delta}{4}\right) A _ {t}. \\ \end{array}
$$
This shows that  $A_{t + 1} \geq -(1 - \gamma)A_t$ . Next, we show that  $A_{t + 1} \leq (1 - \gamma)A_t$ . First, if  $A_{t}g(b_{t}) \leq 2$ ,
$$
\begin{array}{l} A _ {t + 1} = A _ {t} - 2 \eta d ^ {2} \ell^ {\prime} (A _ {t} g (b _ {t})) g (b _ {t}) \leq A _ {t} - \frac {1}{4} \eta d ^ {2} A _ {t} g (b _ {t}) ^ {2} \\ = \left(1 - \frac {(8 - \delta) \pi}{4} g (b _ {t}) ^ {2}\right) A _ {t} \leq \left(1 - \frac {(8 - \delta)}{4} \pi \cdot 0. 0 8 ^ {2}\right) A _ {t} \leq (1 - \gamma) A _ {t}, \\ \end{array}
$$
since we have  $g(b_{t}) > 0.08$ . Next, if  $A_{t}g(b_{t}) \geq 2$ , then
$$
\begin{array}{l} A _ {t + 1} = A _ {t} - 2 \eta d ^ {2} \ell^ {\prime} \left(A _ {t} g (b _ {t})\right) g (b _ {t}) \leq A _ {t} - \frac {1}{2} \eta d ^ {2} g (b _ {t}) = \left(1 - \frac {(8 - \delta) \pi}{2} \frac {g (b _ {t})}{A _ {t}}\right) A _ {t} \\ \leq \left(1 - \frac {(8 - \delta) \pi}{2} \cdot \frac {0 . 0 8}{A _ {0}}\right) A _ {t} \leq (1 - \gamma) A _ {t}. \\ \end{array}
$$
This shows that  $|A_{t + 1}| \leq (1 - \gamma)|A_t|$  for the case  $A_{t} > 0$ . A similar conclusion is obtained for the case  $A_{t} < 0$ . The induction is complete.
E.3 EoS regime
Proof of Theorem 4. The following proof is analogous to the proof of Lemma 7. Assume throughout that  $A_{t} \neq 0$  for all  $t$ . Recall the dynamics for  $b$ :
$$
b _ {t + 1} = b _ {t} - \eta \ell^ {\prime} \left(A _ {t} g (b _ {t})\right) A _ {t} g ^ {\prime} (b _ {t}).
$$
Since  $\ell^{\prime}(s) / s\to 1 / 4$  as  $s\to 0$ , and  $\ell^\prime$  is increasing, this equation implies that if  $\lim \inf_{t\to \infty}|A_t| > 0$  then  $b_{t}$  must keep decreasing until  $\frac{1}{2} d^{2}g(b_{t})^{2} < 2 / \eta$ .
Suppose for the sake of contradiction that there exists  $\varepsilon > 0$  with  $\frac{1}{2} d^2 g(b_t)^2 > (2 + \varepsilon) / \eta$ , for all  $t$ . Let  $\varepsilon' > 0$  be such that  $1 - (2 + \varepsilon)(1 - \varepsilon') < -1$ , i.e.,  $\varepsilon' < \frac{\varepsilon}{2 + \varepsilon}$ . Then, there exists  $\delta > 0$  such  $|A_t| \leq \delta$  implies  $\ell'(A_t g(b_t)) / (A_t g(b_t)) > \frac{1}{4}(1 - \varepsilon')$ , hence
$$
\frac {\left| A _ {t + 1} \right|}{\left| A _ {t} \right|} = \left| 1 - 4 \cdot \frac {1}{4} \left(1 - \varepsilon^ {\prime}\right) \cdot \frac {1}{2} \eta d ^ {2} g \left(b _ {t}\right) ^ {2} \right| > \left| (2 + \varepsilon) \left(1 - \varepsilon^ {\prime}\right) - 1 \right| > 1.
$$
The above means that  $|A_t|$  increases until it exceeds  $\delta$ , i.e.,  $\lim \inf_{t \to \infty} |A_t| \geq \delta$ . This is our desired contradiction and it implies that  $\lim_{t \to \infty} \frac{1}{2} d^2 g(b_t)^2 \leq 2 / \eta$ .
Remark 4. A straightforward calculation yields that when  $(a_{\star}^{-}, a_{\star}^{+}, b_{\star})$  is a global minimizer (i.e.,  $a_{\star}^{-} + a_{\star}^{+} = 0$ ), then  $\lambda_{\max}\big(\nabla^{2}f(a_{\star}^{-}, a_{\star}^{+}, b_{\star})\big) = \frac{1}{2} d^{2}g(b_{\star})^{2}$ . The mean model initialized at  $(A_0,0)$  approximately reaches  $(0,0)$  whose sharpness is  $d^{2}g(0)^{2}/2 = d^{2}/4\pi$ . Hence, the bias learning regime  $2/\eta < d^{2}/(4\pi)$  precisely corresponds to the EoS regime,  $2/\eta < \lambda_{\max}\big(\nabla^{2}f(a_{\star}^{-}, a_{\star}^{+}, b_{\star})\big)$ .