SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
y _ {i} f \left(\mathbf {W} ^ {(\widetilde {t} + 1)}, \mathbf {x} _ {i}\right) \leq y _ {i} f \left(\mathbf {W} ^ {(\widetilde {t})}, \mathbf {x} _ {i}\right) + \frac {3 \eta}{n m} \cdot \left| \ell_ {i} ^ {\prime (\widetilde {t})} \right| \cdot \| \mathbf {x} _ {i} \| _ {2} ^ {2}, \tag {G.5}
$$
which implies that  $y_{i}f(\mathbf{W}^{(\widetilde{t} +1)},\mathbf{x}_{i}) \geq y_{i}f(\mathbf{W}^{(\widetilde{t})},\mathbf{x}_{i}) \geq -c$ . This verifies the first bullet at time  $t = \widetilde{t} +1$ . By subtracting (G.5) from (G.4), we have
$$
\begin{array}{l} y _ {k} f \left(\mathbf {W} ^ {(\widehat {t} + 1)}, \mathbf {x} _ {k}\right) - y _ {i} f \left(\mathbf {W} ^ {(\widehat {t} + 1)}, \mathbf {x} _ {i}\right) \\ \leq y _ {k} f (\mathbf {W} ^ {(\widetilde {t})}, \mathbf {x} _ {k}) - y _ {i} f (\mathbf {W} ^ {(\widetilde {t})}, \mathbf {x} _ {i}) + \frac {3 \eta}{n m} \cdot | \ell_ {k} ^ {\prime (\widetilde {t})} | \cdot \| \mathbf {x} _ {k} \| _ {2} ^ {2} - \frac {\eta}{5 n m} \cdot | \ell_ {i} ^ {\prime (\widetilde {t})} | \cdot \| \mathbf {x} _ {i} \| _ {2} ^ {2}. \\ \end{array}
$$
If  $|\ell_i^{\prime (\widetilde{t})}| / |\ell_k^{\prime (\widetilde{t})}|\geq 15R^2$ , then  $\frac{3\eta}{nm}\cdot |\ell_k^{\prime (\widetilde{t})}|\cdot \| \mathbf{x}_k\| _2^2\leq \frac{\eta}{5nm}\cdot |\ell_i^{\prime (\widetilde{t})}|\cdot \| \mathbf{x}_i\| _2^2$  and hence
$$
y _ {k} f (\mathbf {W} ^ {(\widetilde {t} + 1)}, \mathbf {x} _ {k}) - y _ {i} f (\mathbf {W} ^ {(\widetilde {t} + 1)}, \mathbf {x} _ {i}) \leq y _ {k} f (\mathbf {W} ^ {(\widetilde {t})}, \mathbf {x} _ {k}) - y _ {i} f (\mathbf {W} ^ {(\widetilde {t})}, \mathbf {x} _ {i}) \leq C _ {1}.
$$
If  $|\ell_i^{\prime (\widetilde{t})}| / |\ell_k^{\prime (\widetilde{t})}| < 15R^2$ , then by Lemma H.5
$$
y _ {k} f \left(\mathbf {W} ^ {(\widetilde {t})}, \mathbf {x} _ {k}\right) - y _ {i} f \left(\mathbf {W} ^ {(\widetilde {t})}, \mathbf {x} _ {i}\right) \leq \log \left(4 \left| \ell_ {i} ^ {\prime (\widetilde {t})} \right| / \left| \ell_ {k} ^ {\prime (\widetilde {t})} \right|\right) <   \log (6 0 R ^ {2}),
$$
$$
\begin{array}{l} y _ {k} f (\mathbf {W} ^ {(\widetilde {t} + 1)}, \mathbf {x} _ {k}) - y _ {i} f (\mathbf {W} ^ {(\widetilde {t} + 1)}, \mathbf {x} _ {i}) \leq y _ {k} f (\mathbf {W} ^ {(\widetilde {t})}, \mathbf {x} _ {k}) - y _ {i} f (\mathbf {W} ^ {(\widetilde {t})}, \mathbf {x} _ {i}) + \frac {3 \eta}{n m} \cdot | \ell_ {k} ^ {\prime (\widetilde {t})} | \cdot \| \mathbf {x} _ {k} \| _ {2} ^ {2} \\ \leq \log (6 0 R ^ {2}) + 1. \\ \end{array}
$$
Combining the two cases, we have
$$
y _ {k} f \left(\mathbf {W} ^ {\left(\widetilde {t} + 1\right)}, \mathbf {x} _ {k}\right) - y _ {i} f \left(\mathbf {W} ^ {\left(\widetilde {t} + 1\right)}, \mathbf {x} _ {i}\right) \leq C _ {1},
$$
as long as  $C_1 \geq \max \{4\beta, \log(60R^2) + 1\}$ , which verifies the fourth bullet at time  $t = \widetilde{t} + 1$ . By Lemma H.5, this leads to
$$
\begin{array}{l} \frac {\ell_ {i} ^ {\prime (\widetilde {t} + 1)}}{\ell_ {k} ^ {\prime (\widetilde {t} + 1)}} = \frac {1 + \exp (y _ {k} f (\mathbf {W} ^ {(\widetilde {t} + 1)} , \mathbf {x} _ {k}))}{1 + \exp (y _ {i} f (\mathbf {W} ^ {(\widetilde {t} + 1)} , \mathbf {x} _ {i}))} \\ \leq 2 \left(1 + \exp \left(y _ {k} f \left(\mathbf {W} ^ {\left(\widetilde {t} + 1\right)}, \mathbf {x} _ {k}\right) - y _ {i} f \left(\mathbf {W} ^ {\left(\widetilde {t} + 1\right)}, \mathbf {x} _ {i}\right)\right)\right) \\ \leq 2 \left(1 + \exp \left(C _ {1}\right)\right) \\ \leq C _ {2}, \\ \end{array}
$$
as long as  $C_2 \geq 2(1 + \exp(C_1))$ . For any  $r \in S_i^{(\widetilde{t} + 1)}$ , we have
$$
\begin{array}{l} \langle \mathbf {w} _ {y _ {i}, r} ^ {(\tilde {t} + 2)}, \mathbf {x} _ {i} \rangle = \langle \mathbf {w} _ {y _ {i}, r} ^ {(\tilde {t} + 1)}, \mathbf {x} _ {i} \rangle + \frac {\eta}{n m} \sum_ {i ^ {\prime} = 1} ^ {n} | \ell_ {i ^ {\prime}} ^ {\prime (\tilde {t} + 1)} | \cdot \sigma^ {\prime} (\langle \mathbf {w} _ {y _ {i}, r} ^ {(\tilde {t} + 1)}, \mathbf {x} _ {i} \rangle) \cdot \langle y _ {i ^ {\prime}} \mathbf {x} _ {i ^ {\prime}}, y _ {i} \mathbf {x} _ {i} \rangle \\ = \langle \mathbf {w} _ {y _ {i}, r} ^ {(\widetilde {t} + 1)}, \mathbf {x} _ {i} \rangle + \frac {\eta}{n m} \cdot | \ell_ {i} ^ {\prime (\widetilde {t} + 1)} | \cdot \| \mathbf {x} _ {i} \| _ {2} ^ {2} \\ + \frac {\eta}{n m} \sum_ {i ^ {\prime} \neq i} | \ell_ {i ^ {\prime}} ^ {\prime (\widetilde {t} + 1)} | \cdot \sigma^ {\prime} (\langle \mathbf {w} _ {y _ {i}, r} ^ {(\widetilde {t} + 1)}, \mathbf {x} _ {i} \rangle) \cdot \langle y _ {i ^ {\prime}} \mathbf {x} _ {i ^ {\prime}}, y _ {i} \mathbf {x} _ {i} \rangle \\ \geq \langle \mathbf {w} _ {y _ {i}, r} ^ {(\widetilde {t} + 1)}, \mathbf {x} _ {i} \rangle + \frac {\eta}{n m} \cdot | \ell_ {i} ^ {\prime (\widetilde {t} + 1)} | \cdot \| \mathbf {x} _ {i} \| _ {2} ^ {2} - \frac {\eta}{n m} \sum_ {i ^ {\prime} \neq i} | \ell_ {i ^ {\prime}} ^ {\prime (\widetilde {t} + 1)} | \cdot | \langle \mathbf {x} _ {i ^ {\prime}}, \mathbf {x} _ {i} \rangle | \\ \geq \left\langle \mathbf {w} _ {\mathcal {Y} _ {i}, r} ^ {(\widetilde {t} + 1)}, \mathbf {x} _ {i} \right\rangle \geq 0, \\ \end{array}
$$
where the second equality is by  $\langle \mathbf{w}_{y_i,r}^{(\widetilde{t} +1)},\mathbf{x}_i\rangle \geq 0$ ; the first inequality is by triangle inequality; the second inequality is by  $|\ell_{i}'(\widetilde{t} +1)| / |\ell_{i}'(\widetilde{t} +1)|\leq C_2$  and the condition that  $R_{\mathrm{min}}^2\geq Cnp$ ,  $C$  is a sufficiently large constant. This verifies the fourth bullet at time  $t = \widetilde{t} +1$ .
Notice that  $\| \mathbf{W}^{(t)}\|_F = \Theta (\log t)$  and considering the fact that the difference between any two margins can be bounded by a constant, the difference between any two margins can be bounded by a constant, we can derive the following lemma, which demonstrates that the normalized margin of all the training data points will converge to the same value.
Lemma G.2. For both two-layer ReLU and leaky ReLU neural networks, gradient descent will asymptotically find a neural network such that all the training data points possess the same normalized margin, i.e.,
$$
\lim  _ {t \rightarrow \infty} \left| y _ {i} f \left(\frac {\mathbf {W} ^ {(t)}}{\| \mathbf {W} ^ {(t)} \| _ {F}}, \mathbf {x} _ {i}\right) - y _ {k} f \left(\frac {\mathbf {W} ^ {(t)}}{\| \mathbf {W} ^ {(t)} \| _ {F}}, \mathbf {x} _ {k}\right)\right| = 0,
$$
for any  $i, k \in [n]$ .
By Lemma G.1, we can establish the subsequent lemma regarding the logarithmic rate of increase in margin. This lemma will be beneficial in demonstrating the convergence rate of the training loss in subsequent proofs.
Lemma G.3. There exists time  $T = \Theta(\eta^{-1}R_{\min}^{-2}nm)$  such that the following increasing rate of margin  $y_{i}f(\mathbf{W}^{(t)},\mathbf{x}_{i})$  holds:
$$
\left| y _ {i} f (\mathbf {W} ^ {(t)}, \mathbf {x} _ {i}) - \log t - \log (\eta \| \mathbf {x} _ {i} \| _ {2} ^ {2} / n m) \right| \leq C _ {3},
$$
$$
C _ {4} \cdot \left(\frac {\eta \| \mathbf {x} _ {i} \| _ {2} ^ {2}}{n m} \cdot t\right) ^ {- 1} \leq | \ell_ {i} ^ {\prime (t)} | \leq C _ {5} \cdot \left(\frac {\eta \| \mathbf {x} _ {i} \| _ {2} ^ {2}}{n m} \cdot t\right) ^ {- 1},
$$
for any  $i \in [n]$  and  $t \geq T$ , where  $C_3, C_4, C_5$  are constants.
Proof of Lemma G.3. To prove this, we want to leverage Lemma H.1 and Lemma H.2. To achieve this, we need approximate  $|\ell_i^{\prime (t)}|$  by  $y_{i}f(\mathbf{W}^{(t)},\mathbf{x}_{i})$  . We have
$$
\left| \ell_ {i} ^ {\prime (t)} \right| = \frac {1}{1 + \exp \left(y _ {i} f \left(\mathbf {W} _ {y _ {i}} ^ {(t)} , \mathbf {x} _ {i}\right)\right)} \leq \exp \left(- y _ {i} f \left(\mathbf {W} _ {y _ {i}} ^ {(t)}, \mathbf {x} _ {i}\right)\right), \tag {G.6}
$$
$$
\left| \ell_ {i} ^ {\prime (t)} \right| = \frac {1}{1 + \exp \left(y _ {i} f \left(\mathbf {W} _ {y _ {i}} ^ {(t)}, \mathbf {x} _ {i}\right)\right)} \geq \frac {1}{1 + e ^ {c}} \cdot \exp \left(- y _ {i} f \left(\mathbf {W} _ {y _ {i}} ^ {(t)}, \mathbf {x} _ {i}\right)\right) \tag {G.7}
$$
by  $y_{i}f(\mathbf{W}_{y_{i}}^{(t)},\mathbf{x}_{i}) \geq -c$ . Plugging the upper and lower bounds of  $|\ell_i^{\prime (t)}|$  into (G.4) and (G.5), we obtain
$$
y _ {i} f \left(\mathbf {W} ^ {(t + 1)}, \mathbf {x} _ {i}\right) - y _ {i} f \left(\mathbf {W} ^ {(t)}, \mathbf {x} _ {i}\right) \leq \frac {3 \eta \| \mathbf {x} _ {i} \| _ {2} ^ {2}}{n m} \cdot \exp \left(- y _ {i} f \left(\mathbf {W} _ {y _ {i}} ^ {(t)}, \mathbf {x} _ {i}\right)\right), \tag {G.8}
$$
$$
y _ {i} f \left(\mathbf {W} ^ {(t + 1)}, \mathbf {x} _ {i}\right) - y _ {i} f \left(\mathbf {W} ^ {(t)}, \mathbf {x} _ {i}\right) \geq \frac {\eta \| \mathbf {x} _ {i} \| _ {2} ^ {2}}{5 \left(1 + e ^ {c}\right) n m} \cdot \exp \left(- y _ {i} f \left(\mathbf {W} _ {y _ {i}} ^ {(t)}, \mathbf {x} _ {i}\right)\right). \tag {G.9}
$$
By taking  $x_{t} = y_{i}f(\mathbf{W}^{(t)},\mathbf{x}_{i}) - y_{i}f(\mathbf{W}^{(0)},\mathbf{x}_{i})$  and applying Lemma H.1 to (G.8), we can get
$$
y _ {i} f (\mathbf {W} ^ {(t)}, \mathbf {x} _ {i}) \leq y _ {i} f (\mathbf {W} ^ {(0)}, \mathbf {x} _ {i}) + \log \left(1 + \frac {3 \eta \| \mathbf {x} _ {i} \| _ {2} ^ {2}}{e ^ {y _ {i} f (\mathbf {W} ^ {(0) , \mathbf {x} _ {i}})} n m} \exp \left(\frac {3 \eta \| \mathbf {x} _ {i} \| _ {2} ^ {2}}{e ^ {y _ {i} f (\mathbf {W} ^ {(0) , \mathbf {x} _ {i}})} n m}\right) \cdot t\right).
$$
As long as  $t \geq \frac{e^{y_i f(\mathbf{W}^{(0)},\mathbf{x}_i)}nm}{3\eta\|\mathbf{x}_i\|_2^2}\exp \left(-\frac{3\eta\|\mathbf{x}_i\|_2^2}{e^{y_i f(\mathbf{W}^{(0)},\mathbf{x}_i)}nm}\right) = \Theta (\eta^{-1}R_{\min}^{-2}nm)$ , we have
$$
\begin{array}{l} y _ {i} f \left(\mathbf {W} ^ {(t)}, \mathbf {x} _ {i}\right) \leq y _ {i} f \left(\mathbf {W} ^ {(0)}, \mathbf {x} _ {i}\right) + \log \left(\frac {6 \eta \| \mathbf {x} _ {i} \| _ {2} ^ {2}}{e ^ {y _ {i} f \left(\mathbf {W} ^ {(0) ,} \mathbf {x} _ {i}\right)} n m} \exp \left(\frac {3 \eta \| \mathbf {x} _ {i} \| _ {2} ^ {2}}{e ^ {y _ {i} f \left(\mathbf {W} ^ {(0) ,} \mathbf {x} _ {i}\right)} n m}\right) \cdot t\right) \\ = \log t + \log \left(\frac {\eta \| \mathbf {x} _ {i} \| _ {2} ^ {2}}{n m}\right) + \log 6 + \frac {3 \eta \| \mathbf {x} _ {i} \| _ {2} ^ {2}}{e ^ {y _ {i} f (\mathbf {W} ^ {(0)} , \mathbf {x} _ {i})} n m} \\ \leq \log t + \log \left(\frac {\eta \| \mathbf {x} _ {i} \| _ {2} ^ {2}}{n m}\right) + \log 6 + 1, \\ \end{array}
$$
where the last inequality is by  $y_{i}f(\mathbf{W}^{(0)},\mathbf{x}_{i}) \leq 2\beta$  and  $\eta \leq (CR_{\mathrm{max}}^2 /nm)^{-1}$ ,  $C$  is a sufficiently large constant. By taking  $x_{t} = y_{i}f(\mathbf{W}^{(t)},\mathbf{x}_{i}) - y_{i}f(\mathbf{W}^{(0)},\mathbf{x}_{i})$  and applying Lemma H.2 to (G.9), we can get
$$
\begin{array}{l} y _ {i} f \left(\mathbf {W} ^ {(t)}, \mathbf {x} _ {i}\right) \geq y _ {i} f \left(\mathbf {W} ^ {(0)}, \mathbf {x} _ {i}\right) + \log \left(1 + \frac {\eta \| \mathbf {x} _ {i} \| _ {2} ^ {2}}{5 (1 + e ^ {c}) e ^ {y _ {i} f \left(\mathbf {W} ^ {(0)} , \mathbf {x} _ {i}\right)} n m} \cdot t\right) \\ \geq y _ {i} f \left(\mathbf {W} ^ {(0)}, \mathbf {x} _ {i}\right) + \log \left(\frac {\eta \| \mathbf {x} _ {i} \| _ {2} ^ {2}}{5 (1 + e ^ {c}) e ^ {y _ {i} f \left(\mathbf {W} ^ {(0)} , \mathbf {x} _ {i}\right)} n m} \cdot t\right) \\ = \log t + \log \left(\frac {\eta \| \mathbf {x} _ {i} \| _ {2} ^ {2}}{n m}\right) - \log \left(5 (1 + e ^ {c})\right). \\ \end{array}
$$
$$
\frac {e ^ {y _ {i} f (\mathbf {W} ^ {(0)} , \mathbf {x} _ {i})} n m}{3 \eta \| \mathbf {x} _ {i} \| _ {2} ^ {2}} \exp \left(- \frac {3 \eta \| \mathbf {x} _ {i} \| _ {2} ^ {2}}{e ^ {y _ {i} f (\mathbf {W} ^ {(0)} , \mathbf {x} _ {i})} n m}\right) \leq \frac {e ^ {y _ {i} f (\mathbf {W} ^ {(0)} , \mathbf {x} _ {i})} n m}{3 \eta \| \mathbf {x} _ {i} \| _ {2} ^ {2}} \leq \frac {e ^ {2 \beta}}{3} \eta^ {- 1} R _ {\min } ^ {- 2} n m,
$$
$$
T = \left[ \frac {e ^ {2 \beta}}{3} \eta^ {- 1} R _ {\min } ^ {- 2} n m \right], C _ {3} = \max  \left\{\log 6 + 1, \log \left(5 \left(1 + e ^ {c}\right)\right) \right\}
$$
completes the proof of the first equation. By plugging the margin upper bound into (G.6), we can get
$$
\begin{array}{l} \left| \ell_ {i} ^ {\prime (t)} \right| \leq \exp \left(- y _ {i} f \left(\mathbf {W} _ {y _ {i}} ^ {(t)}, \mathbf {x} _ {i}\right)\right) \\ \leq \exp \left(- \log t - \log (\eta \| \mathbf {x} _ {i} \| _ {2} ^ {2} / n m) + C _ {3}\right) \\ \leq \exp (C _ {3}) \cdot \left(\frac {\eta \| \mathbf {x} _ {i} \| _ {2} ^ {2}}{n m} \cdot t\right) ^ {- 1}. \\ \end{array}
$$
By plugging the margin lower bound into (G.7), we can get
$$
\begin{array}{l} \left| \ell_ {i} ^ {\prime (t)} \right| \geq \frac {1}{1 + e ^ {c}} \cdot \exp \left(- y _ {i} f \left(\mathbf {W} _ {y _ {i}} ^ {(t)}, \mathbf {x} _ {i}\right)\right) \\ \geq \frac {1}{1 + e ^ {c}} \cdot \exp \left(- \log t - \log \left(\eta \| \mathbf {x} _ {i} \| _ {2} ^ {2} / n m\right) - C _ {3}\right) \\ \geq \frac {1}{e ^ {C _ {3}} (1 + e ^ {c})} \cdot \left(\frac {\eta \| \mathbf {x} _ {i} \| _ {2} ^ {2}}{n m} \cdot t\right) ^ {- 1}. \\ \end{array}
$$
Therefore, taking  $C_4 = 1 / e^{C_3}(1 + e^c)$  and  $C_5 = \exp (C_3)$  completes the proof.
Now we give the following lemma about the convergence rate of training loss.
Lemma G.4. For both two-layer ReLU and leaky ReLU networks defined in (G.1), we have the following convergence rate of training loss
$$
L _ {S} (\mathbf {W} ^ {(t)}) = \Theta (t ^ {- 1}).
$$
Proof. Having obtained a lower bound for the margin in Lemma G.3, we can now use it to derive an upper bound for the loss function as follows:
$$
\begin{array}{l} L _ {S} \left(\mathbf {W} ^ {(t)}\right) = \frac {1}{n} \sum_ {i = 1} ^ {n} \ell \left(y _ {i} f \left(\mathbf {W} ^ {(t)}, \mathbf {x} _ {i}\right)\right) \\ = \frac {1}{n} \sum_ {i = 1} ^ {n} \log \left(1 + \exp \left(- y _ {i} f \left(\mathbf {W} ^ {(t)}, \mathbf {x} _ {i}\right)\right)\right) \\ \leq \frac {1}{n} \sum_ {i = 1} ^ {n} \exp \left(- y _ {i} f \left(\mathbf {W} ^ {(t)}, \mathbf {x} _ {i}\right)\right) \\ \leq \frac {1}{n} \sum_ {i = 1} ^ {n} \exp \left(- \log t - \log \left(\eta \| \mathbf {x} _ {i} \| _ {2} ^ {2} / n m\right) + C _ {3}\right) \\ = \frac {1}{n} \sum_ {i = 1} ^ {n} \exp (C _ {3}) \cdot \left(\frac {\eta \| \mathbf {x} _ {i} \| _ {2} ^ {2}}{n m} \cdot t\right) ^ {- 1} \\ = O (t ^ {- 1}). \\ \end{array}
$$
where the first inequality is by  $\log (1 + z)\leq z$ ; the second inequality is by Lemma G.3.
Having obtained an upper bound for the margin in Lemma G.3, we can now use it to derive a lower bound for the loss function as follows:
$$
L _ {S} \left(\mathbf {W} ^ {(t)}\right) = \frac {1}{n} \sum_ {i = 1} ^ {n} \ell \left(y _ {i} f \left(\mathbf {W} ^ {(t)}, \mathbf {x} _ {i}\right)\right)
$$
$$
\begin{array}{l} = \frac {1}{n} \sum_ {i = 1} ^ {n} \log \left(1 + \exp \left(- y _ {i} f \left(\mathbf {W} ^ {(t)}, \mathbf {x} _ {i}\right)\right)\right) \\ \geq \frac {1}{n} \sum_ {i = 1} ^ {n} \exp \left(- y _ {i} f \left(\mathbf {W} ^ {(t)}, \mathbf {x} _ {i}\right)\right) - \exp \left(- 2 y _ {i} f \left(\mathbf {W} ^ {(t)}, \mathbf {x} _ {i}\right)\right) \\ \geq \frac {1}{n} \sum_ {i = 1} ^ {n} \exp \left(- \log t - \log \left(\eta \| \mathbf {x} _ {i} \| _ {2} ^ {2} / n m\right) - C _ {3}\right) \\ - \exp \left(- 2 \left(\log t + \log \left(\eta \| \mathbf {x} _ {i} \| _ {2} ^ {2} / n m\right) - C _ {3}\right)\right) \\ = \frac {1}{n} \sum_ {i = 1} ^ {n} \exp (- C _ {3}) \cdot \left(\frac {\eta \| \mathbf {x} _ {i} \| _ {2} ^ {2}}{n m} \cdot t\right) ^ {- 1} - \exp (2 C _ {3}) \cdot \left(\frac {\eta \| \mathbf {x} _ {i} \| _ {2} ^ {2}}{n m} \cdot t\right) ^ {- 2} \\ = \Omega (t ^ {- 1}). \\ \end{array}
$$
where the first inequality is by  $\log (1 + z) \geq z - z^2 / 2$  for  $z \geq 0$ ; the second inequality is by Lemma G.3. This completes the proof.
In addition to the aforementioned lemmas, in the case of leaky ReLU, assuming convergence in direction, we can demonstrate that the directional limit corresponds to a Karush-Kuhn-Tucker (KKT) point of the max-margin problem. This result is presented in the following lemma.
Lemma G.5. For two-layer leaky ReLU network defined in (G.1), assume that  $\mathbf{W}^{(t)}$  converges in direction, i.e. the limit of  $\mathbf{W}^{(t)} / \| \mathbf{W}^{(t)}\| _F$  exists. Denote  $\lim_{t\to \infty}\mathbf{W}^{(t)} / \| \mathbf{W}^{(t)}\| _F$  as  $\bar{\mathbf{W}}$ . There exists a scaling factor  $\alpha >0$  such that  $\alpha \bar{\mathbf{W}}$  satisfies Karush-Kuhn-Tucker (KKT) conditions of the following max-margin problem:
$$
\min  _ {\mathbf {W}} \frac {1}{2} \| \mathbf {W} \| _ {F} ^ {2}, \quad \text {s . t .} \quad y _ {i} f (\mathbf {W}, \mathbf {x} _ {i}) \geq 1, \forall i \in [ n ]. \tag {G.10}
$$
Proof. We need to prove that there exists  $\lambda_1, \dots, \lambda_n \geq 0$  such that for every  $j \in \{\pm 1\}$  and  $r \in [m]$  we have
$$
\bar {\mathbf {w}} _ {j, r} = \sum_ {i = 1} ^ {n} \lambda_ {i} \nabla_ {\mathbf {w} _ {j, r}} \left(y _ {i} f (\bar {\mathbf {W}}, \mathbf {x} _ {i})\right) = \sum_ {i = 1} ^ {n} \lambda_ {i} y _ {i} j \cdot \sigma^ {\prime} \left(\left\langle \bar {\mathbf {w}} _ {j, r}, \mathbf {x} _ {i} \right\rangle\right) \cdot \mathbf {x} _ {i}. \tag {G.11}
$$
By (5.1), we know that
$$
\begin{array}{l} \bar {\mathbf {w}} _ {j, r} = \lim  _ {t \rightarrow \infty} \frac {\mathbf {w} _ {j , r} ^ {(t)}}{\| \mathbf {W} ^ {(t)} \| _ {F}} \\ = \lim  _ {t \rightarrow \infty} \frac {\mathbf {w} _ {j , r} ^ {(0)}}{\| \mathbf {W} ^ {(t)} \| _ {F}} + \sum_ {i = 1} ^ {n} \frac {\rho_ {j , r , i} ^ {(t)}}{\| \mathbf {W} ^ {(t)} \| _ {F}} \cdot \| \mathbf {x} _ {i} \| _ {2} ^ {- 2} \cdot \mathbf {x} _ {i} \\ = \lim  _ {t \rightarrow \infty} \sum_ {i = 1} ^ {n} \frac {\rho_ {j , r , i} ^ {(t)}}{\| \mathbf {W} ^ {(t)} \| _ {F}} \cdot \| \mathbf {x} _ {i} \| _ {2} ^ {- 2} \cdot \mathbf {x} _ {i} \\ = \sum_ {i = 1} ^ {n} \lim  _ {t \rightarrow \infty} \frac {\rho_ {j , r , i} ^ {(t)}}{\| \mathbf {W} ^ {(t)} \| _ {F}} \cdot \| \mathbf {x} _ {i} \| _ {2} ^ {- 2} \cdot \mathbf {x} _ {i}, \\ \end{array}
$$
where the second equality is by  $\| \mathbf{W}^{(t)}\| _F = \Theta (\log t)$  and the last equality is by the existence of  $\lim_{t\to \infty}\mathbf{W}^{(t)} / \| \mathbf{W}^{(t)}\| _F$  as well as the uniqueness of data-correlated decomposition. By Lemma D.3 and  $\| \mathbf{W}^{(t)}\| _F = \Theta (\log t)$ , we can obtain that
$$
\lim  _ {t \rightarrow \infty} \frac {\rho_ {j , r , i} ^ {(t)}}{\| \mathbf {W} ^ {(t)} \| _ {F}} = \lim  _ {t \rightarrow \infty} \frac {\rho_ {j , r ^ {\prime} , i} ^ {(t)}}{\| \mathbf {W} ^ {(t)} \| _ {F}}, \tag {G.12}
$$
for any  $j\in \{\pm 1\}$ $r,r^{\prime}\in [m]$ $i\in [n]$  , and
$$
\lim  _ {t \rightarrow \infty} \frac {\rho_ {j , r , i} ^ {(t)}}{\| \mathbf {W} ^ {(t)} \| _ {F}} = - \gamma^ {- 1} \cdot \lim  _ {t \rightarrow \infty} \frac {\rho_ {j , r ^ {\prime} , i ^ {\prime}} ^ {(t)}}{\| \mathbf {W} ^ {(t)} \| _ {F}}, \tag {G.13}
$$
for any  $j\in \{\pm 1\}$ $i,i^{\prime}\in [n]$  with  $j = y_{i},j = -y_{i^{\prime}}$  and  $r,r^{\prime}\in [m]$  . Define
$$
S _ {j} = \{i \in [ n ]: y _ {i} = j \}, \lambda_ {i} ^ {\prime} := \lim _ {t \to \infty} \frac {\rho_ {y _ {i} , r , i} ^ {(t)}}{\| \mathbf {W} ^ {(t)} \| _ {F}}.
$$
By (G.12), we know  $\lambda_i^\prime$  is well defined and  $\lambda_i^\prime \geq 0$ . And by (G.13), we know that for any  $r\in [m]$ $i\in [n]$
$$
\lim  _ {t \rightarrow \infty} \frac {\rho_ {- y _ {i} , r , i} ^ {(t)}}{\| \mathbf {W} ^ {(t)} \| _ {F}} = - \gamma \lambda_ {i} ^ {\prime}.
$$
$$
\begin{array}{l} \bar {\mathbf {w}} _ {j, r} = \sum_ {i = 1} ^ {n} \lim _ {t \to \infty} \frac {\rho_ {j , r , i} ^ {(t)}}{\| \mathbf {W} ^ {(t)} \| _ {F}} \cdot \| \mathbf {x} _ {i} \| _ {2} ^ {- 2} \cdot \mathbf {x} _ {i} \\ = \sum_ {i \in S _ {j}} \lim _ {t \to \infty} \frac {\rho_ {j , r , i} ^ {(t)}}{\| \mathbf {W} ^ {(t)} \| _ {F}} \cdot \| \mathbf {x} _ {i} \| _ {2} ^ {- 2} \cdot \mathbf {x} _ {i} + \sum_ {i \in S _ {- j}} \lim _ {t \to \infty} \frac {\rho_ {j , r , i} ^ {(t)}}{\| \mathbf {W} ^ {(t)} \| _ {F}} \cdot \| \mathbf {x} _ {i} \| _ {2} ^ {- 2} \cdot \mathbf {x} _ {i} \\ = \sum_ {i \in S _ {j}} \lambda_ {i} ^ {\prime} \cdot \| \mathbf {x} _ {i} \| _ {2} ^ {- 2} \cdot \mathbf {x} _ {i} - \sum_ {i \in S _ {- j}} \gamma \lambda_ {i} ^ {\prime} \cdot \| \mathbf {x} _ {i} \| _ {2} ^ {- 2} \cdot \mathbf {x} _ {i}. \\ \end{array}
$$
By Lemma C.5, it holds for any  $t \geq T_1$  that
$$
\sigma^ {\prime} (\langle \mathbf {w} _ {j, r} ^ {(t)}, \mathbf {x} _ {i} \rangle) = 1,
$$
$$
\forall j \in \{\pm 1 \}, i \in S _ {j},
$$
$$
\sigma^ {\prime} (\langle \mathbf {w} _ {j, r} ^ {(t)}, \mathbf {x} _ {i} \rangle) = \gamma ,
$$
$$
\forall j \in \{\pm 1 \}, i \in S _ {- j}.
$$
$$
\sigma^ {\prime} (\langle \bar {\mathbf {w}} _ {j, r}, \mathbf {x} _ {i} \rangle) = 1,
$$
$$
\sigma^ {\prime} (\langle \bar {\mathbf {w}} _ {j, r}, \mathbf {x} _ {i} \rangle) = \gamma ,
$$
$$
\begin{array}{l} \bar {\mathbf {w}} _ {j, r} = \sum_ {i \in S _ {j}} \lambda_ {i} ^ {\prime} \cdot \sigma^ {\prime} (\langle \bar {\mathbf {w}} _ {j, r}, \mathbf {x} _ {i} \rangle) \cdot \| \mathbf {x} _ {i} \| _ {2} ^ {- 2} \cdot \mathbf {x} _ {i} - \sum_ {i \in S _ {- j}} \lambda_ {i} ^ {\prime} \cdot \sigma^ {\prime} (\langle \bar {\mathbf {w}} _ {j, r}, \mathbf {x} _ {i} \rangle) \cdot \| \mathbf {x} _ {i} \| _ {2} ^ {- 2} \cdot \mathbf {x} _ {i} \\ = \sum_ {i = 1} ^ {n} \lambda_ {i} ^ {\prime} y _ {i} j \cdot \sigma^ {\prime} (\langle \bar {\mathbf {w}} _ {j, r}, \mathbf {x} _ {i} \rangle) \cdot \| \mathbf {x} _ {i} \| _ {2} ^ {- 2} \cdot \mathbf {x} _ {i}. \\ \end{array}
$$
Taking  $\lambda_{i} = \lambda_{i}^{\prime}\| \mathbf{x}_{i}\|_{2}^{-2}$  completes the proof of (G.11). On the other hand, by Lemma G.2 and the assumption of the existence of  $\mathbf{W}^{(t)} / \| \mathbf{W}^{(t)}\| _F$ , we can get
$$
y _ {i} f (\bar {\mathbf {W}}, \mathbf {x} _ {i}) = y _ {k} f (\bar {\mathbf {W}}, \mathbf {x} _ {k}),
$$
for any  $i, k \in [n]$ . Taking  $\alpha = 1 / y_i f(\bar{\mathbf{W}}, \mathbf{x}_i)$ , we have
$$
y _ {i} f (\alpha \bar {\mathbf {W}}, \mathbf {x} _ {i}) = 1,
$$
for any  $i \in [n]$ , which completes the proof.
H Auxiliary Lemmas
Lemma H.1. Let  $\{x_{t}\}_{t = 0}^{\infty}$  be an non-negative sequence satisfying the following inequality:
$$
x _ {t + 1} - x _ {t} \leq C \cdot e ^ {- x _ {t}}, \forall t \geq 0
$$
$$
x _ {t} \leq \log \left(e ^ {x _ {0}} + C e ^ {C} \cdot t\right).
$$
Proof of Lemma H.1. Given the inequality  $x_{t + 1} - x_t \leq C \cdot e^{-x_t}$  for all  $t \geq 0$ , we want to prove that  $x_T \leq \log (e^{x_0} + Ce^C \cdot T)$  for  $T \geq 0$ . We start by manipulating the inequality as follows:
$$
\begin{array}{l} x _ {t + 1} - x _ {t} \leq C \cdot e ^ {- x _ {t}} \\ \Rightarrow x _ {t + 1} - x _ {t} \leq C \cdot e ^ {- x _ {t + 1} + C} \quad (\text {u s i n g} x _ {t + 1} \text {i n s t e a d} x _ {t}) \\ \Rightarrow \quad e ^ {x _ {t + 1}} \left(x _ {t + 1} - x _ {t}\right) \leq C e ^ {C} \quad (\text {m u l t i p l y i n g b o t h s i d e s b y} e ^ {x _ {t + 1}}). \\ \end{array}
$$
Summing the inequality from  $t = 0$  to  $t = T - 1$ , we get:
$$
\sum_ {t = 0} ^ {T - 1} e ^ {x _ {t + 1}} \left(x _ {t + 1} - x _ {t}\right) \leq C e ^ {C} \cdot T.
$$
Since  $e^x$  is a monotone increasing function, we can approximate the above sum with an integral:
$$
\int_ {x _ {0}} ^ {x _ {T}} e ^ {x} d x \leq C e ^ {C} \cdot T.
$$
Evaluating the integral, we get:
$$
e ^ {x _ {T}} - e ^ {x _ {0}} \leq C e ^ {C} \cdot T.
$$
Rearranging the inequality, we get:
$$
e ^ {x _ {T}} \leq e ^ {x _ {0}} + C e ^ {C} \cdot T.
$$
Taking the natural logarithm of both sides, we get:
$$
x _ {T} \leq \log \left(e ^ {x _ {0}} + C e ^ {C} \cdot T\right).
$$
Therefore, we have shown that  $x_{T} \leq \log (e^{x_{0}} + Ce^{C} \cdot T)$ , as required.
Lemma H.2. Let  $\{x_{t}\}_{t = 0}^{\infty}$  be an sequence satisfying the following inequality:
$$
x _ {t + 1} - x _ {t} \geq C \cdot e ^ {- x _ {t}}, \forall t \geq 0
$$
$$
x _ {t} \geq \log \left(e ^ {x _ {0}} + C \cdot t\right).
$$
Proof of Lemma H.2. Given the inequality  $x_{t + 1} - x_t \geq C \cdot e^{-x_t}$  for all  $t \geq 0$ , we want to prove that  $x_T \geq \log (e^{x_0} + C \cdot T)$  for  $T \geq 0$ . We start by manipulating the inequality as follows: