SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Step III. A rough two-side bound for  $\mathcal{I}(t) / \mathcal{J}(t)$ .
In Step II, we have given a rough upper bound for  $\mathcal{I}(t) / \mathcal{J}(t)$ :  $\mathcal{I}(t) / \mathcal{J}(t) < 1$ ,  $\forall t \geq T_{\mathrm{III}}$ . And we want to derive a lower bound for  $\mathcal{I}(t) / \mathcal{J}(t)$  in this step. We aim to prove:
$$
\mathcal {I} (t) / \mathcal {J} (t) > \frac {1 + \cos \Delta}{1 + \cos \Delta + \epsilon}, \quad \forall t \in [ T _ {\mathrm {I I I}}, + \infty).
$$
First, we define the hitting time:
$$
\tau_ {\mathcal {I} / \mathcal {I}} ^ {t} := \inf  \left\{t \geq T _ {\mathrm {I I I}}: \mathcal {I} (t) \leq \frac {1 + \cos \Delta}{1 + \cos \Delta + \epsilon} \mathcal {J} (t) \right\}.
$$
From Step I, we know
$$
\begin{array}{l} \mathcal {I} \left(T _ {\mathrm {I I I}}\right) - \frac {1 + \cos \Delta}{1 + \cos \Delta + \epsilon} \mathcal {J} \left(T _ {\mathrm {I I I}}\right) > \left(\cos \Delta - \frac {1 + \cos \Delta}{1 + \cos \Delta + \epsilon}\right) \mathcal {J} \left(T _ {\mathrm {I I I}}\right) \\ = \frac {\left(\frac {m _ {+}}{m _ {-}} \cos \Delta - 1\right) \sin^ {2} \Delta}{1 + \cos \Delta + \epsilon} \mathcal {J} (T _ {\mathrm {I I I}}) \geq \frac {\left(\frac {\cos \Delta}{0 . 9 7 7} - 1\right) \sin^ {2} \Delta}{1 + \cos \Delta + \epsilon} \mathcal {J} (T _ {\mathrm {I I I}}) \\ \geq \frac {\left(\frac {0 . 9 8 0}{0 . 9 7 7} - 1\right) \sin^ {2} \Delta}{1 + \cos \Delta + \epsilon} \mathcal {J} (T _ {\mathrm {I I I}}) > 0. \\ \end{array}
$$
From the continuity,  $\tau_{\mathcal{I} / \mathcal{J}}^l$  exists and  $\tau_{\mathcal{I} / \mathcal{J}}^l > T_{\mathrm{III}}$ .
For any  $t \in [T_{\mathrm{III}}, \tau_{\mathcal{I} / \mathcal{J}}^l)$ , we have  $\mathcal{I}(t) - \frac{1 + \cos\Delta}{1 + \cos\Delta + \epsilon}\mathcal{J}(t) > 0$  and
$$
\begin{array}{l} \frac {\mathrm {d}}{\mathrm {d} t} \left(\mathcal {I} (t) - \frac {1 + \cos \Delta}{1 + \cos \Delta + \epsilon} \mathcal {J} (t)\right) \\ = \mathcal {I} (t) \mathcal {J} (t) \cos \Delta \left(1 - \frac {1 + \cos \Delta}{1 + \cos \Delta + \epsilon}\right) - (1 + \epsilon) \mathcal {I} ^ {2} (t) + \frac {1 + \cos \Delta}{1 + \cos \Delta + \epsilon} \mathcal {J} ^ {2} (t) \\ = - \left(\mathcal {I} (t) - \frac {1 + \cos \Delta}{1 + \cos \Delta + \epsilon} \mathcal {J} (t)\right) ((1 + \epsilon) \mathcal {I} (t) + \mathcal {J} (t)) \\ > - (1 + \epsilon) \left(\mathcal {I} (t) - \frac {1 + \cos \Delta}{1 + \cos \Delta + \epsilon} \mathcal {J} (t)\right) (\mathcal {I} (t) + \mathcal {J} (t)) \\ \geq - (1 + \epsilon) (\mathcal {I} (T _ {\mathrm {I I I}}) + \mathcal {J} (T _ {\mathrm {I I I}})) \left(\mathcal {I} (t) - \frac {1 + \cos \Delta}{1 + \cos \Delta + \epsilon} \mathcal {J} (t)\right). \\ \end{array}
$$
We consider the auxiliary ODE:  $\frac{d}{dt}\mathcal{Q}(t) = -(1 + \epsilon)(\mathcal{I}(T_{\mathrm{III}}) + \mathcal{J}(T_{\mathrm{III}}))\mathcal{Q}(t)$ , where  $\mathcal{Q}(T_{\mathrm{III}}) = \mathcal{I}(T_{\mathrm{III}}) - \frac{1 + \cos\Delta}{1 + \cos\Delta + \epsilon}\mathcal{J}(T_{\mathrm{III}}) > 0$ . From the Comparison Principle of ODEs, we have:
$$
\mathcal {I} (t) - \frac {1 + \cos \Delta}{1 + \cos \Delta + \epsilon} \mathcal {J} (t) \geq \mathcal {Q} (t)
$$
$$
= \left(\mathcal {I} (T _ {\mathrm {I I I}}) - \frac {1 + \cos \Delta}{1 + \cos \Delta + \epsilon} \mathcal {J} (T _ {\mathrm {I I I}})\right) \exp \Big (- (1 + \epsilon) (\mathcal {I} (T _ {\mathrm {I I I}}) + \mathcal {J} (T _ {\mathrm {I I I}})) (t - T _ {\mathrm {I I I}}) \Big) > 0, \forall t \in [ T _ {\mathrm {I I I}}, \tau_ {\mathcal {I} / \mathcal {J}} ^ {l}).
$$
$$
\tau_ {\mathcal {I} / \mathcal {I}} ^ {l} = + \infty ;
$$
Hence, we obtain the two-side bound for  $\mathcal{I}(t) / \mathcal{J}(t)$ :
$$
\frac {1 + \cos \Delta}{1 + \cos \Delta + \epsilon} <   \frac {\mathcal {I} (t)}{\mathcal {J} (t)} <   1, \quad \forall t \in [ T _ {\mathrm {I I I}}, + \infty).
$$
Step IV.  $\mathcal{I}(t)\cos \Delta -\mathcal{J}(t)$  and  $\mathcal{I}(t) - \mathcal{J}(t)\cos \Delta$  are both negative.
The estimate on  $\mathcal{I}(t)\cos \Delta -\mathcal{J}(t)$  is straight-forward:
$$
\mathcal {I} (t) \cos \Delta - \mathcal {J} (t) <   \mathcal {I} (t) \cos \Delta - \mathcal {I} (t) <   0.
$$
As for  $\mathcal{I}(t) - \mathcal{J}(t)\cos \Delta$ , we will actually prove a tighter upper bound:
$$
\frac {\mathcal {I} (t)}{\mathcal {J} (t)} <   \cos \Delta .
$$
We need to do finer analysis using the specific dynamics (31). First, we define the following hitting time:
Define the following hitting time
$$
\tau_ {\mathcal {I} / \mathcal {J}} ^ {u} := \inf  \left\{t > T _ {\mathrm {I}}: \mathcal {I} (t) - \mathcal {J} (t) \cos \Delta \geq 0 \right\}.
$$
From  $\mathcal{I}(T_{\mathrm{III}}) - \mathcal{J}(T_{\mathrm{III}})\cos \Delta = 0$ ,  $\frac{\mathrm{d}}{\mathrm{d}t} (\mathcal{I}(T_{\mathrm{III}}) - \mathcal{J}(T_{\mathrm{III}})\cos \Delta) < 0$  and the continuity, we know  $\tau_{\mathcal{I}/\mathcal{J}}^u$  exists and  $\tau_{\mathcal{I}/\mathcal{J}}^u > T_{\mathrm{III}}$ .
Recalling  $\mathcal{I}(t)\cos \Delta -\mathcal{J}(t) <   0$  , we have
$$
\frac {\mathrm {d}}{\mathrm {d} t} \mathcal {I} (t) = \mathcal {I} (t) (\mathcal {I} (t) \cos \Delta - \mathcal {J} (t)) <   0, \quad \forall t \geq T _ {\mathrm {I I I}}.
$$
So we can consider the following dynamics for  $t \in [T_{\mathrm{III}}, \tau_{\mathcal{I} / \mathcal{J}}^u]$ :
$$
\frac {\mathrm {d} \mathcal {I}}{\mathrm {d} \mathcal {J}} = \frac {\mathcal {I} \mathcal {J} \cos \Delta - \mathcal {I} ^ {2} (1 + \epsilon)}{\mathcal {I} \mathcal {J} \cos \Delta - \mathcal {J} ^ {2}} = \frac {\frac {\mathcal {I}}{\mathcal {J}} \cos \Delta - \left(\frac {\mathcal {I}}{\mathcal {J}}\right) ^ {2} (1 + \epsilon)}{\frac {\mathcal {I}}{\mathcal {J}} \cos \Delta - 1}.
$$
If we define  $\mathcal{Z}(t) \coloneqq \frac{\mathcal{I}(t)}{\mathcal{J}(t)}$ , then we have  $\mathrm{d}\mathcal{I} = \mathcal{Z}\mathrm{d}\mathcal{J} + \mathcal{J}\mathrm{d}\mathcal{Z}$ .
$$
\mathcal {J} \frac {\mathrm {d} \mathcal {Z}}{\mathrm {d} \mathcal {J}} = \frac {\mathcal {Z} \cos \Delta - \mathcal {Z} ^ {2} (1 + \epsilon)}{\mathcal {Z} \cos \Delta - 1} - \mathcal {Z} = \frac {\mathcal {Z} (1 + \cos \Delta) - \mathcal {Z} ^ {2} (1 + \cos \Delta + \epsilon)}{\mathcal {Z} \cos \Delta - 1}.
$$
Recalling the result in Step III, we have  $(1 + \cos \Delta) - (1 + \cos \Delta + \epsilon)\mathcal{Z}(t) < 0$  holds for any  $t \geq T_{\mathrm{III}}$ . So the dynamics is equal to:
$$
\begin{array}{l} \frac {\mathrm {d} \mathcal {J}}{\mathcal {J}} = \left(\frac {\mathcal {Z} \cos \Delta - 1}{\mathcal {Z} (1 + \cos \Delta) - \mathcal {Z} ^ {2} (1 + \cos \Delta + \epsilon)}\right) \mathrm {d} \mathcal {Z} \\ = - \frac {1}{1 + \cos \Delta} \left(\frac {1}{\mathcal {Z}} + \frac {\sin^ {2} \Delta + \epsilon}{1 + \cos \Delta - \mathcal {Z} (1 + \cos \Delta + \epsilon)}\right) d \mathcal {Z}. \\ \end{array}
$$
Integrating this equation from  $T_{\mathrm{III}}$  to  $t \in [T_{\mathrm{III}}, \tau_{\mathcal{I} / \mathcal{J}}^u)$ , we have:
$$
\begin{array}{l} \log \left(\frac {\mathcal {J} (t)}{\mathcal {J} (T _ {\mathrm {I I I}})}\right) = - \frac {1}{1 + \cos \Delta} \log \left(\frac {\mathcal {Z} (t)}{\mathcal {Z} (T _ {\mathrm {I I I}})}\right) \\ + \frac {\sin^ {2} \Delta + \epsilon}{(1 + \cos \Delta + \epsilon) (1 + \cos \Delta)} \log \left(\frac {(1 + \cos \Delta + \epsilon) \mathcal {Z} (t) - (1 + \cos \Delta)}{(1 + \cos \Delta + \epsilon) \mathcal {Z} (T _ {\mathrm {I}}) - (1 + \cos \Delta)}\right). \tag {32} \\ \end{array}
$$
If we assume  $\tau_{\mathcal{I} / \mathcal{J}}^u < + \infty$  , the continuity gives us
$$
\lim _ {t \to \tau_ {\mathcal {I} / \mathcal {J}} ^ {u} -} \mathcal {Z} (t) = \lim _ {t \to \tau_ {\mathcal {I} / \mathcal {J}} ^ {u} -} \mathcal {I} (t) / \mathcal {J} (t) = \cos \Delta = \mathcal {I} (T _ {\mathrm {I I I}}) / \mathcal {J} (T _ {\mathrm {I I I}}) = \mathcal {Z} (T _ {\mathrm {I I I}}).
$$
Then letting  $t\to \tau_{\mathcal{I} / \mathcal{I}}^u$  in (32), we have
$$
\lim  _ {t \to \tau_ {\mathcal {I} / \mathcal {J}} ^ {u} -} \log \left(\frac {\mathcal {J} (t)}{\mathcal {J} (T _ {\mathrm {I I I}})}\right) = 0 + 0 = 0,
$$
which means  $\mathcal{J}(\tau_{\mathcal{I} / \mathcal{J}}^u) = \mathcal{J}(T_{\mathrm{III}})$ .
But on the other hand, we have:
$$
\begin{array}{l} \mathcal {J} \left(\tau_ {\mathcal {I} / \mathcal {J}} ^ {u}\right) = \mathcal {J} \left(T _ {\mathrm {I I I}}\right) + \int_ {T _ {\mathrm {I I I}}} ^ {\tau_ {\mathcal {I} / \mathcal {J}} ^ {u}} (\mathcal {I} (t) \mathcal {J} (t) \cos \Delta - \mathcal {J} ^ {2} (t)) \mathrm {d} t \\ = \mathcal {J} \left(T _ {\mathrm {I I I}}\right) + \int_ {T _ {\mathrm {I I I}}} ^ {\tau_ {\mathcal {I} / \mathcal {I}} ^ {u}} \mathcal {J} (t) (\mathcal {I} (t) \cos \Delta - \mathcal {J} (t)) \mathrm {d} t \\ <   \mathcal {J} (T _ {\mathrm {I I I}}) + (\cos \Delta - 1) \int_ {T _ {\mathrm {I I I}}} ^ {\tau_ {\mathcal {I} / \mathcal {I}} ^ {u}} \mathcal {J} (t) \mathcal {I} (t) \mathrm {d} t <   \mathcal {J} (T _ {\mathrm {I I I}}), \\ \end{array}
$$
which leads to a contradiction. Hence, we have proved
$$
\tau_ {\mathcal {I} / \mathcal {I}} ^ {u} = + \infty ;
$$
$$
\mathcal {I} (t) - \mathcal {J} (t) \cos \Delta <   0, \quad \forall t \in [ T _ {\mathrm {H I}}, + \infty).
$$
Moreover, we obtain a sharper two-side bound for  $\mathcal{I}(t) / \mathcal{J}(t)$ :
$$
\frac {1 + \cos \Delta}{1 + \cos \Delta + \epsilon} <   \frac {\mathcal {I} (t)}{\mathcal {J} (t)} <   \cos \Delta , \quad \forall t \in [ T _ {\mathrm {I I I}}, + \infty). \tag {33}
$$
Step V. Tight bound for  $\mathcal{I}(t)$  and  $\mathcal{J}(t)$ .
In this step, we aim to give a tight bound for  $\mathcal{I}(t) + \mathcal{J}(t)$ . With the help of the two-side bound (33), we have
$$
\begin{array}{l} \frac {\mathrm {d}}{\mathrm {d} t} \Big (\mathcal {I} (t) + \mathcal {J} (t) \Big) = 2 \mathcal {I} (t) \mathcal {J} (t) \cos \Delta - \mathcal {I} ^ {2} (t) (1 + \epsilon) - \mathcal {J} ^ {2} (t) \\ = - \left(\mathcal {I} (t) - \mathcal {J} (t)\right) ^ {2} \cos \Delta - (1 - \cos \Delta) \mathcal {J} ^ {2} (t) - \mathcal {I} ^ {2} (t) (1 + \epsilon - \cos \Delta) \\ <   - \left(1 - \cos \Delta\right) \left(\mathcal {I} ^ {2} (t) + \mathcal {I} ^ {2} (t)\right) <   - \frac {\left(1 - \cos \Delta\right) \left(\mathcal {I} (t) + \mathcal {J} (t)\right) ^ {2}}{2} <   - \frac {\Delta^ {2}}{6} \left(\mathcal {I} (t) + \mathcal {J} (t)\right) ^ {2}, \\ \frac {\mathrm {d}}{\mathrm {d} t} \Big (\mathcal {I} (t) + \mathcal {J} (t) \Big) = 2 \mathcal {I} (t) \mathcal {J} (t) \cos \Delta - \mathcal {I} ^ {2} (t) (1 + \epsilon) - \mathcal {J} ^ {2} (t) \\ = - \left(\mathcal {I} (t) - \mathcal {J} (t)\right) ^ {2} \cos \Delta - (1 - \cos \Delta) \mathcal {J} ^ {2} (t) - \mathcal {I} ^ {2} (t) (1 + \epsilon - \cos \Delta) \\ > - \left(1 - \frac {1 + \cos \Delta}{1 + \cos \Delta + \epsilon}\right) ^ {2} \mathcal {J} ^ {2} (t) - (1 - \cos \Delta) \mathcal {J} ^ {2} (t) - \mathcal {I} ^ {2} (t) (1 + \epsilon - \cos \Delta) \\ > - \left(1 + \epsilon - \cos \Delta\right) \left(\mathcal {I} ^ {2} (t) + \mathcal {J} ^ {2} (t)\right) > - \left(\frac {2}{3} + \frac {m _ {+}}{m _ {-}}\right) \Delta^ {2} \left(\mathcal {I} (t) + \mathcal {J} (t)\right) ^ {2} > - 2 \Delta^ {2} \left(\mathcal {I} (t) + \mathcal {J} (t)\right) ^ {2}. \\ \end{array}
$$
For the first inequality, we consider the auxiliary ODE:  $\frac{d}{dt}\mathcal{P}(t) = -\frac{\Delta^2}{6}\mathcal{P}^2 (t)$ , where  $\mathcal{P}(T_{\mathrm{III}}) = \mathcal{I}(T_{\mathrm{III}}) + \mathcal{J}(T_{\mathrm{III}}) > 0$ . From the Comparison Principle of ODEs, we have:
$$
\mathcal {I} (t) + \mathcal {J} (t) \leq \mathcal {P} (t) = \frac {1}{\overline {{\mathcal {I} (T _ {\mathrm {I I I}}) + \mathcal {J} (T _ {\mathrm {I I I}})}}} + \frac {\Delta^ {2}}{6} (t - T _ {\mathrm {I I I}})}, \quad \forall t \geq T _ {\mathrm {I I I}}.
$$
In the same way, we can obtain the lower bound:
$$
\mathcal {I} (t) + \mathcal {J} (t) \geq \frac {1}{\overline {{\mathcal {I} (T _ {\mathrm {I I I}}) + \mathcal {J} (T _ {\mathrm {I I I}})}}} + 2 \Delta^ {2} (t - T _ {\mathrm {I I I}})}, \quad \forall t \geq T _ {\mathrm {I I I}}.
$$
Recalling Step I, we have  $\frac{1}{\mathcal{I}(T_{\mathrm{III}}) + \mathcal{J}(T_{\mathrm{III}})} = \Theta \left(p^{\frac{1}{1 - \alpha\cos\Delta}} / \kappa_2^2\right)$ . Hence, we obtain the tight bound:
$$
\mathcal {I} (t) + \mathcal {J} (t) = \Theta \left(\frac {1}{\frac {\frac {1}{p ^ {1 - \alpha \cos \Delta}}}{\kappa_ {2} ^ {2}} + \Delta^ {2} (t - T _ {\mathrm {I I I}})}\right), \quad \forall t \in [ T _ {\mathrm {I I I}}, + \infty).
$$
Taking (33) into the equation above, we have:
Step VI. The limit of  $\mathcal{I}(t) / \mathcal{J}(t)$ .
By Step V and the proof of Step IV, we know  $\lim_{t\to \infty}\mathcal{J}(t) = 0$  and  $\frac{\mathrm{d}}{\mathrm{d}t}\mathcal{J}(t) < 0$  holds for any  $t > T_{\mathrm{III}}$ .
Then for any  $\epsilon' > 0$ , there exists  $T' > T_{\mathrm{III}}$  such that
$$
\log \left(\frac {\mathcal {J} (t)}{\mathcal {J} (T _ {\mathrm {I I I}})}\right) <   \frac {1 0 0 0}{\Delta^ {2}} \log (1 0 0 0 \epsilon \Delta^ {2}), \quad \forall t > T ^ {\prime}.
$$
Taking it into (32), we obtain that for any  $t > T'$
$$
0 <   (1 + \cos \Delta + \epsilon) \mathcal {Z} (t) - (1 + \cos \Delta) <   \epsilon^ {\prime}.
$$
By the definition of the limit, we get
$$
\lim  _ {t \rightarrow \infty} \frac {\mathcal {I} (t)}{\mathcal {J} (t)} = \frac {1 + \cos \Delta}{1 + \cos \Delta + \epsilon} = \frac {1 + \cos \Delta}{1 + \cos \Delta + \frac {m _ {+}}{m _ {-}} \sin^ {2} \Delta}.
$$
Lemma F.5 (Time and prediction estimate).
(S1). For any  $t \in (T_{\mathrm{III}}, + \infty)$
$$
p e ^ {- f _ {+} (t)} = \Theta \left(\frac {1}{p ^ {\frac {1}{1 - \alpha \cos \Delta}} + \kappa_ {2} ^ {2} \Delta^ {2} (t - T _ {\mathrm {I I I}})}\right), e ^ {f _ {-} (t)} = \Theta \left(\frac {1}{p ^ {\frac {1}{1 - \alpha \cos \Delta}} + \kappa_ {2} ^ {2} \Delta^ {2} (t - T _ {\mathrm {I I I}})}\right);
$$
$$
\mathcal {L} (\pmb {\theta} (t)) = \Theta \left(\frac {1}{p ^ {\frac {1}{1 - \alpha \cos \Delta}} + \kappa_ {2} ^ {2} \Delta^ {2} (t - T _ {\mathrm {I I I}})}\right).
$$
(S2). For any  $t \in (T_{\mathrm{III}}, + \infty)$ ,
$$
\frac {1 + \cos \Delta}{1 + \cos \Delta + \frac {m _ {+}}{m _ {-}} \sin^ {2} \Delta} <   p e ^ {- (f _ {+} (t) + f _ {-} (t))} <   \cos \Delta .
$$
Moreover,  $p e^{-(f_{+}(T_{\mathrm{III}}) + f_{-}(T_{\mathrm{III}}))} = \cos \Delta$  and  $\lim_{t\to \infty}p e^{-(f_{+}(t) + f_{-}(t))} = \frac{1 + \cos\Delta}{1 + \cos\Delta + \frac{m_{+}}{m_{-}}\sin^2\Delta}$ .
(S3). For any  $t \in (T_{\mathrm{III}}, + \infty)$ ,
$$
\langle \boldsymbol {b} _ {k} (t), \boldsymbol {x} _ {+} \rangle > 0, \langle \boldsymbol {b} _ {k} (t), \boldsymbol {x} _ {-} \rangle = 0, k \in \mathcal {K} _ {+};
$$
$$
\langle \boldsymbol {b} _ {k} (t), \boldsymbol {x} _ {+} \rangle > 0, \langle \boldsymbol {b} _ {k} (t), \boldsymbol {x} _ {-} \rangle > 0, k \in \mathcal {K} _ {-}.
$$
(S4) (Time).
$$
T _ {\mathrm {I V}} = T _ {\mathrm {I V}} ^ {*} = + \infty .
$$
Proof of Lemma F.5.
Notice the relationships:  $pe^{-f_{+}(t)} = \kappa_{2}^{2}\frac{m_{-}}{m}\frac{\mathcal{I}(t)}{1 + p}, e^{f_{-}(t)} = \kappa_{2}^{2}\frac{m_{-}}{m}\frac{\mathcal{J}(t)}{1 + p}$  and  $pe^{-(f_{+}(t) + f_{-}(t))} = \mathcal{I}(t) / \mathcal{J}(t)$ . Then Lemma F.4 implies that  $T_{\mathrm{IV}}^* = +\infty$ . Recalling the dynamics in Lemma F.1, then lemma (S3)(S4) hold. Then using Lemma F.4 again, we obtain (S1)(S2).
Proof of Theorem 4.9.
Theorem 4.9 (S1), (S2), and (S3) are obtained in Lemma F.5 (S4), (S3), and (S1), respectively.
F.2 Asymptotic Directional Convergence
In this section, we will study the final convergence direction in our setting. It mainly depends on our prior fine-grained analysis of the training dynamics in Phase IV and the following result about the final convergence direction at the end of training.
Lemma F.6. Let  $f(\cdot; \theta)$  be a homogeneous neural network parameterized by  $\theta$ . Consider minimizing the exponential loss over a binary classification dataset  $\{(x_i, y_i)\}_{i=1}^n$  ( $\| x_i \|_2 \leq 1$ ,  $y_i \in \{\pm 1\}$ ) using Gradient Flow. Assume that there exists time  $t_0$  such that  $\mathcal{L}(\theta(t_0)) < \frac{1}{n}$ . Then,
(I) (Paraphrased from (Lyu and Li, 2019; Ji and Telgarsky, 2020)).  $\pmb{\theta}(t)$  converges in direction to a KKT point (Definition G.3) of the following maximum margin problem:
$$
\min : \frac {1}{2} \| \boldsymbol {\theta} \| _ {2} ^ {2}
$$
$$
\begin{array}{l} \text {s . t .} y _ {i} f (\boldsymbol {x} _ {i}; \boldsymbol {\theta}) \geq 1. \end{array}
$$
$(II)$  (Lyu and Li, 2019; Ji and Telgarsky, 2020)).  $\| \pmb {\theta}(t)\| _2\to \infty$  and  $\mathcal{L}(\pmb {\theta}(t))\rightarrow 0$
(III) (Ji and Telgarsky (2020)).  $-\nabla \mathcal{L}(\pmb{\theta}(t))$  and  $\pmb{\theta}(t)$  converge to the same direction, meaning the angle between  $\pmb{\theta}(t)$  and  $-\nabla \mathcal{L}(\pmb{\theta}(t))$  converges to 0.
Lemma F.7 (Final Convergence Direction). The limit  $\lim_{t\to +\infty}\frac{\pmb{\theta}(t)}{\|\pmb{\theta}(t)\|_2}$  exists, and denoted by  $\overline{\pmb{\theta}} = (\bar{b}_1^\top ,\dots ,\bar{b}_m^\top)^\top \in \mathbb{S}^{md - 1}$ , then it satisfies
$$
\bar {b} _ {k} = \mathbf {0}, \quad \forall k \notin \mathcal {K} _ {+} \cup \mathcal {K} _ {-};
$$
$$
\bar {b} _ {k} = C \left(\boldsymbol {x} _ {+} - \boldsymbol {x} _ {-} \cos \Delta\right), \quad \forall k \in \mathcal {K} _ {+};
$$
$$
\overline {{\boldsymbol {b}}} _ {k} = C \left(\left(1 + \frac {m _ {+} \sin^ {2} \Delta}{m _ {-} (1 + \cos \Delta)}\right) \boldsymbol {x} _ {-} - \boldsymbol {x} _ {+}\right), \quad \forall k \in \mathcal {K} _ {-};
$$
where  $C > 0$  is a scaling constant such that  $\left\| \overline{\pmb{\theta}}\right\| _2 = 1$ . Moreover,  $f(\pmb{x}_{+};\overline{\pmb{\theta}}) = -f(\pmb{x}_{-};\overline{\pmb{\theta}}) > 0$ .
Proof of Lemma F.7.
Let  $\overline{\pmb{\theta}} = (\overline{b}_1^\top, \dots, \overline{b}_m^\top)^\top \in \mathbb{S}^{md-1}$  be the limits point of  $\left\{\frac{\pmb{\theta}(t)}{\|\pmb{\theta}(t)\|_2}: t \geq t_0\right\}$ . From Lemma F.6 (I), we know that there exists a scaling factor  $\alpha > 0$  such that  $\alpha \overline{\pmb{\theta}}$  satisfies KKT conditions (Definition G.3) of the maximum-margin problem
$$
\min : \frac {1}{2} \| \boldsymbol {\theta} \| _ {2} ^ {2} \tag {34}
$$
$$
\mathrm {s . t .} f \left(\boldsymbol {x} _ {+}; \boldsymbol {\theta}\right) \geq 1, f \left(\boldsymbol {x} _ {-}; \boldsymbol {\theta}\right) \leq - 1.
$$
For simplicity, we denote  $\pmb{\theta}^{*} \coloneqq \alpha \bar{\pmb{\theta}}$ , where
$$
\boldsymbol {\theta} ^ {*} = \left(\boldsymbol {b} _ {1} ^ {* \top}, \dots , \boldsymbol {b} _ {m} ^ {* \top}\right) ^ {\top}.
$$
Moreover, let  $\lambda_{+}^{*},\lambda_{-}^{*}\geq 0$  be the corresponding Lagrange multipliers (with respect to  $\pmb{\theta}^{*}$ ) in Definition G.3.
Step I. The rough direction of each neuron.
Recalling the training dynamics about the dead neurons in Theorem 4.1 (S3),
$$
\boldsymbol {b} _ {k} (t) \equiv \boldsymbol {b} _ {k} \left(T _ {1}\right), \langle \boldsymbol {b} _ {k} (t), \boldsymbol {x} _ {+} \rangle \leq 0, \langle \boldsymbol {b} _ {k} (t), \boldsymbol {x} _ {-} \rangle \leq 0, k \in [ m / 2 ] - \mathcal {K} _ {+};
$$
$$
\boldsymbol {b} _ {k} (t) \equiv \boldsymbol {b} _ {k} \left(T _ {\mathrm {I}}\right), \left\langle \boldsymbol {b} _ {k} (t), \boldsymbol {x} _ {+} \right\rangle \leq 0, \left\langle \boldsymbol {b} _ {k} (t), \boldsymbol {x} _ {-} \right\rangle \leq 0, k \in [ m ] - [ m / 2 ] - \mathcal {K} _ {-}.
$$
Noticing Lemma F.6 (II) or Lemma F.5 (S1),  $\| \pmb {\theta}(t)\| _2\to \infty$  , so
$$
\boldsymbol {b} _ {k} ^ {*} = \mathbf {0}, \quad \forall k \notin \mathcal {K} _ {+} \cup \mathcal {K} _ {-}.
$$
Then we only need to focus on  $\pmb{\theta}_k^*$  for  $k\in \mathcal{K}_{+}\cup \mathcal{K}_{-}$
Recalling in Lemma F.5 (S3), for any  $t > T_{\mathrm{IV}}$
$$
\langle \boldsymbol {b} _ {k} ^ {*}, \boldsymbol {x} _ {+} \rangle \geq 0, \langle \boldsymbol {b} _ {k} ^ {*}, \boldsymbol {x} _ {-} \rangle = 0, k \in \mathcal {K} _ {+};
$$
$$
\langle \boldsymbol {b} _ {k} ^ {*}, \boldsymbol {x} _ {+} \rangle \geq 0, \langle \boldsymbol {b} _ {k} ^ {*}, \boldsymbol {x} _ {-} \rangle \geq 0, k \in \mathcal {K} _ {-}.
$$
$$
f \left(\boldsymbol {x} _ {+}; \boldsymbol {\theta} ^ {*}\right) = \frac {\kappa_ {2}}{\sqrt {m}} \Big (\sum_ {k \in \mathcal {K} _ {+}} \sigma \left(\left\langle \boldsymbol {b} _ {k} ^ {*}, \boldsymbol {x} _ {+} \right\rangle\right) - \sum_ {k \in \mathcal {K} _ {-}} \sigma \left(\left\langle \boldsymbol {b} _ {k} ^ {*}, \boldsymbol {x} _ {+} \right\rangle\right) \Big),
$$
$$
f \left(\boldsymbol {x} _ {-}; \boldsymbol {\theta} ^ {*}\right) = \frac {\kappa_ {2}}{\sqrt {m}} \left(\sum_ {k \in \mathcal {K} _ {+}} \sigma \left(\left\langle \boldsymbol {b} _ {k} ^ {*}, \boldsymbol {x} _ {-} \right\rangle\right) - \sum_ {k \in \mathcal {K} _ {-}} \sigma \left(\left\langle \boldsymbol {b} _ {k} ^ {*}, \boldsymbol {x} _ {-} \right\rangle\right)\right) = - \frac {\kappa_ {2}}{\sqrt {m}} \sum_ {k \in \mathcal {K} _ {-}} \sigma \left(\left\langle \boldsymbol {b} _ {k} ^ {*}, \boldsymbol {x} _ {-} \right\rangle\right).
$$
Step II. Determine the direction of the neurons  $k\in \mathcal{K}_{+}$
Since  $\pmb{\theta}^{*}$  is a possible point,  $f(\pmb{x}_{+};\pmb{\theta}^{*})\geq 1$  , which gives us
$$
\sum_ {k \in \mathcal {K} _ {+}} \sigma \left(\left\langle \boldsymbol {b} _ {k} ^ {*}, \boldsymbol {x} _ {+} \right\rangle\right) \geq \frac {\sqrt {m}}{\kappa_ {2}} > 0.
$$
Hence, there exists  $k_{1}\in \mathcal{K}_{+}$  , s.t.  $\langle \pmb{b}_{k_1}^*,\pmb{x}_+\rangle >0$  strictly.
Then we study the neuron  $k_{2} \in \mathcal{K}_{+}$  ( $k_{2} \neq k_{1}$ ). Lemma F.1 and Lemma F.5 (S3) give us that
$$
\begin{array}{l} \langle \boldsymbol {b} _ {k _ {1}} (t), \boldsymbol {x} _ {+} \rangle = \langle \boldsymbol {b} _ {k _ {1}} (T _ {\mathrm {I V}}), \boldsymbol {x} _ {+} \rangle + \int_ {T _ {\mathrm {I V}}} ^ {t} \frac {\kappa_ {2} p e ^ {- f _ {+} (t)}}{\sqrt {m} (1 + p)} \sin^ {2} \Delta \mathrm {d} t \\ = \left\langle \boldsymbol {b} _ {k _ {2}} \left(T _ {\mathrm {I V}}\right), \boldsymbol {x} _ {+} \right\rangle + \int_ {T _ {\mathrm {I V}}} ^ {t} \frac {\kappa_ {2} p e ^ {- f _ {+} (t)}}{\sqrt {m} (1 + p)} \sin^ {2} \Delta \mathrm {d} t + \left(\left\langle \boldsymbol {b} _ {k _ {1}} \left(T _ {\mathrm {I V}}\right), \boldsymbol {x} _ {+} \right\rangle - \left\langle \boldsymbol {b} _ {k _ {2}} \left(T _ {\mathrm {I V}}\right), \boldsymbol {x} _ {+} \right\rangle\right) \\ = \left\langle \boldsymbol {b} _ {k _ {2}} (t), \boldsymbol {x} _ {+} \right\rangle + \Big (\left\langle \boldsymbol {b} _ {k _ {1}} \left(T _ {\mathrm {I V}}\right), \boldsymbol {x} _ {+} \right\rangle - \left\langle \boldsymbol {b} _ {k _ {2}} \left(T _ {\mathrm {I V}}\right), \boldsymbol {x} _ {+} \right\rangle \Big). \\ \end{array}
$$
Multiplying the above formula by  $c / \| \pmb{\theta}(t) \|_2$  and taking  $t$  go to infinity, we obtain
$$
\left\langle \boldsymbol {b} _ {k _ {1}} ^ {*}, \boldsymbol {x} _ {+} \right\rangle = \left\langle \boldsymbol {b} _ {k _ {2}} ^ {*}, \boldsymbol {x} _ {+} \right\rangle > 0.
$$
Due to the arbitrariness of  $k_{2}$ , we know
$$
\boldsymbol {b} _ {k} ^ {*} \neq 0, \quad \langle \boldsymbol {b} _ {k} ^ {*}, \boldsymbol {x} _ {+} \rangle > 0, \quad \langle \boldsymbol {b} _ {k} ^ {*}, \boldsymbol {x} _ {-} \rangle = 0, \quad \forall k \in \mathcal {K} _ {+}.
$$
Then we can write the KKT condition about the gradient of  $\pmb{b}_k^*$  ( $k\in \mathcal{K}_{+}$ ) of Problem (34):
$$
\mathbf {0} \in \boldsymbol {b} _ {k} ^ {*} - \lambda_ {+} ^ {*} \frac {\kappa_ {2}}{\sqrt {m}} \boldsymbol {x} _ {+} + \lambda_ {-} ^ {*} \frac {\kappa_ {2}}{\sqrt {m}} \partial^ {\circ} \sigma (0) \boldsymbol {x} _ {-}.
$$
It is clear that  $\pmb{b}_k^* \in \operatorname{span}\{\pmb{x}_+, \pmb{x}_-\}$ . Then combining two formulations above, we obtain:
$$
\boldsymbol {b} _ {k} ^ {*} = \lambda_ {+} ^ {*} \frac {\kappa_ {2}}{\sqrt {m}} \left(\boldsymbol {x} _ {+} - \boldsymbol {x} _ {-} \cos \Delta\right), \quad \forall k \in \mathcal {K} _ {+}.
$$
Step III. Determine the direction of the neurons  $k\in \mathcal{K}_{-}$