SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \leq \left| \mathcal {K} _ {+} \right| \frac {1 2 \kappa_ {2} \sqrt {\kappa_ {1} \kappa_ {2}}}{m} \left(1 \cdot \frac {p - \cos \Delta}{p + 1} + 2 - 2 + 8. 4 \sqrt {\kappa_ {1} \kappa_ {2}} + \frac {4}{1 + 0 . 7 \left(1 + 9 . 9 \sqrt {\frac {\kappa_ {2}}{\kappa_ {1}}}\right) ^ {1 . 1 5}}\right) \\ \leq 0. 2 9 \cdot 1 2 \kappa_ {2} \sqrt {\kappa_ {1} \kappa_ {2}} \left(1 \cdot \frac {p - \cos \Delta}{p + 1} + 8. 4 \sqrt {\kappa_ {1} \kappa_ {2}} + \frac {4}{1 + 0 . 7 \left(1 + 9 . 9 \sqrt {\frac {\kappa_ {2}}{\kappa_ {1}}}\right) ^ {1 . 1 5}}\right) \\ \stackrel {(6)} {\leq} 3. 4 8 \kappa_ {2} \sqrt {\kappa_ {1} \kappa_ {2}} \Big (1 + 0. 0 8 4 + \frac {4}{1 + 0 . 7 (1 + 9 9) ^ {1 . 1 5}} \Big) \leq 3. 8 5 \kappa_ {2} \sqrt {\kappa_ {1} \kappa_ {2}}. \\ \end{array}
$$
$$
0. 9 7 8 \kappa_ {2} \sqrt {\kappa_ {1} \kappa_ {2}} \Big ((\frac {p - 1}{p + 1}) ^ {2} - 0. 1 1 \Big) \leq f _ {+} (T _ {1}) \leq 3. 8 5 \kappa_ {2} \sqrt {\kappa_ {1} \kappa_ {2}}.
$$
In the same way, we can estimate  $f_{-}(T_{1})$ :
$$
\begin{array}{l} f _ {-} \left(T _ {\mathrm {I}}\right) = \sum_ {k \in \mathcal {K} _ {+}} a _ {k} \sigma \left(\boldsymbol {b} _ {k} \left(T _ {\mathrm {I}}\right) ^ {\top} \boldsymbol {x} _ {-}\right) + \sum_ {k \in \mathcal {K} _ {-}} a _ {k} \sigma \left(\boldsymbol {b} _ {k} \left(T _ {\mathrm {I}}\right) ^ {\top} \boldsymbol {x} _ {-}\right) \\ \geq \sum_ {k \in \mathcal {K} _ {+}} \frac {\kappa_ {2}}{\sqrt {m}} \frac {4 . 6 6 \sqrt {\kappa_ {1} \kappa_ {2}}}{\sqrt {m}} \left\langle \boldsymbol {w} _ {k} (T _ {\mathrm {I}}), \boldsymbol {x} _ {-} \right\rangle - \sum_ {k \in \mathcal {K} _ {-}} \frac {\kappa_ {2}}{\sqrt {m}} \frac {\sqrt {\kappa_ {1} \kappa_ {2}}}{\sqrt {m}} \left(4. 3 \sqrt {\frac {\kappa_ {1}}{\kappa_ {2}}} + \frac {1 1 . 1 \sin \Delta}{1 + p}\right) \left\langle \boldsymbol {w} _ {k} (T _ {\mathrm {I}}), \boldsymbol {x} _ {-} \right\rangle \\ \geq \sum_ {k \in \mathcal {K} _ {+}} \frac {\kappa_ {2}}{\sqrt {m}} \frac {4 . 6 6 \sqrt {\kappa_ {1} \kappa_ {2}}}{\sqrt {m}} \Big (\langle \pmb {\mu}, \pmb {x} _ {-} \rangle - \| \pmb {w} _ {k} (T _ {\mathrm {I}}) - \pmb {\mu} \| \Big) - \sum_ {k \in \mathcal {K} _ {-}} \frac {\kappa_ {2}}{\sqrt {m}} \frac {\sqrt {\kappa_ {1} \kappa_ {2}}}{\sqrt {m}} \Big (0. 4 3 + \frac {1 1 . 1}{1 2} \Big) \sin \Delta \\ \geq \left| \mathcal {K} _ {+} \right| \frac {4 . 6 6 \kappa_ {2} \sqrt {\kappa_ {1} \kappa_ {2}}}{m} \left(\frac {p - 1}{p + 1} \frac {p \cos \Delta - 1}{p + 1} - 2 + 2 \left(1 - 4. 2 \sqrt {\kappa_ {1} \kappa_ {2}} - \frac {2}{1 + 0 . 7 \left(1 + 9 . 9 \sqrt {\frac {\kappa_ {2}}{\kappa_ {1}}}\right) ^ {1 . 1 5}}\right)\right) \\ - \left| \mathcal {K} _ {-} \right| \frac {\kappa_ {2} \sqrt {\kappa_ {1} \kappa_ {2}}}{m} \cdot 1. 3 5 5 \sin \Delta \\ \geq 0. 2 1 \cdot 4. 6 6 \kappa_ {2} \sqrt {\kappa_ {1} \kappa_ {2}} \left(\frac {p - 1}{p + 1} \frac {p \cos \Delta - 1}{p + 1} - 0. 1 1\right) - 0. 2 0 5 \cdot 1. 3 5 5 \sin \Delta \kappa_ {2} \sqrt {\kappa_ {1} \kappa_ {2}} \\ \geq \kappa_ {2} \sqrt {\kappa_ {1} \kappa_ {2}} \left(0. 9 7 8 (\frac {p - 1}{p + 1}) ^ {2} \cos \Delta - 0. 1 1 - 0. 2 8 \sin \Delta\right) \geq \kappa_ {2} \sqrt {\kappa_ {1} \kappa_ {2}} \left(0. 9 4 7 (\frac {p - 1}{p + 1}) ^ {2} \cos \Delta - 0. 1 1 - 0. 0 7\right) \\ \geq 0. 9 4 7 \left(\left(\frac {p - 1}{p + 1}\right) ^ {2} \cos \Delta - 0. 2\right); \\ \end{array}
$$
$$
\begin{array}{l} f _ {-} \left(T _ {\mathrm {I}}\right) = \sum_ {k \in \mathcal {K} _ {+}} a _ {k} \sigma \left(\boldsymbol {b} _ {k} \left(T _ {\mathrm {I}}\right) ^ {\top} \boldsymbol {x} _ {-}\right) + \sum_ {k \in \mathcal {K} _ {-}} a _ {k} \sigma \left(\boldsymbol {b} _ {k} \left(T _ {\mathrm {I}}\right) ^ {\top} \boldsymbol {x} _ {-}\right) \\ \leq \sum_ {k \in \mathcal {K} _ {+}} \frac {\kappa_ {2}}{\sqrt {m}} \frac {1 2 \sqrt {\kappa_ {1} \kappa_ {2}}}{\sqrt {m}} \left\langle \boldsymbol {w} _ {k} (T _ {\mathrm {I}}), \boldsymbol {x} _ {-} \right\rangle - 0 \\ = \sum_ {k \in \mathcal {K} _ {+}} \frac {\kappa_ {2}}{\sqrt {m}} \frac {1 2 \sqrt {\kappa_ {1} \kappa_ {2}}}{\sqrt {m}} \left(\| z \| \langle z, x _ {-} \rangle + 2 - 2 \langle w _ {k} (T _ {\mathrm {I}}), \boldsymbol {\mu} \rangle\right) \\ \leq \left| \mathcal {K} _ {+} \right| \frac {1 2 \kappa_ {2} \sqrt {\kappa_ {1} \kappa_ {2}}}{m} \left(1 \cdot \frac {p \cos \Delta - 1}{p + 1} + 2 - 2 + 8. 4 \sqrt {\kappa_ {1} \kappa_ {2}} + \frac {4}{1 + 0 . 7 \left(1 + 9 . 9 \sqrt {\frac {\kappa_ {2}}{\kappa_ {1}}}\right) ^ {1 . 1 5}}\right) \\ \leq 0. 2 9 \cdot 1 2 \kappa_ {2} \sqrt {\kappa_ {1} \kappa_ {2}} \left(1 \cdot \frac {p \cos \Delta - 1}{p + 1} + 8. 4 \sqrt {\kappa_ {1} \kappa_ {2}} + \frac {4}{1 + 0 . 7 \left(1 + 9 . 9 \sqrt {\frac {\kappa_ {2}}{\kappa_ {1}}}\right) ^ {1 . 1 5}}\right) \\ \stackrel {(6)} {\leq} 3. 4 8 \kappa_ {2} \sqrt {\kappa_ {1} \kappa_ {2}} \left(1 + 0. 0 8 4 + \frac {4}{1 + 0 . 7 (1 + 9 9) ^ {1 . 1 5}}\right) \leq 3. 8 5 \kappa_ {2} \sqrt {\kappa_ {1} \kappa_ {2}}. \\ \end{array}
$$
Them we have:
$$
0. 9 4 7 \left(\left(\frac {p - 1}{p + 1}\right) ^ {2} \cos \Delta - 0. 2\right) \leq f _ {-} \left(T _ {\mathrm {I}}\right) \leq 3. 8 5 \kappa_ {2} \sqrt {\kappa_ {1} \kappa_ {2}}.
$$
Due to  $f_{+}(T_{\mathrm{I}}) > 0$  and  $f_{-}(T_{\mathrm{I}}) > 0$ , we obtain  $\mathrm{ACC}(T_{\mathrm{I}}) = \frac{p}{1 + p}$ .
Moreover, from Theorem C.12 (S1)(S2), we have
$$
0 <   0. 2 5 8 \leq \frac {0 . 0 7 5 m}{0 . 2 9 m} \leq \alpha = \frac {\left| \mathcal {K} _ {-} \right|}{\left| \mathcal {K} _ {+} \right|} = \frac {m _ {-}}{m _ {+}} \leq \frac {0 . 2 0 5 m}{0 . 2 1 m} \leq 0. 9 7 7 <   1. \tag {18}
$$
Remark C.13. The results in the following proofs are all based on the occurrence of the events in Theorem C.12. All of these results use the same settings as Theorem 4.1, except using a stronger condition on the initialization parameters (3) than (6) in Theorem C.12. So they all hold with probability at least  $1 - \delta$ .
D Proofs of Optimization Dynamics in Phase II
Recalling the results in Theorem 4.1, during Phase II, the activation patterns do not change with  $\operatorname{sgn}_k^+(t) = \operatorname{sgn}_k^- (t) = 1$  for  $k\in \mathcal{K}_{+}$  and  $\operatorname{sgn}_k^+(t) = 0$ ,  $\operatorname{sgn}_k^- (t) = 1$  for  $k\in \mathcal{K}_{-}$ . Theorem 4.4 demonstrates that at the end of Phase II, except for one of living positive neuron  $k_{0}\in \mathcal{K}_{+}$  precisely changes its pattern on  $\pmb{x}_{-}$ , all other activation patterns remain unchanged.
Recall that at the end of Phase I, (i) the neuron  $k \notin \mathcal{K}_{+} \cup \mathcal{K}_{-}$  is dead forever; (ii) as for the living neuron  $k \in \mathcal{K}_{+} \cup \mathcal{K}_{-}$ , the activation patterns are:
$$
\operatorname {s g n} _ {k} ^ {+} (t) = \operatorname {s g n} _ {k} ^ {-} (t) = 1 \text {f o r} k \in \mathcal {K} _ {+};
$$
$$
\mathbf {s g n} _ {k} ^ {+} (t) = 0, \mathbf {s g n} _ {k} ^ {-} (t) = 1 \text {f o r} k \in \mathcal {K} _ {-}.
$$
In this section, we will focus on the Phase when the negative neuron  $k \in \mathcal{K}_{-}$  still stays on the manifold  $\mathcal{M}_+^0 \cap \mathcal{M}_-^+$  (hence is still dead for  $\pmb{x}_{+}$ ) and the positive neuron  $k \in \mathcal{K}_{+}$  is still activated for  $\pmb{x}_{-}$ . In general, we aim to estimate the
As stated in the main text, we define as following hitting time:
$$
\begin{array}{l} T _ {\mathrm {I I}} := \inf  \left\{t > T _ {\mathrm {I}}: \exists k \in \mathcal {K} _ {+} \cup \mathcal {K} _ {-}, \operatorname {s g n} _ {k} ^ {+} (t) \neq \operatorname {s g n} _ {k} ^ {+} (T _ {\mathrm {I}}) \text {o r} \operatorname {s g n} _ {k} ^ {-} (t) \neq \operatorname {s g n} _ {k} ^ {-} (T _ {\mathrm {I}}) \right\} \\ = \inf  \left\{t > T _ {\mathrm {I}}: \exists k \in \mathcal {K} _ {+}, \text {s . t .} \langle \boldsymbol {w} _ {k} (t), \boldsymbol {x} _ {+} \rangle \leq 0 \text {o r} \langle \boldsymbol {w} _ {k} (t), \boldsymbol {x} _ {-} \rangle \leq 0, \right. \tag {19} \\ \left. \text {o r} \exists k \in \mathcal {K} _ {-}, \text {s . t .} \langle \boldsymbol {w} _ {k} (t), \boldsymbol {x} _ {+} \rangle \neq 0 \text {o r} \langle \boldsymbol {w} _ {k} (t), \boldsymbol {x} _ {-} \rangle \leq 0 \right\}, \\ \end{array}
$$
and we call  $T_{\mathrm{I}} \leq t \leq T_{\mathrm{II}}$  "Phase II".
First, we define a more relaxed hitting time than  $T_{\mathrm{II}}$ , only about the change of living positive neurons:
$$
T _ {\mathrm {I I}} ^ {+} := \inf  \left\{t > T _ {\mathrm {I}}: \exists k \in \mathcal {K} _ {+}, \text {s . t .} \langle \boldsymbol {w} _ {k} (t), \boldsymbol {x} _ {+} \rangle \leq 0 \text {o r} \langle \boldsymbol {w} _ {k} (t), \boldsymbol {x} _ {-} \rangle \leq 0 \right\}. \tag {20}
$$
Noticing that the changes in activation partitions are essentially caused by the change of discontinuous vector fields, we first define the following auxiliary hitting time:
$$
T _ {\mathrm {I I}} ^ {*} := T _ {\mathrm {I I}} ^ {+} \wedge \inf  \left\{t > T _ {\mathrm {I}}: \langle \boldsymbol {F} _ {+} (t), \boldsymbol {x} _ {+} \rangle \leq 0 \right\},
$$
$$
\text {w h e r e} \boldsymbol {F} _ {+} (t) = \frac {p}{1 + p} e ^ {- f _ {+} (t)} \boldsymbol {x} _ {+} - \frac {1}{1 + p} e ^ {f _ {-} (t)} \boldsymbol {x} _ {-}. \tag {21}
$$
We call  $T_{\mathrm{I}} \leq t \leq T_{\mathrm{II}}^{*}$  "Phase  $\Pi^{*}$ ".
In the subsequent proof, we first meticulously characterize the optimization dynamics in Phase II* and then prove  $T_{\mathrm{II}} = T_{\mathrm{II}}^{+} = T_{\mathrm{II}}^{*}$ . The crucial proof technique is fine-grained prior estimations for 2d ODEs on  $f_{+}(t)$  and  $f_{-}(t)$ , leading to the vector field estimation.
To begin with, we establish the following lemma about the optimization dynamics of living neurons.
Lemma D.1 (Dynamics of living neurons in Phase II*).
In Phase  $II^{*}$ ,  $t \in [T_{\mathrm{I}}, T_{\mathrm{II}}^{*}]$ , we have the following dynamics for each neuron  $k \in \mathcal{K}_{-} \cup \mathcal{K}_{+}$ .
(S1) For positive neuron  $k \in \mathcal{K}_{+}$ , we have:
$$
\boldsymbol {w} _ {k} (t) \in \mathcal {M} _ {+} ^ {+} \cap \mathcal {M} _ {-} ^ {+},
$$
$$
\frac {\mathrm {d} \boldsymbol {b} _ {k} (t)}{\mathrm {d} t} = \frac {\kappa_ {2}}{\sqrt {m}} \boldsymbol {F} _ {+} (t) = \frac {\kappa_ {2}}{\sqrt {m}} \left(\frac {p}{1 + p} e ^ {- f _ {+} (t)} \boldsymbol {x} _ {+} - \frac {1}{1 + p} e ^ {f _ {-} (t)} \boldsymbol {x} _ {-}\right).
$$
(S2). For negative neuron  $k \in \mathcal{K}_{-}$ , we have:
$$
\boldsymbol {w} _ {k} (t) \in \mathcal {M} _ {+} ^ {0} \cap \mathcal {M} _ {-} ^ {+},
$$
Proof of Lemma D.1.
(S1) Let  $k \in \mathcal{K}_{+}$ . Recalling the definition of  $T_{\mathrm{II}}^{*}$ , it holds that  $\langle \pmb{w}_k(t), \pmb{x}_+ \rangle > 0$  and  $\langle \pmb{w}_k(t), \pmb{x}_- \rangle > 0$  for any  $T_{\mathrm{I}} \leq t \leq T_{\mathrm{II}}^{*}$ , so the dynamics hold.
(S2) Let  $k \in \mathcal{K}_{-}$ . Recalling the definition of  $T_{\mathrm{II}}^{*}$ , it holds  $\langle \pmb{F}_{+}(t), \pmb{x}_{+} \rangle > 0$  for any  $T_{\mathrm{I}} \leq t \leq T_{\mathrm{II}}^{*}$ . Due to  $\pmb{w}_{k}(T_{\mathrm{I}}) \in \mathcal{M}_{+}^{0} \cap \mathcal{M}_{-}^{+}$ , we first analysis the vector field around the manifold  $\mathcal{P}_{+}^{0} \cap \mathcal{P}_{-}^{+}$  for  $T_{\mathrm{I}} \leq t \leq T_{\mathrm{II}}^{*}$ .
For any  $\tilde{\pmb{b}}\in \mathcal{P}_+^0\cap \mathcal{P}_-^+$  and  $0 <   \delta_0\ll 1$  , we know that  $\mathcal{P}_+^0\cap \mathcal{P}_-^+$  separates its neighborhood  $\mathcal{B}(\tilde{\pmb{b}},\delta_0)$  into two domains  $\mathcal{G}_{-} = \{\pmb {b}\in \mathcal{B}(\tilde{\pmb{b}},\delta_{0}):\langle \pmb {b},\pmb{x}_{+}\rangle <  0\}$  and  $\mathcal{G}_{+} = \{\pmb {b}\in \mathcal{B}(\tilde{\pmb{b}},\delta_{0}):\langle \pmb {b},\pmb{x}_{+}\rangle >0\}$  Following Definition H.1, we calculate the limited vector field on  $\tilde{\pmb{b}}$  from  $\mathcal{G}_{-}$  and  $\mathcal{G}_{+}$
(i) The limited vector field  $F^{-}$  on  $\tilde{\pmb{b}}$  (from  $\mathcal{G}_{-}$ ):
(iii) Then we calculate the projections of  $\pmb{F}^{-}$  and  $\pmb{F}^{+}$  onto  $\pmb{x}_{+}$  (the normal to the surface  $\mathcal{P}_+^0\cap \mathcal{P}_-^+$ ):
We further define the hitting time to check whether  $\pmb{w}_k(t) \in \mathcal{M}_+^0 \cap \mathcal{M}_-^+$  for  $T_{\mathrm{I}} \leq t \leq T_{\mathrm{II}}^*$ .
$$
\tau_ {-} ^ {+} := T _ {\mathrm {I I}} ^ {*} \wedge \inf  \left\{t > T _ {\mathrm {I}}: \exists k \in \mathcal {K} _ {-}, \text {s . t .} \langle \boldsymbol {w} _ {k} (t), \boldsymbol {x} _ {-} \rangle \leq 0 \right\}.
$$
From the definition of  $T_{\mathrm{II}}^{*}$ , we know  $\langle F_{+}(t), \boldsymbol{x}_{+} \rangle > 0$  for any  $T_{\mathrm{I}} \leq t \leq T_{\mathrm{II}}^{*}$ , so  $F_{N}^{+} < 0$ . And it is clear that  $F_{N}^{-} > 0$ . Hence, the dynamics corresponds to Case (I) in Definition H.1 ( $F_{N}^{-} > 0$  and  $F_{N}^{+} < 0$ ), which means  $\boldsymbol{b}_{k}(t)$  can not leave  $\mathcal{P}_{+}^{0}$  (i.e.,  $\langle \boldsymbol{b}_{k}(t), \boldsymbol{x}_{+} \rangle = 0$ ) for  $t \in [T_{\mathrm{I}}, \tau_{-}^{+}]$ , and the dynamics of  $\boldsymbol{b}_{k}(t)$  satisfies:
$$
\frac {\mathrm {d} \boldsymbol {b}}{\mathrm {d} t} = \alpha \boldsymbol {F} ^ {+} + (1 - \alpha) \boldsymbol {F} ^ {-}, \quad \alpha = \frac {f _ {N} ^ {-}}{f _ {N} ^ {-} - f _ {N} ^ {+}}, t \in [ T _ {\mathrm {I}}, \tau_ {-} ^ {+} ],
$$
$$
\frac {\mathrm {d} \boldsymbol {b} _ {k} (t)}{\mathrm {d} t} \frac {\kappa_ {2} e ^ {f _ {-} (t)}}{\sqrt {m} (1 + p)} \left(\boldsymbol {x} _ {-} - \boldsymbol {x} _ {+} \cos \Delta\right), t \in [ T _ {1}, \tau_ {-} ^ {+} ].
$$
$$
\frac {\mathrm {d} \boldsymbol {w} _ {k} (t)}{\mathrm {d} t} = \frac {\kappa_ {2} e ^ {f _ {-} (t)}}{\rho_ {k} (t) \sqrt {m} (1 + p)} \Big (\boldsymbol {x} _ {-} - \langle \boldsymbol {w} _ {k} (t), \boldsymbol {x} _ {-} \rangle \boldsymbol {w} _ {k} - \boldsymbol {x} _ {+} \cos \Delta \Big).
$$
$$
\frac {\mathrm {d} \rho_ {k} (t)}{\mathrm {d} t} = \frac {\kappa_ {2} e ^ {f _ {-} (t)}}{\sqrt {m} (1 + p)} \langle \boldsymbol {w} _ {k} (t), \boldsymbol {x} _ {-} \rangle .
$$
From this dynamics, for any  $t \in [T_1, \tau_-^+]$ , we have
$$
\begin{array}{l} \langle \boldsymbol {b} _ {k} (t), \boldsymbol {x} _ {-} \rangle = \langle \boldsymbol {b} _ {k} (0), \boldsymbol {x} _ {-} \rangle + \int_ {0} ^ {t} \left\langle \frac {\mathrm {d} \boldsymbol {b} _ {k} (s)}{\mathrm {d} s}, \boldsymbol {x} _ {-} \right\rangle \mathrm {d} s \\ = \left\langle \boldsymbol {b} _ {k} (0), \boldsymbol {x} _ {-} \right\rangle + \int_ {0} ^ {t} \frac {\kappa_ {2} e ^ {f _ {-} (s)}}{\sqrt {m} (1 + p)} \sin^ {2} \Delta \mathrm {d} s > \left\langle \boldsymbol {b} _ {k} (0), \boldsymbol {x} _ {-} \right\rangle > 0, \\ \end{array}
$$
which means that  $T_{\mathrm{II}}^{*} = \tau_{-}^{+}$
Noticing that Lemma D.1 determines the activation patterns for living neurons in Phase II*, the next lemma gives the first-order dynamics of  $f_{+}(t)$  and  $f_{-}(t)$ .
Lemma D.2 (First-order dynamics of predictions in Phase II*).
In Phase II* ( $T_{\mathrm{I}} \leq t \leq T_{\mathrm{II}}^{*}$ ), we have the following dynamics for  $f_{+}(t)$  and  $f_{-}(t)$ :
$$
\frac {\mathrm {d} f _ {+} (t)}{\mathrm {d} t} = \kappa_ {2} ^ {2} \frac {m _ {+}}{m} \Big (\frac {p e ^ {- f _ {+} (t)}}{1 + p} - \frac {e ^ {f _ {-} (t)}}{1 + p} \cos \Delta \Big),
$$
$$
\frac {\mathrm {d} f _ {-} (t)}{\mathrm {d} t} = \kappa_ {2} ^ {2} \frac {m _ {+}}{m} \Big (\frac {p e ^ {- f _ {+} (t)}}{1 + p} \cos \Delta - \frac {e ^ {f _ {-} (t)}}{1 + p} \Big) - \kappa_ {2} ^ {2} \frac {m _ {-}}{m} \frac {e ^ {f _ {-} (t)}}{1 + p} \sin^ {2} \Delta .
$$
Proof of Lemma D.2.
From the definition of  $T_{\mathrm{II}}^{*}$ , for any  $T_{\mathrm{I}} \leq t \leq T_{\mathrm{II}}^{*}$ , we have
$$
f _ {+} (t) = \sum_ {k \in \mathcal {K} _ {+}} \frac {\kappa_ {2}}{\sqrt {m}} \sigma \left(\boldsymbol {b} _ {k} ^ {\top} (t) \boldsymbol {x} _ {+}\right) - \sum_ {k \in \mathcal {K} _ {-}} \frac {\kappa_ {2}}{\sqrt {m}} \sigma \left(\boldsymbol {b} _ {k} ^ {\top} (t) \boldsymbol {x} _ {+}\right) = \sum_ {k \in \mathcal {K} _ {+}} \frac {\kappa_ {2}}{\sqrt {m}} \boldsymbol {b} _ {k} ^ {\top} (t) \boldsymbol {x} _ {+},
$$
$$
f _ {-} (t) = \sum_ {k \in \mathcal {K} _ {+}} \frac {\kappa_ {2}}{\sqrt {m}} \sigma (\boldsymbol {b} _ {k} ^ {\top} (t) \boldsymbol {x} _ {-}) - \sum_ {k \in \mathcal {K} _ {-}} \frac {\kappa_ {2}}{\sqrt {m}} \sigma (\boldsymbol {b} _ {k} ^ {\top} (t) \boldsymbol {x} _ {-}) = \sum_ {k \in \mathcal {K} _ {+}} \frac {\kappa_ {2}}{\sqrt {m}} \boldsymbol {b} _ {k} ^ {\top} (t) \boldsymbol {x} _ {-} - \sum_ {k \in \mathcal {K} _ {-}} \frac {\kappa_ {2}}{\sqrt {m}} \boldsymbol {b} _ {k} ^ {\top} (t) \boldsymbol {x} _ {-}.
$$
With the help of Lemma D.1, we have the dynamics of predictions:
$$
\begin{array}{l} \frac {\mathrm {d} f _ {+} (t)}{\mathrm {d} t} = \sum_ {k \in \mathcal {K} _ {+}} \frac {\kappa_ {2}}{\sqrt {m}} \left\langle \frac {\mathrm {d} \boldsymbol {b} _ {k} (t)}{\mathrm {d} t}, \boldsymbol {x} _ {+} \right\rangle = \frac {\kappa_ {2} ^ {2}}{m} \sum_ {k \in \mathcal {K} _ {+}} \left(\frac {p}{1 + p} e ^ {- f _ {+} (t)} - \frac {1}{1 + p} e ^ {f _ {-} (t)} \cos \Delta\right) \\ = \frac {m _ {+}}{m} \kappa_ {2} ^ {2} \left(\frac {p e ^ {- f _ {+} (t)}}{1 + p} - \frac {e ^ {f _ {-} (t)}}{1 + p} \cos \Delta\right). \\ \end{array}
$$
$$
\begin{array}{l} \frac {\mathrm {d} f _ {-} (t)}{\mathrm {d} t} = \sum_ {k \in \mathcal {K} _ {+}} \frac {\kappa_ {2}}{\sqrt {m}} \left\langle \frac {\mathrm {d} \boldsymbol {b} _ {k} (t)}{\mathrm {d} t}, \boldsymbol {x} _ {-} \right\rangle - \sum_ {k \in \mathcal {K} _ {-}} \frac {\kappa_ {2}}{\sqrt {m}} \left\langle \frac {\mathrm {d} \boldsymbol {b} _ {k} (t)}{\mathrm {d} t}, \boldsymbol {x} _ {-} \right\rangle \\ = \frac {\kappa_ {2} ^ {2}}{m} \sum_ {k \in \mathcal {K} _ {+}} \left(\frac {p e ^ {- f _ {+} (t)}}{1 + p} \cos \Delta - \frac {e ^ {f _ {-} (t)}}{1 + p}\right) - \frac {\kappa_ {2} ^ {2}}{m} \sum_ {k \in \mathcal {K} _ {-}} \frac {e ^ {f _ {-} (t)}}{1 + p} \left(1 - \cos^ {2} \Delta\right) \\ = \frac {m _ {+}}{m} \kappa_ {2} ^ {2} \bigg (\frac {p e ^ {- f _ {+} (t)}}{1 + p} \cos \Delta - \frac {e ^ {f _ {-} (t)}}{1 + p} \bigg) - \frac {m _ {-}}{m} \kappa_ {2} ^ {2} \frac {e ^ {f _ {-} (t)}}{1 + p} \sin^ {2} \Delta . \\ \end{array}
$$
Due to the specificity of the first-order dynamics, the following lemma gives an second-order autonomous dynamics of predictions, which is the core dynamics in this phase.
Lemma D.3 (Second-order Autonomous Dynamics of predictions in Phase II*).
Consider the following two variables:
$$
\left\{ \begin{array}{l} \mathcal {U} (t) := \kappa_ {2} ^ {2} \frac {m _ {+}}{m} \frac {p}{1 + p} e ^ {- f _ {+} (t)}, \\ \mathcal {V} (t) := \kappa_ {2} ^ {2} \frac {m _ {+}}{m} \frac {1}{1 + p} e ^ {f _ {-} (t)}. \end{array} \right.
$$
Then the following autonomous dynamics of  $\mathcal{U}(t)$  and  $\mathcal{V}(t)$  hold in Phase  $II^{*}$  ( $T_{\mathrm{I}} \leq t \leq T_{\mathrm{II}}^{*}$ ):
$$
\left\{ \begin{array}{l} \frac {\mathrm {d} \mathcal {U} (t)}{\mathrm {d} t} = \mathcal {U} (t) \mathcal {V} (t) \cos \Delta - \mathcal {U} ^ {2} (t), \\ \frac {\mathrm {d} \mathcal {V} (t)}{\mathrm {d} t} = \mathcal {U} (t) \mathcal {V} (t) \cos \Delta - \mathcal {V} ^ {2} (t) \left(1 + \alpha \sin^ {2} \Delta\right). \end{array} \right.
$$
Proof of Lemma D.3.
Recall the first-order dynamics in Lemma D.2:
$$
\left\{ \begin{array}{l l} \frac {\mathrm {d} f _ {+} (t)}{\mathrm {d} t} & = \kappa_ {2} ^ {2} \frac {m _ {+}}{m} \left(\frac {p e ^ {- f _ {+} (t)}}{1 + p} - \frac {e ^ {f _ {-} (t)}}{1 + p} \cos \Delta\right), \\ \frac {\mathrm {d} f _ {-} (t)}{\mathrm {d} t} & = \kappa_ {2} ^ {2} \frac {m _ {+}}{m} \left(\frac {p e ^ {- f _ {+} (t)}}{1 + p} \cos \Delta - \frac {e ^ {f _ {-} (t)}}{1 + p}\right) - \kappa_ {2} ^ {2} \frac {m _ {-}}{m} \frac {e ^ {f _ {-} (t)}}{1 + p} \sin^ {2} \Delta . \end{array} \right.
$$
Then this proof is a straight-forward calculation:
$$
\begin{array}{l} \frac {\mathrm {d} \mathcal {U} (t)}{\mathrm {d} t} = \kappa_ {2} ^ {2} \frac {m _ {+}}{m} \frac {p}{1 + p} \frac {\mathrm {d} e ^ {- f _ {+} (t)}}{\mathrm {d} t} = - \kappa_ {2} ^ {2} \frac {m _ {+}}{m} \frac {p}{1 + p} e ^ {- f _ {+} (t)} \frac {\mathrm {d} f _ {+} (t)}{\mathrm {d} t} \\ = - \mathcal {U} (t) \frac {\mathrm {d} f _ {+} (t)}{\mathrm {d} t} = \mathcal {U} (t) \mathcal {V} (t) \cos \Delta - \mathcal {U} ^ {2} (t), \\ \end{array}
$$
$$
\begin{array}{l} \frac {\mathrm {d} \mathcal {V} (t)}{\mathrm {d} t} = \kappa_ {2} ^ {2} \frac {m _ {+}}{m} \frac {1}{1 + p} \frac {\mathrm {d} e ^ {f _ {-} (t)}}{\mathrm {d} t} = \kappa_ {2} ^ {2} \frac {m _ {+}}{m} \frac {1}{1 + p} e ^ {f _ {-} (t)} \frac {\mathrm {d} f _ {-} (t)}{\mathrm {d} t} \\ = \mathcal {V} (t) \frac {\mathrm {d} f _ {-} (t)}{\mathrm {d} t} = \mathcal {U} (t) \mathcal {V} (t) \cos \Delta - \mathcal {V} ^ {2} (t) \left(1 + \alpha \sin^ {2} \Delta\right). \\ \end{array}
$$
Lemma D.3 enlighten us that we only need to study the dynamics of  $\mathcal{U}(t)$  and  $\mathcal{V}(t)$  to study the dynamics in Phase II, where  $\mathcal{U}(t),\mathcal{V}(t)$  satisfies the following autonomous dynamics:
$$
\left\{ \begin{array}{l} \frac {\mathrm {d} \mathcal {U} (t)}{\mathrm {d} t} = \mathcal {U} (t) \mathcal {V} (t) \cos \Delta - \mathcal {U} ^ {2} (t); \\ \frac {\mathrm {d} \mathcal {V} (t)}{\mathrm {d} t} = \mathcal {U} (t) \mathcal {V} (t) \cos \Delta - \mathcal {V} ^ {2} (t) \left(1 + \alpha \sin^ {2} \Delta\right), \end{array} \quad t \geq T _ {1}; \right. \tag {22}
$$
$$
\left\{ \begin{array}{l} \mathcal {U} (T _ {\mathrm {I}}) = \kappa_ {2} ^ {2} \frac {m _ {+}}{m} \frac {p}{1 + p} e ^ {- f _ {+} (T _ {\mathrm {I}})}, \\ \mathcal {V} (T _ {\mathrm {I}}) = \kappa_ {2} ^ {2} \frac {m _ {+}}{m} \frac {1}{1 + p} e ^ {f _ {-} (T _ {\mathrm {I}})}. \end{array} \right.
$$
The next lemma provides a fine-grained prior estimate of the dynamics (22).
Lemma D.4 (Fine-grained prior estimate of the dynamics (22)).
For the dynamics (22), then we have the following results:
(S1).  $\mathcal{U}(T_{\mathrm{I}}) = \Theta (\kappa_2^2)$  and  $\mathcal{V}(T_{\mathrm{I}}) = \Theta \left(\frac{\kappa_2^2}{p}\right)$
(S2). For any  $t \geq T_{\mathrm{I}}$ , we have  $\mathcal{U}(t) > \mathcal{V}(t) > 0$ .
(S3). If we define the hitting time  $\tau_{1} := \inf \left\{t \geq T_{1} : \mathcal{U}(t) \cos \Delta \leq \mathcal{V}(t) (1 + \alpha \sin^{2} \Delta)\right\}$ , then
$$
\mathcal {U} (\tau_ {1}) = \frac {1 + \alpha \sin^ {2} \Delta}{\cos \Delta} \mathcal {V} (\tau_ {1}), \quad \mathcal {V} (\tau_ {1}) = \Theta \left(\kappa_ {2} ^ {2} p ^ {- \frac {1}{1 + \cos \Delta}}\right),
$$
$$
\tau_ {1} = \mathcal {O} \left(\frac {p ^ {\frac {1}{1 + \cos \Delta}} \log (1 / \Delta)}{\kappa_ {2} ^ {2}}\right) = \Omega \left(\frac {p ^ {\frac {1}{1 + \cos \Delta}}}{\kappa_ {2} ^ {2}}\right) = \tilde {\Theta} \left(\frac {p ^ {\frac {1}{1 + \cos \Delta}}}{\kappa_ {2} ^ {2}}\right).
$$
(S4). For any  $t \geq \tau_1$ , we have
$$
1 + \frac {m _ {-}}{2 m _ {+}} \sin^ {2} \Delta <   \frac {\mathcal {U} (t)}{\mathcal {V} (t)} <   \frac {1 + 2 \alpha \sin^ {2} \Delta}{\cos \Delta},
$$
$$
\mathcal {U} (t) = \Theta \left(\frac {1}{\frac {1}{p ^ {\frac {1 + \cos \Delta}{\kappa_ {2} ^ {2}}} + \Delta^ {2} (t - \tau_ {1})}}\right), \quad \mathcal {V} (t) = \Theta \left(\frac {1}{\frac {1}{p ^ {\frac {1 + \cos \Delta}{\kappa_ {2} ^ {2}}} + \Delta^ {2} (t - \tau_ {1})}}\right).
$$
(S5). For any  $t \geq \tau_1$ , we have  $\mathcal{U}(t) - \mathcal{V}(t)\cos \Delta = \Theta (\Delta^2\mathcal{V}(t)) > 0$
(S6). For any  $t \geq \tau_2 = \Theta\left(\frac{p^{\frac{1}{1 + \cos\Delta}}\log(1 / \Delta)}{\kappa_2^2}\right) \geq 2\tau_1$ , we have
$$
\mathcal {U} (t) \cos \Delta - \mathcal {V} (t) = - \Theta (\Delta^ {2} \mathcal {V} (t)) <   0.
$$
Proof of Lemma D.4.
For simplicity, in this proof, we denote
$$
\epsilon := \alpha \sin^ {2} \Delta .
$$
Step I. Preparation. From Theorem C.12 (S4), we know  $0 < f_{+}(T_{\mathrm{I}}),f_{-}(T_{\mathrm{I}})\leq 3.85\kappa_{2}\sqrt{\kappa_{1}\kappa_{2}}\leq$ $\frac{0.04\log(1.1)}{0.04\log(1.1)}$  so