SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Recalling the definition of  $\tau_{1}$  in Lemma D.4, we know  $\mathcal{U}(s)\cos \Delta -\mathcal{V}(s) > 0$  for any  $t\leq \tau_1$ , so  $T_{\mathrm{II}}^{*} > \tau_{1}$ .
From Lemma D.4 (S4), we know
$$
\mathcal {U} (t) = \Theta \left(\frac {1}{\frac {\frac {1}{p ^ {\frac {1}{1 + \cos \Delta}}}}{\kappa_ {2} ^ {2}} + \Delta^ {2} (t - \tau_ {1})}\right), \quad \mathcal {V} (t) = \Theta \left(\frac {1}{\frac {1}{p ^ {\frac {1}{1 + \cos \Delta}}}} + \Delta^ {2} (t - \tau_ {1})\right).
$$
From the proof of Lemma D.4, we know  $\mathcal{U}(T_{\mathrm{I}}) = \Theta (\kappa_2^2)$  and  $\mathcal{V}(T_{\mathrm{I}}) = \Theta \left(\frac{\kappa_2^2}{p}\right)$ .
Therefore, solving
$$
\begin{array}{l} 0 = \langle \pmb {b} _ {k} (T _ {\mathrm {I}}), \pmb {x} _ {-} \rangle + \int_ {T _ {\mathrm {I}}} ^ {T _ {\mathrm {I I}} ^ {*}} \frac {\sqrt {m}}{\kappa_ {2} m _ {+}} \Big (\mathcal {U} (s) \cos \Delta - \mathcal {V} (s) \Big) \mathrm {d} s \\ = \Theta \left(\frac {\sqrt {\kappa_ {1} \kappa_ {2}}}{\sqrt {m}}\right) + \frac {\sqrt {m}}{\kappa_ {2} m _ {+}} \log \left(\frac {\mathcal {U} (T _ {\mathrm {I}}) ^ {\frac {m _ {-} \cos \Delta}{(m _ {-} + m _ {+})}}}{\mathcal {U} (T _ {\mathrm {I I}} ^ {*}) ^ {\frac {m _ {-} \cos \Delta}{(m _ {-} + m _ {+})}}} \cdot \frac {\mathcal {V} (T _ {\mathrm {I I}} ^ {*}) ^ {\frac {m _ {+}}{m _ {-} + m _ {+}}}}{\mathcal {V} (T _ {\mathrm {I}}) ^ {\frac {m _ {+}}{m _ {-} + m _ {+}}}}\right) \\ = \Theta \left(\frac {\sqrt {\kappa_ {1} \kappa_ {2}}}{\sqrt {m}}\right) + \Theta \left(\frac {1}{\kappa_ {2} \sqrt {m}}\right) \log \left(\frac {\mathcal {U} (T _ {\mathrm {I}}) ^ {\frac {m _ {-} \cos \Delta}{(m _ {-} + m _ {+})}}}{\mathcal {U} (T _ {\mathrm {I I}} ^ {*}) ^ {\frac {m _ {-} \cos \Delta}{(m _ {-} + m _ {+})}}} \cdot \frac {\mathcal {V} (T _ {\mathrm {I I}} ^ {*}) ^ {\frac {m _ {+}}{m _ {-} + m _ {+}}}}{\mathcal {V} (T _ {\mathrm {I}}) ^ {\frac {m _ {+}}{m _ {-} + m _ {+}}}}\right), \\ \end{array}
$$
$$
\frac {\mathcal {V} (T _ {\mathrm {I I}} ^ {*}) ^ {\frac {m _ {+}}{m _ {-} + m _ {+}}}}{\mathcal {U} (T _ {\mathrm {I I}} ^ {*}) ^ {\frac {m _ {-} \cos \Delta}{(m _ {-} + m _ {+})}}} \cdot \frac {\mathcal {U} (T _ {\mathrm {I}}) ^ {\frac {m _ {-} \cos \Delta}{(m _ {-} + m _ {+})}}}{\mathcal {V} (T _ {\mathrm {I}}) ^ {\frac {m _ {+}}{m _ {-} + m _ {+}}}} = \exp \left(- \Theta \left(\kappa_ {2} \sqrt {\kappa_ {1} \kappa_ {2}}\right)\right) = \Theta (1).
$$
A straight-forward calculation gives us:
$$
\begin{array}{l} \frac {\mathcal {V} (T _ {\mathrm {I I}} ^ {*}) ^ {\frac {m _ {+}}{m _ {-} + m _ {+}}}}{\mathcal {U} (T _ {\mathrm {I I}} ^ {*}) ^ {\frac {m _ {-} \cos \Delta}{(m _ {-} + m _ {+})}}} \cdot \frac {\mathcal {U} (T _ {\mathrm {I}}) ^ {\frac {m _ {-} \cos \Delta}{(m _ {-} + m _ {+})}}}{\mathcal {V} (T _ {\mathrm {I}}) ^ {\frac {m _ {+}}{m _ {-} + m _ {+}}}} \\ = \Theta \left(\frac {1}{\left(\frac {p ^ {\frac {1}{1 + \cos \Delta}}}{\kappa_ {2} ^ {2}} + \Delta^ {2} (T _ {\mathrm {I I}} ^ {*} - \tau_ {1})\right) ^ {\frac {m _ {+} - m _ {-} \cos \Delta}{m _ {+} + m _ {-}}}} \cdot \frac {\left(\kappa_ {2} ^ {2}\right) ^ {\frac {m _ {-} \cos \Delta}{m _ {+} + m _ {-}}}}{\left(\frac {\kappa_ {2} ^ {2}}{p}\right) ^ {\frac {m _ {+}}{m _ {+} + m _ {-}}}}\right) \\ \end{array}
$$
$$
= \Theta \left(\frac {p ^ {\frac {m _ {+}}{m _ {+} + m _ {-}}}}{\left(p ^ {\frac {1}{1 + \cos \Delta}} + \kappa_ {2} ^ {2} \Delta^ {2} (T _ {\mathrm {I I}} ^ {*} - \tau_ {1})\right) ^ {\frac {m _ {+} - m _ {-} \cos \Delta}{m _ {+} + m _ {-}}}}\right).
$$
$$
T _ {\mathrm {I I}} ^ {*} - \tau_ {1} = \Theta \left(\frac {1}{\kappa_ {2} ^ {2} \Delta^ {2}} \left(p ^ {\frac {m _ {+}}{m _ {+} - m _ {-} \cos \Delta}} - \Theta \left(p ^ {\frac {1}{1 + \cos \Delta}}\right)\right)\right) = \Theta \left(\frac {p ^ {\frac {1}{1 - \alpha \cos \Delta}}}{\kappa_ {2} ^ {2} \Delta^ {2}}\right),
$$
Combining Lemma D.4 (S3), we obtain
$$
T _ {\mathrm {I I}} ^ {+} = T _ {\mathrm {I I}} ^ {*} = \tau_ {1} + \Theta \left(\frac {p ^ {\frac {1}{1 - \alpha \cos \Delta}}}{\kappa_ {2} ^ {2} \Delta^ {2}}\right) = \mathcal {O} \left(\frac {p ^ {\frac {1}{1 + \cos \Delta} \log (1 / \Delta)}}{\kappa_ {2} ^ {2}}\right) + \Theta \left(\frac {p ^ {\frac {1}{1 - \alpha \cos \Delta}}}{\kappa_ {2} ^ {2} \Delta^ {2}}\right) = \Theta \left(\frac {p ^ {\frac {1}{1 - \alpha \cos \Delta}}}{\kappa_ {2} ^ {2} \Delta^ {2}}\right).
$$
Recall the relationship between  $T_{\mathrm{II}}$  and  $T_{\mathrm{II}}^{+}$  (19)(20):
$$
T _ {\mathrm {I I}} = T _ {\mathrm {I I}} ^ {+} \wedge \inf  \{t > T _ {\mathrm {I}}: \exists k \in \mathcal {K} _ {-}, \text {s . t .} \langle \boldsymbol {w} _ {k} (t), \boldsymbol {x} _ {+} \rangle \neq 0 \text {o r} \langle \boldsymbol {w} _ {k} (t), \boldsymbol {x} _ {-} \rangle \leq 0 \}.
$$
Then using Lemma D.1 (S2), we obtain:
$$
T _ {\mathrm {I I}} = T _ {\mathrm {I I}} ^ {+} = \Theta \left(\frac {p ^ {\frac {1}{1 - \alpha \cos \Delta}}}{\kappa_ {2} ^ {2} \Delta^ {2}}\right).
$$
Lemma D.7 (Length of Plateau).
If we define the hitting time  $T_{\mathrm{plat}} \coloneqq \inf \left\{t \in [T_{\mathrm{I}}, T_{\mathrm{II}}] : \mathrm{Acc}(t) = 1\right\}$ , then we have:
(S1).  $T_{\mathrm{plat}} = \Theta \left(\frac{p}{\kappa_2^2\Delta^2}\right)$
(S2).  $\forall t\in [T_{\mathrm{I}},T_{\mathrm{plat}}]$ $\operatorname {Acc}(t)\equiv \frac{p}{1 + p}$
(S3).  $\forall t\in (T_{\mathrm{plat}},T_{\mathrm{II}}],\mathrm{Acc}(t)\equiv 1.$
Proof of Lemma D.7.
$$
T _ {\text {p l a t}} = \inf  \left\{t \in [ T _ {\mathrm {I}}, T _ {\mathrm {I I}} ]: f _ {+} (t) \leq 0 \text {o r} f _ {-} (t) > 0 \right\}.
$$
From Theorem C.12 (S4), we know  $f_{+}(T_{\mathrm{I}}) > 0$  and  $f_{-}(T_{\mathrm{I}}) > 0$ . From Lemma D.3, we have
$$
\mathcal {U} (t) = \kappa_ {2} ^ {2} \frac {m _ {+}}{m} \frac {p}{1 + p} e ^ {- f _ {+} (t)}, \quad \mathcal {V} (t) = \kappa_ {2} ^ {2} \frac {m _ {+}}{m} \frac {1}{1 + p} e ^ {f _ {-} (t)}.
$$
From the proof of Lemma D.4, we know  $\frac{\mathrm{d}}{\mathrm{d}t}\mathcal{U}(t) < 0$ ,  $\forall t \in [T_{\mathrm{I}}, T_{\mathrm{II}}]$ , so
$$
\mathcal {U} (t) \leq \mathcal {U} \left(T _ {\mathrm {I}}\right), \quad f _ {+} (t) \geq f _ {+} \left(T _ {\mathrm {I}}\right) > 0, \forall t \in \left[ T _ {\mathrm {I}}, T _ {\mathrm {I I}} \right].
$$
Recall the definition of  $\tau_{1}$  and  $\tau_{2}$  in Lemma D.4. From the proof of Lemma D.4, we know  $\frac{\mathrm{d}}{\mathrm{d}t}\mathcal{V}(t) > 0$ ,  $\forall t\in [T_1,\tau_1)$ , so
$$
\mathcal {V} (t) \geq \mathcal {V} (T _ {\mathrm {I}}), f _ {-} (t) \geq f _ {-} (T _ {\mathrm {I}}) > 0, \forall t \in [ T _ {\mathrm {I}}, \tau_ {\mathrm {I}} ].
$$
With the help of Lemma D.4 (S4), we know
$$
\mathcal {V} (t) = \Theta \left(\frac {1}{\frac {\frac {1}{p ^ {\frac {1}{1 + \cos \Delta}} + \Delta^ {2}} (t - \tau_ {1})}{\kappa_ {2} ^ {2}}}\right), \forall t \in (\tau_ {1}, + \infty).
$$
Because  $\mathcal{V}(T_{\mathrm{plat}}) = \kappa_2^2\frac{m_+}{m}\frac{1}{1 + p} = \Theta \left(\frac{\kappa_2^2}{p}\right)$ , we have
$$
\Theta \left(\frac {\kappa_ {2} ^ {2}}{p}\right) = \Theta \left(\frac {1}{\frac {\frac {1}{p ^ {1 + \cos \Delta}}}{\kappa_ {2} ^ {2}} + \Delta^ {2} (T _ {\mathrm {p l a t}} - \tau_ {1})}\right), \forall t \in (\tau_ {1}, + \infty).
$$
$$
\begin{array}{l} T _ {\text {p l a t}} = \tau_ {1} + \Theta \left(\frac {p}{\kappa_ {2} ^ {2} \Delta^ {2}} \left(1 - \Theta \left(\frac {1}{p ^ {\frac {\cos \Delta}{1 + \cos \Delta}}}\right)\right)\right) = \tau_ {1} + \Theta \left(\frac {p}{\kappa_ {2} ^ {2} \Delta^ {2}}\right) \\ = \mathcal {O} \left(\frac {p ^ {\frac {1}{1 + \cos \Delta}} \log (1 / \Delta)}{\kappa_ {2} ^ {2}}\right) + \Theta \left(\frac {p}{\kappa_ {2} ^ {2} \Delta^ {2}}\right) = \Theta \left(\frac {p}{\kappa_ {2} ^ {2} \Delta^ {2}}\right). \\ \end{array}
$$
It is easy to verify  $T_{\mathrm{plat}} = \Theta \left(\frac{p}{\kappa_2^2\Delta^2}\right) < T_{\mathrm{II}} = \Theta \left(\frac{p^{\frac{1}{1 - \alpha}\cos\Delta}}{\kappa_2^2\Delta^2}\right)$ .
Because  $\tau_{2} - \tau_{1} = \Theta \left(\frac{\frac{1}{p + \cos\Delta}\log(1 / \Delta)}{\kappa_{2}^{2}}\right)$ , for any  $t\in (\tau_1,\tau_2]$ , we have
$$
\begin{array}{l} \mathcal {V} (t) = \Theta \left(\frac {1}{\frac {\frac {1}{p ^ {1 + \cos \Delta}}}{\kappa_ {2} ^ {2}} + \Delta^ {2} (t - \tau_ {1})}\right) = \Theta \left(\frac {1}{\frac {\frac {1}{p ^ {1 + \cos \Delta}}}{\kappa_ {2} ^ {2}} + \Delta^ {2} (\tau_ {2} - \tau_ {1})}\right) \\ = \Theta \left(\frac {\kappa_ {2} ^ {2}}{p ^ {\frac {1}{1 + \cos \Delta}}}\right) \gg \Theta \left(\frac {\kappa_ {2} ^ {2}}{p}\right) = \mathcal {V} (T _ {\mathrm {p l a t}}). \\ \end{array}
$$
$$
f _ {-} (t) > f _ {-} \left(T _ {\text {p l a t}}\right) = 0, \forall t \in \left(\tau_ {1}, \tau_ {2} \right\rbrack .
$$
From Lemma D.4 (S6), we know that  $\mathcal{U}(t)\cos \Delta -\mathcal{V}(t) < 0,\forall t\in (\tau_2,T_{\mathrm{II}}]$ . Then  $\frac{\mathrm{d}\mathcal{V}(t)}{\mathrm{d}t} < 0$ $\forall t\in (\tau_2,T_{\mathrm{II}}]$ . Thus,
$$
f _ {-} (t) > f _ {-} \left(T _ {\text {p l a t}}\right) = 0, \forall t \in \left(\tau_ {2}, T _ {\text {p l a t}}\right);
$$
$$
f _ {-} (t) <   f _ {-} \left(T _ {\text {p l a t}}\right) = 0, \forall t \in \left(T _ {\text {p l a t}}, T _ {\mathrm {I I}} \right].
$$
Hence, we know
$$
f _ {-} (t) \geq 0, \forall t \in \left[ T _ {\mathrm {I}}, T _ {\mathrm {p l a t}} \right];
$$
$$
f _ {-} (t) <   0, \forall t \in \left(T _ {\mathrm {p l a t}}, T _ {\mathrm {I I}} \right].
$$
In summary, we have proved (S1)(S2)(S3).
Lemma D.8 (Prediction at end of Phase II).
(S1) For the predictions, we have:
$$
\begin{array}{l} e ^ {- f _ {+} (T _ {\mathrm {I I}})} = \Theta \left(p ^ {- \frac {1}{1 - \alpha \cos \Delta}}\right), \quad e ^ {f _ {-} (T _ {\mathrm {I I}})} = \Theta \left(p ^ {- \frac {\alpha \cos \Delta}{1 - \alpha \cos \Delta}}\right), \quad \mathcal {L} (\boldsymbol {\theta} (T _ {\mathrm {I I}})) = \Theta \left(p ^ {- \frac {1}{1 - \alpha \cos \Delta}}\right); \\ \frac {p e ^ {- f _ {+} \left(T _ {\mathrm {I I}}\right)}}{1 + p} - \frac {e ^ {f _ {-} \left(T _ {\mathrm {I I}}\right)}}{1 + p} \cos \Delta = \Theta \left(\Delta^ {2} p ^ {- \frac {1}{1 - \alpha \cos \Delta}}\right), \\ \frac {p e ^ {- f _ {+} \left(T _ {\mathrm {I I}}\right)}}{1 + p} \cos \Delta - \frac {e ^ {f _ {-} \left(T _ {\mathrm {I I}}\right)}}{1 + p} = - \Theta \left(\Delta^ {2} p ^ {- \frac {1}{1 - \alpha \cos \Delta}}\right). \\ \end{array}
$$
(S2). For any  $k \in \mathcal{K}_{+}$ , we have:
$$
\langle \boldsymbol {b} _ {k} (T _ {\mathrm {I I}}), \boldsymbol {x} _ {-} \rangle = \mathcal {O} \left(\frac {\sqrt {\kappa_ {1} \kappa_ {2}}}{\sqrt {m}}\right).
$$
(S3). For any  $k \in \mathcal{K}_{-}$ , we have  $\langle \pmb{b}_k(T_{\mathrm{II}}), \pmb{x}_+ \rangle = 0$
Proof of (S1). Recall the definitions in Lemma D.3:
$$
\left\{ \begin{array}{l} \mathcal {U} (t) = \kappa_ {2} ^ {2} \frac {m _ {+}}{m} \frac {p}{1 + p} e ^ {- f _ {+} (t)}, \\ \mathcal {V} (t) = \kappa_ {2} ^ {2} \frac {m _ {+}}{m} \frac {1}{1 + p} e ^ {f _ {-} (t)}. \end{array} \right.
$$
From Lemma D.4 (S4) and Lemma D.6, we have
$$
\begin{array}{l} \mathcal {U} (t) = \Theta \left(\frac {1}{\frac {1}{p ^ {\frac {1 + \cos \Delta}{\kappa_ {2} ^ {2}}} + \Delta^ {2} (t - \tau_ {1})}}\right), \quad \mathcal {V} (t) = \Theta \left(\frac {1}{\frac {1}{p ^ {\frac {1 + \cos \Delta}{\kappa_ {2} ^ {2}}} + \Delta^ {2} (t - \tau_ {1})}}\right), \\ \tau_ {1} = \mathcal {O} \left(\frac {p ^ {\frac {1}{1 + \cos \Delta}} \log (1 / \Delta)}{\kappa_ {2} ^ {2}}\right), T _ {\mathrm {I I}} = \Theta \left(\frac {p ^ {\frac {1}{1 - \alpha \cos \Delta}}}{\kappa_ {2} ^ {2} \Delta^ {2}}\right). \\ \end{array}
$$
Therefore, we obtain the estimate:
$$
\begin{array}{l} e ^ {- f _ {+} (T _ {\mathrm {I I}})} = \Theta \left(\frac {1}{\kappa_ {2} ^ {2}} \frac {1}{\mathcal {U} (T _ {\mathrm {I I}})}\right) = \Theta \left(\frac {1}{\kappa_ {2} ^ {2}} \frac {1}{\frac {1}{p ^ {\frac {1 + \cos \Delta}{2}}} + \Delta^ {2} \frac {p ^ {\frac {1}{1 - \alpha \cos \Delta}}}{\kappa_ {2} ^ {2} \Delta^ {2}}}\right) = \Theta \left(p ^ {- \frac {1}{1 - \alpha \cos \Delta}}\right), \\ e ^ {f _ {-} (T _ {\mathrm {I I}})} = \Theta \left(\frac {p}{\kappa_ {2} ^ {2}} \frac {1}{\mathcal {V} (T _ {\mathrm {I I}})}\right) = \Theta \left(\frac {p}{\kappa_ {2} ^ {2}} \frac {1}{\frac {1}{p ^ {\frac {1}{1 + \cos \Delta}}} + \Delta^ {2} \frac {p ^ {\frac {1}{1 - \alpha \cos \Delta}}}{\kappa_ {2} ^ {2} \Delta^ {2}}}\right) = \Theta \left(p ^ {- \frac {\alpha \cos \Delta}{1 - \alpha \cos \Delta}}\right). \\ \end{array}
$$
Moreover, Lemma D.4 (S4)(S5)(S6) give us
$$
\begin{array}{l} \mathcal {U} (T _ {\mathrm {I I}}) - \mathcal {V} (T _ {\mathrm {I I}}) \cos \Delta = \Theta (\Delta^ {2} \mathcal {V} (T _ {\mathrm {I I}})) = \Theta (\kappa_ {2} ^ {2} \Delta^ {2} p ^ {- \frac {1}{1 - \alpha \cos \Delta}}), \\ \mathcal {U} (T _ {\mathrm {I I}}) \cos \Delta - \mathcal {V} (T _ {\mathrm {I I}}) = - \Theta (\Delta^ {2} \mathcal {V} (T _ {\mathrm {I I}})) = - \Theta (\kappa_ {2} ^ {2} \Delta^ {2} p ^ {- \frac {1}{1 - \alpha \cos \Delta}}). \\ \end{array}
$$
$$
\frac {p e ^ {- f _ {+} \left(T _ {\mathrm {I I}}\right)}}{1 + p} - \frac {e ^ {f _ {-} \left(T _ {\mathrm {I I}}\right)}}{1 + p} \cos \Delta = \frac {1}{\kappa_ {2} ^ {2}} \left(\mathcal {U} \left(T _ {\mathrm {I I}}\right) - \mathcal {V} \left(T _ {\mathrm {I I}}\right) \cos \Delta\right) = \Theta \left(\Delta^ {2} p ^ {- \frac {1}{1 - \alpha \cos \Delta}}\right),
$$
$$
\frac {p e ^ {- f _ {+} (T _ {\mathrm {I I}})}}{1 + p} \cos \Delta - \frac {e ^ {f _ {-} (T _ {\mathrm {I I}})}}{1 + p} = \frac {1}{\kappa_ {2} ^ {2}} \Big (\mathcal {U} (T _ {\mathrm {I I}}) \cos \Delta - \mathcal {V} (T _ {\mathrm {I I}}) \Big) = - \Theta \left(\Delta^ {2} p ^ {- \frac {1}{1 - \alpha \cos \Delta}}\right).
$$
Proof of (S2). Denote  $\mathcal{K}_{+}^{0} := \left\{k \in \mathcal{K}_{+} : \langle \pmb{w}_{k}(T_{\mathrm{II}}), \pmb{x}_{-} \rangle = 0\right\}$ . From the definition of  $T_{\mathrm{II}}$  and the proof in Phase II, we know that  $\langle \pmb{w}_{k}(T_{\mathrm{II}}), \pmb{x}_{-} \rangle > 0$  holds for any  $k \in \mathcal{K}_{+} - \mathcal{K}_{+}^{0}$ .
From the proof in Lemma D.5, for any  $k \in \mathcal{K}_{+}$ , it holds
$$
\begin{array}{l} \langle \boldsymbol {b} _ {k} (t), \boldsymbol {x} _ {-} \rangle = \left\langle \boldsymbol {b} _ {k} \left(T _ {\mathrm {I}}\right), \boldsymbol {x} _ {-} \right\rangle + \int_ {T _ {\mathrm {I}}} ^ {t} \left\langle \frac {\mathrm {d} \boldsymbol {b} _ {k} (s)}{\mathrm {d} s}, \boldsymbol {x} _ {-} \right\rangle \mathrm {d} s \\ = \langle \pmb {b} _ {k} (T _ {\mathrm {I}}), \pmb {x} _ {-} \rangle + \int_ {T _ {\mathrm {I}}} ^ {t} \frac {\sqrt {m}}{\kappa_ {2} m _ {+}} \Big (\mathcal {U} (s) \cos \Delta - \mathcal {V} (s) \Big) \mathrm {d} s, \forall t \in [ T _ {\mathrm {I}}, T _ {\mathrm {I I}} ]. \\ \end{array}
$$
Thus for any  $k\in \mathcal{K}_{+} - \mathcal{K}_{+}^{0}$  , we have
$$
\langle \boldsymbol {b} _ {k} (T _ {\mathrm {I I}}), \boldsymbol {x} _ {-} \rangle - \langle \boldsymbol {b} _ {k} (T _ {\mathrm {I}}), \boldsymbol {x} _ {-} \rangle = \langle \boldsymbol {b} _ {k _ {0}} (T _ {\mathrm {I I}}), \boldsymbol {x} _ {-} \rangle - \langle \boldsymbol {b} _ {k _ {0}} (T _ {\mathrm {I}}), \boldsymbol {x} _ {-} \rangle = - \langle \boldsymbol {b} _ {k _ {0}} (T _ {\mathrm {I}}), \boldsymbol {x} _ {-} \rangle ,
$$
$$
\left\langle \boldsymbol {b} _ {k} \left(T _ {\mathrm {I I}}\right), \boldsymbol {x} _ {-} \right\rangle = \left\langle \boldsymbol {b} _ {k} \left(T _ {\mathrm {I}}\right), \boldsymbol {x} _ {-} \right\rangle - \left\langle \boldsymbol {b} _ {k _ {0}} \left(T _ {\mathrm {I}}\right), \boldsymbol {x} _ {-} \right\rangle .
$$
From the proof of Lemma D.6, we know
$$
\langle \boldsymbol {b} _ {k} (T _ {\mathrm {I}}), \boldsymbol {x} _ {-} \rangle \leq \frac {1 2 \sqrt {\kappa_ {1} \kappa_ {2}}}{\sqrt {m}} \left(\frac {p \cos \Delta - 1}{\sqrt {p ^ {2} + 1 - 2 p \cos \Delta}} + 2 \sqrt {\epsilon} \sin \Delta\right),
$$
$$
\langle \pmb {b} _ {k} (T _ {\mathrm {I}}), \pmb {x} _ {-} \rangle \geq \frac {4 . 6 6 \sqrt {\kappa_ {1} \kappa_ {2}}}{\sqrt {m}} \left(\frac {p \cos \Delta - 1}{\sqrt {p ^ {2} + 1 - 2 p \cos \Delta}} - 2 \sqrt {\epsilon} \sin \Delta - \epsilon\right),
$$
where  $\epsilon = 4.2\sqrt{\kappa_1\kappa_2} +\frac{2}{1 + 0.7\left(1 + 9.9\sqrt{\frac{\kappa_2}{\kappa_1}}\right)^{1.15}}$  . This means
$$
\langle \boldsymbol {b} _ {k} (T _ {\mathrm {I}}), \boldsymbol {x} _ {-} \rangle = \Theta \left(\frac {\sqrt {\kappa_ {1} \kappa_ {2}}}{\sqrt {m}}\right).
$$
Hence, for any  $k\in \mathcal{K}_{+} - \mathcal{K}_{+}^{0}$
$$
\begin{array}{l} 0 <   \langle \boldsymbol {b} _ {k} (T _ {\mathrm {I I}}), \boldsymbol {x} _ {-} \rangle = | \langle \boldsymbol {b} _ {k} (T _ {\mathrm {I}}), \boldsymbol {x} _ {-} \rangle - \langle \boldsymbol {b} _ {k _ {0}} (T _ {\mathrm {I}}), \boldsymbol {x} _ {-} \rangle | \leq | \langle \boldsymbol {b} _ {k} (T _ {\mathrm {I}}), \boldsymbol {x} _ {-} \rangle | + | \langle \boldsymbol {b} _ {k _ {0}} (T _ {\mathrm {I}}), \boldsymbol {x} _ {-} \rangle | \\ = \Theta \left(\frac {\sqrt {\kappa_ {1} \kappa_ {2}}}{\sqrt {m}}\right) + \Theta \left(\frac {\sqrt {\kappa_ {1} \kappa_ {2}}}{\sqrt {m}}\right) = \Theta \left(\frac {\sqrt {\kappa_ {1} \kappa_ {2}}}{\sqrt {m}}\right). \\ \end{array}
$$
Proof of (S3). Due to the dynamics of the neuron  $k \in \mathcal{K}_{-}$  in Phase II, this conclusion is clear.
As simple corollaries of these lemmas, we can prove two theorems in Phase II.
Proof of Theorem 4.4 and 4.5.
Theorem 4.5 is Lemma D.7. Theorem 4.4 (S1) has been proven in Lemma D.6; Theorem 4.4 (S2) has been proven in Lemma D.8. Additionally, combining (i)  $T_{\mathrm{II}} = T_{\mathrm{II}}^{+}$  in Lemma D.6, (ii) the transformation in Lemma D.5, and (iii) the definition of  $T_{\mathrm{II}}$ , we obtain Theorem 4.4 (S3).
E Proofs of Optimization Dynamics in Phase III
E.1 Optimization Dynamics during Phase Transition
Building upon Phase II, we will demonstrate that within a short time, all the living positive neurons  $\mathcal{K}_{+}$  change their activation patterns, corresponding to a "phase transition". After the phase transition, all the living positive neurons  $k\in \mathcal{K}_{+}$  undergo deactivation for  $\pmb{x}_{-}$ , i.e.,  $\mathrm{sgn}_k^{-}(t)$  changes from 1 to 0, while other activation patterns remain unchanged.
Specifically, we define the hitting time
$$
\begin{array}{l} T _ {\mathrm {I I}} ^ {\mathrm {P T}} := \inf  \left\{t > T _ {\mathrm {I I}}: \forall k \in \mathcal {K} _ {+}, \operatorname {s g n} _ {k} ^ {-} (t) = 0 \right\} \tag {26} \\ = \inf  \left\{t > T _ {\mathrm {I I}}: \forall k \in \mathcal {K} _ {+}, \langle \boldsymbol {w} _ {k} (t), \boldsymbol {x} _ {-} \rangle = 0 \right\}, \\ \end{array}
$$
and we call  $t \in (T_{\mathrm{II}}, T_{\mathrm{II}}^{\mathrm{PT}}]$  "Phase Transition" from Phase II to Phase III.
Notice that the dynamics during phase transition is highly nonlinear with  $|\mathcal{K}_{+}| = \Theta (m)$  changes on activation partitions. Fortunately, we can keep the neurons of  $\mathcal{K}_{+}$  and  $\mathcal{K}_{-}$  close enough respectively in Phase I by using sufficiently small initialization  $\kappa_{1}$ . Moreover, their differences do not enlarge in Phase II. As a result, the phase transition can be completed quickly without significant changes in the vector field.
In order to analyze the dynamics of neurons and vector fields, we introduce the auxiliary hitting time:
$$
\begin{array}{l} T _ {\mathrm {I I}} ^ {\mathrm {P T} *} := T _ {\mathrm {I I}} ^ {\mathrm {P T}} \wedge \inf  \left\{t > T _ {\mathrm {I I}}: \langle \boldsymbol {F} _ {+} (t), \boldsymbol {x} _ {+} \rangle \leq 0 \text {o r} \langle \boldsymbol {F} _ {+} (t), \boldsymbol {x} _ {-} \rangle \geq 0 \right\}; \tag {27} \\ \text {w h e r e} \quad \boldsymbol {F} _ {+} (t) = \frac {p}{1 + p} e ^ {- f _ {+} (t)} \boldsymbol {x} _ {+} - \frac {1}{1 + p} e ^ {f _ {-} (t)} \boldsymbol {x} _ {-}. \\ \end{array}
$$
We call  $T_{\mathrm{II}} \leq t \leq T_{\mathrm{II}}^{\mathrm{PT*}}$  "Phase Transition".
Lemma E.1 (Dynamics of living neurons during Phase Transition*).
In Phase Transition*, i.e.,  $t \in [T_{\mathrm{II}}, T_{\mathrm{II}}^{\mathrm{PT*}}]$ , we have the following dynamics for each neuron  $k \in \mathcal{K}_{-} \cup \mathcal{K}_{+}$ .
(S1). For living negative neuron  $k \in \mathcal{K}_{-}$ , we have:
$$
\frac {\mathrm {d} \boldsymbol {b} _ {k} (t)}{\mathrm {d} t} = \frac {\kappa_ {2} e ^ {f _ {-} (t)}}{\sqrt {m} (1 + p)} \Big (\boldsymbol {x} _ {-} - \boldsymbol {x} _ {+} \cos \Delta \Big).
$$
(S2) For living positive neuron  $k \in \mathcal{K}_{+}$ , we define the hitting time:
$$
T _ {\mathrm {I I}, k} ^ {\mathrm {P T} *} := \inf  \left\{t > T _ {\mathrm {I I}}: \langle \boldsymbol {w} _ {k} (t), \boldsymbol {x} _ {-} \rangle = 0 \right\} \wedge \inf  \left\{t > T _ {\mathrm {I I}}: \langle \boldsymbol {F} _ {+} (t), \boldsymbol {x} _ {+} \rangle \leq 0 \text {o r} \langle \boldsymbol {F} _ {+} (t), \boldsymbol {x} _ {-} \rangle \geq 0 \right\}.
$$
Then it holds that:
$$
(P 0) T _ {\mathrm {I I}} ^ {\mathrm {P T} *} = \max  _ {k \in \mathcal {K} _ {+}} T _ {\mathrm {I I}, k} ^ {\mathrm {P T} *};
$$
(P1) For any  $t \in [T_{\mathrm{II}}, T_{\mathrm{II},k}^{\mathrm{PT*}})$ , we have
$$
\frac {\mathrm {d} \boldsymbol {b} _ {k} (t)}{\mathrm {d} t} = \frac {\kappa_ {2}}{\sqrt {m}} \boldsymbol {F} _ {+} (t);
$$
(P2) If  $T_{\mathrm{II},k}^{\mathrm{PT*}} < T_{\mathrm{II}}^{\mathrm{PT*}}$  strictly, then for any  $t \in [T_{\mathrm{II},k}^{\mathrm{PT*}}, T_{\mathrm{II}}^{\mathrm{PT*}}]$ , we have  $\mathbf{w}_k(t) \in \mathcal{M}_+^+ \cap \mathcal{M}_-^0$  and
$$
\frac {\mathrm {d} \boldsymbol {b} _ {k} (t)}{\mathrm {d} t} = \frac {\kappa_ {2} p e ^ {- f _ {+} (t)}}{\sqrt {m} (1 + p)} \left(\boldsymbol {x} _ {+} - \boldsymbol {x} _ {-} \cos \Delta\right).
$$
(P3) Regardless of the relationship between  $T_{\mathrm{II},k}^{\mathrm{PT*}}$  and  $T_{\mathrm{II}}^{\mathrm{PT*}}$ , for any  $t \in [T_{\mathrm{II}}, T_{\mathrm{II}}^{\mathrm{PT*}}]$ , we have
$$
\langle \boldsymbol {w} _ {k} (t), \boldsymbol {x} _ {+} \rangle > 0, \quad \langle \boldsymbol {w} _ {k} (t), \boldsymbol {x} _ {-} \rangle \geq 0.
$$
Proof of Lemma E.1.
Proof of (S1). Recalling the definition of  $T_{\mathrm{II}}^{\mathrm{PT*}}$ , it holds  $\langle \pmb{F}_{+}(t),\pmb{x}_{+}\rangle >0$  for any  $T_{\mathrm{II}}\leq t\leq T_{\mathrm{II}}^{\mathrm{PT*}}$ . So (S1) can be proved in the same way as employed in the proof of Lemma D.1 (S2) and is omitted.
Proof of (S2)(P0) and (S2)(P1). (S2)(P0) is obvious. Moreover, for any  $k \in \mathcal{K}_{+}$  and  $t \in [T_{\mathrm{II}}, T_{\mathrm{II},k}^{\mathrm{PT*}})$ , we have  $\langle \pmb{w}_k(t), \pmb{x}_+ \rangle > 0$  and  $\langle \pmb{w}_k(t), \pmb{x}_- \rangle > 0$  for any  $T_{\mathrm{I}} \leq t \leq T_{\mathrm{II}}^{*}$ , so (S2)(P1) can be proved in the same method as shown in the proof of Lemma D.1 (S1), and we have the dynamics:
$$
\frac {\mathrm {d} \boldsymbol {b} _ {k} (t)}{\mathrm {d} t} = \frac {\kappa_ {2}}{\sqrt {m}} \boldsymbol {F} _ {+} (t), \quad t \in [ T _ {\mathrm {I I}}, T _ {\mathrm {I I}, k} ^ {\mathrm {P T} *}).
$$
Additionally, recalling the definition of  $T_{\mathrm{II},k}^{\mathrm{PT*}}$ , we know  $\langle \pmb{F}_{+}(t),\pmb{x}_{+}\rangle >0$  holds for any  $t\in [T_{\mathrm{II}},T_{\mathrm{II},k}^{\mathrm{PT*}})$ . Combining the dynamics of  $\pmb {b}_k(t)$ , we further have:
$$
\begin{array}{l} \langle \boldsymbol {b} _ {k} (t), \boldsymbol {x} _ {+} \rangle = \langle \boldsymbol {b} _ {k} (T _ {\mathrm {I I}}), \boldsymbol {x} _ {+} \rangle + \frac {\kappa_ {2}}{\sqrt {m}} \int_ {T _ {\mathrm {I I}}} ^ {t} \langle \boldsymbol {F} _ {+} (s), \boldsymbol {x} _ {+} \rangle \mathrm {d} s \\ > \left\langle \boldsymbol {b} _ {k} \left(T _ {\mathrm {I I}}\right), \boldsymbol {x} _ {+} \right\rangle > 0, \quad \forall t \in \left[ T _ {\mathrm {I I}}, T _ {\mathrm {I I}, k} ^ {\mathrm {P T} *} \right]. \\ \end{array}
$$
Proof of (S2)(P2). Let  $k \in \mathcal{K}_+$ . If  $T_{\mathrm{II},k}^{\mathrm{PT*}} < T_{\mathrm{II}}^{\mathrm{PT*}}$ , we have the following results:
Step I.  $\pmb{w}_k(T_{\mathrm{II},k}^{\mathrm{PT*}}) \in \mathcal{M}_+^+ \cap \mathcal{M}_-^0$ .