SlowGuess/NeurIPS_2023_OCR huggingface dataset:
then  $\kappa_2^2\Delta^2 p^{-\frac{1}{1 - \alpha\cos\Delta}}\delta_T = \mathcal{O}(1)$  and
$$
| \langle \boldsymbol {F} _ {+} (t), \boldsymbol {x} _ {+} \rangle - \langle \boldsymbol {F} _ {+} (T _ {\mathrm {I I}}), \boldsymbol {x} _ {+} \rangle | = \mathcal {O} \left(\Delta^ {2} p ^ {- \frac {1}{1 - \alpha \cos \Delta}}\right) \leq \frac {1}{2} Q _ {1} \Delta^ {2} p ^ {- \frac {1}{1 - \alpha \cos \Delta}},
$$
$$
\begin{array}{l} \langle \boldsymbol {F} _ {+} (t), \boldsymbol {x} _ {+} \rangle \geq \langle \boldsymbol {F} _ {+} (T _ {\mathrm {I I}}), \boldsymbol {x} _ {+} \rangle - | \langle \boldsymbol {F} _ {+} (t), \boldsymbol {x} _ {+} \rangle - \langle \boldsymbol {F} _ {+} (T _ {\mathrm {I I}}), \boldsymbol {x} _ {+} \rangle | \\ \geq \frac {1}{2} Q _ {1} \Delta^ {2} p ^ {- \frac {1}{1 - \alpha \cos \Delta}} = \Omega \left(\Delta^ {2} p ^ {- \frac {1}{1 - \alpha \cos \Delta}}\right), \quad \forall t \in [ T _ {\mathrm {I I}}, T _ {\mathrm {f i x}} \wedge T _ {\mathrm {I I}} ^ {\mathrm {P T *}} ]. \\ \end{array}
$$
Step II. Bounding the term  $\langle F_{+}(t),\pmb{x}_{-}\rangle$
The proof can be completed by the method analogous to that used in Step I, and we omit it. The result is
$$
\begin{array}{l} \langle \boldsymbol {F} _ {+} (t), \boldsymbol {x} _ {-} \rangle \geq \langle \boldsymbol {F} _ {+} (T _ {\Pi}), \boldsymbol {x} _ {-} \rangle + | \langle \boldsymbol {F} _ {+} (t), \boldsymbol {x} _ {+} \rangle - \langle \boldsymbol {F} _ {+} (T _ {\Pi}), \boldsymbol {x} _ {-} \rangle | \\ \geq - \frac {1}{2} Q _ {2} \Delta^ {2} p ^ {- \frac {1}{1 - \alpha \cos \Delta}} \equiv - \Omega \left(\Delta^ {2} p ^ {- \frac {1}{1 - \alpha \cos \Delta}}\right), \quad \forall t \in [ T _ {\mathrm {I I}}, T _ {\mathrm {f i x}} \wedge T _ {\mathrm {I I}} ^ {\mathrm {P T *}} ]. \\ \end{array}
$$
Lemma E.4 (The end of Phase Transition).
If we choose  $\kappa_{1},\kappa_{2}$  s.t  $\kappa_{2} = \mathcal{O}(1)$  and  $\sqrt{\frac{\kappa_1}{\kappa_2}} = \mathcal{O}\left(\Delta^4\right)$  (3), then it holds that
(S1) (Time).
$$
T _ {\mathrm {I I}} ^ {\mathrm {P T}} = T _ {\mathrm {I I}} ^ {\mathrm {P T} *} = T _ {\mathrm {I I}} + \mathcal {O} \left(\sqrt {\frac {\kappa_ {1}}{\kappa_ {2}}} \frac {p ^ {\frac {1}{1 - \alpha \cos \Delta}}}{\Delta^ {2}}\right) = \left(1 + \mathcal {O} \left(\sqrt {\kappa_ {1} \kappa_ {2} ^ {3}}\right)\right) T _ {\mathrm {I I}};
$$
(S2) (Prediction).
$$
\begin{array}{l} e ^ {- f _ {+} (T _ {\mathrm {I I}} ^ {\mathrm {P T}})} = \Theta \left(p ^ {- \frac {1}{1 - \alpha \cos \Delta}}\right), e ^ {f _ {-} (T _ {\mathrm {I I}} ^ {\mathrm {P T}})} = \Theta \left(p ^ {- \frac {\alpha \cos \Delta}{1 - \alpha \cos \Delta}}\right); \\ \frac {p e ^ {- f _ {+} (T _ {\mathrm {I I}} ^ {\mathrm {P T}})}}{1 + p} - \frac {e ^ {f _ {-} (T _ {\mathrm {I I}} ^ {\mathrm {P T}})}}{1 + p} \cos \Delta = \Theta \left(\Delta^ {2} p ^ {- \frac {1}{1 - \alpha \cos \Delta}}\right), \\ \frac {p e ^ {- f _ {+} (T _ {\mathrm {I I}} ^ {\mathrm {P T}})}}{1 + p} \cos \Delta - \frac {e ^ {f _ {-} (T _ {\mathrm {I I}} ^ {\mathrm {P T}})}}{1 + p} = - \Theta \left(\Delta^ {2} p ^ {- \frac {1}{1 - \alpha \cos \Delta}}\right). \\ \end{array}
$$
(S3) (Activation patterns).
$$
\left\langle \boldsymbol {w} _ {k} \left(T _ {\mathrm {I I}} ^ {\mathrm {P T}}\right), \boldsymbol {x} _ {+} \right\rangle > 0, \left\langle \boldsymbol {w} _ {k} \left(T _ {\mathrm {I I}} ^ {\mathrm {P T}}\right), \boldsymbol {x} _ {-} \right\rangle = 0, \forall k \in \mathcal {K} _ {+};
$$
$$
\left\langle \boldsymbol {w} _ {k} \left(T _ {\mathrm {I I}} ^ {\mathrm {P T}}\right), \boldsymbol {x} _ {+} \right\rangle = 0, \left\langle \boldsymbol {w} _ {k} \left(T _ {\mathrm {I I}} ^ {\mathrm {P T}}\right), \boldsymbol {x} _ {-} \right\rangle > 0, \forall k \in \mathcal {K} _ {-}.
$$
Step I. Time Estimate. Let  $k \in \mathcal{K}_{+}$ .
Recalling the definition of  $T_{\mathrm{II},k}^{\mathrm{PT*}}$  in Lemma E.1, Lemma E.1 (S2)(P0) also gives us
$$
T _ {\mathrm {I I}} ^ {\mathrm {P T} *} = \max _ {k \in \mathcal {K} _ {+}} T _ {\mathrm {I I}, k} ^ {\mathrm {P T} *}.
$$
From Lemma D.8 (S2), we know that there exists an absolute constant  $Q_{3} > 0$ , s.t.  $0 \leq \langle \pmb{b}_{k}(T_{\mathrm{II}}), \pmb{x}_{-} \rangle \leq Q_{3} \frac{\sqrt{\kappa_{1} \kappa_{2}}}{\sqrt{m}}$ . And we let  $Q_{2} > 0$  be the absolute constant  $Q_{2}$  in Lemma E.3.
First, we choose the time
$$
T _ {\mathrm {f i x}} = T _ {\mathrm {I I}} + \frac {3 Q _ {3}}{Q _ {2} \Delta^ {2}} \sqrt {\frac {\kappa_ {1}}{\kappa_ {2}}} p ^ {\frac {1}{1 - \alpha \cos \Delta}}.
$$
then we choose  $\kappa_{1},\kappa_{2}$  s.t.
$$
\kappa_ {2} = \mathcal {O} (1), \quad \sqrt {\frac {\kappa_ {1}}{\kappa_ {2}}} = \mathcal {O} \left(\Delta^ {4}\right).
$$
It can ensure
$$
\kappa_ {2} ^ {2} \Big (T _ {\mathrm {f i x}} \wedge T _ {\mathrm {I I}} ^ {\mathrm {P T *}} - T _ {\mathrm {I I}} \Big) p ^ {- \frac {1}{1 - \alpha \cos \Delta}} = \Theta \left(\frac {\kappa_ {2} ^ {2}}{\Delta^ {2}} \sqrt {\frac {\kappa_ {1}}{\kappa_ {2}}}\right) = \mathcal {O} (\Delta^ {2}), \quad \kappa_ {2} ^ {2} \sqrt {\frac {\kappa_ {1}}{\kappa_ {2}}} = \mathcal {O} (\Delta^ {2}).
$$
Then according to Lemma E.3, it follows that
$$
\langle \pmb {F} _ {+} (t), \pmb {x} _ {-} \rangle \leq - \frac {Q _ {2}}{2} \Delta^ {2} p ^ {- \frac {1}{1 - \alpha \cos \Delta}}, \quad \forall t \in [ T _ {\mathrm {I I}}, T _ {\mathrm {I I}, k} ^ {\mathrm {P T *}} \wedge T _ {\mathrm {f i x}}).
$$
Now we consider the dynamics for  $t \in [T_{\mathrm{II}}, T_{\mathrm{II},k}^{\mathrm{PT*}} \wedge T_{\mathrm{fix}})$ .
Recalling lemma E.1, we have
$$
\begin{array}{l} \left\langle \boldsymbol {b} _ {k} \left(T _ {\mathrm {I I}, k} ^ {\mathrm {P T} *} \wedge T _ {\mathrm {f i x}}\right), \boldsymbol {x} _ {-} \right\rangle = \left\langle \boldsymbol {b} _ {k} \left(T _ {\mathrm {I I}}\right), \boldsymbol {x} _ {-} \right\rangle + \int_ {T _ {\mathrm {I I}}} ^ {t} \left\langle \frac {\mathrm {d} \boldsymbol {b} _ {k} (s)}{\mathrm {d} s}, \boldsymbol {x} _ {+} \right\rangle \mathrm {d} s \\ = \left\langle \boldsymbol {b} _ {k} \left(T _ {\mathrm {I I}}\right), \boldsymbol {x} _ {-} \right\rangle + \frac {\kappa_ {2}}{\sqrt {m}} \int_ {T _ {\mathrm {I I}}} ^ {T _ {\mathrm {I I}, k} ^ {\mathrm {P T *}} \wedge T _ {\mathrm {f i x}}} \left\langle \boldsymbol {F} _ {+} (s), \boldsymbol {x} _ {-} \right\rangle \mathrm {d} s \\ \leq Q _ {3} \frac {\sqrt {\kappa_ {1} \kappa_ {2}}}{\sqrt {m}} - \frac {Q _ {2}}{2} \Delta^ {2} p ^ {- \frac {1}{1 - \alpha \cos \Delta}} \left(T _ {\mathrm {I I}, k} ^ {\mathrm {P T} *} \wedge T _ {\mathrm {f i x}} - T _ {\mathrm {I I}}\right) \\ \leq Q _ {3} \frac {\sqrt {\kappa_ {1} \kappa_ {2}}}{\sqrt {m}} - \frac {Q _ {2}}{2} \Delta^ {2} p ^ {- \frac {1}{1 - \alpha \cos \Delta}} \left((T _ {\mathrm {I I I}, k} - T _ {\mathrm {I I}}) \wedge \frac {3 Q _ {3}}{Q _ {2} \Delta^ {2}} \sqrt {\frac {\kappa_ {1}}{\kappa_ {2}}} p ^ {\frac {1}{1 - \alpha \cos \Delta}}\right). \\ \end{array}
$$
We claim  $T_{\mathrm{II},k}^{\mathrm{PT*}} - T_{\mathrm{II}} \leq \frac{2Q_3}{Q_2\Delta^2}\sqrt{\frac{\kappa_1}{\kappa_2}} p^{\frac{1}{1 - \alpha\cos\Delta}}$ . If otherwise, then
$$
\left\langle \boldsymbol {b} _ {k} \left(T _ {\mathrm {I I}, k} ^ {\mathrm {P T} *} \wedge T _ {\mathrm {f i x}}\right), \boldsymbol {x} _ {-} \right\rangle <   Q _ {3} \frac {\sqrt {\kappa_ {1} \kappa_ {2}}}{\sqrt {m}} - Q _ {3} \frac {\sqrt {\kappa_ {1} \kappa_ {2}}}{\sqrt {m}} = 0.
$$
From the definition of  $T_{\mathrm{II},k}^{\mathrm{PT*}}$ , we know  $T_{\mathrm{II},k}^{\mathrm{PT*}} < T_{\mathrm{II},k}^{\mathrm{PT*}} \wedge T_{\mathrm{fix}}$ , which leads to a contradiction.
therefore, we have proved that for any  $k\in \mathcal{K}_{+}$
$$
T _ {\mathrm {I I}, k} ^ {\mathrm {P T} *} \wedge T _ {\mathrm {f i x}} = T _ {\mathrm {I I}, k} ^ {\mathrm {P T} *};
$$
$$
T _ {\mathrm {I I}, k} ^ {\mathrm {P T} *} \leq T _ {\mathrm {I I}} + \frac {2 Q _ {3}}{Q _ {2} \Delta^ {2}} \sqrt {\frac {\kappa_ {1}}{\kappa_ {2}}} p ^ {\frac {1}{1 - \alpha} \cos \Delta}.
$$
With the help of Lemma E.1 (S2)(P0), we obtain
$$
T _ {\mathrm {I I}} ^ {\mathrm {P T} *} \wedge T _ {\mathrm {f i x}} = T _ {\mathrm {I I}} ^ {\mathrm {P T} *};
$$
$$
T _ {\mathrm {I I}} ^ {\mathrm {P T} *} = \max  _ {k \in \mathcal {K} _ {+}} T _ {\mathrm {I I}, k} ^ {\mathrm {P T} *} \leq T _ {\mathrm {I I}} + \frac {2 Q _ {3}}{Q _ {2} \Delta^ {2}} \sqrt {\frac {\kappa_ {1}}{\kappa_ {2}}} p ^ {\frac {1}{1 - \alpha \cos \Delta}}.
$$
Recalling Lemma E.3,  $\langle \pmb{F}_{+}(t),\pmb{x}_{+}\rangle >0$  and  $\langle \pmb {F}_+(t),\pmb {x}_-\rangle < 0$  hold for any  $t\in [T_{\mathrm{II}},T_{\mathrm{II}}^{\mathrm{PT}*}\wedge T_{\mathrm{fix}}] = [T_{\mathrm{II}},T_{\mathrm{II}}^{\mathrm{PT}*}]$ . From the definitions of  $T_{\mathrm{II}}^{\mathrm{PT}}$  and  $T_{\mathrm{II}}^{\mathrm{PT}*}$  (26)(27), we obtain
$$
T _ {\mathrm {I I}} ^ {\mathrm {P T}} = T _ {\mathrm {I I}} ^ {\mathrm {P T} *}.
$$
In conclusion, we have proved:
$$
T _ {\mathrm {I I}} ^ {\mathrm {P T}} = T _ {\mathrm {I I}} ^ {\mathrm {P T} *} = T _ {\mathrm {I I}} + \mathcal {O} \left(\sqrt {\frac {\kappa_ {1}}{\kappa_ {2}}} \frac {p ^ {\frac {1}{1 - \alpha \cos \Delta}}}{\Delta^ {2}}\right) \stackrel {{\text {L e m m a}}} {{=}} ^ {\mathrm {D . 6}} \left(1 + \mathcal {O} \left(\sqrt {\kappa_ {1} \kappa_ {2} ^ {3}}\right)\right) T _ {\mathrm {I I}}.
$$
Step II. Prediction Estimate. Step I gives us the result:
$$
\delta_ {T} := T _ {\mathrm {I I}} ^ {\mathrm {P T}} - T _ {\mathrm {I I}} = \mathcal {O} \left(\sqrt {\frac {\kappa_ {1}}{\kappa_ {2}}} \frac {p ^ {\frac {1}{1 - \alpha \cos \Delta}}}{\Delta^ {2}}\right).
$$
Recalling the proof of Lemma E.2, we know
$$
\begin{array}{l} - \frac {C _ {1} + C _ {0} e ^ {f _ {-} (T _ {\mathrm {I I}})} \delta_ {T}}{1 + C _ {0} e ^ {f _ {-} (T _ {\mathrm {I I}})} \delta_ {T}} \leq e ^ {f _ {-} (t) - f _ {-} (T _ {\mathrm {I I}})} - 1 \leq 0, \\ - C _ {2} \delta_ {T} \leq e ^ {f _ {+} (T _ {\mathrm {H}}) - f _ {+} (t)} - 1 \leq 0. \\ \end{array}
$$
$$
C _ {0} = \Theta \left(\frac {\kappa_ {2} ^ {2} \Delta^ {2}}{p}\right), C _ {1} = \mathcal {O} \left(\kappa_ {2} \sqrt {\kappa_ {1} \kappa_ {2}}\right), C _ {2} = \Theta \left(\kappa_ {2} ^ {2} p ^ {- \frac {1}{1 - \alpha \cos \Delta}}\right).
$$
Then a straightforward calculation gives us:
$$
\begin{array}{l} 0 \geq e ^ {f _ {-} (t) - f _ {-} \left(T _ {\mathrm {I I}}\right)} - 1 = - \mathcal {O} \left(\kappa_ {2} ^ {2} \sqrt {\frac {\kappa_ {1}}{\kappa_ {2}}}\right) - \mathcal {O} \left(\kappa_ {2} ^ {2} \sqrt {\frac {\kappa_ {1}}{\kappa_ {2}}}\right) = - \mathcal {O} \left(\kappa_ {2} ^ {2} \sqrt {\frac {\kappa_ {1}}{\kappa_ {2}}}\right) \\ 0 \geq e ^ {f _ {+} (T _ {\mathrm {I I}}) - f _ {+} (T _ {\mathrm {I I}} ^ {\mathrm {P T}})} - 1 = - \mathcal {O} \left(\kappa_ {2} ^ {2} \sqrt {\frac {\kappa_ {1}}{\kappa_ {2}}} \frac {1}{\Delta^ {2}}\right). \\ \end{array}
$$
With the help of Lemma D.8, we obtain the prediction estimate at the end of Phase III:
$$
\begin{array}{l} e ^ {- f _ {+} (T _ {\mathrm {I I}} ^ {\mathrm {P T}})} = e ^ {- f _ {+} (T _ {\mathrm {I I}})} e ^ {f _ {+} (T _ {\mathrm {I I}}) - f _ {+} (T _ {\mathrm {I I}} ^ {\mathrm {P T}})} = \Theta \left(e ^ {- f _ {+} (T _ {\mathrm {I I}})}\right) = \Theta \left(p ^ {- \frac {1}{1 - \alpha \cos \Delta}}\right), \\ e ^ {f _ {-} (T _ {\mathrm {I I}} ^ {\mathrm {P T}})} = e ^ {f _ {-} (T _ {\mathrm {I I}})} e ^ {f _ {-} (T _ {\mathrm {I I}} ^ {\mathrm {P T}}) - f _ {-} (T _ {\mathrm {I I}})} = \Theta \left(e ^ {f _ {-} (T _ {\mathrm {I I}})}\right) = \Theta \left(p ^ {- \frac {\alpha \cos \Delta}{1 - \alpha \cos \Delta}}\right). \\ \end{array}
$$
$$
\begin{array}{l} \left| \left(\frac {p e ^ {- f _ {+} \left(T _ {\mathrm {I I}} ^ {\mathrm {P T}}\right)}}{1 + p} - \frac {e ^ {f _ {-} \left(T _ {\mathrm {I I}} ^ {\mathrm {P T}}\right)}}{1 + p} \cos \Delta\right) - \left(\frac {p e ^ {- f _ {+} \left(T _ {\mathrm {I I}}\right)}}{1 + p} - \frac {e ^ {f _ {-} \left(T _ {\mathrm {I I}}\right)}}{1 + p} \cos \Delta\right) \right| \\ \leq \left| \frac {p e ^ {- f _ {+} (T _ {\mathrm {I I}})}}{1 + p} \right| \left| \frac {\frac {p e ^ {- f _ {+} (T _ {\mathrm {I I}} ^ {\mathrm {P T}})}}{1 + p}}{\frac {p e ^ {- f _ {+} (T _ {\mathrm {I I}})}}{1 + p}} - 1 \right| + \left| \frac {e ^ {f _ {-} (T _ {\mathrm {I I}})} \cos \Delta}}{1 + p} \right| \left| \frac {\frac {e ^ {f _ {-} (T _ {\mathrm {I I}} ^ {\mathrm {P T}})} \cos \Delta}{1 + p}}{\frac {e ^ {f _ {-} (T _ {\mathrm {I I}})}}{1 + p} \cos \Delta} - 1 \right| \\ = \mathcal {O} \left(p ^ {- \frac {1}{1 - \alpha \cos \Delta}} \kappa_ {2} ^ {2} \sqrt {\frac {\kappa_ {1}}{\kappa_ {2}}} \frac {1}{\Delta^ {2}}\right) + \mathcal {O} \left(p ^ {- \frac {1}{1 - \alpha \cos \Delta}} \kappa_ {2} ^ {2} \sqrt {\frac {\kappa_ {1}}{\kappa_ {2}}}\right) \\ = \mathcal {O} \left(p ^ {- \frac {1}{1 - \alpha \cos \Delta}} \kappa_ {2} ^ {2} \sqrt {\frac {\kappa_ {1}}{\kappa_ {2}}} \frac {1}{\Delta^ {2}}\right) \stackrel {{\kappa_ {1} \kappa_ {2} ^ {3} = \mathcal {O} (\Delta^ {8})}} {{=}} \mathcal {O} \left(\Delta^ {2} p ^ {- \frac {1}{1 - \alpha \cos \Delta}}\right), \\ \end{array}
$$
$$
\frac {p e ^ {- f _ {+} (T _ {\mathrm {I I}} ^ {\mathrm {P T}})}}{1 + p} - \frac {e ^ {f _ {-} (T _ {\mathrm {I I}} ^ {\mathrm {P T}})}}{1 + p} \cos \Delta = \Theta \left(\Delta^ {2} p ^ {- \frac {1}{1 - \alpha \cos \Delta}}\right).
$$
In the same way, we can obtain
$$
\frac {p e ^ {- f _ {+} \left(T _ {\mathrm {I I}} ^ {\mathrm {P T}}\right)}}{1 + p} \cos \Delta - \frac {e ^ {f _ {-} \left(T _ {\mathrm {I I}} ^ {\mathrm {P T}}\right)}}{1 + p} = - \Theta \left(\Delta^ {2} p ^ {- \frac {1}{1 - \alpha \cos \Delta}}\right).
$$
Step III. Activation Patterns.
Recall our proofs in Step I, we know that
$$
\left\langle \boldsymbol {w} _ {k} \left(T _ {\mathrm {I I}} ^ {\mathrm {P T}}\right), \boldsymbol {x} _ {-} \right\rangle = 0, \forall k \in \mathcal {K} _ {+}.
$$
Moreover, from the dynamics in Lemma E.1 (S1) and (S2)(P3), we obtain:
$$
\left\langle \boldsymbol {w} _ {k} \left(T _ {\mathrm {I I}} ^ {\mathrm {P T}}\right), \boldsymbol {x} _ {+} \right\rangle > 0, \forall k \in \mathcal {K} _ {+};
$$
Proof of Theorem 4.6.
Theorem 4.6 (S1) has been proven in Lemma E.4 (S1), and Theorem 4.6 (S2) has been proven in Lemma E.4 (S3).
E.2 Optimization Dynamics after Phase Transition
After Phase Transition  $(t > T_{\mathrm{II}}^{\mathrm{PT}})$ , we study the dynamics before the patterns of living neurons change again. Specifically, we define the following hitting time
$$
\begin{array}{l} T _ {\mathrm {I I I}} := \inf  \left\{t > T _ {\mathrm {I I}} ^ {\mathrm {P T}}: \exists k \in \mathcal {K} _ {+} \cup \mathcal {K} _ {-}, \operatorname {s g n} _ {k} ^ {+} (t) \neq \operatorname {s g n} _ {k} ^ {+} (T _ {\mathrm {I}}) \text {o r} \operatorname {s g n} _ {k} ^ {-} (t) \neq \operatorname {s g n} _ {k} ^ {-} (T _ {\mathrm {I}}) \right\} \\ = \inf  \left\{t > T _ {\mathrm {I I}} ^ {\mathrm {P T}}: \exists k \in \mathcal {K} _ {+}, \text {s . t .} \langle \boldsymbol {w} _ {k} (t), \boldsymbol {x} _ {+} \rangle \leq 0 \text {o r} \langle \boldsymbol {w} _ {k} (t), \boldsymbol {x} _ {-} \rangle \neq 0; \right. \tag {28} \\ \left. \text {o r} \exists k \in \mathcal {K} _ {-}, \text {s . t .} \langle \boldsymbol {w} _ {k} (t), \boldsymbol {x} _ {+} \rangle \neq 0 \text {o r} \langle \boldsymbol {w} _ {k} (t), \boldsymbol {x} _ {-} \rangle \leq 0 \right\}, \\ \end{array}
$$
and we call  $t\in (T_{\mathrm{II}}^{\mathrm{PT}},T_{\mathrm{III}})$  "L-Phase III".
Moreover, we call  $t \in [T_{\mathrm{II}}, T_{\mathrm{III}})$  "Phase III", i.e., "Phase Transition" + "L-Phase III".
$$
T _ {\mathrm {I I I}} ^ {*} := T _ {\mathrm {I I I}} \wedge \inf  \left\{t > T _ {\mathrm {I I}} ^ {\mathrm {P T}}: \langle \boldsymbol {F} _ {+} (t), \boldsymbol {x} _ {+} \rangle \leq 0 \text {o r} \langle \boldsymbol {F} _ {+} (t), \boldsymbol {x} _ {-} \rangle \geq 0 \right\}, \tag {29}
$$
$$
\text {w h e r e} \quad \boldsymbol {F} _ {+} (t) = \frac {p}{1 + p} e ^ {- f _ {+} (t)} \boldsymbol {x} _ {+} - \frac {1}{1 + p} e ^ {f _ {-} (t)} \boldsymbol {x} _ {-}.
$$
We call  $t\in (T_{\mathrm{II}}^{\mathrm{PT}},T_{\mathrm{III}}^{*})$  "L-Phase III\*".
Due to the almost simplest activation patterns, this phase is easier to analyze, and we only need to estimate the time and size of the changes in the vector field. Nevertheless, our challenge is to prove that all living negative neurons simultaneously change their activation patterns at  $T_{\mathrm{III}}^*$ , which also implies that  $T_{\mathrm{III}} = T_{\mathrm{III}}^*$ .
Lemma E.5 (Dynamics of activate neurons during L-Phase III*).
In  $L$ -Phase III*  $(t \in [T_{\mathrm{II}}^{\mathrm{PT}}, T_{\mathrm{III}}^*)$ ), we have the following dynamics for each neuron  $k \in \mathcal{K}_{-} \cup \mathcal{K}_{+}$ .
(S1). For negative neuron  $k \in \mathcal{K}_{-}$ , we have:
(S2) For positive neuron  $k \in \mathcal{K}_{+}$ , we have:
$$
\boldsymbol {w} _ {k} (t) \in \mathcal {M} _ {+} ^ {+} \cap \mathcal {M} _ {-} ^ {0},
$$
Proof of Lemma E.5.
From the definition of  $T_{\mathrm{III}}^{*}$ , we know that  $\langle F_{+}(t),\pmb{x}_{+}\rangle >0$  and  $\langle F_{+}(t),\pmb{x}_{-}\rangle < 0$  hold for any  $t\in [T_{\mathrm{II}}^{\mathrm{PT}},T_{\mathrm{III}}^{*})$ . Moreover, Lemma E.4 ensures that for  $k\in \mathcal{K}_{+}$ ,  $\pmb{w}_k(T_{\mathrm{II}}^{\mathrm{PT}})\in \mathcal{M}_+^0\cap \mathcal{M}_-^+$ ; for  $k\in \mathcal{K}_{-}$ ,  $\pmb{w}_k(T_{\mathrm{II}}^{\mathrm{PT}})\in \mathcal{M}_+^+\cap \mathcal{M}_-^0$ . Hence, this lemma can be proved in the same way as shown in the proof of Lemma E.1 (S1) and (S2)(P2). We do not repeat it here.
Lemma E.6 (Time and prediction estimate at the end of L-Phase III*). (S1) (Time).
$$
T _ {\mathrm {I I I}} ^ {*} = T _ {\mathrm {I I}} ^ {\mathrm {P T}} + \Theta \left(\frac {p ^ {\frac {1}{1 - \alpha \cos \Delta}}}{\kappa_ {2} ^ {2}}\right) = \left(1 + \Theta (\Delta^ {2})\right) T _ {\mathrm {I I}} ^ {\mathrm {P T}} = \left(1 + \Theta (\Delta^ {2})\right) T _ {\mathrm {I I}};
$$
$$
\begin{array}{l} e ^ {- f _ {+} (T _ {\mathrm {I I I}} ^ {*})} = \Theta \left(p ^ {- \frac {1}{1 - \alpha \cos \Delta}}\right), \quad e ^ {f _ {-} (T _ {\mathrm {I I I}} ^ {*})} = \Theta \left(p ^ {- \frac {\alpha \cos \Delta}{1 - \alpha \cos \Delta}}\right); \\ \frac {p e ^ {- f _ {+} \left(T _ {\mathrm {I I I}} ^ {*}\right)}}{1 + p} - \frac {e ^ {f _ {-} \left(T _ {\mathrm {I I I}} ^ {*}\right)}}{1 + p} \cos \Delta = 0, \\ \frac {p e ^ {- f _ {+} (T _ {\mathrm {I I}} ^ {\mathrm {P T}})}}{1 + p} \cos \Delta - \frac {e ^ {f _ {-} (T _ {\mathrm {I I}} ^ {\mathrm {P T}})}}{1 + p} = - \Theta \left(\Delta^ {2} p ^ {- \frac {1}{1 - \alpha \cos \Delta}}\right). \\ \end{array}
$$
Proof of Lemma E.6.
Step I. Explicit Solution to  $f_{+}(t)$  and  $f_{-}(t)$ .
For any  $t\in [T_{\mathrm{II}}^{\mathrm{PT}},T_{\mathrm{III}}^{*})$  , we have:
$$
f _ {+} (t) = \frac {\kappa_ {2}}{\sqrt {m}} \sum_ {k \in \mathcal {K} _ {+}} \boldsymbol {b} _ {k} ^ {\top} (t) \boldsymbol {x} _ {+},
$$
$$
f _ {-} (t) = - \frac {\kappa_ {2}}{\sqrt {m}} \sum_ {k \in \mathcal {K} _ {-}} \boldsymbol {b} _ {k} ^ {\top} (t) \boldsymbol {x} _ {-}.
$$
Let us consider the dynamics of  $f_{+}(t)$  and  $f_{-}(t)$ . With the help of Lemma E.5, these two dynamics are nearly independent:
$$
\frac {\mathrm {d} f _ {+} (t)}{\mathrm {d} t} = = \frac {\kappa_ {2}}{\sqrt {m}} \sum_ {k \in \mathcal {K} _ {+}} \left\langle \frac {\kappa_ {2} p e ^ {- f _ {+} (t)}}{\sqrt {m} (1 + p)} \Big (\pmb {x} _ {+} - \pmb {x} _ {-} \cos \Delta \Big), \pmb {x} _ {+} \right\rangle = \frac {\kappa_ {2} ^ {2} m _ {+} p \sin^ {2} \Delta}{m (1 + p)} e ^ {- f _ {+} (t)},
$$
$$
\frac {\mathrm {d} f _ {-} (t)}{\mathrm {d} t} = = - \frac {\kappa_ {2}}{\sqrt {m}} \sum_ {k \in \mathcal {K} _ {-}} \left\langle \frac {\kappa_ {2} e ^ {f _ {-} (t)}}{\sqrt {m} (1 + p)} \Big (\pmb {x} _ {-} - \pmb {x} _ {+} \cos \Delta \Big), \pmb {x} _ {-} \right\rangle = - \frac {\kappa_ {2} ^ {2} m _ {-} \sin^ {2} \Delta}{m (1 + p)} e ^ {f _ {-} (t)}.
$$
Their solutions are:
$$
e ^ {- f _ {+} (t)} = \frac {e ^ {- f _ {+} (T _ {\mathrm {I I}} ^ {\mathrm {P T}})}}{1 + e ^ {- f _ {+} (T _ {\mathrm {I I}} ^ {\mathrm {P T}})} \frac {\kappa_ {2} ^ {2} m _ {+} p \sin^ {2} \Delta}{m (1 + p)} (t - T _ {\mathrm {I I}} ^ {\mathrm {P T}})},
$$
$$
e ^ {f _ {-} (t)} = \frac {e ^ {f _ {-} (T _ {\mathrm {I I}} ^ {\mathrm {P T}})}}{1 + e ^ {f _ {-} (T _ {\mathrm {I I}} ^ {\mathrm {P T}})} \frac {\kappa_ {2} ^ {2} m _ {-} \sin^ {2} \Delta}{m (1 + p)} (t - T _ {\mathrm {I I}} ^ {\mathrm {P T}})}.
$$
Step II. Time Estimate of  $T_{\mathrm{III}}^*$ .
For simplicity, we denote  $G_{+} \coloneqq \frac{\kappa_{2}^{2}m_{+}p\sin^{2}\Delta}{m(1 + p)}$  and  $G_{-} \coloneqq \frac{\kappa_{2}^{2}m_{-}\sin^{2}\Delta}{m(1 + p)}$ .
First, we consider the evolution of the vector field  $\langle F_{+}(t),x_{-}\rangle$  ..
$$
\langle \boldsymbol {F} _ {+} (t), \boldsymbol {x} _ {-} \rangle = \frac {p e ^ {- f _ {+} (t)}}{1 + p} \cos \Delta - \frac {e ^ {f _ {-} (t)}}{1 + p}
$$
$$
\begin{array}{l} = \frac {1}{1 + p} \left(\frac {p e ^ {- f _ {+} \left(T _ {\mathrm {I I}} ^ {\mathrm {P T}}\right)} \cos \Delta}{1 + e ^ {- f _ {+} \left(T _ {\mathrm {I I}} ^ {\mathrm {P T}}\right)} G _ {+} \left(t - T _ {\mathrm {I I}} ^ {\mathrm {P T}}\right)} - \frac {e ^ {f _ {-} \left(T _ {\mathrm {I I}} ^ {\mathrm {P T}}\right)}}{1 + e ^ {f _ {-} \left(T _ {\mathrm {I I}} ^ {\mathrm {P T}}\right)} G _ {-} \left(t - T _ {\mathrm {I I}} ^ {\mathrm {P T}}\right)}\right) \\ = \frac {\left(p e ^ {- f _ {+} \left(T _ {\mathrm {I I}} ^ {\mathrm {P T}}\right)} \cos \Delta - e ^ {f _ {-} \left(T _ {\mathrm {I I}} ^ {\mathrm {P T}}\right)}\right) + e ^ {f _ {-} \left(T _ {\mathrm {I I}} ^ {\mathrm {P T}} - f _ {+} \left(T _ {\mathrm {I I}} ^ {\mathrm {P T}}\right)\right)} \left(p G _ {-} \cos \Delta - G _ {+}\right) \left(t - T _ {\mathrm {I I}} ^ {\mathrm {P T}}\right)}{\left(1 + p\right) \left(1 + e ^ {- f _ {+} \left(T _ {\mathrm {I I}} ^ {\mathrm {P T}}\right)} G _ {+} \left(t - T _ {\mathrm {I I}} ^ {\mathrm {P T}}\right)\right) \left(1 + e ^ {f _ {-} \left(T _ {\mathrm {I I}} ^ {\mathrm {P T}}\right)} G _ {-} \left(t - T _ {\mathrm {I I}} ^ {\mathrm {P T}}\right)\right)} \\ = \frac {(1 + p) \left\langle \boldsymbol {F} _ {+} (T _ {\mathrm {I I}} ^ {\mathrm {P T}}) , \boldsymbol {x} _ {-} \right\rangle + e ^ {f _ {-} (T _ {\mathrm {I I}} ^ {\mathrm {P T}} - f _ {+} (T _ {\mathrm {I I}} ^ {\mathrm {P T}})} (p G _ {-} \cos \Delta - G _ {+}) (t - T _ {\mathrm {I I}} ^ {\mathrm {P T}})}{(1 + p) (1 + e ^ {- f _ {+} (T _ {\mathrm {I I}} ^ {\mathrm {P T}})} G _ {+} (t - T _ {\mathrm {I I}} ^ {\mathrm {P T}})) (1 + e ^ {f _ {-} (T _ {\mathrm {I I}} ^ {\mathrm {P T}})} G _ {-} (t - T _ {\mathrm {I I}} ^ {\mathrm {P T}}))} <   0. \\ \end{array}
$$
Hence, the hitting time  $T_{\mathrm{III}}^{*}$  can be converted to the following  $T_{\mathrm{III}}^{**}$ :
$$
T _ {\mathrm {I I I}} ^ {*} = T _ {\mathrm {I I I}} ^ {* *} := T _ {\mathrm {I I I}} \wedge \inf  \left\{t > T _ {\mathrm {I I}} ^ {\mathrm {P T}}: \langle \boldsymbol {F} _ {+} (t), \boldsymbol {x} _ {+} \rangle \leq 0 \right\}.
$$
Then we consider  $\langle \pmb{F}_{+}(t),\pmb{x}_{+}\rangle$  ..