SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \langle \boldsymbol {F} _ {+} (t), \boldsymbol {x} _ {+} \rangle = \frac {p e ^ {- f _ {+} (t)}}{1 + p} - \frac {e ^ {f _ {-} (t)}}{1 + p} \cos \Delta \\ = \frac {1}{1 + p} \left(\frac {p e ^ {- f _ {+} \left(T _ {\mathrm {I I}} ^ {\mathrm {P T}}\right)}}{1 + e ^ {- f _ {+} \left(T _ {\mathrm {I I}} ^ {\mathrm {P T}}\right)} G _ {+} \left(t - T _ {\mathrm {I I}} ^ {\mathrm {P T}}\right)} - \frac {e ^ {f _ {-} \left(T _ {\mathrm {I I}} ^ {\mathrm {P T}}\right)} \cos \Delta}{1 + e ^ {f _ {-} \left(T _ {\mathrm {I I}} ^ {\mathrm {P T}}\right)} G _ {-} \left(t - T _ {\mathrm {I I}} ^ {\mathrm {P T}}\right)}\right) \\ = \frac {\left(p e ^ {- f _ {+} \left(T _ {\mathrm {I I}} ^ {\mathrm {P T}}\right)} - e ^ {f _ {-} \left(T _ {\mathrm {I I}} ^ {\mathrm {P T}}\right)} \cos \Delta\right) + e ^ {f _ {-} \left(T _ {\mathrm {I I}} ^ {\mathrm {P T}} - f _ {+} \left(T _ {\mathrm {I I}} ^ {\mathrm {P T}}\right)\right)} \left(p G _ {-} - G _ {+} \cos \Delta\right) \left(t - T _ {\mathrm {I I}} ^ {\mathrm {P T}}\right)}{\left(1 + p\right) \left(1 + e ^ {- f _ {+} \left(T _ {\mathrm {I I}} ^ {\mathrm {P T}}\right)} G _ {+} \left(t - T _ {\mathrm {I I}} ^ {\mathrm {P T}}\right)\right) \left(1 + e ^ {f _ {-} \left(T _ {\mathrm {I I}} ^ {\mathrm {P T}}\right)} G _ {-} \left(t - T _ {\mathrm {I I}} ^ {\mathrm {P T}}\right)\right)} \\ = \frac {(1 + p) \left\langle \boldsymbol {F} _ {+} (T _ {\mathrm {I I}} ^ {\mathrm {P T}}) , \boldsymbol {x} _ {+} \right\rangle + e ^ {f _ {-} (T _ {\mathrm {I I}} ^ {\mathrm {P T}}) - f _ {+} (T _ {\mathrm {I I}} ^ {\mathrm {P T}})} (p G _ {-} - G _ {+} \cos \Delta) (t - T _ {\mathrm {I I}} ^ {\mathrm {P T}})}{(1 + p) (1 + e ^ {- f _ {+} (T _ {\mathrm {I I}} ^ {\mathrm {P T}})} G _ {+} (t - T _ {\mathrm {I I}} ^ {\mathrm {P T}})) (1 + e ^ {f _ {-} (T _ {\mathrm {I I}} ^ {\mathrm {P T}})} G _ {-} (t - T _ {\mathrm {I I}} ^ {\mathrm {P T}}))}. \\ \end{array}
$$
From Lemma E.4, we know
$$
\begin{array}{l} (1 + p) \left\langle \boldsymbol {F} _ {+} \left(T _ {\mathrm {I I}} ^ {\mathrm {P T}}\right), \boldsymbol {x} _ {+} \right\rangle = (1 + p) \left(\frac {p e ^ {- f _ {+} (t)}}{1 + p} - \frac {e ^ {f _ {-} (t)}}{1 + p} \cos \Delta\right) = \Theta \left(\Delta^ {2} p ^ {- \frac {\alpha \cos \Delta}{1 - \alpha \cos \Delta}}\right), \\ e ^ {f _ {-} (T _ {\mathrm {I I}} ^ {\mathrm {P T}}) - f _ {+} (T _ {\mathrm {I I}} ^ {\mathrm {P T}})} = \Theta \left(e ^ {f _ {-} (T _ {\mathrm {I + I I}}) - f _ {+} (T _ {\mathrm {I + I I}})}\right) = \Theta \left(p ^ {- \frac {1 + \alpha \cos \Delta}{1 - \alpha \cos \Delta}}\right), \\ p G _ {-} - G _ {+} \cos \Delta = \frac {\kappa_ {2} ^ {2} p \sin^ {2} \Delta}{1 + p} \frac {(m _ {-} - m _ {+} \cos \Delta)}{m} = - \Theta (\kappa_ {2} ^ {2} \Delta^ {2}). \\ \end{array}
$$
These imply the hitting time:
$$
T _ {\mathrm {I I I}} ^ {*} = T _ {\mathrm {I I I}} ^ {* *} = T _ {\mathrm {I I}} ^ {\mathrm {P T}} + \Theta \left(\frac {\Delta^ {2} p ^ {- \frac {\alpha \cos \Delta}{1 - \alpha \cos \Delta}}}{p ^ {- \frac {1 + \alpha \cos \Delta}{1 - \alpha \cos \Delta}} \kappa_ {2} ^ {2} \Delta^ {2}}\right) = T _ {\mathrm {I I}} ^ {\mathrm {P T}} + \Theta \left(\frac {p ^ {\frac {1}{1 - \alpha \cos \Delta}}}{\kappa_ {2} ^ {2}}\right).
$$
Step III. Prediction estimate.
From the explicit solution in Step I and the time estimate in Step II, it is easy to verify
$$
e ^ {- f _ {+} (T _ {\mathrm {I I I}} ^ {*})} = \frac {e ^ {- f _ {+} (T _ {\mathrm {I I}} ^ {\mathrm {P T}})}}{1 + e ^ {- f _ {+} (T _ {\mathrm {I I}} ^ {\mathrm {P T}})} \frac {\kappa_ {2} ^ {2} m _ {+} p \sin^ {2} \Delta}{m (1 + p)} (T _ {\mathrm {I I I}} ^ {*} - T _ {\mathrm {I I}} ^ {\mathrm {P T}})} = \Theta \left(p ^ {- \frac {1}{1 - \alpha \cos \Delta}}\right),
$$
$$
e ^ {f _ {-} (T _ {\mathrm {I I I}} ^ {*})} = \frac {e ^ {f _ {-} (T _ {\mathrm {I I}} ^ {\mathrm {P T}})}}{1 + e ^ {f _ {-} (T _ {\mathrm {I I}} ^ {\mathrm {P T}})} \frac {\kappa_ {2} ^ {2} m _ {-} \sin^ {2} \Delta}{m (1 + p)} (T _ {\mathrm {I I I}} ^ {*} - T _ {\mathrm {I I}} ^ {\mathrm {P T}})} = \Theta \left(p ^ {- \frac {\alpha \cos \Delta}{1 - \alpha \cos \Delta}}\right).
$$
Recalling the calculation in Step II, we have:
$$
\left\langle \boldsymbol {F} _ {+} \left(T _ {\mathrm {I I I}} ^ {*}\right), \boldsymbol {x} _ {+} \right\rangle = \frac {p e ^ {- f _ {+} \left(T _ {\mathrm {I I I}} ^ {*}\right)}}{1 + p} - \frac {e ^ {f _ {-} \left(T _ {\mathrm {I I I}} ^ {*}\right)}}{1 + p} \cos \Delta = 0,
$$
$$
\left\langle \boldsymbol {F} _ {+} \left(T _ {\mathrm {I I I}} ^ {*}\right), \boldsymbol {x} _ {-} \right\rangle = \frac {p e ^ {- f _ {+} \left(T _ {\mathrm {I I I}} ^ {*}\right)}}{1 + p} \cos \Delta - \frac {e ^ {f _ {-} \left(T _ {\mathrm {I I I}} ^ {*}\right)}}{1 + p}
$$
$$
\begin{array}{l} = \frac {(1 + p) \left\langle \boldsymbol {F} _ {+} \left(T _ {\mathrm {I I}} ^ {\mathrm {P T}}\right) , \boldsymbol {x} _ {-} \right\rangle + e ^ {f _ {-} \left(T _ {\mathrm {I I}} ^ {\mathrm {P T}}\right) - f _ {+} \left(T _ {\mathrm {I I}} ^ {\mathrm {P T}}\right)} \left(p G _ {-} \cos \Delta - G _ {+}\right) \left(T _ {\mathrm {I I I}} ^ {*} - T _ {\mathrm {I I}} ^ {\mathrm {P T}}\right)}{(1 + p) \left(1 + e ^ {- f _ {+} \left(T _ {\mathrm {I I}} ^ {\mathrm {P T}}\right)} G _ {+} \left(T _ {\mathrm {I I I}} ^ {*} - T _ {\mathrm {I I}} ^ {\mathrm {P T}}\right)\right) \left(1 + e ^ {f _ {-} \left(T _ {\mathrm {I I}} ^ {\mathrm {P T}}\right)} G _ {-} \left(T _ {\mathrm {I I I}} ^ {*} - T _ {\mathrm {I I}} ^ {\mathrm {P T}}\right)\right)} \\ = \Theta \left(\frac {- \Delta^ {2} p ^ {- \frac {\alpha \cos \Delta}{1 - \alpha \cos \Delta}} - \Delta^ {2} p ^ {- \frac {\alpha \cos \Delta}{1 - \alpha \cos \Delta}}}{p (1 + \Theta (\Delta^ {2})) (1 + \Theta (\Delta^ {2}))}\right) = - \Theta (\Delta^ {2} p ^ {- \frac {1}{1 - \alpha \cos \Delta}}). \\ \end{array}
$$
Lemma E.7 (Hitting time relationship). If we define the following hitting time:
$$
T _ {\mathrm {I I I}} ^ {\mathrm {W}} = \inf  \left\{t > T _ {\mathrm {I I}} ^ {\mathrm {P T}}: \forall k \in \mathcal {K} _ {-}, \langle \boldsymbol {w} _ {k} (t), \boldsymbol {x} _ {+} \rangle > 0 \right\},
$$
then it holds that  $T_{\mathrm{III}} = T_{\mathrm{III}}^{*} = T_{\mathrm{III}}^{\mathrm{W}}$ .
Proof of Lemma E.7.
We define the following hitting time:
$$
\begin{array}{l} T _ {\mathrm {I I I}} ^ {\mathrm {F}} = \inf  \left\{t > T _ {\mathrm {I I}} ^ {\mathrm {P T}}: \langle \boldsymbol {F} _ {+} (t), \boldsymbol {x} _ {+} \rangle \leq 0 \right\}; \\ T _ {\mathrm {I I I}} ^ {\mathrm {N}} = \inf  \left\{t > T _ {\mathrm {I I}} ^ {\mathrm {P T}}: \exists k \in \mathcal {K} _ {-}, \text {s . t .} \langle \boldsymbol {w} _ {k} (t), \boldsymbol {x} _ {+} \rangle > 0 \right\}, \\ T _ {\mathrm {I I I}} ^ {\mathrm {W}} = \inf  \left\{t > T _ {\mathrm {I I}} ^ {\mathrm {P T}}: \forall k \in \mathcal {K} _ {-}, \langle \boldsymbol {w} _ {k} (t), \boldsymbol {x} _ {+} \rangle > 0 \right\}, \\ \end{array}
$$
From the proof in Lemma E.4, we know  $\langle \pmb{F}_{+}(T_{\mathrm{III}}^{*}),\pmb{x}_{-}\rangle < 0$ . From the continuity of  $\langle \pmb{F}_{+}(\cdot),\pmb{x}_{-}\rangle$  we know that there exists  $\tau_{1} > 0$ , such that  $\langle \pmb{F}_{+}(t),\pmb{x}_{-}\rangle < 0$  holds for any  $t\in [T_{\mathrm{III}}^{*},T_{\mathrm{III}}^{*} + \tau_{1})$ . Then in the same way as the proof of Lemma E.5 (S2), we know that for  $k\in \mathcal{K}_+$ ,  $\pmb {w}(t)\in \mathcal{M}_{+}^{+}\cap \mathcal{M}_{-}^{0}$  for any  $t\in [T_{\mathrm{III}}^{*},T_{\mathrm{III}}^{*} + \tau_{1})$ .
Recalling that for any  $k \in \mathcal{K}_{-}$ ,  $\langle \pmb{b}_k(T_{\mathrm{III}}^*), \pmb{x}_{-} \rangle > 0$ , from the continuity, we know that there exists  $\tau_2 > 0$  such that  $\langle \pmb{b}_k(t), \pmb{x}_{-} \rangle > 0$  holds for any  $t \in [T_{\mathrm{III}}^*, T_{\mathrm{III}}^* + \tau_2)$ .
$$
T _ {\mathrm {I I I}} ^ {*} = T _ {\mathrm {I I I}} ^ {\mathrm {F}} \wedge T _ {\mathrm {I I I}} ^ {\mathrm {N}} = \inf  \left\{t > T _ {\mathrm {I I}} ^ {\mathrm {P T}}: \langle \boldsymbol {F} _ {+} (t), \boldsymbol {x} _ {+} \rangle \leq 0 \text {o r} \exists k \in \mathcal {K} _ {-}, \text {s . t .} \langle \boldsymbol {w} _ {k} (t), \boldsymbol {x} _ {+} \rangle \neq 0 \right\}.
$$
It is obvious that  $T_{\mathrm{III}}^{\mathrm{N}} \geq T_{\mathrm{III}}^{\mathrm{F}} \wedge T_{\mathrm{III}}^{\mathrm{N}} = T_{\mathrm{III}}^{*}$ . Now we prove  $T_{\mathrm{III}}^{\mathrm{N}} = T_{\mathrm{III}}^{*}$ .
If we assume  $T_{\mathrm{III}}^{\mathrm{N}} > T_{\mathrm{III}}^{*}$  strictly, then the dynamics about  $f_{+}(t)$  and  $f_{-}(t)$  in the proof (Step I) of Lemma E.6 still hold for any  $t \in [T_{\mathrm{III}}^{*}, T_{\mathrm{III}}^{\mathrm{N}})$ . Using the same calculate about  $\langle F_{+}(t), x_{+} \rangle$  in the proof (Step II, III) of Lemma E.6, we can obtain:  $\langle F_{+}(t), x_{+} \rangle < 0$ ,  $t \in [T_{\mathrm{III}}^{*}, T_{\mathrm{III}}^{\mathrm{N}})$ .
Then we consider the vector field around the manifold  $\mathcal{M}_{+}^{0}\cap \mathcal{M}_{-}^{+}$  for  $t\in [T_{\mathrm{III}}^{*},T_{\mathrm{III}}^{\mathrm{N}})$ . In the same way as the proof of Lemma E.1 (S1), we can prove that the two-side projections onto  $\pmb{x}_{+}$  (the normal to the surface  $\mathcal{M}_+^+\cap \mathcal{M}_-^0$ ) satisfies  $f_{N}^{+}(t,\tilde{\boldsymbol{w}}),f_{N}^{-}(t,\tilde{\boldsymbol{w}}) > 0$  for any  $t\in [T_{\mathrm{III}}^{*},T_{\mathrm{III}}^{*} + \tau_{1})$  which satisfies (Case II) in Definition H.1. This implies that  $\pmb {w}_k(t)$  enters the manifold  $\mathcal{M}_{+}^{+}$ , i.e.,  $\langle \pmb {w}_k(t),\pmb {x}_+\rangle >0$  for any  $t\in [T_{\mathrm{III}}^{*},T_{\mathrm{III}}^{\mathrm{N}})$ , which is contradict to the definition of  $T_{\mathrm{III}}^{\mathrm{N}}$ . Hence, we have proved
$$
T _ {\mathrm {I I I}} ^ {\mathrm {N}} = T _ {\mathrm {I I I}} ^ {\mathrm {N}} \wedge T _ {\mathrm {I I I}} ^ {\mathrm {F}} = T _ {\mathrm {I I I}} ^ {*}.
$$
Noticing that the change of activation patterns of  $\mathbf{sgn}_k^+(t)$  ( $k \in \mathcal{K}_{-}$ ) is due to the change of the vector field  $\langle \pmb{F}_+ (t), \pmb{x}_+ \rangle$ , it is easy to verify that  $T_{\mathrm{III}}^{\mathrm{F}} = T_{\mathrm{III}}^{\mathrm{N}} \wedge T_{\mathrm{III}}^{\mathrm{F}}$ . Then we have  $T_{\mathrm{III}}^{\mathrm{N}} = T_{\mathrm{III}}^{\mathrm{F}} = T_{\mathrm{III}}^{\mathrm{N}} \wedge T_{\mathrm{III}}^{\mathrm{F}} = T_{\mathrm{III}}^*$ .
Moreover, noticing that  $T_{\mathrm{III}} \leq T_{\mathrm{III}}^{\mathrm{N}}$  and  $T_{\mathrm{III}}^{*} \leq T_{\mathrm{III}}$ , we obtain  $T_{\mathrm{III}} = T_{\mathrm{III}}^{*} = T_{\mathrm{III}}^{\mathrm{N}} = T_{\mathrm{III}}^{\mathrm{F}}$ .
Lastly, noticing that all living negative neurons ( $k \in \mathcal{K}_{-}$ ) belong to  $\mathcal{M}_{+}^{0} \cap \mathcal{M}_{-}^{+}$  at time  $T_{\mathrm{III}}$ . As discussed above, for each living negative neuron  $k \in \mathcal{K}_{-}$ , the vector field near  $\pmb{b}_k(T_{\mathrm{III}})$  is the same, with  $f_N^{-} > 0$  and  $f_N^{+} = 0$  in Definition H.1 (Case II). Hence, each living positive neuron  $\pmb{w}_k$  leaves from  $\mathcal{M}_{+}^{0}$  and enter  $\mathcal{M}_{+}^{+}$  instantly at  $T_{\mathrm{III}}$ , which means  $T_{\mathrm{III}}^{\mathrm{W}} = T_{\mathrm{III}}^{\mathrm{N}}$ .
Hence, we have proved  $T_{\mathrm{III}} = T_{\mathrm{III}}^{*} = T_{\mathrm{III}}^{\mathrm{W}} = T_{\mathrm{III}}^{\mathrm{N}} = T_{\mathrm{III}}^{\mathrm{F}}$ .
Proof of Theorem 4.7.
Combining Lemma E.6 and E.7, we obtain  $T_{\mathrm{III}} = \left(1 + \Theta (\Delta^{2})\right)T_{\mathrm{II}}$ .
F Proofs of Optimization Dynamics in Phase IV
Proof of Theorem 4.8.
From Lemma E.7, we know that all living negative neuron  $k \in \mathcal{K}_{-}$  simultaneously change their patterns on  $\pmb{x}_{+}$  at  $T_{\mathrm{III}}$ :  $\lim_{t \to T_{\mathrm{III}}^{-}} \operatorname{sgn}_k^+(t) = 0$ ,  $\lim_{t \to T_{\mathrm{III}}^{+}} \operatorname{sgn}_k^+(t) = 1$ . Moreover, from our proof of
Lemma E.7, we know that other activation patterns remain unchanged at  $T_{\mathrm{III}}$ .
$$
T _ {\mathrm {I V}} := \inf  \{t > T _ {\mathrm {I I I}}: \exists k \in \mathcal {K} _ {+} \cup \mathcal {K} _ {-}, \mathbf {s g n} _ {k} ^ {+} (t) \neq \lim  _ {s \to T _ {\mathrm {I I I}} ^ {+}} \mathbf {s g n} _ {k} ^ {+} (s) \text {o r} \mathbf {s g n} _ {k} ^ {-} (t) \neq \lim  _ {s \to T _ {\mathrm {I I I}} ^ {+}} \mathbf {s g n} _ {k} ^ {-} (s) \},
$$
and we call  $t\in (T_{\mathrm{III}},T_{\mathrm{IV}})$  Phase IV.
$$
T _ {\mathrm {I V}} ^ {*} := \inf  \left\{t > T _ {\mathrm {I I I}}: \langle \boldsymbol {F} _ {+} (t), \boldsymbol {x} _ {+} \rangle > 0, \text {o r} \langle \boldsymbol {F} _ {+} (t), \boldsymbol {x} _ {-} \rangle > 0 \right\}, \tag {30}
$$
where  $\pmb{F}_{+}(t) = \frac{p}{1 + p} e^{-f_{+}(t)}\pmb{x}_{+} - \frac{1}{1 + p} e^{f_{-}(t)}\pmb{x}_{-},$
and we call  $t \in (T_{\mathrm{III}}, T_{\mathrm{IV}}^*)$  Phase IV*.
First, we will provide meticulous prior estimations for 2d ODEs on  $f_{+}(t)$  and  $f_{-}(t)$ , similar to Phase II, which can imply  $T_{\mathrm{IV}}^{*} = +\infty$ . Additionally, we can prove  $T_{\mathrm{IV}} = T_{\mathrm{IV}}^{*}$ . Lastly, with the help of our fine-grained analysis for the 2D dynamics and the results in (Lyu and Li, 2019; Ji and Telgarsky, 2020), we can determine the unique convergent direction from numerous KKT directions.
F.1 Non-asymptotic Analysis of Optimization Dynamics in Phase IV*
Lemma F.1 (Dynamics of activate neurons in Phase IV*).
In Phase IV\*  $(t\in (T_{\mathrm{III}},T_{\mathrm{IV}}^{*}))$  , we have the following dynamics for each neuron  $k\in \mathcal{K}_{-}\cup \mathcal{K}_{+}$
$$
\frac {\mathrm {d} \pmb {b} _ {k} (t)}{\mathrm {d} t} = - \frac {\kappa_ {2}}{\sqrt {m}} \pmb {F} _ {+} (t) = - \frac {\kappa_ {2}}{\sqrt {m}} \left(\frac {p}{1 + p} e ^ {- f _ {+} (t)} \pmb {x} _ {+} - \frac {1}{1 + p} e ^ {f _ {-} (t)} \pmb {x} _ {-}\right).
$$
$$
\frac {\mathrm {d} \pmb {b} _ {k} (t)}{\mathrm {d} t} = \frac {\kappa_ {2} p e ^ {- f _ {+} (t)}}{\sqrt {m} (1 + p)} \Big (\pmb {x} _ {+} - \pmb {x} _ {-} \cos \Delta \Big).
$$
Proof of Lemma F.1.
Using the definition of  $T_{\mathrm{IV}}^*$ , this lemma can be proved in the same way as shown in the proof of Lemma D.1, E.1 and E.5.
The next lemma gives the first-order dynamics of  $f_{+}(t)$  and  $f_{-}(t)$ .
Lemma F.2 (First-order Dynamics of predictions in Phase IV*).
In Phase IV* ( $T_{\mathrm{III}} \leq t \leq T_{\mathrm{IV}}^*$ ), we have the following dynamics for  $f_+(t)$  and  $f_-(t)$ :
$$
\frac {\mathrm {d} f _ {+} (t)}{\mathrm {d} t} = \kappa_ {2} ^ {2} \frac {m _ {+}}{m} \frac {p e ^ {- f _ {+} (t)}}{1 + p} \sin^ {2} \Delta + \kappa_ {2} ^ {2} \frac {m _ {-}}{m} \left(\frac {p e ^ {- f _ {+} (t)}}{1 + p} - \frac {e ^ {f - (t)} \cos \Delta}{1 + p}\right),
$$
$$
\frac {\mathrm {d} f _ {-} (t)}{\mathrm {d} t} = \kappa_ {2} ^ {2} \frac {m _ {-}}{m} \left(\frac {p e ^ {- f _ {+} (t)}}{1 + p} \cos \Delta - \frac {e ^ {f _ {-} (t)}}{1 + p}\right).
$$
Proof of Lemma F.2.
From the definition of  $T_{\mathrm{IV}}$ , for any  $T_{\mathrm{III}} \leq t \leq T_{\mathrm{IV}}$ , we have
$$
f _ {+} (t) = \sum_ {k \in \mathcal {K} _ {+}} \frac {\kappa_ {2}}{\sqrt {m}} \boldsymbol {b} _ {k} ^ {\top} (t) \boldsymbol {x} _ {+} - \sum_ {k \in \mathcal {K} _ {-}} \frac {\kappa_ {2}}{\sqrt {m}} \boldsymbol {b} _ {k} ^ {\top} (t) \boldsymbol {x} _ {+},
$$
$$
f _ {-} (t) = - \sum_ {k \in \mathcal {K} _ {-}} \frac {\kappa_ {2}}{\sqrt {m}} \boldsymbol {b} _ {k} ^ {\top} (t) \boldsymbol {x} _ {-}.
$$
With the help of Lemma F.1, we have the dynamics of predictions:
$$
\begin{array}{l} \frac {\mathrm {d} f _ {+} (t)}{\mathrm {d} t} = \sum_ {k \in \mathcal {K} _ {+}} \frac {\kappa_ {2}}{\sqrt {m}} \left\langle \frac {\mathrm {d} \boldsymbol {b} _ {k} (t)}{\mathrm {d} t}, \boldsymbol {x} _ {+} \right\rangle - \sum_ {k \in \mathcal {K} _ {-}} \frac {\kappa_ {2}}{\sqrt {m}} \left\langle \frac {\mathrm {d} \boldsymbol {b} _ {k} (t)}{\mathrm {d} t}, \boldsymbol {x} _ {+} \right\rangle \\ = \frac {\kappa_ {2} ^ {2}}{m} \sum_ {k \in \mathcal {K} _ {+}} \frac {p}{1 + p} e ^ {- f _ {+} (t)} \left(1 - \cos^ {2} \Delta\right) - \frac {\kappa_ {2} ^ {2}}{m} \sum_ {k \in \mathcal {K} _ {-}} \left(\frac {\cos \Delta}{1 + p} e ^ {f _ {-} (t)} - \frac {p}{1 + p} e ^ {- f _ {+} (t)}\right) \\ = \frac {m _ {+}}{m} \kappa_ {2} ^ {2} \frac {p}{1 + p} e ^ {- f _ {+} (t)} \sin^ {2} \Delta + \frac {m _ {-}}{m} \kappa_ {2} ^ {2} \left(\frac {p e ^ {- f _ {+} (t)}}{1 + p} - \frac {\cos \Delta}{1 + p} e ^ {f _ {-} (t)}\right). \\ \end{array}
$$
$$
\begin{array}{l} \frac {\mathrm {d} f _ {-} (t)}{\mathrm {d} t} = - \sum_ {k \in \mathcal {K} _ {-}} \frac {\kappa_ {2}}{\sqrt {m}} \left\langle \frac {\mathrm {d} \boldsymbol {b} _ {k} (t)}{\mathrm {d} t}, \boldsymbol {x} _ {-} \right\rangle = \frac {\kappa_ {2} ^ {2}}{m} \sum_ {k \in \mathcal {K} _ {-}} \left(\frac {p}{1 + p} e ^ {- f _ {+} (t)} \cos \Delta - \frac {1}{1 + p} e ^ {f _ {-} (t)}\right) \\ = \frac {m _ {-}}{m} \kappa_ {2} ^ {2} \left(\frac {p}{1 + p} e ^ {- f _ {+} (t)} \cos \Delta - \frac {1}{1 + p} e ^ {f _ {-} (t)}\right). \\ \end{array}
$$
Following the proof in Phase II, we focus on the dynamics about predictions. Due to the specificity of the first-order dynamics, the following lemma gives an second-order autonomous dynamics of predictions.
Lemma F.3 (Second-order Autonomous Dynamics of predictions in Phase IV*). If we consider the following two variables:
$$
\left\{ \begin{array}{l} \mathcal {I} (t) := \kappa_ {2} ^ {2} \frac {m _ {-}}{m} \frac {p}{1 + p} e ^ {- f _ {+} (t)}, \\ \mathcal {J} (t) := \kappa_ {2} ^ {2} \frac {m _ {-}}{m} \frac {1}{1 + p} e ^ {f _ {-} (t)}, \end{array} \right.
$$
then the following autonomous dynamics of  $\mathcal{U}(t)$  and  $\mathcal{V}(t)$  hold in Phase IV* ( $T_{\mathrm{III}} \leq t \leq T_{\mathrm{IV}}^*$ ):
$$
\left\{ \begin{array}{l} \frac {\mathrm {d} \mathcal {I} (t)}{\mathrm {d} t} = \mathcal {I} (t) \mathcal {J} (t) \cos \Delta - \mathcal {I} ^ {2} (t) \left(1 + \frac {m _ {+}}{m _ {-}} \sin^ {2} \Delta\right), \\ \frac {\mathrm {d} \mathcal {J} (t)}{\mathrm {d} t} = \mathcal {I} (t) \mathcal {J} (t) \cos \Delta - \mathcal {J} ^ {2} (t). \end{array} \right.
$$
With the help of the first-order dynamics in Lemma F.2, the proof is straight-forward.
Lemma F.3 enlighten us that we only need to study the dynamics of  $\mathcal{I}(t)$  and  $\mathcal{J}(t)$  to study the dynamics in Phase IV*, where  $\mathcal{I}(t),\mathcal{J}(t)$  satisfies the following autonomous dynamics:
$$
\left\{ \begin{array}{l} \frac {\mathrm {d} \mathcal {I} (t)}{\mathrm {d} t} = \mathcal {I} (t) \mathcal {J} (t) \cos \Delta - \mathcal {I} ^ {2} (t) \left(1 + \frac {m _ {+}}{m _ {-}} \sin^ {2} \Delta\right), \\ \frac {\mathrm {d} \mathcal {J} (t)}{\mathrm {d} t} = \mathcal {I} (t) \mathcal {J} (t) \cos \Delta - \mathcal {J} ^ {2} (t), \end{array} \quad t \geq T _ {\mathrm {I I I}}; \right. \tag {31}
$$
$$
\left\{ \begin{array}{l} \mathcal {I} (T _ {\mathrm {I I I}}) = \kappa_ {2} ^ {2} \frac {m _ {-}}{m} \frac {p}{1 + p} e ^ {- f _ {+} (T _ {\mathrm {I I I}})}, \\ \mathcal {J} (T _ {\mathrm {I I I}}) = \kappa_ {2} ^ {2} \frac {m _ {-}}{m} \frac {1}{1 + p} e ^ {f _ {-} (T _ {\mathrm {I I I}})}. \end{array} \right.
$$
The next lemma studies the dynamics (31) for any  $t \in [T_{\mathrm{III}}, +\infty)$ .
Lemma F.4 (Fine-grained analysis of the dynamics (31)).
For the dynamics (31), we have the following results:
(S1). Initialization.
$$
\mathcal {I} (T _ {\mathrm {I I I}}) = \Theta \left(\kappa_ {2} ^ {2} p ^ {- \frac {1}{1 - \alpha \cos \Delta}}\right), \quad \mathcal {J} (T _ {\mathrm {I I I}}) = \Theta \left(\kappa_ {2} ^ {2} p ^ {- \frac {1}{1 - \alpha \cos \Delta}}\right),
$$
$$
\mathcal {I} (T _ {\mathrm {I I I}}) - \mathcal {J} (T _ {\mathrm {I I I}}) \cos \Delta = 0, \quad \mathcal {I} (T _ {\mathrm {I I I}}) \cos \Delta - \mathcal {J} (T _ {\mathrm {I I I}}) = - \Theta \left(\kappa_ {2} ^ {2} \Delta^ {2} p ^ {- \frac {1}{1 - \alpha \cos \Delta}}\right).
$$
(S2). Fine-grained two-side bound for  $\mathcal{I}(t) / \mathcal{J}(t)$
$$
\frac {1 + \cos \Delta}{1 + \cos \Delta + \frac {m _ {+}}{m _ {-}} \sin^ {2} \Delta} <   \frac {\mathcal {I} (t)}{\mathcal {J} (t)} <   \cos \Delta , \quad \forall t \in [ T _ {\mathrm {I I I}}, + \infty).
$$
(S3). The limit of  $\mathcal{I}(t) / \mathcal{J}(t)$
$$
\lim _ {t \to \infty} \frac {\mathcal {I} (t)}{\mathcal {J} (t)} = \frac {1 + \cos \Delta}{1 + \cos \Delta + \frac {m _ {+}}{m _ {-}} \sin^ {2} \Delta}.
$$
(S4). Tight estimate of  $\mathcal{I}(t)$  and  $\mathcal{J}(t)$ .
$$
\mathcal {I} (t) = \Theta \left(\frac {1}{\frac {\frac {1}{p ^ {1 - \alpha} \cos \Delta}}{\kappa_ {2} ^ {2}} + \Delta^ {2} (t - T _ {\mathrm {I I I}})}\right), \quad \forall t \in [ T _ {\mathrm {I I I}}, + \infty);
$$
$$
\mathcal {J} (t) = \Theta \left(\frac {1}{\frac {1}{p ^ {\frac {1 - \alpha \cos \Delta}{\kappa_ {2} ^ {2}}} + \Delta^ {2}} (t - T _ {\mathrm {I I I}})}\right), \quad \forall t \in [ T _ {\mathrm {I I I}}, + \infty).
$$
Proof of Lemma F.4.
$$
\epsilon := \frac {m _ {+}}{m _ {-}} \sin^ {2} \Delta .
$$
Step I. Preparation. Recalling Lemma E.6, we have:
Step II. A rough estimate on  $\mathcal{I}(t)$  and  $\mathcal{J}(t)$ . In this step, we aim to prove:
$$
\mathcal {J} (t) > \mathcal {I} (t) > 0, \quad \mathcal {I} (t) + \mathcal {J} (t) \leq \mathcal {I} \left(T _ {\mathrm {I I I}}\right) + \mathcal {J} \left(T _ {\mathrm {I I I}}\right), \quad \forall t \in \left[ T _ {\mathrm {I I I}}, \infty\right).
$$
First, from the definition of  $\mathcal{I}(t)$  and  $\mathcal{J}(t)$ , we have  $\mathcal{I}(t) > 0$  and  $\mathcal{J}(t) > 0$ .
Then we consider the dynamics of  $\mathcal{I}(t) + \mathcal{J}(t)$ . From
$$
\begin{array}{l} \frac {\mathrm {d}}{\mathrm {d} t} \Big (\mathcal {I} (t) + \mathcal {J} (t) \Big) = 2 \mathcal {I} (t) \mathcal {J} (t) \cos \Delta - \mathcal {I} ^ {2} (t) (1 + \epsilon) - \mathcal {J} ^ {2} (t) \\ = - \left(\mathcal {I} (t) - \mathcal {J} (t)\right) ^ {2} \cos \Delta - (1 - \cos \Delta) \mathcal {J} ^ {2} (t) - \mathcal {I} ^ {2} (t) (1 + \epsilon - \cos \Delta) <   0, \\ \end{array}
$$
$$
\mathcal {I} (t) + \mathcal {J} (t) \leq \mathcal {I} \left(T _ {\mathrm {I I I}}\right) + \mathcal {J} \left(T _ {\mathrm {I I I}}\right), \quad \forall t \geq T _ {\mathrm {I I I}}.
$$
Then we consider the dynamics of  $\mathcal{J}(t) - \mathcal{I}(t)$ . We define the hitting time
$$
\tau_ {\mathcal {I} - \mathcal {I}} := \inf  \left\{t \geq T _ {\mathrm {I I I}}: \mathcal {J} (t) \leq \mathcal {I} (t) \right\}.
$$
From Step I, we know  $\mathcal{J}(T_{\mathrm{III}}) - \mathcal{I}(T_{\mathrm{III}}) = (1 - \cos \Delta)\mathcal{J}(T_{\mathrm{III}}) > 0$ . From the continuity,  $\tau_{\mathcal{J} - \mathcal{I}}$  exists and  $\tau_{\mathcal{J} - \mathcal{I}} > T_{\mathrm{III}}$ .
For any  $t \in [T_{\mathrm{III}}, \tau_{\mathcal{J} - \mathcal{I}})$ , we have  $\mathcal{J}(t) - \mathcal{I}(t) > 0$  and
$$
\begin{array}{l} \frac {\mathrm {d}}{\mathrm {d} t} \Big (\mathcal {J} (t) - \mathcal {I} (t) \Big) = - \mathcal {J} ^ {2} (t) + \mathcal {I} ^ {2} (t) (1 + \epsilon) = - (\mathcal {J} (t) + \mathcal {I} (t)) (\mathcal {J} (t) - \mathcal {I} (t)) + \epsilon \mathcal {I} ^ {2} (t) \\ > - \left(\mathcal {J} (t) + \mathcal {I} (t)\right) \left(\mathcal {J} (t) - \mathcal {I} (t)\right) \geq - \left(\mathcal {J} \left(T _ {\mathrm {I I I}}\right) + \mathcal {I} \left(T _ {\mathrm {I I I}}\right)\right) \left(\mathcal {J} (t) - \mathcal {I} (t)\right), \\ \end{array}
$$
We consider the auxiliary ODE:  $\frac{d}{dt}\mathcal{P}(t) = -(\mathcal{J}(T_{\mathrm{III}}) + \mathcal{I}(T_{\mathrm{III}}))\mathcal{P}(t)$ , where  $\mathcal{P}(T_{\mathrm{III}}) = \mathcal{J}(T_{\mathrm{III}}) - \mathcal{I}(T_{\mathrm{III}}) > 0$ . From the Comparison Principle of ODEs, we have:
$$
\mathcal {J} (t) - \mathcal {I} (t) \geq \mathcal {P} (t) = (\mathcal {J} (T _ {\mathrm {I I I}}) - \mathcal {I} (T _ {\mathrm {I I I}})) \exp \Big (- (\mathcal {J} (T _ {\mathrm {I I I}}) + \mathcal {I} (T _ {\mathrm {I I I}})) (t - T _ {\mathrm {I I I}}) \Big) > 0, \forall t \in [ T _ {\mathrm {I}}, \tau_ {\mathcal {U} - \mathcal {V}}).
$$
From the definition of  $\tau_{\mathcal{I} - \mathcal{I}}$  , we have proved
$$
\tau_ {\mathcal {I} - \mathcal {I}} = + \infty ;
$$
$$
\mathcal {J} (t) > \mathcal {I} (t), \forall t \in [ T _ {\mathrm {H I}}, + \infty).
$$