SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Since  $\pmb{\theta}^{*}$  is a possible point,  $f(\pmb{x}_{-};\pmb{\theta}^{*})\leq -1$  , which gives us
$$
\sum_ {k \in \mathcal {K} _ {-}} \sigma \left(\left\langle \boldsymbol {b} _ {k} ^ {*}, \boldsymbol {x} _ {-} \right\rangle\right) \geq \frac {\sqrt {m}}{\kappa_ {2}} > 0.
$$
Hence, there exists  $k_{1}\in \mathcal{K}_{-}$  , s.t.  $\langle \pmb {b}_{k_1}^*,\pmb {x}_-\rangle >0$  strictly.
Then we study the neuron  $k_{2} \in \mathcal{K}_{-}$  ( $k_{2} \neq k_{1}$ ). Lemma F.1 and Lemma F.5 (S3) give us that
$$
\begin{array}{l} \langle \boldsymbol {b} _ {k _ {1}} (t), \boldsymbol {x} _ {-} \rangle = \langle \boldsymbol {b} _ {k _ {1}} (T _ {\mathrm {I V}}), \boldsymbol {x} _ {+} \rangle + \int_ {T _ {\mathrm {I V}}} ^ {t} \frac {\kappa_ {2}}{\sqrt {m}} \left(\frac {1}{1 + p} e ^ {f _ {-} (t)} - \frac {p}{1 + p} e ^ {- f _ {+} (t)} \cos \Delta\right) \mathrm {d} t \\ = \left\langle \boldsymbol {b} _ {k _ {2}} \left(T _ {\mathrm {I V}}\right), \boldsymbol {x} _ {-} \right\rangle + \int_ {T _ {\mathrm {I V}}} ^ {t} \frac {\kappa_ {2}}{\sqrt {m}} \left(\frac {1}{1 + p} e ^ {f _ {-} (t)} - \frac {p}{1 + p} e ^ {- f _ {+} (t)} \cos \Delta\right) d t + \left(\left\langle \boldsymbol {b} _ {k _ {1}} \left(T _ {\mathrm {I V}}\right), \boldsymbol {x} _ {-} \right\rangle - \left\langle \boldsymbol {b} _ {k _ {2}} \left(T _ {\mathrm {I V}}\right), \boldsymbol {x} _ {-} \right\rangle\right) \\ = \left\langle \boldsymbol {b} _ {k _ {2}} (t), \boldsymbol {x} _ {-} \right\rangle + \Bigg (\left\langle \boldsymbol {b} _ {k _ {1}} \left(T _ {\mathrm {I V}}\right), \boldsymbol {x} _ {-} \right\rangle - \left\langle \boldsymbol {b} _ {k _ {2}} \left(T _ {\mathrm {I V}}\right), \boldsymbol {x} _ {-} \right\rangle \Bigg). \\ \end{array}
$$
$$
\left\langle \boldsymbol {b} _ {k _ {1}} ^ {*}, \boldsymbol {x} _ {-} \right\rangle = \left\langle \boldsymbol {b} _ {k _ {2}} ^ {*}, \boldsymbol {x} _ {-} \right\rangle > 0.
$$
$$
\boldsymbol {b} _ {k} ^ {*} \neq 0, \quad \langle \boldsymbol {b} _ {k} ^ {*}, \boldsymbol {x} _ {-} \rangle > 0, \quad \langle \boldsymbol {b} _ {k} ^ {*}, \boldsymbol {x} _ {+} \rangle \geq 0, \quad \forall k \in \mathcal {K} _ {-}.
$$
The next difficulty in this step is to determine whether  $\langle \pmb{b}_k^*,\pmb{x}_{+}\rangle$  can be 0. To prove this, we will use our fine-grained analysis of training dynamics (Lemma F.5) and Lemma F.6 (III).
Let  $k \in \mathcal{K}_{-}$ . Recalling the dynamics of  $\pmb{b}_k(t)$  in Lemma F.1, we know
$$
\begin{array}{l} - \frac {\partial \mathcal {L} (\boldsymbol {\theta} (t))}{\partial \boldsymbol {b} _ {k}} = - \frac {\kappa_ {2}}{\sqrt {m}} \left(\frac {p}{1 + p} e ^ {- f _ {+} (t)} \boldsymbol {x} _ {+} - \frac {1}{1 + p} e ^ {f _ {-} (t)} \boldsymbol {x} _ {-}\right) \\ = \frac {\kappa_ {2}}{\sqrt {m}} \frac {e ^ {f _ {-} (t)}}{1 + p} \left(\boldsymbol {x} _ {-} - p e ^ {- (f _ {+} (t) + f _ {-} (t))} \boldsymbol {x} _ {+}\right). \\ \end{array}
$$
Recalling Lemma Lemma F.5 (S2),  $\lim_{t\to \infty}pe^{-(f_{+}(t) + f_{-}(t))} = \frac{1 + \cos\Delta}{1 + \cos\Delta + \frac{m_{+}}{m_{-}}\sin^{2}\Delta}$ . Then using Lemma F.6 (III), there exists  $c_{1} > 0$ , s.t.
$$
\boldsymbol {b} _ {k} ^ {*} = c _ {1} \left(\boldsymbol {x} _ {-} - \frac {1 + \cos \Delta}{1 + \cos \Delta + \frac {m _ {+}}{m _ {-}} \sin^ {2} \Delta} \boldsymbol {x} _ {+}\right).
$$
Hence, we have proved
$$
\langle \pmb {b} _ {k} ^ {*}, \pmb {x} _ {+} \rangle > 0, \quad \forall k \in \mathcal {K} _ {-}.
$$
Then writing the KKT condition about the gradient of  $\pmb{b}_k^*$  of Problem (34):
$$
\mathbf {0} = \pmb {b} _ {k} ^ {*} + \lambda_ {+} ^ {*} \frac {\kappa_ {2}}{\sqrt {m}} \pmb {x} _ {+} - \lambda_ {-} ^ {*} \frac {\kappa_ {2}}{\sqrt {m}} \pmb {x} _ {-}, \quad \forall k \in \mathcal {K} _ {-}.
$$
Combining the two equations about  $\pmb{b}_k^*$ , we obtain
$$
\begin{array}{l} \boldsymbol {b} _ {k} ^ {*} = \lambda_ {-} ^ {*} \frac {\kappa_ {2}}{\sqrt {m}} \left(\boldsymbol {x} _ {-} - \frac {1 + \cos \Delta}{1 + \cos \Delta + \frac {m _ {+}}{m _ {-}} \sin^ {2} \Delta} \boldsymbol {x} _ {+}\right), \quad \forall k \in \mathcal {K} _ {-}; \\ \frac {\lambda_ {+} ^ {*}}{\lambda_ {-} ^ {*}} = \frac {1 + \cos \Delta}{1 + \cos \Delta + \frac {m _ {+}}{m _ {-}} \sin^ {2} \Delta}. \\ \end{array}
$$
In summary, we have proved the final convergence direction  $\overline{\pmb{\theta}} = (\bar{\pmb{b}}_1^\top ,\dots ,\bar{\pmb{b}}_m^\top)^\top \in \mathbb{S}^{md - 1}$  satisfies
$$
\begin{array}{l} \bar {b} _ {k} = \mathbf {0}, \quad \forall k \notin \mathcal {K} _ {+} \cup \mathcal {K} _ {-}; \\ \bar {\boldsymbol {b}} _ {k} = C \left(\boldsymbol {x} _ {+} - \boldsymbol {x} _ {-} \cos \Delta\right), \quad \forall k \in \mathcal {K} _ {+}; \\ \end{array}
$$
where  $C > 0$  is a scaling constant such that  $\left\| \overline{\boldsymbol{\theta}}\right\| _2 = 1$
Moreover, a straight-forward calculation gives us that
$$
f \left(\boldsymbol {x} _ {+}; \overline {{\boldsymbol {\theta}}}\right) = - f \left(\boldsymbol {x} _ {-}; \overline {{\boldsymbol {\theta}}}\right) > 0.
$$
Proof of Theorem 4.10.
Lemma F.7 implies Theorem 4.10 directly.
Proof of Theorem 4.11.
From Lemma F.7, the final convergence direction  $\overline{\pmb{\theta}} = (\bar{b}_1^\top ,\dots ,\bar{b}_m^\top)^\top \in \mathbb{S}^{md - 1}$  satisfies
$$
\bar {b} _ {k} = 0, \quad \forall k \notin \mathcal {K} _ {+} \cup \mathcal {K} _ {-};
$$
$$
\bar {\boldsymbol {b}} _ {k} = C \left(\boldsymbol {x} _ {+} - \boldsymbol {x} _ {-} \cos \Delta\right), \quad \forall k \in \mathcal {K} _ {+};
$$
$$
\bar {\boldsymbol {b}} _ {k} = C \left(\left(1 + \frac {m _ {+} \sin^ {2} \Delta}{m _ {-} (1 + \cos \Delta)}\right) \boldsymbol {x} _ {-} - \boldsymbol {x} _ {+}\right), \quad \forall k \in \mathcal {K} _ {-};
$$
where  $C > 0$  is a scaling constant such that  $\left\| \overline{\pmb{\theta}}\right\| _2 = 1$  and  $f_{+}(\overline{\pmb{\theta}}) = -f_{-}(\overline{\pmb{\theta}}) > 0$
It is easy to verify that There exists a scaling factor  $C_1 > 0$  such that  $f_{+}(\hat{\pmb{\theta}}) = -f_{-}(\hat{\pmb{\theta}}) = 1$ , where  $\hat{\pmb{\theta}} = C_1\overline{\pmb{\theta}}$ . For simplicity, we denote  $Q \coloneqq CC_1$ , then  $\hat{\pmb{\theta}} = (\hat{\pmb{b}}_1^\top, \dots, \hat{\pmb{b}}_m^\top)^\top \in \mathbb{R}^{md}$  satisfies
$$
\hat {\boldsymbol {b}} _ {k} = \mathbf {0}, \quad \forall k \notin \mathcal {K} _ {+} \cup \mathcal {K} _ {-};
$$
$$
\hat {\boldsymbol {b}} _ {k} = Q \left(\boldsymbol {x} _ {+} - \boldsymbol {x} _ {-} \cos \Delta\right), \quad \forall k \in \mathcal {K} _ {+};
$$
$$
\hat {\boldsymbol {b}} _ {k} = Q \left(\left(1 + \frac {m _ {+} \sin^ {2} \Delta}{m _ {-} (1 + \cos \Delta)}\right) \boldsymbol {x} _ {-} - \boldsymbol {x} _ {+}\right), \quad \forall k \in \mathcal {K} _ {-};
$$
$$
f _ {+} (\hat {\boldsymbol {\theta}}) = - f _ {-} (\hat {\boldsymbol {\theta}}) = 1.
$$
Therefore,  $\hat{\theta}$  is a feasible point of Problem (34). Moreover, from
$$
\begin{array}{l} - 1 = - \sum_ {k \in \mathcal {K} _ {-}} \frac {\kappa_ {2}}{\sqrt {m}} \left\langle \hat {\boldsymbol {b}} _ {k}, \boldsymbol {x} _ {-} \right\rangle \\ = - \frac {\kappa_ {2}}{\sqrt {m}} m _ {-} \left\langle Q \left(\left(1 + \frac {m _ {+} \sin^ {2} \Delta}{m _ {-} (1 + \cos \Delta)}\right) \boldsymbol {x} _ {-} - \boldsymbol {x} _ {+}\right), \boldsymbol {x} _ {-} \right\rangle \\ = - Q \frac {\kappa_ {2}}{\sqrt {m}} m _ {-} \left(1 - \cos \Delta + \frac {m _ {+} \sin^ {2} \Delta}{m _ {-} (1 + \cos \Delta)}\right) \\ = - Q \frac {\kappa_ {2}}{\sqrt {m}} \left(m _ {-} (1 - \cos \Delta) + m _ {+} \frac {\sin^ {2} \Delta}{1 + \cos \Delta}\right), \\ \end{array}
$$
$$
\frac {\kappa_ {2}}{\sqrt {m}} Q = \frac {1}{m _ {-} (1 - \cos \Delta) + m _ {+} \frac {\sin^ {2} \Delta}{1 + \cos \Delta}}.
$$
For any  $\epsilon \geq 0$ , now we consider another solution  $\hat{\pmb{\theta}} (\epsilon)$  near  $\hat{\pmb{\theta}}$ :  $\hat{\pmb{\theta}} (\epsilon) = (\hat{\pmb{b}}_1^\top (\epsilon),\dots ,\hat{\pmb{b}}_m^\top (\epsilon))^{\top}$ , where
$$
\hat {\boldsymbol {b}} _ {k} (\epsilon) = \mathbf {0}, \quad \forall k \notin \mathcal {K} _ {+} \cup \mathcal {K} _ {-};
$$
$$
\hat {\boldsymbol {b}} _ {k} (\epsilon) = Q \left(\boldsymbol {x} _ {+} - \boldsymbol {x} _ {-} \cos \Delta\right) - Q \epsilon \left(\boldsymbol {x} _ {+} - \boldsymbol {x} _ {-} \cos \Delta\right), \quad \forall k \in \mathcal {K} _ {+};
$$
$$
\hat {\boldsymbol {b}} _ {k} (\epsilon) = Q \left(\left(1 + \frac {m _ {+} \sin^ {2} \Delta}{m _ {-} (1 + \cos \Delta)}\right) \boldsymbol {x} _ {-} - \boldsymbol {x} _ {+}\right) + Q \epsilon \left(\boldsymbol {x} _ {+} - \boldsymbol {x} _ {-} \cos \Delta\right), \quad \forall k \in \mathcal {K} _ {-}.
$$
and it holds that  $\hat{\pmb{\theta}}(0) = \hat{\pmb{\theta}}$ . Moreover, it is easy to verify that  $\hat{\pmb{\theta}}(\epsilon)$  is also a feasible point of Problem (34).
$$
f _ {-} (\hat {\boldsymbol {\theta}} (\epsilon)) = f _ {-} (\hat {\boldsymbol {\theta}} (0)) = - 1;
$$
$$
f _ {+} (\hat {\boldsymbol {\theta}} (\epsilon)) = f _ {+} (\hat {\boldsymbol {\theta}} (0)) = 1.
$$
Then we compare the norm of  $\hat{\pmb{\theta}} (\epsilon)$  and  $\hat{\pmb{\theta}} (0)$
$$
\begin{array}{l} \left\| \hat {\boldsymbol {\theta}} (\epsilon) \right\| ^ {2} = m _ {+} \left(Q \left(\boldsymbol {x} _ {+} - \boldsymbol {x} _ {-} \cos \Delta\right) - Q \epsilon \left(\boldsymbol {x} _ {+} - \boldsymbol {x} _ {-} \cos \Delta\right)\right) ^ {2} \\ + m _ {-} \left(Q \left(\left(1 + \frac {m _ {+} \sin^ {2} \Delta}{m _ {-} (1 + \cos \Delta)}\right) \boldsymbol {x} _ {-} - \boldsymbol {x} _ {+}\right) + Q \epsilon \left(\boldsymbol {x} _ {+} - \boldsymbol {x} _ {-} \cos \Delta\right)\right) ^ {2}, \\ \end{array}
$$
At  $\epsilon = 0$ , we can calculate that
$$
\begin{array}{l} \left. \frac {\mathrm {d} \left\| \hat {\boldsymbol {\theta}} (\epsilon) \right\| ^ {2}}{2 m _ {+} Q ^ {2} \mathrm {d} \epsilon} \right| _ {\epsilon = 0} \\ = \frac {1}{m _ {+}} \left(- m _ {+} \left\| \boldsymbol {x} _ {+} - \boldsymbol {x} _ {-} \cos \Delta \right\| ^ {2} + m _ {-} \left\langle \left(1 + \frac {m _ {+} \sin^ {2} \Delta}{m _ {-} (1 + \cos \Delta)}\right) \boldsymbol {x} _ {-} - \boldsymbol {x} _ {+}, \boldsymbol {x} _ {+} - \boldsymbol {x} _ {-} \cos \Delta \right\rangle\right) \\ = \left\langle \left(\frac {m _ {-}}{m _ {+}} + \frac {\sin^ {2} \Delta}{1 + \cos \Delta} + \cos \Delta\right) \boldsymbol {x} _ {-} - (1 + \frac {m _ {-}}{m _ {+}}) \boldsymbol {x} _ {+}, \boldsymbol {x} _ {+} - \boldsymbol {x} _ {-} \cos \Delta \right\rangle \\ \end{array}
$$
Denote  $\alpha = \frac{m_{-}}{m_{+}}$ $\begin{array}{r}\mathbf{\sigma} = \left(\alpha +\frac{\sin^2\Delta}{1 + \cos\Delta} +\cos \Delta\right)\pmb{x}_{-} - (1 + \alpha)\pmb{x}_{+},\pmb{x}_{+} - \pmb{x}_{-}\cos \Delta \end{array}$
$$
\begin{array}{l} = - \left(\alpha + \frac {\sin^ {2} \Delta}{1 + \cos \Delta} + \cos \Delta\right) \cos \Delta - (1 + \alpha) \\ + \left(\alpha + \frac {\sin^ {2} \Delta}{1 + \cos \Delta} + \cos \Delta\right) \cos \Delta + (1 + \alpha) \cos^ {2} \Delta \\ = - (1 + \alpha) \sin^ {2} \Delta <   0 \\ \end{array}
$$
Then combining the continuity of  $\frac{\mathrm{d}\|\hat{\pmb{\theta}}(\epsilon)\|^2}{\mathrm{d}\epsilon}$ , there exists  $\delta > 0$  such that the following inequality holds:
$$
\left\| \hat {\boldsymbol {\theta}} (\epsilon) \right\| ^ {2} <   \left\| \hat {\boldsymbol {\theta}} (0) \right\| ^ {2}, \forall \epsilon \in (0, \delta).
$$
Hence, we have proved that  $\overline{\theta}$  is not a local optimal direction of the max-margin problem (34).
G Clarke Subdifferential and KKT Conditions for Non-smooth Optimization
Definition G.1 (Clarke's Subdifferential (Clarke et al., 2008)). For a locally Lipschitz function  $\mathcal{L}:\Omega \to \mathbb{R}$ , the Clarke's subdifferential at  $\theta \in \Omega$  is the convex set
$$
\partial^ {\circ} \mathcal {L} (\boldsymbol {\theta}) := \operatorname {c o n v} \left\{\lim  _ {i \to \infty} \nabla \mathcal {L} (\boldsymbol {\theta} _ {i}): \lim  _ {i \to \infty} \boldsymbol {\theta} _ {i} = \boldsymbol {\theta}, \mathcal {L} \text {i s d i f f e r e n t i a l a t} \boldsymbol {\theta} _ {i} \right\}.
$$
Remark G.2. Notice that if  $\mathcal{L}$  is continuously differentiable at  $\pmb{\theta}$ , then  $\partial^{\circ}\mathcal{L}(\pmb{\theta}) = \{\nabla \mathcal{L}(\pmb{\theta})\}$  is unique. However, for discontinuous differentiable points of  $\mathcal{L}$ , the differential inclusion flow  $\frac{\mathrm{d}\pmb{\theta}}{\mathrm{d}t} \in \partial^{\circ}\mathcal{L}(\pmb{\theta})$  defined by Definition G.1 may not be unique. To study a more specific dynamics, we also utilize Definition H.1 to determine GF at some of such points.
Now we review the definition of Karush-Kuhn-Tucker (KKT) conditions for non-smooth optimization problems (Dutta et al., 2013). Consider the following constrained optimization problem (P):
$$
\min  _ {\boldsymbol {x} \in \mathbb {R} ^ {d}}: f (\boldsymbol {x})
$$
$$
\text {s . t .} g _ {i} (\boldsymbol {x}) \leq 0, \quad \forall i \in [ N ]
$$
where  $f, g_1, \dots, g_N: \mathbb{R}^d \to \mathbb{R}$  are locally Lipschitz functions. We say that  $\pmb{x} \in \mathbb{R}$  is a feasible point of (P) if  $\pmb{x}$  satisfies  $g_i(\pmb{x}) \leq 0$  for all  $i \in [N]$ .
Definition G.3 (KKT Point for Non-smooth Optimization). We say that a feasible point of (P) is a KKT point if there exists  $\lambda_1, \dots, \lambda_N \geq 0$  such that
$$
\begin{array}{l} 1. \mathbf {0} \in \partial^ {\circ} f (\boldsymbol {x}) + \sum_ {i \in [ N ]} \lambda_ {i} \partial^ {\circ} g _ {i} (\boldsymbol {x}); \\ 2. \forall i \in [ N ], \lambda_ {i} g _ {i} (\boldsymbol {x}) = 0. \\ \end{array}
$$
H Solution of Discontinuous System
In this section, we add some supplements about the definitions of solutions of discontinuous systems, which can overcome non-uniqueness of GF trajectories (2) to some extent. Many definitions of solutions of differential equations with discontinuous systems have been proposed. In this paper, we adopt a widely used definition of the solutions in Chapter 2.4 in (Filippov, 2013).
Definition H.1 (Solutions of Discontinuous Systems, Chapter 2.4 in (Filippov, 2013)). Consider a  $n$ -dimensional equation or a system  $(\boldsymbol{x} \in \mathbb{R}^n)$ :  $\frac{\mathrm{d}\boldsymbol{x}}{\mathrm{d}t} = \boldsymbol{f}(\boldsymbol{x})$  with a piecewise continuous function  $\boldsymbol{f}$  in a domain  $G$ . We aim to define the dynamics near some discontinuous regions.
Let the function  $\pmb{f}$  be discontinuous on a smooth surface  $S$  given by the equation  $\phi(\pmb{x}) = 0$ . Let  $\pmb{x}^* \in S$  and the surface  $S$  separate the neighborhood of  $\pmb{x}^*$  into domains  $G^-$  and  $G^+$ . Let the function  $\pmb{f}(\pmb{x})$  have the limit values:
$$
\boldsymbol {f} ^ {-} (\boldsymbol {x} ^ {*}) := \lim  _ {\boldsymbol {x} \in G ^ {-}, \boldsymbol {x} \to \boldsymbol {x} ^ {*}} \boldsymbol {f} (\boldsymbol {x}), \quad \boldsymbol {f} ^ {+} (\boldsymbol {x} ^ {*}) := \lim  _ {\boldsymbol {x} \in G ^ {+}, \boldsymbol {x} ^ {*} \to \boldsymbol {x} ^ {*}} \boldsymbol {f} (\boldsymbol {x}).
$$
Here one should distinguish between two main cases. Let  $f_{N}^{-}(\boldsymbol{x}^{*})$  and  $f_{N}^{+}(\boldsymbol{x}^{*})$  be projections of the vectors  $\boldsymbol{f}^{-}(\boldsymbol{x}^{*})$  and  $\boldsymbol{f}^{+}(\boldsymbol{x}^{*})$  onto the normal to the surface  $S$  at the point  $\boldsymbol{x}^{*}$ , where the normal is directed towards the domain  $G^{+}$ .
(Case I). If the vectors  $\pmb{f}(\pmb{x}^{*})$  are directed to the surface  $S$  on both sides, i.e.  $f_{N}^{-}(\pmb{x}^{*}) > 0$ ,  $f_{N}^{+}(\pmb{x}^{*}) < 0$ , then the solution the solution starting from  $\pmb{x}^{*}$  can not leave  $S$  for some time. Moreover, its dynamics on  $S$  can be defined in the following way:
$$
\frac {\mathrm {d} \boldsymbol {x}}{\mathrm {d} t} = \boldsymbol {f} ^ {0} (\boldsymbol {x}),
$$
$$
\text {w h e r e} \boldsymbol {f} ^ {0} (\boldsymbol {x}) = \alpha \boldsymbol {f} ^ {+} (\boldsymbol {x}) + (1 - \alpha) \boldsymbol {f} ^ {-} (\boldsymbol {x}), \quad \alpha = \frac {f _ {N} ^ {-} (\boldsymbol {x})}{f _ {N} ^ {-} (\boldsymbol {x}) - f _ {N} ^ {+} (\boldsymbol {x})}.
$$
(Case II). If  $f_{N}^{-}(\boldsymbol{x}^{*}) \geq 0$ ,  $f_{N}^{+}(\boldsymbol{x}^{*}) \geq 0$ , but  $f_{N}^{-}(\boldsymbol{x}^{*})$  and  $f_{N}^{+}(\boldsymbol{x}^{*})$  are not both 0, then the solution starting from  $\boldsymbol{x}^{*}$  passes from one side of the surface  $S$  to the other instantly.
(Case III). If  $f_{N}^{-}(\pmb{x}^{*}) < 0$ ,  $f_{N}^{+}(\pmb{x}^{*}) > 0$ , then the dynamics is defined in the similar way as (Case I).
(Case IV). If  $f_{N}^{-}(\boldsymbol{x}^{*}) \leq 0$ ,  $f_{N}^{+}(\boldsymbol{x}^{*}) \leq 0$ , but  $f_{N}^{-}(\boldsymbol{x}^{*})$  and  $f_{N}^{+}(\boldsymbol{x}^{*})$  are not both 0, then the dynamics is defined in the similar way as (Case II).
Remark H.2. Notice that Definition H.1 overcomes non-uniqueness of GF trajectories to some extent. It is worth noting that Definition H.1 and Definition G.1 are compatible and specifically, the dynamics defined in Definition H.1 (Case I, III) lie in the convex hull defined in Definition G.1.
Remark H.3. In (Lyu et al., 2021), the non-branching starting point Assumption is employed to address the technical challenge of non-uniqueness in GF trajectories. By comparison, in this work, we do not need this assumption. We adopt Definition H.1 to uniquely determine the Gradient Flow trajectories theoretically near some discontinuous differential regions, such as "Ridge", "Valley", and "Refraction edge" discussed in Section I.2 in (Lyu et al., 2021).
I Some Basic Inequalities
Lemma I.1 (Hoeffding's Inequality). Let  $X_{1}, \dots, X_{n}$  are independent random variables, and  $X_{i} \in [a_{i}, b_{i}]$  for any  $i \in [n]$ . Define  $\bar{X} = \frac{1}{n} \sum_{i=1}^{n} X_{i}$ . Then for any  $\epsilon > 0$ , we have the following probability inequalities:
$$
\mathbb {P} \Big (\bar {X} - \mathbb {E} [ \bar {X} ] \geq \epsilon \Big) \leq \exp \Big (- \frac {2 n ^ {2} \epsilon^ {2}}{\sum_ {i = 1} ^ {n} (b _ {i} - a _ {i}) ^ {2}} \Big),
$$
$$
\mathbb {P} \Big (\bar {X} - \mathbb {E} [ \bar {X} ] \leq - \epsilon \Big) \leq \exp \Big (- \frac {2 n ^ {2} \epsilon^ {2}}{\sum_ {i = 1} ^ {n} (b _ {i} - a _ {i}) ^ {2}} \Big).
$$
Lemma I.2. Consider  $\pmb{x}_1, \pmb{x}_2, \pmb{y} \in \mathbb{S}^{d-1}$ , where  $\langle \pmb{x}_1, \pmb{x}_2 \rangle = \cos \Delta (\Delta \in (0, \pi/2))$ . If  $\langle \pmb{y}, \pmb{x}_1 \rangle \geq 0$  and  $\langle \pmb{y}, \pmb{x}_2 \rangle \leq 0$ , then we have  $0 \leq \langle \pmb{y}, \pmb{x}_1 \rangle \leq \sin \Delta$  and  $-\sin \Delta \leq \langle \pmb{y}, \pmb{x}_2 \rangle \leq 0$ .
Proof of Lemma 1.2.
Denote  $\mathcal{M}_{\pmb{x}}\coloneqq \operatorname {span}\{\pmb{x}_1,\pmb{x}_2\}$ . We can do the orthogonal decomposition of  $\pmb{y}$ :
$$
\boldsymbol {y} = \boldsymbol {y} _ {\mathcal {M}} + \boldsymbol {y} _ {\mathcal {M}} ^ {\perp},
$$
where  $\pmb{y}_{\mathcal{M}} \in \mathcal{M}_{\pmb{x}}$  and  $\pmb{y}_{\mathcal{M}}^{\perp} \perp \mathcal{M}_{\pmb{x}}$ . From  $\pmb{y} \in \operatorname{span}\{\pmb{x}_1, \pmb{x}_2\}$ , there exist  $\alpha, \beta \in \mathbb{R}$ , s.t.  $\pmb{y}_{\mathcal{M}} = \alpha \pmb{x}_1 + \beta \pmb{x}_2$ .
Due to the orthogonal decomposition, we know  $\| \pmb{y}_{\mathcal{M}}\| \leq 1$ , which means  $\alpha^2 +\beta^2 +2\alpha \beta \cos \Delta \leq 1$ . Noticing  $\alpha^2 +\beta^2 +2\alpha \beta \cos \Delta = (\alpha +\beta \cos \Delta)^2 +\alpha^2\sin^2\Delta$ , we know  $\alpha^2\sin^2\Delta \leq 1$
Due to  $\langle \pmb{y},\pmb{x}_1\rangle \geq 0$  and  $\langle \pmb {y},\pmb {x}_2\rangle \leq 0$  , we have  $\langle \pmb {y}_{\mathcal{M}},\pmb {x}_1\rangle \geq 0$  and  $\langle \pmb {y}_{\mathcal{M}},\pmb {x}_2\rangle \leq 0$  , which means
$$
\alpha + \beta \cos \Delta \geq 0, \quad \alpha \cos \Delta + \beta \leq 0.
$$
So  $\alpha \geq 0$  and  $\alpha \sin \Delta \geq 0$ . Recalling  $\alpha^2 \cos^2 \Delta \leq 1$ , we know  $0 \leq \alpha \sin \Delta \leq 1$ . Hence, we have  $\alpha + \beta \cos \Delta \leq \alpha - \alpha \cos^2 \Delta = \alpha \sin^2 \Delta \leq \sin \Delta$ , i.e.  $\langle \pmb{y}, \pmb{x}_1 \rangle \leq \sin \Delta$ .
In the same way, we have  $-\sin \Delta \leq \langle \pmb{y}, \pmb{x}_2 \rangle \leq 0$ .
Lemma I.3. If  $p \geq 5$ , we have  $\|z\| \geq \frac{p - 1}{p + 1} \geq \frac{2}{3}$ .
Proof of Lemma 1.3.
$$
\begin{array}{l} \| \boldsymbol {z} \| ^ {2} = \left\| \frac {1}{n} \sum_ {i = 1} ^ {n} y _ {i} \boldsymbol {x} _ {i} \right\| ^ {2} = \left\| \frac {p}{1 + p} \boldsymbol {x} _ {+} - \frac {1}{1 + p} \boldsymbol {x} _ {-} \right\| ^ {2} = \left(\frac {p}{1 + p}\right) ^ {2} + \left(\frac {1}{1 + p}\right) ^ {2} - \frac {2 p}{(1 + p) ^ {2}} \langle \boldsymbol {x} _ {+}, \boldsymbol {x} _ {-} \rangle \\ = \left(\frac {p}{1 + p} + \frac {1}{1 + p}\right) ^ {2} - \frac {2 p}{(1 + p) ^ {2}} (\cos \Delta + 1) \geq 1 - \frac {2 p}{(p + 1) ^ {2}} \cdot 2 = \left(\frac {p - 1}{p + 1}\right) ^ {2}. \\ \end{array}
$$
Lemma I.4. Let  $\pmb{w} \in \mathbb{S}^{d-1}$ . If  $\langle \pmb{w}, \pmb{\mu} \rangle \geq 1 - \epsilon$  ( $\epsilon \in (0,1)$ ),  $p \geq 5$  and  $\cos \Delta \geq 4/5$ , then we have
$$
- 2 \sqrt {\epsilon} \sin \Delta - \epsilon \leq \langle \boldsymbol {w}, \boldsymbol {x} _ {-} \rangle - \frac {p \cos \Delta - 1}{\sqrt {p ^ {2} + 1 - 2 p \cos \Delta}} \leq 2 \sqrt {\epsilon} \sin \Delta .
$$
Proof of Lemma 1.4.
$$
\langle \boldsymbol {w}, \boldsymbol {x} _ {-} \rangle = \langle \boldsymbol {w}, \boldsymbol {\mu} \rangle + \langle \boldsymbol {w}, \boldsymbol {x} _ {-} - \boldsymbol {\mu} \rangle = \langle \boldsymbol {w}, \boldsymbol {\mu} \rangle + \langle \boldsymbol {\mu}, \boldsymbol {x} _ {-} - \boldsymbol {\mu} \rangle + \langle \boldsymbol {w} - \boldsymbol {\mu}, \boldsymbol {x} _ {-} - \boldsymbol {\mu} \rangle .
$$
$$
\langle \pmb {\mu}, \pmb {x} _ {-} - \pmb {\mu} \rangle = \langle \pmb {\mu}, \pmb {x} _ {-} \rangle - 1 = \left\langle \frac {p \pmb {x} _ {+} - \pmb {x} _ {-}}{\| p \pmb {x} _ {+} - \pmb {x} _ {-} \|}, \pmb {x} _ {-} \right\rangle - 1 = \frac {p \cos \Delta - 1}{\sqrt {p ^ {2} + 1 - 2 p \cos \Delta}} - 1;
$$
$$
\left\| \boldsymbol {x} _ {-} - \boldsymbol {\mu} \right\| = \sqrt {2 - 2 \langle \boldsymbol {\mu} , \boldsymbol {x} _ {-} \rangle} = \sqrt {2 - 2 \frac {p \cos \Delta - 1}{\sqrt {p ^ {2} + 1 - 2 p \cos \Delta}}};
$$
$$
\| \boldsymbol {w} - \boldsymbol {\mu} \| = \sqrt {2 - 2 \langle \boldsymbol {w} , \boldsymbol {\mu} \rangle}.
$$
$$
\begin{array}{l} | \langle \boldsymbol {w} - \boldsymbol {\mu}, \boldsymbol {x} _ {-} - \boldsymbol {\mu} \rangle | \leq \| \boldsymbol {w} - \boldsymbol {\mu} \| \| \boldsymbol {x} _ {-} - \boldsymbol {\mu} \| \\ = \sqrt {2 - 2 \frac {p \cos \Delta - 1}{\sqrt {p ^ {2} + 1 - 2 p \cos \Delta}}} \sqrt {2 - 2 \langle \boldsymbol {w} , \boldsymbol {\mu} \rangle} \leq \sqrt {2 - 2 \frac {p \cos \Delta - 1}{\sqrt {p ^ {2} + 1 - 2 p \cos \Delta}}} \sqrt {2 \epsilon} \\ \leq \frac {\sqrt {2}}{\sqrt [ 4 ]{p ^ {2} + 1 - 2 p \cos \Delta}} \sqrt {\frac {p ^ {2} \sin^ {2} \Delta}{\sqrt {p ^ {2} + 1 - 2 p \cos \Delta} + p \cos \Delta - 1}} \sqrt {2 \epsilon} \\ \leq \frac {2 \sqrt {\epsilon}}{\sqrt {p - 1}} \frac {p \sin \Delta}{\sqrt {p - 1} + p \cos \Delta - 1} \leq 2 \sqrt {\epsilon} \sin \Delta . \\ \end{array}
$$
Then we have the bound:
$$
\begin{array}{l} \langle \boldsymbol {w}, \boldsymbol {x} _ {-} \rangle \leq \frac {p \cos \Delta - 1}{\sqrt {p ^ {2} + 1 - 2 p \cos \Delta}} - 1 + \langle \boldsymbol {w}, \boldsymbol {\mu} \rangle + | \langle \boldsymbol {w} - \boldsymbol {\mu}, \boldsymbol {x} _ {-} - \boldsymbol {\mu} \rangle | \\ \leq \frac {p \cos \Delta - 1}{\sqrt {p ^ {2} + 1 - 2 p \cos \Delta}} + 2 \sqrt {\epsilon} \sin \Delta , \\ \langle \boldsymbol {w}, \boldsymbol {x} _ {-} \rangle \geq \frac {p \cos \Delta - 1}{\sqrt {p ^ {2} + 1 - 2 p \cos \Delta}} - 1 + \langle \boldsymbol {w}, \boldsymbol {\mu} \rangle - | \langle \boldsymbol {w} - \boldsymbol {\mu}, \boldsymbol {x} _ {-} - \boldsymbol {\mu} \rangle | \\ \geq \frac {p \cos \Delta - 1}{\sqrt {p ^ {2} + 1 - 2 p \cos \Delta}} - \epsilon - 2 \sqrt {\epsilon} \sin \Delta . \\ \end{array}
$$