SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\hat {\mathbf {w}} = \sum_ {i \in \mathcal {S}} \alpha_ {i} \cdot y _ {i} \mathbf {x} _ {i}, \quad \alpha_ {i} > 0.
$$
Note that as long as  $\sum_{i\in S}\alpha_i = 1 / \gamma^2$  , we have
$$
\mathcal {P} (\hat {\mathbf {w}}) = \sum_ {i \in \mathcal {S}} \alpha_ {i} \cdot y _ {i} \mathcal {P} (\mathbf {x} _ {i}) = \gamma \cdot \frac {1}{\gamma^ {2}} = \frac {1}{\gamma}.
$$
Now consider  $\bar{\mathcal{P}}$ . Note that  $\bar{\mathcal{P}}(\hat{\mathbf{w}}) = 0$  by the choice of  $\bar{\mathcal{P}}$ , then apply  $\bar{\mathcal{P}}$  on both sides of the above equation, we get
$$
0 = \bar {\mathcal {P}} (\hat {\mathbf {w}}) = \sum_ {i \in \mathcal {S}} \alpha_ {i} \cdot y _ {i} \bar {\mathcal {P}} (\mathbf {x} _ {i}). \tag {7}
$$
Under (7), (6) is equivalent to the following condition on  $\bar{\mathcal{P}} (\tilde{\mathbf{w}})$ :
$$
0 = \sum_ {i \in \mathcal {S}} \exp \left(- y _ {i} \cdot \left\langle \bar {\mathcal {P}} (\mathbf {x} _ {i}), \bar {\mathcal {P}} (\tilde {\mathbf {w}}) \right\rangle\right) \cdot y _ {i} \bar {\mathcal {P}} (\mathbf {x} _ {i}),
$$
which is precisely the first-order condition of
$$
\bar {\mathcal {P}} (\tilde {\mathbf {w}}) \in \arg \min  G (\cdot), \text {w h e r e} G (\mathbf {v}) := \sum_ {i \in \mathcal {S}} \exp \big (- y _ {i} \cdot \left\langle \bar {\mathcal {P}} (\mathbf {x} _ {i}), \mathbf {v} \right\rangle \big).
$$
Hence we have shown that: the condition that  $\tilde{\mathbf{w}}$  satisfies (1) is equivalent to the condition that  $\bar{\mathcal{P}} (\tilde{\mathbf{w}})$  minimizes the strongly convex potential  $G(\cdot)$ .
B The Behaviors of Constant-Stepsize GD
B.1 Notation Simplifications
Without loss of generality, we assume that
$$
y _ {i} = 1, \quad i = 1, \ldots , n.
$$
Otherwise, we replace  $y_{i}$  with 1 and  $\mathbf{x}_i$  with  $y_{i}\cdot \mathbf{x}_{i}$ , respectively, and the following analysis does not change.
Then the risk becomes
$$
L (\mathbf {w}) := \sum_ {i = 1} ^ {n} \log \left(1 + e ^ {- \mathbf {w} ^ {\top} \mathbf {x} _ {i}}\right).
$$
Rotating the hard-margin SVM solution. Note that the (GD) iterates (under linear models) are rotation equivariant. Specifically, let  $\mathbf{R}$  be an orthogonal matrix, then applying  $\mathbf{R}$  on both sides of (GD), we obtain
$$
\begin{array}{l} \mathbf {R} \mathbf {w} _ {t + 1} = \mathbf {R} \mathbf {w} _ {t} + \eta \sum_ {i = 1} ^ {n} \left(1 - s \left(\mathbf {x} _ {i} ^ {\top} \mathbf {w} _ {t}\right)\right) \cdot \mathbf {R} \mathbf {x} _ {i} \\ = \mathbf {R} \mathbf {w} _ {t} + \eta \sum_ {i = 1} ^ {n} \left(1 - s \left(\left(\mathbf {R} \mathbf {x} _ {i}\right) ^ {\top} \left(\mathbf {R} \mathbf {w} _ {t}\right)\right)\right) \cdot \mathbf {R} \mathbf {x} _ {i}, \\ \end{array}
$$
which is equivalent to the GD iterates under changes of variables,  $\mathbf{w}\gets \mathbf{R}\mathbf{w}$  and  $\mathbf{x}\leftarrow \mathbf{R}\mathbf{x}$ .
Therefore, without loss of generality, we can apply a rotation to the dataset such that  $\hat{\mathbf{w}}\parallel \mathbf{e}_1$ . Then for  $\mathbf{v}\in \mathbb{R}^d$
$$
\mathcal {P} \mathbf {v} = \mathbf {v} [ 1 ] \in \mathbb {R}, \quad \bar {\mathcal {P}} \mathbf {v} = \mathbf {v} [ 2: d ] \in \mathbb {R} ^ {d - 1}.
$$
Slightly abusing notations, in what follows we will write
$$
\mathbf {x} _ {i} = \left(x _ {i}, \bar {\mathbf {x}} _ {i}\right) ^ {\top} \in \mathbb {R} \oplus \mathbb {R} ^ {d - 1}, \quad i = 1, \dots , n,
$$
$$
x _ {i} := \mathbf {x} _ {i} [ 1 ] \in \mathbb {R}, \quad \bar {\mathbf {x}} _ {i} := \mathbf {x} _ {i} [ 2: d ] \in \mathbb {R} ^ {d - 1}.
$$
Similarly, we define
$$
\mathbf {w} = (w, \bar {\mathbf {w}}) ^ {\top} \in \mathbb {R} \oplus \mathbb {R} ^ {d - 1}.
$$
$$
\mathbf {x} _ {i} ^ {\top} \mathbf {w} = x _ {i} w _ {i} + \bar {\mathbf {x}} _ {i} ^ {\top} \bar {\mathbf {w}}.
$$
So the loss can be written as:
$$
L (w, \bar {\mathbf {w}}) := \sum_ {i = 1} ^ {n} \log \left(1 + e ^ {- w x _ {i} - \bar {\mathbf {w}} ^ {\top} \bar {\mathbf {x}} _ {i}}\right).
$$
So (GD) can be written as:
$$
w _ {0} = 0, \quad w _ {t} = w _ {t - 1} - \eta \cdot \nabla_ {w} L (w _ {t - 1}, \bar {\mathbf {w}} _ {t - 1}), \quad t \geq 1;
$$
$$
\bar {\mathbf {w}} _ {0} = 0, \quad \bar {\mathbf {w}} _ {t} = \bar {\mathbf {w}} _ {t - 1} - \eta \cdot \nabla_ {\bar {\mathbf {w}}} L \left(w _ {t - 1}, \bar {\mathbf {w}} _ {t - 1}\right), \quad t \geq 1. \tag {8}
$$
The above two recursions capture the GD iterates projecting to the max-margin and non-separable subspaces, respectively.
B.2 Boundedness of GD in the Non-Separable Subspace
We first show that  $(\bar{\mathbf{w}}_t)_{t\geq 0}$  stay bounded for every fixed stepsize  $\eta$
Lemma B.1 (Positiveness of  $w_{t}$ ). Suppose that Assumption1 holds. Consider  $(w_{t})_{t\geq 0}$  defined by (8) with constant stepsize  $\eta >0$ . Then for every  $t\geq 0$ , it holds that  $w_{t}\geq 0$ .
$$
w _ {0} = 0, \quad w _ {t} = w _ {t - 1} - \eta \cdot \nabla_ {w} L (w _ {t - 1}, \bar {\mathbf {w}} _ {t - 1}), \quad t \geq 1.
$$
We only need to show that  $\nabla_w L(w, \bar{\mathbf{w}}) \leq 0$ . This is because
$$
\begin{array}{l} \nabla_ {w} L (w, \bar {\mathbf {w}}) = - \sum_ {i = 1} ^ {n} \frac {1}{1 + e ^ {w x _ {i} + \bar {\mathbf {w}} ^ {\top} \bar {\mathbf {x}} _ {i}}} \cdot x _ {i} \\ <   0. \quad \text {s i n c e} x _ {i} \geq \gamma > 0 \text {b y} \\ \end{array}
$$
Lemma B.2 (A recursion of  $\| \bar{\mathbf{w}}_t\| _2$ ). Suppose that Assumptions 1, 2, and 3 hold. Consider  $(\bar{\mathbf{w}}_t)_{t\geq 0}$  defined by (8) with constant stepsize  $\eta >0$ . Then for every  $t\geq 0$ , there exists  $j\in [n]$  such that
$$
\| \bar {\mathbf {w}} _ {t + 1} \| _ {2} ^ {2} \leq \| \bar {\mathbf {w}} _ {t} \| _ {2} ^ {2} + 2 \eta e ^ {- w _ {t} \gamma} \cdot \left(n - \frac {b \cdot \| \bar {\mathbf {w}} _ {t} \| _ {2}}{4}\right) + \frac {\eta}{1 + e ^ {w _ {t} x _ {j} + \bar {\mathbf {w}} _ {t} ^ {\top} \bar {\mathbf {x}} _ {j}}} \cdot \left(\eta n ^ {2} - b \cdot \| \bar {\mathbf {w}} _ {t} \| _ {2}\right).
$$
As a direct consequence,
$$
\left\| \bar {\mathbf {w}} _ {t} \right\| _ {2} \geq \max  \left\{4 n / b, \eta n ^ {2} / b \right\} \quad i m p l i e s t h a t \quad \left\| \bar {\mathbf {w}} _ {t + 1} \right\| _ {2} \leq \left\| \bar {\mathbf {w}} _ {t} \right\| _ {2}.
$$
Proof. We first make a few useful notations. Fix a time index  $t$ .
- Let  $k$  be the index of the "most negatively classified" support sample, i.e.,
$$
k := \arg \min _ {i \in S} \{\langle \bar {\mathbf {w}} _ {t}, \bar {\mathbf {x}} _ {i} \rangle \},
$$
then by Definition 2 it holds that
$$
\left\langle \bar {\mathbf {w}} _ {t}, \bar {\mathbf {x}} _ {k} \right\rangle \leq - b \cdot \left\| \bar {\mathbf {w}} _ {t} \right\| _ {2}. \tag {9}
$$
- Let  $j$  be the index of the "most negatively classified" sample, i.e.,
$$
j := \arg \min  _ {1 \leq i \leq n} \left\{w _ {t} x _ {i} + \langle \bar {\mathbf {w}} _ {t}, \bar {\mathbf {x}} _ {i} \rangle \right\}.
$$
$$
w _ {t} x _ {j} + \left\langle \bar {\mathbf {w}} _ {t}, \bar {\mathbf {x}} _ {j} \right\rangle \leq w _ {t} x _ {i} + \left\langle \bar {\mathbf {w}} _ {t}, \bar {\mathbf {x}} _ {i} \right\rangle \text {f o r e v e r y} i \in [ n ]. \tag {10}
$$
In particular, we must have
$$
\langle \bar {\mathbf {w}} _ {t}, \bar {\mathbf {x}} _ {j} \rangle \leq - b \| \bar {\mathbf {w}} _ {t} \| _ {2}, \tag {11}
$$
$$
\begin{array}{l} w _ {t} \gamma + \left\langle \bar {\mathbf {w}} _ {t}, \bar {\mathbf {x}} _ {j} \right\rangle \leq w _ {t} x _ {j} + \left\langle \bar {\mathbf {w}} _ {t}, \bar {\mathbf {x}} _ {j} \right\rangle \quad \text {b y} \\ \leq \min  _ {i \in \mathcal {S}} \left\{w _ {t} x _ {i} + \langle \bar {\mathbf {w}} _ {t}, \bar {\mathbf {x}} _ {i} \rangle \right\} \quad \text {b y} (1 0) \\ = w _ {t} \gamma + \min  _ {i \in \mathcal {S}} \left\{\langle \bar {\mathbf {w}} _ {t}, \bar {\mathbf {x}} _ {i} \rangle \right\} \quad \text {b y} \\ \leq w _ {t} \gamma - b \| \bar {\mathbf {w}} _ {t} \| _ {2}. \quad \text {b y} \\ \end{array}
$$
We remark that it is possible that  $k = j$ .
Step 0: an iterate norm recursion. Recall that
$$
\bar {\mathbf {w}} _ {t + 1} = \bar {\mathbf {w}} _ {t} - \eta \nabla_ {\bar {\mathbf {w}}} L (w _ {t}, \bar {\mathbf {w}} _ {t}), \quad \nabla_ {\bar {\mathbf {w}}} L (w _ {t}, \bar {\mathbf {w}} _ {t}) = - \sum_ {i = 1} ^ {n} \frac {1}{1 + e ^ {w _ {t} x _ {i} + \bar {\mathbf {w}} _ {t} ^ {\top} \bar {\mathbf {x}} _ {i}}} \cdot \bar {\mathbf {x}} _ {i}.
$$
$$
\left\| \bar {\mathbf {w}} _ {t + 1} \right\| _ {2} ^ {2} = \left\| \bar {\mathbf {w}} _ {t} \right\| _ {2} ^ {2} - 2 \eta \cdot \left\langle \bar {\mathbf {w}} _ {t}, \nabla_ {\bar {\mathbf {w}}} L \left(w _ {t}, \bar {\mathbf {w}} _ {t}\right) \right\rangle + \eta^ {2} \cdot \left\| \nabla_ {\bar {\mathbf {w}}} L \left(w _ {t}, \bar {\mathbf {w}} _ {t}\right) \right\| _ {2} ^ {2}.
$$
Step 1: gradient norm bounds. By definition, we have
$$
\begin{array}{l} \left\| \nabla_ {\bar {\mathbf {w}}} L (w _ {t}, \bar {\mathbf {w}} _ {t}) \right\| _ {2} = \left\| \sum_ {i = 1} ^ {n} \frac {1}{1 + e ^ {w _ {t} x _ {i}} + \bar {\mathbf {w}} _ {t} ^ {\top} \bar {\mathbf {x}} _ {i}} \cdot \bar {\mathbf {x}} _ {i} \right\| _ {2} \\ \leq \sum_ {i = 1} ^ {n} \frac {1}{1 + e ^ {w _ {t} x _ {i} + \bar {\mathbf {w}} _ {t} ^ {\top} \bar {\mathbf {x}} _ {i}}} \cdot \| \bar {\mathbf {x}} _ {i} \| _ {2} \\ \leq \sum_ {i = 1} ^ {n} \frac {1}{1 + e ^ {w _ {t} x _ {i}} + \bar {\mathbf {w}} _ {t} ^ {\top} \bar {\mathbf {x}} _ {i}} \quad \text {b y} \\ \leq \frac {n}{1 + e ^ {w _ {t} x _ {j}} + \bar {\mathbf {w}} _ {t} ^ {\top} \bar {\mathbf {x}} _ {j}} \quad \text {b y} (1 0) \tag {12} \\ \end{array}
$$
$$
\leq n. \tag {13}
$$
Here, we use a property of logistic loss that the gradient is uniformly bounded. Therefore, we have
$$
\begin{array}{l} \left\| \nabla_ {\bar {\mathbf {w}} _ {t}} L \left(w _ {t}, \bar {\mathbf {w}} _ {t}\right) \right\| _ {2} ^ {2} \leq \left(\frac {n}{1 + e ^ {w _ {t} x _ {j}} + \bar {\mathbf {w}} _ {t} ^ {\top} \bar {\mathbf {x}} _ {j}}\right) \cdot \left\| \nabla_ {\bar {\mathbf {w}} _ {t}} L \left(w _ {t}, \bar {\mathbf {w}} _ {t}\right) \right\| _ {2} \quad \text {b y} (12) \\ \leq \frac {n ^ {2}}{1 + e ^ {w _ {t} x _ {j} + \bar {\mathbf {w}} _ {t} ^ {\top} \bar {\mathbf {x}} _ {j}}}. (14) \\ \end{array}
$$
Step 2: cross-term bounds. We aim to show that the negative parts in the cross-term can cancel both the positive parts in the cross-term and the squared gradient norm term.
Note that the following holds for either  $j = k$  or  $j \neq k$ :
$$
\begin{array}{l} - \left\langle \bar {\mathbf {w}} _ {t}, \nabla_ {\bar {\mathbf {w}} _ {t}} L \left(w _ {t}, \bar {\mathbf {w}} _ {t}\right) \right\rangle \\ = \sum_ {i = 1} ^ {n} \frac {1}{1 + e ^ {w _ {t} x _ {i} + \bar {\mathbf {w}} _ {t} ^ {\top} \bar {\mathbf {x}} _ {i}}} \cdot \bar {\mathbf {w}} _ {t} ^ {\top} \bar {\mathbf {x}} _ {i} \\ \leq \sum_ {\bar {\mathbf {w}} _ {t} ^ {\top} \bar {\mathbf {x}} _ {i} > 0} \frac {1}{1 + e ^ {w _ {t} x _ {i} + \bar {\mathbf {w}} _ {t} ^ {\top} \bar {\mathbf {x}} _ {i}}} \cdot \bar {\mathbf {w}} _ {t} ^ {\top} \bar {\mathbf {x}} _ {i} \tag {15} \\ + \frac {1}{2} \cdot \frac {1}{1 + e ^ {w _ {t} x _ {j} + \bar {\mathbf {w}} _ {t} ^ {\top} \bar {\mathbf {x}} _ {j}}} \cdot \bar {\mathbf {w}} _ {t} ^ {\top} \bar {\mathbf {x}} _ {j} + \frac {1}{2} \cdot \frac {1}{1 + e ^ {w _ {t} x _ {k} + \bar {\mathbf {w}} _ {t} ^ {\top} \bar {\mathbf {x}} _ {k}}} \cdot \bar {\mathbf {w}} _ {t} ^ {\top} \bar {\mathbf {x}} _ {k}. \\ \end{array}
$$
The first term in (15) can be bounded by
$$
\begin{array}{l} \sum_ {\bar {\mathbf {w}} _ {t} ^ {\top} \bar {\mathbf {x}} _ {i} > 0} \frac {1}{1 + e ^ {w _ {t} x _ {i} + \bar {\mathbf {w}} _ {t} ^ {\top} \bar {\mathbf {x}} _ {i}}} \cdot \bar {\mathbf {w}} _ {t} ^ {\top} \bar {\mathbf {x}} _ {i} \\ = \sum_ {\bar {\mathbf {w}} _ {t} ^ {\top} \bar {\mathbf {x}} _ {i} > 0} \frac {e ^ {- w _ {t} x _ {i}}}{1 + e ^ {- w _ {t} x _ {i} - \bar {\mathbf {w}} _ {t} ^ {\top} \bar {\mathbf {x}} _ {i}}} \cdot e ^ {- \bar {\mathbf {w}} _ {t} ^ {\top} \bar {\mathbf {x}} _ {i}} \cdot \bar {\mathbf {w}} _ {t} ^ {\top} \bar {\mathbf {x}} _ {i} \\ \leq \sum_ {\bar {\mathbf {w}} _ {t} ^ {\top} \bar {\mathbf {x}} _ {i} > 0} \frac {e ^ {- w _ {t} x _ {i}}}{1 + e ^ {- w _ {t} x _ {i} - \bar {\mathbf {w}} _ {t} ^ {\top} \bar {\mathbf {x}} _ {i}}} \quad \text {s i n c e} e ^ {- t} \cdot t \leq 1 \\ \leq \sum_ {\bar {\mathbf {w}} _ {t} ^ {\top} \bar {\mathbf {x}} _ {i} > 0} e ^ {- w _ {t} x _ {i}} \\ \leq n e ^ {- \gamma w _ {t}}. \quad \text {s i n c e} x _ {i} \geq \gamma \text {f o r} i \in [ n ] \tag {16} \\ \end{array}
$$
The second term in (15) can be bounded by
$$
\frac {1}{2} \cdot \frac {1}{1 + e ^ {w _ {t} x _ {j}} + \bar {\mathbf {w}} _ {t} ^ {\top} \bar {\mathbf {x}} _ {j}} \cdot \bar {\mathbf {w}} _ {t} ^ {\top} \bar {\mathbf {x}} _ {j} \leq \frac {1}{2} \cdot \frac {- b \cdot \| \bar {\mathbf {w}} _ {t} \| _ {2}}{1 + e ^ {w _ {t} x _ {j}} + \bar {\mathbf {w}} _ {t} ^ {\top} \bar {\mathbf {x}} _ {j}}. \tag {17}
$$
The third term in (15) can be bounded by
$$
\begin{array}{l} \frac {1}{2} \cdot \frac {1}{1 + e ^ {w _ {t} x _ {k} + \bar {\mathbf {w}} _ {t} ^ {\top} \bar {\mathbf {x}} _ {k}}} \cdot \bar {\mathbf {w}} _ {t} ^ {\top} \bar {\mathbf {x}} _ {k} \leq \frac {1}{2} \cdot \frac {- b \cdot \| \bar {\mathbf {w}} _ {t} \| _ {2}}{1 + e ^ {w _ {t} \gamma + \bar {\mathbf {w}} _ {t} ^ {\top} \bar {\mathbf {x}} _ {k}}} \quad \text {b y (9) a n d t h e c h o i c e o f k} \\ = \frac {- b \cdot \| \bar {\mathbf {w}} _ {t} \| _ {2}}{2} \cdot \frac {e ^ {- w _ {t} \gamma}}{e ^ {- w _ {t} \gamma} + e ^ {\bar {\mathbf {w}} _ {t} ^ {\top} \bar {\mathbf {x}} _ {k}}} \\ \leq \frac {- b \cdot \| \bar {\mathbf {w}} _ {t} \| _ {2}}{2} \cdot \frac {e ^ {- w _ {t} \gamma}}{2}, \quad \text {s i n c e} e ^ {- w _ {t} \gamma}, e ^ {\bar {\mathbf {w}} _ {t} ^ {\top} \bar {\mathbf {x}} _ {k}} \leq 1 \tag {18} \\ \end{array}
$$
$$
w _ {t} \gamma \geq 0, \quad \text {b y}
$$
$$
\bar {\mathbf {w}} _ {t} ^ {\top} \bar {\mathbf {x}} _ {k} \leq 0. \quad \text {b y}
$$
Now, bringing (16), (17), and (18) into (15), we obtain
$$
- \left\langle \bar {\mathbf {w}} _ {t}, \nabla_ {\bar {\mathbf {w}} _ {t}} L \left(w _ {t}, \bar {\mathbf {w}} _ {t}\right) \right\rangle \leq e ^ {- w _ {t} \gamma} \cdot \left(n - \frac {b \cdot \| \bar {\mathbf {w}} _ {t} \| _ {2}}{4}\right) - \frac {b \cdot \| \bar {\mathbf {w}} _ {t} \| _ {2}}{2} \cdot \frac {1}{1 + e ^ {w _ {t} x _ {j} + \bar {\mathbf {w}} _ {t} ^ {\top} \bar {\mathbf {x}} _ {j}}}. \tag {19}
$$
Here, we use a property of logistic loss that the gradients from incorrectly classified samples dominate the gradients from correctly classified samples.
Step 3: iterate norm recursion bounds. Using (14) and (19), we can obtain
$$
\begin{array}{l} \left\| \bar {\mathbf {w}} _ {t + 1} \right\| _ {2} ^ {2} = \left\| \bar {\mathbf {w}} _ {t} \right\| _ {2} ^ {2} - 2 \eta \cdot \left\langle \bar {\mathbf {w}} _ {t}, \nabla_ {\bar {\mathbf {w}}} L \left(w _ {t}, \bar {\mathbf {w}} _ {t}\right) \right\rangle + \eta^ {2} \cdot \left\| \nabla_ {\bar {\mathbf {w}}} L \left(w _ {t}, \bar {\mathbf {w}} _ {t}\right) \right\| _ {2} ^ {2} \\ \leq \| \bar {\mathbf {w}} _ {t} \| _ {2} ^ {2} + 2 \eta e ^ {- w _ {t} \gamma} \cdot \left(n - \frac {b \cdot \| \bar {\mathbf {w}} _ {t} \| _ {2}}{4}\right) - \eta b \cdot \| \bar {\mathbf {w}} _ {t} \| _ {2} \cdot \frac {1}{1 + e ^ {w _ {t} x _ {j}} + \bar {\mathbf {w}} _ {t} ^ {\top} \bar {\mathbf {x}} _ {j}} \\ + \eta^ {2} \cdot \frac {n ^ {2}}{1 + e ^ {w _ {t} x _ {j} + \bar {\mathbf {w}} _ {t} ^ {\top} \bar {\mathbf {x}} _ {j}}} \\ = \| \bar {\mathbf {w}} _ {t} \| _ {2} ^ {2} + 2 \eta e ^ {- w _ {t} \gamma} \cdot \left(n - \frac {b \cdot \| \bar {\mathbf {w}} _ {t} \| _ {2}}{4}\right) + \frac {\eta}{1 + e ^ {w _ {t} x _ {j} + \bar {\mathbf {w}} _ {t} \top} \bar {\mathbf {x}} _ {j}} \cdot \left(\eta n ^ {2} - b \cdot \| \bar {\mathbf {w}} _ {t} \| _ {2}\right). \\ \end{array}
$$
We have completed the proof.
Lemma B.3 (Boundedness of  $\bar{\mathbf{w}}$ ). Suppose that Assumptions 1, 2, and 3 hold. Consider  $(\bar{\mathbf{w}}_t)_{t\geq 0}$  defined by (8) with constant stepsize  $\eta >0$ . Then for every  $t\geq 0$ , it holds that
$$
\left\| \bar {\mathbf {w}} _ {t} \right\| _ {2} \leq W _ {\max } := \max  \left\{4 n / b, \eta n ^ {2} / b \right\} + \eta n.
$$
Proof. We prove the claim by induction. Clearly,  $\| \bar{\mathbf{w}}_0\| _2 = 0\leq \max \{4n / b,\eta n^2 /b\} +\eta n$ . Now suppose that
$$
\left\| \bar {\mathbf {w}} _ {t} \right\| _ {2} \leq \max  \left\{4 n / b, \eta n ^ {2} / b \right\} + \eta n,
$$
and discuss the following two cases:
1. If  $\| \bar{\mathbf{w}}_t\| _2\leq \max \{4n / b,\eta n^2 /b\}$ , then
$$
\begin{array}{l} \| \bar {\mathbf {w}} _ {t + 1} \| _ {2} \leq \| \bar {\mathbf {w}} _ {t} \| _ {2} + \| \eta \cdot \nabla_ {\bar {\mathbf {w}}} L (w _ {t}, \bar {\mathbf {w}} _ {t}) \| _ {2} \quad \text {b y} \\ \leq \left\| \bar {\mathbf {w}} _ {t} \right\| _ {2} + \eta n \tag {13} \\ \leq \max  \{4 n / b, \eta n ^ {2} / b \} + \eta n. \\ \end{array}
$$
2. Else, we have
$$
\max  \left\{4 n / b, \eta n ^ {2} / b \right\} \leq \| \bar {\mathbf {w}} _ {t} \| _ {2} \leq \max  \left\{4 n / b, \eta n ^ {2} / b \right\} + \eta n,
$$
$$
\begin{array}{l} \| \bar {\mathbf {w}} _ {t + 1} \| _ {2} \leq \| \bar {\mathbf {w}} _ {t} \| _ {2} \quad \text {b y} \\ \leq \max  \{4 n / b, \eta n ^ {2} / b \} + \eta n. \\ \end{array}
$$
B.3 Divergence of GD in the Max-Margin Subspace
Definition 3 (Some loss measurements in the non-separable subspace). Under Assumptions 1, 2, and 3, we define the following notations:
(A) Define two loss functions
$$
G (\bar {\mathbf {w}}) := \sum_ {i \in \mathcal {S}} e ^ {- \bar {\mathbf {w}} ^ {\top} \bar {\mathbf {x}} _ {i}}, \qquad H (\bar {\mathbf {w}}) := \sum_ {i \notin \mathcal {S}} e ^ {- \bar {\mathbf {w}} ^ {\top} \bar {\mathbf {x}} _ {i}}.
$$
In the case where  $S = [n]$ , we define  $H(\bar{\mathbf{w}}) = 0$ .
(B) Define
$$
G _ {\min } := \min  _ {\bar {\mathbf {w}} \in \mathbb {R} ^ {d - 1}} G (\bar {\mathbf {w}}),
$$
It is clear that  $G_{\mathrm{min}} \geq 1$  since  $(\bar{\mathbf{x}}_i)_{i \in S}$  are non-separable by Definition 2.
(C) Define
$$
\bar {\mathbf {w}} _ {*} := \arg \min  _ {\bar {\mathbf {w}} \in \mathbb {R} ^ {d - 1}} G (\bar {\mathbf {w}}).
$$
It is clear that  $G(\bar{\mathbf{w}}_{*}) = G_{\min}$ . Moreover, it holds that  $\| \bar{\mathbf{w}}_{*}\|_{2}\leq W_{\max}$  by Lemma B.4.
(D) Recall that  $\| \bar{\mathbf{w}}_t\| _2\leq W_{\max}$  according to Lemma B.3. We then define
$$
G _ {\max } := \sup  _ {\| \bar {\mathbf {w}} \| _ {2} \leq W _ {\max }} G (\bar {\mathbf {w}}), \quad H _ {\max } := \sup  _ {\| \bar {\mathbf {w}} \| _ {2} \leq W _ {\max }} H (\bar {\mathbf {w}}).
$$
It is clear that
$$
G (\bar {\mathbf {w}} _ {t}) \leq G _ {\max }, \quad H (\bar {\mathbf {w}} _ {t}) \leq H _ {\max },
$$
and that  $G_{\mathrm{max}}$ ,  $H_{\mathrm{max}}$  are polynomials on  $e^{\eta}$ ,  $e^n$ , and  $e^{1 / b}$ , and are independent of  $t$ .
Lemma B.4. For the  $\bar{\mathbf{w}}_*$  in Definition 3, it holds that
$$
\| \bar {\mathbf {w}} _ {*} \| _ {2} \leq \frac {\log (n)}{b} \leq W _ {\max } \cdot
$$
Proof. By Definition 2, there exists  $j \in S$  such that
$$
\bar {\mathbf {w}} _ {*} ^ {\top} \bar {\mathbf {x}} _ {j} \leq - b \cdot \| \bar {\mathbf {w}} _ {*} \| _ {2},
$$
$$
G (\bar {\mathbf {w}} _ {*}) = \sum_ {i \in \mathcal {S}} e ^ {- \bar {\mathbf {w}} _ {*} ^ {\top} \bar {\mathbf {x}} _ {i}} \geq e ^ {- \bar {\mathbf {w}} _ {*} ^ {\top} \bar {\mathbf {x}} _ {j}} \geq e ^ {b \cdot \| \bar {\mathbf {w}} _ {*} \| _ {2}}.
$$
On the other hand, by the definition of  $\bar{\mathbf{w}}_*$ , we have
$$
G (\bar {\mathbf {w}} _ {*}) \leq G (0) = n.
$$
Therefore, we have  $e^{b\cdot \|\bar{\mathbf{w}}_*\|_2} \leq n$ , that is,  $\| \bar{\mathbf{w}}_*\|_2 \leq \log(n) / b \leq W_{\max}$ .
We now consider  $(w_{t})_{t\geq 0}$
Lemma B.5. Suppose Assumptions 1, 2, and 3 hold. Then for every  $t \geq 0$ , it holds that
$$
\begin{array}{l} w _ {t + 1} \geq w _ {t} + \frac {\eta \gamma}{2} \cdot \min  \left\{1, e ^ {- \gamma w _ {t}} \cdot G (\bar {\mathbf {w}} _ {t}) \right\}, \\ w _ {t + 1} \leq w _ {t} + \eta \cdot \min  \left\{\gamma n, \gamma \cdot e ^ {- \gamma w _ {t}} \cdot G (\bar {\mathbf {w}} _ {t}) + \eta \cdot e ^ {- \theta w _ {t}} \cdot H (\bar {\mathbf {w}} _ {t}) \right\}. \\ \end{array}
$$
$$
w _ {t + 1} = w _ {t} - \eta \cdot \nabla_ {w} L (w _ {t}, \bar {\mathbf {w}} _ {t}), \qquad \nabla_ {w} L (w _ {t}, \bar {\mathbf {w}} _ {t}) = - \sum_ {i = 1} ^ {n} \frac {e ^ {- x _ {i} w _ {t} - \bar {\mathbf {x}} _ {i} ^ {\top} \bar {\mathbf {w}} _ {t}}}{1 + e ^ {- x _ {i} w _ {t} - \bar {\mathbf {x}} _ {i} ^ {\top} \bar {\mathbf {w}} _ {t}}} \cdot x _ {i}.
$$
We only need to provide upper and lower bounds on  $-\nabla_w L(w_t, \bar{\mathbf{w}}_t)$ . The lower bound is because:
$$
\begin{array}{l} - \nabla_ {w} L (w _ {t}, \bar {\mathbf {w}} _ {t}) = \sum_ {i = 1} ^ {n} \frac {e ^ {- x _ {i} w _ {t} - \bar {\mathbf {x}} _ {i} ^ {\top} \bar {\mathbf {w}} _ {t}}}{1 + e ^ {- x _ {i} w _ {t} - \bar {\mathbf {x}} _ {i} ^ {\top} \bar {\mathbf {w}} _ {t}}} \cdot x _ {i} \\ \geq \sum_ {i \in \mathcal {S}} \frac {e ^ {- x _ {i} w _ {t} - \bar {\mathbf {x}} _ {i} ^ {\top} \bar {\mathbf {w}} _ {t}}}{1 + e ^ {- x _ {i} w _ {t} - \bar {\mathbf {x}} _ {i} ^ {\top} \bar {\mathbf {w}} _ {t}}} \cdot x _ {i} \quad \text {s i n c e} x _ {i} \geq \gamma > 0 \text {b y D e f i n i t i o n 1} \\ = \sum_ {i \in \mathcal {S}} \frac {e ^ {- \gamma w _ {t} - \bar {\mathbf {x}} _ {i} ^ {\top} \bar {\mathbf {w}} _ {t}}}{1 + e ^ {- \gamma w _ {t} - \bar {\mathbf {x}} _ {i} ^ {\top} \bar {\mathbf {w}} _ {t}}} \cdot \gamma \quad \text {s i n c e} x _ {i} = \gamma \text {f o r} i \in \mathcal {S} \\ \geq \frac {\gamma}{2} \cdot \sum_ {i \in S} \min  \left\{1, e ^ {- \gamma w _ {t} - \bar {\mathbf {x}} _ {i} ^ {\top} \bar {\mathbf {w}} _ {t}} \right\} \quad \text {s i n c e} e ^ {t} / (1 + e ^ {t}) \geq 0. 5 \min  \left\{1, e ^ {t} \right\} \\ \geq \frac {\gamma}{2} \cdot \min  \left\{1, e ^ {- \gamma w _ {t}} \cdot \sum_ {i \in \mathcal {S}} e ^ {- \bar {\mathbf {x}} _ {i} ^ {\top} \bar {\mathbf {w}} _ {t}} \right\} \\ = \frac {\gamma}{2} \cdot \min  \left\{1, e ^ {- \gamma w _ {t}} \cdot G (\bar {\mathbf {w}} _ {t}) \right\}. \\ \end{array}
$$
The upper bound is because:
$$
- \nabla_ {w} L (w _ {t}, \bar {\mathbf {w}} _ {t}) = \sum_ {i = 1} ^ {n} \frac {e ^ {- x _ {i} w _ {t} - \bar {\mathbf {x}} _ {i} ^ {\top} \bar {\mathbf {w}} _ {t}}}{1 + e ^ {- x _ {i} w _ {t} - \bar {\mathbf {x}} _ {i} ^ {\top} \bar {\mathbf {w}} _ {t}}} \cdot x _ {i}
$$