SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Jingfeng Wu
Baltimore, MD 21218
uuujf@jhu.edu
Vladimir Braverman
vb21@rice.edu
Princeton, NJ 08544
jasonlee@princeton.edu
Recent research has observed that in machine learning optimization, gradient descent (GD) often operates at the edge of stability (EoS) [Cohen et al., 2021], where the stepsizes are set to be large, resulting in non-monotonic losses induced by the GD iterates. This paper studies the convergence and implicit bias of constant-stepsize GD for logistic regression on linearly separable data in the EoS regime. Despite the presence of local oscillations, we prove that the logistic loss can be minimized by GD with any constant stepsize over a long time scale. Furthermore, we prove that with any constant stepsize, the GD iterates tend to infinity when projected to a max-margin direction (the hard-margin SVM direction) and converge to a fixed vector that minimizes a strongly convex potential when projected to the orthogonal complement of the max-margin direction. In contrast, we also show that in the EoS regime, GD iterates may diverge catastrophically under the exponential loss, highlighting the superiority of the logistic loss. These theoretical findings are in line with numerical simulations and complement existing theories on the convergence and implicit bias of GD for logistic regression, which are only applicable when the stepsizes are sufficiently small.
Gradient descent (GD) is a foundational algorithm for machine learning optimization that motivates many popular algorithms. Theoretically, the behavior of GD is well understood when the stepsize is small. In this regard, one of the most classic results is the descent lemma (see, e.g., Section 1.2.3 in Nesterov et al. [2018]):
Lemma (Descent lemma, simplified version). Suppose that  $\sup_{\mathbf{w}}\left\| \nabla^2 L(\mathbf{w})\right\| _2\leq \beta^1$  , then
$$
L (\mathbf {w} _ {+}) \leq L (\mathbf {w}) - \eta \cdot (1 - \eta \beta / 2) \cdot \| \nabla L (\mathbf {w}) \| _ {2} ^ {2}, \quad w h e r e \mathbf {w} _ {+} := \mathbf {w} - \eta \cdot \nabla L (\mathbf {w}).
$$
When the targeted function is smooth (such as logistic regression) and the stepsize is small  $(0 < \eta < \beta / 2)$ , the descent lemma ensures a monotonic decrease of the function value by performing each GD step. Building upon this, a sequence of iterates produced by GD with small step sizes provably minimizes the function value in various settings (see, e.g., Lan [2020]).
For a more modern example, a recent line of research has established the implicit bias of GD with small step sizes (see Soudry et al. [2018], Ji and Telgarsky [2018] and references thereafter). Specifically, they consider GD for optimizing logistic regression (besides other loss functions) on
The uniformly bounded Hessian norm condition is stated for simplicity and can be relaxed in many ways. For example, it can be replaced by requiring  $L(\cdot)$  to be  $\beta$ -smooth. For another example, it can also be replaced with  $\sup_{0 \leq \lambda \leq 1} \left\| \nabla^2 L(\lambda \cdot \mathbf{w} + (1 - \lambda) \cdot \mathbf{w}_+) \right\|_2 \leq \beta$ .
(a) Training loss
(b) Sharpness
Figure 1: The behaviors of GD for optimizing a neural network. We randomly sample 1,000 data from the MNIST dataset and then use GD to train a 4-layer fully connected network to fit those data. We use the cross-entropy loss, i.e., the multi-class version of the logistic loss. The sub-figures (a), (b), and (c) report the training loss, test accuracy, and sharpness (i.e.,  $\| \nabla L(\mathbf{w}_t)\| _2$ ) along the GD trajectories, respectively. The red curves correspond to GD with a large stepsize  $\eta = 0.1$  , where the training losses oscillate locally and the sharpnesses can exceed  $2 / \eta = 20$  . The green curves correspond to GD with a small stepsize  $\eta = 0.01$  , where the training losses decrease monotonically and the sharpnesses are always below  $2 / \eta = 200$  . Moreover, (c) suggests that large-stepsize GD achieves better test accuracy than small-stepsize GD, consistent with larger-scale deep learning experiments [Goyal et al., 2017]. More details of the experiments can be found in Appendix D.
(c) Test accuracy
linearly separable data. When the stepsizes are sufficiently small, the GD iterates are shown to decrease the risk monotonically (by a variant of the descent lemma); moreover, the GD iterates tend to align with a direction that maximizes the  $\ell_2$ -margin of the data [Soudry et al., 2018, Ji and Telgarsky, 2018]. The margin-maximization bias of small-stepsize GD sheds important light on understanding the statistical benefits of GD, as a large margin solution often generalizes well [Bartlett et al., 2017, Neyshabur et al., 2018].
Nonetheless, in practical machine learning optimization, especially in deep learning, the empirical risk (or training loss) often varies non-monotonically (while being minimized in the long run) — the local risk oscillation is not only caused by the algorithmic randomness but is more an effect of using large step sizes, as it happens for deterministic GD (with large step sizes) as well [Wu et al., 2018, Xing et al., 2018, Lewkowycz et al., 2020, Cohen et al., 2021]. This phenomenon is showcased in Figures 1(a) and 2(a), and is referred to by Cohen et al. [2021] as the edge of stability (EoS). The observation sets a non-negligible gap between practical and theoretical GD setups, where in practice, GD is run with large step sizes that lead to local risk oscillations, but in theory, GD is only considered with sufficiently small step sizes, predicting a monotonic risk descent (with a few exceptions, which will be discussed later in Section 2). A tension remains to be resolved:
Is the convergence of risk under local oscillation merely a "lucky" occurrence, or is it predictable based on theory?
Contributions. In this work, we study the behaviors of GD in the EoS regime in arguably the simplest setting for machine learning optimization — logistic regression on linearly separable data. We show that with any constant stepsize, while the induced risks may oscillate locally, GD must minimize the risk in the long run at a rate of  $\mathcal{O}(1 / t)$ , where  $t$  is the number of iterates. In addition, we show that the direction of the GD iterates (with any constant stepsize) must align with a max-margin direction (the hard-margin SVM direction) at a rate of  $\mathcal{O}(1 / \log(t))$ . These results explain how GD minimizes a risk non-monotonically, and complement existing theories [Soudry et al., 2018, Ji and Telgarsky, 2018] on the convergence and implicit bias of GD, which are only applicable when the stepsizes are sufficiently small.
Some additional notable contributions are
1. We also show that, when projected to the orthogonal complement of the max-margin direction, the GD iterates (with any constant stepsize) converge to a fixed vector that minimizes a strongly convex potential at a rate of  $\mathcal{O}(1 / \log(t))$ . This characterization is conceptually more interpretable than an existing version [Soudry et al., 2018].
2. We show that in the EoS regime, GD can diverge catastrophically if the logistic loss is replaced by the exponential loss. This is in stark contrast to the small-stepsize regime, where the behaviors
of GD are known to be similar under any exponentially-tailed losses including both the logistic and exponential losses [Soudry et al., 2018, Ji and Telgarsky, 2018]. The difference in the EoS regime provides insights into why the logistic loss is preferable to the exponential loss in practice.
3. From a technical perspective, we develop a new approach for analyzing GD with large step sizes. Our approach views the GD iterates as a coupling of two orthogonal iterates, one along a max-margin direction and the other along the orthogonal complement of the max-margin direction. The former iterates tend to infinity and the latter iterates approximate "imaginary" GD iterates that minimize a strongly convex potential with a decaying stepsize scheduler, controlled by the former iterates. Our techniques for analyzing large-stepsize GD can be of independent interest.
In this section, we discuss papers related to our work.
Implicit bias. We first review a set of papers on the implicit bias of GD (with small step sizes).
Along this line, Soudry et al. [2018] are the very first to show that GD converges along a max-margin direction when minimizing the empirical risk of an exponentially-tailed loss function (such as the logistic and exponential losses), a linear model, and linearly separable data. Then, an alternative analysis is provided by Ji and Telgarsky [2018], which also deals with non-separable data. These two works directly motivate us for considering GD for logistic regression on linearly separable data. However, there are at least three notable differences between our work and theirs. Firstly, their results only apply to GD with small step sizes, while our results apply to GD with any constant stepsize. Secondly, their theories predict no difference between the logistic and exponential losses (as they are limited to the small-stepsize regime). Quite surprisingly, we prove that in the EoS regime, GD can diverge catastrophically under the exponential loss. Thirdly, from a technical viewpoint, their implicit bias analysis is built upon the risk convergence analysis, which further relies on a monotonic risk descent argument, hence only applies to small step sizes. In comparison, we come up with a new approach that allows analyzing the implicit bias under risk oscillations; the long-term risk convergence is simply a consequence of the implicit bias results. Hence our techniques can accommodate any constant stepsize. See Section 5 for more discussions.
Subsequent works have extended the results by Soudry et al. [2018], Ji and Telgarsky [2018] to other algorithms such as momentum-based GD [Gunasekar et al., 2018a, Ji et al., 2021] and SGD [Nacson et al., 2019c], and homogenous but non-linear models [Gunasekar et al., 2017, Ji and Telgarsky, 2019, Gunasekar et al., 2018b, Nacson et al., 2019a, Lyu and Li, 2020] and non-homogenous models [Nacson et al., 2019a]. All these theories require the stepsizes to be small or even infinitesimal, in a regime away from our focus, the EoS regime.
It is worth noting that Nacson et al. [2019b] consider GD with an increasing stepsize scheduler that achieves a faster margin-maximization rate than constant-stepsize GD. However, their stepsize at each iteration is still appropriately small, resulting in a monotonic risk descent by a variant of the descent lemma.
Edge of stability. The risk oscillation phenomenon has been observed in several deep learning papers [Wu et al., 2018, Xing et al., 2018, Lewkowycz et al., 2020], and the work by Cohen et al. [2021] coins the term, edge of stability (EoS), that formally refers to it. In the remainder of this part, we focus on reviewing the current theoretical progress in understanding EoS.
Zhu et al. [2023] rigorously characterize EoS for a two-dimensional function  $(u,v)\mapsto (u^{2}v^{2} - 1)^{2}$ . Chen and Bruna [2022] study EoS for a one-dimensional function  $u\mapsto (u^2 -1)^2$  and for a special two-layer single-neuron network. Similar to these two works, Kreisler et al. [2023] study EoS in a 1-dimensional linear network. Ahn et al. [2022a] consider functions  $(u,v)\mapsto \ell (uv)$ , where  $\ell$  is assumed to be convex, even, and Lipschitz; notably, they show a statistical gap between the small-stepsize regime and the EoS regime. Finally, Even et al. [2023], Andriushchenko et al. [2023] consider the regularization effect of large stepsizes in a diagonal linear network. Compared to their settings, our problem, i.e., logistic regression, is a natural machine-learning problem with fewer artifacts (if any).
EoS has also been theoretically investigated for general functions [Ma et al., 2022, Ahn et al., 2022b, Damian et al., 2022, Wang et al., 2022b], but these theories are often subject to subtle assumptions that are hard to interpret or verify. Specifically, Ma et al. [2022] require the function to grow
"subquadratically". Ahn et al. [2022b] assume the existence of a "forward invariant subset" near the set of minima of the function. Damian et al. [2022] assume a negative correlation between the gradient direction and the largest eigenvalue direction of the Hessian. Wang et al. [2022b] consider a two-layer neural network but require the norm of the last layer parameter and the sharpness to change in the same direction along the GD trajectory. Indirectly connected to EoS, the work by Kong and Tao [2020] shows a chaotic behavior of GD with a non-small stepsize when optimizing a "multi-scale" loss function. In comparison, our assumptions are more natural and interpretable.
Besides, the work by Lyu et al. [2022] considers EoS induced by GD for scale-invariant loss, e.g., a network with normalization layers and weight decay, and the work by Wang et al. [2022a] shows a balancing effect in matrix factorization induced by GD with a constant stepsize that is nearly  $4 / \|\nabla^2 L(\mathbf{w}_0)\|_2$  and is larger than  $2 / \|\nabla^2 L(\mathbf{w}_0)\|_2$ . The objectives in their works are different from ours, i.e., logistic regression.
The unstable convergence has also been studied for normalized GD [Arora et al., 2022] and regularized GD [Bartlett et al., 2022]. These algorithms are apart from our focus on the vanilla GD.
Finally, the work by Liu et al. [2023] considers logistic regression with non-separable data (such that the objective is strongly convex), where GD with sufficiently large stepsize diverges. In contrast, we consider logistic regression with separable data, where GD with an arbitrarily large stepsize still converges.
We use  $\mathbf{x} \in \mathbb{R}^d$  to denote a feature vector and  $y \in \{\pm 1\}$  to denote a binary label, respectively. Let  $(\mathbf{x}_i, y_i)_{i=1}^n$  be a set of training data. Throughout the paper, we assume that  $(\mathbf{x}_i, y_i)_{i=1}^n$  is linearly separable [Soudry et al., 2018].
Assumption 1 (Linear separability). Assume there is  $\mathbf{w} \in \mathbb{R}^d$  such that  $y_i \mathbf{x}_i^\top \mathbf{w} > 0$  for  $i = 1, \ldots, n$ .
Let  $\mathbf{w} \in \mathbb{R}^d$  be the parameter of a linear model. In logistic regression, we aim to minimize the following empirical risk
$$
L (\mathbf {w}) := \sum_ {i = 1} ^ {n} \log \left(1 + \exp (- y _ {i} \cdot \langle \mathbf {x} _ {i}, \mathbf {w} \rangle)\right), \quad \mathbf {w} \in \mathbb {R} ^ {d}.
$$
We study a sequence of iterates  $(\mathbf{w}_t)_{t\geq 0}$  produced by constant-stepsize gradient descent (GD), where  $\mathbf{w}_0$  denotes the initialization and the remaining iterates are sequentially generated by:
$$
\mathbf {w} _ {t} = \mathbf {w} _ {t - 1} - \eta \cdot \nabla L \left(\mathbf {w} _ {t - 1}\right), \quad t \geq 1, \tag {GD}
$$
where  $\eta > 0$  is a constant stepsize. We are especially interested in a regime where  $\eta$  is very large such that  $L(\mathbf{w}_t)$  oscillates as a function of  $t$ . For the simplicity of presentation, we will assume that  $\mathbf{w}_0 = 0$ . Our results can be easily extended to allow general initialization.
The following notations are useful for presenting our results.
Definition 1 (Margins and support vectors). Under Assumption 1, define the following notations:
(A) Let  $\gamma$  be the max- $\ell_2$ -margin (or max-margin in short), i.e.,
$$
\gamma := \max  _ {\| \mathbf {w} \| _ {2} = 1} \min  _ {i \in [ n ]} y _ {i} \cdot \left\langle \mathbf {x} _ {i}, \mathbf {w} \right\rangle .
$$
(B) Let  $\hat{\mathbf{w}}$  be the hard-margin support-vector-machine (SVM) solution, i.e.,
$$
\hat {\mathbf {w}} := \arg \min  _ {\mathbf {w} \in \mathbb {R} ^ {d}} \| \mathbf {w} \| _ {2}, \text {s . t .} y _ {i} \cdot \left\langle \mathbf {x} _ {i}, \mathbf {w} \right\rangle \geq 1, i = 1, \ldots , n.
$$
It is clear that  $\hat{\mathbf{w}}$  exists and is uniquely defined (see, e.g., Section 5.2 in Mohri et al. [2018]). Note that  $\| \hat{\mathbf{w}} \|_2 = 1 / \gamma$  and  $\hat{\mathbf{w}} / \| \hat{\mathbf{w}} \|_2$  is a max-margin direction. Also note that by duality,  $\hat{\mathbf{w}}$  can be written as (see, e.g., Section 5.2 in Mohri et al. [2018])
$$
\hat {\mathbf {w}} = \sum_ {i \in \mathcal {S}} \alpha_ {i} \cdot y _ {i} \mathbf {x} _ {i}, \quad \alpha_ {i} \geq 0.
$$
(C) Let  $S$  be the set of indexes of the support vectors, i.e.,
$$
\mathcal {S} := \left\{i \in [ n ]: y _ {i} \cdot \langle \mathbf {x} _ {i}, \hat {\mathbf {w}} / \| \hat {\mathbf {w}} \| _ {2} \rangle = \gamma \right\}.
$$
(D) If there exists non-support vector  $(S\subsetneq [n])$  , let  $\theta$  be the second smallest margin, i.e.,
$$
\theta := \min _ {i \notin S} y _ {i} \cdot \langle \mathbf {x} _ {i}, \hat {\mathbf {w}} / \| \hat {\mathbf {w}} \| _ {2} \rangle .
$$
It is clear from the definitions that  $\theta >\gamma >0$  . In addition, from the definitions we have
$$
\sum_ {i \in \mathcal {S}} \alpha_ {i} = \sum_ {i \in \mathcal {S}} \alpha_ {i} \cdot y _ {i} \mathbf {x} _ {i} ^ {\top} \hat {\mathbf {w}} = \| \hat {\mathbf {w}} \| _ {2} ^ {2} = \frac {1}{\gamma^ {2}}.
$$
In addition to Assumption 1, we make the following two mild assumptions to facilitate our analysis.
Assumption 2 (Regularity conditions). Assume that:
(A)  $\| \mathbf{x}_i\| _2\leq 1$ $i = 1,\dots ,n$
$(B)$ $\mathrm{rank}\{\mathbf{x}_i,i = 1,\dots ,n\} = d.$
Assumption 2 is only made for the convenience of presentation. In particular, Assumption 2(A) can be made true for any dataset by scaling the data vectors with a factor of  $\max_i\| \mathbf{x}_i\| _2$ . Without Assumption 2(B), our theorems still hold under a minor revision by replacing all the vectors of interests with their projections to  $\operatorname {span}\{\mathbf{x}_i,i = 1,\dots ,n\}$ .
Assumption 3 (Non-degenerate data). In addition to Assumption 1, assume that
(A)  $\operatorname{rank}\{\mathbf{x}_i, i \in \mathcal{S}\} = \operatorname{rank}\{\mathbf{x}_i, i = 1, \dots, n\}$ .
(B) There exist  $\alpha_{i} > 0, i\in S$  such that  $\hat{\mathbf{w}} = \sum_{i\in S}\alpha_{i}\cdot y_{i}\mathbf{x}_{i}$ .
Assumption 3 has been used in Soudry et al. [2018] (see their Theorem 4), which requires that the support vectors span the dataset and are associated with strictly positive dual variables. Assumption 3(B) holds almost surely for every linearly separable dataset sampled from a continuous distribution according to Appendix B in Soudry et al. [2018]. Assumption 3 provides convenience to our analysis, but we conjecture it might not be necessary. Removing/relaxing Assumption 3 is left as a future work.
3.1 Space Decomposition
Conceptually, our analysis is built on a novel space decomposition viewpoint, which relies on the following lemma.
Lemma 3.1 (Non-separable subspace). Suppose that Assumptions 1, 2, and 3 hold. Then  $(\mathbf{x}_i, y_i)_{i \in S}$  is not linearly separable in the subspace orthogonal to the max-margin direction  $\hat{\mathbf{w}} / \|\hat{\mathbf{w}}\|_2$ . That is, for every  $\mathbf{v}$  such that  $\langle \mathbf{v}, \hat{\mathbf{w}} \rangle = 0$ , there exist  $i, j \in S$  such that  $y_i \cdot \langle \mathbf{x}_i, \mathbf{v} \rangle < 0$ ,  $y_j \cdot \langle \mathbf{x}_j, \mathbf{v} \rangle > 0$ .
Proof of Lemma 3.1. By Assumption 3 and  $\langle \mathbf{v},\hat{\mathbf{w}}\rangle = 0$  we have
$$
0 = \langle \mathbf {v}, \hat {\mathbf {w}} \rangle = \sum_ {i \in \mathcal {S}} \alpha_ {i} \cdot y _ {i} \mathbf {x} _ {i} ^ {\top} \mathbf {v}.
$$
By Assumptions 2 and 3 we have
$$
\operatorname {r a n k} \left\{y _ {i} \mathbf {x} _ {i}, i \in \mathcal {S} \right\} = \operatorname {r a n k} \left\{\mathbf {x} _ {i}, i \in \mathcal {S} \right\} = \operatorname {r a n k} \left\{\mathbf {x} _ {i}, i = 1, \dots , n \right\} = d,
$$
so there must exist  $i \in S$  such that  $y_{i} \mathbf{x}_{i}^{\top} \mathbf{v} \neq 0$ . Without loss of generality, assume that  $y_{i} \mathbf{x}_{i}^{\top} \mathbf{v} < 0$ . Then since  $\alpha_{i} > 0$  for  $i \in S$  by Assumption 3, there must exist  $j \in S$  such that  $y_{j} \mathbf{x}_{j}^{\top} \mathbf{v} > 0$ .
Lemma 3.1 shows that, although the dataset can be (linearly) separated by  $\hat{\mathbf{w}}$ , it cannot be separated by any vector orthogonal to  $\hat{\mathbf{w}}$ . This motivates us to decompose the  $d$ -dimensional ambient space into a 1-dimensional "separable" subspace and a  $(d - 1)$ -dimensional "non-separable" subspace. This idea is formally realized as follows.
Fix  $d - 1$  orthogonal vectors  $\mathbf{f}_1,\ldots ,\mathbf{f}_{d - 1}\in \mathbb{R}^d$  such that  $\left(\hat{\mathbf{w}} /\| \hat{\mathbf{w}}\| _2,\mathbf{f}_1,\dots ,\mathbf{f}_{d - 1}\right)$  forms an orthogonal basis of the ambient space  $\mathbb{R}^d$ . Then define two projection operators:
$$
\mathcal {P} \colon \mathbb {R} ^ {d} \to \mathbb {R} \quad \text {g i v e n b y} \quad \mathbf {v} \mapsto \mathbf {v} ^ {\top} \hat {\mathbf {w}} / \| \hat {\mathbf {w}} \| _ {2},
$$
$$
\bar {\mathcal {P}} \colon \mathbb {R} ^ {d} \to \mathbb {R} ^ {d - 1} \quad \text {g i v e n b y} \quad \mathbf {v} \mapsto (\mathbf {v} ^ {\top} \mathbf {f} _ {1}, \ldots , \mathbf {v} ^ {\top} \mathbf {f} _ {d - 1}).
$$
The two operators together define a natural space decomposition, i.e.,  $\mathbb{R}^d = \mathcal{P}(\mathbb{R}^d) \oplus \mathcal{P}(\mathbb{R}^d)$ . Moreover,  $\left(\mathcal{P}(\mathbf{x}_i), y_i\right)_{i=1}^n$  are linearly separable with an max- $\ell_2$ -margin  $\gamma$  according to Definition 1, and  $\left(\bar{\mathcal{P}}(\mathbf{x}_i), y_i\right)_{i \in S}$  (hence  $\left(\bar{\mathcal{P}}(\mathbf{x}_i), y_i\right)_{i=1}^n$ ) are non-separable according to Lemma 3.1. So the decomposition of space can also be understood as the decomposition of data features into "max-margin features" and "non-separable features".
In what follows, we will call  $\mathcal{P}(\mathbb{R}^d)$  the max-margin subspace and  $\bar{\mathcal{P}} (\mathbb{R}^d)$  the non-separable subspace, respectively. In addition, we define a "margin offset" that quantifies to what extent the "non-separable features" are not separable.
Definition 2 (Margin offset for the non-separable features). Under Assumptions 1, 2, and 3, it holds that  $\left(\mathcal{P}(\mathbf{x}_i),y_i\right)_{i\in S}$  is non-separable. Let  $b$  be a margin offset such that
$$
- b := \max  _ {\bar {\mathbf {w}} \in \mathbb {R} ^ {d - 1}, \| \bar {\mathbf {w}} \| _ {2} = 1} \min  _ {i \in \mathcal {S}} y _ {i} \cdot \left\langle \bar {\mathcal {P}} (\mathbf {x} _ {i}), \bar {\mathbf {w}} \right\rangle .
$$
Then  $b > 0$  due to the non-separability. The definition immediately implies that:
$$
f o r e v e r y \bar {\mathbf {v}} \in \mathbb {R} ^ {d - 1}, \text {t h e r e e x i s t s} i \in \mathcal {S} \text {s u c h t h a t} y _ {i} \cdot \left\langle \bar {\mathcal {P}} (\mathbf {x} _ {i}), \bar {\mathbf {v}} \right\rangle \leq - b \cdot \| \bar {\mathbf {v}} \| _ {2}.
$$
Comparison to Ji and Telgarsky [2018]. The work by Ji and Telgarsky [2018] also conducts space decomposition (see their Section 2). However, our approach is completely different from theirs. Firstly, they consider a non-separable dataset but we consider a linearly separable dataset. Secondly, at a higher level, they decompose the "dataset" (into two subsets), while we decompose the "features" (into two kinds of features). More specifically, Ji and Telgarsky [2018] first group the non-separable dataset into the "maximal linearly separable subset" and the complement, non-separable subset, then decompose the ambient space according to the subspace spanned by the non-separable subset and its orthogonal complement. In comparison, we consider a linearly separable dataset and decompose the ambient space according to a max-margin direction (i.e.,  $\mathcal{P}$ ) and its orthogonal complement (i.e.,  $\bar{\mathcal{P}}$ ).
We are now ready to present our main results. All proofs are deferred to Appendix C. To begin with, we provide the following theorem that captures the behaviors of constant-stepsize GD for logistic regression on linearly separable data.