Title: Large Stepsizes Accelerate Gradient Descent for Regularized Logistic Regression
Venue: NeurIPS 2025
Accepted: yes
Keywords: GD

large stepsize

logistic regression

EoS

regularization

acceleration
Abstract: We study *gradient descent* (GD) with a constant stepsize for $\ell_2$-regularized logistic regression with linearly separable data. Classical theory suggests small stepsizes to ensure monotonic reduction of the optimization objective, achieving exponential convergence in $\widetilde{\mathcal{O}}(\kappa)$ steps with $\kappa$ being the condition number. Surprisingly, we show that this can be *accelerated* to $\widetilde{\mathcal{O}}(\sqrt{\kappa})$ by simply using a large stepsize---for which the objective evolves *nonmonotonically*. The acceleration brought by large stepsizes extends to minimizing the population risk for separable distributions, improving on the best-known upper bounds on the number of steps to reach a near-optimum. Finally, we characterize the largest stepsize for the local convergence of GD, which also determines the global convergence in special scenarios. Our results extend the analysis of Wu et al. (2024) from convex settings with minimizers at infinity to strongly convex cases with finite minimizers.


Peer reviews:
comments: SUMMARY:
This paper studies gradient descent (GD) for $\ell_2$-regularized logistic regression with linearly separable data under a large constant stepsize. Considering different regimes of the regularization strength, this paper shows various accelerating behavior induced by the large stepsize, particularly about the jump between stable phase (where the loss decreases monotonically) and edge of stability (EoS) phase (the loss does not decrease monotonically) where phase transition points are indentified. Moreover, the critical threshold on the convergent stepsizes is also discussed.

STRENGTHS AND WEAKNESSES:
### Strengths
* This paper is well written and clear to read.
* This paper studies the interplay of large stepsize and EoS for $\ell_2$-regularized logistic regression, which is novel.
The result on the different regimes of stepsizes is very interesting and provide more understandings for the study of EoS.
* The theorems in this paper are clear and seem to be well supported by the proofs (though I haven't checked all the details).


### Weaknesses

* Numerical experiments can be included to further justify the theoretical results. Demonstrating a training path can provide readers with more intuitions.
* Filling out the missing parts of the full picture of the theoretical results as shown in Figure 1 would make the theory more complete.

QUESTIONS:
1. Can you discuss more on the assumptions? Does similar result hold if the separability condition is not satisfied?

2. What is the explicit dependence of the constants $C_1,C_2,C_3$ in the theorems 1, 4 and 7 on the dataset? How large will they be?

3. Can the result and techniques be extended to other models such as generalized linear regression?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 4
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-30

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper shows that gradient descent (GD) with a properly chosen large constant step size achieves fast convergence for $\\ell_2$-regularized logistic regression when the data is linearly separable. This thereby extends our understanding of the optimization benefit of GD operating at the edge of stability regime. In particular, this work builds on a recent result by Wu et al. (2024), which showed that a large constant step size can accelerate GD in the unregularized logistic regression setting. Although the two results appear related, the proof techniques developed in the unregularized setting do not directly extend to the regularized case considered here, due to the presence of a finite minimizer. \n\nFor a small regularization parameter $\\lambda = O(1/n^2)$, Theorem 1 establishes that large step GD achieves a convergence rate of $O(\\ln(1/\\epsilon)/\\sqrt{\\lambda})$ matching that of Nesterov's momentum. Moreover, the analysis shows that GD enters a stable phase with linear convergence after a finite number of steps. While the acceleration is theoretically appealing, the authors correctly note that such a small regularization level may be inadequate in the presence of statistical noise. \n\nTo address this limitation, this paper further investigates the regime of larger regularization and statistical uncertainty. When $\\lambda=O(1)$ and the step size is set to $\\eta=O(1/\\lambda^{1/3})$, GD achieves the complexity of $O\\(\\ln(1/\\epsilon)/\\lambda^{2/3})$, which improves over small-step GD but remains slower than Nesterov's momentum. In Section 3, the authors further investigate the benefits of large step under statistical uncertainty. Based on the best-known population risk upper bound in Proposition 6, the authors argue that the largest reduction in population risk is obtained when choosing $\\lambda=1/n$ and $\\eta=(\\gamma^2n)^{1/3}$. However, under this setup, the rate does not match the fast rate provided by Nesterov's momentum.\n\nFinally, this paper completes its analysis by establishing that $1/\\sqrt{\\lambda}$ is the critical threshold between (local) convergence and divergence of large-step GD, under a mild support vectors condition.\n\nSTRENGTHS AND WEAKNESSES:\n- S1: As noted in Section 2.3, while the main claims resemble those of Wu et al. (2024), the proof technique developed in this paper for the regularized setting is novel and nontrivial.\n- S2: Despite the limited benefit of large step size under strong regularization and statistical uncertainty, this paper offers a comprehensive theoretical analysis of large step GD for regularized logistic regression.\n- S3: This paper is clearly written and well-structured. The main results appear technically sound, although I have not carefully checked the proofs.\n\n- W1: Related to S2, the rate matching the optimal rate of Nesterov's momentum is achieved only in the small regularization regime, which limits the practical relevance of the result under statistical uncertainty.\n\nQUESTIONS:\n- Page 2 line 64: Is there a reference that supports this statement? If this claim is supported by this paper, I think it would be helpful to add a brief explanation or clarification at this point.\n- Page 3 line 117: Why is there a gap in the large step size analysis between the separable and non-separable cases? Is it due to logistic regression being strictly convex but not strongly convex under strictly non-separable data? What would happen if one adds regularization in the strictly non-separable case?\n- Page 4 line 151: Could you provide some intuition behind the transition in behavior at the step size $1/\\sqrt{\\lambda}$?\n- Page 8 Table 1: Do the theoretical comparisons in Table 1 align with the empirical observations?\n- Page 9 line 365: Could you clarify the last two future directions? They were somewhat vague and difficult to interpret.\n\n- Minor comments: Unlike the main texts, the conclusion does not explicitly mention the requirement of small regularization for matching the rate of Nesterov's momentum. I suggest including this. In addition, when referring to (Cai et al., 2024), something other than linearly separable data should be mentioned.\n\nLIMITATIONS:\nThe paper clearly states its limitations at the end of each section.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn the paper (Cohen et al., 2020), a phenomenon ``edge of stability’’ (EoS) regime is\ninvestigated under the training of neural networks (NN). The EoS indicates that the NN\ntraining can still converge if using a relatively large learning rate.\n\nAnother paper (Wu et al., 2024) utilizes a non-regularized logistic regression to investigate the\nEoS regime. The current paper expand the results to regularized logistic regression\nproblems.\n\nSTRENGTHS AND WEAKNESSES:\n#### Strengths\n\n- This paper is written clearly.\n- This paper utilizes a simpler problem (logistic regression) to explain the Eos regime issue.\n\n#### Weaknesses\n\n- A concern is that this paper may have only theoretical values. For logistic regression, gradient descent is not an effective method. People have shown that quasi Newton (e.g., LBFGS), truncated Newton (via conjugat gradient methods) or even coordiate descent methods are better choices in different scenarios. Moreover, the paper concerns about linearly separable data, a situation not common in practice.\n\n- The proofs in the appendices skip many details. \n  Thus, it is hard to verify the correctness of the proofs.\n  For example,\n  a) In line 818, it seems that the right hand side lacks the term $$ -4 \\eta \\nabla L(w_t)^T w_t. $$\n  b) In line 828, the left hand side is equal to $$ \\eta^2 [ \\nabla L (w_t)^T w^* + \\| \\nabla L(w_t) \\|^2 ], $$\n     but why this term is less than zero?\n  c) In line 864, why do we have the second inequality\n     $$ \\frac{e}{n} \\sum_{i=1}^{n} \\left( \\frac{1}{1+exp(y_i x_{i}^{T} w_t )} \\right)^{1-\\alpha \\lambda \\eta} \\leq e \\left( \\frac{1}{n} \\sum_{i=1}^{n} \\frac{1}{1+exp(y_i x_{i}^{T} w_t)} \\right)^{1-\\alpha \\lambda \\eta} ? $$\n  \n- The paper does not contain any experiment to verify their proofs.\n\nQUESTIONS:\nSee \"weaknesses\"\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis manuscript analyzes gradient descent with large constant step sizes for $\\ell_2$-regularized logistic regression on linearly separable data. Building on a series of “edge-of-stability” (EoS) works that studied the unregularized case, the authors show that the same unstable dynamics can accelerate convergence in the strongly convex setting induced by $\\ell_2$ regularization. Their choice of stepsize and regularization scaling is motivated by minimizing the generalization bound presented in Proposition 6.\n\nSpecifically, the manuscript shows that the iteration complexity depends on the scale of the regularization parameter $\\lambda$. When $\\lambda \\leq O(n^{-2})$, gradient descent achieves $O(1/\\sqrt{\\lambda})$ iteration complexity, matching the convergence guarantee of Nesterov’s accelerated gradient method. For general $\\lambda$, they obtain an iteration complexity of $O(\\lambda^{-2/3})$.\n\nIn addition, the authors establish a lower bound on the iteration complexity when the loss function is monotone, showing that this regime requires $\\Omega(1/\\lambda)$ iterations, which demonstrates a separation between non monotonic and monotonic dynamics of GD.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n* The manuscript extends an existing line of work to a relevant and practically motivated setting. Unlike the unregularized case, where the minimizer lies at infinity, this setup requires genuinely new technical arguments to handle the $\\ell_2$-regularized objective.\n\n**Weaknesses:**\n* I find the way the contributions are introduced in the abstract somewhat misleading. While it is true that the proposed method can match Nesterov’s accelerated gradient convergence rate, as discussed in Section 6, this happens only in the regime $\\lambda \\leq O(n^{-2})$, which is arguably not the most practically relevant case. In contrast, in the more interesting regime $\\lambda = O(n^{-1})$, the result yields an iteration complexity of $O(\\lambda^{-2/3})$, which remains slower than Nesterov’s method. I would recommend revising the abstract and introduction to clarify this distinction.\n* More broadly, although the overall result is slower than a known baseline (Nesterov’s momentum), the paper still offers an interesting contribution given the emerging relevance of edge-of-stability dynamics in practice.\n* Finally, the presentation and discussion of the results could be improved for clarity; see my questions and comments below.\n\nQUESTIONS:\n**Some discussions:**\n\n* Both learning rates in Theorem 1 and Theorem 4 require a priori knowledge of the margin parameter $\\gamma$. I would expect that if this parameter is not known, the generalization bounds in Table 1 could become suboptimal. Could you discuss how realistic this assumption is in practice? Additionally, it would be helpful to compare the results to adaptive gradient methods (e.g., Zhang et al. \\[2025]) in the scenario where $\\gamma$ is unavailable.\n* The convergence rate predicted by\\ Theorem 1 is worse than that of Theorem 4 in the range $O(n^{-3/2}) \\leq \\lambda \\leq O(n^{-1})$, suggesting there may be room for improving the analysis of Theorem 1. What causes this discrepancy? Clarifying this point could be helpful for readers.\n\n**Suggestions for improving the presentation:**\n\n* It would be helpful to cite the original sources and formally state the results of the baseline methods referenced in Table 1. In its current form, it is not immediately clear that “GD” refers to the results developed in this manuscript.\n* In the last sentence of the conclusion,  [1] could be a good reference to include.\n\n[1] Cohen, J., Damian, A., Talwalkar, A., Kolter, Z., & Lee, J.D. (2024). Understanding Optimization in Deep Learning with Central Flows. ArXiv, abs/2410.24206.\n\nLIMITATIONS:\nSee my points in Weaknesses and Questions part.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-07"}