Title: Sparse Polyak: an adaptive step size rule for high-dimensional M-estimation
Venue: NeurIPS 2025
Accepted: yes
Keywords: high-dimensional statistics

non-convex optimization

adaptive step size
Abstract: We propose and study Sparse Polyak, a variant of Polyak’s adaptive step size, designed to solve high-dimensional statistical estimation problems where the problem dimension is allowed to grow much faster than the sample size. In such settings, the standard Polyak step size performs poorly, requiring an increasing number of iterations to achieve optimal statistical precision-even when, the problem remains well conditioned and/or the achievable precision itself does not degrade with problem size. We trace this limitation to a mismatch in how smoothness is measured: in high dimensions, it is no longer effective to estimate the Lipschitz smoothness constant. Instead, it is more appropriate to estimate the smoothness restricted to specific directions relevant to the problem (restricted Lipschitz smoothnes constant). Sparse Polyak overcomes this issue by modifying the step size to estimate the restricted Lipschitz smoothness constant. We support our approach with both theoretical analysis and numerical experiments, demonstrating its improved performance.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes Sparse Polyak, a novel adaptive step-size rule for the Iterative Hard Thresholding algorithm, aimed at solving sparse high-dimensional M-estimation problems, particularly where the dimension $d$ significantly exceeds the sample size $n$. Traditional Polyak step sizes degrade in such settings due to poor estimation of the global Lipschitz constant. The proposed method instead estimates a restricted Lipschitz smoothness constant by focusing on significant descent directions relevant to the sparsity constraint. The authors present a thorough theoretical analysis that establishes the linear convergence of the proposed algorithm under restricted strong convexity and smoothness assumptions, a result comparable to that of IHT with an optimal fixed step size. Notably, the convergence rate only depends on the restricted condition number, which does not grow with dimension in many specific statistical problems, enabling rate invariance under high-dimensional settings. Exact recovery of the support set is established under additional SNR conditions. Experiments are conducted on both synthetic and real datasets to support the theoretical findings.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The paper provides nontrivial extensions to the convergence theory of IHT with Polyak step-size in the high-dimensional regime. The main theorem and its corollaries are rigorously proved.\n2. The experiments are carefully designed and demonstrate the algorithm's advantage over the original Polyak step size, consistency with the optimal fixed step size, and rate invariance under high dimensions, thereby providing sufficient support for the theory.\n3. The paper connects its results to classical models, such as sparse logistic regression and low-rank matrix regression, demonstrating broad applicability.\n\nWeaknesses:\n1. Corollary 1 requires a relatively strong minimum signal condition for exact support recovery. In many practical applications, signal amplitudes are not well-separated.\n2. The paper assumes that each per-sample loss function is convex, which ensures that the empirical risk is convex. However, this assumption excludes a large class of modern M-estimation problems involving non-convex losses (e.g., Tukey’s biweight, logistic loss with non-convex regularizers, or robust statistics). In such settings, the optimization problem becomes non-convex both due to the objective and the constraint, and it is unclear whether the proposed Sparse Polyak step-size remains effective or stable. This limitation restricts the scope of the method to convex loss settings, and the authors do not discuss potential extensions or failure modes under non-convex losses, which are increasingly relevant in robust learning and signal processing applications.\n\nQUESTIONS:\n1.  Can the authors discuss whether partial or approximate support recovery guarantees could be obtained under relaxed assumptions (e.g., $\\Vert \\theta^\\star_{\\min} \\Vert$ below the SNR threshold)?\n2. Can the authors comment on whether Sparse Polyak could be effective in dense or moderately sparse settings? Will it degrade significantly?\n3. How to choose an appropriate $\\hat{f}$ in practice? When the loss function is always nonnegative, can we always set it to zero and control the iteration complexity only by stopping criteria?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper considers a high-dimensional statistical estimation problem $\\min_{\\theta} f(\\theta) = \\frac{1}{n} \\sum_{i=1}^n \\ell(z_i,\\theta)$ where sparsity of $ \\theta $ is at most $s$. They focus on the case when the dimensionality grows much faster than the sample size, i.e. $d/n \\rightarrow \\infty$. The problem is non convex even if $\\ell$ is strongly convex/smooth due to the sparsity constraint. IHT is an algorithm that has been known to efficiently find solutions to this. The algorithm repeatedly applies projected gradient descent where the projection is via a hard thresholding operator. The stepsize used is $1/\\overline{L}$ where $\\overline{L}$ is the restricted Lipschitz constant, i.e. smoothness constant corresponding to certain important directions.\n\nThe paper looks at 3 questions: 1. Do already existing approaches to tune the step size work? 2. Can they achieve the same of better guarantess by choosing an optimal step size? 3. Can they guarantee rate invariance as $d/n \\rightarrow \\infty$?\n\nThe main contribution is an adaptive step size rule that performs well in high-dimensions and maintaines rate invariance. They show simulations that prove existing approaches do not work well in high-dimensions.\n\nKey idea & algorithm\n1. Measure smoothness only along sparse directions -- The authors replace the full gradient norm in Polyak’s rule with the norm of the gradient hard-thresholded to the allowed sparsity level s, yielding an estimate of the restricted Lipschitz constant.\n2. Sparse Polyak step size  followed by one projected-gradient (Iterative Hard Thresholding) step .\n\nTheoretical guarantees\n1. Linear convergence to a neighbourhood whose radius matches the optimal statistical precision, at a rate $O(1/\\kappa)$ where $\\kappa $is the restricted condition number (Theorem 1) .\n2. Rate invariance: if $\\kappa$ and the achievable precision stay constant while $ d/n \\rightarrow \\infty$, the required number of iterations remains constant—matching fixed-step IHT with oracle knowledge of the restricted smoothness constant .\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: \n\n 1. First adaptive step-size rule that retains IHT’s constant-iteration property in the high-dimensional regime.\n2. General proof technique that may extend to other adaptive rules.\n\nQUESTIONS:\n1. Your theory assumes access to a lower bound on $f(\\theta^*)$. In practice you estimate it once per outer loop. How sensitive are convergence and statistical accuracy to misspecifying that bound?\n2. Convergence is linear with rate $1/\\kappa$. the restricted condition number. Have you observed cases where $\\overline{\\kappa}$ is large even though sparsity holds, and does Sparse Polyak still help relative to classical Polyak there?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a sparse adaptation of the classical Polyak stepsize rule for high-dimensional sparse estimation problems. Specifically, the classical Polyak stepsize is given by $\\gamma_t=\\frac{f(\\theta_t)-f(\\theta^\\ast)}{\\|\\nabla f(\\theta_t)\\|^2}$, and the proposed modification replaces the denominator with the squared norm of the top-$s$ hard-thresholded gradient, i.e., $\\gamma_t=\\frac{f(\\theta_t)-f(\\theta^\\ast)}{\\|\\mathrm{HT}_s(\\nabla f(\\theta_t)) \\|^2}$. This modification is intended to better align the stepsize with the underlying sparsity structure of the target solution. The paper provides a convergence theorem showing that, under this modified stepsize rule, gradient descent converges linearly to a neighborhood of the optimum. The authors also present numerical experiments to examine empirically the proposed method.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The motivation for adapting the Polyak stepsize to sparse settings is clearly explained, and the method is easy to implement.\n- Using the top-$s$ hard-thresholded gradient in the stepsize rule is a smart and intuitive adaptation that aligns well with the underlying sparse structure of the problem. It allows the stepsize to focus on informative coordinates and avoids distortion from noise in irrelevant dimensions.\n- The authors prove linear convergence for their algorithm. This provides theoretical support for the proposed method in high-dimensional settings.\n\nWeaknesses:\n- The paper does not clearly define the original Polyak stepsize rule. For readers who may not be familiar with the literature, it would be helpful to include the classical definition early on to better contextualize the proposed modification.\n- While the sparse adaptation is well-motivated, the linear convergence result is somewhat expected from the form of the update. The theoretical result, though important, may not be surprising to readers familiar with sparse optimization theory. While the authors mention that the proof technique may itself be of interest, it is not clear where the novelty lies. It would strengthen the paper if the authors\ncould more clearly highlight any innovative aspects of the analysis.\n- It would be helpful to include comparisons with other sparse-aware baselines, such as top-$k$ SGD, to better contextualize the proposed method's benefits.\n\nQUESTIONS:\n- Could the authors clarify where the novelty in the convergence proof lies?\n- Have the authors considered comparing the proposed method to other sparse optimization methods, such as top-$k$ SGD?\n- In practice, how should $s$ be chosen, especially when $s^*$ is unknown? Is there a data-driven or adaptive way to tune this parameter?\n- On line 39, “the same” appears to be repeated twice.\n- I found the sentence on lines 67–69 a bit unclear. Could the authors consider rephrasing or elaborating on its intended meaning?\n\nLIMITATIONS:\nThe work is theoretical and methodological in nature and does not appear to raise direct societal concerns.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe problem studies minimizing the average of convex losses under the constraint of having at most $s$ non-zero element. The Iterative Hard Thresholding algorithm has been found to a some desirable property called rate invariance: under commonly used statistical models, and as long as the number of samples scales as $s \\log d$ (instead of $d$ which is the usual scaling in the dense setting), the number of steps required for achieving optimal statistical precision is independent of $d$ and $n$. This however requires a step size that depends on the restricted Lipschitz smoothness (RSS) constant which is in general unknown to the algorithm. The question then is if existing methods with adaptive step size already achieve this kind of guarantee.\n\nThe paper shows empirically that using Polyak’s step size leads to an error that grows with the dimension, thereby ruling out the possibility that the rate invariance property holds in this setting.\n\nInstead, the proposed method  tries to estimate the restricted Lipschitz smoothness. The resulting algorithm is termed sparse Polyak, as it can be viewed as projected gradient descent with a modified version of Polyak’s step size, where the gradient in the denominator is replaced by its hard-thresholded version. The main theorem shows that under the standard assumptions on $f$ used in prior work, convergence is similar to when the RSS constant is known and that the rate invariance property holds. The corollaries specialize the results for specific statistical learning models. The corollaries include logistic regression and matrix regression. The authors also relax some of the conditions in order to get results for generalized linear models for which the original conditions do not hold. The paper also contains numerical experiments that show an advantage of the sparse Polyak method.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The paper makes progress on an important problem. Taking advantage of sparsity to avoid performance degrading by the dimension is highly desirable.\n-  The main result is general and applicable to many specific problems, as shown in Section 4.\n- I liked the evidence shown in Figure 6 of the experiments showing that the adaptive step-size makes more - aggressive progress because it uses local information.\n\nWeaknesses (mostly about writing and presentation):\n-  Is question (i) from the beginning of page 2 intended to be a more general question that subsumes (ii) and (iii)? If not, what does (i) ask that is not already covered by (ii) and (iii)?\n-  On my first read, I was confused by the description of problem (1) as “statistical,” given that no references to the underlying data distribution were made early on. It would help to clarify from the beginning which quantities (e.g., the RSS parameter) depend on the distribution of the $z_i$ and in what way (e.g., in expectation vs. with high probability). In the current version of the paper the first time these aspects are talked about is in the paper is in Section 4.\n- It is stated that the main theorem is almost equivalent to that of Jain et al. 2014. However the theorem there is stated in a different way (it talks about closeness in $f$ values rather than closeness of the parameters). It would be helpful to explain how to translate between the two and to clarify what the additive term $1/(80\\kappa)$ corresponds to in the guarantee from Jain et al.\n- In Lemma 1, $\\hat{f} = f(\\theta*)$ is input to the algorithm thus it is implicitly assumed that $f(\\theta*)$ is known.\n\n\nOverall I found some aspects of the presentation confusing but I am overall satisfied with the results of this paper. That said, I am not familiar with the literature, so my opinion is based mostly on the main body of the paper and a quick look at some of the cited works.\n\nQUESTIONS:\n- Does the algorithm work for any possible initialization $\\theta_0$ being an $s$-sparse vector?\n- If one wants to achieve $f(\\theta_t) - \\hat{f}$ being upper bounded by a desired value, how do they choose $t$ (this is related to the question about the main theorem in the \"Weaknesses\" paragraph)? Similar question for the Corollaries. \n- For Lemma 1, $\\|x_i\\| \\leq 1$ implicitly means that $\\Sigma$ must be upper bounded. Is there a way to relax this condition and possibly have the norm of $\\Sigma$ appear in more places in the lemma guarantee?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}