Title: Accelerated Distance-adaptive Methods for Hölder Smooth and Convex Optimization
Venue: NeurIPS 2025
Accepted: yes
Keywords: Convex optimization

Stochastic optimization

parameter-free optimization
Abstract: This paper introduces new parameter-free first-order methods for convex optimization problems in which the objective function exhibits Hölder smoothness. Inspired by the recently proposed distance-over-gradient (DOG) technique, we propose an accelerated distance-adaptive method which achieves optimal anytime convergence rates for Hölder smooth problems without requiring prior knowledge of smoothness parameters or explicit parameter tuning. Importantly, our parameter-free approach removes the necessity of specifying target accuracy in advance, addressing a significant limitation found in the universal fast gradient methods(Nesterov,2015).
We further present a parameter-free accelerated method that eliminates the need for line-search procedures and extend it to convex stochastic optimization. Preliminary experimental results highlight the effectiveness of our approach in convex nonsmooth problems and its advantages over existing parameter-free or accelerated methods.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors introduce a parameter-free optimization algorithm for convex and Hölder smooth objectives, applicable in both deterministic and stochastic settings. The proposed algorithm removes the need for prior knowledge of key problem parameters such as the distance to the optimal solution or Lipschitz constants. It dynamically adapts its step size to achieve near-optimal convergence rates. Numerical experiments demonstrate robustness to misspecification of initial distance estimates and consistent performance across a wide range of scales.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n1. Quality\nThe theoretical framework is rigorous and builds upon well-established results in convex optimization. The key innovation lies in combining acceleration with distance adaptation to achieve strong guarntees without specifying the target accuracy in advance. \n\nThe extension to stochastic optimization via the balance equation (15) circumvents the line search requirement while maintaining theoretical guarantees, though at the cost of requiring bounded domains.\n\nEmpirical evaluation demonstrates practical utility and robustness, especially regarding sensitivity to initialization. However, the experimental scope could be broader in terms of problem scales and domains.\n\n2. Clarity \nThe paper is well-structured and accessible to readers with background in optimization theory. The motivation behind replacing fixed $\\varepsilon$ with the dynamic $\\eta_k$ is clearly explained and theoretically justified.\n\n3.  Significance\nRemoving dependence on unknown problem parameters while maintaining acceleration is an interesting and useful contribution, in particularly in anytime settings, where classical stopping criteria are not easily applied. The stochastic extension fills an important gap.\n\n4. Originality\n\nThe authors propose a novel approach to parameter-free optimization by combining acceleration with dynamic distance adaptation. The key insight of using variable target levels instead of a fixed one is new and theoretically sound.\n\nIt extends previous work on parameter-free methods (e.g. DOG) in a non-trivial way by incorporating acceleration and Hölder smoothness. The line search procedure and balance equation for stochastic settings represent meaningful technical contributions.\n\n\nWeaknesses\n\n1. Quality \nThe assumption of a bounded domain (Assumption 1) in the stochastic case significantly limits applicability. Many machine learning problems are naturally posed on unbounded domains (e.g., unconstrained neural network training). The authors acknowledge this but don't provide clear guidance on relaxation strategies.\n\nThe convergence bounds contain logarithmic terms that, while claimed to be \"near-optimal,\" lack clear evidence that tighter analysis is impossible. The dependence on initial distance estimation introduces a practical parameter despite \"parameter-free\" claims.\n\nThe local Hölder smoothness assumption, while more general than global smoothness, still requires the optimization trajectory to remain in the required domain, which may not hold if $\\bar{r}$ is poorly chosen initially.\n\n2. Clarity\nThe notation becomes dense in places (e.g., the interplay between βₖ, r̄ₖ, and Aₖ in Algorithm 1), potentially hindering implementation. The line search termination criteria could be specified more precisely.\n\nThe relationship between Algorithms 1 and 2 could be better explained—particularly how the balance equation replaces line search and the computational trade-offs involved.\n\n3. Significance\nWhile theoretically sound, the practical advantages over existing adaptive methods (Adam, AdaGrad variants) in machine learning contexts remain unclear (in what practical scenarios does algorithmic complexity justify the theoretical improvements over simpler adaptive methods?)\n\nThe bounded domain requirement for stochastic optimization limits immediate applicability to modern deep learning.\n\nThe experimental evaluation, while demonstrating robustness, focuses on relatively small-scale convex problems. Large-scale nonconvex evaluations would strengthen the significance claims.\n\n4. Originality\nThe technical contributions, while solid, represent incremental advances over DOG/DADAPT rather than fundamental breakthroughs. The core distance adaptation idea predates this work, and the main novelty is in the specific combination with acceleration.\n\nQUESTIONS:\nHow would the algorithm perform if the boundedness assumption (Assumption 1) were relaxed or violated? Could adaptive domain expansion strategies maintain convergence guarantees in unbounded settings?\n\nCan you provide explicit comparisons of computational cost per iteration between your method and DOG/DADAPT? \n\nAbout the logarithmic factors in your convergence rates: are these fundamental to the distance adaptation approach, or artifacts of the current analysis that could potentially be tightened?\n\nFor the stochastic case, how does your balance equation approach compare empirically with variance-reduced methods or adaptive step-size rules like AdaGrad/Adam on unbounded problems?\n\nLIMITATIONS:\nThe bounded domain assumption for stochastic optimization significantly restricts applicability. Many practical machine learning problems (neural network training, sparse regression) naturally involve unbounded feasible sets.\n\nWhile claiming to be \"parameter-free,\" the method still requires choosing initial distance estimate $\\bar{r}$, and performance depends on this choice being reasonable ($\\bar{r}\\leq 4D_0$). The line search adds computational overhead per iteration.\n\nThe convergence bounds include logarithmic terms whose necessity is not clearly established. Compared to optimal rates, this represents a potentially improvable gap.\n\nEvaluation focuses on convex problems with moderate dimensions. Modern machine learning applications involving high-dimensional, nonconvex, unbounded problems are not addressed. The claimed robustness would be more convincing with large-scale evaluations.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents an accelerated gradient method for Hölder-smooth functions that does not require any difficult input parameters, such as the Hölder smoothness constant or the distance to a solution. The algorithm has an almost optimal (up to log factors) any-time convergence rate. The stochastic version of the algorithm relies on the assumption of a bounded domain instead of a line search.\n\nThe paper is well written, with clear explanations of how the proof works. However, I did not check the proofs in the appendix.\n\nSTRENGTHS AND WEAKNESSES:\nMy main concern is that I don't know which practical functions this method is applicable to. It seems that the authors don't know either. Softmax and least-squares are smooth problems. Matrix games can be solved using primal-dual methods in time complexity of $O(1/T)$ instead of $O(1/\\sqrt T)$.\n\n### Experiments.\n\n- **Softmax.** Did authors mean that we first fix $x=0$ and used it to define $a_i$? Otherwise, the function $f$ will become quite non-convex and I don't see why 0 will be its solution.\n\n- **Matrix game.** I find it a bit suspicious.  For the matrix game problem we can estimate easily\nall constants we want. Also acceleration is this case ceases to exist. Even good\nsubgradient algorithm should work equally well on this problem. So to me it\nlooks as either intentional not-optimal implementation of other algorithms or\nsome cherry-picking. Or then there must some clear reasons of such start\ncontrast in the performance.\n\nQUESTIONS:\n### Minor questions/typos\n\n- This paper https://arxiv.org/abs/2402.06271 may be relevant.\n\n- l.16: \"where\" is missing\n\n- l.104: Why do we introduce the Bregman distance at all?\n\n- l.166: I think error bounds in optimization community have a bit different meaning. To me that condition looks as the \"Hölder\" descent lemma.\n\n- l. 143: \"that\" is missing\n\n- l. 159 \"smooth smoothness\"\n\n- l. 254: \"t\" is missing\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper propose a novel parameter-free first-order method (called AGDA) for solving Hölder smooth composite convex optimization problems. It is proven that AGDA achieves a near-optimal convergence rate, up to logarithmic factors, without the need to specify the target accuracy in advance. Additionally, the paper eliminates the necessity of line-search procedures in AGDA and introduces AGDA-LSFM. Under bounded domain assumption, a convergence result for AGDA-LSFM is established. Preliminary experiments demonstrate the effectiveness of both AGDA and AGDA-LSFM.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\n1）The proposed AGDA is novel, combining Nesterov’s dual averaging technique with a distance-adaptive approach.\n\n2）AGDA is shown to achieve a near-optimal convergence rate for solving Hölder-smooth composite convex optimization problems without requiring prior knowledge of smoothness parameters or explicit parameter tuning.\n\n3）The effectiveness of AGDA and AGDA-LSFM is validated through numerical experiments.\n\n**Weaknesses:**\n\n1）The line search procedure used in AGDA introduces additional computational overhead per iteration, which may reduce the algorithm’s overall efficiency.\n\n2）In Line 56, the paper claims, ``In addition, we propose a line-search-free accelerated method that achieves optimal convergence rates ... \" but it lacks clarity following Theorem 3.\n\n3）The paper does not sufficiently discuss the proposed methods in comparison to the restarted variant of SGD, such as the recent results in Attia, A. and Koren, T. (How Free is Parameter-Free Stochastic Optimization?) and Khaled, A. and Jin, C. (Tuning-Free Stochastic Optimization).\n\n4）Although the numerical results are interesting, the paper lacks further experimental results on large-scale model training, which would provide valuable insights.\n\nQUESTIONS:\nQ1. Why introduce the general Bregman distance at the beginning of Section 2 but not use it later?\n\nQ2. Why is $\\eta_k$ chosen in the special form, differing from the earlier approach? What is the underlying insight?\n\nQ3. What is the price of the automatic initialization strategy for $\\bar{r}$ discussed in Remark 3 and Appendix?\n\nQ4. In Fig. 1 (Middle and Right), why does FGM with $\\epsilon=1$ perform better than FGM with $\\epsilon=0.01$?\n\nQ5. Why are standard statistical results not reported for the experiment in the stochastic setting?\n\nTypos:\n\n- Line 16: if $g$ is real-valued, then its domain is the whole space.\n  \n- Eq. (8): $k$ should start from $0$.\n  \n- Should $\\bar{r}$ be introduced after Eq. (4)?\n  \n- In Algorithm 1, should $z^0$ be $v^0$?\n  \n- Line 8 in Algorithm 1 and Line 147: There are two different sums in the definition of $v^{k+1}$ ? By the way, $v_{k+1}$ should be $v^{k+1}$.\n  \n- Proposition 2: It seems that $D_k$ has no definition.\n  \n- Remark 2: A reference is missing for the lower bound.\n  \n- Line 4 of Algorithm 2: $\\nabla f$ should be a stochastic gradient.\n  \n- What is $f$ for the matrix game?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces new parameter-free optimization algorithms.\nThe paper then analyzes the convergence rate for the objective function, which is convex and exhibits Hölder smoothness.\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths\n1. The paper introduces the AGDA algorithm and analyzes the convergence rate bound of this algorithm for the deterministic case. The resulting convergence rate bound is near optimal.\n2. To avoid the problems that the line search in the AGDA algorithm causes for the stochastic case, the paper introduces the AGDA LSFM algorithm, which does not contain a line search. The paper then analyzes the convergence rate bound of this algorithm for the stochastic case and achieves an in-expectation bound. In the case that the domain is bounded, has a diameter $D$, and that $\\\\| x_0-x^* \\\\|=\\Theta(D)$, then the in-expectation bound is near optimal.\n\n# Weaknesses\n1. The convergence rate bounds require that the number of steps $K$ will be $K\\ge\\Omega(\\text{log}(D/\\bar{r}))$.\n    1. In this regard, the paper only mentions this constraint in the proofs that are written in the appendix. The paper should mention this clearly in the theorems themself.\n2. As mentioned in the paper, the AGDA algorithm contains a line search and thus is inappropriate for the stochastic case.\n3. The AGDA LSFM algorithm requires a bounded domain with diameter $D$. The convergence rate bound is dependent on $D$ instead of $\\\\| x_0-x^* \\\\|$, and thus can be very large in the case that $\\\\| x_0-x^* \\\\|\\neq\\Theta(D)$.\n\nQUESTIONS:\n# Questions\n1. Lemma 7 shows that $x^i,y^i,v^i$ remain inside the ball around $x^*$. Can't Theorem 1 be proven directly from Lemma 7 instead of being proven using Proposition 2?\n\n# Suggestions\n1. In my opinion, the paper should explain more clearly why the algorithm only needs that the objective function is locally Hölder smooth. This is related to question 1.\n2. In Theorem 2, instead of using $z^K$, which is an arg-min, it would probably be better to define $k^\\*$ as in Theorem 3 and use $y^{k^\\*}$.\n3. Missing citation of [1], which introduced a parameter-free algorithm that achieves near-optimal results for both the nonsmooth and smooth cases.\n4. There are some typos in the paper:\n    1. In equation (4), you forgot to write the Hölder smoothness constant.\n    2. In line 184, you accidentally wrote $f$ inside the bound.\n    3.  In Algorithm 2, line 2, you wrote $z$ instead of $v$.\n    4. In Algorithm 2, line 4, you accidentally wrote $\\nabla$ instead of $\\tilde{\\nabla}$.\n    5. In equation (17), you wrote $z^{k^\\*}$ instead of $y^{k^\\*}$.\n\n[1] Itai Kreisler, Maor Ivgi, Oliver Hinder, and Yair Carmon. Accelerated parameter-free stochastic\noptimization. In Conference on Learning Theory (COLT), 2024.\n\nLIMITATIONS:\nyes, except what I mentioned in weakness 1.1.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-10"}