Title: Accelerating Optimization via Differentiable Stopping Time
Venue: NeurIPS 2025
Accepted: yes
Keywords: optimization

implicit differentiation

hyperparameter tuning

differential equations
Abstract: A common approach for accelerating optimization algorithms is to minimize the loss achieved in a fixed time, which enables a differentiable framework with respect to the algorithm's hyperparameters. In contrast, the complementary objective of minimizing the time to reach a target loss is traditionally considered non-differentiable. To address this limitation, we propose a differentiable discrete stopping time and theoretically justify it based on its connection to continuous differential equations. We design an efficient algorithm to compute its sensitivities, thereby enabling a new differentiable formulation for directly accelerating algorithms. We demonstrate its effectiveness in applications such as online hyperparameter tuning and learning to optimize. Our proposed methods show superior performance in comprehensive experiments across various problems, which confirms their effectiveness.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a computationally tractable way of differentiating stopping times associated with a given stopping criterion for an optimization problem. Some potential applications for the proposed differentiable stopping time are given, followed by experiments to illustrate further potential benefits of adopting it.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: The idea of differentiating the stopping time is nice and could be important in applications. There is a good discussion of such potential applications, some experiments to validate the idea and the theoretical results are reasonably well presented.\nWeaknesses:\nThe presentation can be improved significantly in my view. Although the title quite accurately captures the spirit of the paper, reading the abstract and the introduction was more confusing than informative and I only started to get a better understanding of the topic and contributions of the paper when I read section 2 in the first pass. For example, the sentences in the abstract are only mildly connected. The use of the term \"dual\" is confusing, since it has different meanings in different contexts. Likewise some sentences like \"An efficient algorithm is designed to backpropagate through it\" could be clarified.\nAlso, I believe the experiments could be described a little better. I understand and agree that their point is to serve as proof-of-concept rather than provide definite evidence of achieving state-of-the-art, but some further details would be helpful to get a better sense of what is happening. In particular, the training procedure of the optimizers in the \"Learning to Optimize\" experiment could be explained in more detail.\nFor example, the description of a similar experiment in [29] is easier to understand.\nFinally, I think the results from experiment \"Validation of Theorems 2 and Proposition 1\" could be discussed in more detail. For example, as the proposed Euler method becomes more computationally advantageous than the adaptive ODE solver method, the discrepancy between them also decreases. So, an assessment of the impact of the relative error on the quality of the learned optimizer would help to compare the two approaches more fairly.\nMinor points:\n- In the main text, define W2,inf norms or at least say they are Sobolev norms.\n- Line 161. Clarify what the limit of |x_k -x(t_k)| is taken wrt. What is the connection for future improvement of (9)?\n- Perhaps consider moving Algorithm 2 to the appendix? The two-column presentation is somewhat distracting.\n\nQUESTIONS:\n7.1: Could K_max in the objective (10) be replaced with N_J? Having K_max in the summation possibly accounts for performance after the stopping criterion is achieved, does that mean the criterion is not enforced to stop the optimization run? In [29] the authors report that optimizing only for the last loss f(x_{K_max}) gives better results. In that case, would N_J still lead to better performance or would it have the same effect as f(x_{K_max})?\n7.2: On Fig 2.a, why do smaller epsilon lead to smaller relative error? There is less transient as the solution settle?\n7.3: Is there a way to quantify the impact of the relative error on the quality of the learned optimizer? Is there a range of step-sizes in which the adaptive ODE solver approach is more advantageous? That would help to compare the two approaches more fairly and get a better sense of the potential significance of the paper.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nTo accelerate the convergence of iterative optimization algorithms, this paper models the dynamic of continuous steps at optimum to variable by ODE and discretizes it to recover true iterations. Rigorous derivative construction and demonstration of the approximation between discretized and continuous steps are proposed. Moreover, an explicit temporal derivative calculation method is proposed to efficiently alleviate redundant derivative calculations. The proposed method can be straightforwardly applied to the learning to optimize  (L2O) framework and to accelerate existing numerical algorithms, whose effectiveness is evaluated on synthetic optimization problems.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. This paper is well-written and well-organized. First, the motivation is clearly presented. Second, a big picture of the discretization of ODE for continuous step size is illustrated. Then, the discretization is achieved by Euler approximation. After that, an efficient algorithm is proposed to calculate the acceleration formulation. Further, the proposed method is utilized to accelerate learning to optimize (L2O) and Adam.\n2. This paper proposes a simple but straightforward modeling method for the number of step minimization, where the derivative of step to variable is formulated to illustrate the dynamic of steps. The proposed definition of the optimal step where a small objective emerges connects the dynamic of the objective with variable to the dynamic of steps.\n3. This paper proposes an efficient and explicit derivative calculation method for each step. The method takes an induction methodology and eliminates redundant computation in back-propagation through time.\n\nWeaknesses:\n1. Experiments are not enough. First, the optimization problems in the experiments are all synthetic. Since the proposed method is not limited to solving convex optimization problems, some real-world optimization problems (even non-convex ones) will be better. Second, the presented improvement in experimental results for L2O is not significant.\n2. Figure 1 lacks explanation.\n\nQUESTIONS:\n1. Why should L2O insert the extra minimization of number of step? Is it feasible to directly optimize meta-learning process with the proposed differentiable stopping time?\n2. Is the improvement of proposed method limited? What is $h$ for L2O? As in the configuration in line 865, to align with proposed $h\\mathcal{A}$ framework, the L2O model should shrink its original update by $\\frac{1}{h}$. \n3. In the demonstration for Theorem 2, the boundary is made up from second-order Taylor Expansion. Is the equality trivial?\n4. Does Definition 1 require a continuous property of $J$ to hold the implicit function theory?\n\nLIMITATIONS:\nNo", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a novel training paradigm for learning-based optimization solvers. Rather than differentiating the objective function at a fixed iteration, the authors propose to differentiate with respect to the stopping time required to reach a given accuracy. The proposed method is mathematically well-grounded and is evaluated on standard benchmark problems, where it demonstrates certain advantages over existing approaches.\n\nSTRENGTHS AND WEAKNESSES:\nThe paper is clearly written, and the theoretical derivations are rigorous and convincing. My questions primarily concern the experimental results:\n\n(L2O-DM Performance). All baseline L2O methods—including L2O-DM—were trained using 500 iterations, correct? If so, it is puzzling that in Figure 3, L2O-DM achieves its best performance at iteration 10 rather than near the training horizon of 500. Does this pattern also appear on the training set? Has such behavior been reported in prior literature? (To my knowledge, this is not a commonly observed phenomenon.)\n\n(L2O-RNNprop vs. L2O-RNNprop-Time). Still referring to Figure 3, the performance gap between L2O-RNNprop and L2O-RNNprop-Time appears to peak around accuracy $10^{-5}$, which is consistent with the training tolerance of L2O-RNNprop-Time. Furthermore, the performance gap narrows around iteration 500, matching its training horizon. If this interpretation is correct, it would be helpful to include a brief discussion of these observations in the paper.\n\n(Generalization Beyond Training Scope). It is interesting to note that while L2O-RNNprop-Time is trained with a target tolerance of $10^{-5}$, it generalizes well to higher-accuracy regimes. In contrast, L2O-RNNprop seems unable to improve further beyond the accuracy it achieves at 500 iterations, even when allowed more iterations. Could you elaborate on why the proposed training method leads to better generalization in this regard?\n\nQUESTIONS:\nSee \"Strengths And Weaknesses\"\n\nLIMITATIONS:\nSee \"Strengths And Weaknesses\"", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}