Accelerating Optimization via Differentiable Stopping Time
Authors: Zhonglin Xie
Yiman Fong
Haoran Yuan
Zaiwen Wen
Venue: NeurIPS 2025 spotlight ()
Keywords: optimization
implicit differentiation
hyperparameter tuning
differential equations
Primary area: optimization

Abstract: A common approach for accelerating optimization algorithms is to minimize the loss achieved in a fixed time, which enables a differentiable framework with respect to the algorithm's hyperparameters. In contrast, the complementary objective of minimizing the time to reach a target loss is traditionally considered non-differentiable. To address this limitation, we propose a differentiable discrete stopping time and theoretically justify it based on its connection to continuous differential equations. We design an efficient algorithm to compute its sensitivities, thereby enabling a new differentiable formulation for directly accelerating algorithms. We demonstrate its effectiveness in applications such as online hyperparameter tuning and learning to optimize. Our proposed methods show superior performance in comprehensive experiments across various problems, which confirms their effectiveness.

Accelerating Optimization via Differentiable Stopping Time
Zhonglin Xie
Beijing International Center for Mathematical Research Peking University zlxi zlxie@pku. edu.
Yiman Fong
Haoran Yuan School Mathematical Science
Department of Industrial Engineering
Tsinghua University
Peking University yuanhr@stu, pku. edu. cn
fangym230mails tsinghua, edu. cn
Zaiwen Wen
Beijing International Center for Mathematical Research Peking University
wenzwěpku. edu.
Abstract
A common approach for accelerating optimization algorithmss minimize the loss achieved in fixed time, which enables differentiable framework with respect the algorithm's hyperparameters. In contrast, the complementary ob- jective of minimizing the time to reach a target loss is traditionally considered non-differentiable. address this limitation, we propose differentiable discrete stopping time and theoretically justify it based on its connection to continuous differential equations. We design an efficient algorithm to compute its sensitivi- ties, thereby enabling new differentiable formulation for directly accelerating algorithms. We demonstrate effectiveness applications such as online hyper- parameter tuning and learning to optimize. Our proposed methods show superior performance comprehensive experiments across various problems, which con- firms their effectiveness.
Introduction
Optimization algorithms are fundamental to a wide range of applications, including operations research (1) the training of large language models (2), and decision-making in financial markets [3. Consequently, significant research effort has been dedicated to accelerating these algorithms. common formulation for algorithm design and tuning, prevalent in areas like hyperparameter 4 and learning to optimize (L20) (5 is to minimize the objective function value achieved after fixed number of iterations or predetermined time budget. This approach often leads to differentiable training objectives with respect to algorithmic hyperparameters, enabling gradient-based optimization of the algorithm itself.
optimization
However, this formulation does not directly optimize the number of iterations required to reach desired performance level target loss, which is often the practical goal in deployment. This complementary objective, minimizing the time to reach a target loss, is traditionally perceived as non-differentiable with respect to algorithm parameters, stopping time is typically integer-valued
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

or non-smooth function of parameters, addressed only conceptually or via zeroth-order optimization methods [6]
To overcome this fundamental challenge and enable the direct, gradient-based optimization of convergence speed towards a target accuracy for iterative algorithms, this paper introduces the concept of differentiable stopping time. We propose a comprehensive framework that allows for the computation of sensitivities of the number of iterations required to reach a stopping criterion with respect to algorithm parameters. Our main contributions are summarized as follows:
We formulate a new class of differentiable objectives for algorithm acceleration, aiming to directly minimize the number of iterations or computational time required to achieve a target performance. This is supported by a theoretical framework that establishes the differentiability of discrete stopping time via connection between discrete-time iterative algorithms and continuous- time dynamics, leveraging tools from the theory of continuous stopping times.
We develop a memory-efficient and scalable algorithm for computing sensitivities of discrete stopping time, enabling effective hackpropagation through iterative procedures. Our exper- imental results validate the accuracy and efficiency of the proposed method, particularly in high-dimensional settings, and show clear advantages over approaches relying on exact ordinary differential equation solvers.
We demonstrate the applicability of differentiable stopping time in practical applications, including L20 and the online adaptation of optimizer hyperparameters. These case studies show that differentiable stopping time can be seamlessly integrated into existing frameworks, and our empirical evaluations suggest that it provides principled and effective lens for understanding and improving algorithmic acceleration.
1.1 Related Work
ODE Perspective of Accelerated Methods. Offering a continuous-time view of optimization algorithms, this perspective provides both theoretical insights and practical improvements. The foundational work (7] established a connection between Nesterov's accelerated gradient method and a second-order ordinary differential equation, introducing dynamical systems viewpoint for understanding acceleration. Building on this, accelcration phenomena have been analyzed through high-resolution differential equations S revealing deeper insights into optimization dynamics. The symplectic discretization of these high-resolution ODEs [9] has also been explored, leading to practi- cal acceleration techniques with theoretical guarantees. A Lyapunov analysis of accelerated gradient methods was developed in [10, extending the framework to stochastic settings. For optimization on parametric manifolds, accelerated natural gradient descent methods have been formulated in based on the ODE perspective.
Implicit Differentiation in Deep Learning. This technique enables efficient gradient computation through complex optimization procedures. A modular framework for implicit differentiation was presented in 12], unifying existing approaches and introducing new methods for optimization prob- lems. In non-smooth settings, [33 developed a robust theory of nonsmooth implicit differentiation with applications to machine learning and optimization. Implicit differentiation has also been applied to train iterative refinement algorithms [14,, treating object representations as fixed points. For non-smooth convex learning problems, fast hyperparameter selection methods have been developed using implicit differentiation (15) Training techniques for implicit models that match or surpass traditional approaches have been explored in [16), leveraging implicit differentiation. Implicit bias in overparameterized bilevel optimization has been investigated in [17), providing insights into the behavior of implicit differentiation in high-dimensional settings. In optimal control, implicit differentiation for learning problems has been revisited in [88, where new methods for differentiating optimization-basee controllers are proposed.
Learning Optimize. This emerging paradigm leverages machine learning techniques to design optimization algorithms. A comprehensive overview of L20 methods [19 categorizes the landscape and establishes benchmarks for future research. The scalability of L.20 to large-scale optimization problems has been explored in (20), showing that learned optimizers can effectively train large neural networks. To enhance the robustness of learned optimizers, policy imitation techniques were introduced in (11. significantly improving L20 model performance. Generalization capabilities

---

have been studied by developing provable bounds for unseen optimization tasks 122] In [3] meta- learning approaches are proposed for fast adlaataapn of learned optimizers. The theoretical foundations of 1.20 have been strengthened through convergence guarantees for robust learned optimization algorithms [24]. Furthermore, 1.20 has been extended to the design of acceleration methods by leveraging an ODE perspective of optimization algorithms (25].
2 Differentiable Stopping Time: From Continuous to Discrete
We consider an iterative algorithm that arises from the discretization of an underlying continuous- time dynamical system. Let A(0, x, t) be a function that defines the instantaneous negative rate of change for the state E Rd at time parameterized by 0. The input 0 could represent, for example, parameters of a step size schedule or weights of a learnable optimizer. Given to and To E R the continuous-ting dynamics are given by the ordinary differential equation (ODE)
=(1) - A xxt), t), with initial condition a<to) z0.
The trajectory st) aims to minimize funetion f(x), and A is typically related to fax Applying the forward Euler discretization method to the ODE (T) with a time step h > 0 yields the iterative algorithm
Tk+1 = I& h.((,, Fg, tx).
(2)
where Ty is the approximation of x(22), and l to + kh. We emphasize that h serves as the discretization step for the ODE. The "effective step size" of the optimization algorithm at iteration
is h times any scaling factors embedded within A(0, 2k, 4x). We provide two simple examples of (2) as follows. This model also captures more sophisticated algorithms, such as the gradient method with momentum and LSTM- --ddeddaedddd-- learnable optimizers, illustrated in Appendix D
Figure D provides visual intuition for these concepts. Figure la illustrates how the hyperparameter 0 influences the optimization path. The solid lines are the idealized continuous trajectories from the ODE, while the dotted lines with markers show the actual discrete steps of the algorithm. The level sets of the stopping criterion are shown concentric ellipses. Changing 0 from 0.5 (red) to 2.5 (green) alters the trajectory, changing where and when the path intersects the stopping criterion. Figure Tb demonstrates a central idea of our work: the stopping time (T' for continuous. N, for discrete) is a smooth, differentiable function the hyperparameter The close alignment between the discrete stopping times (N,, transparent markers) and their continuous counterparts (T), solid lines) visually validates our ODE-based approximation and shows the key property our method exploits.
1.2
Guutiraces (Sopptng Orivetto))
T. (aak))
(Inangansa)
15
-10
Put
2-02
&
2w02
Ciatiiaana Plax IT 1.511
Tr [Custimans Floo Time)
Strpp)
0.5
Therstn Path 25. OAN (N.
30
0.5
0.5
3.5
3.0
11.5
1..0
0.4
Pammette))
(a) Trajectory vs discrete path
(b) Stopping time vs parameters
Figure 1: Illustration of the differentiable stopping time on f(11, x2) = 0.5z² + 2xz and A(z, 0, f) (i) Effect of 0 on continuous and discrete stopping time T,, N₂ for different s values.
Rescaled Gradient Flow. common instance the rescaled gradient flow, where A incorporates time-dependent and daremeter-deeeeddent scaling factor a(0, for the gradient of the objective
3

---

function f(x). In this case
A(0, t)t.. a0. a(O.t)Vf(z(t))
The ODE becomes ==t) = -a0,
The parameters 0 might define the functional form of c.g., if a0,, 010 01e (2(t-ta), then 0 = (01, 02). The effective step size for the discretized iteration Ix+1 = Ih ha(0, (x) Vf) is a(0, fz).
Learned Optimizer. Another relevant scenario involves the optimizer using parametric model, such as a neural network. Let (:; R R R + IR' denote a neural network parame- terized by weights 0. This network could learn, for instance, a diagonal preconditioning matrix 1; (xr, (z))), 1)).. Then, the function A is defined
A(0, a(), t) (x(g(((((t)) Vf(((t)), ((z((t)
(4)
The discretized update would then use this learned preconditioned gradient.
2.1 Differentiating the Continuous Stopping Time
We first give a formal definition of the continuous stopping time.
Definition 1 (Continuous Stopping Time). Given a continuously differentiable function J. for stopping criterion defined by the condition J(x) the continuous stopping time is the first time that the trajectory reaches it:
T5 Fo, e): in J(x(t)) S e, to xx()) solves I
(5)
When [(((t) never reaches the targer E, we set T, (6, Fo. E) +00.
Now, we present theorem that establishes conditions for the differentiability of the stopping time T, with respect to parameters or initial condition zg.
Theorem 1 (Differentiability of Continuous Stopping Time). stopping time such that (x())) = E. We assume that the function A(0, x, is continuously differentiable with respect I, and t. Additionally, the stopping criterion function (xx is assumed to be continuously differentiable with respect to a. Finally, it is assumed that the time derivative of the criterion function along the trajectory does not vanish at time T, that
10, be the continuous
|( -V](((T) &(T) #0.
di
Then, the solution x(t; 0, xo) of the ODE system is continuously differentiable with respect to its arguments 0 and To for t in neighborhood T. The stopping time T, vo. continuously differentiable with respect to (and zo) in neighborhood of the given (6, zo) where T, < co. Specifically, its derivatives with respect to a component 0 and Fy are given by
(u(T (T(0) OT, 00
OT, VJ(x(T)) / Ozg
(6)
V=(((T))) T(T)
VJ(x(T)) ((T)
The differentiability of rT) with respect 0 and Zo is guaranteed by the smooth dependence of solutions on initial conditions and parameters. The terms (x(T) and (Tx are sensitivities of the state at time 7 with respect to 0 and respectively, which can be obtained by solving the corresponding sensitivity equations or via adjoint methods. The proof is an application of the implicit function theorem, which is deferred to Appendix A For the differentiability under weaker conditions,
one may refer to (25] Proposition 2], which confirms the path differentiability (26 when A involves non-smooth components.
2.2 Differentiable Discrete Stopping Time: An Effective Approximation
The continuous stopping time Ty is an ideal measure of algorithm efficiency. However, backpropa- gating through it requires solving forward and backward (adjoint) differential equations numerically. This can incur significant computational overhead, and many the detailed steps evaluated by an ODE solver might be considered "wasted" compared to the coarser steps of the original iterative algorithm (2). We now return to the discrete iteration and propose an efficient approach to approximate VoT and V TT.

---

Definition 2 (Discrete Stopping Time). For a stopping criterion (z) E and the iterative algorithm the discrete stopping time the smallest integer such that J (I ) 3 E:
(5 1o. E): min(n ((xn) s,n 2 0, fxn ( satisfies 2
If J > for all n 2 0, we set NJ +00.
While NN is inherently an integer, to enable its use in gradient- optimization of or 20. we seek
meaningful way define its sensitivity to these parameters. Our approach is inspired by Theorem
I We conceptualize as a continuous variable for which the condition J (6, xo)) = holds
exactly at the stopping time. Formally differentiating this identity with respect to 8 gives
dIN êN
dIN J(ZN) J(ZN ON
dzn
0z VJ(EN)"
+ 00
+ VVNNN)
ON 00
h
00
00
Under suitable regularity assumptions, this approximation allows us to define the sensitivity of the discrete stopping time.
Definition 3 (Sensitivity of the Discrete Stopping Time). Assume that the conditions of Theorem ! hold. Let N (0, zo, E) denote the discrete stopping time. Then, the sensitivities of N with respect to and xo are defined as
0N AVJ(EN))
/ 000
AVJ(EN) ( ax aN Ozo
(8)
00
Since Definition 2 ensures - < O, the above expressions are well-defined. Beyond being a natural symbolic differentiation of the discrete stopping condition, this definition also serves as an effective approximation of the gradient of the continuous stopping time. The next theorem formalizes this connection by quantifying the approximation error between the sensitivities of the discrete stopping time and the gradients of the continuous stopping time.
Theorem 2 (Approximation Error for Gradient of Stopping Time). Let J(x) = E be stopping criterion, and let h > 0 be a time step size. Assume the discrete stopping index N, satisfies T3 E (lo + (N, - 1)h, ln + Nıh), where TT is the (continuous) stopping time and to is the initial time. Suppose that the function A is twice continuously disferentiable. We assume that regarded as a function of (0, t), has uniformly bounded W2, (Sobolev) norms with respect to (0. t) in neighborhood of 0 x [to. to + Nih,,, and that J. regarded as function of x. has a uniformly bounded W2 norm in neighborhood of x xTTr). Furthermore, suppose the boundary condition ({(((T))) (TT)) + 0 holds. Then, for sufficiently small h, the following holds
x(1), 1),
1 x0, E) VoN,(0, e)ll = O(h).
This theorem demonstrates that Definition provides an approximation to the gradient of the continuous stopping time. That is, VoN, converges to VoT, as h - 0. An analogous result holds for the gradient with respect to Zo- This result serves as a theoretical justification for using the symbolic discrete sensitivity (8) as a surrogate for the continuous counterpart. Our O(h) error bound in (9) relies the standard local error analysis of the Euler method. However,
note
connection to a non-trivial result from (25), which proves for certain forms of A that the global error || xx(x)) ]ll can converge to zero as k- oc
with a fixed, non-vanishing step size h. While proving this for our more general framework is beyond the current scope, it suggests that the approximation in (9) may be more accurate than the local analysis implies, paving the way for future work to establish stronger error bounds that do not require I + 0.
2.3 Efficient Computation of the Sensitivity
The primary challenge in X is computing the numerator term V.((NN) ( TOnn/06. If the function A and the iteration process are implemented within an automatic differentiation framework (e.g.. PyTorch, TensorFlow) where A might be learnable nn nn. Module, then the numerator can be obtained by unrolling the computation graph and applying backpropagation. However, this can be unstable and memory-intensive for large N. Other methods include finite differences or stochastic gradient estimators, which are inexact.
5

---

Alternatively, the discrete adjoint method provides a memory-efficient way to compute the re- quired vector-Jacobian products vJ(xN) (0xz /00) and vz(zx) (0IN without form- ing the Jacobians explicitly. This method involves a forward pass to compute the trajectory : followed by a backward pass that propagates adjoint (or co-state) vectors. Let ak+1 (x 0) Ik h.A(0, FT. (2) be the iterative update. Algorithm ( outlines the pro- cedure to compute the term Sg V [zx)) (ŌzN/ /00) and Seo = V (zxx)) (ŌzN /0zo). The correctness of Algorithm I is established by the following theorem. The proof is presented in
Appendix(C
Algorithm Discrete Adjoint Method for Sensitivity Components
1: Input: Forward trajectory { parameters 0, J(NN), time step h, initial time lo-
2: Output: So VJ((N)) (ONN/ /80) and Smo Va(N)) (@xx/Oma).
3: x < (=
D Initialize adjoint vector
4: Sg f- (vector of same size as 0).
Initialize sensitivity component for
5: for k = N downto 0 do
6: is < to + kh.
7: Sg + So (..
A.
A Accumulate contribution to Sg
86
8:
X [ li (.(0.z(.tt.
A.
D Propagate adjoint vector backward
Usk
9: end for
10: Sep + A.
b> After the loop, À represents [J(xN)
11: return Sg. Szo
Proposition 1 (Discrete Adjoint Method), Let the sequence 2o. EN be generated by 2k+1 2 hA(0, ægg tx). The quantities Sg and Szo computed by Algorithm are equal ( (dzN (0xx/00) /00) and V [xx) ( /dxo), respectively.
Once So and S., are computed using Algorithm I they are plugged into expression (8) This approach computes the required numerators efficiently by only requiring storage for the forward trajectory } and the current adjoint vector X, making its memory footprint O(Nd + d). which is typically much smaller than O(N x memory for graph) needed for naive unrolling. The computational cost is roughly proportional to N times the cost of evaluating A and its relevant partial derivatives (or VJPs). The overall procedure to compute N, would first run the forward pass to find N and store { then call Algorithm to get Sa, and finally assemble the components using (8).
Applications of Differentiable Stopping Time
In this section, explore two applications of the differentiable discrete stopping time: L20 and online adaptation of learning rates (or other optimizer parameters). The ability to differentiate N, allows us directly optimize for algorithmic efficiency towards target suboptimality.
3.1 L20 with Differentiable Stopping Time
In 1.20, the objective to learn an optimization algorithm, denoted by 2 parameterized by that performs efficiently across a distribution of optimization tasks. Traditional L2O approaches often aim to minimize sum of objective function values over a predetermined number of steps. While this provides a dense reward signal, may not directly optimize for the speed reach specific target precision e. To overcome this limitation, the L20 training objective can be augmented with the stopping time
F Σ
min 0
L(0) = [00)=E5DD,,oe mo
wwf(xx) AN AN,
(10)
where Dy, Dr, are distributions of and zo, respectively. Kmmx is a maximum horizon for the sum, stopping criterion depending on f, wh are weights, X is a balancing hyperparameter, and N5 zo.5) is the discrete stopping time. The parameters 0 are then updated using stochastic

---

optimization method such as stochastic gradient descent or Adam. The update follows the rule
Kina
new Gold
71.20
V
WKJ (2x))
+XVoN)(0,z0.e))
(11)
where 1/220 is the meta-learning rate. Combining these two losses contributions provides richer training signal that values both the quality the optimization path and the overall convergence speed.
Suppose (xxx > (I holds for all k, another interesting result comes from the identity
K
Kanx
vf(x)))xx ((xx))) [(xx) - ((
(xx)
(12)
Kmax
Σ k=0
00
We emphasize that the operator 0/00 directly applies to the variable A while d/de will unroll the intermediate variable and apply chain rule. The identity (12) reveals that optimizing the weighted loss sum with wı. = equals to minimize the sum of stopping times greedily with stopping criterion and natural weights ( ((rk)
3.2 Online Adaptation Optimizer Parameters via Stopping Time
Online adaptation of optimizer hyperparameters 0g (for 2K+11 G(z,, 0x)) can be triggered by an adaptive criterion =)) This criterion, potentially adaptive itself, signals when to update 0z. N is stopping time from reference Vref (last adaptation point or z0) until &o is met at Feurrent Upon meeting & at ( Feurren), the sensitivity 0N/a0 /00 of the stopping time to hyperparameters active within [Tiet, #k+11 is key. Theoretically, 0N/00 is found by backpropagating through all steps from T6+1 to Trefs yielding principled multi-step signal for adjusting
Calculating the full multi-step /2/ to Tref is often costly. Practical methods may truncate this dependency. The simplest truncation considers only the immediate impact of 0g on 2k+11- For this single-step proxy Nk, its sensitivity, given Tx+1
I - Assee A(Ox. 2b, tx) and decreasing (z). is
ON₂ 00
Ssaep (I (OA(()) Ik. (okktt)//// /08)
(13)
[(I+11) - (IK)
For Adam's learning rate og (where FK-1 Ih andi. A ardg, hasp O2 dr, and 0A/0az = da), the one-step truncated sensitivity Ss from (13)
(with J(x) = f(x)) becomes
vf(xx+1)
S f(e141) f(xx)
(14)
Se is the practical signal for adjusting og. If f( < f(xx (negative denominator), 0g is updated by
0 0% -hadaptSk:
(15)
with Radapt as the adaptation rate. Specifically, if V f((+x1)) > 0, then S₃ < 0, increasing as; V ( dg < 0, then S > 0, decreasing ag., The full procedure for Adam with Online LR Adaptation (Adam-OLA, is provided in Algorithm in the Appendix D
4 Experiments
Validation of Theorems and Proposition 1 To validate the effectiveness and efficiency of our differentiable discrete stopping time approach, we conduct experiments on a high-dimensional
quadratic optimization problem. We minimize f(z) x Qz/2, E Rd with d E {102, 10 _ 10*3 and condition number 100. The optimization algorithm uses forward Euler discretization (2) of I where A incorporates diagonal preconditioner (4) with 10d learnable parameters. The stopping
criterion is [/+[()) S with E (10-3, 3 10-1 TO T0- We compare the sensitivity of the discrete
stopping time V VeN computed using Algorithm against the gradient of the continuous stopping

---

time VoT' (ground truth), computed via torchdiffeq [27 through an adaptive ODE solver. We vary d. E, and h.
We evaluate effectiveness and efficiency using two primary metries, Relative Error quantifies the accuracy of VoN, approximation of VoTy A smaller error indicates better accuracy, with (h) magnitude expected (Theorem 2). Results are shown in Figure 2a NFE Ratio measures the computational cost efficiency, defined as the number of function evaluations (NFE) for Algorithm to compute V versus the adaptive ODE solver to compute Vol's. A ratio < 1 indicates the discrete approach's forward simulation cheaper. Results are shown Figure 2b The numbers of The math formulae of these quantities are
Euler NFE and ODE NFE are presented in Appendix D
|| VoTaIl 112
Fuler NFE
Relative Error
NFE Ratio = ODE NFE
VoT la |V
By analyzing the relative error and NFE ratio across varying parameters, our experiments demon- strate that the discrete sensitivity provides an accurate approximation while requiring substantially fewer function evaluations for the forward pass, highlighting its efficiency and suitability for high- dimensional problems compared to methods relying precise ODE solves for the stopping time gradient.
Notably, smaller stopping thresholds e also lead to lower relative error. Intuitively, smaller values of lead to longer optimization trajectories that settle closer the optimum, where the dynamics smoother and the discrete approximation becomes more accurate. However, a deeper explanation supported by the theoretical analysis in [25], which shows that || 2(tx)]l can decrease increases, even under a fixed step size. This directly explains the trend: smaller leads to larger k. which in turn reduces the discrepancy between the discrete and continuous trajectories, and thus the relative error. This interpretation is also reinforced by Figure Ta As the optimization progresses, the distance between the discrete iterates and the continuous path visibly decreases. In particular, the discrete and continuous trajectories gradually align as they approach the stopping region, further supporting the claim that gradient approximation becomes more accurate near convergence.
Problem Conigunntio direw2?? cw10-1
Problem ganiion
- dinw10 c=10-
0.030
423 cw10-1
+ dim=10" c=10"
din- 302 m-aie5
0.8
dim e-1p->
0/735
dimm10)) a-32-4
NFF Balio (Fiee ReiieFFdee (00F)
dim-10) e-10-t
dim a03, 20-d
din 10! e 10-1
Baldve Brock
dim-10f cw10
0.TM
di0m125 2=32-4
0.6
4im=10f c=10-
die cw30
dim-10f cw10
9.3
03
0.003
0,002
Step
[h)
(a) Relative Error vs Time Step Size
(b) NFE Ratio Time Step Size
Figure 2: Experimental results comparing the discrete and continuous stopping time gradients across varying problem dimensions, stopping thresholds, and time step sizes. shows the relative error of the discrete gradient approximation. (b) shows the computational cost ratio.
Learning to Optimize. We consider logistic regression problem with synthetic data
1 Σ log (1 exp(-xxup y-w, 2)),
min f(x): rER
where w; Rd denotes the i-th data sample, and y1 E {0,1] the corresponding label. We consider two 1.20 optimizers: 1.20-DM [28) and L20-RNNProp [29]. Both employ two-layer I.STM with a hidden state size 30 to predict coordinate-wise updates. The data generation process and the architecture of the L20 optimizers are detailed in Appendix D The training setup follows

---

that of [29]. Specifically, the feature dimension is set to d = 512, and the number of samples is 256. In each training step, we use a mini matchbathh consisting of 64 optimization problems. The total number of training steps is 500. In each of these steps, batch of 64 optimization problems is sampled, and the learned optimizers are unrolled for horizon of max 100 iterations to compute the training loss. We divide the sequence into 5 segments of 20 steps each and apply truncated backpropagation through time (BPTT) for training. The weights in (10) are set as wk = 1/ max- Two loss functions are considered. The first corresponds to setting À = 0 in (10), resulting in average loss across iterations. To demonstrate the benefit of incorporating the stopping time
penalty, also set À and use the stopping criterion f(z+-1) f2x) S 10 This can be reformulated into the standard form by augmenting the state variable as 2 = (Ik xk-1) and defining (() f=[dd+ 1 2d) f(=[1 d)).
The test results are summarized in Figure 3 L20-DM refers to the 1.20-DM optimizer. L20-RNN -20-RNNrrop and L20-RRNNropp-inee denote the L220-RNNroOp optimizer with and without the stopping time penalty, respectively. Since L.20-DM does not reach the stopping criterion within the maximum number of steps, we do not evaluate its performance under the stopping time penalty, For comparison with manually designed optimizers, GD represents gradient descent, NAG denotes Nesterov's accel- erated gradient method, and Adam is well-known adaptive optimizer. All classical methods use fixed step size of 1/L, where L is the Lipschitz constant of (z) estimated at the initial point øg- Our results show clear acceleration toward reaching the target stopping criterion. In Figure Bal evaluate on a problem of the same size, d = 512, n = 256. In Figure 3b) we test on a fourfold larger instance with d 2048, to meet the stopping criterion is reduced by hundreds of steps, and the learned optimizers generalize robustly to larger-scale problems.
1024. Both experiments indicate that the number of iterations required
10
107
107
20-4
30
10
20
10.2
120-BAMprop-TTn
300-bMppo---Tre
36-1
10-1
L2G-RNMprop
MAG
10
10.9
10'
20
10+
10"
10"
101
10c
10
Iterations
Iterations
Train and the size prohlems.
(b) Test 4x larger prohlems than training.
Figure 3: Test results of different optimizers on logistic regression: Function value versus iteration.
Online Learning Rate Adaptation. We tested Algorithm smooth support vector machine (SVM) problems [30], using datasets from LIBSVM BI]. HB denotes the heavy-ball method, and NAG-SC refers the Nesterov accelerated gradient method tailored for strongly convex objectives. Adagrad an adaptive gradient algorithm that scales the learning rate per coordinate based on historical gradient information. Adam-HD is an influential extension of Adam (32 in the context of online learning rate adaptation; it updates the base learning rate of Adam at each iteration using hyper-gradient technique. The remaining abbreviations retain their previously defined meanings. The results presented in Figure 2 demonstrate that Algorithm lemommetosssllllnlnnli outperforms the baseline methods, particularly in the later stages of convergence. Further comparisons across multiple datasets, as well as detailed descriptions of hyperparameter settings for the baselines, provided in Appendix D
5 Conclusion
In this work, we introduced the concept of a differentiable discrete stopping time for iterative algorithms, establishing link between continuous time dynamics and their discrete approximations. We proposed an efficient method using the discrete adjoint principle to compute the sensitivity of the discrete stopping time. Our experiments demonstrate that this approach provides an accurate

---

ALA AdaDivo
18
Adsin Ad2tv110
1071
107
1/- fmrelffar
19
12"
ALDIVOLA
AseCred
Adam
10
10"
KAGISC
1000
200
400
400
400
1100
Iterations
Iterations
(a) ala
(b) a2a
Figure 4: Comparison different optimizers on smooth SVM: Function value versus iteration. Here, Jmin denotes the minimum function value achieved across all iterations for each optimizer.
gradient approximation while requiring substantially fewer function evaluations for the forward pass compared to methods relying on continuous ODE solves, proving efficient and scalable for high-dimensional problems. This allows direct optimization of algorithms for convergence speed, with potential applications L20 and online adaptation.
However, we note that employing a forward Euler discretization with fixed time step may be too coarse for the algorithmic design. This limitation also reflected in the error bound estimated in Theorem 2 In future work, we plan to explore more tailored algorithmic designs for A alongside more sophisticated discretization schemes such as symplectic integrators or methods that incorporate higher-order information. Such approaches may enable more accurate control of the global error and allow for wider range stable time steps during discretization.
Acknowledgement
This research was supported in part by the National Natural Science Foundation of China under the grant numbers 1233 1010 and 12288101. National Key Research and Development Program of China under the grant number 2024YFA 101 2902, and the Natural Science Foundation of Beijing, China under the grant number 7230002.
References
[1] Hamdy A Taha and Hamdy Taha. Operations research: an introduction, volume Prentice hall Upper Saddle River, NJ, 1997.
[2] Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jian- Yun Nie, and Ji-Rong Wen. A survey of large language models, 2025. URL RRLhttps:: /tapp:///rzivv. www.ziivoorg/baaa.... brg/aba/
2803. 18223
[3] Keith Pilbeam. Finance and financial markets. Bloomsbury Publishing, 2018.
[4] Matthias Feurer and Frank Hutter. Hyperparameter optimization. Springer International Publishing, 2019.
[5] Xiaohan Chen, Jialin Liu, and Wotao Yin. Learning to optimize: tutorial for continuous and mixed-integer optimization. Science China Mathematics, 191-1221, 2024.
[6] Arkadij Semenovič Nemirovskij and David Borisovich Yudin. Problem complexity and method efficiency in optimization. Wiley-Interscience, 1983.
10

---

[7] Weijie Su, Stephen Boyd, and Emmanuel J Candes. A differential equation for modeling nesterov's 's accelerated gradient method: Theory and insights. Journal of Machine Learning Research, 17(1):5312-5354, 2016.
[8] Bin Shi, Simon S Du, Michael Jordan, and Weijie J Su. Understanding the acceleration phenomenon via high-resolution differential equations. Mathematical Progrumming, 194: 313-351, 2022.
[9] Bin Shi, Simon S Du, Weijie Su, and Michael Jordan. Acceleration via symplectic discretiza- tion of high-resolution differential equations. In Advances in Neural Information Processing Systems, pages 5745-5753, 2019.
[10] Mathieu Laborde and Adam Oberman. A Iyарunov analysis for accelerated gradient methods: From deterministic stochastic case. In International Conference on Artificial Intelligence and Statistics, pages 602-612. PMLR, 2020.
[11] Chenyi Li, Shuchen Zhu, Zhonglin Xie, and Zaiwen Wen. Accelerated natural gradient method for parametric manifold optimization, 2025. htttps://arxiv..rrg//////// 2504 05753
[12] Mathieu Blondel, Quentin Berthet, Marco Cuturi, Roy Frostig, Stephan Hoyer, Felipe Llinares- López, Fabian Pedregosa, and Jean-Philippe Vert. Efficient and modular implicit differentiation. In Advances in Neural Information Processing Systems, volume 35, pages 14502-145144 2022.
[13] Jérôme Bolte, Tam Le, Edouard Pauwels, and Jean- Jean-Philippe Vert. Nonsmooth implicit differen- tiation for machine-lcarning and optimization. Advances in Neural Information Processing Systems, volume 34, pages 11913-11924, 2021.
[14] Michael Chang, Thomas Griffiths, and Sergey Levine. Object representations fixed points: Training iterative refinement algorithms with implicit differentiation. In Advances in Neural Information Processing Systems, volume 35, pages (883-22849) 2022.
[15] Quentin Bertrand, Quentin Klopfenstein, Mathieu Massias, Mathicu Blondel, Gael Varoquaux, Alexandre Gramfort, and Joseph Salmon. Implicit differentiation for fast hyperparameter selection in non-smooth convex learning. Journal of Machine Learning Research, 23(1): 7710-7749, 2022.
[16] Zhengyang Geng, Xin- Yu Zhang, Shaoyuan Bai, Yiran Wang, and Zhouchen Lin. On training implicit models. In Advances in Neural Information Processing Systems, volume 34, pages 3575, 2021.
[17] Paul Vicol, Jonathan Lorraine, Fabian Pedregosa, On implicit bias in overparameterized bilevel optimization. In International Conference Machine Learning, pages 22137-22161. PMLR, 2022.
Pérez-Rus Ruaa and Pierre Ablin.
[18] Ming Xu, Timothy L Molloy, and Stephen Gould. Revisiting implicit differentiation for learning problems in optimal control. Advances Neural Information Processing Systems, volume 36, pages 66428 66441, 2023.
[19] Tianlong Chen, Xiaohan Chen, Wuyang Chen, Howard Heaton, Jialin Liu, Zhangyang Wang, and Wotao Yin. Learning to optimize: A primer and a benchmark. Journal of Machine Learning Research, 23:1-20, 2022.
[20] Xinyang Chen, Tianlong Chen, Yinghua Cheng, Wuyang Chen, Xiaoyang Xiao, Ziyi Lu, and Zhangyang Wang. Scalable learning to optimize: A learned optimizer can train big models. In European Conference on Computer Vision, pages 377-394. Springer, 2022.
[21] Tianlong Chen, Weiyi Zhang, Zhou Jingyang, Shiyu Wang, Wei Zhang, and Zhangyang Wang. Training stronger baselines for learning optimize. In Advances in Neural Information Processing Systems, volume 33, pages 10658-10669, 2020.
[22] Jiayi Yang, Tianlong Chen, Muxin Zhu, Fengxiang He, Dacheng Tao, and Zhangyang Wang. Learning to generalize provably in learning to optimize. In International Conference on Machine Learning, pages 39496-39519 PMLR, 2023.

---

[23] Jiayi Yang. Xinyang Chen, Tianlong Chen, Zhangyang Wang, and Yingbin Liang. M-12o: Towards generalizable soaminggto-optmmizz by test-time fast adaptatatn arXiv preprint arXiv:2303.00033 2023.
[24] Qi Song, Weiyang Lin, Jingyi Wang, and Hao Xu. Towards robust learning to optimize with theoretical guarantees. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2024.
[25] Zhonglin Xie, Wotao Yin, and Zaiwen Wen. ODE-based Learning to Optimize, 2024. URL attps://arxiv.orrt org/ wtwgg////////g////ae 2406 02006
[26] Jérôme Bolte and Edouard Pauwels. Conservative set valued fields, automatic differentiation, stochastic gradient methods and deep learning. Mathematical Programming, 188:19-51, 2021. 2021.
[27] Ricky T. Q. Chen, Yulia Rubanova, Jesse Bettencourt, and David Duvenaud. Neural ordinary differential equations. In Advances in Neural Information Processing Systems, volume 31, 2018.
[28] Marcin Andrychowicz, Misha Denil, Sergio Gomez Colmenarejo, Matthew W. Hoffman, David Pfau, Tom Schaul, and Nando de Freitas. Learning to learn by gradient descent by gradient descent. In Daniel D. Lee, Masashi Sugiyama, Ulrike von Luxburg, Isabelle Guyon, and Roman Garnett, editors, Advances in Neural Information Processing Systems 29; Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain. pages 3981-3989, 2016. URL 01877688885112288844255555653833 Absrratt-tbtaatt...t..
neurips wwwpcc/ppppppeeeeeee hhaper/2016/has/
[29] Kaifeng Lv, Shunhua Jiang, and Jian Li. Learning gradient descent: Better generalization and longer horizons. In Doina Precup and Yee Whye Tch, editors, Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, volume 70 of Proceedings of Machine Learning Research, pages 2247-2255. PMLR, 2017. URL http:://proceeddng.. mlr wwtpps//o/evvv///// html
[30] Yuh-Jye Lee and I.. Mangasarian. SSVM: smooth support vector machine for classification. Comput. Optim. Appl., 20(1):5-22, 2001, doi: 10.102//A:0112213321377 URL https
//doi.org/10. 1023 A: 1011215321374
[31] Chih-Chung Chang and Chih-Jen Lin. LIBSVM: A library for support vector machines. ACM Transactions on Intelligent Systems and Technology, 2:27:1 2011, Software available
http::/www.cat csie. ntu edu. "w/""jjin/libsvw
[32] Atilim Gunes Baydin, Robert Cornish, David Cartinez-Rubboo Mark Schmidt, and Frank Wood. Online learning rate adaptation with hypergradient descent. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018. URL /ttpp:://openreview net/
forum?id=BkrsAzll
[33] Ya-Chi Chu, Wenzhi Gao, Yinyu Ye, and Madeleine Udell. Provable and practical online learning rate adaptation with hypergradient descent, 2025. URI.bttps://arra org/ / 2502. 11229
[34] Jialin Liu, Xiaohan Chen, Zhangyang Wang, Wotao Yin, and HanQin Cai. Towards constituting mathematical structures for learning to optimize. In Proceedings of the 40th International Conference Machine Learning, pages 21426-21449, 2023.
12

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Our main contributions are discussed in Section 1 as well as the abstract.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We have discussed the limitations and future work in Section 5
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
13

---

For each theorem and proposition, we clearly state the assumptions and present the corresponding proof in appendix.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The hyperparameters for reproducing the experiments are listed in Section and Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
14

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [No]
Justification:
We do not provide the code during submission stage.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
//nips //ipss.cc/pill wttpcccpppppppppeeeeeeeeeeeeeeee
httaa//gguudeee/eeeeabiiiisssss /Cc////oddiiisssseeeeeee ssiiiiiiccyyyyyyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification:
All experimental details are presented in Section and Appendix D
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We test different problems with different data and compare the proposed algorithms with multiple baselines.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for
example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
15

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (c.g. negative emror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: See Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: Yes, conform the NeurlPS code of ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
16

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We have explicitly cited the code used in Appendix D
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
17

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
[Nstification:NN]]
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
18

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy for what should or should not be described.
19

---

A Proof of Theorem 1
Proof. Consider function G(O, To. t) (x(t;00zz)) we have
By the definition of T (0,00, E).
G(O, zo. T)
G(8, zu. (, zo. c)) 0.
Computing the partial derivatives yields
aG (z(x))
8x
00
00'
and
0G
= VJ(
0t
Using the implicit function theorem, we conclude that T can be expressed locally as a continuously differentiable function f or zo. We now differentiate G with respect & and zo. which yields
d d0 (0, T))
d d0 (z(T(0,, 20, e);0, V=(x)*
Oz
0T
0
+
+TT
00
00
and
d ao,
Or
0T
0 dro Rearranging these equations leads
(x(T zo, c); VJ(x)]
+ (T)
dæa
Ozo
0xo
VJ(z) (T) OT
Ox 00 J
8T (VJ(x)) (((7)) (xx))
Ox
and hence
00
00
00
as well as
OT = [z)
8T
dx Ozo
Ja(x) ((T)
and hence
(VJ(x) ((T))
V [xx)
Ozo
Oxo"
Oro
The above equations complete the proof.
B Proof of Theorem 2
We first present a basic analysis in numerical ODEs.
Proposition 2 (Error analysis of the forward Euler method). Let f : R. x R -+ R be a function defined by (xr, () + [(x 1). Suppose the following assumptions hold
1. There exists constant Lu > 0 such that ())( f(x2,1)) S Lall<r - x2ll for all 21, F2, and t.
2. There exists a constant L, > such that (lf(x,tt) f((xtt)) s Lelti - t2l for all x, ti. and iz.
3. There exists constant M 0 such that [] 0)ll < M for all a and I.
Given an initial condition z(to) = Zo and a fixed stepsize h, we consider the sequence generated by the forward Euler method as
Fk+1 + lff tx), ts = to + kh.
Then, for any positive integer k. the error ek = 3s x((R)) satisfies
|ex|l S h
Lt
2 M
Tox (e-chi-1))
Proof. We begin by expressing the error step & +
Ck+1 Ik+1 2(tx+1) Ex + h (f(z,, (4) f + +hf(x(tt), x(te+1).
20

---

Applying Lipschitz continuity, we obtain the inequality
|ex+1ll & (1+ L + [|x((x) + (f(x(t)), x(t++1)]l
The second term on the right-hand side can be expressed in integral form as
[ + h((((42)))) -00---(((111 x((+11)l -111 [S(x(((),)) - [(((4)),
which is bounded above by
1Te (++z [f(=(t), f(x(tt),tt)) (((())))))(( dt
1/ [f(x(t), f(x()))))
Substituting the assumptions, we estimate the first integral
/
tx) - f(x((t)), s Lx
||x(t) att)))= S M MLgh?,
where the last inequality follows from the Lagrange mean value theorem, which implies that
t++1 [x((t)
Mk+1
x(tx)lldtt
|((())][t- tu ldt
plkll
[If(x(EE)))]ll tx|dt
xx-1
S
M
It tx|dt = Mh²
Similarly, for the second integral, we derive the bound as
fx+1
(
cltt
1 Lnh2
(((=())t))=f(ee f(x(t), tw) dt
s L1
It --txddt
Combining these inequalities, obtain
|cc41|| S (1 + [exxl + h
(L. + ML2).
Finally, using the initial error ep = O, we conclude that the global enror satisfies
|ex|
h 2
M
Le Tox
(eLndkk - 1).
This completes the proof.
Proof of the Theorem. The following conditions are assumed throughout our analysis. First, the function A is twice continuously differentiable, i.e., A E C². Second, A itself, together with all partial derivatives of A (such as Ze
29 A. 8²A and the gradient and Hessian of (i.e., VJ and V²J), Doat uniformly bounded by constants A, Az, Ag, Ae, Ag.a. Az.z Ar,t, As, J1, and J2, respectively. Third, we assume the boundary condition (x((x(T)) 2(T)) = 8.
For clarity and brevity, our main theorem states the regularity and boundedness assumptions using Sobolev norms; specifically, we require that A(e,, xx)t, t), regarded as function of (0, t). has uniformly bounded 2,00 norms with respect to (0.t) a neighborhood of f x [tu, to + h), and that J, regarded
function of z, has uniformly bounded W norm in a neighborhood of (TT)). In this proof, we equivalently expand these assumptions by explicitly introducing uniform bounds for A. its partial derivatives (with respect to 0, L, etc.), and for the gradient V and Hessian V2J, denoted by A, Agy Ag, A,, Ag.z, Ar,a Az,t, Ag,ts Jn. and J2, respectively. This explicit formulation purely for notational convenience in the analysis, it allows to refer directly to these quantities
21

---

in the derivations, especially during Taylor expansions and error estimates. We emphasize that these detailed bounds can be derived from the W2.00 norm boundedness assumed in the theorem statement.
Without loss of generality, we only prove the case for the 1,2 norm. We first recall the form and the definition of the derivative. They given by
V((((T)) d=(T)
VoI VoTr(0, 20.
vJ(x(T)) a(T)
daN 66
V&N = VoN,(0,20, J(NN) J(NN-I)
We consider the iteration
Tk+1 = I hA(0, tk)t).
Differentiating with respect to 0, we obtain
0ok+1
a Or
0xk
h 8
I
A(B, Tkg tx)
A(0, A(0,
0A
00 00
where the initial condition Also, we consider the flow
Ong holds.
i(t) x()
Differentiating with respect to 0, we obtain
d @x
ar(()
dt 00 = 0e ((0, - 0x A(6, ((0,,x(t) t)
(16)
00
where the initial condition a=(te) O holds. Let u(t) az(()
20 It is easy to observe that the iteration
00 above corresponds to the forward Euler method for solving the ODE
dt (tt) = de A(0, xt),
((),,
We now proceed to show that u(t), for E [to T". is bounded by some constant M > 0. Let 8 B(I) 20 A(0, xx),(((()))))) 1), and C(t) Dh A(0, xx(), (). Then we can derive that
d v = 2u u = 2u T + 2u Cu.
dt
dt
Therefore, we have the bound
d - BI + < |B|| + (1 + 2|C S Ag + (Ao 2A
dt
Applying the Gronwall inequality, for every E (Loo TJ. we obtain the following estimate
Ag (T( 1) & M.
= =vo)) S V Ao + 2Ag
(17)
Employing Proposition
2 we obtain that
||
JaN 36
Na(Nh) 80
is bounded by
h MAa Ao M(Ar, + AAxx)) + Ag. AAex)
(cALITHI-to)-1)
As
Let
M(Ar, + AAz.a) + Ao,t + AAax MAr + Ao
P/4 2
(18)
A
Noticing that Bm(t) 09
is bounded by Ag A.M according to [6 and that IT - Nhl S h,
deduce that
(rr(N))
(x(T)
< (Ag + A.
00
00
22

---

Let duN
Then we obtain the estimate
|elll I (Ao + AgM c1)h.
(19)
Similarly, by Proposition
2 we know that
IN
x(Nh)|
-
h
A +
Ap Ag
(d=(T+I-w)-1)
Let
A,
( -1)
02 2
+
(20)
Ag
Since x(t) -A(0,x(t), 1) is bounded by A and T Nbl S h, it follows that || - -x(7)]] S (A + 02)h.
Let 62 VJ(xN) (z((())) Since || is bounded by J2. we obtain the estimate
|22 S (2 + c2)
(21)
The Taylor expansion yields
[ZN) (IN-11) VJ((N-1) (xN - 1N-
(eN zN-1) V> 2N-1).
Combining this with the fact that ON - EN-1 -h (N,, and that [|V is bounded by J2, we obtain
J(NN) J(NN-1)
1 ((((EN-I)TA(B,NNN-11) $2 h.
$
h
Let
J(NN) - J(NN-1) + VJ(EN-1) A(0,NN-111 1 (((x(T))ee N(N + (T)). We have just derived
and e3 VJ(ZN-1)
les - hA
(22)
As in the previous estimate for 62, obtain
||al| < J2(A + c2)h.
(23)
Furthermore, we have
||e|| I
- A(0, (-)) + (T), - (T),
< A+(A+ + c2)h Aph.
(24)
Substituting the definitions of these error terms into the expression for VoN, we obtain
(VJ(x(T)) + T2) (Oz(T) + 61
VoN (VJ(x(T)) + ea)) (+(T) - E4) +
Recall that
VoT VJ(x(T)) T dz(T) (J(x(T))T&())
89
Comparing these two expressions and combining the estimates from 19 together with the assumptions, arrive the final estimate that
23
and
24
VOT - VoN < Rh + O(h2),
where
R R=
JJM 82
[Ji(AL+ A.
02)) +
3 A AJ202
2
+ (Ji + A.M + MJ2(A +
Here, the constants refer to those defined in completes the proof.
T8
20
and the assumptions stated earlier. This
D
23

---

C Proof of Proposition 1
Proof. We aim to compute Se, = vJ(IN)" dex for each component O2 of 8, and Sao VJ(xN) the adjoint (co-state) vectors X E Rd for i = O,...,N such that A The base case is at k = N,
20, T dzx Let (00,0)) [N vo)). We are interested Vgl. and Vx L. Define
Uzo
0J(xx) [ T day dze
drg
0J(xN) VJ(NN).
AN
(25)
deN
For < N, nN depends on Fyk through 24+1- Using the chain rule
0J(xN) (J(N)) 0
Oz
0ro+1
Oz
In terms of our adjoints, we have
Ork+1
A = X
0rk
Given #k+1 = ha(00 æR. tx), the Jacobian is recursion for the adjoints is
8m O2ea
= I (A(00m)))))
Thus, the backward
0A(0, Dk tx)
AT = N (1-h-
(26)
Ōrk
or A -11- 0.A0.ag.e Bag
Ax+1- The loop in Algorithm implements this recursion. At the beginning of iteration (loop index in algorithm, representing the step from Fy to #k+1), the variable in the algorithm holds 1 from our derivation.
Now consider the derivative with respect to a parameter 0,. (xx depends on 0g through all am for m < N where Tm is influenced by Oz. Hence,
aJ(zN)
N-1 (J(N))
Σ
(0xm+1 00,
00,
0mm-1
m=0
explicit
where (axm+1/00 (000)xxpiiitt means differentiating Tm+1 = Im while holding 3'm. fixed
hA(0. &m bm) with respect to 0,
0r 00,
OA(0, Xm, tm)
-
00,
explicit
Thus, it holds
0J(IN)
0.A(8, I'm tm)
Σ m+1
(27)
00,-
00,
The loop runs from h = N - 1 down to 0. For each k in the loop, the term added is -h( 8A10 (x)() A Summing these terms gives (VJ(zN) dey
Finally, for the sensitivity with respect to xp-
aJ(IN) arg
Ag.
After the loop in Algorithm finishes (i.e., after the iteration for k: = 0), the variable will have been updated using \ and (0 thus holding Ag
dize

---

D Details of Experiments
Algorithm 2 Adam-OLA
1: Input: Zg. (kg, f. V..
2: Params:3]] (2, stab; Thudape Cdesc-
3: mo, Uo -- -0,0; 0; k + 0; Cleurr + a0-
4: Jnef t- To: fref + f(xa);; N Nupdates O.
5: for k = 0. 0.1.... until convergence do
6: Sk [i((x))
7:
m f- m1mm + (1 - 31)g
8:
uk+1 - 320R= (1 B2)gg
m t m2+1/(1 g
9:
10:
0K+1 + ukt1/(1 (//t1)
11:
de - fx-11/ /e/l/VV+++ Ük+11 + Estab)
12:
JK - [(Ix)
Ik+1 t Ik Aqurdkk
14:
fK+1 + f(xx+1)
if fret fk+1 Edose Nupdacss then
16:
gh-41 - [f(Ik+1)
17:
ASsep -- [K+1 S
A Seep # 00
19:
S& (gk+1 / stpp
20:
acur f- Oceur Taddpp
21:
end if
22:
Frel t- 3k+1; frel t fk+1
23:
N Nupdates N Nupdates +
24:
end
25:
end for
26:
Return T2
Implementation Details. We adopt the official implementation of 33 for the online learning rate adaptation experiments, and the codebase from 34 for L20 experiments. They all follow the MIT License as specified in their respective GitHub repositories. All experiments are conducted on a workstation running Ubuntu with a 12-core Intel Xeon Platinum 8458P CPU (2.7GHz, 44 threads), one NVIDIA RTX 4090 GPU with 24GB memory, and 60GB of RAM. We note that, for both experimental setups, we have made moderate modifications to the original implementations to better align with the goals of our study. However, the focus of this work is explore the potential applications of stopping time in optimization rather than to achieve statc-of-the-arr performance across all settings, we did not perform extensive hyperparameter tuning for the stopping time-based algorithms under different configurations. This choice may explain why our method does not reach SOTA performance some scenarios.
NFEs of different solvers. Figure 3] that the NFE for an adaptive solver is mainly influenced by the stopping criterion. Since does not accept a prespecified time step size, all of the statistics remain the same for different h.
Hyperparameters of Baselines. Adagrad is adaptive gradient algorithm that adjusts learn- ing rates per coordinate based on historical gradient information. The learning rate is set 10-8 For Heavy-Ball method (HB), the mo-
3 E {10-3, $0102 10-1 1.0,, 10.0, with mentum parameter selected from the set {0.1, 0.5, 0.9, 1.03. Adam-HD is a notable variant of Adam (32). which employs hypergradient-basee scheme adaptively update the base learning rate at each iteration an online fashion. For Adam-HD HHD the hyperparameter used to update the learning rate is chosen from the set [10 10 10 10 $). All other abbreviations follow their previously defined roles within the 1,20 framework. Adam-OLA and Adam-HD are all based on the
25

---

Prokiem
Problem
- din=1tf p=10"
100
--- dim- p=10
din UP 1ird
dim 1tt
din=10t c-10-
- dim=101 p=10-
din=10f c=10
dim110 c=10
dim-10t sw1ir>
dim 10f 10nk
Edle NFE
din 1iI 1015
600
dim-10ff cw10
din-10f cw10
dim
e-10-=
4%
190
2X
5.096
11.06
01,010
C1014
0.011
2.066
0.055
01010
0,002
C.011
0.035
Time Step Sin (h)
Time Stap Stn
(a) NFE of the adaptive ODE solver.
(b) NFE of the Euler discretization.
Figure 5: NFEs of different solvers.
classical Adam, where (31,82) (0.9, 0.999) and e = 10 10-8 The initial learning rate for Adam is selected from the set {10-3, -0 ,,002,10-1 10.0, 1/LJ. L is the Lipschitz constant of V (x)) estimated at the initial point zg. The maximum number iterations set to 1000, with stopping criterion tolerance of 10-
Table Hyperparameter settings for Adam-OL OLA on different datasets. The parameter controls the learning rate adaptation magnitude, and specifies the sufficient decrease threshold for triggering learning rate update.
Dataset (Experiment)
e (Learning Rate Update)
(Descent Threshold)
ala (exp_svm)
10-2
10-5
a2a (exp_svm)
10 3
x 10
a3a (exp_svm)
x 10-5
x 10-1
w3a (exp_svm)
0.005
10-9
Formulation of the Smooth SVM. In this work, we consider the problem of binary classification using smooth variant of the SVM, where the non-smooth hinge loss replaced by its squared counterpart to enable efficient gradient-based optimization. Given dataset {[,, y:) with feature vectors x; E IRd and binary labels yi E {-1, +1}, the objective function takes the form
(ww)
Σ 1=1
maax(0, - you x;:)] + 2 |w]
2
where > O is regularization parameter. This formulation preserves the margin- maximiini behavior of the original SVM while allowing for stable and differentiable optimization. We further incorporate intercept term into the model by appending constant feature each input vector. The resulting problem is solved using first-order methods with step size determined via an estimate of the gradient's Lipschitz constant.
More Examples of Online Learning Rate Adaptation. We report the performance of Algorithm 2 and other baseline methods. Our method shows consistent improvement in the later stage of the convergence.
Data Synthetic Setting for L20. The data is synthetically generated. We first sample sparse ground truth vector x* 8d with prescribed sparsity level s. and then sample W gaxd with standard normal entries. The binary labels are generated via
1/ 1 (w) JuTx+22 202 i I....,
26

---

(fc fmmelfimn
fmallffaa
1/-
ASSPYCEA AsaCred
ASDIV
10"
AseCred
Adam AssiviiD
Adam
90"
KAGISC
200
1000
200
400
400
400
1000
Iterationn
Iterations
(a) a3a
(b) w3a
Figure 6: Comparison different optimizers on smooth SVM: Function value versus iteration. Here, Jmin denotes the minimum function value achieved across all iterations for each optimizer.
A small proportion of labels are flipped to simulate noise.
Architectures of L20 Optimizers. We now provide two examples of learned optimizers formulated within this framework, drawing from seminal works in the field. These learned optimizers typically
output a direct parameter update U3 such that Ix+1 IR + UA. To fit the continuous-tiin dynamical system framework where TA-1 Here, Wopt denotes the parameters of the learned optimizer itself, 2g are the parameters being optimized, and h is the discretization step size from the underlying ODE.
h.A(w opt Tk, tx),, we define A( ((Wopt Zk. tx.) = -U, /
Detailed L,00 Training Procedure. The training of the L20 optimizers follows the paradigm described in [29]. The goal is to learn the parameters 0 of the optimizer by minimizing the expected loss defined in Equation (0).. The training process consists of 500 steps. In each step, we sample
mini-batch of 64 distinct logistic regression problems. For each problem in the batch, we unroll the learned optimizer for K max 100 iterations, starting from random initialization zg. The loss for that problem is computed based on the trajectory {x 1100 according to Equation (TO). To manage memory and computational cost, we use Truncated Backpropagation Through Time (BPTT), dividing the 100-step trajectory into 5 segments of 20 steps each. Gradients are computed for each segment and then averaged. The final gradient for the parameters is the average of the gradients computed across all 64 problems in the mini-batch. This gradient is then used to update 0 with the Adam optimizer.
LSTM-based Optimizer. The influential work by Andrychowicz et al. (28) introduced an optimizer based on Long Short-Term Memory (LSTM) network, which we denote as Mw.... This optimizer operates coordinate-wise meaning small, shared-weight LSTM is applied to each parameter (coor- dinate) of the function (x) being optimized. For each coordinate, the LSTM takes the corresponding component of the gradient V o (z())) and its own previous state, state(t), as input to compute the parameter update component (t) Mw (=)() state(t)). The term state(t) for each coordi- nate's s LSTM, typically multi-layer LSTM (e.g.. two layers as used in the paper). consists of tuple of (cell state, hidden state) pairs for each layer, i.e., ((cr,) 1t,1), (Cr,2 hx,2)) )) for two-layer LSTM. These states allow the optimizer to accumulate information over the optimization trajectory, akin to momentum. The function then defined
A(w opt; x (1), = - -yw. (V
(28)
Here, Wopt are the learnable weights of the shared LSTM optimizer.
RNNprop Optimizer. Building on similar principles, Lv et al. 29 proposed the RNNprop optimizer. This optimizer also typically uses coordinate-wiis multi-layer LSTM (e.g.. two-layer) its core recurrent unit. Before the gradient information v)(x(t)) is fed to the RNN, it undergoes a preprocessing step, P. This preprocessing involves calculating Adam-like statistics, such as estimates of the first and second moments of the gradients, x(1) (vn((), i(()), which are then used to normalize the current gradient and provide historical context. The preprocessed features, P(V (z(()), s(t),, along with the RNN's previous state, state(t), input to the RNN, Similar
27

---

to the LSTM-optimizer described above, state(t) for each coordinate's s RNN consists of the (cell state, hidden state) tuples for each of its layers. The output of the RNN then passed through scaled hyperbolic tangent function to produce the final update (t). Let this entire update-generating function Uwaps (Vf()())), s(t), state(t)). The corresponding function is
1 Uwapp (Vf(x(t)). statee)))))))
A(w opt t)
(29)
where Uwape (-) can be more specifically written as a tanh (NN(((((((((( s(t)) state(t); Wopt)). The parameters Wopt encompass those for the preprocessing module P and the RNN, and is scaling hyperparameter.
28


Reviews:
[{"review_id": "Dq7I3aGu0s", "reviewer": "Reviewer_zF24", "summary": "This paper introduces a novel training paradigm for learning-based optimization solvers. Rather than differentiating the objective function at a fixed iteration, the authors propose to differentiate with respect to the stopping time required to reach a given accuracy. The proposed method is mathematically well-grounded and is evaluated on standard benchmark problems, where it demonstrates certain advantages over existing approaches.", "questions": "See \"Strengths And Weaknesses\"", "limitations": "See \"Strengths And Weaknesses\"", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "The paper is clearly written, and the theoretical derivations are rigorous and convincing. My questions primarily concern the experimental results:\n\n(L2O-DM Performance). All baseline L2O methods—including L2O-DM—were trained using 500 iterations, correct? If so, it is puzzling that in Figure 3, L2O-DM achieves its best performance at iteration 10 rather than near the training horizon of 500. Does this pattern also appear on the training set? Has such behavior been reported in prior literature? (To my knowledge, this is not a commonly observed phenomenon.)\n\n(L2O-RNNprop vs. L2O-RNNprop-Time). Still referring to Figure 3, the performance gap between L2O-RNNprop and L2O-RNNprop-Time appears to peak around accuracy $10^{-5}$, which is consistent with the training tolerance of L2O-RNNprop-Time. Furthermore, the performance gap narrows around iteration 500, matching its training horizon. If this interpretation is correct, it would be helpful to include a brief discussion of these observations in the paper.\n\n(Generalization Beyond Training Scope). It is interesting to note that while L2O-RNNprop-Time is trained with a target tolerance of $10^{-5}$, it generalizes well to higher-accuracy regimes. In contrast, L2O-RNNprop seems unable to improve further beyond the accuracy it achieves at 500 iterations, even when allowed more iterations. Could you elaborate on why the proposed training method leads to better generalization in this regard?", "was_revised": true, "final_justification": "The authors have addressed my concerns, so I will keep my positive feedback", "rebuttal": "We sincerely appreciate the time and effort the reviewers have dedicated to evaluating our paper. We are encouraged by the positive feedback and grateful for the thoughtful and constructive suggestions.\n\n**L2O-DM Performance**  \nThe observation is accurate. This issue is reported in several studies, including Figure 6 and Figure 8 of [1], where L2O-DM consistently fails to decrease the loss function. Similar phenomena are also observed in Figure 2 and Figure 5 of [2]. This behavior also appears in the training set, where performance is similar to that in testing: the loss decreases during the first 10 steps and then begins to increase. These works confirm the weak generalization of the L2O-DM method, which stems from the intrinsic limitation of model-free L2O approaches. Their inability to capture gradient information, causing them to rely heavily on the alignment between gradient directions and the expressive space of the model-free optimizer. When the gradient is small, the noise inherent in model-free methods may overwhelm the gradient signal, leading to divergence.\n\nThere is a minor clarification (which does not affect the argument): in Line 246, we train the L2O optimizers for 500 steps, and during training, at each step, the L2O optimizers are allowed to run for $K_{\\text{max}} = 100$ steps to obtain the training loss. We will emphasize this point in the revised manuscript.\n\n**L2O-RNNprop vs. L2O-RNNprop-Time**  \nThe interpretation is largely accurate. The peak in the performance gap indeed aligns with the accuracy threshold used to determine the stopping criterion during training. However, we would like to clarify that the training horizon of L2O-RNNprop-Time is 100 iterations. The observed narrowing of the performance gap around iteration 500 is primarily due to the loss approaching a plateau, where gradients become very small and further improvement is limited. We will include a brief discussion in the revised version to clarify this point and aid interpretation.\n\n**Generalization Beyond Training Scope**  \nThe improved generalization of L2O-RNNprop-Time can be attributed to its training objective, which shifts the optimization focus from minimizing the loss over a fixed number of steps to minimizing a differentiable stopping time under a specified target precision. This design encourages the optimizer to prioritize rapid convergence to a desired accuracy, rather than just short-term loss reduction, thereby fostering a more robust update policy that generalizes better to longer horizons and higher-accuracy regimes.\n\nPrevious studies such as Lv et al. [2] show that focusing exclusively on the final‑step loss $f(x_{K_{\\max}})$ can sometimes improve generalization by prioritizing the accuracy of the ultimate iterate. Our approach differs in that it retains the intermediate loss signals, while still incorporating an explicit focus on the final target accuracy through the stopping time penalty. This hybrid objective provides a richer training signal, helping the learned optimizer balance early progress with long term convergence, and often better exploit local convergence behaviors once a target precision is reached. We believe this complementary use of trajectory information and precision-specific emphasis contributes to the observed generalization gains.\n\n**Reference**  \n[1] T. Chen et al., \"Learning to optimize: A primer and a benchmark,\" *J. Mach. Learn. Res.*, vol. 23, no. 189, pp. 1–59, 2022.  \n[2] K. Lv, S. Jiang, and J. Li, \"Learning gradient descent: Better generalization and longer horizons,\" in *Proc. Int. Conf. Mach. Learn. (ICML)*, 2017, pp. 2247–2255."}, {"review_id": "T9ACX9td10", "reviewer": "Reviewer_Hhor", "summary": "To accelerate the convergence of iterative optimization algorithms, this paper models the dynamic of continuous steps at optimum to variable by ODE and discretizes it to recover true iterations. Rigorous derivative construction and demonstration of the approximation between discretized and continuous steps are proposed. Moreover, an explicit temporal derivative calculation method is proposed to efficiently alleviate redundant derivative calculations. The proposed method can be straightforwardly applied to the learning to optimize  (L2O) framework and to accelerate existing numerical algorithms, whose effectiveness is evaluated on synthetic optimization problems.", "questions": "1. Why should L2O insert the extra minimization of number of step? Is it feasible to directly optimize meta-learning process with the proposed differentiable stopping time?\n2. Is the improvement of proposed method limited? What is $h$ for L2O? As in the configuration in line 865, to align with proposed $h\\mathcal{A}$ framework, the L2O model should shrink its original update by $\\frac{1}{h}$. \n3. In the demonstration for Theorem 2, the boundary is made up from second-order Taylor Expansion. Is the equality trivial?\n4. Does Definition 1 require a continuous property of $J$ to hold the implicit function theory?", "limitations": "No", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Strengths:\n1. This paper is well-written and well-organized. First, the motivation is clearly presented. Second, a big picture of the discretization of ODE for continuous step size is illustrated. Then, the discretization is achieved by Euler approximation. After that, an efficient algorithm is proposed to calculate the acceleration formulation. Further, the proposed method is utilized to accelerate learning to optimize (L2O) and Adam.\n2. This paper proposes a simple but straightforward modeling method for the number of step minimization, where the derivative of step to variable is formulated to illustrate the dynamic of steps. The proposed definition of the optimal step where a small objective emerges connects the dynamic of the objective with variable to the dynamic of steps.\n3. This paper proposes an efficient and explicit derivative calculation method for each step. The method takes an induction methodology and eliminates redundant computation in back-propagation through time.\n\nWeaknesses:\n1. Experiments are not enough. First, the optimization problems in the experiments are all synthetic. Since the proposed method is not limited to solving convex optimization problems, some real-world optimization problems (even non-convex ones) will be better. Second, the presented improvement in experimental results for L2O is not significant.\n2. Figure 1 lacks explanation.", "was_revised": true, "final_justification": "My final rating is 5. This is a well-written paper. The paper proposes a theoretically guaranteed plugin component to improve the convergence of existing L2O methods, which provides a new aspect of L2O.", "rebuttal": "## Regarding the Experiments (Weaknesses 1)\n\nWe appreciate the reviewer's suggestion to include more complex, real-world, and non-convex problems.\n\n- **Inclusion of Real-World Benchmarks:** We would like to gently clarify that not all experiments are purely synthetic. The \"Online Learning Rate Adaptation\" experiments were conducted on **smooth Support Vector Machine (SVM) problems using standard benchmark datasets from LIBSVM**. We will revise our manuscript to add more results of L2O on real-world dataset, such as datasets from LIBSVM. We will also include the results on the generalized Rastrigin function, a non-convex optimiation problem [1]:\n$$\nf_q(\\boldsymbol{x})=\\frac{1}{2}\\left\\\\|\\boldsymbol{A}_q \\boldsymbol{x}-\\boldsymbol{b}_q\\right\\\\|_2^2-\\alpha \\boldsymbol{c}_q \\cos (2 \\pi \\boldsymbol{x})+\\alpha n,\n$$\nwhere $\\boldsymbol{A}_q \\in \\mathbb{R}^{n \\times n}, \\boldsymbol{b}_q \\in \\mathbb{R}^{n \\times 1}$ and $\\boldsymbol{c}_q \\in \\mathbb{R}^{n \\times 1}$ are parameters whose elements are sampled i.i.d. from $\\mathcal{N}(0,1)$.\n- **Significance of the L2O Results:** We acknowledge the reviewer’s concern on the significance of improvement, but emphasize that the key advance lies in the *qualitative change* in optimization behavior. Unlike the baseline (L2O-RNNprop), which exhibits strong initial progress but quickly stagnates due to a short-sighted policy, our method (L2O-RNNprop-Time) avoids stagnation by learning to prioritize rapid convergence to a target precision through a differentiable stopping time objective. This shift encourages the optimizer to exploit favorable local geometries and transition effectively into efficient local convergence. It leads to sustained progress and final objective values nearly an order of magnitude lower. Crucially, this training strategy enhances generalization, as seen in Figure 3b, where our method scales to problems four times larger than training, outperforming the baseline significantly. By combining intermediate loss signals with differentiable stopping time, our approach learns more robust and adaptable update rules that generalize across horizons and scales.\n\n## Regarding the Explanation of Figure 1 (Weakness 2)\n\nWe agree that the explanation of Figure 1 in the main text could be more explicit, and we appreciate the opportunity to clarify its purpose. The figure is designed to provide intuitive visual support for the core concepts introduced in Section 2.\n\n- **Figure 1a (Trajectory vs. discrete path):** This panel illustrates how the algorithm’s hyperparameter, $\\theta$, directly influences the optimization path. The solid lines represent the idealized continuous trajectories derived from the corresponding ODE, while the dotted lines with markers depict the actual discrete updates of the algorithm. The blue ellipses indicate the level sets of the stopping criterion (e.g., $\\\\|\\nabla f(x)\\\\|= c$). As shown, varying $\\theta$ alters the trajectory from 0.5 (red) to 2.5 (green), leading to different intersection points with the stopping condition, both in location and timing. This highlights the sensitivity of the solution path to hyperparameter choices.\n- **Figure 1b (Stopping time vs. parameters):** This plot captures a central idea of our work: the stopping time (denoted $T_J$ for the continuous case and $N_J$ for the discrete) is a *function* of the hyperparameter $\\theta$. Each curve corresponds to a different stopping tolerance $\\epsilon$. The close alignment between the discrete stopping times $N_J$ (shown with transparent markers) and their continuous approximations $T_J$ (solid lines) provides visual evidence for the validity of our ODE-based approximation. Importantly, the smoothness and differentiability of these curves with respect to $\\theta$ are key properties that our method exploits to enable efficient hyperparameter optimization.\n\nWe will revise the manuscript to include a more detailed walkthrough of Figure 1 within Section 2, ensuring stronger integration between the visual illustrations and the theoretical framework. This enhancement will help readers better connect the geometric intuition with the formal definitions.\n\n## Regarding the Question 1\n\nThis approach is feasible and is verified in [2], which only uses the differentiable stopping time. However, this method still relies on the ODE and thus requires heavy computational overhead, hindering its application to large-scale problems. In this paper, we find it is also feasible to use only the stopping time. However, baseline L2O methods utilize information gathered over a fixed number of steps for training—a standard paradigm.  To conduct a fair comparison with other L2O methods, we insert the extra minimization of number of step. By inserting the stopping time, we do not introduce more information than the baseline methods, especially since all information needed to differentiate the stopping time is already included in the $K_{\\max}$ steps. This confirms the effectiveness of our method.\n\n## Regarding the Question 2\n\nFor the specific L2O-RNNprop and L2O-RNNprop-Time methods, we strictly follow the setting in [3, Eq. 7], where the RNN outputs a single vector $x_{\\text{out}}$, and the update increment is:\n$$\n\\Delta \\theta_t=\\alpha \\tanh \\left({x}_{\\mathrm{out}}\\right)\n$$\n\nThis formula acts as a form of gradient clipping, bounding all effective step sizes by the preset parameter $\\alpha$. In our experiments, we set $\\alpha=0.1$.\n\nTherefore, the effective step size is inherently capped at 0.1. This is a common characteristic in L2O methods, many of which have a relatively small, built-in effective step size. As a result, our proposed method can be applied to them directly without modification while maintaining stability, ensuring performance is not hindered.\n\nA general principle for increasing efficiency is to select a step size $h$ that is as large as possible, while still ensuring the stability of both the algorithm and the backpropagation for the differentiable stopping time. This principle aligns with the conventional wisdom in optimization: a larger step size often leads to faster convergence and, consequently, less computation.\n\nFor the L2O experiments, the step size $h$ is a conceptual element of the framework, used to connect discrete-time learned optimizers to our continuous-time theory. As detailed in Appendix D, learned optimizers typically output a direct update $U_k$ such that $x_{k+1} = x_k + U_k$. To align this with our framework's formulation, $x_{k+1} = x_k - h\\mathcal{A}(\\theta, x_k, t_k)$, we define $\\mathcal{A}(\\theta,x_k,t_k) = -U_k/h$.\n\nWhen this definition of $\\mathcal{A}$ is substituted back into the discretization formula, the $h$ in the numerator and denominator cancels out, recovering the original update:\n$$\nx_{k+1} = x_k - h(-U_k/h) = x_k + U_k\n$$\nThis formulation allows the theory of differentiable stopping time to be applied without altering the behavior of existing L2O models, even though the gradient calculation for the stopping time depends on this conceptual $h$.\n\n## Regarding the Question 3\n\nThe proof technique for Theorem 2 is an application of Taylor's theorem for estimating the local error of a forward Euler scheme. However, the **key contribution lies in the implication** of this result within our framework. The $O(h)$ error bound in our theorem relies on the conventional assumption that the global error between the discrete and continuous states, $\\\\|x_N - x(T)\\\\|$, is also $O(h)$.\n\nThis is where we connect to a **highly non-trivial result** from the literature. As we cite, [2] proves that for certain forms of the optimization algorithm $\\mathcal{A}$, the global error $\\\\|x_k - x(t_k)\\\\|$ can converge to zero even with a constant, non-vanishing step size $h$. This phenomenon is also illustrated in our Figure 1, where the discrete path (dotted line) progressively aligns with the continuous trajectory.\n\nOur framework considers a more general form of $\\mathcal{A}$ than that in [2]. While a formal proof of this stronger convergence for our general case is beyond the current scope, our strong numerical results suggest these favorable properties may carry over. Investigating the precise conditions for this convergence in our broader framework is an important direction for future work. We will revise the remark follows Theorem 2 to make this distinction clearer.\n\n## Regarding the Question 4\nYes, the continuous differentiability of the function $J(x)$ is a required condition. While Definition 1 itself only defines the stopping time, the subsequent Theorem 1, which establishes the differentiability of the stopping time, relies on this property. To exclude the queer situation where $J$ is a discontinuous function and a gap in the stopping time may occur, we will revise our manuscript to explicitly state the continuity of $J$ with respect to $x$ in Definition 1.\n\n## Reference\n\n[1] T. Chen et al., \"Learning to optimize: A primer and a benchmark,\" *J. Mach. Learn. Res.*, vol. 23, no. 189, pp. 1–59, 2022.\n\n[2] Z. Xie, W. Yin, and Z. Wen, \"ODE-based Learning to Optimize,\" 2024, *arXiv:2406.02006*.\n\n[3] K. Lv, S. Jiang, and J. Li, \"Learning gradient descent: Better generalization and longer horizons,\" in *Proc. Int. Conf. Mach. Learn. (ICML)*, 2017, pp. 2247–2255."}, {"review_id": "iUJrx89YjX", "reviewer": "Reviewer_fnrf", "summary": "This paper proposes a computationally tractable way of differentiating stopping times associated with a given stopping criterion for an optimization problem. Some potential applications for the proposed differentiable stopping time are given, followed by experiments to illustrate further potential benefits of adopting it.", "questions": "7.1: Could K_max in the objective (10) be replaced with N_J? Having K_max in the summation possibly accounts for performance after the stopping criterion is achieved, does that mean the criterion is not enforced to stop the optimization run? In [29] the authors report that optimizing only for the last loss f(x_{K_max}) gives better results. In that case, would N_J still lead to better performance or would it have the same effect as f(x_{K_max})?\n7.2: On Fig 2.a, why do smaller epsilon lead to smaller relative error? There is less transient as the solution settle?\n7.3: Is there a way to quantify the impact of the relative error on the quality of the learned optimizer? Is there a range of step-sizes in which the adaptive ODE solver approach is more advantageous? That would help to compare the two approaches more fairly and get a better sense of the potential significance of the paper.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths: The idea of differentiating the stopping time is nice and could be important in applications. There is a good discussion of such potential applications, some experiments to validate the idea and the theoretical results are reasonably well presented.\nWeaknesses:\nThe presentation can be improved significantly in my view. Although the title quite accurately captures the spirit of the paper, reading the abstract and the introduction was more confusing than informative and I only started to get a better understanding of the topic and contributions of the paper when I read section 2 in the first pass. For example, the sentences in the abstract are only mildly connected. The use of the term \"dual\" is confusing, since it has different meanings in different contexts. Likewise some sentences like \"An efficient algorithm is designed to backpropagate through it\" could be clarified.\nAlso, I believe the experiments could be described a little better. I understand and agree that their point is to serve as proof-of-concept rather than provide definite evidence of achieving state-of-the-art, but some further details would be helpful to get a better sense of what is happening. In particular, the training procedure of the optimizers in the \"Learning to Optimize\" experiment could be explained in more detail.\nFor example, the description of a similar experiment in [29] is easier to understand.\nFinally, I think the results from experiment \"Validation of Theorems 2 and Proposition 1\" could be discussed in more detail. For example, as the proposed Euler method becomes more computationally advantageous than the adaptive ODE solver method, the discrepancy between them also decreases. So, an assessment of the impact of the relative error on the quality of the learned optimizer would help to compare the two approaches more fairly.\nMinor points:\n- In the main text, define W2,inf norms or at least say they are Sobolev norms.\n- Line 161. Clarify what the limit of |x_k -x(t_k)| is taken wrt. What is the connection for future improvement of (9)?\n- Perhaps consider moving Algorithm 2 to the appendix? The two-column presentation is somewhat distracting.", "was_revised": true, "final_justification": "This is a technically solid paper with an interesting core idea that could have broader impact. The authors have responded responsibly to feedback and committed to meaningful improvements. However, the evaluation limitations and questions about practical significance that motivated my original score remain largely unchanged. overall I am generally supportive of this work and happy to see it will get accepted.", "rebuttal": "Thank you for the thoughtful and constructive review. The core idea being recognized as potentially important is greatly encouraging. The detailed feedback regarding presentation and clarity is much appreciated, and each point will be addressed carefully in the revision.\n\n## Response to Identified Weaknesses\n\n**Abstract and Introduction:**\n\nThe abstract and introduction will be rewritten to present a clearer narrative. The revised version will begin by highlighting a key practical challenge, namely minimizing time to solution in optimization, which remains difficult due to the non-differentiability of stopping times. This will be followed by a precise statement of the main contribution, a theoretically grounded differentiable stopping time. The goal is to clearly convey the motivation, core idea, and downstream impact from the outset.\n\n**Terminology Clarification:**\n\nThe use of the term “dual” was intended to describe the contrast between minimizing loss within a fixed budget and minimizing the time needed to reach a target loss. To avoid ambiguity, this term will be replaced with a more precise description. Additionally, the phrase “backpropagate through it” will be clarified to indicate the use of the chain rule in computing gradients of a final objective with respect to algorithm hyperparameters, where the differentiable stopping time serves as an intermediate variable.\n\n**Training Details in the Learning to Optimize (L2O) Experiment:**\n\nAn expanded explanation of the L2O training procedure will be included in the appendix, following the structure provided in [2]. This addition will include details on the meta objective optimization from Equation (10), the mini batching strategy across different optimization problems, the use of Truncated Backpropagation Through Time (BPTT), and the method for combining gradients from the standard loss and the stopping time penalty.\n\n**Discussion of the Validation Experiment:**\n\nThe validation experiment demonstrates that the proposed method becomes more efficient relative to adaptive ODE solvers as the step size increases, at the cost of higher relative error. While the current study serves as a proof of concept for Theorem 2 and Proposition 1, the suggestion to assess the downstream impact of this relative error is insightful. The revised manuscript will include a discussion clarifying that the small relative error observed empirically suggests that the approximation is sufficiently accurate for training learned optimizers, while also emphasizing that a suitable step size is needed to be carefully selected to avoid instability when differentiating the stopping time.\n\n## Response to Minor Comments\n\n- **$W^{2,\\infty}$ Norms:** A clarification will be added to the main text indicating that these are Sobolev norms.\n- **Clarification of Line 161:** The statement regarding the limit of $\\\\|x_k - x(t_k)\\\\|$ refers to the result in [1], which shows convergence under certain conditions as $k \\to \\infty$, even with fixed step size $h$. This is relevant because it suggests the potential to strengthen the $O(h)$ error bound in Equation (9). If similar convergence guarantees could be extended to the present framework, it would indicate that the proposed gradient approximation remains accurate at larger step sizes. This connection will be made explicit in the revision.\n- **Algorithm 2 Placement:** To improve readability, Algorithm 2 will be moved to the appendix, avoiding formatting issues in the main two column layout.\n\n## Response to Question 1: Objective Formulation in L2O\n\n**Replacing $K_{\\max}$ with $N_J$:**\n\nUsing $N_J$ in place of $K_{\\max}$ within the summation in Equation (10) is indeed feasible. In fact, it is also possible to train the optimizer using only $N_J$ as the objective. This design has been explored in prior work such as ODE-based Learning to Optimize [1], where $N_J$ alone is used to guide learning. However, that method depends on solving an ODE system, which limits its scalability to large problems.\n\nIn the present work, summing up to $K_{\\max}$ was adopted to maintain consistency with standard settings in the Learning to Optimize literature, where training is typically conducted over a fixed number of iterations. This choice ensures that the training horizon and the amount of information available to the optimizer remain consistent across different methods. Importantly, the computation of $N_J$ and its gradient relies only on information from iterations $1$ through $K_{\\max}$, so including the full trajectory up to $K_{\\max}$ allows fair comparison between our approach and existing baselines. This design choice and its implications will be discussed in more detail in the revised manuscript.\n\n**Optimizing $f(x_{K_{\\max}})$ vs. $N_J$:**\n\nMinimizing $f(x_{K_{\\max}})$ provides a sparse but targeted training signal that focuses only on the final outcome. In contrast, minimizing $N_J$ offers trajectory level feedback that promotes efficiency throughout the optimization process. These objectives are complementary.\n\nWhile [2] reports that using only $f(x_{K_{\\max}})$ can improve generalization, the relative advantage of $f(x_{K_{\\max}})$ versus $N_J$ depends on the specific problem setting. When the goal is to minimize the number of iterations required to reach a given level of accuracy, $N_J$ provides a more informative and useful signal. This is often the case in classical numerical optimization problems. On the other hand, when the final performance after a fixed number of steps is the primary concern, as is often the case in machine learning applications, optimizing $f(x_{K_{\\max}})$ may be more effective.\n\nThese two training signals reflect different priorities and can be viewed as complementary. In fact, combining them, for example through $\\min_\\theta [f(x_{K_{\\max}}) + \\lambda N_J]$, can encourage both rapid convergence and low final error. The revised version of the paper will include a dedicated discussion to clarify these tradeoffs and to explain the suitability of different objectives under various practical scenarios.\n\n## Response to Question 2: Effect of Epsilon on Relative Error\n\nIntuitively, smaller values of $\\epsilon$ lead to longer optimization trajectories that settle closer to the optimum, where the dynamics are smoother and the discrete approximation becomes more accurate. However, a deeper explanation is supported by our results. As noted in the **Clarification of Line 161**, the theoretical analysis in [1] shows that $\\\\|x_k - x(t_k)\\\\|$ can decrease as $k$ increases, even under a fixed step size. This directly explains the trend observed in Figure 2: smaller $\\epsilon$ leads to larger $k$, which in turn reduces the discrepancy between the discrete and continuous trajectories, and thus the relative error.\n\nThis interpretation is also reinforced by Figure 1(a). As the optimization progresses, the distance between the discrete iterates and the continuous path visibly decreases. In particular, the discrete and continuous trajectories gradually align as they approach the stopping region, further supporting the claim that gradient approximation becomes more accurate near convergence.\n\nThese observations validate the remark made after Theorem 2 and provide additional evidence that the proposed method remains accurate even at larger step sizes. The revised version of the paper will emphasize this connection more clearly.\n\n## Response to Question 3: Relative Error and Step Size Tradeoff\n\nWe appreciate the reviewer’s insightful question. To address it, we begin by clarifying that in many L2O methods such as RNNprop, the effective step size is internally controlled. Specifically, the learned optimizer outputs a vector \\$\\Delta \\theta\\_t = \\alpha \\tanh(x\\_{\\text{out}})\\$, which inherently bounds the step size by a small constant \\$\\alpha\\$ (set to \\$0.1\\$ in our experiments). As a result, even though our method introduces a step size parameter \\$h\\$ to align the discrete updates with the continuous-time framework, this \\$h\\$ is purely technical and does not alter the effective behavior of the optimizer. In fact, it cancels out analytically when back-substituted, as detailed in Appendix D.\n\nThe reviewer's question about the relationship between gradient approximation error and optimizer quality is important. As mentioned in the response to Question 2, empirical observations indicate that the discrepancy between the discrete and continuous trajectories decreases as iterations progress, even for fixed \\$h\\$. This suggests that the error does not grow linearly with \\$h\\$ as predicted by Theorem 2, provided that the optimizer remains stable. Under this stability condition, a larger \\$h\\$ often leads to faster convergence and lower overall computational cost, without sacrificing solution quality.\n\nHowever, if the chosen \\$h\\$ is too large and causes the learned optimizer to diverge, then using an adaptive ODE solver becomes advantageous. Adaptive solvers can automatically reject unstable steps and adjust the step size to ensure convergence. This makes them a robust fallback when stability is not guaranteed. We will revise the paper to highlight this tradeoff and to make explicit the conditions under which each approach is preferable.\n\n\n## References\n\n[1] Z. Xie, W. Yin, and Z. Wen, “ODE-based Learning to Optimize,” 2024, *arXiv:2406.02006*.\n\n[2] K. Lv, S. Jiang, and J. Li, “Learning gradient descent: Better generalization and longer horizons,” in *Proc. Int. Conf. Mach. Learn. (ICML)*, 2017, pp. 2247–2255."}]

Author rebuttal:

Meta-review: To accelerate the convergence of iterative optimization algorithms, this paper models the optimization dynamic of continuous steps by ODE, computes the derivatives of stopping time with respect to learnable parameters, and proposes approximation algorithms for the practical discrete use case. The paper provides guarantee for the discrete approximation, and the proposed method can be straightforwardly applied to the learning to optimize (L2O) framework and to accelerate existing numerical algorithms. The reviewers think the method is original, and has good theoretical support and practical applications.
Decision: Accept (spotlight)