Stepsize anything: A unified learning rate schedule for budgeted-iteration training
Authors: Anda Tang
Yiming Dong
Yutao Zeng
zhou Xun
Zhouchen Lin
Venue: NeurIPS 2025 poster ()
Keywords: learning rate schedule
budgeted-iteration training
networks optimization
Primary area: deep_learning

Abstract: The expanding computational costs and limited resources underscore the critical need for budgeted-iteration training, which aims to achieve optimal learning within predetermined iteration budgets. While learning rate schedules fundamentally govern the performance of different networks and tasks, particularly in budgeted-iteration scenarios, their design remains largely heuristic, lacking theoretical foundations.   In addition, the optimal learning rate schedule requires extensive trial-and-error selection, making the training process inefficient. In this work, we propose the Unified Budget-Aware (UBA) schedule, a theoretically grounded learning rate schedule that consistently outperforms commonly-used schedules among diverse architectures and tasks under different constrained training budgets.  First, we bridge the gap by constructing a novel training budget-aware optimization framework, which explicitly accounts for the robustness to landscape curvature variations.  From this framework, we derive the UBA schedule, controlled by a single hyper-parameter $\varphi$ that provides a trade-off between flexibility and simplicity, eliminating the need for per-network numerical optimization.  Moreover, we establish a theoretical connection between $\varphi$ and the condition number, adding interpretation and justification to our approach.   Besides, we prove the convergence for different values of $\varphi$. We offer practical guidelines for $\varphi$ selection via theoretical analysis and empirical results.   Extensive experimental results show that UBA $\textit{consistently surpasses}$ the commonly-used schedules across diverse vision and language tasks, spanning network architectures (e.g., ResNet, OLMo) and scales, under different training-iteration budgets.

Stepsize Anything: A Unified Learning Rate Schedule for Budgeted-Iteration Training
Anda Tang' Yiming Dong Yutao Zeng²
Xun Zhou²
Zhouchen Lin
State Key Lab of General AI, School of Intelligence Science and Technology, Peking University
BByteDance Seed
Institute for Artificial Intelligence, Peking University Pazhou Laboratory (Huangpu), Guangzhou, Guangdong, China tanganda@pku. edu. cn, yimingdong https/ddonnmmoouooooooooooo ml@outoook. com, yimiinddong_mllltuloook yutao . zww.eottooo.... com, zhouxun@bytedance com, zlin@pku edu.
Abstract
The expanding computational costs and limited resources underscore the criti- cal need for budgeted-iteration training. which aims achieve optimal learning within predetermined iteration budgets. While learning rate schedules funda- mentally govern the performance of different networks and tasks, particularly in budgeted-iteration scenarios, their design remains largely heuristic, lacking theoret- ical foundations. In addition, the optimal learning rate schedule requires extensive trial-and-erro selection, making the training process inefficient. In this work, we propose the Unified Budget-Aware (UBA) schedule, a theoretically grounded learning rate schedule that consistently outperforms commonly-un schedules among diverse architectures and tasks under different constrained training budgets. First, we bridge the gap by constructing novel training budget-aware optimization framework, which explicitly accounts for the robustness landscape curvature variations. From this framework, we derive the UBA schedule, controlled by a single hyper-parameter y that provides trade-off between flexibility and simplic- ity, eliminating the need for per-network numerical optimization. Moreover, we establish a theoretical connection between ; and the condition number, adding interpretation and justification to our approach. Besides, we prove the convergence for different values of (p. We offer practical guidelines for its selection via theoreti- cal analysis and empirical results. Extensive experimental results show that UBA consistently surpasses the commonly-un language tasks, spanning network architectures (e.g., ResNet, OLMo) and scales, under different training-iteratio budgets.
schedules across diverse vision and
Introduction
Deep learning has achieved remarkable success across wide range of domains, including computer vision and natural language processing. However, despite continual advancements in hardware the training cost of neural networks has increased dramatically due to the As a result, resource constraints, including computational power, memory, energy consumption, and time budgets, emerging significant bottlenecks in the training process [4155) These challenges highlight the pressing need for budgeted training, which aims to achieve optimal model performance under fixed hardware and limited time.
technologies 45 53
growing scale of models and datasets [4] g 47 48).
Work was done during an internship ByteDance Seed. Corresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

While existing budgeted training studies broadly address resource efficiency, a critical yet under- explored direction within budgeted training is achieving the best possible model performance under strictly fixed iteration constraints. This scenario is common and practically significant where practi- tioners work under limited computational or time budgets [30, and in extreme cases, models have to be completed within few training iterations due to resource exhaustion. To formalize this specific research problem, we introduce the term sedgeted-iteration training', distinguishing it from the
broader scope of budgeted training.
Budgeted-iteration iteration training has received growing attention in research, given its significant real- world applicability. Several studies have developed relevant techniques that align with its goals. Smith et al. [43] propose the cyclical learning rate schedule (CLR), improving accuracy in fewer iterations without tuning [44]. Li et al. 30) introduce budget-award adaptations for existing learning rate schedules. Chen et al. [5] propose a novel learning rate schedule called Reflected Exponential (REX).
These approaches are primarily based on learning rate designing. Learning rate scheduling highlights key advantages: (i) plays critical role in general training of diverse neural architectures across tasks. (ii) It has demonstrated suitable and competitive in fixed training iteration budgets [20]. (iii) It is plug-and-play, requiring minimal adjustments to the underlying model architecture, which makes them easily adaptable to various deep learning frameworks. Leveraging these advantages, we adopt the learning rate design approach for sudgeted-iteratinn training.
Despite their advantages, most learning rate schedules, whether tailored for budgeted-iteration train- ing or standard training performance, are still heuristic and lack rigorous theoretical grounding. In addition, existing schedules typically rely on manually designed rules or empirical tuning. Conse- quently, selecting an optimal schedule often involves extensive trial-and-error, incurring substantial cost in terms of time and computation (34]
A natural question arises: Does there exist a theoretically grounded, unified schedule that eliminates heuristic selection while maintaining robust performance across tasks, networks, scales and training
budgets?
In this paper, we provide an affirmative answer by proposing a theoretically grounded schedule. The proposed learning rate schedule should consistently outperforms existing schedules among diverse architectures and tasks under different constrained training budgets. By doing so, it avoids choosing suitable learning rate schedule after multiple trials for network training.
To achieve this, we first bridge the gap by constructing a unified budget-aware training optimization framework, which incorporates the robustness to landscape curvature variations induced by data distribution, sampling, network architectures and optimization. The framework employs a min-max optimization approach to guarantee performance under worst-case conditions, since the minimization of learning rate is operated on an upper bound. Then we obtain numerical solutions by gradient projection methods. To eliminate the need for repeated numerical optimization when applying our method to different networks, we propose a universal parametric function that approximates numerical solutions. We nominate the resulting schedule Unified Budget-Aware (UBA) schedule. It requires tuning only single hyper-parameter ço, reducing the overhead of per-network numerical optimization. Moreover, establish a theoretical connection between op and the condition number, adding interpretation and optimization difficulty-awan convergence for different values of yp. These theoretical analysis along with empirical results offer practical guidelines for selection. We evaluate UBA through comprehensive experiments across vision and language tasks, spanning diverse architectures and iteration budgets. Specifically, for vision tasks, the UBA schedule demonstrates consistent superiority over basclines across all evaluated datasets and model scales under varying training iterations. For language tasks, we validate the effectiveness of UBA through extensive benchmarks with OLMo model(36M, 73M, 1 150M and 300M parameters). Results show that UBA achieves of-te-of-the-arr art performance across on approximately half of the benchmarks, and consistently outperforms baselines on the average scores. Moreover, our performance improvements come with negligible computational overhead, which constitutes one of its key advantages.
theoretical grounding. Besides, we prove the
Main contributions:
1.
We construct a unified budget-aware training optimization framework that inherently adapts to landscape curvature variations, enhancing training robustness of leaning rates.

---

2. We propose the Unified Budget-Aware (UBA) schedule from our constructed optimiza- tion problem, controlled by single aaper-paramttr yp that provides trade-off between flexibility and simplicity, i.e. adaptive curvature adjustment and minimal tuning cost.
3. We prove the convergence under yp and derive practical guidelines for its selection through analysis and experiments. Besides, theoretical analysis and empirical results show that is related to optimization difficulty such condition number.
4. We perform experiments and demonstrate that UBA surpasses the commonly-used schedules across diverse vision and language tasks, spanning network architectures (e.g., ResNet, OLMo) and scales, under different training-iteration budgets. Moreover, our performance improvements come with negligible computational overhead, which constitutes one of its key advantages.
2 Related work
Budgeted training Researchers face significant challenges in achieving optimal model performance under fixed hardware and limited time. To address these challenges, the concept of budgeted training has gained increasing attention, exploring techniques including computation efficiency, model compression, training stability and convergence improvement [1].. It focuses on: (i) emphasizing the allocation of resources, such as the balance between the model size and the amount data
2 6 24 28 (ii) finding optimal configurations improving performance within the given compute or time budget, such as memory efficiency and computation reduction [17 [26] 31 [8], optimization learning rate schedules [5 30 44]. batch size [17] and other weight averaging method [26] (7).
Within the context of budgeted training, a key area that remains under-explored achieving the best possible model performance within a fixed number of iterations, i.e. 'budgeted-iteration -iteteei--t training'. In this domain, learning rate scheduling based methods are particularly aligned with the objectives of nudgeted-iteration training. Smith et al.
[33 propose a new learning rate schedule, named cyclical learning rates (CLR). It improves accuracy in fewer iterations without tuning and is relevant to super- convergence phenomenon 44. Li et al. [30] introduce an alternative setting of existing learning rate schedules for budgeted training. Chen et al. [5] propose the reflected exponential schedule (REX) via a profile and sampling fashion. Zhang et al. (57] introduce the finite horizon optimization framework and apply to linear programming, which optimizes the algorithm performance under a strict iteration budget. Learning rate based approaches achieve robust and under various lengths of training iterations, which corresponds to our purpose in this work, i.e. nudgeted-iteration training [4. Besides, this approach is plug-and-play, requiring no substantial alterations to the underlying model structure, making it readily adaptable to various deep network frameworks. Therefore, we explore the proper learning rate schedule to achieve budgeted-iteratin training.
results
Learning rate schedule The learning rate plays a pivotal role in controlling the optimization process during network training. The common scheme is the step decay schedule. A typical instance decreases the learning rate by a decaying scalar 0.1 after 50% epochs and by a decaying scalar 0.01 after 75% epochs (21). Then Loshchilov et al. [32] observe that sharp decreases may prevent models from escaping local minima and propose the cosine schedule function, which is the most popular schedule for language model pretraining.
Pan et al. (36] propose Eigencurve, and demonstrate that minimax-optimal convergence can be achieved for quadratic objectives when the Hessian's eigenvalue distribution exhibits substantial skewness. Defazio et al. [I] developed the wwwreticallyggroundee framework for learning rate scheduling, establishing optimal linear decay schedules and refinements through rigorous mathemati- cal derivation. Their work additionally provides the most extensive empirical evaluation of scheduling approaches to date. Although some schedules include the CLR 43) REX [5), Varmup-Stable- -ecay (WSD) (5 and schedule from multi-power law (34), there is no consensus on the optimal choice. In addition, the detail can be found in Appendix including some works focus adaptive learning
rate methods.
Learning rate design is not only significant in general training, but also critical in andgeted-iteration training which still remains a topic of debate. Some analyses advocate for small, constant learning rates to ensure stability and convergence [12.. On the contrast, one prevailing hypothesis suggests that
3

---

large learning rates may facilitate crossing over sharp local minima in the optimization landscape [56]. Despite the lack of comprehensive theoretical explanations, range of learning rate schedules inspired by the above analyses as heuristic guidelines has been widely adopted in practice, using variable learning rates to budgeted-iteratin training [0) In this work, we explore learning rate schedule from optimization problem tailored to budgeted-iteration training, aiming to balance iteration budget constraints and generalization.
3 Budgeted-iteration training
3.1 Finite optimization under limited training iterations
To design a mme-size-fits-all learning rate schedule, we construct a robust optimization model of learning rates across varying training conditions (see more conceptual illustration in Appendix [A]. Specifically, we aim to guarantee minimal loss within constrained training iterations under the worst-case conditions, proposing a budget-iteration-awaa framework for learning rate optimization.
Definition (Finite optimization); Let f(W, D) denote the function parameterized by a given neural network with parameters W E EN on the dataset D. Let denote the data sampling the dataset D, and let F be a function class. C be the loss function. Let 17 be the learning rate at the f-th iteration, T be maximum number of training iterations, and be the current learning step. A finite optimization
min
max -1-1 fET
((f(Wr,E)),
Wt - VVV-VVE(((W..))) s.t. W
1= 0,1,2..... 1
In the optimization model the constraint represents the stochastic gradient descent process. The maximizing of L(J(WT, represents the worst-case among the training process the network f. Then it minimizes the worst-case loss within given iterations, embodying its budget-aware property. By formulating the problem
min-max optimization, we identify learning rates m) (t 2, T 1) that are resilient to the uncertainties introduced by different training configuration, uniformly throughout the optimization trajectory.
The challenge lies characterizing the f within the optimization process. f primarily determined by variations in parameter configuration, datasets characteristics, batch ordering and network architec- ture. From an optimization standpoint, we assume that the characteristics can be captured by the loss landscape of f. Therefore, we can analyze f by approximating its loss using a quadratic expansion around nearby strict local optima. Specifically, during the optimization process, the loss surface in the vicinity of the optimization trajectory be approximated by sequence of strict local optima (or at least by one optimum), denoted as (*) (k- 1,2, K, K E Z with the final optimum represented as W This approach enables us to capture the key features of the loss surface near these points. Therefore, the trajectory of optimization is impacted by the characteristics of the nearby strict local optima, since the optimization process inherently shaped by the loss landscape and the key features of the surface are captured by these optima. Consequently, we can derive the learning rate within the optimization model by the information of these nearby strict local optima.
shaped by these factors
According to the second-order necessary condition for the strict minimum Wi(k), this approx- imation is achieved through the positive semi-definite Hessian matrix H. (²) NNN. In addition, the optimization problem (I) will be programmed sequentially. Then ob- jective function of the optimization problem
can be reformulated as ((((W(())
(Wr.,) - Wr()) (E) (Wr.. - -W() 1,2 K). Given that the networks un- der consideration possess sufficient capacity to fit the data, the loss at the optimal point for different f can be made small enough. Thus the term (W)()())) <)) can be reasonably neglected in our analysis. We obtain the following sequential optimization problem.

---

2 WTx11 - - f (5) (WTE11- -
min m+7, 024Th P
max f(E)
i+1 Wt - / (5) (Wr
(
w(())
s.t.
t= Tk + 1, Tx +2, 2.... Tk+1 (A:
1,2,... , ,..K, -K-1) 1),
where Tj and Tx -
By the first constraint Wi+1 Wi mHH (E) (W, W(k)), we can obtain (see Appendix for derivation)
] |W - -
TR
Wi - W7(1k) -W/k0/ S
max 2A
II 1+Tr
(1 (1- (1-ma(tt)
(3)
where x(+) (i = 1, N) are the eigenvalues of (5) around the k-th optimum, which N denote N linearly independent eigenvectors and 8 are the coefficients corresponding to the eigenvector components. Since the term [Wr, (h( | is certain constant, the optimization process of weights WT
satisfy 0 < A (k) X (8) (F(E) A for all i, u; (k)
(i
T++1
equivalent to the least upper bound of the
(1-n) Thus, the optimization of learning 6=1+7
II
rate schedule can be formulated follow,
Ta+1
II
(1 (1- (1-m))())
min m+ N2+ PT..
max A SA SA 1=1
s.t.
7++Ta T2 Me++1 k == 1.2...... K,
E [mmn 7maxx]
To solve the constrained min- min-max problem we adopt an iterative projected gradient method that alternates between minimizing over the variables 1/e and maximizing over the parameters X, To avoid repeated numerical optimization, we fit the solutions with a parametric function. Details regarding the numerical solution and curve fitting process provided in Appendix B Then, obtain the 7/ within the interval E [1 + Tk. Tx+1l as follows
2
1 + cos
[2(--T+)-1m
2(T.11 Tx) + (k 1) 1)
T) (1/ma Mnnin)
7min:
(5)
24p + (2 - (p)
1 + cos
[2((-%))-]l + (A. 1) 2(T
where oo is the dyper-paramete controlling the variation speed of the learning rate ?/--
When setting (K > 2), UBA extends to a multi-phase formulation, which offers three potential advantages. Hierarchical optimization: by partitioning the budget-aware optimization into multiple phases, each near different local minima, it improves approximation accuracy and captures the dynamic features of the loss surface. (ii) helps escape saddle points, (iii) generalizes the single- phase method, allowing for future extensions. Notably, single-phase approach (K = 2) remains effective, as the robust budget-aware model inherently derives the optimal schedule function, as shown in Figure [(a) For consistency with common practice and to better isolate scheduling effects, this paper focuses primarily on the single-phase implementation. The multi-phase approach is also compared in Appendix
E.9 with its schedule shown in Figure [(b)
We name the proposed schedule Unified Budget-Aware (UBA) schedule for the following reasons. The robust budget-awe aware model minimizes the loss function uniformly across the optimization trajectory, resulting in stable performance. UBA provides reliable, unified choice for practitioners, eliminating the need for case-by-case baseline comparisons and delivering consistent superiority across datasets, architectures, and training budgets. Lastly, UBA can uniformly approximate the behavior of existing schedules through simple parameter adjustments.
5

---

2.8
Iam na 2el
scheddut:
25
p.0
p.25
0
25
100
125
50
1.0 125
(a)
(b)
Figure 1: Evolution of the learning rate in UBA schedule across training iterations.
3.2 Theoretical analysis
Proposition 1. The fit function 5 is the exact closed-form solution to the min-max optimization problem:
Te+1 II
2
min Mo M24
(M)
max SX
1
(k)
(1)
+
(R)
(K)
(6)
1=1+T
when the hyper-parameter yp are determined by x(a) and (k) through the relation yp 2 x and
(k
Tmax = 1, Mmin O.
The min-max model in Proposition [ represents a special case of our generalized optimization framework. We show that UBA is the exact solution to this special case optimization problem when
the learning rate scaled by A
provides a theoretical foundation for our
choice of learning rate instead of choosing it heuristically empirically, adding rigor and justification to our approach. Moreover, in this case, is linked condition number. It indicates how the learning rate is shaped by the local curvature of the model, which could guide the optimization process more effectively. The relationship between şo and condition number suggests an adaptive nature for the learning rate. In regions with sharp curvatures (large condition number), &p reduces the learning rate more rapidly to avoid overshooting, Conversely. in flatter regions, ip allows the learning rate to remain large for several iterations, facilitating faster convergence. The transformation of the learning rate trend is shown in Figure [] This is a step toward establishing a principled connection between learning rate and local loss landscape geometry along with optimization difficulty. By this special case, we generalize that is related to optimization difficulty. The empirical results support these conclusions, with further details in Section 4.3 and Appendix E.6
Proposition 2. Consider the training process within the interval t E [1 + Tk, Tet1). When the hypee--paaanete is set sufficiently close to 2, the proposed learning rate scheduling formula reduces to the cosine learning rate schedule.
foredulllgformula
By Proposition T the hyper-parameter ;p is related to the optimization difficulty. In that case, the optimization difficulty is explicitly quantified as the condition number and & 2n. where X
Furthermore, Proposition shows that when & approaches the proposed learning rate schedule converges to the standard cosine schedule. It means that, in regions where the optimization difficulty manageable (i.e., the curvature relatively flat), setting y R 2 allows the learning rate schedule to naturally reduce to the cosine form. Besides, our schedule can approximate the behavior of existing schedules (c.g.step decay, cosine annealing, cyclic schedule or Rex schedule) through simple parameter adjustments, shown in Table The detail can be found in Appendix C More importantly, it outperforms these schedules by training convergence and final accuracy, supporting results can be
found in experiment Sections 4.1 and 4.2 and Appendix E.5
Theorem Let n, = HH (W, - WW)) - -H (E) (We W(() be the stochastic curvature noise
(x
introduced by sampling at iteration t. Assume that the sampling Hessian satisfies: EEg [n.nz]]] T 1
42.5 (Amas mman)
1t000
[2(1 mx) 1/- (Te41 (T441-Tk)
o² H²H(K) for some constant a. Denote T
11
If we

---

Table 1: The adaptive simulation of existing schedules.
Schedule
Parameter adjustments
Schedule
Parameter adjustments
Cosine
x = 2
Step
4p == 0, Mmmx 0.5*,
Tk+1 Te decaying step. = 1,2,
Exponential
g 30
Cyclic
y= 2, + 1,1k T. =cyclic step ,A 1, 2, ...
Rex
y 0.8
OneCycle
yp = 2, f- 1,72 Ti = pet_start step
the loss uncertainty introduced by stochastic gradient method within the interval E [1 + Tk. Tx+1 can be bounded by two terms,
set the learning rate the proposed form (
For y > 2:
E ((F(WL,E) - - L(F(W)())
4(Tx+ - Tx) (4 -2) Tk) ]" -xpp (xxp--2A"" (k)
4(TM-1 Th.) + (gp 2)TT
Mminl (t Tk.) !0 [Wi+1 1h 112
N
[4(Tx+1 - Tk:) (pp (2-2)mii Ts)) (7)
+o²
Σ ti
(2 exp (-21 Mmnin (t (t1))
A(Tx+1 - TT: + (pp - 2) - Tk)
i=1+Th
j=1
For & <
E ((f(We.)) [(f(WT(()) [E)]
(24p + 2x -pp)
(2-p)= (t Tk 0.5)
(T++1 -6)
exp -21 (k)
Mmin (1. Tx)) 1(4) || W
(2p + 2 (p)) + 2(Tx+1
(2-4)m
(8)
N
(1(1)²² exp (-2 Mmin -i))
(2pp + 2x ppx)
(1z-1- -x)) - T3 0.5)
(2-9)
+
(2pp + 2 pT))
(2-0)
TT 0.5)
1
j=1
(T++1 Tx.
4 Experiment results
We conduct comprehensive evaluations along three dimensions: (i) Modality diversity, including vision and language tasks; (ii) Training budget, including model scales (small large) and training iterations (short to long); (iii) Ablation studies, such as parameter sensitivity and cross optimizer performance. This evaluation strategy ensures the robustness and generalization of our findings.
Our implementation strictly follows the PyTorch official API, i.e. it is inherited from the base class torch. owthh.hpp......m. certimm.r_sse scheduler LRSchedduer (Appendix EE3). It maintains full compatibility with all torch. optim optimizers. The complete production-ready code is publicly available (see Appendix E.3 for link).
key advantage of UBA lies in its critical parameter s. While an optimal & can further improve model performance, we intentionally fix its value across all tasks and architectures to ensure fair evaluation. We fix yp for SGD and Adam respectively (see learning rate variations in Figure [(a)). The rationale for these choices is systematically analyzed in ablation study [.3 Since each schedule has a different optimal learning rate in practice, to thoroughly address this concern, we conduct additional experiments to evaluate the performance of all schedules with different learning
rates in Appendix E.8
Baselines Research on budgeted-iteran iiteration training remains limited, with most existing approaches
focusing on learning rate scheduling strategies [5] 30 44]. To provide fair comparison, we
adopt several widely used and empirically effective learning rate schedules as baselines: Step(SS)
[6]. Cosine(CS) 32 Cyclical (CLR) and OneCycle (1C) 43 44 Budgeted training(BT)
mathematically equivalent linear decay) [30] and Reflected Exponential (REX) [5) Details of
these basclines provided in Appendix E.
7

---

Table 2: Validation accuracy for vision classification tasks. We present validation accuracy on vision benchmarks (e.g., CIFAR10/100 and ImageNet) using different architectures (ResNet18, ResNet34, ResNet50) under fixed epoch budgets of 25%, 50%, and 100% of the maximum training epochs.
training budget (epoch(9))
Dataset
Network
Method
(25%)
1.0(50%)
300 100%)
SS
92.71=0..243
93.45:10.422)
93.61=0.33055
CS
94.211
94.24:0.2419
95.52:#:.2433
CLR
92.13::0.2610
92.9910055459
94.92:10.388
CI
ResNet18
IC
94.23++.188.
95.03++0.1355
95.35++0.2884
BT
94.18+0.3354
94.79++04460
95.68-+0.1168
REX
94.49++0.3225
95.10++0.199 1929
95.45-+0.1350
UBA(ours)
94.57:00.281
95.26:t0.2150
95.65=+0.1589
75 (25%)
150(50%)
300(100%)
SS
71.09-0..266
75.34.10.1800
77.24110.808
CS
64.12:t0.188
69.84 09.8480
74.29=t0.5958
CLR
73.33=#0.5635
73.41::0.2835
74.74+0.224 2266
CIFARI00 ResNet34
IC
73.38=0.3222
75.04:+0.638)
77 7 2239
BT
72.75=+0.2611
75.35++0.2423
78.58
REX
73.64:0..635
75.14
18.04
UBA(ours)
74.60110.108
76.63:10.2678
78.95=10.2818
75 (25%)
150(50%)
300(100%)
SS
74.74110.136
77.24
78.56:10.6433
CS
75.84:L0.2200
77.95110.2164
19.07710.0438
CLR
73.90+1.4409
76.04+
17.84-11.30
ImageNet
ResNet50
IC
75.83++0.7792
77.74++0.5218
78 0.
BT
75.90=
77.81+0.4624
78 18.86+002214
REX
75.85 77.66.10.8853
78.66 10.2828
UBA(ours)
76.29+004031
78.26++0.1640
79.39=
4.1 Experiments for vision classification tasks
We evaluate our proposed UBA schedule on vision benchmarks (e.g.. CIFAR 10/100 and ImageNet) using different architectures (ResNet18. ResNet34, ResNet50). In addition, we independently train models using fixed epoch budgets of 25%, 50%, and 100% of the maximum training epochs, without reusing or interpolating results from longer training runs. This setup ensures that each budget setting is evaluated in isolation, thereby preserving the integrity of comparisons across low and high training budgets. Table presents the validation accuracy across different training budgets, comparing UBA against six baseline schedules (see detailed results in Appendix F.4
We find that: (i) UBA demonstrates the strongest performance across all training budgets, achieving superior results on both small-scale (CIFAR with ResNe118/34) and large-scale (ImageNet with ResNet50) benchmarks. This consistent improvement highlights the generalizability across model and dataset scales. (ii) UBA outperforms baselines not only at 100% training budget but also at 25% and 50% iteration budgets, demonstrating its budget efficiency, i.e. effectiveness in computation -onouattioo-oostaainneed scenarios. (iii)Notably, UBA shows robust performance even while the second- best schedule varies depending on both the datasets, architectures and training budgets. For practitioners seeking reliable schedules without extensive method selection, UBA provides default- strong choice. UBA eliminates the need for case-by-cass baseline comparison and delivers stable superiority and reliability regardless of datasets, architectures, training budgets and scales.
4.2 Experiments for language models
We evaluate UBA schedule on the OLMo[19 a truly open language model based on decoder-only transformer architecture, across diverse benchmarks in language model evaluation. Since large language models commonly provide multiple scales accommodate different compute constraints. We adjust both model size and training steps explore budgets. We evaluate UBA OLMo networks

---

(C4)
50-
45.0
17.5
3.2
3.3
3.50
340
37.5
45.0
52-
3.3
35.0
3.0
32.5
35.0
20
200 (8)
27.5
100
130
Tokens
Tokens (B)
Tolkens
Figure 2: Training dynamics and performance for language tasks under 150B tokens on 300M OLMo. We present the training loss, validation loss, and downstream performance on HSWAG and ARC-E, demonstrating that UBA schedule achieves superior performance.
Table 3: Performance comparison between UBA and the best-performing baseline schedules on OLMo. We report the results of top-performing baseline schedule for the corresponding benchmark and model scale. The names of the top pop-performing baseline schedules are listed below the results. Bold values indicate UBA's superiority (See overall results of each schedule Appendix E.5).
Benchmark((accuracy %)
Size
Sched.
PIQA
HSWAG
OBQA SciQ
ARC-E
ARC-C
COPA
SIQA
SOC
OTH
Avg.
Best
61.15
27.91
27.80
68.
45.26
23.41
63.00
41.45
25.18
29.86
40.92
36M
baseline
(CLR) (IC)
(REX)
(REX)
(SS)
(IC)
(SS)
(CLR)
(REX)
(IC)
(REX)
UBA
60.39
27.98
27.40
68.20
45.79
21.74
63.00
40.69
24.33
30.14
40.97
Best
62.46
30.22
28.80
72.60
47.54
26.42
66.00
41.97
26.32
29.34
42.53
73M
baseline
(REX) (REX)
(CS)
(CS)
(IC)
(IC)
(BT)
(BT)
(SS)
(REX)
(CS)
UBA
63.17
30.09
28.80
74.30
45.79
22.74
65.00
41.45
27.13
28.85
42.73
150M
Best
66.00
35.72
32.80
78.20
53.16
26.42
69.00
43.71
27.68
32.64
45.91
baseline
(CS)
(REX)
(IC)
(BT)
(IC)
(BT)
(REX)
(CLR)
(REX)
(REX)
(REX)
UBA
65.23
35.50
29.80
78.30
50.35
27.42
67.00
43.50
29.29
32.72
45.91
300M
Best
70.18
46.30
33.40
84,40
57.72
28.43
72.00
44.58
29.50
36.74
49.70
baseline
(REX) (REX)
(CS)
(REX)
(REX)
(REX)
(BT)
(CS)
(REX)
(REX)
(REX)
UBA
69.48 46.44
34.60
83,90
60.35
29.10
72.00
44.42
28.53
35.41
50.42
spanning four parameter scales: 36M, 73M, 151M, and 300M, covering normal to large-scale models. Due to space limitation, we defer details such as benchmarks introduction, experimental setting and overall results in Appendix H.5).
From Table UBA achieves state-of-the-an art performance on approximately 50% of the benchmarks across all scales while the second-best schedule varies. Furthermore, it achieves consistent superior average performance among all baselines. highlights the stable superiority and reliability of UBA. providing defaull-strong choice. Moreover, it demonstrates significant improvements in (00-73M(+1177) and ARC-E-300M (00MM(+2.63)) highlighting its ability to enhance generalization across diverse benchmarks. Besides, in Figure 2 UBA consistently achieves lower training loss and validation loss throughout the training. indicating the efficient training ability and downstream performance enhancement. Notably, while task-specific tuning of yp can further improve model performance, we intentionally fix its value across all tasks and architectures to ensure fair evaluation. Remarkably, with this universal &o setting, UBA consistently outperforms baselines diverse benchmarks, demonstrating inherent robustness to varying benchmarks and model scales.
4.3 Ablation study
Performance across different optimizers UBA originates from optimization problem under gradient descent dynamics. While modern optimizers (e.g., AdamW (33) introduce momentum and adaptive mechanisms, they maintain the fundamental property of gradient-based updates. To verify the cross-optimizer performance, we conduct ablation studies between SGD and AdamW

---

optimizers(see detailed results in Appendix(E66) SGD and AdamW optimizers, demonstrating the rrossopptmizzer robustness of UBA. This highlights its broad upplicability beyond standard gradient descent.
The results show that UBA achieves SOTAs on both
Parameter analysis of & We perform sensitivity analysis of yp E (0.25, 0.5, 1.0, 2.5, 5, 10} in equation ( which controls the variation speed of learning rate (see detailed setting. results and analyses in Appendix [.7). Figure [1 shows that Adam W prefers smaller yp while SGD requires larger yp. We attribute this phenomenon to the preconditioning effect of AdamW. As the relationship formalized in Proposition D] smaller yp is favored for low optimization difficult, while larger yp is beneficial for difficult optimization. AdamW adapts the learning rate by scaling gradients with the second moment estimate V0z, implicitly reducing the condition number the optimization landscape. Thus the schedule with smaller şo is preferred. In contrast, SGD lacks such adaptive mechanisms, thus larger yp is more suitable for this situation. This alignment between theory and experiment underscores the importance of tailoring y to the optimizer's 's characteristics. Overall, y is related generalized optimization difficulty, where optimization precondition effect, datasets distribution and network architecture are all related to optimization difficulty.
Performance across different periods Our schedule has a periodic phase-based learning rate adjustment setting, where the learning rate the k-th phase is dynamically determined by the k-th local minimum of the loss landscape. To validate our scheduling strategy, we conduct experiments by varying K (see detailed results in Appendix (E.9). The results suggests that multi-phase scheduling captures the dynamic features of the loss surface more finely, but it needs careful selection for ip, where yp reflects optimization difficulty. However, selecting optimal ş values per phase for multi-phase remains non-trivial, which motivates future work on automated landseape-aware tuning.
Conclusion
In this paper, we construct unified budget-aware training optimization framework that inherently adapts landscape curvature variations, enhancing training robustness. From this optimization frame- work, we propose the Unified Budget-Aware (UBA) schedule. Extensive experiments demonstrate UBA consistently surpasses the commonly-used schedules across diverse vision and language tasks, spanning network architectures (c.g., ResNet, OLMo) and scales, under different moining-iteratton budgets. Moreover, our performance improvements come with negligible computational overhead, which constitutes one of its key advantages.
Theoretically and empirically, we observe that the parameter y correlates with the optimization difficulty of the training process, influences the optimal choice of yp and UBA's performance. However, explicit relationship between yo and optimization difficulty remains unexplored and no established evaluation metric exists to quantify the optimization difficulty. These limitations motivate future work on optimization difficulty-award yp tuning.
Despite these open questions, the effécttiveness, implementation simplicity and ease of tuning make the UBA a practical, must-try schedule for deep learning practitioners. We hope this work could motivate more studies on learning rate scheduling.
6 Acknowledgments and disclosure of funding
Z. Lin was supported by National Key R&D Program of China (20222D0160300), the NSF China (No. 62276004) and the State Key Laboratory of General Artificial Intelligence.
10

---

References
[1] Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Cojocaru, Mérouane Debbah, Étienne Goffinet, Daniel Hesslow, Julien Launay. Quentin Malartic, et al. The Falcon series of open language models. CoRR, 2023.
[2] Eric Arazo, Diego Ortego, Paul Albert, Noel E O'Connor, and Kevin MeGuinness. How important is importance sampling for deep budgeted training? arXiv preprint arXiv:21 14283, 2021.
[3] Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. PIQA: Reasoning about physical common- sense in natural language. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 7432-7439, 2020.
[4] Tom B Brown. I Language models are few-shot learners. In Advances in Neural Information Processing Systems, pages 1877-1901, 2020.
[5] John Chen, Cameron Wolfe, and Tasos Kyrillidis. REX: Revisiting budgeted training with an improved schedule. Proceedings of Machine Learning and Systems, 4:64-76, 2022.
[6] Tianlong Chen, Yu Cheng, Zhe Gan, Jingjing Liu, and Zhangyang Wang. Data-efficient GAN training beyond (just) augmentations: A lottery ticket perspective. Advances in Neural Information Processing Systems, 34:20941 20955, 2021.
[7] Xiangning Chen, Chen Liang. Da Huang, Esteban Real, Kaiyuan Wang. Hieu Pham, Xuanyi Dong, Thang Luong, Cho-Jui Hsich, Yifeng Lu, et al. Symbolic discovery of optimization algorithms. Advances in Neural Information Processing Systems, 36, 2024.
[8] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, al. PaLM: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1-113, 2023.
[9] Christopher Clark, Kenton Lee, Ming-Wes Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. BoolQ:
Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pages
2019.
[10] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? Try ARC, the A12 reasoning challenge. arXiv preprint arXiv: 05577, 2018.
[11] Aaron Defazio, Ashok Cutkosky, Harsh Mehta, and Konstantin Mishchenko. Optimal linear decay learning rate schedules and further refinements. arXiv preprint arXiv:2310.078// 2023.
[12] Simon Du, Jason Lee, Yuandong Tian, Aarti Singh, and Barnabas Poczos. Gradient descent learns one- hidden-layer CNN: Don't be afraid of spurious local minima. In International Conference on Machine Learning, pages 1339-1348. PMLR, 2018
[13] John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of machine learning research, 12(7), 2011.
[14] Donald A. Flanders and George Shortley. Numerical determination of fundamental modes. Journal of Applied Physics, 21(12): 123-3333333,332 1951.
[15] Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding. and Jeffrey Hsu. A framework for few-shot language model evaluation. December 2023. URL htttss///eeoddo wwwp//eeoor////66eee.0..
[16] Rong Ge, Sham M Kakade, Rahul Kidambi, and Praneeth Netrapalli. The step decay schedule: A near optimal, geometrically decaying learning rate procedure for least squares. In Advances Neural Information Processing Systems, volume 32, 2019.
[17] Jonas Geiping and Tom Goldstein. Cramming: International Conference on Machine Learning, pages 11117-11143, PMLR, 2023.
Training language model single GPU in one day. In
[18] Andrew Gordon, Zornitsa Kozareva, and Melissa Roemmele. Semeval-2012 task 7: Choice of plausible alternatives: An evaluation commonsense causal reasoning. In * SEM 2012: The First Joint Conference on Lexical and Computational Gemantics-Volume I: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation (SemEval 2012). pages 394 398, 2012.

---

[19] Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord. Ananya Harsh Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang. et al. OLMo: Accelerating the science of language models, arXiv preprint urXiv:2402.0088. 00838, 2024.
[20] Alex Hägele, Elie Bakouch, Atli Kosson, Leandro Von Werra, Martin Jaggi, et al. Scaling laws and compute-optimal training beyond fixed training durations. Advances in Neural Information Processing Systems, 77:76232-76264, 2024.
[21] Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun, Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 770-778, 2016.
[22] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprini wrXiv22009.03300 2020.
[23] Geoffrey Hinton, Nitish Srivastava, and Kevin Swersky. Neural networks for machine learning lecture 6a overview of mini-batch gradient descent. Cited on, 14(8):2, 2012.
[24] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint 2rXiv:2203.15556 2022.
[25] Shengding Hu, Yuge Tu, Xu Han, Chaoqun He. Ganqu Cui, Xiang Long, Zhi Zheng, Yewei Fang. Yuxiang Huang. Weilin Zhao, et al. Minicpm: Unveiling the potential of small language models with scalable training strategies. arXiv preprint arXiv:2404.006395, 2024.
[26] Peter Izsak, Moshe Berchansky, and Omer Levy. How to train BERT with academic budget. arXiv preprint arXiv:2104.07705, 2021.
[27] Jean Kaddour. Stop wasting my time! Saving days of imagenet and bert training with latest weight averaging. arXiv preprint arXiv:2209. 14981, 2022.
[28] Krishnateja Killamsetty, Sivasubramanian Durga, Ganesh Ramakrishnan, Abir De and Rishabh Iyer. Grad- match: Gradient matching based data subset selection for efficient deep model training. In International Conference on Machine Learning, pages 5464-5474. PMLR, 2021.
[29] Diederik Kingma. Adam: A method for stochastic optimization. arXiv preprint arXiv: 141.69141..6980, 2014.
[30] Mengtian Li, Ersin Yumer, and Deva Ramanan. Budgeted training: Rethinking deep neural network training under resource constraints. arXiv preprint arXiv:1905.04753, 2019.
[31] Zhuohan Li, Eric Wallace, Sheng Shen, Kevin Lin, Kurt Keutzer, Dan Klein, and Joey Gonzalez. Train big, then compress: Rethinking model size for efficient training and inference of transformers. International Conference on machine learning, pages 9958-5988. PMLR, 2020.
[32] Ilya Loshchilov and Frank Hutter. SGDR: Stochastic gradient descent with warm restarts. International Conference on Learning Representations, 2017.
[33] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019.
[34] Kairong Luo, Haodong Wen, Shengding Hu, Zhenbo Sun, Zhiyuan Liu, Maosong Sun, Kaifeng Lyu, and Wenguang Chen. A multi-power law for loss curve prediction across learning rate schedules. In International Conference on Learning Representations, 2025.
[35] Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can suit of armor conduct electricity? new dataset for open book question answering. arXiv preprint arXiv: 18X9 (ao::0:000000000 02789, 2018.
[36] Rui Pan, Haishan Yc, and Tong Zhang. Figencurve: Optimal learning rate schedule for SGD on quadratic objectives with skewed hessian spectrums. arXiv preprint erXiv:21 14109, 2021.
[37] Rui Pan, Shizhe Diao, Jianlin Chen, and Tong Zhang, EXtremeBERT: A toolkit for accelerating pretraining of customized BERT. CoRR, 2022.
[38] Xuran Pan, Xuan Jin, Yuan He, Shiji Song, Gao Huang, et al. Budgeted training for vision transformer. In The Eleventh International Conference on Learning Representations, 2022.
[39] Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99-100 2021.
12

---

[40] Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi. Socialiqa: Commonsense reasoning about social interactions. arXiv preprint arXiv:1904.09728, 2019.
[41] Li Shen, Yan Sun, Zhiyuan Yu, Liang Ding, Xinmei Tian, and Dacheng Tao. On efficient training of large-scale deep learning models: A literature review. arXiv preprint arXiv:2304. 03589, 2023.
[42] Yong Shi, Anda Tang, Lingfeng Niu, and Ruizhi Zhou. Sparse optimization guided pruning for neural networks. Newrocomputing. 574:127280, 2024.
[43] Leslic N Smith. Cyclical learning rates for training neural networks. In 2017 IEEF Winter Conference on Applications of Computer Vision, pages 464-472. IEEE, 2017.
[44] Leslie N Smith and Nicholay Topin. Super-convergence: Very fast training of neural networks using large learning rates. In Artificial intelligence and machine learning for multi-ommin operations applications, volume 11006 pages 336. SPIE, 2019.
[45] Vivienne Sze, Yu-Hsin Chen, Joel Emer, Amr Suleiman, and Zhengdong Zhang. Hardware for machine learning: Challenges and opportunities. 2017 IEEE castom integrated circuits conference, pages 1-8. IEEE, 2017.
[46] Alon Talmor. Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. Commonsensega: question answering challenge targeting commonsense knowledge. arXiv preprint 1rXiv:1811.00933 2018.
[47] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International conference on machine learning, pages
PMLR, 2021.
[48] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint ar3iv:2307 230.000888. 2023.
[49] Ashish Vaswani, Noam Shazeer. Niki Parmar, Jakob Uszkoreit, Llion Jones. Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances Neural Information Processing Systems, 30, 2017.
[50] Johannes Welbl, Nelson F Liu, and Matt Gardner. Crowdsourcing multiple choice science questions. arXiv preprint (Xxx:1777.00609 2017.
[51] Ashia C Wilson, Rebecca Roelofs, Mitchell Stern, Nati Srebro, and Benjamin Recht. The marginal value of adaptive gradient methods machine learning. Advances in Neural Information Processing Systems, 30, 2017.
[52] Xingyu Xie, Pan Zhou, Huan Li, Zhouchen Lin, Shuicheng Yan. Adan: Adaptive nesterov momentum algorithm for faster optimizing deep models. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.
[53] Kh Shahriya Zaman, Mamun Bin Ibne Reaz, Sawal Hamid Md Ali, Ahmad Ashrif Bakar, and Muhammad Enamul Hoque Chowdhury. Custom hardware architectures for deep learning on portable devices: review. IEEE Transactions on Neural Networks and Learning Systems, 33(1 (1 1 160668 6088, 2021.
[54] Matthew D Zeiler. Adadelta: an adaptive learning rate method. arXiv preprint arXiv: 1212.5711 2012.
[55] Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:I905 078307 2019.
[56] Jian Zhang, Lei Qi. Yinghuan Shi, and Yang Gao. Exploring flat minima for domain generalization with large learning rates. IEEE Transactions on Knowledge and Data Engineering, 2024.
[57] Yushun Zhang, Dmiity Rybin, and Zhi-Quan Luo. Finite horizon optimization: Framework and applications. arXiv preprint arXiv:2412.21063 2024.
[58] Xunyu Zhu, Jian Li, Yong Liu, Can Ma, and Weiping Wang. A survey on model compression for large language models. Transactions of the Association for Computational Linguistics, 12:1556-155 1577, 2024.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main claims in the abstract and introduction (e.g.. UBA consistently surpasses the commonly-used schedules on vision and language tasks and sets new SoTAs for many networks and frameworks, e.g. ResNet and OL,o, under different training iterations.) are fully aligned with the paper's contributions and scope. We provide an in-depth experiments and analyses the proposed learning schedule across vision and language tasks, networks, scales and training budgets.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We explicitly discuss limitations and future work in Section S Besides, the assumptions can be found in Appendix A We test our method across vision and language tasks, various scales of datasets and networks, under different iteration budgets.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point any strong assumptions and how robust the results to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
14

---

3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: The full set of assumptions are in Appendix proof are in Appendix and
A The complete derivations and
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofss provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions the paper (regardless whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: In Appendix D carefully describe provide complete information needed to
reproduce the experimental results, including hyper-parameters(imee learning rates, batch sizes, configurations of optimizer and schedules) for all tasks, implementation versions(such as PyTorch 144.1ccul1y on CIFAR), hardware configurations(GUU models, memory), full results of ablation studies (see Appendix [E). Code and data are also available.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
15

---

(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We provide open access to the code in the Appendix. Moreover, we provide sufficient instructions to faithfully reproduce the main experimental results in Appendix E
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ wttppssccpppbllc////
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
These details are described in Appendix
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Due to the high computational cost of training large-scale models (e.g.. Trans- formers on ImageNet), we report results from single run with fixed random seeds. To ensure reliability, we list experimental setting and provide code for deterministic reproduc- tion. To maintain uniform evaluation protocols across all experiments (large/small-scale),,
16

---

we prioritized identical statistical reporting standards. This avoids selective rigor that could mislead comparisons. Besides, we conduct parameter analyses improve reliability, More- over, small-scale results align with the expectations and results on large-scale experiments, reducing the need for empirical variance quantification. However, we provide error bars of curve fitting on numerical solution.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The computation resource is described in Appendix E carefully describe provide complete information including type of compute workers and memory.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics htttps://neurd: httppp/pc.ppppppppttttccccteeeea
Answer: [Yes]
Justification:
We conform with the NeurIPS code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
17

---

The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
The paper discusses societal impacts in Section 5 It highlights potential benefits, such improving model performance and offering practical scheduling strategy for practitioners, which may inspire further research on learning rate scheduling, Since we note that the proposed method is a generic optimization algorithm with no explicit negative societal impacts, indirect risks such as computational costs from sub-optimal hyper- parameter selection are not thoroughly examined. We provide the hyper-paramett selection guideline through analyses to mitigate this indirect risk.
Justification:
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
18

---

12. Licenses for existing assets
Question:
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All assets are properly cited.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode com datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
19

---

15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/teview based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
Declaration LLM usage
Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development this research does not involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM for what should or should not be described.
20

---

(b)
Figure 3: Conceptual illustration: visualization of the motivations behind the proposed modeling approaches. (a) shows the rationale for the modeling approach in ( while (b) explains the modeling approach for 2 The solid line represents the optimization trajectory, while the dashed circles indicate the local approximation around corresponding minima.
Appendix
A Methodology
Conceptual illustration The optimization landscape of neural networks is highly dynamic due to data distribution and sampling, and sensitive to various factors, such network architecture, optimization and training process. The black-and-white surfaces reflect the loss landscape processing one batch, while the colored surface represents the landscape following the next batch. The substantial variations in the landscape between consecutive batches significantly affect the training trajectory, making challenging to design a one-size-fits-al learning rate strategy that can effectively adapt to such aim to develop strategy robust approach be scheduled leaning
Figure ((a) illustrates the inherent uncertainties during the
fluctuations. Therefore, rate across diverse optimization landscapes. We construct andgetiteeaation-a aware framework for learning rate optimization model that guarantees minimal loss within constrained training iterations under the worst-case conditions.
We analyze loss landscape by quadratic approximation around nearby strict local optima. This approach enables us to capture the key features of the loss surface near these points. Therefore, the trajectory of optimization impacted by the characteristics of the nearby strict local optima, since the optimization process is inherently shaped by the loss landscape and the key features the surface captured by these optima, illustrated in Figure 3(b) Consequently, rate within the optimization model by the information of these nearby strict local optima.
derive the learning
Assumptions The optimization behavior of a neural network fundamentally governed by geometry of its loss landscape, which emerges from the interaction of parameter configuration, datasets characteristics, sampling, network architecture, optimization preconditioning effect. In the vicinity of any strict local minimum, the loss surface can be approximated by quadratic method whose curvature is determined by the local Hessian matrix. Thus the optimization trajectory is consequently dominated by the geometric properties (e.g., Hessian features, curvature profiles) of the nearest strict local minima.
Many tasks train models using epochs, where one epoch represents a full pass through the datasets. Under this circumstance, an iteration occurs when the model processes a single batch. While number of iterations per epoch varies with batch size, training with fixed epoch budget becomes equivalent to using a fixed iteration count when batch size remains constant. For consistency, this paper uses iterations
primary unit of optimization model construction and networks training.
21

---

The sampling Hessian satisfies: Eg [m
o² for some constant This assumption is followed by the work T6 36.. Here, o² measures the degree of Hessian inconsistency across data samples, i.e., the variance in the per-sample Hessian matrices. Besides, a smaller o² indicates more stable curvature landscape between each data sampling, leading lower noise amplification during optimization.
Optimization problem derivation for Section 3.1
Considering the first constraint W++1 W
n (5) (We - (()) in min- max optimization
2 we can reformulate it as
W W(() = - ( (w.-wik - W()
(9)
where I denotes the identity matrix. By iteratively applying this equation ( we obtain
(Wn.. W((k)
(10)
I (1- (14 ... (T. ( (Wr+1- -
Since the matrix (T) (E) is positive semi-definite, it possesses N eigenvalues x1mh 1th (f(E), 2 (f(E)) ((1)) (f(E)) abbreviated as 1!th, which satisfy 0 < 1 - (f({) <
X for all i. Here, A/() and x denote the bounds the non-zero eigenvalues of the Hessian around
k-th optimum. also satisfies ( - where is the learning rate to guarantee the convergence [56)
Moreover, there exist N linearly independent eigenvectors u, (x)) (f(E), un (f(E)). "N (f(E)) abbreviated as u.
(R) which can form the basis for the dimension vector space. Then we have (k) (i 1,2, ,N) and the initial deviation from the optimal solution can N
/ (8) (E)u ( u;
be expressed as WT. (R( = Σ
(K) where are the coefficients corresponding to the
8,
eigenvector components. Thus, we obtain
WT 10/0011
N Σ
Tx+1 II 1=++T
(1 (1-ma(to)
am1
Te+t
S 1=1
8
(k) 2 11
(4)
112
max
πI (1 (1- [=1+1
(11)
x (A)
Th+1 πI (1-m2((a) 1=1+T
Wr. - -Vrrol
max X (x) (x!
22

---

B Numerical solution and curve fitting
To solve the constrained min-max problem 44 follow,
Tk+11
II
(1-m)
min 7+47y 124
max
1=++T
s.t.
71T 72+ Mea+1
Thains Truax)
k: 1
K
we adopt an iterative projected gradient method that alternates between minimizing over the variables Me and maximizing over the parameters x Specifically, at each iteration, we update de with fixed (k) as follows
Th. f- P Plymin Praxs - av an (TT, (
For fixed Th, we update (( via
(A50) 200) (A.
(k) + BV X Sen L(m 1!11))
where P is the projection over box constraints, and a and 33 are step sizes.
This approach requires per-network numerical optimization to determine optimal learning rates case-by-case. To circumvent repeated numerical optimization for learning rate scheduling whenever network is trained, we propose to fit these solutions with parametric function universally across networks.
We note that the solution of optimization model is an unordered set (7 72+T. mr meaning that any permutation of these values yields the same objective value. To facilitate function approximation and improve the training stability of the network, we sort the solution set in both descending and ascending orders before fitting them with smooth function. This pre-processing step preserves the optimality of the solution while enhancing the stability and interpretability of the fitted function.
Due to the symmetry between the descending and ascending orders, and without loss of generality, we first fit function to the descending order solution. The corresponding function for the ascending order can then be easily obtained via a simple transformation.
Our key observation is that the numerical solutions under varying λo, An configurations exhibit two characteristic decay modes. As shown in Figure 4 these transformations manifest through two distinct mechanisms: (i) Fradual--o-acceleerate decay: gentle initial decrease transitioning to rapid drop (as shown in Figures ((d) 4[e)) and 4(f)); (ii) rapid-to-gradual decay: Initial steep decline followed by slow decay (as shown in Figures 4 4(h) and 4()). These pattern can be formulated as transformations of base function, which resembles the curvature variation of the transformed L
norm modulated by a parameter (42]
We first isolate the base function by examining the limiting case where x R Ag In this regime, the numerical solution within each interval [1 + Tg. Tk41] manifests cosine-like profile (Figures 4(a) ((b) and ( suggesting the base form:
[ - (T))-TT)-- 1]
1 + cos
2 [T -Tk)
Then we construct the fitting function as:
(2((t-Tx)-13
+ cos
3(Tx+1-TT)
(Amax
7min
7mins
(12)
b+ 1 + cos
(((t-Tz)-1p) 2(T TT
where {a, b, c} control learning rate decay modes.
In addition, through systematic fitting across nine configurations (Table we discover a universal scaling relationship among the parameters:
b 2c
2.
(13)
a
23

---

Table 4: The parameter details of curve fitting.
hyper-parameters
fitted parameters
objective value(log)
4
Mmmin
Tmmax
X
(4)
numerical solution
by fitting function
10
1.89
2.60
0.26
0.51
0.00
1.99
792.67
750.83
10
2.39
2.61
4.16
7.35
0.53
798.38
770.38
0
10
5.29
5.61
4.84
7.32
1.28
1.56
1118.79
1105.80
0
10
18.89
260.11
1124.67
212.28
1011.94
0.19
2933.34 34
2933.62
0
5
28.89
200.11
755.91
85.01
719.40
0.12
2607.11
2591.51
150.11
0.27
0.05
0.24
0.17
2031.66
2075.50
10
10.01
150.11
77.45
684.15
-268.20
8.40
2373.58
2431.42
2
10
18.89
50.11
0.11
2.52
-1.15
21.02
2007.68 668
2044.84
3
10
20.01
180.11
0.07
3.07
-1.47
38.34
2604.98
2632.21
Table lists the fitted parameters {a, b, c), where the mean absolute error 0.04. Figure 5 visualizes the variability of the 6+2s by shading the region within #1 standard deviation from the theoretical value in Figure 5 The empirical results falls within + of the predicted value (2.0 + 0.04) in most
of cases, demonstrating strong agreement with the assumption relation [3]
This empirical relationship freedom follows.
13) allows us to reduce the original 3-parameter system to degrees of
2a 1 + cos
2((1-Ta)-11 2(11+1
-2k]
(Trax Tuii
Tmin (14)
22(1-72 To 1-)-))1111 2(Tx41 -Tb)
2b + (2a -b) b)
cos
Actually, a I-parameter system
2((t-Te)-11
+ cos
2(Tn+) - -Ta)
(Mmax Mminn)
+ Mminn
(15)
2p (2 :p)
21(- 1-)-1117 2
where yP : b/a. We list the value of fitted parameter yp in Table Since the equation (4
and the equation (15) are mathematically equivalent, we adopt the equation (15) as the canonical representation throughout this work.
Then the corresponding function for the ascending order can then be easily obtained via a phase shift of as follow,
2 1 + cos
(2((t-Tx)-1]= + + 1
(1max - 7minn)
Mmins
(16)
24 + (2 - v) | cos
((t-Tx)-116 + 2(T 72)
In summary, we obtain the 77 within the interval t [1 + Tk. T 1 as follows
2 + cos
(2(t-Tx)-11 + (k 1) 2(T4+1 -T)
T) (Tmax Mmin
7min
(17)
2Gp +
(p)
cos
12(c-1 1|mT 2( 1 Tx)
(k: 1)
where yp is the dyper-parameter controlling the variation speed of T/.
24

---

Brows
20
13
50
10
(a)
(b)
(c)
12
25 15
35
15
(d)
(e)
(1)
11
g
(h)
Figure 4: The curve fitting numerical solutions.
2.10
2.05
(b + 2c)la
755.91
2.00
0.07
1124.67
0.29
1.95
0.27
77,46
1.90
vauer2
panmmttr (4.
standadd deviatin stardard desiorion
10
10 0
10' Parameter a
10²
103
Figure 5: Visualization of relation between parameter a, and c.
25

---

cos achocule p-2
p-30
5.0
3.8
oosenettall shesue
0.8
Nam
0.4
0.4
25
50
77
100
125
25
50
100
125
25
50
75
100
125
(a)
(b)
(c)
Miru 2....
0.3
0.4
2.4
0.2-
5.2
50
75
25
50
50
(d)
(e)
(f)
Figure 6: Adaptive simulation by UBA schedule.
C
Adaptive approximation of existing schedules
Our schedule has adaptability owing to the parameter ya, enabling it approximate the behavior of existing schedules (e.g.step decay, cosine annealing, cyclic schedule or Rex schedule) through simple parameter adjustments. More importantly, it outperforms these schedules by jointly optimizing training stability and final accuracy.
We visualize the adaptive simulation of in Figure 6 By setting parameter io 2, the UBA schedule degenerates to a cosine schedule (Figure 6(a)). By setting o to large value such as 30, the UBA schedule mimics exponential schedule (Figure ((b). By setting & to small value such 0.8, the UBA schedule mimics Rex schedule (Figure [(c).. By setting to O, the proposed schedule degenerates constant. Therefore,
control 7/max piece-wise value depending on iteration, and the UBA schedule degenerates to a step schedule (Figure ((d). By setting l to any integer larger than 1, the UBA schedule possesses cyclic characters, thus mimics cyclic schedule (Figure ((() 6(1)). Since the OneCycle learning rate schedule can be viewed as single-period version of cyclical learning rates, the UBA schedule mimics the OneCycle approach by emulating the cyclical schedule.
26

---

D Propositions and lemmas
D.1 Proof of Proposition 1
Proposition
The
fitted
function
is
the exact
closed-fore fored-form
solution
2 1+0
1/ Tk
to the min-max optimization problem:
1111- 2
2 (2-1p)
I+ons
Ta+1 II
min
max
(1
(K)
(18)
1111 T2 1221 0
1=+TT,
when the hyper-parameter ip are determined by x1 and (K) through the relation ip = 2
and
Tmax = 1, Mmin = O.
Proof. According the theorem in 14] among all polynomials of degree in μ that take the value (.(
+1 at = 1, the Chebyshev polynomial S, (p) = C.(d) is the unique solution that minimizes the maximum absolute value over the interval (-1, +1). Here, Cn(u) cos(n arccos(()) is the n-th order Chebyshev polynomial.
We define the polynomial Q(A! (*)) as:
Te+:
Qa
II
1(h)
71 +
A (R)
A
(1)
Nu
t-++1
which is degree Th+1 Ta polynomial in (,A)).
Next, we introduce the variable transformation:
-21 x + (1)
(R)
X
which maps the interval ((m) < 1
(k)
A
(R)
to the interval -1 S y 1, such that x(a)
2
corresponds to 7 -1 and X 1 (k) corresponds to 1.
The polynomial P(7) Q(1,1) (1) now satisfies the condition in the theorem from 14 i.e.
(k)
x
P
since
(0)) 1.
(e)
(k) A
Auu
Therefore, the original min-max problem
Teta
1
(k)
WT.
mmi
max (R SX
-
(k)
X
(ec
+
A
T=
equivalent the problem of finding polynomial in y of degree T maximum absolute value in the interval -1 S < 1.
Ti that minimizes the
By the theorem in [4], the desired solution to the problem is the Chebyshev polynomial:
Cre -Tt (?)
ST,11-T(77-
ST,
+x (K) X15 -
CT.IL-Th
To match P(y) ST.41 (7), we equate the corresponding zeros. The roots of (()(()) are given by:
x(5)
1
for t= 1,2, ... Tk+1 Th.
17t
+
27

---

On the other hand, the zeros of P()) correspond to the values of y where the polynomial P(y) vanishes, and these zeros are given by:
1 (k) (R)
7t
An (R)
21 (R)
The + x (A
)
for t= 1,,,,.... Tk+1 Tk.
The zeros of the Chebyshev polynomial S14f1 T2 (7) are given by the values:
(2(t Th) - 1)sT
for t= 1,,,..... Tk+1 Tk.
2(Tx+1 - Th)
Equating the zeros of ST -Ta (7) and P(y), we have:
\ + Au (A)
2
(2(t-
(2(t - TTa) -1)x
cos
X X (k)
(A (R)
1/ (10)
2(Tx+1 Tkk)
TJt
Solving this equation for The we obtain:
(2(1-Tx)-11)
(1 + cos
2[Te+1 -ib)
(())
(
1
[ cos
(2(t-Ta)-1 1)z 2/72+111 1x
D.2 Proof of Proposition] 2
Consider the training process within the interval [1 Th, TR41). When the hyper-parameter up is set sufficiently close to 2, the proposed learning rate scheduling formula reduces the cosine learning rate schedule.
Proposition
Proof.
1 + cos
2((t-T))-13
2(T2+1-1,
lim (7max 7minn) x-2
+ Mmin
24p (2 1 :p) +
2 (G-TT)) (--T)))111
2(2 1
1 1
(2(1 Th. - 11)T )) M/nin
-(mmax T/nin) (
+ cos(
2(Tx+1 Tx)
(t (t-Tis -TTR)=
π 2(Tx41 Th)
(19)
2 (Nmax Mminn (1 cosl
cos(
Th+1 TT
+ sin( (t - -Tk)
sin(
+ Tmnin
Tk+1 Th
2(Tx41
Thk
(t
Tx)
(7max - ) (1 + cosl
)) + 7min
Tk+1 Tí
The final approximation holds because each training phase typically contains many iterations, i.e., T x= 0. Consequently, sin( 2(T..) [(T..)-75) 0 and cos( ((77.) xz 1.
making
D.3 Proof of Theorem
Lemma 1. Let ni be the learning rate at the iteration j E [1 Tk,
t). defined by the proposed form 5 Let X be the eigenvalues of the Hessian matrix H (k) (E) at the k-th strict minimum (k).
40.00 (Amax Amin))
1+cos
[2(1-T%)-1p
(Te-1-Ti)
Denote T =
2
Then for any E [1 + Tg. T 1). the
following inequalities hold,
28

---

Case 1 (0 > 2):
II (1 - [
j=1+T
(20)
S <exp -21, (k) Mmmi (t - -ia)
4(Tx+1 - Tx.) + (4p 2)
4(Tx+1 - Tx) + (4p - 2) (L Tk)
Case (4 2):
t
II j=1+Tk
(1 (1- 1-
(21)
(2p + 2T (T)
[Tl+1 - (1 - Ta 0.5)
(2-4)
$ <exp(
Mmina (ixtt)maill (tT)))
(2.p 2T (PT)) + 2(Tx+1
(2-2)5
Proof. By the fact that - x I exp(-xx) have
II
[1-)
exp
-2 Σ
m3 1
(22)
j=1+TR
j=1+Tk
Substituting the expression for ni by the proposed form
5 yields:
exp
2
Σ
j=1+T
2(1 + cos
(2(j-T.)-13)
22 (k)
2(11+1 16)
exp
(Jmax Tminn)
Mmin
j=1+%
24 + (2 - () + cos
(2(1-Te)-1)s
2
(23)
-2x Mmin (t 1- (-TT))
exp
2 ( + cos
((((-TT))-)m
- 2x (k)
2(Tx-1 Tn,)
exp
(7max
Mmmnn)
2G + (2 - 4p) (1 + cos
(2( To 1115 1)5 2( Tk)
j=1+%
The behavior of this term depends on the value of the parameter y. We consider the following cases:
y 2, - 0
Since
(2(t Ti)) 1)TT
cos (2(j Tk) 1)TT
(2 (2(j- - Th) 1)TT)
cos(
2 1
2(Tx+1 Tk)
2(Tx+1 Tk)
2(TT+1 Tkk
(24)
(j - -Tw)
>1
(Tk+1 Tk)
we have
Σ
j=1+T, 24 + (2 - () (1 (1+cos
(2(j-Tu)-
2(Tx+1 - I1a)
Σ
(25)
j j=1+T, 24 + (2 v) (1 (1+1-5
(1-Tt))
(Tet1-Te)
1
j=1+T,
4 + (p-2)m
(Tk11-Ta
(j Tx)
29

---

Then the equality 23
becomes
exp
-2 Mmin (t - -Tb)
(k)
2 (1 11 cos
(2(j-Ti)-13)
22 (k)
2 Tnc
exp
(7max - Mmin))
20 + (2 (p) 1 + cos
(((--t)--11) 1)x
j=1+%
2(Tx 2(Tx11-T2
(( Tx) 1)TT
-21, (K)
Muii (t - Ib)) exp
- -41 (K) (1umax Mmiu)
I+ ( cos
S exp
2([Tx+1 Tk)
(26)
1=1+T 2p (2-1) (1 cos
1
2(2x+1 -2k)
< exp (-21(4) (k)
Mmmin (1 (1- -Ta) exp
-41,0) (1max - 7min))
1 +css cos
(2(t ((2(t-T+) 1) 2(7)-1 Tkk)
Ti:)
Σ
4
(f-2)z
(j - Tx:)
j=1+7,
(Te+1 1141-17) Tr)
Note that the function h(j)
monotone decreasing over j E [1+Tk, t).
(ύー2) (1-T))
so we can bound the summation from below by the integral:
Σ
+ (p-2)x (j - Tx)
j=1+%
Tkt1 Th
6+1
dj
4 +
(p (p-2)7
2)
(j - Txx)
141
Tk41 -a1-T- Tr
(27)
2
dj
4 + (p-22)n (j - (TT)
T2
1+1,
(Tk+1 - Tix) In
[(TT+1 - TTk + (1p - 2)m(1 1 Tkk)
(4p 2)xx
4(Tx+1 - Tx) + (4p -2)
Thus the inequality (26) becomes
(2(1-
(T)) Ti) 1)
exp (-21 (k)
Mmin (t - -Tkk)) exp
-41, (b) (Mmax Mmiin)
+ cos
1 1
2(TT - Tk)
Σ
4+ (x-2)5
(Ta11 (6:11-T)) (j (T))
/=1+%
< exp
-2x Mmin
(1) (1 - -Ti)) exp 42 (1) (Tmmx (Thmax Mmin) (1+ cos
(16)
(2(t- Th) - 1)
2(Tk+11 -Tx)
(28)
(k+11-70 In
4(Tx+1 - -Ta) + (sp - 2)
(4 - -
4(Tx41
Tx) + (4 - 2) Tik)
44TT+4 4(Tx+1 - Tx) + (1p -2)TT
exxp
(22x Mmin (t - -Tx))
T2. + (1p 2)T(( Th)
< exp
(2x
(k) Mmin (t Tk)
4(Tx+1 - TT) + (1p - 2)TT
4(Th+] Tx) + (p 2)T(( (1- Th
The last inequality is because x
S X and Tr -
AT2x -T)+(e-2)- 2)
$ 1.
30

---

y 2, 2
- p 2 0 Since
(2(j (2(j Tk) 1)TT
(2(j Tk) 1)TT
+ cos(
- π
(29)
2(Tx+1 - Tk)
2(Tx+1 TT)
We obtain
exp
-21 (k)
Mmmn (t - Tk))
2 + cos
(2(5-1h)-1))
-21 (k)
2( Txd
exp
(mmax 1min)
24 + (2 (0) 1+ cos
(2 11x 2(T Tx)
/=++7k
(2(t -T)) 1)T
S exp
-21, Munin (t Th)) exp
-11, (fmax 7mii))
(Tmaxx
| cos
2(Tx+1 - Tk)
(30)
1=1+T 24 + (2- - -1))(1 + cos
(2(1-Ta)-113
2(124) -1)
(2(t (2(t- (T)) 1)T
(2x (k) Mmin (1 - Th.) exp
-41 (1) (Amax Mmmin)
< exp
I + cos
2(T Taa)
(2.p 2n - -p
(2-0)a
(j - 0.5)
jm 1114%
[T Th.
Because function l(()) == (2.427--23) (012--a))
monotone increasing the
17++1 2gJ (i 1--77 -0.5)
12-5)
range [1 + Tk. too), then it holds that
j=1+T, (2ip + 2n (pt)
(T - 1t) (j Ti - 0.5)
(2 p)-
dj
(31)
(2ip + 2T -pp)
(Tx.1 Tx) (j - T3 0.5)
12-4)
(Tk+1 Thb)
(24p + 2nT -pp))=
(2-2)a (Tx TT)
Tk: - 0.5)
In
(2 ()
(2pp + vpx) + 2 TT TT)
12 12-5p
31

---

Thus, we have
(2 T-) 1)
(-2 (R) Mmmn (t - (t-T)) exp -11 (1 (Mmax - Mmiin) 1+ cos
exp
2(Th-1 - Ta)
Σ
(2pp + 2n - (p - [T441 TxJ (j - -Te 0.5)
(2-4)T
j=1+7
-21, (8) / (t- -Tx)) exp 42 (Nmax (Thmax Mmii))
1 (1+ cos
(2(4- Th.) -1)
1)TT
I exp
2(TT+1 - Txk)
(Tk+1 Tk) In
(2p 2n (T)
(Th Tx)
(2-0)T
Tx= 0.5)
(2 Tz)T
(2-$2 (2p + 2x (px) 2(12+1 1 TT)
(32)
S exp
-21, "min (t (t-
(2pp + 2T (T) (11+11 -1x]
(2- (t Ti: -0.5)
0.5)
(2p + 2n -(TT)= + 2(T -Tx)
(2-4)xx
S Exp -21, (1:) (1))
Mmii (1 (1- - (T-)))
(2p + 2TT -p (Te+1 - [x]
12-5)x
Tx - 0.5)
(2p + 2n (pt) + -12
(2-5)x
The last inequality is because x(m) S ((4) and
(2p+ pp)
1 7
(2 (2-
(t- Th 00.5)
S 1.
(2p+2m pm k+9
Theorem I Let = (1 (W₂- -
be the stochastic curvature noise introduced by sampling at iteration t. Assume that the sampling Hessian satisfies: Eg [neng] 3
(E)
4x/²) (Maas
mmin))
14000
(2(e-2x)-11 2 -.)
(Te11-Tx)
If we set the learning rate as the proposed form ( the loss uncertainty introduced by stochastic gradient
o ( for some constant a. Denote T
(g-2)xx
method within the interval t E [1 + Tk, T T+1l can be bounded by two terms,
For & >
E -((())))))))))
4(Th-1 Th)+++++ + (pp 2)
- -2)=(( - Tx. .expl texp(-21!* Mminn (t - -Tx)) A [ 1+T, - 10
1(Tx+] - Tk) +
+o Σ n M (2)1)) exp -2x (k)
N
Mmin (t ( (1-)))
4(TT+1 - Tis) + (1p 2) - Tio)
(33)
4(Tx+1 - Tx) + (4p - 2) - TT)
1=1/72
j=1
For 40 < <:
E ((f(We, C(F(W(()) [0)]
(2ip + 2T (pt)
(Th11 Tt) (t Th 0.5)
(2-p)x
exp -21 Mmin (t TTk) {T)))a!00)
(k)
21(0 (.01|16) | 1T-144)
(2ip + 2t
(oT) + (2-
2[7 T
(34)
+
Σ
N
2 exp (22 Tmin - -i)) (1("))²²
(24p + 2x (pt)
(Tx 1-i1-TT) Tx)
(2-1)
(t T3 - 0.5)
(2ip + 2n upt)
(2- p)=
(i Ti - 0.5)
i Tx
j=1
(T++1 Twj
32

---

Proof. Let ( denote full batch Hessian at k-th minimum, such that { denote n4 ( (We W -00--43 (c) (E) (Wr W()), which indicates the bias. The uncertainty brought by stochastic gradient method is measured as follows,
(5)) = and
W41 W/(k) Wi (V(()) nelly (8) (E) (W; -
-W, ((()) mlll (W, - k
+ / (Wr - vT(c)) - , (5) (W, -
(I mH (W, - + ,,t
(35)
(I n H (0)) (T 1/ -1
11(b1) ... ...(1 (T 1/1 - H1(0) (We-o 1- w(x)
+ Σ (T- - m1 ; ([ 1/- (1)) ... (T - 1/it1 (1,10))110 mim
p(()) p({b) ((*) (W, 1-g ((((k) Σ
p( p(A)) p!() Piti
i=t-q
where p((k) = (1 (+))
According to the approximation approach of the objective function 1 in Section 3.1 i.e. L(f(W,E)) L(f(W(()) .)) + [W - w(()) ( (E) W(()), the loss uncertainty in- + Tx T is quantified as
troduced by stochastic gradient method within the interval E[(WT.,, W H (Wr. - W())), where the loss function is approximated by leveraging
the full batch Hessian.
E (C((W.)))) -(S(WW)) .0)] TE. [ ((h))) (W, [7(k))]
(())
((k) (()) ...))))) (W1+7, W(h))+ +
Σ p()) (())
((K)
= E
26-1-1
Mams
i=1-Tk
HM
(11)p(M)) p( 1-TT (W1+T, - -W(())+ +
Σ p!() ((h) ((.)
i=
(36)
E
...P(h) (W; - w((k)) T Tp(4) ... (k)
H1 -
(k) p(())
1+T, (W, - v(((t))
(We w/(1)) T p(4) ... ...pph) (1(0))
Σ (()) p(A)]
+ 2E
s+1 17m24
1=1+T
T
+ E
Σ (=1+T
((())
((K)
mmm
Σ ((k)
p(k)
s+1 npn:
/=1+T
Since E [n.] = O, (W, - W(())T P(N)
p!") ((
Σ (=14T
((())
... P = 0. Mean-
14T,
while, we notice the data samplings between different iteration are independent, thus E [(n.)
33

---

0 ( We obtain
Σ
p(h)
pll)
1 Met
H1hh) Σ p(())
pll)) i+i man
i=1
Tk
i=LITe
E
Mem! p p( p(()) P/.t n5
(37)
iF=I+T
PM p!(A) H(() p( ... ... pffinan))
E
n:n,
Therefore, Eq. (6) can be reformulated as follows,
E
Σ
p(()) PLA
1-1
1974 (1 Σ p(()) p((k)
1774
i=+
i=i+Te
(Wr - ((A)) T p(AT
(W))T
... P.P()) 11 p(()... p( I-T, (W, - w()))]
(38)
nm (()) ((()) H p( pm
+
E
( 11
= B + V,
where we denote B == [W, (x())) p(h)
(..P(())H()) ... ...,(), (Wr - w())
1+r," ... H 11 p())..... p...p..())m) variance term.
bias
term and V = Σ [nRT p(h] p(0)
1=177h
(k) Denote uq for the N dimension vector space. Then we have H (k) (E) X u (i = 1,2, ... ..)...N) and
iE (1,2, N} are linearly independent eigenvectors. which can form the basis (k)
the initial deviation from the optimal solution can be expressed as W. (K()) Σ
(K) where
(1) are the coefficients corresponding to the eigenvector components. Combined with Lemma we obtain
Case (0 2):
B (We - W7(k) T 11t. ...(()) p((k) (())... Pitz (W: - W((A))
(I - ( ... -II 1144 11 (1) (WWTT -W()]]] =1(1) -1--ma
(1 n 11 ... ... 1-14T, 11 ( (WI4T -W((A))
N
Σ
(8)
(0)
1lu 11x(00) (1)
II
(1 {m?
j=1+ 1+7
N
(39)
Σ
(s) (1) 111
(k) 118 exp -2x (k) Mmin (t Tk))
4(Tx+1 Tx) (pp 2)TT
4(7k+1 TT) + (p 2t Tk)
4(Tx+1 - Tx) (p 2)
4(Tk41 - Tx) + (p TT li:)
exp -2x (k)
Mmin (1 Tx)) |
W1+T,
11
34

---

In addition,
1 Σ 1=
T P ... PP( 11 p!()) p.()) mons)
V
E
s+1
1 tr n2 p(N) ... p!()) H p(()) .... p(42n))) m3s)
i=14
E tr
p(()
p(0) H p(h)) ... p(t) p
[+1
im11
nttr p(k) 1+1 ... pp(k) H (k) p(k) pff E (nmm.)
1
(40)
2
11t1r
pll) i+1
... p!l) H1 p(k)) p(A) [|]
(1++Ty
N
o²
2
(k)
II (1 (1- ( 2
1/4
1=1
q=i+1
2
2
Σ
( ) cxp (-2, (k) maiin (t - 1)
1/1
(=+TT
1=1
4(Tx+11 - Tx) + (4p 2) - Tk)
(Tx=1= - Tx) (p 2) Tx)
where the second equality comes from eyclic property of trace and the first inequality comes from Eg [mmn?]] ] = o² (
Case 2 (gp 2):
B = (W,- - 17/(1)) T Tp(4) ...P!') p(())p (())...p(1)) Pin. (W: - (7(1))
[1 (1) ... - nnt H1 ( -W(N]]
1/00 (1 - il - ... (1 - m 11()) (W2+21 - ((()))
N
Σ (s) ( 110)
(K) II [11 (1-1) (1-n)-((1) {
j=1+Tc
(s)
(1)
(1)
exp -2x (k) ( (t- -Tx))
Mmmi
(41)
(2p + 2T (p))
(T2+1 T5) (t T3= - 0.5)
(2-2)m
(2ip + 2n (TT) + (2-4)T
-116
(2p + 2n - (pt) (2- (t - TT - 0.5)
(12+1 1
(2ip + 2 - (pT) + 2[TT+11 -Tú)
(2-9)8
expp
-2x (k)
Umiaa (1 Tx)) X (1) W W1+T. - 1/
35

---

In addition,
V Σ
E n p(f ((( p(A) p( m
1
Σ
1 tr n p(()) p(()) 1T
p(()) pp
p!4 m1
1
Σ
m tr [p(4] p(()) H (k) p((k) p(() p non?
1=
Σ
t
p(()]
p((k) H ( p(()) P(42 E (n,m?))
i=
(42)
2
Σ
T/
p(l)) H ( p(h) p(#) 1m
i=14T
N
22)
II (1 (1-17 m a=i+1
im
j=l
(k) 2
exp -2x, (k) Mainn (t - i)
M/
5=1
(2p + 2T (p)))
(2-4)T
TT 0.5) -r
(Tx+1- Th.
(2:p + 2T (pT)
(2 (2-
Tq 0.5)
(71-1 Tx.
where the second equality comes from cyclic property of trace and the first inequality comes from Eg [nmn]] o² H()) (k)
In this proof, we employ assumption Eg [nmn 1 0² (1) used in the work [16 36 Moreover, we acknowledge that the idea of the proof is inspired by the approach introduced in [6 36
36

---

E Experimental details
E.1 Baselines
We detail the baselines as follows,
Step schedule(SS) reduces the learning rate in piecewise manner. It implements discrete learning rate drops at predefined epoch intervals following schedule g(t) = det/t, J. where ts is the predefined decaying step and (d 2 d is predefined decaying scalar [6]] A typical implementation decreases the learning rate by a factor of 0.1 after 50% of the epochs and further by factor of 0.01 after 75% of the epochs [2I]. However, it requires careful tuning of step timing since abrupt changes may destabilize optimization. In this paper, we employ such setting of step schedule (x 0.1 after 50% and x 0.01 after 75%) for all our experiments.
Cosine schedule(CS) controls the 1-th iteration learning rate d following the mathematical formulation as 1/: T/min + (7Jo 1000m (1 (FF where 7 is total iterations. It provides smooth transition between learning rates, shown improve final model accuracy step schedule. Moreover, it is proposed to mitigate abrupt decreases in the learning rate that could hinder models from escaping local minima. This approach has demonstrated effectiveness, particularly in transformer training [32]. Besides, it includes restart mechanisms where T becomes the cyclic period.
Cyclical Learning Rates (CLR) oscillates between base learning rate Mmni and maximal learning rate 7/mar with triangular and triangular2 policies. The cyclic property help traverse loss landscape barrier, so that it enables neural networks to train significantly faster, often by order of magnitude, compared to standard training methods 43
Besides, the commonly-un OneCycle learning rate schedule can be seen as the one period version of Cyclical Learning Rates. In our experiments, this baseline CLR includes Cyclical and OneCycle Schedules(1C), For CLR, we set linear mode for each period and period is two.
44
Budgeted training(BT) conducted by the proposed budget-aware setting of existing learning rate schedule under epoch budget [30]. It reformulates standard schedules such as linear, step and cosine schedule as functions of remaining budget. The classic linear decay version is formulated as 1/; тjo x (1 t/T' where T is total iterations. It is particularly effective for hypper-paaameter tuning.
Reflected Exponential (REX) controls the t-th iteration learning rate 7) following the 1-t/T mathematical formulation ?/- =!o (.5+0..5x1-1/21) It is inspired by the performance
of delayed linear schedules, aggressively reducing the learning rate near the end of training and reflecting an exponential decay [5]
E.2 Evaluation benchmarks of language task
We evaluate models on a diverse set of open benchmarks. The benchmarks are listed in Table 5 For language models evaluation, we adopt the [15] for standardized performance comparisons.
E.3
Implementation Details
Consistent with CIFAR, ImageNet and OLMo practices, we use step-wise scheduling for all training procedures.
Vision tasks on CIFAR CIFARI0/100 are simple benchmark datasets that are widely used for quick and efficient evaluation of deep learning tasks. CIFARIO consists of a training split of 50000 examples and a test split of 10000 examples. Each example is 32x32 color image in 10 classes (airplane, automobile, bird, cat, deer, dog, frog, horse, ship, truck). CIFAR100 comprises 60000 32 x22 pixels color images which are divided into 100 classes. Each class contains 600 images and the classes are grouped into super-classes. Each image comes with a 'fine' label (the class to which it belongs) and a "coarse" label (the super-class to which it belongs). We use the torch. optim SCD and torch. optim AdamW API to configure the optimizer. For vision tasks Section 4.T we all adopt the SGD optimizer. For ablation studies in Section
adopt the AdamW W optimizer. We
37

---

Table 5: Evaluation benchmarks of language task
Datasets name
abbreviation
ARC-Easy 10
ARC-E
ARC-Challenge 10
ARC-C
HellaSwag 555
HSWAG
PIQA [
PIQA
WinoGrande 339
WG
OpenbookQA 33
OBQA
BoolQ 9
BoolQ
SeiQ
50
SciQ
CОPA 18
COPA
CommonsenseQA 46
CSQA
SocialIQA 40
SIQA
MMLU stem 22
STEM
MMLU humanities
22
HUM
MMLU social sci 22
SOC
MMLU other 22
OTH
initialize the learning to 1e-1, the batch size to 128. Each epoch consists of 50000/128 = 391 steps. We leave the weight decay value as 5e-4, and complete the training task on Py PyTorch 2.4.1+cul18 with RTX 5880 and RTX 3090.
Vision tasks ImageNet To evaluate the scalability of our schedule larger-scale dataset, conduct experiments on the ImageNet dataset, a dataset that more closely reflects real-world visual recognition challenges. ImageNet consists of approximately 1.28 million training images and 50000 validation images across 1000 classes, which also come with an official dataset split. This benchmark, particularly with the ResNet-50 architecture, has been extensively studied and is widely regarded as standard evaluation setup. We adopt the common ResNet50 architecture, and separately train it with different learning rate schedulers. To provide comprehensive evaluation, we conduct experiments using both the classical SGD optimizer and the popularized AdamW optimizer. For the experiments with SGD, we set the peak learning rate to 0.5 and apply the weight decay factor of lc-4. For the experiments using AdamW, the peak learning rate is set to 3e-3, with decoupled weight decay of 0.02. Following default configuration which requires learning rate warmup, integrate 10% warmup phases of the total training tokens into all schedules. For schedules that inherently include warmup-like behavior, such as the 1C schedule, where the learning rate initially rises from zero to a target value, we retain their original settings during the ascending phase. We set the batch size to 2048 and complete these training tasks with NVIDIA A100 GPUs.
Language tasks For language tasks, we use the OLMo model. OLMo [9 decoder-only transformer-basel language model that builds upon the vanilla Transformer architecture [49]. It incorporates several key improvements inspired by recent large language models (LLMMs) such as PaLM 8 the LLaMA family [48. OpenLM and Falcon [1] Notably, OLMo stands out as truly open language model, offering full transparency through its open training data, released training/evaluation code, intermediate model checkpoints, and comprehensive training logs, making a valuable resource reproducibility and further research.
For the training hyper-parameters on OLMo, we primarily adopt the configuration outlined in OLMo
We conduct experiment on H100 and A100. Following default configuration which requires learning rate warmup. we integrate 10% warmup phases of the total training tokens into all schedules. For schedules that inherently include warmup-like behavior, such the 1C schedule, where the learning rate initially rises from zero to a target value, we retain their original settings during the ascending phase. Besides, large language models commonly provide multiple scales to accommodate different compute constraints. We adjust both model size and training steps to explore budgets. ratio (D/N) of 500 across scales. The specific configurations are follows, 36M-parameter model is trained on 50 billion tokens, 73M parameter model trained on billion tokens, 151 M-parmmeter model is trained on 75 billion tokens, 300M-parameter model is trained on 150 billion tokens. This scaling strategy allows us
19
To ensure robust conclusions, we maintain data-to-paramett
38

---

Table 6: Configurations of OLMo
Model scale
36M
73M
151M
300M
token number
50B
50B
75B
150B
Layers (L)
12
16
20
24
Hidden dim (D)
512
640
768
1024
Attention heads
10
8
Inner dim (H)
1280
1536
2240
2688
Vocab size (V)
100,352
to systematically study the relationship between model size, training data, and performance while controlling for the D/N ratio. The OLMo model is trained using AdamW optimizer. The detailed configuration are shown 6
Our implementation strictly follows the PyTorch official API, i.e. it is inherited from the base class
torch. optinn mecim... 1r LRsschee
LRScheduler
It maintains full compatibility with all
torch.or torch.optim
optimizers. The complete production-ready code is publicly available.
The code of UBA schedule is available at htttps://githi com/Ttt-answer// .it,
E.4 Overall experiment results for Vision Classification Tasks
Overall results for the vision classification tasks are presented in Table depicting validation accuracy vision benchmarks (e.g.. CIFAR 10/100 and ImageNet) using different architectures RReNNe118 18, ResNet34, ResNet50). We conduct several independent runs for error bars. Due to computational constraints, we use two independent experimental results for ImageNNe-ResSeefee our proposed UBA schedule outperform baselines, achieving the highest validation accuracy over both small-scale and large-scale benchmarks across all training budgets.
It is obvious that
Our analysis reveals the following key findings. (i) On task, UBA shows the strongest performance across all training budgets. On the large-scale ImageNet benchmark with ResNet50, UBA maintains consistent superiority. The consistent improvements on both small-scale and large-scale benchmarks highlight UBA's generalizability across scales and data regimes. (ii) UBA outperforms baselines not only at 100% training budget but also at 25% and 50% iteration budgets, demonstrating its budget efficiency, .ffectfcccss in computation computation-coosttainned scenarios. (iii)Notably, UBA shows robust performance even while the second-best schedule varies depending on both the datasets, architectures and training budgets. This instability among competing schedules suggests their performance is highly sensitive to datasets-architecture, characteristies and training dynamics different learning phases. Therefore, for practitioners seeking reliable schedules without extensive method selection, UBA provides default-strong choice. UBA climinates the need for case-by-cad base baseline comparison and delivers stable superiority and reliability regardless of datasets, architectures, training budgets and scales.
and SETARI00-ResNeae
E.5
Overall experiment results for language models
For most tasks, we conduct a comprehensive evaluation across six representative baselines to ensure broad coverage. For large model with 300M parameters, we focus our comparison the top three performing baselines due to the computational tractability.
The overall performance of the OLMo models is presented in Table g We also plot training curves in Figures 7 and 10 showing training and validation losses along with downstream evaluation results across model sizes ranging from 36M to 300M parameters and training data scales from 50B 150B tokens. shown, UBA schedule consistently achieves lower training and validation losses while delivering superior downstream performance.
39

---

Table 7: Generalization accuracy.
training budget (epoch(%))
Dataset
Network
Method
75 (25%)
1.50(50%)
300(100%)
92.71=:0:223
93.45:10.4822
93.61=0.3035
94.21=0.5116
94.2444022419
95.52==0.2433
CLR
92.13=0.2610
92.99+00.5559
94.92=0.3811
CIFARIO
ResNet18
IC
94.23=0.1838
95.03++0.135
95.35=0.200
BT
94.1810.3346
94.79.10.4600
95.68
REX
94.49-1033225
95.10:10.1929
95.45.1011350
UBA(ours)
94.57=0.3281
95.26:10.2150
95.65=0.1589
75 (25%)
150(50%)
300(100%)
SS
71.09=0.2676
75.34:10.180
77.24=0.1800
CS
64.12==0.1800
69. 0.7848
74.29=0.5958
73.33
03.415-0.2835
74.74-0.2266
CIFARI00
ResNNe33
IC
73.38-
75.04++0.6382
77.86-0.2239
BT
72.75=002261
75.35++0..242
78.58
REX
73.64 +0.5635
75.14110.3110
0..04.0..130
UBA(ours)
74.60=0.1508
76.63++-.2678
78.95
75 (25%)
150(50%)
300(100%)
SS
74.74=0.1
77.24+0.3860
78 78.56=0.64433
CS
75.84=0.2220
07.95++0.2144
79.07=000438
CLR
73.90= 1.4609
76.04+1..6419
77 1.3053
ImageNet
ResNet50
7..83
77.74100.2118
78. 78.90_0.2996
BT
75.90 -0.2786
77.2110.4624
78.86-: 0.2814
REX
75.85
77.66 +-0.8853
78.66 10.2828
UBA(ours)
76.29- 0.4031
78.26:1 0.1640
79.39 1.399-1000 0.0170
From T UBA achieves state-of-the-uns performance on approximately 50% of the benchmarks across all scales while the second-best schedule varies. Furthermore, achieves consistent superior average performance among all baselines. highlights the stable superiority and reliability of UBA, providing default-strong choice. Moreover, it demonstrates significant improvements in SciQ-73M(+1.7) and ARC-EE300MM+2.6333 highlighting its ability to enhance generalization across diverse benchmarks. Besides, in Figure 2 UBA consistently achieves lower training loss and validation loss throughout the training, indicating the efficient training ability and downstream performance enhancement. Notably, while task-specific tuning of can further improve model performance, we intentionally fix its value across all tasks and architectures to ensure fair evaluation. Remarkably, with this universal oo setting. UBA consistently outperforms baselines diverse benchmarks, demonstrating inherent robustness to varying benchmarks and model scales.
E.6 Performance across different optimizers
Our scheduling strategy originates from the solution to optimization problem under gradient descent dynamics. While modern optimizer (e.g., AdamW 33 introduce additional momentum and adaptive mechanisms, they maintain the fundamental property of performing gradient-based updates. To verify the performance of schedule, we conduct cross-optimizer ablation studies. We evaluate the model on CIFAR100 and ImageNet datasets using AdamW optimizer. For AdamW, 0.003 and weight decay of 0.0001. We report the validation accuracy for each configuration. The results are summarized in Table
a learning rate of
As shown in Table our UBA schedule achieves state-of-the-an validation accuracy on both SGD and AdamW optimizers, consistently outperforming baselines across CIFAR100-Resh (00-ResNet34 and ImageNet-RRsNee5: benchmarks. This demonstrates that although our schedule is theoretically derived from standard gradient descent dynamics, it generalizes effectively to modern optimizers like Adam - despite their additional momentum and adaptive mechanisms - highlighting broad applicability.
40

---

Table 8: Performance comparison on OLMo model
Benchmark(accuracyy %)
Size Sched.
PIQA
HSWAG OBQA SciQ
ARC-E
ARC-C
COPA
SIQA
SOC
OTH
Avg.
CS
59.74
27.36
26.80
66.50
44.74
21.74
60.00
40.58
24.33
29.09
40.09
BT
59.41
27.33
27.40
67.20
43.68
21.40
59.00
39.82
23.48
29.13
39.79
SS
60.12
27.63
27.20
65.30
45,26
22.41
63.00
40.89
23.87
29.62
40.53
36M
REX
60.17
27.82
27.80
68.10
45.09
22.41
62.00
40.63
25.18
30.02
40.92
CLR
61.15
27.21
27.60
67.50
42.11
22.07
62.00
41.45
24.29
29.38
40.48
60.17
27.91
25.80
67,40
43.68
23,41
55.00
40.02
22.26
29.86
39.55
UBA
60.39
27.98
27.40
68.20
45.79
21.74
63.00
40.69
24.33
30.14
40.97
CS
63.06
29.68
28.80
72.60
45.09
25.08
65.00
41.56
26.24
28.17 17
42.53
61.70
29.79
28.00
71.40
47.54
23.41
66.00
41.97
25.05
26.48
42.13
73M
SS
61.53
29.30
28.20
69.40
47.19
24.41
65.00
41.76
26.32
28.33
42.14
REX
62,46
30.22
27.60
72.20
45.09
26.09
65.00
41.81
25.52
29.34
42.53
CLR
62.30
29.38
27.80
70.40
45.44
24.08
62.00
42.02
25.05
29.05
41.75
IC
61,81
29.90
27.80
71.10
47.54
26.42
63.00
40.79
24.37
28.81
42.15
UBA
63.17
30.09
28.80
74.30
45.79
22.74
65.00
41.45
27.13
28.85
42.73
CS
66.00
35.19
32.00
76.60
49.82
25.42
67.00
42.84
26.24
32.27
45.34
BT
64.85
34,95
29.80
78.20
48.95
26.42
67.00
42.99
26.83
31.87
45.19
150M
65.45
34.76
30.60
77.10
49.65
24.75
68.00
42.37
26.58
31.59
45.09
REX
65.56
35.72
29.40
78.30
52.46
25.08
69.00
43.30
27.68
32.64
45.91
CLR
65.07
34.76
31.60
77.20
50.70
25.08
68.00
43.71
27.30
31.23
45.47
IC
65.29
35.16
32.80
76.30
53.16
25.75
67.00
43.45
27.64
30.62
45.72
UBA
65.23
35.50
29.80
78.30
50.35
27.42
67.00
43.50
29.29
32.72
45.91
200M
CS
68,88
45.32
33.40
83,20
57.19
27.76
69.00
44.58
29.25
34.57
49.32
BT
69.64
45.21
30.80
83.70
57.37
26.42
72.00
43.96
28.66
35.49
49.33
REX
70.18
46.30
33.00
84.40
57.72
28.43
67.00
43.71
29.50
36.74
49.70
UBA
69.48
46.44
34.60
83,90
60.35
29.10
72.00
44.42
28.53
35.41
50.42
E.7 Parameter analysis of
In this subsection, perform sensitivity analysis of the key parameter &p in equation 5 which controls the variation speed of learning rate. Our experiments systematically evaluate (0.25, 0.5, 1.0, 2.5, 5, 10) across different optimization scenarios. The results are plotted in Figure and listed Table
Observations From the experimental results, we find an interesting dichotomy of Phe- nomenon(a): By choose proper y, UBA demonstrates consistent performance gains all baselines across the full spectrum of training budgets (25%, 50% and 100%), irrespective of optimizer choice (SGD/AdamW). This suggests the effectiveness of UBA is orthogonal to specific optimization strategies. Besides, with optimizer changed, UBA maintains comparable accuracy on the same dataset-architecture, which shows that UBA exhaustively exploit the model's capacity regardless of optimizer variants. Phenomenon(b): As shown the best performance for AdamW achieved with y whereas SGD performs optimally with 2.5. On ImageNet dataset, the best performance for Adam W is achieved with & = 0.5, whereas SGD performs optimally with o 10. Our experiments reveal a notable trend: when using AdamW, smaller yp yields better performance, whereas larger is preferred for SGD.
and Table|T0) on CIFAR 100 dataset,
Analysis Furthermore, we intend to explain the dual behavior of &p through theoretical analyses (Proposition I and Theorem []. We attribute Phenomenon (a) to UBA's optimal scheduling dynamics, governed by the value. Moreover, Phenomenon (b) reveals fundamental dichotomy in value selection between optimizers. We attribute this phenomenon(b) to the preconditioning effect of AdamW. Unlike SGD, AdamW adapts the learning rate by scaling gradients with the square root of the second moment estimate /rz. regions where gradients de are large, the surrounding landscapes steep. The denominator /u, , is also large, effectively reducing the learning rate. This adaptive
41

---

Table 9: Cross-optimizer ablation studies. We list validation accuracy for vision classification tasks across SGD and AdamW.
Dataset
training budget (epoch(S6))
Optimizer
Schedule
-network
30 (25%)
60(50%)
120(100%)
SS
70.85
75.58
77.28
CS
63.88
68.84
70.43
CLR
72.63
73.81
74.80
SGD
IC
73.72
75,53
77.90
BT
72.53
75.40
78.49
CIFARI00
REX
73.12
75.48
77.99
UBA(ours)p
74.57
76.68
78.97
-ResNer34
SS
64.01
70.93
73.65
CS
62.56
68.91
72.98
CLR
62.87
70.13
73.22
AdamW
IC
64.27
71.60
74.06
BT
64.01
71.44
74.34
REX
65.19
71.72
74.08
UBA(ours)p = 0.5
64.44
72.10
74.48
75 (25%)
150(50%)
300(100%))
SS
74.84
76.96
78.10
CS
75.99
77.79
79.10
CLR
72.86
74.88
76.91
SGD
IC
75.28
77.37
78.79
BT
75.71
77.48
78.66
ImageNet
REX
75.28
77.03
78.46
UBA(ours)p = 5
76.00
77.99
79.32
-ResNe150
SS
74.64
77.51
79.01
CS
75.68
78.10
79.04
CLR
74.93
77.20
78.76
Adam W
IC
76.38
78.
79.21
BT
76.10
78.14
79.05
REX
76.42
78.28
78.86
UBA(ours) y = 76.57
78.37
79.38
Table 10: Cross-ip ablation studies.
UBA
Dataset
Network
Epoch
Optimizer
kp=025
4p=0.5
6==1
2==2.5
yp=5
p==0
SGD
75.10
75.89
76.31
76.40
76.68
75.64
CIFAR100
ResNet34
60
Adam
72.10
71.34
71.54
71.21
70.40
69.01
SGD
78.35
78.75
78,95
79.24
79.32
79.40
ImageNet
ResNet50
300
AdamW
79.26
79.38
79.12
79.10
78.72
78.47
behavior mimics preconditioning, implicitly lowering the condition number of the optimization landscape.
Our theoretical framework (Proposition T establishes a connection between parameter &p and the condition number of the landscape around local minima: smaller yp is favored for well-conditioned problems (low condition number), while larger &o is beneficial for large condition number scenarios. Since preconditioning effect of AdamW naturally mitigates ill-conditioning, is preferred. In contrast, SGD lacks such adaptive mechanisms, thus larger is more suitable for this situation. This alignment between theory and experiment underscores the importance of tailoring yo to the optimizer's characteristics. Thus, recommend selecting higher values of ip when
schedule with smaller
42

---

Table 11: Performance across different initial learning rates.
Validation Accuracy
Dataset
Network
Schedule
1r=0.1
lr=0.01
Ir=0.001
77.28:10.240
73.48=0.4227
67.9811..143
74.11+0.7690
74.66-0.5888
69.18-+0.736
CLR
74.77+0.1880
74.26=0.3523
68.84-+04613
CIFARI00
ResNet34
1C
77.73=0.18 1868
15.80=0.1701
69.20=t0.2152
BT
78.53:10.0751
77.1110.1126
16.
REX
78.11:10.1124
77.40
69.88:10.1253
UBA(ours)
79.02:+0.2838
77.59=0.3522
70.50=0.0862
facing challenging optimization difficulty, while employing lower yp values in scenarios with smoother optimization difficulty.
E.8
Performance across different initial learning rates
Since each schedule will have a different optimal learning rate in practice, to thoroughly address this concern, conducted additional experiments to evaluate the performance of all schedules with different learning rates.
We conducted auxiliary experiments on CIFAR100 SEFAR100-ResNee ResNet34 for runs. For computational efficiency, these experiments were run for slightly fewer epochs than the main experiments, with results reported over three independent runs. Through current observation and search, we identified 0.1 as the proper initial learning rate for most schedules. use 0.01 and 0.001 to confirm this choice maximizes validation accuracy across methods. Additional smaller larger values such as 0.3 or 0.0001 were explored in preliminary studies, showing consistent trends but lower peak accuracy. Thus we do not choose them for comparison.
The experimental results are listed in the following table IT We report results of different base learning rate and compare the performance with optimal learning rate of each schedule.
E.9 Performance across different periods
Our schedule has periodic phase-based learning rate adjustment setting, where the learning rate at the k-th phase dynamically determined by the k-th local minimum of the loss landscape. In the original conception, this design captures the optimization dynamics around successive local minima, with the hyper-parameted related to the difficulty of each phase. Specifically, &p controls the trade-off between exploration (large steps) and exploitation (small steps) within each phase.
To validate our scheduling strategy, we conduct experiments by varying K. We observe that when roughly setting ço constant in each phase, performance degrades as training progresses. This suggests that a fixed yp strategy cannot adapt to the changing landscape of multi-phase scheduling. Since the optimization difficulty should decrease during training process, we drop yp as phase increases. By finely evaluate jo in each phase, it achieves performance improvement, confirming the need for dynamic control. It makes sense since multi-phase captures the dynamic features of the loss surface more finely, it needs careful selection for ş,, where reflects optimization difficulty. These results highlight fundamental trade-off: when &p values per phase for multi-phas scheduling can be finely evaluated, multi-phase scheduling better captures the loss landscape's 's non-stationary behavior than single-phar phase scheduling. However, selecting optimal oo values per phase for multi -hase remains non non-trivial, which motivates future work on automated landscape-awe aware yp tuning.
43

---

Table 12: Performance on multi-phase version UBA.
Multi-phan
Dataset
Network
Original
5
76.54
75.37
75.05
CIFAR 100
ResNet34
76.68
5 0.8
76.31
76.60
76.45
Related work
Budgeted training Researchers face significant challenges in achieving optimal model performance under fixed hardware and limited time. To address this, budgeted training has emerged as a broad research domain focusing on optimizing performance while minimizing resource usage. This research spans multiple aspects, including computation efficiency, model compression, training stability, convergence improvement, and optimization [T.. By strategically integrating efficient training approaches, budgeted training advances cost-effective model development, offering pathways to achieve high performance under realistic constraints.
Budgeted training can be divided from three main perspectives: data, model, and optimization. From the data perspective, budgeted training emphasizes the allocation of resources between the dataset and the algorithms that process it, such as the balance between the model size and the amount of data 2 24 28). From the model perspective, budgeted training searches the optimal number of model parameters within the given compute budget [17 26 31 38). Beyond data and model perspective,
optimization based methods guide budgeted training based learning rate schedules 5 30 43 44
batch size 17 and other weight averaging method [26] 27).
Among these three aspects, optimization baasid methods particularly aligned with the objectives of dedgeted-iteration training. this domain, learning rate scheduling based methods are particularly aligned with the objectives of budgeted-iteration interation training. Smith et al. 43 propose new learning rate schedule, named cyclical learning rates (CLR). It improves accuracy in fewer iterations without tuning introduce alternative setting of propose the reflected exponential schedule (REX) via profile and sampling fashion. Learning rate based approaches achieve robust and high-peu high-performing results under various lengths of training iterations, which corresponds to our purpose in this work, i.e. budgeted-iteration training. Besides, this approach plug-and-play, requiring no substantial alterations to the underlying model structure, making it readily adaptable to various deep network frameworks. Therefore, we explore the proper learning rate schedule to achieve budgeted-iteratio training.
and is relevant super-converrence phenomenon 44 et al. (0
existing learning rate schedules for budgeted training. Chen et al. S
Learning rate schedule The learning rate plays a pivotal role in controlling the non-convex optimization process during network training. Generally, the learning rate is gradually adjusted progressively alongside the iterations, which can be represented as 17 = g(t) where g is the schedule function which control the variation of learning rate, t is the current iteration and 17o is the initial learning rate.
Learning rate schedules have been extensively studied to improve training efficiency and model per- formance. The common scheme is the step decay schedule. Its schedule function can be represented as g(t) = dle/ts.. where f, is the predefined decaying step and (d4 2 d,+1) is predefined decaying scalar. A typical instance decreases the learning rate by decaying scalar 0. after 50% epochs and by decaying scalar 0.01 after 75% epochs (21). Then Loshchilov et al. [32] observe that sharp decreases may prevent models from escaping local minima. Thus they propose cosine schedule function } (1 + cos( (x) )), where is the predefined stage restart the learning rate schedule, which has proven effective in transformer training. In addition, cosine schedule is commonly-used schedule nowadays.
Pan et al. [36] propose Eigencurve, and demonstrate that minimax-opting convergence can be achieved for quadratic objectives when the Hessian's is eigenvalue distribution exhibits substantial skewness. Pan et al. 37 further introduce a learning rate schedule called elastic step decay as " /2 if E (1-7) [1-r+)T, (1 pk+1)T) to accelerate the pre-training for Berts, where and are hyper- that control learning rate interval. Defazio et al. [T] developed the theoretically-
44

---

grounded framework for learning rate scheduling, establishing optimal linear decay schedules and refinements through rigorous mathematical derivation. Their work additionally provides the most extensive empirical evaluation of scheduling approaches to date. Hu et al. [25] propose Warmup-
(eab)-))eay(WSD) for large model pretraining and promoting continuous training. WSD consists of three consecutive phases: warmup, stable constant, and decay. Luo et al. 34 discover new schedule that outperforms the cosine schedule, resembling WSD but with lower final loss.
Adaptive learning rate Compared to schedule methods, adaptive learning rate based methods focus on the learning rate adaptation based on local loss landscape statistics. Typical methods includes AdaGrad [1], Adadelta [54), RMSprop [23] and Adam [29]. These methods have been shown stabilizing the training process while effectively optimizing learning rate. Then Loshchilov et al. [33 propose AdamW optimizer which improves the performance in the transformer training. Recently, Chen et al. [7 discovers Lion optimizer for more memory-efficieney, Xie et al. [52] propose the Adan optimizer which implements 50% faster training than Adam and Adam commonly-user networks. Despite these improvements, some studies suggest that adaptive methods may under- perform momentum SGD in terms of generalization 30 511. critical factor for budgeted-iteration -teratioo training.
Learning rate design is not only significant in general training, but also critical in audgeted-iteration training which still remains a topic of debate. Some analyses advocate for small, constant learning rates to ensure stability and convergence [12]. On the contrast, prevailing hypothesis suggests that large learning rates may facilitate crossing over sharp local minima the optimization landscape [56). Despite the lack of comprehensive theoretical explanations, range of learning rate schedules inspired by the above analyses as heuristic guidelines has been widely adopted in practice, using variable learning rates to budgeted- budgeted-iteration iteration training [0] In this work, explore learning rate schedule from optimization problem tailored to budgeted-iterann iteration training, aiming to balance iteration budget constraints and generalization.
45

---

Training loss
Validation loss (C4)
Validation loss (Books)
Validation loss (CC)
4.75
5.00
4.50
5.00
4.75
4.55
4.75
4.50
4.00
4.25
4.25
3.75
4.09
4.00
3.50
10
20
3.75
20
(peS2o)
(Reddit)
(Stack)
Validation (Wikipedia)
6.50
4.00
4.00
4.00
3.50
3.00
10
20
20
40
10
20
30
40
50
20
20
20
10
20
20
40
50
Validation loss (ICE)
Validation loss (M2D2)
Validation loss (Pile)
Validation loss (Wikitext)
4.80
5.00
104
IC.
4.50
4.40
3.50
4.00
6.00
4.00
Acc. (%)
humanities Acc. (%)
Acc. (%)
Acc. (%)
27.5
24
24
20-
25.5
27-
26.0
25.5
22
100
20
30
40
20
30
50
10
20
30
40
50
10
20
30
40
50
MMLU Acc. (%)
HellaSwag (%)
SciQ Acc. (%)
ARC- ARC-Challenge Acc. (%)
25.5
100
24.5
26.0
25.5
40
ARC-Easy Acc. (%)
PIQA Acc. (%)
WinoGrande Acc. (%)
OpenbookQA Acc. (%)
45-
58
27.5
56
27.0
50.0
26.5
10
20
30
40
50
20
30
400
50
69.5
10
20
30
40
26.0
10
20
30
40
50
BoolQ
COPA
Commansense[A (%)
SocialiQA (%)
60.
0.2
40.5
60
46.0
58
39.5
45
56
35.0
54
10L
38.5
15.
50
20
Figure 7: Overall performance for language tasks on OLMo-36M.
46

---

Training loss
Validation loss (C4)
Validation loss (Books)
Validation loss (CC)
7.00
6,90
6.00
5.30
5.00
5.00
4.00
20
30
34
50
10
20
30
40
Validation less (peS2o)
Validation loss (Reddit)
Validation loss (Stack)
Validation loss (Wikipedia)
5.00
5.50
16a
6.00
5.00
5.00
4.00
4.00
4.00
2.00
Validation loss (ICE)
Validation loss (M2D2)
Validation less (Pile)
Validation loss (Wikitext)
7.00
6.00
6.00
CC1
164
K0A
5.00
5.00
4.00
4,00
3.00
20
30
MMLU
MMLU humanities
MMLU
26
25
24
26
24
24
10
10
MMLU Var Acc. (%))
HellaSwap Acc. (%)
SciQ Acc. (%)
ARC-Challenge Acc. (%)
26.0
28
1/1
23.5
25.0
27
40
26
23
24.5
10
20
30
ARC-Easy (%)
(%)
OpenbookOA (%)
62.5
51
28
40-
27
35
55.0
26
50.0
25
10
20
30
20
30
50
10
20
30
40
20
30
50
BoalQ Acc. (%)
COPA Acc. (%)
CommonsensaQA Acc. (%)
SociallOA Acc. (%)
60-
60.0
56
57.5
24
12
LCAA
55.0
15A4
Tokens (B)
of Tikens
#/ Tokens
Tolcens (B)
Figure 8: Overall performance for language tasks on OLMo-73M.
47

---

Training
(C4)
(Books)
(CC)
4.5
5.00
4.00
4.00
4.00
3.5
3.50
20
40
60
20
40
60
20
40
50
40
60
Validation loss (peS2o)
Validation loss (Reddit)
Validation loss (Stack)
Validation loss (Wikipedia)
4.50
4.00
10
4,00
1-0
3.0
3.50
3.00
2.00
3.00
Validation loss (ICE)
Validation loss (M2D2)
Validation (Pile)
Validation (Wikitext)
5.00
4.75
4.30
5.00
4.50
4.50
4.00
3.75
3.50
20
40
0b
3.50
20
45
60
20
40
00
3.00
40
60
MMLU Acc. (%)
MMLU humanities Acc.
MMLU social sci (%)
MMLU other Acc.
26.0
20.
25.5
25.0
26
22
102
26
MMLU Var Acc. (%)
HellaSwag Acc. (%)
SciQ Acc. (%)
ARC-Challenge Acc. (%)
34
32
30
26
25
40
03
20
43
20
80
40
02
ARC-Easy (%)
PIQA (%)
WinoGrande Acc. (%)
OpenbookQA (%)
64
30
60
40
58
BoolQ Acc. (%)
COPA Acc. (%)
Commonsensa@A Acc. (%)
SociallQA Acc. (%)
55
50
28
45
55
26
186
24
40
43
23
40
83
# of Tokens (B)
of Tokens (B)
of Tokens (B)
of Tokens (B)
Figure 9: Overall performance for language tasks OLMo--10M
48

---

Training
Validation
Validation (Books)
Validation
5.00
5.00
3.00
4.0
4.50
4.59
4.00
4.00
4.00
3.50
3.50
3.0
50
100
150
30
100
150
50
150
150
50
100
150
Validation less (pe52o)
Validation loss (Reddit)
Validation less (Stack)
Validation loss (Wikipedia)
4.50
4.00
4.0
3.50
3.00
3.50
2.50
3.50
32
150
150
130
Validation loss (ICE)
Validation loss (M2D2)
Validation loss (Pile)
Validation loss (Wikitext)
5.50
5.00
4.50
5.00
4.50
4.50
4.00
3.00
3.50
3.00
3.00
humanities
MMLU
MMLU Acc.
27
26
30
25
24
26
55
100
150
30
105
1.50
50
100
150
50
150
150
MMLU Var Acc. (%)
Hella HellaSwag Acc. (%)
SciQ Acc. (%)
ARC- ARC-Challenge Acc. (%)
30
45
40
27
40
24
25
50
100
150
150
150
ARC-Easy Ace. (%)
PIOA Acc. (%)
GrnnGrande Acc. (%)
OpenbookOA (%)
32
60
52
15:
30
BoolQ Acc. (%)
COPA Acc. (%)
SocialiQA Acc. (%)
60
35
30
180 of Tokens (B)
35
105
150
50
150
155
50
150
150
# of Tokens (B)
Tokens (B)
of Tokens (B)
Figure 10: Overall performance for language tasks on OLMo-300M.
49

---

CIFAR100 (SGD)
ImageNet (SGD)
80.0
77
2.50, 76.78%
10.00, 79.40%
Accuracy (%)
Accuracy (%)
79.0
78.5
78.0
74
@ value
@ value
(a)
CIFAR100 (AdamW)
ImageNet (AdamW)
80.0
Accuracy (%)
72.5
P-1 LOD, 72.34%
Accuracy (%)
4
0.50, 79.38%
72.0
79.0
71.5
71.0
78.0
0.25
0.25
2.5
value
value
(b)
Figure 11: Visualization of cross-pp performance.
50


Reviews:
[{"review_id": "Q1gOksm3hJ", "reviewer": "Reviewer_jwKN", "summary": "This paper proposes a Unified Budget-Aware (UBA) learning rate schedule designed to address training under limited computational resources and varying model scales. UBA requires tuning only a single hyperparameter, φ, which controls the learning rate step size and balances adaptivity with simplicity. Despite its minimal tuning cost, the method consistently yields marginal but robust improvements across a range of models, tasks, and training iteration budgets.", "questions": "1. Given the marginal improvements reported, how does the proposed UBA schedule compare to baseline schedules under more challenging or large-scale tasks where training budgets are more constrained?\n\n2. Can the authors provide statistical significance analyses (e.g., mean and standard deviation across multiple runs) to validate that the reported improvements are consistent and not due to random fluctuations?\n\n3. While UBA is claimed to generalize across model scales, have the authors considered evaluating it on a broader set of architectures (e.g., ResNet101, ViT, or Transformer-based models) to better support this claim?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths\n\n1. The paper provides a clear and elegant theoretical analysis of step size selection under finite-iteration training budgets, culminating in a unified learning rate schedule governed by a single hyperparameter, φ. This formulation is both principled and practical, offering a clean balance between theoretical rigour and ease of application.\n\nWeaknesses\n\n2. The performance improvements over baseline learning rate schedules are marginal across tasks and architectures, raising questions about the practical impact of the proposed method.\n\n3. Experimental results are reported using single-value metrics without accompanying variance or standard deviation. Given the small performance gains, statistical measures (e.g., mean ± std over multiple runs) are necessary to assess whether improvements are significant and reproducible.\n\n4. While the authors claim UBA generalises across model scales, the experimental validation spans a relatively narrow range—from ResNet-18 to ResNet-50—limiting the strength of the claim. Broader evaluation on larger or more diverse model families would strengthen this argument.", "was_revised": true, "final_justification": "After carefully review the response provided by author, they has sufficient empirical observation on different settings and models, although the UBA still marginal improvement from previous setting, but it consist improve on different tasks. Such empirical evidence might benefit future work in community. Given that I have improve my rate to 4.", "rebuttal": "*$\\mathbf{Weakness\\enspace 1}$:The performance improvements over baseline learning rate schedules are marginal across tasks and architectures, raising questions about the practical impact of the proposed method.*\n\n*$\\mathbf{Question\\enspace 1}$ Given the marginal improvements reported, how does the proposed UBA schedule compare to baseline schedules under more challenging or large-scale tasks where training budgets are more constrained?*\n\n**Answer:** We appreciate the reviewer’s observation. While the gains may appear modest at first glance, we argue that UBA’s consistent improvements with negligible computational costs across all tested scenarios demonstrate its significant practical value, particularly in large-scale applications. \nBelow, we clarify five key aspects supporting this claim:\n\n- **Unified improvements across all settings.** Unlike task-specific optimizations, UBA consistently outperforms baseline schedules  without any per-task tuning. Moreover, UBA reliably improves performance in every tested scenario across  **diverse architectures (CNNs, Transformers), varying model scales (from 36M to 300M parameters), multiple benchmarks (vision, language)**. This reliability aligns perfectly with our core thesis: UBA provides a universal and task-agnostic solution for training dynamics optimization.\n\n- **Extensive validation on LLM (OLMo).** Our experiments on OLMo-36M, OLMo-73M, OLMo-150M, OLMo-300M (Sec. 4.2)  address the scenario of constrained training budgets through four key dimensions: (1) The OLMo family is a **modern Transformer architecture**. Moreover, Transformer-based models are known to be sensitive to optimization dynamics, making consistent improvements more meaningful. It is representative of **\"challenging or large-scale tasks\"**.  (2)  The progressive scaling from **36M to 300M** parameters systematically tests the method's robustness across different budget constraints. (3) We conduct sufficient overtrain experiments. All models are trained with **D/N ratios >500×**, far above Chinchilla's 20× sufficiency threshold. It ensures stable final performance, not just early convergence. (4) **Our study encompasses 40-50 experimental runs, cumulatively consuming 30,000 A100 GPU-hours—a substantial investment for a research paper** (for comparison, DeepSeek-V3 utilized 3M H800 GPU-hours). This scale validates robustness and scalability.\n*Overall, UBA  consistently improves final accuracy across all OLMo models (see Sec. 4.2 and Appendix E.5).  As a result, this systematic evaluation on this modern Transformer architecture provides particularly compelling evidence for UBA's practical utility and practical impact.*\n\n- **Significant improvements in LLM training scenarios.** Although the performance gain of UBA may appear numerically modest on some tasks at first glance, our method achieves significant improvements across multiple benchmarks. For instance, it can be see in Figure 2 in the current paper, UBA matches baseline performance with 25B fewer tokens (125B vs. 150B) on HellaSwag Benchmark, representing a **20% reduction in training cost**. Moreover, the same target accuracy is achieved using 33% less data (100B vs. 150B tokens) on ARC-E Benchmark, while maintaining identical model configurations , representing a **33% reduction in training cost**. These improvements become particularly impactful when considering scaling laws, i.e. the logarithmic scaling of LLM performance with data volume.\n\n- **Performance gains v.s. Scale.** The significance of performance gains must be assessed relative to both model parameter scale and training token volume. What appears as a marginal improvement at small scales can represent substantial absolute gains when scaled to foundation model sizes. Although some metric improvements appear numerically modest in absolute terms, their scaled impact becomes substantial given the massive parameter counts and token volumes characteristic of modern LLMs. \n\n- **Negligible additional costs.**  Our improvements come with negligible trade-offs in computational overhead and memory footprint while maintaining training stability. \n\nWe will clarify these points in the revision to better highlight UBA’s generalization and practical utility.\n\n*$\\mathbf{Weakness\\enspace 2}$: Experimental results are reported using single-value metrics without accompanying variance or standard deviation. Given the small performance gains, statistical measures (e.g., mean ± std over multiple runs) are necessary to assess whether improvements are significant and reproducible.*\n\n*$\\mathbf{Question\\enspace 2}$: Can the authors provide statistical significance analyses (e.g., mean and standard deviation across multiple runs) to validate that the reported improvements are consistent and not due to random fluctuations?*\n\n**Answer:** We thank the reviewer for this important suggestion. To rigorously address the concern about random fluctuations, we provide the following statistical evidence from our experiments:\n\n**On vision tasks**: We conduct 5 independent runs for cifar10-ResNet18 and cifar100-ResNet34. Due to computational constraints and limited time within the rebuttal period, we use two independent experimental results for ImageNet-ResNet50.  We report the validation accuracy by mean accuracy ± standard deviation. Due to character limit, part of results are listed in the following table, demonstrating consistent performance. We will include these statistical validation in the next version of paper. \n\n| Dataset   | Network  | Method | 25% Budget   | 50% Budget  | 100% Budget  |\n|-|-|-|-|-|-|\n| CIFAR100|ResNet34|SS|71.092 ± 0.2676| 75.348 ± 0.1802|77.244±0.1808|\n|||CS|64.124 ± 0.1808|69.840 ± 0.7580|74.295±0.5958|\n|||CLR|73.336 ± 0.5635|73.412 ± 0.2835|74.748±0.2266|\n|||1C|73.384 ± 0.3227|75.036 ± 0.6382|77.860±0.2239 |\n|||BT|72.750 ± 0.2611|75.346 ± 0.2423|78.578±0.2705 |\n|||REX|73.642 ± 0.5635|75.136 ± 0.3110|78.040±0.1430  |\n|||**UBA**| **74.604 ± 0.1508**|**76.626 ± 0.2678**|**78.948 ± 0.2818**|\n| ImageNet|ResNet50|SS|74.738 ± 0.1386|77.235 ± 0.3861|78.559 ± 0.6435|\n|||CS|75.839 ± 0.2220|77.945 ± 0.2164|79.067 ± 0.0438|\n|||CLR|73.899 ± 1.4609|76.037 ± 1.6419|77.837 ± 1.3053|\n|||1C|75.831 ± 0.7792|77.743 ± 0.5218|78.997 ± 0.2956|\n|||BT|75.907 ± 0.2786|77.809 ± 0.4624|78.855 ± 0.2814|\n|||REX|75.853 ± 0.8047|77.658 ± 0.8853|78.662 ± 0.2828|\n|||**UBA**| **76.289 ± 0.4031**|**78.258 ± 0.1640**|**79.388 ± 0.0170**|\n\n**On LLM training**: Given the high cost of training large language models (LLMs), running multiple experimental trials is often impractical and rarely adopted in practice. **Our study encompasses 40-50 experimental runs, cumulatively consuming 30,000 A100 GPU-hours, which is a substantial investment for a research paper** (for comparison, DeepSeek-V3 utilized 3M H800 GPU-hours). This scale validates robustness and scalability.\n\nMore critically, experiment reliability can be assessed through the data-to-parameters ratio (D/N ratio). As Chinchilla scaling laws show, a D/N ratio of 20 is already sufficient for stable convergence.\n**Our LLM experiments far exceed this threshold** (e.g., 50B/36M ≈ 1389, 50B/73M ≈ 685, 75B/151M ≈ 497, 150B/300M ≈ 500), indicating that each parameter receives ample training data. This high D/N regime guarantees: (a) stable convergence, not just early convergence, and  (b) representative single-run results with minimal variance.\nThis approach follows modern LLM training standards. Seminal works like LLaMA-2 and OLMo similarly omit multi-trial reporting when D/N ratios exceed 200, as their publications show.\n\n*$\\mathbf{Weakness\\enspace 3}$: While the authors claim UBA generalizes across model scales, the experimental validation spans a relatively narrow range—from ResNet-18 to ResNet-50—limiting the strength of the claim. Broader evaluation on larger or more diverse model families would strengthen this argument.*\n\n*$\\mathbf{Question\\enspace 3}$: While UBA is claimed to generalize across model scales, have the authors considered evaluating it on a broader set of architectures (e.g., ResNet101, ViT, or Transformer-based models) to better support this claim?*\n\n**Answer:** We appreciate the reviewer's valuable feedback regarding the generalization across model scales. \nWe would like to clarify that the current manuscript **already includes comprehensive experiments on Transformer architectures (OLMo family)**. In detail, we have already conducted experiments on VGG16, ResNet-18, ResNet-34, ResNet-50 ( CNN architectures), **OLMo-36M, OLMo-73M, OLMo-150M, OLMo-300M (Transformer architectures with varying token number and depth/width configurations)**,  in the current version of paper. \n \nThe OLMo experiments provide particularly compelling evidence because they represent modern Transformer architectures widely used in practice. Moreover, the 8× parameter range (from 36M to 300M) exceeds the scale variation in our vision experiments. Furthermore, Transformers' known sensitivity to optimization makes consistent improvements more meaningful, and the consistent leading results on all OLMo sizes test true generalization. \n\n**Our study encompasses 40-50 experimental runs, cumulatively consuming 30,000 A100 GPU-hours—a substantial investment for a research paper** (for comparison, DeepSeek-V3 utilized 3M H800 GPU-hours). This scale validates robustness and scalability. Moreover, we conduct sufficient overtrain experiments. All models are trained with **D/N ratios >500×**, far above Chinchilla's 20× sufficiency threshold. It ensures stable final performance, not just early convergence.\n\n*Overall, UBA  consistently improves final accuracy across all OLMo models (see Sec. 4.2 and Appendix E.5).  As a result, this systematic evaluation on this modern Transformer architecture provides particularly compelling evidence for UBA's practical utility and practical impact.*\n \nTo further strengthen our claims, we will highlight these points in the revised manuscript."}, {"review_id": "PqzAIUsGeV", "reviewer": "Reviewer_DPSV", "summary": "This paper proposes a unified budget-aware learning rate scheduling strategy Unified Budget-Aware schedule, which aims to solve the problem of how to efficiently train models under a fixed training iteration budget. The author constructs a theory-driven optimization framework, designs a scheduling function that only requires a hyperparameter $\\varphi$ to control the worst case, and establishes the connection between it and the optimization condition number. Experiments on a large number of vision and language tasks have shown that UBA outperforms existing mainstream scheduling strategies under different models, tasks, and budgets, and has good generalization and robustness.", "questions": "Questions:\n\n1: Have the authors evaluated whether the local quadratic approximation still applies in the early stages of training?\n\n2: $\\varphi$ is fixed and related to the condition number. Have the authors considered adapting $\\varphi$ based on the optimization state?\n\n3: Although the authors have verified that it is effective on both SGD and AdamW, the scheduling strategy itself is currently independent of the optimizer. Have the authors considered combining the $\\varphi$ or $η_t$ design with the state term in adaptive gradient methods?", "limitations": "Partially. While the limitations are addressed in the conclusion, more discussion on the applicability of the quadratic approximation would improve the clarity and transparency of the work. No societal risks are apparent.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n1: This paper derives a UBA schedule based on a well-motivated min-max optimization framework with a rigorous approach.\n\n2: This paper proves the proposed scheduler is an exact solution to a specific min-max problem and analyzes its convergence in noisy environments.\n\n3: Experiments in vision and language tasks show that the method in this paper outperforms baseline models under various settings.\n\nWeaknesses:\n\n1: The method relies on a local quadratic approximation of the loss surface, which is valid for models with smooth structures, but its applicability is still questionable in transformers with small batch sizes in the early training stage.\n\n2: Although the authors achieved good results by fixing $\\varphi$, there is still a lack of robustness analysis on how to adaptively select $\\varphi$ or when $\\varphi$ is selected incorrectly.", "was_revised": true, "final_justification": "The authors provided solid experimental validation addressing the applicability of their method in early training stages, and offered a clear sensitivity analysis for the key parameter. I also appreciate the preliminary work on adaptive scheduling based on optimizer states. These responses address my main concerns, and I am raising my score to 4.", "rebuttal": "*$\\mathbf{Weakness\\enspace 1}$:  The method relies on a local quadratic approximation of the loss surface, which is valid for models with smooth structures, but its applicability is still questionable in transformers with small batch sizes in the early training stage.*\n\n**Answer:** We sincerely appreciate the reviewer's insightful comment regarding the validity of our method  when the loss landscape may exhibit less smooth characteristics. We fully agree that this requires careful analysis. However, our empirical results demonstrate that the applicability of our proposed schedule remains robust despite these potential challenges. Below we provide detailed explanations:\n \n- To prevent the approximation error from compromising practical applicability, we employ robust optimization design. Our approach employs minimax optimization to determine learning rates, optimizing for worst-case scenarios within the iteration process. First, this formulation mitigates early training approximation errors by operating on an upper bound. Second, the explicit solution avoids direct network value computation or local quadratic approximations.\n- We incorporate standard warmup phases (linear warmup from 0 to maximum LR). This industry-standard practice  effectively circumvents early-stage approximation issues before applying UBA. Empirical results on OLMo demonstrate UBA's effectiveness in early training stages.\n- Due to the complexity of NN loss landscapes (affected by architecture, data distribution, etc.), direct evaluation is intractable. We use our experiment results to demonstrate its applicability in Transformers in early training. Comprehensive LLM experiments validate UBA's performance across model sizes (36M-300M). The training curves in both **main text (Figure 2) and Appendix (Figure 7,8,9,10) provide visual confirmation of UBA's robust behavior on transformer based model in early training**. We list the following table to present quantitative comparisons with  baseline scheduling methods, showing consistent stability of our approach and competitive performance throughout early training phases.\n\n| Training | Size | Scheduler | HSWAG | ARC-C | COPA |\n|-|-|-|-|-|-|\n| 10%|36M  | BT|25.8  | 22.4  | 55.0 |\n|||CS    | 26.0  | 21.8  | 55.0 |\n|||UBA       | **26.4**  | **22.8**  | **58.0** |\n||73M|BT |**25.9**  | 23.3  | **56.0** |\n|||CS|**25.9**| 23.7  | 55.0 |\n|||UBA|**25.9**  | **24.4**  | **56.0** |\n||150M  | BT| 27.0  | 21.6  | 56.0 |\n|||CS| **27.5**  | 21.0  | 57.0 |\n|||UBA| 27.0  | **24.0** | **58.0** |\n||300M  |BT| 25.3  | 25.2  | 55.0 |\n|||CS| **25.4**  | **25.3**  | **57.0** |\n||| UBA  | 25.2  | 23.0  | **57.0** |\n\n- In the future, we will enhance UBA through $\\varphi$ universal flexibility for non-smooth adaptation. Dynamic adjustment for high-curvature regions will enable adaptive optimization \"modes\".\n- The revised manuscript will include  discussion on method applicability.\n\n*$\\mathbf{Weakness\\enspace 2}$:    Although the authors achieved good results by fixing $\\varphi$ , there is still a lack of robustness analysis on how to adaptively select $\\varphi$  or when $\\varphi$ is selected incorrectly.*\n\n**Answer:** We sincerely appreciate your valuable suggestions, which we have addressed from both **experimental** and **methodological** perspectives.\n \n**Experimental validation:**\n- Our original manuscript **already includes a comprehensive sensitivity analysis of the key parameter $\\varphi$**. Our experiments systematically evaluate $\\varphi \\in $ {0.25, 0.5, 1.0, 2.5,5,10} across different optimization scenarios.  **The results are plotted in Figure 11 and listed in Table 11 in Appendix E.7. In addition, Sec. 4.3 provides the analysis of $\\varphi$'s impact.** \n- **The experimental results demonstrate that, UBA consistently achieves optimal performance across different tasks (CIFAR100, ImageNet) and optimizers (AdamW, SGD)**. Moreover, from our parameter robustness analysis, we recommend selecting higher values of $\\varphi$ when facing challenging optimization difficulty, while employing lower $\\varphi$ values in scenarios with smoother optimization difficulty.\n- Notably, **even with suboptimal $\\varphi$ selection,** CIFAR100-SGD: Accuracy remains **above 76.3** (best 76.68) for $\\varphi \\in$  {1.0, 2.5,5} . ImageNet-AdamW: Accuracy stays **above 79.10** (best 79.38) for $\\varphi \\in$ {0.25, 0.5, 1.0, 2.5}. This confirms our method's **robustness to parameter selection, especially in the situation that when $\\varphi$ is selected incorrectly**, as you mentioned.\n \n**Methodological development:**\n- Regarding **adaptive $\\varphi$ selection** during training (as mentioned in Sec. 5), We've made **recent progress** by incorporating first and second-order moment estimates into the UBA optimization framework. Preliminary experiments show modest improvements over fixed-$\\varphi$ approaches. The results are listed in the following table.\n\n||| Epoch| Optimizer  | $\\eta_0$   | Original | Adaptive $\\varphi$ |\n|-|-|-|-|-|-|-|\n|||| SGD| 0.1  | 79.0| 79.15|\n| CIFAR100  | ResNet34 | 300   | AdamW     | 0.1| 75.94 | 77.42|\n||||| 0.01| 73.98| 75.98|\n||||| 0.001| 74.27| 75.16|\n\nThe results show that, although adaptive $\\varphi$ improves performance marginally under SGD optimizer, the results under AdamW show **promising potential of adaptive $\\varphi$ usage on UBA**. Moreover, current implementation requires further refinement, which we're actively pursuing. \n\nIn the  revision, we will highlight the parameter robustness results more prominently. Moreover, we  will include discussion of dynamic $\\varphi$ selection in the revised manuscript.\n\n*$\\mathbf{Question\\enspace 1}$: Have the authors evaluated whether the local quadratic approximation still applies in the early stages of training?*\n\nAnswer:  We sincerely appreciate the reviewer's insightful comment.  \n\n- Methodological Robustness: Our UBA learning rate is derived as an explicit solution from the optimization problem, avoiding direct computation of network-dependent values or local quadratic approximations in early training. Moreover, the robust min-max framework mitigates approximation errors of local quadratic approximation. \n- Empirical Validation: Due to the complexity of NN loss landscapes (affected by architecture, data distribution, etc.), direct evaluation is intractable. Instead, **we empirically validate UBA’s applicability in early training using OLMo**, a state-of-the-art Transformer model sensitive to optimization dynamics. **We plot training curves of validation losses and accuracy on OLMo between different sizes(36M, 73M, 150M and 300M)**. The training curves in both **main text (Figure 2) and Appendix (Figure 7,8,9,10)** provide visual confirmation of UBA's robust behavior on Transformer based models in early training . We list the validation accuracy in the 10% training stage as an example **in the table of answer to \"Weakness 1\", showing consistent stability of our approach and competitive performance throughout early training phases**. \n\nTo further strengthen our claims, we will highlight these results in the revised manuscript.\n\n*$\\mathbf{Question\\enspace 2}$: $\\varphi$ is fixed and related to condition number. Have the authors considered adapting $\\varphi$ based on the optimization state?*\n\n**Answer:** We sincerely appreciate your insightful suggestions, which resonate strongly with our planned future work directions. Given that this study establishes $\\varphi$ as an indicator of optimization difficulty, a natural extension is to investigate difficulty representations derived from optimizer states, including gradients, momentum terms, second-order moments, and temporal progression\n- Our current exploration focuses on two methodological advancements. First, **we develop a mechanism that incorporates time steps information into min-max optimization**. The numerical results generate a characteristic warmup-stable-decay pattern that we integrate with the UBA schedule. Initial validation shows  performance gains across some tasks. \n- Second, we introduce first and second-order moment estimators into the UBA framework. Preliminary experiments demonstrate improvements over fixed $\\varphi$ baselines. **Current  performance gain demonstrates promising potential of adaptive $\\varphi$ design**.\nThe findings will be will be included in the discussion section in our upcoming revision.\n\n*$\\mathbf{Question\\enspace 3}$: Although the authors have verified that it is effective on both SGD and AdamW, the scheduling strategy itself is currently independent of the optimizer. Have the authors considered combining the $\\varphi$ or $\\eta_t$ design with the state term in adaptive gradient methods?*\n\n**Answer:** We sincerely appreciate your valuable suggestions, which closely align with our research directions.\n- Building on UBA's empirical validation, we're extending the framework to incorporate adaptive gradient methods as additional constraints in our optimization framework.\n- Regarding dynamic $\\varphi$ selection during training (as mentioned in our Sec.5), We've made recent progress by incorporating first and second-order moment estimates into the UBA optimization framework. Preliminary experiments show modest improvements over fixed-$\\varphi$ approaches. The results are listed in the table (see the second table, in answer of \"Weakness 2\").\n- Adaptive $\\varphi$ demonstrates stronger synergy with AdamW than SGD, suggesting better compatibility with adaptive optimizers.\n- In the future, we will refine implementation for more robust gains, developing advanced moment estimation techniques and theoretical  analysis and broader validation.\n\n*$\\mathbf{Limitations}$: Partially. While the limitations are addressed in the conclusion......*\n\n**Answer:** We sincerely appreciate this constructive feedback. We will strengthen our discussion including the above answers of weaknesses, such as the quadratic approximation's applicability, in the revised manuscript."}, {"review_id": "CwmgbWsSrA", "reviewer": "Reviewer_LjTY", "summary": "This paper proposes a quadratic framework for deriving a family of learning rates schedules parameterized by a $\\varphi$ parameter. The schedules show strong results for ResNet training.", "questions": "N/A", "limitations": "Introducing an additional tunable parameter into your learning rate schedule is a clear limitation. I think it's discussed in reasonable depth.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "This paper has some interesting ideas but is very limited by a compressed presentation and the lack of proper citations to relevant work.\n\nThe idea of deriving a learning rate schedule numerically optimized for a quadratic has been explored before by Pan et. al (2021) in their Eigencurve paper. This Eigencurve paper is cited in the appendix, so the authors are aware of it. It should be in the body of the paper, and given the extreme similarity in the development framework and concept, the Eigencurve work needs to be properly credited. They also show that cosine-like schedule shapes arise from their framework.\nDefazio et. al. (2023)'s work on theoretically motivated linear decay schedules should also be mentioned, as it's the only other completely theory driven framework for learning rate scheduling, and is currently the largest comparison of learning rate scheduling approaches.\n\nThe experimental results in this paper are strong and encouraging, but it's hard to trust the results given the following limitations:\n - No error bars or standard errors are reported. ResNet experiments are famously noisy, and the differences between different schedules are often at the same level as the noise. Highlighting UBA as the winning method for 75 epoch ResNet-50 training with accuracy 76.00 compared to cosines schedule's 75.99 is not appropriate as the run-to-run variability is more than 0.2%!\n - The use of a fixed learning rate for all schedules doesn't give fair comparisons. This is a major limitation. Generally each schedule will have a different optimal learning rate in practice. The choice of this learning rate is not motivated, and may favor certain schedules over others.\n- Is the budgeted iteration method listed actually the linear decay schedule? this should be labelled as linear decay if so.\n- The introduction is too long and and largely restates the abstract. It could be shortened significantly to make more room for discussion of related work (I see you have a longer related work section in the Appendix that you could use some of).\n\nOverall I like the ideas in this paper and I think it's a good extension of the work of Pan et. al, and I would generally support acceptance if the authors commit to the improvements mentioned above, and a proper phrasing of their work within the context of the existing literature.", "was_revised": true, "final_justification": "I have increased my score. I hope that the authors can run more seeds for the final camera ready to get comprehensive error bars for the larger imagenet runs.\nAlso, a finer granularity learning rate sweep would be nice for cifar10.", "rebuttal": "*$\\mathbf{Comment\\enspace 1}$: ... Pan et. al (2021) in their Eigencurve paper. ... Defazio et. al. (2023)'s work...*\n\n**Answer:** We sincerely appreciate the reviewer's insightful comments regarding prior theoretical works on learning rate scheduling. We fully acknowledge that both Pan et al. (2021)'s Eigencurve and Defazio et al. (2023)'s linear decay framework represents seminal contributions to theory-driven schedule. \n**In the revision, we will cite the Eigencurve in the body of the paper. Moreover, we elevate discussion of Pan et. al (2021) Eigencurve and  Defazio et. al. (2023)'s work.** The revision is listed as follows.\n\n\"Pan et al. propose Eigencurve,  and demonstrate that minimax-optimal convergence can be achieved for quadratic objectives when the Hessian's eigenvalue distribution exhibits substantial skewness. Pan et al. propose Extremebert, and further introduce a learning rate schedule called elastic step decay to accelerate the pre-training for Berts. Defazio et al. developed the theoretically-grounded framework for learning rate scheduling, establishing optimal linear decay schedules and refinements through rigorous mathematical derivation. Their work additionally provides the most extensive empirical evaluation of scheduling approaches to date. \n\nWhile Eigencurve and our work both employ quadratic optimization, our approach differs in: (a) incorporating robust min-max optimization, (b) constructing an adaptive learning rate schedule controlled by a hyper-parameter \\varphi, (c) establishing the connection between \\varphi and optimization difficulty. \"\n\nWe will clarify these points in the revision.\n\n*$\\mathbf{Comment\\enspace 2}$: No error bars or standard errors are reported......*\n\n**Answer:** We sincerely appreciate the reviewer's important concern regarding experimental reliability. We provide statistical validation (mean ± std over multiple runs) across tasks:\n\nOn vision tasks. We conduct 5 independent runs for cifar10/100. Due to computational constraints and limited time within the rebuttal period, we use two independent experimental results for ImageNet-ResNet50 . We report the validation accuracy by mean accuracy ± standard deviation. Due to character limit, part of results are listed in the following table. The results demonstrate consistent performance of UBA. Full results will be tabulated in revised manuscript.\n\n| Dataset   | Network  | Method | 25% Budget   | 50% Budget  | 100% Budget  |\n|-|-|-|-|-|-|\n| CIFAR100|ResNet34|SS|71.092 ± 0.2676| 75.348 ± 0.1802|77.244±0.1808|\n|||CS|64.124 ± 0.1808|69.840 ± 0.7580|74.295±0.5958|\n|||CLR|73.336 ± 0.5635|73.412 ± 0.2835|74.748±0.2266|\n|||1C|73.384 ± 0.3227|75.036 ± 0.6382|77.860±0.2239 |\n|||BT|72.750 ± 0.2611|75.346 ± 0.2423|78.578±0.2705 |\n|||REX|73.642 ± 0.5635|75.136 ± 0.3110|78.040±0.1430  |\n|||**UBA**| **74.604 ± 0.1508**|**76.626 ± 0.2678**|**78.948 ± 0.2818**|\n| ImageNet|ResNet50|SS|74.738 ± 0.1386|77.235 ± 0.3861|78.559 ± 0.6435|\n|||CS|75.839 ± 0.2220|77.945 ± 0.2164|79.067 ± 0.0438|\n|||CLR|73.899 ± 1.4609|76.037 ± 1.6419|77.837 ± 1.3053|\n|||1C|75.831 ± 0.7792|77.743 ± 0.5218|78.997 ± 0.2956|\n|||BT|75.907 ± 0.2786|77.809 ± 0.4624|78.855 ± 0.2814|\n|||REX|75.853 ± 0.8047|77.658 ± 0.8853|78.662 ± 0.2828|\n|||**UBA**| **76.289 ± 0.4031**|**78.258 ± 0.1640**|**79.388 ± 0.0170**|\n\nOn LLM training. \n- **Computational Reality:** Given the exorbitant costs with single  run  for LLMs,  multiple experimental trials is often impractical and rarely adopted in practice. **However, our study encompasses 40-50 experimental runs, cumulatively consuming 30,000 A100 GPU-hours.** It is a substantial investment for a research paper (for comparison, DeepSeek-V3 utilized 3M H800 GPU-hours). This scale validates robustness and scalability. \n- **Empirical Reliability Criteria:** The reliability of such experiments can be effectively assessed through the data-parameters ratio (D/N ratio). As Chinchilla scaling laws saying, a D/N ratio of 20 is already sufficient. Notably, seminal works like LLaMA-2 and OLMo omit multi-trial reporting when operating at D/N ratios >200. \n**In our LLM experiments, the D/N ratios exceed this threshold** (50B/73M ≈ 685, 75B/151M ≈ 497, 150B/300M ≈ 500), indicating that each parameter has been allocated ample data. **This high D/N regime ensures: (a) stable final performance, not just early convergence, and  (b) representative single-run results with minimal fluctuation.** This practice aligns with contemporary  LLM training paradigms. \n\n*$\\mathbf{Comment\\enspace 3}$: The use of a fixed learning rate for all schedules doesn't give fair comparisons. ...*\n\n**Answer:** We sincerely appreciate the reviewer's insightful critique regarding learning rate selection. We address this concern through **methodological clarification** and **additional experimental validation**:\n\n**Methodological clarification**:\nOur original design rationale follows two points to ensure fairness:\n- According to Learning rate scaling law, the optimal learning rate scales as a power-law with the size of model, batch size and data. Thus, the predicted optimal learning rates fall within a narrow range and the peak performance differences with this narrow range may vary slightly.\n- To ensure rigorous experimental control, all schedules in one task use the same base learning rate. However, this rate was determined through multiple-run search to enable peak performance for most schedules. Such standardization isolates the scheduler's effect from other variables.\n\nHowever, **we fully agree with your insightful observation that each schedule will have a different optimal learning rate in practice.** Therefore, to thoroughly address this concern, we conducted additional experiments to evaluate the performance of all schedules with different learning rates. \n\n**Extended validation**:\nWe operate experiments on CIFAR100-ResNet34 for 3 runs. \nThrough current observation and search, we identified 0.1 as the proper initial learning rate for most schedules. We use  0.01 and 0.001 to confirm this choice maximizes validation accuracy across methods. Additional smaller or larger values such as 0.3 or 0.0001 were explored in preliminary studies, showing consistent trends but lower peak accuracy. Thus we do not choose them for comparison. \n\nThe experimental results are listed in the following table. We report results of different base learning rate and compare the performance with optimal learning rate of each schedule. \n\n| Dataset | Network | Method| lr=0.1| lr=0.01| lr=0.001 | Performance on optimal lr |\n|-|-|-|-|-|-|-|\n| CIFAR100| ResNet-34(100% budget) | SS|77.28±0.2401|73.48±0.4267|67.98±0.1473| 77.28±0.2401 |\n||| CS| 74.11±0.7690| 74.66±0.5888| 69.18±0.3736| 74.11±0.7690|\n||| CLR| 74.77±0.1861|74.26±0.3523|68.84±0.4613|74.77±0.1861|\n|||1C| 77.73±0.1868| 75.80±0.1701| 69.20±0.2152| 77.73±0.1868|\n|||BT (linear decay) |78.53±0.0751| 77.11±0.1026| 68.84±0.4613|78.53±0.0751|\n|||REX| 78.11±0.1124|77.40±0.4477| 69.88±0.1253|78.11±0.1124|\n|||**UBA**| **79.02±0.2838**| **77.59±0.3522**| **70.50±0.0862**| **79.02±0.2838**|\n\nFrom the table, we can see that compared with schedule performance on corresponding optimal learning rate, **UBA also achieve the leading place, demonstrating the superiority regardless of the base learning rate.**\n\nIn the revision, we will detail this analysis of varying base learning rate with the results table.\n\n*$\\mathbf{Comment\\enspace 4}$: Is BT actually the linear decay?...*\n\n**Answer:** We thank the reviewer for careful reading. The \"BT\" method is mathematically equivalent to linear decay scheduling. The original \"budgeted training\" naming reflected its application context, but we will standardize to \"Linear Decay\" in the revised manuscript to avoid confusion.\n\n*$\\mathbf{Comment\\enspace 5}$: The introduction is too long ...*\n\n**Answer:** Thank you for your suggestion. **We will shorten the introduction and incorporate relevant material from the Appendix' to strengthen the discussion of related work.** The main points of revision is listed as follows.\n\n\"Pan et al. propose Eigencurve,  and demonstrate that minimax-optimal convergence can be achieved for quadratic objectives......Their work additionally provides the most extensive empirical evaluation of scheduling approaches to date. \n\nWhile Eigencurve and our work...... optimization difficulty. \"\n(We respectfully note that **the complete revision details appear in our response to Comment 1**. We have omitted the detailed revision content here due to character limit while ensuring all substantive changes are documented.)\n\n*$\\mathbf{Comment\\enspace 6}$: Overall I like the ideas in this paper...I would generally support acceptance if the authors commit to the improvements mentioned above, and a proper phrasing of their work within the context of the existing literature.*\n\n**Answer:** Thank you for your positive feedback and constructive suggestions. We sincerely appreciate your support and will **carefully address all the requested improvements, including refining the paper’s positioning within the existing literature.**\n\nAs you recommend, we make the following improvements:\n- We have addressed all the points raised in your review, as detailed in our responses above.\n- We have revised the manuscript to better position our work within the existing literature. Specifically, we properly credit the work of Pan et. al (2021)  and  Defazio et al.(2023). We also clarify how our approach differs in handling adaptive learning rate  design and establishing the meaning of parameter.\n\n*$\\mathbf{Limitations}$: ...tunable parameter...*\n\n**Answer:** Thanks for this observation. We agree with your opinion. Thus, we conduct parameter analysis, demonstrating it robustness (Sec.4.3 and Appendix E.7). Moreover, we give a guideline for parameter selection (Appendix E.7). Furthermore, we discuss the potential of $\\varphi$ to achieve adaptive improvement (Sec. 5) ."}, {"review_id": "2pzq7T5K4x", "reviewer": "Reviewer_a6Zw", "summary": "This paper proposes a theoretical framework for a unified learning rate schedule that is budget-iteration aware. The resulting schedule adapts to the loss landscape of the model being trained and better leverages the available training budget across different training conditions. This is done by addressing the worst-case loss function and approximating the optimization trajectory across the loss landscape by approximating it with a sequence of local optima. The authors include a comprehensive theoretical analysis of the proposed framework and the related learning rate schedule. The experimental section extensively explores a wide range of datasets and domains.", "questions": "See weaknesses.", "limitations": "As listed in the weaknesses, the authors do not analyze the computational overhead from the proposed approach.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- The proposed method and framework are well motivated and based on a strong theoretical analysis and an intuitive concept. \n- The paper has extensive theoretical and experimental justifications and analysis of the proposed approach.\n\nWeaknesses:\n- The practicality of the proposed method is not explored or discussed. The authors mention that practitioners can easily leverage this method, but do not provide a discussion about the implementation.\n- The paper focuses on training efficiency, but the authors do not discuss the additional computational cost that the proposed learning rate schedule brings.\n- The abstract and the introduction of the paper address the work from a high-level perspective and include scarce details of the proposed approach. The readers would benefit from more introductory details of the proposed theoretical framework and the proposed learning rate schedule in the initial parts of the paper.\n- The text should be reviewed for typos and grammar. See the formatting section for more details on this.", "was_revised": true, "final_justification": "I increased my score. The author's rebuttal addresses the main concerns raised by my review, and the response to the other reviewers is also sound.", "rebuttal": "*$\\mathbf{Weakness\\enspace 1}$:The practicality of the proposed method is not explored or discussed. The authors mention that practitioners can easily leverage this method, but do not provide a discussion about implementation.*\n\n**Answer:** We thank the reviewer for raising this important point. **The implementation practicality has been thoroughly addressed in our submission**:\n\n- Our implementation **strictly follows the PyTorch official API**, i.e. it is inherited from the base class `torch.optim.lr_scheduler.LRScheduler` (Appendix E.3). It maintains full compatibility with all `torch.optim` optimizers. **The complete production-ready code is publicly available (see Appendix E.3 for link).**\n- We conduct parameter analysis for $\\varphi$. It provides guidance for $\\varphi$ selection, which makes practitioners leverage this method more easily.\nTo further clarify, we give an example snippet of implementing UBA schedule:\n\n`import torch`\n\n`from custom_schedulers import UBA`\n\n`scheduler = UBA(optimizer,  num_warmup_steps,    num_training_steps,    varphi,    max_lr=0.1,    min_lr=0)`\n\n`scheduler.step()`\n\nIn the revised manuscript, we will expand Appendix E.3 with more implementation details of UBA. \n\n*$\\mathbf{Weakness\\enspace 2}$:   The paper focuses on training efficiency, but the authors do not discuss the additional computational cost that the proposed learning rate schedule brings.*\n\n**Answer:** We appreciate the reviewer’s attention to computational efficiency. The proposed UBA schedule is designed to be **negligible computational overhead**.  Our **performance improvements come with negligible computational overhead**, which constitutes one of its key advantages:\n\nUBA introduces **only 6-15 scalar arithmetic operations per update**, which is compared to standard learning rate schedules (4-8 ops per iteration in standard schedulers like cosine annealing). \nSince *the operations of learning rate scheduling are **orders of magnitude smaller** than the FLOPs required for forward/backward passes in typical neural networks*, these extra operations for UBA is negligible computational overhead.\n\nMoreover, UBA requires minimal additional memory allocations or gradient computations required.\n\nWe will include a dedicated subsection in revised manuscript on Computational Efficiency.\n\n*$\\mathbf{Weakness\\enspace 3}$:  The abstract and the introduction of the paper address the work from a high-level perspective and include scarce details of the proposed approach. The readers would benefit from more introductory details of the proposed theoretical framework and the proposed learning rate schedule in the initial parts of the paper.*\n\n**Answer:** We sincerely appreciate this constructive suggestion to improve the paper's accessibility. In the revised manuscript, we will make the following enhancements to the abstract and introduction:\n\n- The  Unified Budget-Aware schedule framework unifies robustness to curvature variations caused by data distribution shifts, mini-batch sampling noise, architecture-induced landscape properties and optimization dynamics. To achieve this, we **introduce an implicit random variable to model uncertainty in the objective function**. Moreover, we **theoretically transform the problem into an eigenvalue formulation**.\n- The framework also employs a minimax optimization approach to guarantee performance under worst-case conditions, since **the minimization of learning rate is operated on an upper bound.**\n- The optimization \n\\begin{aligned}\n\\min_{\\eta_{1+T_{k}}, \\eta_{2+T_{k}},..., \\eta_{T_{k+1}}} \n&\\max_{\\lambda_{l}^{(k)} \\leq \\lambda_i^{(k)} \\leq \\lambda_{u}^{(k)}} \n\\prod_{t=1+T_{k}}^{T_{k+1}} \\left[(1-\\eta_{t}\\lambda_i^{(k)}) \\right]^2 \\\\\n\\text{s.t.} \\quad & \\eta_{1+T_{k}}, \\eta_{2+T_{k}},..., \\eta_{T_{k+1}} \\in [\\eta_{\\min},\\eta_{\\max}], \\\\\n& k=1,2,\\dots K\n\\end{aligned}\nis solved by gradient projection method. Multiple initializations are used to explore the solution space. Despite varying initial conditions, numerical solutions exhibit consistent patterns. **These patterns are captured through a parametric approximation.**\n- By approximating numerical solutions via **a universal parametric function** (with single hyperparameter $\\varphi$ linked to condition numbers), UBA eliminates per-network retraining while guaranteeing convergence.\n- The hyperparameter **$\\varphi$ governs the functional form of the scheduling trajectory.** Through $\\varphi$ modulation, the learning rate schedule can exhibit: **(a) Early slow decay → rapid descent → asymptotic convergence (b) Initial rapid drop → gradual decline → terminal smoothing.** Moreover, $\\varphi$ directly controls the **transition rates between these phases**, enabling precise adaptation to optimization dynamics.\n\n\n*$\\mathbf{Weakness\\enspace 4}$:  The text should be reviewed for typos and grammar. See the formatting section for more details on this.*\n\n**Answer:** We sincerely appreciate the reviewer's careful reading and valuable feedback. We have thoroughly addressed the concerns regarding text polishing:\n\n- We corrected all identified typos. Parts of changes are listed as follows.\n\nL24:  ...across various across a wide range of ... → ...across a wide range of ...\n\nL137: we aim to guarantees ... → we aim to guarantee ...\n\nL140: $\\xi$ denotes the data sampling on the dataset $D$→ Let $\\xi$ denote the data sampling on the dataset $D$.\n\nL194: compared in Appendix. → compared in Appendix E.8\n\nL195: for reasons. →for the following reasons.\n\nL227: is not large. → is manageable.\n\nL301: Experiments 11→ Figure 11.\n\n- We have implemented all corrections in the revised manuscript, which has been proofread by two  colleagues."}]

Author rebuttal:

Meta-review: This work introduces an original and novel approach for selecting and tuning hyperparameters for training neural networks under a constrained budget. The method is elegant, theoretically grounded, and demonstrates strong practical performance - which are important strengths of the paper.

Concerns were raised regarding the scale of the experiments, but in my view, an ImageNet evaluation is sufficient for a novel idea of this kind. More computationally intensive studies can naturally be deferred to followup work. The rebuttal effectively addressed the reviewers’ questions, and all acknowledged the paper’s merits: it is original, well-motivated, and empirically validated that should bring some attention at NeurIPS.
Decision: Accept (poster)