AdaLRS: Loss-Guided Adaptive Learning Rate Search for Efficient Foundation Model Pretraining
Authors: Hongyuan Dong
Dingkang Yang
LiangXiao
ChaoFeng
Ran Jiao
Venue: NeurIPS 2025 poster ()
Keywords: Foundation Model Pretraining
Hyperparameter Search
Learning Rate Search
Primary area: deep_learning

Abstract: Learning rate is widely regarded as crucial for effective foundation model pretraining.
Recent research explores and demonstrates the transferability of learning rate configurations across varying model and dataset sizes, etc. 
Nevertheless, these approaches are constrained to specific training scenarios and typically necessitate extensive hyperparameter tuning on proxy models.
In this work, we propose \textbf{AdaLRS}, a plug-in-and-play adaptive learning rate search algorithm that conducts online optimal learning rate search via optimizing loss descent velocities.
We provide theoretical and experimental analyzes to show that foundation model pretraining loss and its descent velocity are both convex and share the same optimal learning rate. 
Relying solely on training loss dynamics, AdaLRS involves few extra computations to guide the search process, and its convergence is guaranteed via theoretical analysis. 
Experiments on both LLM and VLM pretraining show that AdaLRS adjusts suboptimal learning rates to the neighborhood of optimum with marked efficiency and effectiveness, with model performance improved accordingly. 
We also show the robust generalizability of AdaLRS across varying training scenarios, such as different model sizes, training paradigms, base learning rate scheduler choices, and hyperparameter settings.

AdaLRS: Loss-Guided Adaptive Learning Rate Search for Efficient Foundation Model Pretraining
Hongyuan Dong, Dingkang Yang, Xiao Liang, Chao Feng', .Jio Ran
ByteDance Inc.
d_ Lousia@icloud. com, yangdingkag@bbteaaaeee. com, liangxiao.. .aaagxiao..lxxbytedddeee... ixxbbytedacce.com
chaofeng wwwbyzedanceeee......ee..a com, ranjiaodyytedance. com
Abstract
Learning rate is widely regarded crucial for effective foundation model pretrain- ing. Recent research explores and demonstrates the transferability of learning rate configurations across varying model and dataset sizes, etc. Nevertheless, these approaches are constrained to specific training scenarios and typically necessi- tate extensive hyperparameter tuning on proxy models. In this work, we propose AdaLRS, a plug-in-ann aat--ndd-pl play adaptive learning rate search algorithm that conducts online optimal learning rate search via optimizing loss descent velocities. We provide theoretical and experimental analyzes to show that foundation model pre- training loss and its descent velocity are both convex and share the same optimal learning rate. Relying solely on training loss dynamics, AdaLRS involves few extra computations to guide the search process, and its convergence guaranteed via theoretical analysis. Experiments on both LLM and VLM pretraining show that AdaLRS adjusts suboptimal learning rates to the neighborhood of optimum with marked efficiency and effectiveness, with model performance improved accord- ingly. We also show the robust generalizability of AdaLRS across varying training scenarios, such different model sizes, training paradigms, base learning rate scheduler choices, and hyperparameter settings.
Introduction
Learning rate (LR) is regarded as a critical hyperparameter for foundation model pretraining, e.g. Large Language Model (LLM), Vision Language Model (VLM), etc. With the development of foundation model pretraining techniques [1, 29, 46, 50, 12, 11, 16, 25), series of studies seek to find the optimal learning rate for effective pretraining [39, 49, 26, 35]. Although reducing the costs of hyperparameter searches to some extent, these methods are bound specific pretraining scenarios or necessitate assource-consumin hyperparameter searches on proxy models to take effect.
One line of work predicts the optimal learning rate settings directly via summarizing model perfor- mance dynamics w.r.f hyperparameter settings. Under certain model structure designs, the optimal learning can be expressed a function of series of training hyperparameters [24, 5, 26, 35), such as model size, batch size, training budget, etc. Nevertheless, the resulting correlation laws lack desired generalizability new training scenarios, where large-scale learning rate search experiments are required to establish the model performance dynamic patterns.
Another line of research explores transferring hyperparameter search results obtained from proxy models to larger ones. Tensor Program series work [48, 49] proves the zero-shot transferability of hyperparameter settings across model sizes for transformer models. As long the model design
EEmail corresponding
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

meets certain criteria, learning rate search on certain proxy models can be transferred to larger models directly, reducing the hyperparameter search cost significantly. However, the learning search process on proxy models is still time- and resource-consumiig techniques can applied to accelerate the search process [4, 41, 28], large number of independent runs are still required to approximate the optimum, resulting in additional training resource overhead.
Although auto hyperparameter learning
In this work, we propose AdaLRS, a plug-in-anddplas learning rate search algorithm that optimizes the loss descent velocity to find the optimal learning rate in a single run. We formulate the optimization of training loss and loss descent velocity w.r.t LR in foundation model pretraining tasks as convex problems, and show that they share the same optimum by experiment results. Based on these observations, AdaLRS adjusts the base learning rate sequentially according to training loss dynamies to approximate the optimum. We provide theoretical proof for the convergence and geometric error decay of AdaLRS,, and show that AdaLRS effectively tackles various foundation model pretraining tasks. It adjusts inappropriate LRs effectively in single runs, and improves model performance significantly for different model sizes, training paradigms, and base scheduler choices.
To summarize, our contributions can be listed as follows:
We provide theoretical and experimental analysis to demonstrate that the foundation model pre- training loss and its slop w.r.f. LR are convex and share the same optimum.
We propose AdaLRS, an learning rate search algorithm which performs online LR adjustment to optimize loss descent velocity, approximating the optimal learning rate in a single run. We provide theoretical analysis for its convergence and the geometric error decay in the search process.
We conduct experiments in both LLM and VLM pretraining tasks with varying base learning rates. AdaLRS not only demonstrates marked effectiveness finding the optimal LR and improving model performance, but also exhibits desired generalizability in different pretraining tasks.
2 AdaLRS
In this section, we provide formulations for the proposed AdaLRS algorithm and present proof of its convergence and geometric error decay in the optimal LR search process.
2.1 Formulations of AdaLRS Algorithm
Previous work has discussed the convexity of training loss optimization w.r. learning rate set- tings [38]. In this work, we further hypothesize that the optimization of the training loss and loss curve slope in foundation model pretraining tasks are both near-convex and share the same optimal learning rate, which is supported by the experiment results shown in Section 2.2. In this way, can approximate the optimal learning rate by optimizing the velocity of loss descent within a single run.
Formulation.
Denoting the learning rate at training step as Te. and the LR upscaling and downscaling factors as a' and 8", we formulate the workflow of AdaLRS as follows:
a'n if o( > v(n)) + 2e
(loss slope increases 1), I'm if of mp) < vmn) - 2 (loss slope decreases 4);
Qc+k
(I)
otherwise.
v(-) indicates the estimated loss curve slope obtained from k-step window, while stands for the estimation error between u and the true loss descent velocity V. a' and Ø are rectified LR scaling factors which satisfy a' max(X a, 1), independent real numbers and À E (0, 1) is a decay factor. We validate the multiplicatively in- dependent design of LR scaling factors (for all integers m, n, a 3m Appendix B.
m.1()*0.1) where a, 3 > are two multiplicatively
n 0) in
Workflow. During model training, AdaLRS monitors the loss curve slope with the least squares method [7], and attempts to upscale the learning rate when the loss curve slope decays. After the upscaling adjustment, we compare the loss curve slope with that before upscaling. As shown in Equation 1, if the estimated loss slope increases more than 2e after the adjustment, the upscaling is regarded as valid and the adjustment will be retained. On the other hand, once the validation fails

---

under condition m:) < (m.) - 2e, the upscaling adjustment is reverted, with downscaling factor applied the base learning rate. As a result, AdaLRS able to conduct an online search for the optimal learning rate within single run, performing marked efficiency and effectiveness.
We also introduce several tricks to improve the stability of the Ad algorithm. In the LR upscaling process, we adopt an early stopping operation if the training loss rises over the largest loss value in the loss records, followed by several trial training steps performed after upscaling. This is to ensure that the loss slopes before and after the trial adjustment can be compared fairly at similar loss levels. Additionally, the learning rate search process is restricted to certain step ratios in the early training stage, ensuring a stable LR decay process for pretraining. To tackle extremely high learning rate settings, we also introduce boundary condition that lowers the LR if the loss increases for two consecutive windows. We refer to Appendix A for a detailed formulation of AdaLRS in pseudocode.
2.2
Convergence Analysis
2.2.1 Theoretical Hypotheses
We make two hypotheses to prove the convergence of the proposed AdaLRS algorithm.
Hypothesis The optimization of the training loss w.E.t learning rate foundation model pretrain-
ing is convex, and share the same optimum with loss descent velocity optimization. Let V (T) and L(n) be the loss descent velocity and loss value function w.r.. the learning rate 17, this hypothesis means that there exists an optimal learning rate 1ÿ" so that:
and
0L < 0,
Vn < n¹
Zm
(2)
OV
0 and
0L lm
0,
UU 17
This is a relatively strong assumption, and therefore we conduct both theoretical and experimental analyses to support it. Consider simplified foundation model pretraining task with constant learning rate and SGD optimizer, we formulate the update rule for model parameter & as:
+++ = the T,,,
where Le is the training loss at time step t. The expected loss descent velocity per step (using Taylor expansion) is:
E | =z x - CLip 2
where CLap is the Lipschitz constant of the gradient. When the learning rate is small, the first term dominates the expectation, and a smaller v) leads to smaller decrease in loss. When the learning rate is large, on the other hand, the second term cannot be neglected. It contributes to suppress the expected loss descent value. Differentiating w.r.t. 17 and setting to zero for the extreme:
① Om
(-n+ CLiP n7² 0 nj" 2
1
(mn
(5)
CLip
This shows that the loss descent velocity w.r.t. 17 is convex function. It is intuitive to infer that a higher loss descent velocity leads to a lower training loss, and vice versa, which proves that the optimization of training loss is also convex problem.
Apart from theoretical analyses, we also conduct small-scale pretraining experiments in LLM and VLM pretraining to verify this hypothesis. For L.M pretraining, we use Qwen2.5-1.55 and Qwen2.55 7B [43] models with randomly initialized parameters, and train them from scratch on 8M samples from the SlimPajama [42] dataset. A series of exponentially larger learning rates in [2e 6 Se 3] are applied to investigate the dynamics of training loss and loss curve slope under varying LR settings. For VLM pretraining,
conduct experiments on 2B model with the architecture described in SAIL-VL [16]. Approximately 8M data consisting of caption and OCR samples are used for model training, with learning rates configured in [2e-5,22 2.
The resulted training loss curves are shown in Figures ((a)(b with training loss and loss descent velocity dynamics w.r.f varying LR settings illustrated in Figures (d))e) and (g)())i), respectively. As Figures (f) shows, pretraining losses different training steps exhibit strong correlations with learning rate settings, forming series of convex curves with constant optima. To investigate
3

---

Protralning
LLM Pretraining
Pretralning
Tuan ng Daa
Teating 2tap
Tulning Stop
(a)
(b)
(c)
Pretraining Loss VS
7B Protraining Loss LR
VLM Protraining Loss
(d)
(e)
(l)
28 LLM Pretraining Loss Slope VS UR
7B LLM Pretraining Loss Slope VS LR
2B VLM Pretraining Loss Slope VS LR
Loge Conne
(h)
Figure 1: Training loss and loss descent velocity dynamics w.r.f. varying LR settings for LLM and VLM pretraining. Figures (()&()( show the training losses and LR trajectories with cosine LRS, while Figures (I) illustrate how training loss varies across different LR settings through the training process. Figures (g) are loss slope dynamics at varying loss levels, obtained from experiments with constant learning rates.
how learning rate influences the loss descent velocity, we conduct several sets of experiments with constant LR values. We show the correlation between loss curve slopes and learning rate settings in Figures (h))))) and the corresponding curves are not only convex, but also sharing the same optima as the training loss curves. Some data points Figures (g)) are not displayed because the corresponding inappropriate learning rate cannot optimize the model to such loss levels. These experiment results strongly support the validity of Hypothesis 1.
In practice, V is also influenced by the current loss value and optimization states. As described in Section 1, AdaLRS performs trial training steps before comparing loss slopes to eliminate the influence of loss levels, and we assume that the difference in optimization states is limited between consecutive training steps. As result, we present V as a univariate function in our proof.
Let vi denote the empirical estimate of ((nt), computed over a sliding window of training steps. Denoting the estimation error bound as i.e. Vt > 0, |0 - V(no) < e, there exists a sliding window length k such that e can be sufficiently small.
Hypothesis 2.
This guarantees that AdaLRS converges to a relatively narrow neighborhood of the optimum 17"
2.2.2 Convergence
Equipped with the above hypotheses, we seek to prove the following theorem:

---

Theorem 2.1. The learning rate sequence {me} generated by the algorithm converges almost surely to the e-neighborhood of the optimal learning rate Ne(n") _ {n: In n1' < e), i.e.
lim P(m 17" e)= =1. 1-000
(6)
Proposition 2.2. Tf, Te, aa 1)mc < <n- - E, 3T oo such that Tb4T moves towards 1j",, and vice versa.
Since we gradually push the the loss slope decay threshold to (detailed in Appedix A), t 0 will almost surely trigger the LR upscaling adjustment in finite steps. We denote the time step as T, with the loss descent velocity before after trial LR adjustment as UT and UT+ respectively. The LR upscaling adjustment is kept and only kept when UT4 vr > 2e, which can be reformulated as (nr++) <T+ V(nr) éT 2e, where TT and ET stand for loss slope estimatiin error corresponding steps. Substituting gives V (+TT+*) V (mr) > 2e-eT+ke eT > 2e |eT+* -lerl > which means the upscaling condition in Equation ensures an increase the true loss descent velocity. According to Hypothesis 1, the upscaling adjustment should be retained under this condition. For 77t, (8 8, > 11" + e cases, we can derive that trial LR upscaling adjustment gives lower loss descent velocity. A downscaling adjustment should and will occur under such circumstance.
Proposition 2.3. 37 oc. for Vt T the gap between np. and n² is bounded by the decaying LR scaling factors.
Mm, n¹ - <
T"' e'
d'
3
where B' max(X and a' max(Xa,)) 1) are decaying LR adjustment factor
In practice, the value of À is set relatively large, ensuring that the learning rate 7) converges to the neighborhood of ry* at the current LR adjustment magnitude before the scaling factor becomes too small. Since the desired LR adjustment is guaranteed by Proposition 2.2, we begin with X'=e < n'+e In this way, the learning rate 77, either oscillates within this range, or attempts to search mn-=33 o'=e| respectively for higher or lower learning rate the learning rate in [ te te a') and [ a' o
Ne 3
gr Br
values. Since we have te > 1f" +e always holds true, the LR upscaling attempts will trigger 8'
LR downscaling according to Proposition 2.2, maintaining the LR value given in Proposition 2.3. Similarly, learning rate downscaling attempts will be rejected by Proposition 2.2.
Proposition 2.3 indicates that n is bounded to the neighborhood of 17" by decaying LR adjustment factors. As the LR adjustment factor bound narrows, e.g. lim o' 1-000 3' = 1 and lim (-220 a² = 1, 7 will fall into the e-neighborhood of 17' eventually. Therefore, we have Theorem 2.1 proved.
2.3 Complexity Analysis
We introduce the following theorem to demonstrate AdaLRS's geometric error decay.
Theorem 2.4. There exists 7 E 10, such that for TJe & Ne(n"), |N+* - 171 S 7/m 17" when LR adjustment is triggered.
For The & (1)",, let %a E (2"--'ne, 1) for the learning rate upscaling process. Since y moves towards 17" in the upscaling process, the following inequation holds for the whole upscaling process:
Vt > 0, 17" a'T
ry" a'no
|M++ - 17' < 7/me - 1
(8)
17" 17. 17" 70
Similarly, "A can be selected from 33'm-n". -"" 1) for the LR downscaling process. Selecting
=
max(Taa To). Theorem 2.4 is proved directly.
Theorem 2.4 implies the geometric error decay of the AdaLRS algorithm bounded by y. Let R be the range of the learning rate search space. AdaLRS is able to approximate the optimal learning rate within (log R) adjustments. As a result, for foundation model pretraining tasks with data size larger than Olog R), AdaLRS reaches the optimum neighborhood within single run. The effectiveness of AdaLRS surpasses traditional auto hyperparameter search algorithms significantly, which often require extensive independent experiments to take effect.
5

---

Table I: Detailed hyperparameters for the main experiments. "Fit". "Large", and "Small" refer appropriate, too large, and too small learning rates, respectively. "BSZ" stands for batch size.
2B LLM
7B LLM
2B VLM
Hyperparameter
Fit
Large
Small
Fit
Large
Small
Fit
Large
Small
Leaming Rate
2x
2e-3
2e-5
2e
2e-3
2--5
Se-3
de-1
2--4
BS7./ Micro BS7.
1024/512
2048/512
2048/1024
Window Size k
2500
2000
1000
Data Composition
Detail Caption & OCR 0.1,0.4]
SlimPajuma Train Set
SlimPajama Train Set [0.1,0.35]
Search Step Ratio
[,.15
3 Experiments
3.1 Experiment Setup
Model Training. We conduct experiments on both LLM and VLM pretraining. Qwen2.5-1.5p
and Qwen2.5-7B [43] are adopted for LLM pretraining, with parameters randomly initialized via He initialization [21], Glorot initialization [18], and etc. We use a total of approximately 64M samples from the SlimPajama [42] dataset for LLM training from scratch, with all samples shuffled randomly. For VLM pretraining, on the other hand, we adopt the model structure of SAIL-VL [16]. with Intern ViT-300M [13] and Qwen2.5-1.5P adopted as backbone models. We use collection of detail caption and image OCR data to train the vision-to-language projector from scratch. Detail caption datasets are curated via similar recaption procedure as described in SAIL-VEE VL [16], while OCR data is collected from a series of opensource datasets [6, 45, 20].
To demonstrate the effectiveness of the AdaLRS algorithm, set different learning rates for each training task, i.e., learning rates too small, too large, and appropriate for pretraining. For each setting, we train baseline model and model optimized by AdaLRS for fair comparison. We set the upscaling factor a, downscaling factor 3, and decaying factor À as 3. 2, and 0.99 in all experiments for LR adjustment effectiveness. Approximately 120B and 160B tokens are used for LLM and VLM pretraining, with roughly 10,000 and 20,000 9 910B NPU hours consumed for 2B and 7B model pretraining experiments. A cosine learning rate scheduler [32] is applied in the main experiments. Detailed model training recipes are elaborated in Table 1.
Evaluation. To demonstrate the effectiveness of the proposed AdaLRS algorithm, we show the learning rate search and training loss dynamics of our foundation model training experiments. For LLM pretraining, we quantify the performance advantage of models trained with AdaLRS with final training loss and perplexity (PPL) computed on SlimPajama [42] train, validation, and test splits. We also conduct lightweight SFT on 6M samples from the Infinity-Instru dataset [27], and evaluate the downstream model performance on open-ended generation tasks such as Alpaca-Gen and KNIGHT-Gen [47]. For VLM experiments, however, there no widely accepted benchmark or metric for pretrained VLM evaluation. Therefore, we tune the pretrained VLM on 3M samples from Infinity-MM [19] stage3 data with all parameters unfrozen, and use a series of publicly available VLM benchmarks for model evaluation, such as LLa [30], MM [51], MMStar [10],
VQA [34], OCRBench [31], TextVQA [40], DetailCaps [17], etc.
3.2 Main Results
AdaLRS Approximates the Appropriate Learning Rate Effectively. As shown in Figure 2, the proposed AdaLRS algorithm (i.e., blue lines) pushes inappropriate learning rates (i.e., gray lines) towards the optimal ones (i.e., red dashed lines) effectively. For models trained with appropriate initial LR settings (Figures 2g)), AdaLRS conducts trial attempts to adjust the learning rates and eventually converges to the vicinity of the optimum. For excessively large small LR settings (Figures (()))))) the adjusted learning rates produced by AdaLRS demonstrate marked advantages in consistency with the optimum compared with baseline experiments.
Although exhibiting compatibility with varying LR settings, several undesired LR scaling operations can be observed in Figures (d))..) We attribute them to the aggressive upscaling and downscaling factor settings and the limited training steps for L.R adjustment, which can be mitigated by milder LR scaling factors or longer LR search processes.

---

28 LLM LR Trajectories (At LR)
28 LIM LR Trajectories O darge LAI
2B LIM LR Trajectories (Smail (R)
Training Stop
Taning Stop
Tran Stop
(a)
(b)
(c)
75 Trajectories
78 LIM Trajectories O.arge
75 LUM Trajectories (Small LR)
LR
Taxiee
Tra Training Step
Ba0 Bo0 Boi Tannin
200 mice Nolo
Traning Shep
(d)
(e)
(f)
28 VLM Trajectories LR)
VLM LR Vajectodes (Large LR)
VIM LR Trajectories (5mal
Bon dont chin 1865
A0n 2id0 Bid cides (7d0)
(h)
Figure 2: AdaLRS's Learning rate adjustment process in foundation model pretraining under differnt LR settings. "Fit LR" refers to learning rate appropriate for the pretraining task estimated by pilot study results. Dashed curves represent failed LR upscaling attempts.
Table 2: Evaluation results of our LLM pretraining experiments. We use "/" to separate performance scores of models trained with AdaLRS and baselines. Black and gray numbers indicate better or worse performance.
2B LLM
7B LLM
Benchmark
FiL
Large
Small
Large
Small
Training Loss
2.62/2.55
5.21/5.32
2.56/3.50
2.38/2.39
5.07/5.11
2.38/2.74
Training PPL
13.72/12.65
183, 283.28/205.00
12.88/33.21
10.84/10.88
158.54/165.0
10.76/15.10
Validation PPL.
13.51/12.42
183.94/ 983.94/204.97
12.66/32.69
10.67/10 10.
158.22/
10.61/15.30
Test PPL
13.51/12.43
183. 183.70/201.68
12.66/32.06
10.69/10.74
158.16/ 118.16/165.61
1665.61
10.63/15.32
Alpucn-Gen KNIGHT-GGen 10.13/11.29
17.29/18.56
7.11/6.09 06.09
17.10/15.40
21. 26/21/11.
6.35/5.55
21.15/20.66
3.96/2.02
10.81/8.36
13.62/13.35
4.08/3388
13.53/122.99
AdaLRS Accelerates Pretraining Convergence Significantly. As shown in the top-right corner of each subfigure in Figure 2, models trained with AdaLRS demonstrate significant advantages in training losses. For learning rates too small or too large, models trained with vanilla cosine LRS suffer from slow loss descent velocities, with severe training instability observed under certain circumstances (Figure 2(h).. AdaLRS not only improves training loss convergence effectively, but also aids such training instability problems. In experiments with appropriate learning rate settings, AdaLRS introduces slight loss spikes in model training, but the resulted training loss remains close to the baseline, exhibiting desired stability.
We also illustrate the acceleration ratio of our method for inappropriate LR settings. As shown in the training loss curves of Figures X())Xe)(((hOO) AdaLRS experiments reach the final training loss of the baselines at early training stages. For excessively small learning rates, AdaLRS surpasses the basclines at approximately 50% training steps, while more than 30% training costs can be reduced for
7

---

Table 3: Evaluation results for the our 2B VLM experiments. DetailCaps-481 used evaluate the basic visual understanding ability of the pretrained model, while other benchmarks scores are evaluated with the SFT model.
Setting LLaVABench
MMVet
MMStur
Du
OCRBench
TextVQA
DetailCups--550
Averuge
LR 39.5/38.5
34,58/32 02
48.67 49.53
77. 99 78. 00
7188735
64.89,16 690/63.77 63
G5. /6G, 30
56, 16/ 80
Large
86.8/35.7
$147/30 47..0.00
57.53/57.4 44.3/
12
631/606
60,32/58.08
49.02/10 07.08
38.96/47.666 6.67
Small LR
11.3/392
36. 15/30.23
18.67 19.20
71.75/ 777 47
T30/689
61.85/57.88
56.65 /53.51
37.5 17.31/33 77
WSD Scheduler LR Trajectories (Fit UR)
WSD Scheduler LR Tajectortes (Large LR)
WSD Scheduler LR Trajectories (Small LR)
UA VAW
AdaLRS's s Learning rate adjustment process in 2B LLM pretraining with WSD scheduler. We refer to Figure for denotation definitions.
Figure 3:
large LR settings. Although the LR upscaling attempts may introduce extra training costs, significant training loss advantages are still witnessed in models trained with AdaLRS under exactly the same training budgets.
It is also worth noticing that although AdaLRS adjusts learning rates reasonably for large LR settings, the training loss remains relatively high throughout the training process. We attribute this problem to the disruption of model initialization parameters due to the overly large learning rates. However, AdaLRS still reaches the vicinity of the optimal learning rate in a single run under such circumstances. which still improves the efficiency of the learning rate search process.
AdaLRS Enhances Pretrained Foundation Model Performance Markedly. To demonstrate the model performance advantage of training with AdaLRS, we provide PPL and downstream task evaluation results for LLM experiments in Table 2. AdaLRS not only surpasses baseline LLMs pretrained with suboptimal learning rates, but also achieves comparable model performance with Fit LR baselines. It is worth noticing that AdaLRS improves model performance significantly for excessively small LRs, even surpassing the Fit LR baselines in 7B LLM pretraining. This observation implies that starting with relatively small LR setting, AdaLRS can potentially achieve both the optimal learning rate and model performance for unexplored pretraining tasks within a single run.
For VLM pretraining, we show VQA benchmark scores of the SFT VLM models in Table 3. Models pretrained with the AdaLRS algorithm demonstrate remarkable performance advantages across most of the benchmarks, which encompass visual understanding tasks in both natural scenes and OCR- related tasks. Similar to experiments on LLM pretraining, we also observe that AdaLRS starting from small learning rate yields better performance than Fit LR baselines, which validates the effectiveness of the A algorithm in unexplored pretraining tasks.
3.3 Extending AdaLRS to WSD Scheduler
Since AdaLRS simply applies scaling factors on the learning rates produced by the scheduler and conducts LR search only in the early training stages, we can apply it to many mainstream schedulers seamlessly. Considering the generalizability of the conclusion and experiment efficiency, we use WSD [22] scheduler instead of Cosine scheduler in our experiments, to demonstrate the compatibility of AdaLRS with other schedulers. We use a linear decay function for the final 10% training steps in the implementation of the WSD scheduler. Experiment results are shown in Figure 3. Similar to the experiment results on cosine learning rate schedulers shown in Figure ((a)(b((c) AdaLRS adjusts unreasonable LR settings effectively, and shows desired stability for appropriate configurations.
3.4 Robustness Across Hyperparameter Settings
In this part, we investigate the model performance dynamics with different Adal RRS hyperparameters. We take 2B LLM pretraining as example and conduct experiments with small and large learning
8

---

Table 4: Model performance of 2B LLMs trained with different AdaLRS hyperparameters. a, B, and À refer to LR upscaling factor, downscaling factor, and decay factor, respectively. Baselines with AS disabled are shown with " as hyperparameters.
Small
Large
e/9 X
3/2
2/1.67
1,5/1443
2/1.67
2/1.67
3/2
2/1.07
1.5/1.43
2/1.67
0.90
0.30
0.90
0.95
0.9
0.90
0.99
0.90
0.9
Final LR
2,0e-5
3.Ge
3.1=
1.5e-1
3.1e
2.2-
2.0g
3.8e-1
5.2=-4
7.2e-1
7.9c-1
Training Loss
3.07
2.55
2.55
2.58
2.54
2.54
5.30
4.94
5.10
5.15
5.08
Validation
21.50
12.87
12.75
13:11
12.61
12.68
201.54
140.13
163.95
172.01
159.35
Test PPL
21.61
12.89
12.77
13:13
12.63
12.70
201.36
140.07
163.81
171.85
159:47
28 LLM Protrain wio Backtracking
2B VLM Continual Pretraining
28 VLM Continual Pretraining
Waining Stop
Tarning Stop
Train Stop
(a)
(b)
(c)
Figure 4: Ablation studies for the backtracking downscaling strategy (a) and the training dynamics of AdaLRS on VLM continual pretraining (b)
rate settings, respectively. As shown in Table 4, models trained with AdaLRS demonstrate marked advantages in training loss and evaluation PPI. compared to baselines, showing strong robustness across varying hyperparameter combinations. We also observe clear trend that, as the LR scaling and decay factors decrease, the final learning rate adjusted by AdaLRS tends to decrease for small- LR settings and increase for large-LR settings, respectively. In the experiments with a, 3, À = 2, 1.67, 0.99 and a, 3, À = 2, 1.67, 0.95, both settings yield nearly the same final learning rate, despite their different hyperparameters. This is because LR overshooting in the former triggers a downscaling adjustment, whereas the 0.95 decay factor in the latter prevents such adjustment.
3.5 Backtracking LR Downscaling Strategy Ablation
In this part, we evaluate the effectiveness of our backtracking LR downscaling strategy, which restores model training states before LR downscaling for training stability. We show model training dynamics w/o the backtracking strategy applied on the 2B LLM pretraining task in Figure 4, and the scaling factors are set the same mentioned in Section 3.1.
As shown in Figure 4(a), the training loss curve moves upwards during LR upscaling attempts. which may introduce instability in model parameter distribution. Although AdaLRS adjusts the learning rate downwards to approximately 38% of the baseline, the training loss remains relatively high in the entire training process. At the end of training, AdaLRS algorithm without backtracking yields higher training loss than the baseline. We refer to Figure 2(b) for training loss dynamics with the backtracking strategy applied. Loss upward movements and underlying disruptive parameter updates are eliminated by restoring model states. Comparing Figure 4(a) and Figure 2(b), the backtracking strategy improves the resulting model training loss by a large margin.
3.6 AdaLRS for Continual Pretraining
In this part, we explore the compatibility of the proposed AdaLRS algorithm with the continual pre- training paradigm. Different from pretraining foundation models from scratch, disruptive parameter updates may result in catastrophic forgetting and harm model performance markedly in continual pretraining [33]. We take VLM continual pretraining as an example, updating both the projector and vision encoder simultaneously to enable more model capacity used for visual understanding. It is a common practice to improve VLM model performance [2, 50, 11]. To be specific, we conduct 2B VLM continual pretraining from the model checkpoint after from-scratch pretraining, i.e., the baseline model in Figure 2(g). We locate the optimal learning rate in this stage at approximately 55 with 77 and 8e - used as small and large I.R, respectively. The same set of VLM pretraining datasets is used in this stage, and the experiment results are shown in Figure 4(b)
9

---

As expected, AdaLRS adjusts learning rates towards the optimum effectively in VLM continual pretraining. For the small learning rate experiment shown in Figure 4(b). AdaLRS improves the training loss by a large margin from 0.8851 to 0.8286. However, although Adal LRS downscales the large learning rate in Figure (c), the final training loss remains higher than 1.88. The L.R downscaling adjustments fail to improve model performance and even introduce severer loss oscillation, resulting in higher training loss. These observations show that AdaLRS performs promisingly for continual foundation model training with small initial learning rates, but can not eliminate the catastrophic forgetting problem caused by large learning rates.
4 Related Work
Optimal Learning Rate Prediction in Foundation Model Pretraining. Due to the prohibitively expensive pretraining costs, large number of research seeks to predict the optimal learning rate for large scale LLM pretraining. This line of work typically focuses on summarizing model performance dynamics as a function w.t.t. hyperparameter settings [24, 5, 8, 26], such as model size, batch size, training budget, etc. MM1 [35] further extends this approach to VLM pretraining, discovering the dependency of optimal LR on the number of model parameters. Although reducing hyperparameter search costs for larger scale model training, these methods require hundreds even thousands of pretraining experiments on smaller models to form an optimal learning rate expression. What's worse, the resulted optimal LR expressions are often restricted by certain model structure data composition designs to take effect, suffering from poor generalizability.
Transferring Hyperparameter across Model Sizes. Tensor Program series work studies the transferability of hyperparameter settings across varying model sizes. Recently proposed μPP [48] and uTransfer [49] prove the transferability of hyperparameter settings (including learning rate) across Multi-Layer Perceptron [37] (MLP) and Transformers [44, 14, 9]. Powered by μ Trannfer optimal learning rate search can be performed at proxy models and then transferred to larger ones [15, 22], reducing the search cost by a large margin. However, searching for the optimal learning rate on proxy models still be resourge-cnnsuming. and the optimal learning rate may shift across model sizes in complicated model designs.
Auto Hyperparameter Search. This line of work treats hyperparameter search as an optimization problem. Since learning rate a non-differentiable variable in model training. researchers propose to search for optimal learning rates via grid/random search [4]. Bayesian optimization [41], multi-armed bandit algorithm [28], cvolution-bad based method [23], reinforcement learning [36]. etc. Nevertheless, despite the search cost reduction compared with brutal search, these methods still require extensive independent runs to establish the underlying optimal learning rate prediction model.
It is worth noticing that recent work Hypergradient Descent (HD) [3] also conducts online LR adjustments. However, HD serves as a learning rate scheduler in model training and is only verified to be effective against constant schedulers in small neural networks. AdaLRS differs from HD in that our method is a LR search algorithm which is compatible with modern base LRSs, such as cosine and WSD schedulers, and is examined to be effective for modern foundation model pretraining.
Conclusions
In this paper, we propose AdaLRS, an online optimal learning rate search algorithm which optimizes the loss descent velocity to approximate the optimal learning rate. To validate this algorithm, we provide both theoretical and experimental analyzes to show the convexity of foundation model pretraining loss and its slope, as well as their shared optimum in LR. We also provide convergence and complexity analysis of AdaLRS to show its effectiveness and efficiency. Experiments in a series of foundation model pretraining tasks demonstrates that AdaLRS effectively adjusts inappropriate learning rates to the vicinity of the optimum in a single run and accelerates model convergence speed by a large margin. Starting from excessively small learning rates, AdaLRS achieves comparable and even superior performance with baseline models trained with near-optimal learning rates. AdaLRS is validated to improve model performance with varying model sizes, training paradigms, and base schedulers, exhibiting promising potential to be applied in unexplored pretraining tasks. We refer to Appendix C for the discussion of the limitations.
10

---

References
[1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Tlge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXivv2303.08774, 2023. 1
[2] Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308.. 0223308..299 /2966, 2023. 9
[3] Atilim Gunes Baydin, Robert Cornish, David Martinez Rubio, Mark Schmidt, and Frank Wood. Online learning rate adaptation with hypergradient descent. arXiv preprint arXiv:1703.04782 2017. 10
[4] James Bergstra and Yoshua Bengio. Random search for hyper-paramett optimization. The journal of machine learning research, 13(1):281-306 2012. 2, 10
[5] Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, Chengqi Deng. Honghui Ding, Kai Dong, Qiushi Du, Zhe Fu, et al. Deepseek Ilm: Scaling open-source language models with longtermism. arXiv preprint arXiv:2401.02955 2024. 1, 10
[6] Ali Furkan Biten, Ruben Tito, Lluis Gomez, Ernest Valveny, and Dimosthenis Karatzas. Ocr-idl: Oer annotations for industry document library dataset. In European Conference on Computer Vision, pages 241-252 Springer, 2022.
[7] Åke Björck. Least squares methods. Handbook of numerical analysis, 1:465-652, 1990. 2, 15
[8] Johan Bjorck, Alon Benhaim, Vishrav Chaudhary, Furu Wei, and Xia Song. Scaling optimal Ir across token horizons. arXiv preprint arXiv:2409.19911 19913,, 2024. 10
[9] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020. 10
[10] Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, al. Are the right way for evaluating large vision-language models? arXiv preprint arXiv:2403.20330, 2024. 6
[11] Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shen- glong Ye, Hao Tian, Zhaoyang Liu, al. Expanding performance boundaries of apen-sooucee multimodal models with model, data, and test-time scaling, arXiv preprint 1r200:2412.05271, 2024. 1,9
[12] Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites. arXiv preprint arXiv:2404.1682 2024.1
[13] Zhe Chen, Jiannan WL, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, Bin Li, Ping Luo, Tong Lu, Yu Qiao, and Jifeng Dai. Internvl: Scaling up vision foundation models and aligning for generic visual- lingul-iiggissic tasks. arXiv preprint arXiv:23/2.1422 2023. 6
[14] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.0458 04805, 2018. 10
[15] Nolan Dey, Gurpreet Gosal, Hemant Khachane, William Marshall, Ribhu Pathria, Marvin Tom, Joel Hestness, al. Cerebras-gpt: Open compute-optimal language models trained on the cerebras wafer-scale cluster. arXiv preprint arXiv:2304.03200 2023. 10
[16] Hongyuan Dong. Zijian Kang, Weijie Yin, Xiao Liang, Chao Feng, and Jiao Ran. Scalable vision language model training via high quality data curation. arXiv preprint arXiv:2501.055 05552,, 2025. 1, 3,
11

---

[17] Hongyuan Dong, Jiawen Li, Bohong Wu, Jiacong Wang, Yuan Zhang, and Haoyuan Guo. Benchmarking and improving detail image caption. arXiv preprint arXiv:2405. 19092, 2024.
[18] Xavier Glorot and Yoshua Bengio. Understanding the diffieulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intel- ligence and statistics, pages 249-256. JMLR Workshop and Conference Proceedings, 2010. 6
[19] Shuhao Gu, Jialing Zhang, Siyuan Zhou, Kevin Yu, Zhaohu Xing, Liangdong Wang, Zhou Cao, Jintao Jia, Zhuoyi Zhang. Yixuan Wang, et al. Infinity-mmn Scaling multimodal performance with large-scale and high-quality instruction data. arXiv preprint arXiv:2410. 18558, 2024. 6
[20] Ankush Gupta, Andrea Vedaldi, and Andrew Zisserman. Synthetie data for text localisation in natural images. In IEEE Conference Computer Vision and Pattern Recognition, 2016. 6
[21] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference computer vision, pages 1026-1034, 2015. 6
[22] Shengding Hu, Yuge Tu, Xu Han, Chaoqun He, Ganqu Cui. Xiang Long, Zhi Zheng, Yewei Fang, Yuxiang Huang, Weilin Zhao, al. Minicpm: Unveiling the potential of small language models with scalable training strategies. arXiv preprint vrXiv:2404.06399 2024. 8, 10
[23] Max Jaderberg, Valentin Dalibard, Simon Osindero, Wojciech M Czarnecki, Jeff Donahue, Ali Razavi, Oriol Vinyals, Tim Green, lain Dunning, Karen Simonyan, et al. Population based training of neural networks. arXiv preprint arXiv: 1,09116, 2017. 10
[24] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint ar xrXiv:2001.08361, 2020. 1, 10
[25] Weixian Lei, Jiacong Wang. Haochen Wang, Xiangtai Li, Jun Hao Liew, Jiashi Feng, and Zilong Huang. The scalability of simplicity: Empirical analysis of vision-language learning with a single transformer. arXiv preprint arXiv:2504. 10462, 2025.
[26] Houyi Li, Wenzhen Zheng, Jingeheng Hu, Qiufeng Wang. Hanshan Zhang, Zili Wang, Shi- jie Xuyang, Yuantao Fan, Shuigeng Zhou, Xiangyu Zhang, et al. Predictable scale: Part i-optimal hyperparameter scaling law in large language model pretraining. arXiv preprint arXiv:2503.04711 2025. 1, 10
[27] Jijie Li, Li Du, Hanyu Zhao, Bo-wen Zhang, Liangdong Wang, Boyan Gao, Guang Liu, and Yonghua Lin. Infinity instruct: Scaling instruction selection and synthesis enhance language models. arXiv preprint arXiv:2506. / /1, 2025. 6
[28] Lisha Li, Kevin Jamieson, Giulia DeSalvo, Afshin Rostamizadeh, and Ameet Talwalkar. Hy- perband: A novel bandit-based approach to hyperparameter optimization. Journal of Machine Learning Research, 18(18)):1-558 2018. 2, 10
[29] Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19453 19437, 2024. 1
[30] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jac Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024. 6
[31] Yuliang Liu, Zhang Li, Mingxin Huang, Biao Yang, Wenwen Yu, Chunyuan Li, Xu-Cheng Yin, Cheng-Lin Liu, Lianwen Jin, and Xiang Bai. Ocrbench: on the hidden mystery of ocr in large multimodal models. Science China Information Sciences, 67(12):220102, 2024.
[32] Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint 1rXiv: 03988, 2016. 6
[33] Yun Luo, Zhen Yang, Fandong Meng, Yafu Li, Jie Zhou, and Yue Zhang. An empirical study of catastrophic forgetting in large language models during continual fine-tuning. arXiv preprint 27877 2023. 9
12

---

[34] Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. Doevqa: A dataset for vqa on document images. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 2200-2209, 2021. 6
[35] Brandon McKinzie, Zhe Gan, Jean- -hinlppil Fauconnier, Sam Dodge, Bowen Zhang, Philipp Dufter, Dhruti Shah, Xianzhi Du, Futang Peng, Anton Belyi, et al. Mml: methods, analysis and insights from multimodal IIm pre-training. In European Conference Computer Vision, pages 10
304-323, Springer, 2024.
[36] Othmane Mounjid and Charles-Albert Lehalle. Improving reinforcement learning algorithms: Towards optimal learning rate policies. Mathematical Finance, 34(2):588-622 621, 2024. 10
[37] Marius-Constantii Popescu, Valentina E Balas, Liliana Perescu-Popeseee and Nikos Mastorakis. Multilayer perceptron and neural networks. WSEAS Transactions on Circuits and Systems, 8(7):579-588, 2009. 10
[38] Fabian Schaipp, Alexander Hägele, Adrien Taylor, Umut Simsekli, and Francis Bach. The surprising agreement between convex optimization theory and learning-rate scheduling for large model training. arXiv preprint arXiv:2501.18965 2025. 2
[39] Yikang Shen, Matthew Stallone, Mayank Mishra, Gaoyuan Zhang. Shawn Tan, Aditya Prasad, Adriana Meza Soria, David D Cox, and Rameswar Panda. Power scheduler: A batch size and token number agnostic learning rate scheduler. arXiv preprint arXiv:2408 233.3::990..3359 2024. 1
[40] Amanpreet Singh, Vivek Natarjan, Meet Shah, Yu Jiang, Xinlei Chen, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 8317-8326, 2019. 6
[41] Jasper Snoek, Hugo Larochelle, and Ryan P Adams. Practical bayesian optimization of machine learning algorithms. Advances in neural information processing systems, 25, 2012. 2, 10
[42] Daria Soboleva, Faisal Al-Khateeb, Robert Myers, Jacob R Steeves, Joel Hestness, and Nolan Dey. SlimPajama: A 627B token cleaned and deduplicated version of RedPajama. sl aimpaaama---627766
https: / http///wwcccccc///// cewebras......... net/blog/ 1---662bbtkkenceeaaaaeeee and -cleennd-and-dddulliicieee- teudd-ddupllaattdeeerrisoooo ofderrr ama,
2023. 3,
[43] Qwen Team. Qwen2.5: A party of foundation models, September 2024. 3, 6
[44] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017. 10
[45] Zilong Wang, Mingjie Zhan, Xuebo Liu, and Ding Liang. Docstruct: A multimodal method extract hierarchy structure in document for general form understanding. arXiv preprint arXiv:2010.116855 2020. 6
[46] An Yang. Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.1511 15115, 2024.
[47] Dingkang Yang. Dongling Xiao, Jinjie Wei, Mingcheng Li, Zhaoyu Chen, Ke Li, and Lihua Zhang. Improving factuality in large language models via decoding-ting time hallucinatory and truth- ful comparators. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 25606 256144, 2025. 6
[48] Greg Yang and Edward J Hu. Tensor programs iv: Feature learning in infinite-wddth neural networks. In International Conference on Machine Learning, pages 1 11727-11737. PMLR, 2021. 1, 10
[49] Greg Yang, Edward Hu, Igor Babuschkin, Szymon Sidor, Xiaodong Liu, David Farhi, Nick Ryder, Jakub Pachocki, Weizhu Chen, and Jianfeng Gao. Tensor programs v: Tuning large neural networks via zero-shot hyperparameter transfer. arXiv preprint Ar0iv:2003.0m 22.00666666, 2022.
1, 10
13

---

[50] Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, et al. Minicpm-v: gpt-4v level milm on your phone. arXiv preprint
2024. 1,9
[51] Weihao Yu, Zhengyuan Yang. Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang. and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. In International conference machine learning. PMLR, 2024. 6
14

---

A AdaLRS Algorithm
In this section, we show the workflow of the proposed AdaLRS algorithm in detail. As shown in Algorithm I, AdaLRS tracks the velocity of loss descent in pretraining tasks and computes the velocity as the loss curve slope via the least squares method [7]. Learning rate upscaling is triggered when the velocity decays, followed by several validation steps to measure the loss descent velocity after LR adjustment. We then compare the loss descent velocity after LR adjustment with that corresponding to similar loss values before upscaling. If the velocity becomes larger, the upscaling adjustment is retained; otherwise, the model and optimizer states are restored the exact step before LR upsealing. followed by LR downscaling adjustment instead. Restoring training states is designed stabilize the training process, where the upscaled learning rate may disrupt model parameter distribution during training. After several loops of learning rate adjustment, the resulting learning rate eventually falls in the neighborhood of the optimum.
Algorithm 1: Adaptive Learning Rate Scheduling (AdaLRS)
Input:
Initial learning rate ?u. Upscale factor 1, Downscale factor 3 (ged (a, (3) 1), Scale decay factor < 1, Loss slope inspection window size k. Loss slope decay threshold 0 < 0 < 1, Loss slope estimation error e, AdaLRS start and end step Estart: fend
Output: Optimal learning rate 17"
Function AdaLRS (r)o. O2 8, k; 0, start, lend)
Initialize: i <- 0, 7p < l/o. 0 < 0g
Initialize loss history H 0
while tswrr < < tend do
// 1ololect loss observations
Run training steps with {mh M+t}
Record losses {le,... 44.4
Update H. t HU f4e
// 2. Calculate loss descent velocity 4 least_ heast_square__ (rruure_fi(((1144__.___ // Check velocity degradation
(k: (++)))
if [H]] 2 2k and i < 0 then
// Trigger upscale phase
Run & up-scaling steps with Temp gradually up-scaled to oT);
// Find comparable historica cal records
Run another & validation steps with loss records Caew (1426+11 on letar)
Eet argmin IC Lnowl
DCH
// Compare velocities
if Unew Uny + 2e then 1]++ +- max (a>2; )m)
Keep up-scaled LR
else if Unow Unn The
2e then
Th+* (3
// Downscale after failed upscale
Restore model states to step +
else
Restore model states to step + k
H. 0
0.6g
else if [7[| 2k: and Uh, 0r 0 then
Th++ & (ax
// Downscale if loss rises consecutively
H. 0 0, On Restore model states step
else
7hth f- 7t
// Maintain current LR
0
9+1 2
1/ Narrow the decay threshold
- t+
return ?)
15

---

In the algorithm pseudo code, we omit the following details for presentation simplicity: 1) the learning rate 7/ updated by base scheduler according to the training step t, multiplied by a scaling factor from As; 2) during learning rate up-scaling, the training loss elevates be higher than the maximum loss value in loss history; 3) for corner cases where min ج Loow exceeds certain threshold, LR up-scaling or down-scaling are triggered if DCH the resulted training loss is higher or lower than all history records.
early stopping mechanism is triggered if
B Proof for the Validity of LR Scaling Factor Values
In this section, we demonstrate the validity of the multiplicatively independent design of L.R scaling factors.
Theorem B.1. Let a. be multiplicatively independent real numbers (for all integers m, n,
ath 8 = m = 0), for any positive rational number and E >> 0, there exist integers m, 2 O, such that:
10 g <.
We start by pointing out that In is irrational. This is rather straightforward because if ing is rational, then a = 8r, where p, q are positive integers, which contradicts with the multiplicatively independent design, and thus I In
must be irrational.
Then we apply Kronecker's Theorem. Let In For any real number c and é' > O, there exist integers m, n N such that |m0 satisfying:
In
é' Choose
Then there exist m, n N lm
In
In a
Inr
<e.
(10)
In 3
In
Multiplying through by In B, we get:
mlna - n In lnrl
é' In
(11)
Finally, let c' g B
for sufficiently small c', by the continuity of the exponential function:
m
r = |r| mm al -1
In
(12)
g
Using Equation 11, we bound:
a
r < |r| In =c.
(13)
8
which proves Theorem B.I.
C Limitations
Despite the promising optimal learning rate search effectiveness and performance improvement, AdaLRS fails achieve comparable results with appropriate LR baselines with excessively large LR settings. As discussed in Section 3.2 and Section 3.6, we attribute this problem to the disruptive parameter updates performed by large learning rates. This result restricts the application of AdaLRS on arbitrary LR settings. We recommend to apply AdaLRS with relatively small initial learning rates for optimal performance, which is validated to work smoothly in both from-scratch and continual pretraining, and leave this problem to be solved in future works.
Moreover, the design of AdaLRS prioritizes the generalizability to unexplored foundation model pretraining tasks, and therefore we do not study the extent to which AdaLRS can approximate the optimal LR in this work. We will explore the impact of the scaling factor designs and learning rate search ranges to demonstrate the fine- -he-ggaided convergence of AdaLRS in our future work.
We also point out that the near-optimal learning rates used in Fit LR experiments in Section 3 are obtained from grid search pilot experiments. We search for the optimal LR setting with exponentially larger LR in given range, such as [2e- 6, 4e 6, Se 6, 2e 5,... Se 3] for LLM experiments. Although sufficiently precise for our experiments, the resulting approximated search result may introduce slight variance in optimal LR estimation.
16

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justificcton right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No] ", it is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Our abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
17

---

Justification:
We discuss the limitations of our work in Appendix C.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approxinations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proof's
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: We provide theoretical proof and experiment results to validate our hypotheses and claims in Section 2.2.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We have included necessary details for experiments.
Guidelines:
18

---

The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open- source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome deseribe the particular way they provide for reproducibilityy In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
We will release the proposed benchmark and code after the paper is accepted.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips cc/
Beeeiii/guudes//oeeeeue httpbmmmsssoooooiyiy.ioc.. for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: //n cts.pss.cc/uubii/gguuuu. gesds//////eeeeeee/e /CodeSubmian ss cassoonooll. onoolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
19

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification:
We show all necessary training and test details in paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification:
We follow the same experiment settings with baseline works.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide compute resource details in Section 3.1.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
20

---

The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: //nerrip///.......// cc cccpublic/eeeeececcceecceeceeeee c lnes?
Answer: [Yes]
Justification: The research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There is societal impact that work may cause. Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that there no societal impact of the work performed.
the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploymeett of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our paper poses such risks.
21

---

Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All the creators original owners of assets used in the paper credited properly, and the license and terms of use explicitly mentioned and are respected properly. All datasets we use are from internet open source datasets under CC-BY licenses and are cited properly.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
Answer: [No]
Justification:
No new assets introduced in our paper.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
22

---

Answer: [NA]
Justification:
Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing. editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [No]
Justification: LLMs are not used in any of the core methods in our research.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non non-standard components.
Please refer to LLM policy (https: //w/rniris....// for what should or should not be described.
23


Reviews:
[{"review_id": "zxrWprv1ah", "reviewer": "Reviewer_6tFv", "summary": "The paper introduces AdaLRS, a plug-and-play algorithm for online adaptive learning rate search in foundation model pretraining. Unlike existing methods that rely on offline tuning or proxy models, AdaLRS optimizes the loss descent velocity to adjust learning rates dynamically during training. The authors provide theoretical proofs of convergence and geometric error decay and demonstrate the algorithm’s effectiveness across diverse settings—covering both LLMs and VLMs, varying model sizes, training paradigms, and learning rate schedules. Experiments show AdaLRS can significantly accelerate convergence, improve model performance, and generalize well across tasks.", "questions": "The experiments focus primarily on training loss, with less emphasis on downstream or task-specific metrics. Although loss and accuracy are strongly correlated, sometimes they may not align perfectly with final task-specific metrics in some applications.", "limitations": "Yes.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 3, "strengths_and_weaknesses": "# Strengths\n1. The paper is well-written and easy to follow.\n2. The method is simple but effective.\n3. The experiments are comprehensive covering both LLM and VLM backbones.\n4. The authors provide theoretical analysis along with good empirical performance.\n\n# Weaknesses\n1. The method introduces more hyper-parameters (i.e., upscaling factor α, downscaling factor β, and decaying factor λ). The authors did not provide sensitivity experiments on these hyper-parameters so it remains unclear if we still need to tune these. If so the method still needs extensive hyper-parameter tuning.", "was_revised": true, "final_justification": "The authors have addressed my concerns during the rebuttal and I will keep my score.", "rebuttal": "We appreciate the reviewer's insightful feedback. Please allow us to address the reviewer's concerns point to point. \n\n> **1. The choice of hyperparameters**\n\nThe default setting $\\alpha$=3, $\\beta$=2, and $\\gamma$=0.99 in the paper is chosen for simplicity, and we did not introduce human labor in hyperparameter tuning. \nTo validate the robustness of AdaLRS, we further conduct experiments under various hyperparameter settings. \nStarting from a 2e-5 LR for 2B LLM pretraining with WSD scheduler, we show the LR dynamics in the 64M-sample pretraining experimets: \n| Experiment |$\\alpha$/$\\beta$/$\\gamma$ | Step/LR/Scale        |                |                |                |                |                |                | Test PPL |\n|----------------|----------------|----------------|----------------|----------------|----------------|----------------|----------------|----------------|----------------|\n| Baseline | - | 100/2e-5/-  | -  | - | - | - | - | - | 43.77 |\n| A | 3/2/0.99 | 100/2e-5/-  | 20100/6e-5/↑  | 28100/1.8e-4/↑| 35100/3.6e-4/↑ | 42100/7.19e-4/↑ | 50100/3.6e-4/↓   | - | 16.94 |\n| B | 2/1.67/0.99 | 100/2e-5/-    | 20100/4e-5/↑    | 28100/8e-5/↑     | 36100/1.6e-4/↑      | 44100/3.2e-4/↑   | 51100/5.16e-4/↑ | 57100/3.1e-4/↓ | 18.79 |\n| C | 1.5/1.43/0.99 | 100/2e-5/-    | 20100/3e-5/↑    | 28100/4.5e-5/↑   | 36100/6.75e-4/↑  | 44100/1.01e-4/↑  | 52100/1.52e-4/↑ | - | 21.90 |\n| D | 2/1.67/0.95  | 100/2e-5/-    | 20100/4e-5/↑    | 28100/7.6e-5/↑   | 36100/1.37e-4/↑  | 44100/2.35e-4/↑  | 51100/3.09e-4/↑ | - | 19.58 |\n| E | 2/1.67/0.9 | 100/2e-5/-    | 20100/4e-5/↑    | 28100/7.2e-5/↑   | 36100/1.16e-4/↑  | 44100/1.7e-4/↑   | 51100/2.17e-4/↑ | - | 19.70 |\n\nIn this table, Exp Baseline, A correspond to the \"Small LR\" setting for 2B LLM pretrain in our paper, and Exp B, C are conducted with different LR scaling factor values. Exp D, E adopt smaller decaying factor, which suppresses the \"overshooting\" behavior observed Exp B. Overall, depite the varying hyperparameter settings, learning rates converge to the neighborhood of the optimum 2e-4 robustly.\n\nWe also choose the model checkpoints at 60,000 step (150,000 in total) to evaluate their test PPL, and the results show that AdaLRS achieves improved model performance over the baseline under various hyperparameter settings. \nWe will include the complete loss dynamic figures and evaluation results in the revised version of our paper. \n\n\n> **2. Downstream task performance**\n\nTo investigate the generalization of AdaLRS's improved pretraining loss, we further conduct a lightweight 6M-sample SFT with the Infinity-Instrcut dataset [1] for the 2B LLM experiments. \nWe evaluate the SFT model performance on open-ended question-answering tasks, such as Alpaca-Gen and KNIGHT-Gen [2], which aligns with real-world LLM applications. \nWe follow previous work [2] to use Rouge-1, Rouge-2, and Rouge-L as evaluation metrics.\n\n| Experiment | AlpacaGen_rouge1 | AlpacaGen_rouge2 | AlpacaGen_rougeL | KNIGHTGen_rouge1 | KNIGHTGen_rouge2 | KNIGHTGen_rougeL |\n|-------------------|:----------------:|:----------------:|:----------------:|:----------------:|:----------------:|:----------------:|\n| Small LR          | 24.03            | 5.41             | 15.40            | 10.02            | 1.23             | 8.36             |\n|   **+ AdaLRS**   | *28.77*            | *6.39*             | *17.10*            | *13.33*            | *2.95*             | *10.81*            |\n| Large LR          | 8.49             | 0.41             | 6.09             | 2.41             | 0.11             | 2.02             |\n| **+ AdaLRS**   | *10.14*            | *0.52*             | *7.11*             | *4.92*             | *0.11*             | *3.96*             |\n| Fit LR            | *29.99*            | *7.66*             | *18.56*            | *14.02*            | *2.78*             | *11.29*            |\n| **+ AdaLRS**     | 28.49            | 6.71             | 17.29            | 12.65            | 2.61             | 10.13            |\n\nAs shown in the table, SFT models from AdaLRS pretrained checkpoints achieve significantly higher benchmark scores than baseline models. \nFor fit LRs, AdaLRS pretrained models suffer from slight performance degradation, which coincides with the pretraining loss shown in Table 2 of our paper. \n\nThese results demonstrate the performance generalizability of LLMs pretrained with AdaLRS. For VLM experiments on downstream benchmarks, we refer to Table 3 in our paper for more details.\n\n[1] Li J, Du L, Zhao H, et al. Infinity Instruct: Scaling Instruction Selection and Synthesis to Enhance Language Models[J]. arXiv preprint arXiv:2506.11116, 2025.\n\n[2] Yang D, Xiao D, Wei J, et al. Improving factuality in large language models via decoding-time hallucinatory and truthful comparators[C]//Proceedings of the AAAI Conference on Artificial Intelligence. 2025, 39(24): 25606-25614."}, {"review_id": "nL5XN7t1FG", "reviewer": "Reviewer_5CYH", "summary": "The paper introduces AdaLRS, a novel adaptive learning rate (LR) search algorithm specifically designed for efficient pretraining of foundation models (such as large language models (LLMs) and vision-language models (VLMs)). The authors provide theoretical proofs for convergence and geometric error decay of the AdaLRS algorithm, and also conducted extensive experiments on LLM (Qwen2.5 models) and VLM (SAIL-VL) pretraining tasks, demonstrating that AdaLRS consistently finds optimal learning rates across various initial conditions and model sizes.", "questions": "Authors attribute the poor performance of AdaLRS under large initial learning rate (LR) settings to disruptive parameter updates. It’s unclear why the baseline methods, which utilize same large initial learning rates, do not suffer from these disruptive parameter updates. Could the authors please explain this discrepancy?", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "The work addresses a significant practical challenge: efficiently finding optimal learning rates without extensive tuning or costly experiments. AdaLRS provides a practical, plug-and-play solution applicable across different model architectures. This work also provides theoretical proofs and analysis of convergence and geometric error decay, significantly enhancing the method’s effectiveness in accelerating the training convergence.\nOne limitation is that, without statistical validation (statistical significance analysis), it's unclear whether the reported improvements over baselines are statistically significant.", "was_revised": true, "final_justification": "Many thanks for the author’s rebuttal comments. They have addressed some of my concerns, but not all issues have been fully resolved. I have increased my rating regarding the paper's quality.", "rebuttal": "We appreciate the reviewer's useful suggestions. Please allow us to clarify our methodologies and conclusions point to point. \n\n> **1. Statistical confidence analysis**\n\nSince large scale foundation model pretraining is resource-consuming, we opt to conduct repetive experiments at 8M-sample scale for 2B LLM pretraining for statistical confidence analysis. \nWe use 5 different random seeds for model training, and report the final pretraining losses. \n\n| Experiment | 1 | 2 | 3 | 4 | 5 |\n| -- | -- | -- | -- | -- | -- |\n| Pretraining Loss | 4.721 | 4.714 | 4.734 | 4.733 | 4.713 |\n\nThe mean value and standard deviation of the pretraining losses are 4.723 and 0.009, respectively.\nAs shown in Table 2 of our paper, the training loss improvement of AdaLRS experiments all exceed the 3σ range, except for the Fit LR experiment for 7B LLM pretraining. \nThese results indicate that the AdaLRS pretraining loss improvement is statistically significant.\n\n\n> **2. Disruptive parameter updates in large LR experiments**\n\nThe disruptive parameter updates is introduced by the large LR setting rather than the AdaLRS algorihhm, and therefore the large LR baseline models indeed suffer from similar performance degradations. As shown in Figure 2 and Table 2, 3 of our paper, large LR baseline experiments converge to relatively high pretraining losses and suffer from performance degradation in evaluation results. Despite the degraded model performance, AdaLRS still reaches the optimal learning rates and improves pretraining convergence significantly, demonstrating its effectiveness in optimal LR searching."}, {"review_id": "r4PrVsN9Uo", "reviewer": "Reviewer_9vf5", "summary": "This paper introduces AdaLRS, an algorithm that automatically finds optimal learning rates during pretraining. The main idea is to use the slope of the training loss to dynamically adjust the learning rate to maximize this velocity within a single training run. The algorithm detects when the loss slope degrades and tests if changes in learning rate would decrease the loss faster. The authors provide a theoretical convergence analysis based on the convexity of loss and its slope. The algorithm was validated on LLM and VLM pre-training at 2B-7B scales when trained from scratch or in a continual training setting. The authors also checked its compatibility with learning rate schedules.", "questions": "Please see weakeneses.", "limitations": "Please see Weaknesses.\n\n**Additional comments**\n* Please use the same scale for the x-axis of the middle and bottom rows of Figure 1.", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 1, "significance": 2, "originality": 3, "strengths_and_weaknesses": "> **Strengths**\n* **Clear motivation**: Optimal LR is difficult to find at scale. This makes finding optimal LR during a single run well motivated.\n* **Large-scale validation**: The experiments are performed at large scales.\n* **Clear main idea**: The core idea of adjusting LR based on the slope of the loss is intuitive and well-motivated\n\n> **Weaknesses**\n* **Poor readability**: The paper is difficult to follow with inconsistent notation and unclear algorithmic descriptions. I think the paper needs a major rewrite.\n* **Algorithmic inconsistencies**: There seems to be a contradiction between Equation 1 and Algorithm 1 in the Appendix. Equation 1 suggests simultaneously increasing or decreasing LR, while Algorithm 1 follows a sequential if-else statement. Furthermore, Equation 1 is using $e$ for the error, but suddenly $\\theta$ is introduced in Algorithm 1.\n* **Timeline ambiguity**:  It is unclear when LR is changed (at step t or t+k). If there is a k-step lookahead, it should be clearly stated in the main text, as it increases the computational cost.\n* **Excessive complexity**: While the central idea is clear, the method is overly complicated. It introduces 5-6 hyperparameters, and their reliability is not well tested. \n* **Convexity assumptions**: The \"convexity\" discussed in the paper is trivial. Of course, small LRs give slow descent and training diverges at large LRs. Therefore, an optimal LR exists trivially. Are the authors claiming anything beyond it? \n* **Logical inconsistency in equation 1**: Equation 1 says that reduce the LR if the loss slope is decreasing. Why would one select an LR where the loss is decreasing slowly? Are the authors assuming that loss is increasing in this case?\n* **Narrow applicability**: Method assumes stable loss curves without spikes, which is unrealistic for large-scale pretraining\n* **Limited scope**: In the Appendix, the authors discuss that the method only works well with small LR baselines.\n* **Chicken-and-egg problem**: How does one know a priori whether their LR is \"small enough\" for the method to work?", "was_revised": true, "final_justification": "The rebuttal has addressed a few of the concerns I had. Accordingly, I have increased my score from 2 to 3. \n\nWhy am I not recommending for acceptance?\n\n1. I feel like the paper needs a major rewrite: among the 6 papers I reviewed this cycle, I have spent the most time understanding the details of this paper. As the high-level idea of the paper is quite simple, I believe the presentation can be improved.\n2. While the overall idea is simple, the resultant method appears to be overly complicated with multiple new hparams. I am unsure if the community would adopt the method.\n\nSince my score is still borderline, I am happy to discuss it with other reviewers and the ACs.", "rebuttal": "We appreciate the reviewer's insightful feedback. Please allow us to clarify our methodologies point to point. \n\n> **1. Paper readability** \n\nWe thank the reviewer for the careful reading. The potentially confusing notations are introduced to prove the convergence of AdaLRS. We consider potential statistical error in loss slope estimation and several other details for broader generalizability of the proof. For better readibility, we provided a simplified workflow description of the proposed AdaLRS algorithm in Section 2.1 workflow. \nWe admit certain algorithmic details can be further clarified, and will adjust corresponding descriptions accordingly in the revised version.\n\nBelow is a simplified workflow description of the proposed AdaLRS algorithm, and we hope it can help address potential confusion:\n1. AdaLRS attempts to upscale the LR when the loss descent velocity decreases, which is triggered by a threshold $\\theta$ shown in Algorithm 1 of the paper.\n2. k attemping training steps are performed to estimate the loss slope after LR upscaling.\n3. If the loss descent velocity increases after LR upscaling attemption, this adjustment is retained; otherwise it is discarded and replaced by a LR downscaling adjustment. \n4. Either a LR upscaling or downscaling adjustment is performed, a decaying factor $\\gamma$ is applied to the LR scaling factors, ensuring the convergence of LR adjustments.\n\n\n\n> **2. Algorithmic representations**\n\nAs shown in Equation 1 and Algorithm 1, the definition of $e$ remains consistent, which is the loss slope estimation error introduced to ensure the generalizability of our proof for AdaLRS's convergence. The threshold $\\theta$ introduced in Algorithm 1 is set to automatically trigger LR upscaling attemptions for various pretraining tasks. We elaborate the necessity and importance of this threshold in point 4.\n\n\n\n> **3. Additional computation cost**\n\nWe mention that the attempted LR upscaling will be reverted if the resulted loss descent velocity decreases in Section 2.1 workflow. In Section 3.2, we further discuss the acceleration ratio of AdaLRS in different pretraining scenarios, and we observe that the k-step lookahead does not undermine the convergence speed and loss improvement of AdaLRS. Even for large LR settings, where a series of roll-backs are performed, models trained with AdaLRS still achieve an over 30% acceleration ratio to reach the same loss level as baselines. It is also worth noticing that the roll-backs do not undermine the efficiency of AdaLRS in optimal LR searching. Since we only conduct LR search in 30% of the training steps, the maximum computational cost of AdaLRS is 130% of the baseline training cost, which still outperforms the traditional LR searching methods markedly.\n\n\n\n\n> **4. Hyperparamter**\n\nBesides the LR scaling factors $\\alpha$, $\\beta$, and $\\gamma$, we introduce only two additional hyperparameters $\\theta$ and $e$ in our algorithm. The threshold $\\theta$ provides an automatical way to trigger LR upscaling attemptions, regardless of the training scenario or scale. We set $\\theta$ as 0.9 in our experiments, i.e., an attempted LR upscaling is triggered whenever the loss descent velocity decreases by at least 10%. This simple strategy ensures steady loss descent with high velocity, as well as timely LR upscaling when the velocity decays. More importantly, this method achieves adaptive LR upscaling attempts across various pretraining tasks, regardless of the training scenario. The loss slope estimation error $e$, on the other hand, is incorporated to ensure the generalizability of our proof for AdaLRS's convergence, which is shown in Section 2.2.\n\nFor LR scaling factors $\\alpha$, $\\beta$, and $\\gamma$, the first two scaling factors are necassary for AdaLRS, and $\\gamma$ ensures the convergence of AdaLRS to the small enough neighborhood of the optimal LR. We further conduct experiments to demonstrate the importance of the scaling factors and AdaLRS's robustness under various hyperparameter settings.\nStarting from a 2e-5 LR for 2B LLM pretraining with WSD scheduler, we show the LR dynamics in the 64M-sample pretraining experimets: \n| Experiment |$\\alpha$/$\\beta$/$\\gamma$ | Step/LR/Scale        |                |                |                |                |                |                | Test PPL |\n|----------------|----------------|----------------|----------------|----------------|----------------|----------------|----------------|----------------|----------------|\n| Baseline | - | 100/2e-5 -  | -  | - | - | - | - | - | 43.77 |\n| A | 3/2/0.99 | 100/2e-5/↑  | 20100/6e-5/↑  | 28100/1.8e-4/↑| 35100/3.6e-4/↑ | 42100/7.19e-4/↑ | 50100/3.6e-4 ↓   | - | 16.94 |\n| B | 2/1.67/0.99 | 100/2e-5/↑    | 20100/4e-5/↑    | 28100/8e-5/↑     | 36100/1.6e-4/↑      | 44100/3.2e-4/↑   | 51100/5.16e-4/↑ | 57100/3.1e-4 ↓ | 18.79 |\n| C | 1.5/1.43/0.99 | 100/2e-5/↑    | 20100/3e-5/↑    | 28100/4.5e-5/↑   | 36100/6.75e-4/↑  | 44100/1.01e-4/↑  | 52100/1.52e-4/↑ | - | 21.90 |\n| D | 2/1.67/0.95  | 100/2e-5/↑    | 20100/4e-5/↑    | 28100/7.6e-5/↑   | 36100/1.37e-4/↑  | 44100/2.35e-4/↑  | 51100/3.09e-4/↑ | - | 19.58 |\n| E | 2/1.67/0.9 | 100/2e-5/↑    | 20100/4e-5/↑    | 28100/7.2e-5/↑   | 36100/1.16e-4/↑  | 44100/1.7e-4/↑   | 51100/2.17e-4/↑ | - | 19.70 |\n\nIn this table, Exp Baseline, A correspond to the \"Small LR\" setting for 2B LLM pretrain in our paper, and Exp B, C are conducted with different LR scaling factor values. Exp D, E adopt smaller decaying factor, which suppresses the \"overshooting\" behavior observed Exp B. Overall, depite the varying hyperparameter settings, learning rates converge to the neighborhood of the optimum 2e-4 robustly.\n\nWe also choose the model checkpoints at 60,000 step (150,000 in total) to evaluate their test PPL, and the results show that AdaLRS achieves improved model performance over the baseline under various hyperparameter settings. \nWe will include the complete loss dynamic figures and evaluation results in the revised version of our paper. \n\n> **5. Convexity assumptions**\n\nThe convexity assumptions is indeed intuitive, but they have not been validated systematically across different foundation model pretraining setting. In this work, we not only validated this convexity in various pretraining scenarios, but also unveil that the loss and loss slope optimization share the same optimum. The shared optimum builds up the keystone of the convergence of AdaLRS algorithm, which is discussed in Section 2.2.2.\n\n\n> **6. Logics of Equation 1**\n\nAs described in Section 2.1 workflow, the proposed AdaLRS algorithm downscales the LR if the attempted LR upscaling results in decreased loss descent velocity. The condition in Equation 1 Line 2 is indeed a typo, and it should be $v(\\alpha'η_t) < v(η_t) − 2e$. We thank the reviewer for the careful reading and will correct it in the revised version.\n\n\n> **7. Loss spikes in pretraining**\n\nLoss spikes are indeed inevitable in pretraining. Actually, we also encounter them in our experiments, but these loss spikes do not undermine the applicability of AdaLRS. Firstly, the gradual LR scaling strategy shown in Algorithm 1 suppresses a number of loss spikes. Secondly, a simple loss smoothing strategy is found to be sufficient to mitigate the influence of loss spikes. As a result, despite the presence of loss spikes, AdaLRS can still achieve expected performance improvement in relatively large scale pretraining.\n\n\n> **8. Application scope**\n\nThis issue is discussed as one of AdaLRS's limitation in Appendix C. We argue that it does not undermine the effectiveness of AdaLRS in optimal LR searching. As shown in Figure 2 and Table 2, 3, AdaLRS can achieve significant performance improvement in pretraining under both small LR and large LR settings. We also discussed in Section 3.2 about the effectiveness of AdaLRS in large LR settings, presenting that even if AdaLRS cannot achieve optimal model performance in a single run, it still finds the optimal LR efficiently, outperforming traditional LR searching methods markedly. \n\n\n> **9. Chicken-and-egg problem**\n\n1. As discussed in the point above, AdaLRS finds optimal LR efficiently and effectively, regardless of the initial LR setting. Therefore, although the chicken-and-egg problem may hinder us from reaching the optimal model performance in a single run, it does not undermine the applicability of AdaLRS in optimal LR searching.\n2. In practice, researchers can refer to existing opensource works for LR setting estimation. For example, 1e-3 and 2e-4 are often used as initial LRs for VLM and LLM pretraining [1,2], respectively. Since AdaLRS approximates the optimal LR with geometric error decay, which is proved in Section 2.3 of the paper, it allows a wide range for small LR choices. As a result, the influence of the chicken-and-egg problem is limited.\n\n[1] Li B, Zhang Y, Guo D, et al. Llava-onevision: Easy visual task transfer[J]. arXiv preprint arXiv:2408.03326, 2024.\n\n[2] Dey N, Gosal G, Khachane H, et al. Cerebras-gpt: Open compute-optimal language models trained on the cerebras wafer-scale cluster[J]. arXiv preprint arXiv:2304.03208, 2023."}, {"review_id": "4DS8hwYyUu", "reviewer": "Reviewer_oJfS", "summary": "This paper introduces AdaLRS, a plug-and-play adaptive learning rate search algorithm that approximates the optimal learning rate online within a single full training process by optimizing the loss descent velocity.\n\nThe authors first demonstrate through small-scale pre-training experiments that in Large Language Model (LLM) and Vision-Language Model (VLM) pre-training, the training loss and its slope exhibit an approximately convex relationship with the learning rate and share the same optimal point. Based on this empirical observation, AdaLRS monitors the loss slope within a sliding window during the early stages of training. If the velocity decreases, it attempts to increase the learning rate; if this proves unsuccessful, it rolls back and reduces the learning rate.\n\nThe paper provides a theoretical proof of convergence and geometric error decay. Experiments covering 2B/7B LLMs and a 2B VLM, with various initial learning rates, Cosine and WSD schedulers, and in a continued pre-training scenario, all show that AdaLRS can quickly guide an unsuitable learning rate towards the optimal region and accelerate convergence. In some cases, it can even surpass the final performance of a manually tuned \"best\" learning rate.", "questions": "You briefly distinguish AdaLRS from Hypergradient Descent (HD) in the related work section. To make the distinction more concrete, would it be possible to provide a small-scale empirical comparison against HD? Showing how HD performs with a modern scheduler (like cosine) and how AdaLRS differs in practice, even on a smaller task, could more forcefully highlight the unique advantages of your proposed method for foundation model pretraining.\n\nAdaLRS repeatedly saves and restores checkpoints to enable roll-backs, yet Table 1 omits the associated memory and wall-time costs. Could you quantify these overheads and discuss feasibility at > 70 B parameter scale?", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "I am overall supportive of this paper. Although I am not an expert in LR tuning domain, I feel that this method is practical.\n\nThe manuscript is straightforward to read: its core idea—adjusting the learning rate online by monitoring the loss-descent velocity—is both simple and intuitive, making the method easy to grasp and potentially easy to reproduce. The authors run large-scale experiments across multiple model sizes (2 B–7 B parameters), two modalities (LLM and VLM), and several learning-rate schedulers. This breadth of evidence strengthens the paper’s empirical claims. The related-work section situates AdaLRS convincingly among classic hyper-parameter optimization, hyper-gradient, and adaptive-LR methods, highlighting both overlap and distinctions. Because AdaLRS can be plugged into an existing training loop with minimal modifications, practitioners who frequently struggle with manual LR tuning can benefit immediately.\n\nI also find some minor weaknesses:\n- Hypothesis 1—that both the loss and its slope are convex in the learning rate and share the same optimum—is supported only empirically. Without a stronger theoretical justification, the method’s robustness outside the tested settings remains uncertain.\n- In several plots (e.g., Figure 1), the training-loss curves and LR trajectories are rendered so small that individual trends are difficult to discern. Enlarging the sub-plots or using insets would improve readability.\n- Figure 1 currently offers little explanatory detail. Expanding the caption (e.g., clarifying what each curve represents and why the trajectories matter) would help readers understand the take-away message without referring back to the main text.", "was_revised": true, "final_justification": "I raised my score from 4 to 5.\n\n**Why not a lower score?**\n\n1. Finding the optimal learning rate (LR) is a timely and relevant challenge in large model pre-training.\n\n2. This paper provides an adequate empirical study along with analytical discussions, offering non-trivial insights for the community.\n\n**Why not a higher score?**\n\n1. The proposed method introduces additional hyperparameters that may require tuning. In light of the next point, this limitation gains added significance.\n\n2. The scalability of the method remains unclear, yet it is crucial for real-world applications.\n\n3. Although optimizing the LR is a timely issue, it represents only a small aspect of large model pre-training, which often intersects with other challenges (e.g., loss spikes, as noted by other reviewers). This suggests the method may be somewhat idealized or primarily experimental in nature.", "rebuttal": "We appreciate the reviewer's insightful feedback. Please allow us to clarify our methodologies point to point. \n\n> **1. Theoretical justification for Hypothesis 1** \n\n1. The convexity of loss descent velocity w.r.t. LR\n\n    Assume $L(\\theta)$ is the loss function to be minimized, and we use SGD with the update rule:\n    $$\n    \\theta_{t+1} = \\theta_t - \\eta \\nabla L(\\theta_t)\n    $$\n    where $\\eta$ is the learning rate (LR).\n\n    Intuitively, if $\\eta$ is too small, updates are small, and the loss decreases slowly; if $\\eta$ is too large, the updates may overshoot, causing oscillation or even divergence.\n\n    The expected loss descent velocity per step (using Taylor expansion) is:\n    $$\n    \\mathbb{E}\\left[ L(\\theta_{t+1}) - L(\\theta_t) \\right] \n    \\approx -\\eta \\| \\nabla L(\\theta_t) \\|^2 + \\frac{L}{2} \\eta^2 \\| \\nabla L(\\theta_t) \\|^2\n    $$\n    where $L$ is the Lipschitz constant of the gradient. When the learning rate is small, the first term dominates the expectation, and a smaller $\\eta$ leads to a smaller decrease in loss. When the learning rate is large, on the other hand, the second term cannot be neglected. It contributes to suppress the expected loss descent value. \n\n    Differentiating with respect to $\\eta$ and setting to zero for the extremum:\n    $$\n    \\frac{\\partial}{\\partial \\eta} \\left( -\\eta + \\frac{L}{2} \\eta^2 \\right) = -1 + L \\eta = 0 \\implies \\eta^* = \\frac{1}{L}\n    $$\n    This shows that the loss descent velocity with respect to $\\eta$ is a convex function (has only one maximum).\n\n2. Shared optimum across loss and loss slope optimization\n\n    In Figure 1, we show that the optimum for both training loss and its slope remain the same across most of the training process. Based on this observation, we summarize the reason why the optimal learning rate for fastest loss decrease and lowest final loss generally coincide as follows:\n      - Optimial loss slope -> optimal loss: the best learning rate discovered via the largest loss slope leads to the largest loss descent per step, resulting in the lowest attainable final loss.\n      - Optimial loss -> optimal loss slope: if a learning rate cannot achieve the largest loss slope, the resulted final training loss must be suboptimal, too, because the LR with the largest loss slope will always produce lower training loss. As a result, the optimal training loss must be achieved by a learning rate with the largest loss slope.\n\n    The conclusions above can be generalized to any learning rate values: learning rate with faster loss descent velocity always achieve lower training loss. Since we have justified that the loss slope optimization is convex, it can be easily dedued that the training loss optimization is also a convex problem. \n\n\n> **2. Details about Figure 1**\n\nWe thank the reviewer for the careful reading and useful suggestions, and we will supplement enlarged figures as well as detailed caption texts in the revised version of the paper.\n\nWe also would like to further clarify the content and main conclusions drawn from Figure 1:\n1. Subplot a,b,c show the training loss curves and LR trajectories for different pretraining scenarios.\n2. In Subplot e,f,g, each curve represents the training loss dynamics across different LR settings at a certain training step.\n3. In Subplot h,i,j, each curve indicates the loss slope dynamics across different LR setting at a certain loss value.\n4. The curve groups shown in Subplot e,f,g and h,i,j demonstrate consistent optima across the training process, and the optimal LR for the fastest loss descent and lowest final loss generally coincide.\n\n\n> **3. Comparison with Hypergradient Descent (HD)**\n\nWe thank the reviewer for the careful reading. In fact, the Hypergradient Descent (HD) and AdaLRS algorithms serve different purposes, and therefore they are not comparable. We list the differences between the two methods below:\n1. Main idea\n   - HD: dynamically adjusts learning rate across the entire training process. \n   - AdaLRS: searches for the optimal learning rate *scale* at the beginning steps of pretraining. \n2. Workflow\n   - HD: adjusts LR dynamically across the training process, serving as a learning rate scheduler itself.\n   - AdaLRS: adjusts only the learning rate *scale*, which is multiplied with base LRs from an existing scheduler. \n3. Compatibility with modern learning rate schedulers\n   - HD: cannot be used with any learning rate scheduler.\n   - AdaLRS: compatible with a series of modern learning rate schedulers, such as cosine scheduler, WSD scheduler, and potentially more alternatives.\n\n\n\n> **4. Additional costs in checkpoint restoration and rollback**\n\n1. Memory and wall-time costs of checkpoints restoration\n\n   To the best of our knowledge, modern foundation model pretraining frameworks, such as deepspeed, support checkpoint restoration in parallel with model training. In our experiments, we find out that the memory and wall-time costs of checkpoint restoration are negligible, and do not block the training process.\n\n2. Memory and wall-time costs of roll-backs\n\n   Assuming a extreme case where all the LR upscaling attempts are failed, every training step will be performed twice (once in the look-ahead attempt, and another in alternative LR downscaling process). As discussed in Section 2.1 workflow, the LR adjustment process is restricted to certain training steps, which is the 10%-40% part of the training process. Therefore, the maximum time consumption of an entire training process will be 130% of the baseline training time. It is still much faster than traditional LR searching methods.\n\n3. Feasibility at 70B parameter scale\n\n   The design of AdaLRS is based solely on the training loss dynamics, which unveils promising scalibilities. In our early experiments in 72B LLM and VLM training, AdaLRS exhibits expected LR adjustment effectiveness. We will validate the applicability of AdaLRS at 70B models at scale in our future work, and supplement corresponding details in the revised version of our paper."}]

Author rebuttal:

Meta-review: This paper proposes AdaLRS, a simple yet effective method for online adaptive learning rate search in foundation model pretraining. The method dynamically adjusts learning rates by monitoring loss descent velocity, with some theoretical evidence for its convergence. The experiments are comprehensive, covering both LLM and VLM pretraining under diverse settings, and reviewers generally agree that the approach is well-motivated, empirically validated, and has certain practical value.

Reviewers raised concerns regarding several aspects, including too many hyperparameters, sensitivity to hyperparameters, the statistical significance of improvements, downstream evaluation, and clarity of presentation. The rebuttal provided additional experiments and clarifications that largely addressed these issues, though some issues in writing remain. Overall, the strengths outweigh the weaknesses, and I recommend acceptance, while encouraging the authors to improve the readability of the paper in future revisions.
Decision: Accept (poster)