Title: AdaLRS: Loss-Guided Adaptive Learning Rate Search for Efficient Foundation Model Pretraining
Venue: NeurIPS 2025
Accepted: yes
Keywords: Foundation Model Pretraining

Hyperparameter Search

Learning Rate Search
Abstract: Learning rate is widely regarded as crucial for effective foundation model pretraining.
Recent research explores and demonstrates the transferability of learning rate configurations across varying model and dataset sizes, etc. 
Nevertheless, these approaches are constrained to specific training scenarios and typically necessitate extensive hyperparameter tuning on proxy models.
In this work, we propose \textbf{AdaLRS}, a plug-in-and-play adaptive learning rate search algorithm that conducts online optimal learning rate search via optimizing loss descent velocities.
We provide theoretical and experimental analyzes to show that foundation model pretraining loss and its descent velocity are both convex and share the same optimal learning rate. 
Relying solely on training loss dynamics, AdaLRS involves few extra computations to guide the search process, and its convergence is guaranteed via theoretical analysis. 
Experiments on both LLM and VLM pretraining show that AdaLRS adjusts suboptimal learning rates to the neighborhood of optimum with marked efficiency and effectiveness, with model performance improved accordingly. 
We also show the robust generalizability of AdaLRS across varying training scenarios, such as different model sizes, training paradigms, base learning rate scheduler choices, and hyperparameter settings.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces AdaLRS, a plug-and-play adaptive learning rate search algorithm that approximates the optimal learning rate online within a single full training process by optimizing the loss descent velocity.\n\nThe authors first demonstrate through small-scale pre-training experiments that in Large Language Model (LLM) and Vision-Language Model (VLM) pre-training, the training loss and its slope exhibit an approximately convex relationship with the learning rate and share the same optimal point. Based on this empirical observation, AdaLRS monitors the loss slope within a sliding window during the early stages of training. If the velocity decreases, it attempts to increase the learning rate; if this proves unsuccessful, it rolls back and reduces the learning rate.\n\nThe paper provides a theoretical proof of convergence and geometric error decay. Experiments covering 2B/7B LLMs and a 2B VLM, with various initial learning rates, Cosine and WSD schedulers, and in a continued pre-training scenario, all show that AdaLRS can quickly guide an unsuitable learning rate towards the optimal region and accelerate convergence. In some cases, it can even surpass the final performance of a manually tuned \"best\" learning rate.\n\nSTRENGTHS AND WEAKNESSES:\nI am overall supportive of this paper. Although I am not an expert in LR tuning domain, I feel that this method is practical.\n\nThe manuscript is straightforward to read: its core idea—adjusting the learning rate online by monitoring the loss-descent velocity—is both simple and intuitive, making the method easy to grasp and potentially easy to reproduce. The authors run large-scale experiments across multiple model sizes (2 B–7 B parameters), two modalities (LLM and VLM), and several learning-rate schedulers. This breadth of evidence strengthens the paper’s empirical claims. The related-work section situates AdaLRS convincingly among classic hyper-parameter optimization, hyper-gradient, and adaptive-LR methods, highlighting both overlap and distinctions. Because AdaLRS can be plugged into an existing training loop with minimal modifications, practitioners who frequently struggle with manual LR tuning can benefit immediately.\n\nI also find some minor weaknesses:\n- Hypothesis 1—that both the loss and its slope are convex in the learning rate and share the same optimum—is supported only empirically. Without a stronger theoretical justification, the method’s robustness outside the tested settings remains uncertain.\n- In several plots (e.g., Figure 1), the training-loss curves and LR trajectories are rendered so small that individual trends are difficult to discern. Enlarging the sub-plots or using insets would improve readability.\n- Figure 1 currently offers little explanatory detail. Expanding the caption (e.g., clarifying what each curve represents and why the trajectories matter) would help readers understand the take-away message without referring back to the main text.\n\nQUESTIONS:\nYou briefly distinguish AdaLRS from Hypergradient Descent (HD) in the related work section. To make the distinction more concrete, would it be possible to provide a small-scale empirical comparison against HD? Showing how HD performs with a modern scheduler (like cosine) and how AdaLRS differs in practice, even on a smaller task, could more forcefully highlight the unique advantages of your proposed method for foundation model pretraining.\n\nAdaLRS repeatedly saves and restores checkpoints to enable roll-backs, yet Table 1 omits the associated memory and wall-time costs. Could you quantify these overheads and discuss feasibility at > 70 B parameter scale?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces AdaLRS, an algorithm that automatically finds optimal learning rates during pretraining. The main idea is to use the slope of the training loss to dynamically adjust the learning rate to maximize this velocity within a single training run. The algorithm detects when the loss slope degrades and tests if changes in learning rate would decrease the loss faster. The authors provide a theoretical convergence analysis based on the convexity of loss and its slope. The algorithm was validated on LLM and VLM pre-training at 2B-7B scales when trained from scratch or in a continual training setting. The authors also checked its compatibility with learning rate schedules.\n\nSTRENGTHS AND WEAKNESSES:\n> **Strengths**\n* **Clear motivation**: Optimal LR is difficult to find at scale. This makes finding optimal LR during a single run well motivated.\n* **Large-scale validation**: The experiments are performed at large scales.\n* **Clear main idea**: The core idea of adjusting LR based on the slope of the loss is intuitive and well-motivated\n\n> **Weaknesses**\n* **Poor readability**: The paper is difficult to follow with inconsistent notation and unclear algorithmic descriptions. I think the paper needs a major rewrite.\n* **Algorithmic inconsistencies**: There seems to be a contradiction between Equation 1 and Algorithm 1 in the Appendix. Equation 1 suggests simultaneously increasing or decreasing LR, while Algorithm 1 follows a sequential if-else statement. Furthermore, Equation 1 is using $e$ for the error, but suddenly $\\theta$ is introduced in Algorithm 1.\n* **Timeline ambiguity**:  It is unclear when LR is changed (at step t or t+k). If there is a k-step lookahead, it should be clearly stated in the main text, as it increases the computational cost.\n* **Excessive complexity**: While the central idea is clear, the method is overly complicated. It introduces 5-6 hyperparameters, and their reliability is not well tested. \n* **Convexity assumptions**: The \"convexity\" discussed in the paper is trivial. Of course, small LRs give slow descent and training diverges at large LRs. Therefore, an optimal LR exists trivially. Are the authors claiming anything beyond it? \n* **Logical inconsistency in equation 1**: Equation 1 says that reduce the LR if the loss slope is decreasing. Why would one select an LR where the loss is decreasing slowly? Are the authors assuming that loss is increasing in this case?\n* **Narrow applicability**: Method assumes stable loss curves without spikes, which is unrealistic for large-scale pretraining\n* **Limited scope**: In the Appendix, the authors discuss that the method only works well with small LR baselines.\n* **Chicken-and-egg problem**: How does one know a priori whether their LR is \"small enough\" for the method to work?\n\nQUESTIONS:\nPlease see weakeneses.\n\nLIMITATIONS:\nPlease see Weaknesses.\n\n**Additional comments**\n* Please use the same scale for the x-axis of the middle and bottom rows of Figure 1.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "CLARITY": 1, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-29"}

comments: SUMMARY:
The paper introduces AdaLRS, a novel adaptive learning rate (LR) search algorithm specifically designed for efficient pretraining of foundation models (such as large language models (LLMs) and vision-language models (VLMs)). The authors provide theoretical proofs for convergence and geometric error decay of the AdaLRS algorithm, and also conducted extensive experiments on LLM (Qwen2.5 models) and VLM (SAIL-VL) pretraining tasks, demonstrating that AdaLRS consistently finds optimal learning rates across various initial conditions and model sizes.

STRENGTHS AND WEAKNESSES:
The work addresses a significant practical challenge: efficiently finding optimal learning rates without extensive tuning or costly experiments. AdaLRS provides a practical, plug-and-play solution applicable across different model architectures. This work also provides theoretical proofs and analysis of convergence and geometric error decay, significantly enhancing the method’s effectiveness in accelerating the training convergence.
One limitation is that, without statistical validation (statistical significance analysis), it's unclear whether the reported improvements over baselines are statistically significant.

QUESTIONS:
Authors attribute the poor performance of AdaLRS under large initial learning rate (LR) settings to disruptive parameter updates. It’s unclear why the baseline methods, which utilize same large initial learning rates, do not suffer from these disruptive parameter updates. Could the authors please explain this discrepancy?

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 2
DATE: 2025-07-03

comments: SUMMARY:
The paper introduces AdaLRS, a plug-and-play algorithm for online adaptive learning rate search in foundation model pretraining. Unlike existing methods that rely on offline tuning or proxy models, AdaLRS optimizes the loss descent velocity to adjust learning rates dynamically during training. The authors provide theoretical proofs of convergence and geometric error decay and demonstrate the algorithm’s effectiveness across diverse settings—covering both LLMs and VLMs, varying model sizes, training paradigms, and learning rate schedules. Experiments show AdaLRS can significantly accelerate convergence, improve model performance, and generalize well across tasks.

STRENGTHS AND WEAKNESSES:
# Strengths
1. The paper is well-written and easy to follow.
2. The method is simple but effective.
3. The experiments are comprehensive covering both LLM and VLM backbones.
4. The authors provide theoretical analysis along with good empirical performance.

# Weaknesses
1. The method introduces more hyper-parameters (i.e., upscaling factor α, downscaling factor β, and decaying factor λ). The authors did not provide sensitivity experiments on these hyper-parameters so it remains unclear if we still need to tune these. If so the method still needs extensive hyper-parameter tuning.

QUESTIONS:
The experiments focus primarily on training loss, with less emphasis on downstream or task-specific metrics. Although loss and accuracy are strongly correlated, sometimes they may not align perfectly with final task-specific metrics in some applications.

LIMITATIONS:
Yes.
TITLE: Official Review
CLARITY: 4
IMPACT: 4
ORIGINALITY: 3
DATE: 2025-07-03