Title: Efficient Hyperparameter Tuning via Trajectory Invariance Principle
Primary area: foundation or frontier models, including LLMs
Keywords: pre-training, batch size, scaling laws, optimization, hyperparameter tuning, large language models

Abstract: As hyperparameter tuning becomes 
increasingly costly at scale, efficient tuning methods are essential. Yet principles for guiding hyperparameter tuning remain limited.
In this work, we seek to establish such principles by considering a broad range of hyperparameters, including batch size, learning rate, and weight decay.
We identify a phenomenon we call \emph{trajectory invariance}, where pre-training loss curves, gradient noise, and gradient norm exhibit invariance--closely overlapping--with respect to a quantity that combines learning rate and weight decay. This phenomenon effectively reduces the original two-dimensional hyperparameter space to one dimension, yielding an efficient tuning rule: follow the salient direction revealed by trajectory invariance. Furthermore, we refine previous scaling laws and challenge several existing viewpoints.
Overall, our work proposes new principles for efficient tuning and inspires future research on scaling laws.