Title: Temperature-Driven Escape Explains Critical Learning Rates in Adaptive Optimization
Primary area: optimization
Keywords: Critical learning rate, Effective temperature, Adaptive optimization, Neural network training

Abstract: The learning rate is a central control parameter in neural network training, even for adaptive optimizers such as Adam, which reduce sensitivity to gradient magnitude differences across parameters. It can either trap models in sharp minima or guide convergence toward flatter regions, yet its selection has largely remained empirical. Here, we introduce a temperature-driven escape framework that grounds learning-rate dynamics in statistical physics. By decomposing Adam’s mini-batch updates into deterministic drift and stochastic fluctuations, we define an effective temperature $T_\text{eff}$ induced by gradient noise and apply Kramers’ escape theory to derive a closed-form expression for the critical learning rate $\alpha_c$. Experiments on vision tasks (MNIST, CIFAR-10 with MLP, CNN, ResNet) and language tasks (SST-2 with BERT, GPT-2, TinyLlama) show that the theoretically predicted $\alpha_c$ achieves better generalization, whereas far deviations from this critical value lead to degraded performance. Beyond initialization, re-estimating $\alpha_c$ also serves as a diagnostic tool for probing the nature of minima reached by well-trained models. Our results elevate the learning rate from an empirical hyperparameter to a theoretically principled quantity, providing both a predictive rule for initialization and a new perspective on optimizer dynamics.