ICLR 2025

Published as a conference paper at ICLR 2025
SCALING LAWS FOR PRECISION
Tanishq Kumar∗1
Zachary Ankner* 3,4
Benjamin F. Spector2
Blake Bordelon1
Niklas Muennighoff2
Mansheej Paul4
Cengiz Pehlevan1
Christopher R´e2
Aditi Raghunathan5
1Harvard University
2Stanford University
3MIT
4Databricks
5Carnegie Mellon University
ABSTRACT
Low precision training and inference affect both the quality and cost of language
models, but current scaling laws do not account for this. In this work, we devise
“precision-aware” scaling laws for both training and inference. We propose that
training in lower precision reduces the model’s effective parameter count, allow-
ing us to predict the additional loss incurred from training in low precision and
post-train quantization. For inference, we find that the degradation introduced by
post-training quantization increases as models are trained on more data, eventu-
ally making additional pretraining data actively harmful. For training, our scaling
laws allow us to predict the loss of a model with different parts in different preci-
sions, and suggest that training larger models in lower precision may be compute
optimal. We unify the scaling laws for post and pretraining quantization to arrive
at a single functional form that predicts degradation from training and inference in
varied precisions. We fit on over 465 pretraining runs and validate our predictions
on model sizes up to 1.7B parameters trained on up to 26B tokens.
1
INTRODUCTION
Scale has emerged as a central driver of progress in deep learning (Brown et al., 2020). Key work
on scaling (Kaplan et al., 2020; Hoffmann et al., 2022) studied tradeoffs between model/dataset
size to balance performance and compute. However, the precision in which models are trained and
served is an important third factor that contributes to both cost and performance. Deep learning
is trending towards lower precision: current frontier models like the Llama-3 series are trained in
BF16 (Dubey et al., 2024), an...