Communication-Efficient Language Model Training Scales Reliably and
  Robustly: Scaling Laws for DiLoCo
Authors: name: Zachary Charles, name: Gabriel Teston, name: Lucio Dery, name: Keith Rush, name: Nova Fallen, name: Zachary Garrett, name: Arthur Szlam, name: Arthur Douillard

As we scale to more massive machine learning models, the frequent
synchronization demands inherent in data-parallel approaches create significant
slowdowns, posing a critical challenge to further scaling. Recent work develops
an approach (DiLoCo) that relaxes synchronization demands without compromising
model quality. However, these works do not carefully analyze how DiLoCo's
behavior changes with model size. In this work, we study the scaling law
behavior of DiLoCo when training LLMs under a fixed compute budget. We focus on
how algorithmic factors, including number of model replicas, hyperparameters,
and token budget affect training in ways that can be accurately predicted via
scaling laws. We find that DiLoCo scales both predictably and robustly with
model size. When well-tuned, DiLoCo scales better than data-parallel training
with model size, and can outperform data-parallel training even at small model
sizes. Our results showcase a more general set of benefits of DiLoCo than
previously documented, including increased optimal batch sizes, improved
downstream generalization with scale, and improved evaluation loss for a fixed
token budget.