Streaming DiLoCo with overlapping communication: Towards a Distributed
  Free Lunch
Authors: name: Arthur Douillard, name: Yanislav Donchev, name: Keith Rush, name: Satyen Kale, name: Zachary Charles, name: Zachary Garrett, name: Gabriel Teston, name: Dave Lacey, name: Ross McIlroy, name: Jiajun Shen, name: Alexandre Ramé, name: Arthur Szlam, name: Marc'Aurelio Ranzato, name: Paul Barham

Training of large language models (LLMs) is typically distributed across a
large number of accelerators to reduce training time. Since internal states and
parameter gradients need to be exchanged at each and every single gradient
step, all devices need to be co-located using low-latency high-bandwidth
communication links to support the required high volume of exchanged bits.
Recently, distributed algorithms like DiLoCo have relaxed such co-location
constraint: accelerators can be grouped into ``workers'', where
synchronizations between workers only occur infrequently. This in turn means
that workers can afford being connected by lower bandwidth communication links
without affecting learning quality. However, in these methods, communication
across workers still requires the same peak bandwidth as before, as the
synchronizations require all parameters to be exchanged across all workers. In
this paper, we improve DiLoCo in three ways. First, we synchronize only subsets
of parameters in sequence, rather than all at once, which greatly reduces peak
bandwidth. Second, we allow workers to continue training while synchronizing,
which decreases wall clock time. Third, we quantize the data exchanged by
workers, which further reduces bandwidth across workers. By properly combining
these modifications, we show experimentally that we can distribute training of
billion-scale parameters and reach similar quality as before, but reducing
required bandwidth by two orders of magnitude.