SlowGuess/NeurIPS_2023_OCR huggingface dataset:
To that end, servers periodically run a load balancing procedure and switch to new blocks if necessary. Formally, servers choose blocks so as to maximize the total system throughput (tokens per second). Each server periodically announces its blocks and empirically measured throughput to a distributed hash table (Maymounkov & Mazieres, 2002). When a new server joins, it uses this information to identify a contiguous interval<sup>7</sup> of blocks that would increase the total system throughput the most.
This interval is always contiguous, since splitting it would harm the inference latency.
Since peers may leave or fail at any time, all nodes periodically check if launching a rebalancing procedure would significantly improve the overall throughput. If it is the case, they switch layers until the throughput becomes near-optimal. In particular, if all peers serving certain blocks suddenly leave the system, this procedure quickly redistributes the remaining resources to close the emerged gaps.
We provide a detailed description of the load balancing algorithms in Appendix D and validate their properties in experiments reported in Appendix E.
3.4 Parameter-efficient fine-tuning
While LLMs achieve high quality on many problems with simple prompt engineering (Brown et al., 2020), they often need training to achieve the best results. Traditionally, this is done by fine-tuning all model parameters on the downstream task. However, for extremely large models, this strategy becomes impractical due to hardware requirements. For example, fine-tuning BLOOM-176B with Adam would require almost 3 TB of GPU memory to store the model, gradients, and optimizer states.
Fortunately, parameter-efficient fine-tuning methods have been developed that keep most of the pretrained model intact. Some of them choose a subset of existing parameters to update (Sung et al., 2021; Guo et al., 2021) while others augment the model with additional trainable weights (Hu et al., 2021; Houlsby et al., 2019; Liu et al., 2021b; Lester et al., 2021; Liu et al., 2021a, 2022b). Despite their lower memory requirements, parameter-efficient approaches are often competitive with full model fine-tuning (Hu et al., 2021; Liu et al., 2021a; Yong & Nikoulina, 2022) and even outperform it in low-data regimes (Liu et al., 2022a). Another appealing property of these approaches for our use-case is that they allow rapidly switching a pretrained LLM between adapters.
By focusing on parameter-efficient fine-tuning, we are able to simplify the system design by making clients responsible for storing their trainable parameters (see Figure 1). Servers can run backpropagation through their layers and return gradients with respect to activations, but they do not update the server-side parameters. Even when client communicates learned values (e.g. soft prompts) to a server, the server treats these values same as input activations. Thus, a server can simultaneously run different fine-tuning tasks without them interfering with one another. This design choice also allows users to define custom adapters in simple PyTorch without having network engineering expertise.
Unlike inference, fine-tuning forward and backward passes process the entire batch at one go and do not need to store past attention caches between successive client requests. Thus, in case of a failure, we can discard the incomplete forward/backward pass and just repeat the previous forward/backward pass request. This algorithm behaves similarly to the cache-less baseline from Section 4.1.
3.5 Implementation details
Since our main intended use-case is running on inexpensive low-end devices, we need to work around their capabilities. In terms of raw FLOPs, even consumer-grade GPUs like GeForce RTX 3070 could run a complete inference step of BLOOM-176B in less than a second (NVIDIA, 2020). However, the GPU memory can only hold a small fraction of model layers: running naively would require 44 RTX 3070 GPUs and 44 communication rounds. To make this more efficient, we use quantization to store more parameters per GPU, reducing the number of consecutive devices and communication rounds.
One option for quantization is to use 8-bit mixed matrix decomposition for matrix multiplication to quantize the weights to 8-bit precision and reduce the memory footprint compared to 16-bit weights, as suggested in Dettmers et al. (2022a). This decomposition separates hidden states and weights into two portions: about  $0.1\%$  of 16-bit outlier and  $99.9\%$  of 8-bit regular values, which roughly halves the memory footprint with negligible effect on the model quality (see evaluations in Appendix A). Another option is to use the 4-bit NormalFloat format (Dettmers et al., 2023).
To send less data between subsequent pipeline stages, we apply dynamic blockwise quantization (Dettmers et al., 2022b) to the hidden states before pipeline-parallel communication, which halves the bandwidth requirements without any noticeable effect on generation quality (Ryabinin et al., 2021). During fine-tuning, we also take advantage of gradient checkpointing (Griewank & Walther, 2000; Chen et al., 2016) and half precision to reduce VRAM usage — both are standard practice for large language models (Narayanan et al., 2021; Brown et al., 2020; Athlur et al., 2022). In experiments, we apply the same optimizations to baseline systems for a fair comparison.
Table 1: Sequential inference speed (steps/second) of BLOOM (7.1B) with varying failure rates. A failure rate  $p$  means that sending any set of activations to the next stage of the pipeline fails with probability  $p$ . Missing values mean that the algorithm did not finish within 1 hour.
Inference Algorithm	128 tokens, failure rate:	1024 tokens, failure rate:
0	1e-4	1e-3	1e-2	0	1e-4	1e-3	1e-2
Caching with restarts	17.1	16.7	12	0.18	15.5	11.8	0.48	-
Cache-less inference	3.44	3.44	3.44	3.44	0.89	0.89	0.89	0.89
Algorithm 1 (ours)	11.4	11.4	10.6	3.38	10.7	10.7	7.76	2.17
4.1 Inference with unreliable servers
First, we conduct small-scale preliminary experiments to test the fault-tolerant generation algorithm described in Section 3.2. For these experiments, we use a smaller BLOOM model with 7.1 billion parameters (BigScience, 2022b). This model contains 30 transformer blocks with hidden size 4096. We compare our algorithm with baselines when generating a single sequence of length 512. For simplicity, we run all computations and communications in single precision and disregard word embeddings and logits for this set of experiments. We measure the time to run a certain number of tokens through all blocks and simulate failures by resetting pipeline stages at a certain rate.
We compare three inference strategies:
1. Caching with restarts, which refers to standard inference with servers storing attention caches. On failure, it restarts the entire generation from scratch since the failed server's caches are lost.
2. Cache-less inference, which reruns past tokens on every step. On failure, it restarts only the last generation step.
3. Algorithm 1, which is specifically designed for fault-tolerant inference.
All runs use four pipeline stages with (8, 7, 8, 7) model layers per pipeline stage. Each pipeline stage is served by a single GeForce 1080 Ti GPU; the four GPUs are running in a single system with dual Xeon Gold 6148 CPU, 12 DDR4 LRDIMM sticks with 64 GB each. The system has 16 dedicated PCIe Gen. 3 lanes per GPU in dual root configuration, without using PCIe switches. Each stage runs in an isolated Docker containers with virtual network interfaces, but there is no limit to communication bandwidth for this experiment. We repeat all experiments 50 times and report the average time. The adjusted standard deviation never exceeds  $0.2\%$ . We use the pipeline parallelism implementation from Megatron-DeepSpeed (BigScience et al., 2022) for the cache-less baseline.
We report performance measurements in Table 1. Unlike baselines, our algorithm provides reasonable performance in all tested conditions, especially for higher failure rates (common for communicating over the Internet, using spot/preemptible instances or unreliable hardware). Caching with restarts is most efficient for inference without failures, with our algorithm being somewhat slower due to less mature implementation. Finally, the cache-less inference can be competitive for short sequences (128 tokens), but slows down considerably on 1024 tokens, which agrees with our intuition from 3.1.
We provide plots showing additional evaluations for a wider range of failure rates (up to  $5\%$ ) and sequence lengths (up to 2048 tokens) in Appendix F (Figure 3).
4.2 Experiments for Llama 2 (70B) and BLOOM (176B)
In this section, we evaluate our system on more practical tasks of running Llama 2 (70B) (Touvron et al., 2023b) and BLOOM (176B) (BigScience, 2022a). First, we consider servers running in a network with controlled bandwidth and latency<sup>8</sup>. We measure performance for (a) Llama 2 distributed across 3 servers with a T4 GPU each, (b) BLOOM distributed across 3 servers with an A100 (80 GB) GPU each, and (c) BLOOM distributed across 10 servers with an RTX 3090 GPU each. We use 4-bit NormalFloat quantization (Dettmers et al., 2023) for Llama 2 and 8-bit matrix decomposition (Dettmers et al., 2022a) for BLOOM in all evaluations including the baselines below.
<sup>8</sup>We simulate network conditions using tc qdisc.
Table 2: Performance of Llama 2 (70B) sequential inference steps and parallel forward passes. The network parameters refer to bidirectional bandwidth and round-trip latency (RTT).
GPUs	Clients	Bandwidth	RTT	Sequential inference (steps/s, each client)	Parallel forward (tokens/s, each client)
Sequence length	Batch size
128	2048	1×128	64×128
3× T4 (16 GB)	1	1 Gbit/s	< 5 ms	2.29	2.02	45.4	155.1
1	100 Mbit/s	< 5 ms	2.29	2.01	37.5	140.2
1	100 Mbit/s	100 ms	1.57	1.44	23.7	128.7
3	1 Gbit/s	< 5 ms	2.02	1.74	21.2	124.2
-	Offloading	0.139	0.139	18.0	139.9
Table 3: Performance of BLOOM (176B) sequential inference steps and parallel forward passes.
GPUs	Clients	Bandwidth	RTT	Sequential inference (steps/s, each client)	Parallel forward (tokens/s, each client)
Sequence length	Batch size
128	2048	1×128	64×128
3× A100 (80 GB)	1	1 Gbit/s	< 5 ms	1.71	1.54	70.0	253.6
1	100 Mbit/s	< 5 ms	1.66	1.49	56.4	182.0
1	100 Mbit/s	100 ms	1.23	1.11	19.7	112.2
3	1 Gbit/s	< 5 ms	1.65	1.49	-	-
-	Offloading	0.0495	0.0495	2.5	152.4
-	Local PP (NVLink)	2.46	2.28	98.4	279.5
10× RTX 3090 (24 GB)	1	1 Gbit/s	< 5 ms	1.65	1.54	59.1	230.1
3	1 Gbit/s	< 5 ms	1.65	1.54	54.7	221.4
10	1 Gbit/s	< 5 ms	1.17	1.01	31.0	131.0
10	100 Mbit/s	< 5 ms	1.05	0.99	20.1	28.1
10	100 Mbit/s	100 ms	0.34	0.33	6.5	16.8
-	Offloading	0.0427	0.0427	2.2	109.3
12× heterogeneous (virtual servers)	1	1 Gbit/s	< 5 ms	1.24	1.06	37.9	180.0
1	100 Mbit/s	< 5 ms	1.24	1.05	25.6	66.0
1	100 Mbit/s	100 ms	0.57	0.53	5.8	44.3
12	1 Gbit/s	< 5 ms	0.90	0.86	-	-
14× heterogeneous	1	Real-world setup	0.83	0.79	32.6	179.4
Theoretical-best	-	Offloading	0.18	0.18	2.7	170.3
We report performance of:
- Sequential (autoregressive) inference for batch size 1 (i.e., each step generates 1 token). It is measured in generation steps per second a client can do and shows the generation latency.
- Parallel forward passes for batches of 128-token sequences<sup>9</sup>. It is measured in tokens per second a client can process. This shows the system's throughput during batch processing and fine-tuning.
Since the backward pass performance depends on a set of trainable weights, batch size, and other hyperparameters, we report its performance in different setups separately in Appendix G.
Concurrent clients. We also investigate the effect of having concurrent clients. We assume that each server belongs to a different person, and multiple people (possibly, all of them) are interested in running inference or fine-tuning at the same time. In order to do that, they run the client interacting with our distributed system. The client runs on the same machine, uses 8 CPU cores and no GPU. We report the speed of sequential inference and parallel forward passes that each client gets on average.
9Internally, large batches are split into micro-batches of 1024 tokens each to minimize pipeline bubbles.
Offloading baseline. We also evaluate parameter offloading, where each user runs independently on a single GPU, swapping parameters from CPU memory. First, we report the actual throughput of RAM offloading in case of DeepSpeed with default recommended parameters and enabled pin_memory (gives  $1.2 - 2 \times$  speedup). Next, we report the theoretical-best throughput the offloading baseline can reach for BLOOM. It is calculated as a maximal throughput in the best hardware setup possible (CPU RAM offloading via PCIe 4.0 with 16 PCIe lanes), assuming infinite GPU performance. The calculations are detailed in Appendix B.
Local pipeline parallelism (NVLink). Next, we report performance for BLOOM running on a server with  $3 \times$  A100 (80 GB) GPUs. In this setup, a single server has enough GPU memory to load the entire model, which provides an upper bound for performance reachable with these GPUs. This setup runs pipeline-parallelism from DeepSpeed v0.7.7.
Heterogeneous servers. To validate that our system works on heterogeneous hardware, we simulate 12 heterogeneous devices by partitioning each A100 (80 GB) into several virtual servers (3 large and 1 small). We get 9 servers hosting 7 blocks each, one server with 3 blocks and two more servers with 2 blocks (70 blocks in total, as required for BLOOM). Additionally, we benchmark the system on real heterogeneous GPUs with diverse compute capabilities in the "Real-world setup" below.
Real-world setup. Finally, we benchmark BLOOM in a real-world setup with 14 smaller servers holding  $2 \times$  RTX 3060,  $4 \times$  2080Ti,  $2 \times$  3090,  $2 \times$  A4000, and  $4 \times$  A5000 GPUs. These are personal servers and servers from university labs, spread across Europe and North America and connected to the Internet at speeds of 100-1000 Mbit/s. Four of the servers operate from behind firewalls<sup>10</sup>.
Analysis. We report the results for Llama 2 in Table 2 and for BLOOM in Table 3. For inference, performance does not depend much on bandwidth or sequence length but degrades with higher latency. In turn, fine-tuning forward passes for large batches are affected by both bandwidth and latency.
We can see that the offloading baseline is about an order of magnitude slower than our system for inference, both in practice and in the theoretical-best setup assuming an infinite GPU performance. For parallel forward passes, offloading is competitive if networking is limited to 100 Mbit/s or has high latency. In other cases, our algorithm offers higher throughput than offloading for training.
Crucially, our system significantly outperforms offloading even when each GPU node runs its own client doing single-batch inference at the same time. Thus, given the same hardware, a group of researchers will get much better inference speed by collaborating over the Internet using our system compared to each of them running offloading independently.
Finally, the real-world setup turns out to be slower than the A100 benchmarks due to slower hardware. Still, our algorithm outperforms offloading even when communicating between different continents.
Additional experiments. We conduct two additional experiments to test individual components of our system. We evaluate the load balancing from 3.3 in isolation in Appendix E. We also evaluate the performance of model compression from Section 3.5 in Appendix A. To reiterate, for each model, we use the same compression strategy in our system and all baselines. Finally, we perform a qualitative evaluation of fault tolerance by shutting down random servers during inference and fine-tuning to verify that the algorithm produces correct outputs and gradients.
In this paper, we introduced a novel fault-tolerant algorithm for inferencing large language models. On top of it, we introduced a decentralized system for running LLMs on distributed unreliable devices connected over the Internet, which significantly outperforms other approaches to running inference on consumer-grade hardware. We demonstrated that the proposed system can scale to the largest publicly available language model with hundreds of billions of trainable parameters.
While our work is focused on technical aspects, it is important to consider limitations of our approach, such as privacy of data processed by outside peers, as well as broader impact of making LLMs more accessible. We discuss these issues and outline directions for future work in Appendix H.
<sup>10</sup>We use the Circuit Relay protocol from libp2p (libp2p, 2022) to traverse NATs and firewalls.
A Quality and efficiency of BLOOM with 8-bit quantization
As shown in Table 4, this method has little effect on LLM quality for major benchmarks. In terms of inference time, Table 5 demonstrates that quantization has about  $5\%$  of overhead with batch size 1 (20 tokens), but becomes negligible for larger batches.
B Estimating theoretical best throughput with RAM offloading
In this estimate, we use the best possible hardware setup for offloading: CPU RAM offloading via PCIe 4.0 with 16 PCIe lanes per GPU. In 8-bit, the model uses 1 GB of memory per billion parameters, and PCIe 4.0 with 16 lanes has a throughput of 256 Gbit/s. We assume an offloading latency of zero in the upper bound estimation. As such, offloading 176B parameters takes at least:
$$
\frac {1 7 6 \mathrm {G B} \cdot 8}{2 5 6 \mathrm {G b i t / s}} = 5. 5 \text {s e c o n d s}
$$
This gives the upper bound of  $1 / 5.5 \approx 0.18$  tokens/s for the inference speed.
C Extension to beam search algorithms
There are several variations of beam-search algorithm used for language model inference, including standard beam search, diverse beam search, constrained beam search, and more. A common thread between those algorithms is that they maintain a fixed number  $k$  of candidate sequences between steps. These sequences are informally referred to as the "beam". On every step, these algorithms generate possible continuations of sequences in the previous beam, then use some fitness criterion to select  $k$  of these continuations for the next beam.
From a computational point of view, this procedure is similar to simple "greedy" inference with a batch of  $k$  sequences. However, there is one important difference: unlike batched inference, beam search algorithms can "shuffle" candidate sequences between steps. In other words, 3rd best sequence from time step  $t$  can produce 1st or 2nd (or any other) sequence on the next step. Furthermore, a single sequence on time step  $t$  can produce multiple sequences selected for step  $t + 1$ .
Since different beam search variations use different criteria for selecting top sequences, we need a generic algorithm that can fit any criterion. In our system, we implement this by allowing clients to reorder server-side attention cache after each step. Formally, a client can send a list of at most  $k$  integers in range  $[1, k]$ , where i-th index specifies which previous attention cache should be used when generating  $i$ -th sequence of the next beam.
For instance, when given indices  $[2, 2, 1, 3, 2]$ , a server will use 2nd best sequence from step  $t$  to produce the new 1st, 3rd and 5th best sequences. Previous 1st and 3rd best sequences go to 3rd and 4th places, respectively. Finally, previous 4th and 5th sequences are discarded. From a technical point of view, servers implement this reordering by reordering attention cache with the specified indices (torch.gather operation) immediately before performing an inference step.
Table 4: Zero-shot accuracy for BLOOM-176B and OPT-175B with 8-bit and 16-bit weights.
Model	Bits	HellaSwag	LAMBADA	WinoGrande	Avg
BLOOM	16	73.0	67.2	70.1	70.1
8	72.8	68.1	70.1	70.3
OPT	16	78.5	74.7	72.6	75.3
8	78.5	74.6	71.7	74.9
Table 5: Generation throughput (tokens/s) for BLOOM-176B with 8-bit and 16-bit weights on  $8\times$  A100 GPUs.
Weights	Batch size
1	8	32
16-bit	4.18	31.3	100.6
8-bit	3.95	29.4	95.8
D Details of the server load balancing algorithms
Measuring throughput. Before joining for the first time, each server measures its Internet connection throughput (in tokens/second, using one of public web APIs for doing that) and GPU throughput (in tokens/second, using a small benchmark running several forward passes). The minimum of these values becomes the overall server throughput, which is then cached for future runs.
Initial block assignment. We assume that each server holds a segment of consecutive transformer blocks to minimize inference latency. Clients may request to perform a forward or backward pass for the whole segment of blocks or its subsegment, if necessary. Normally, each server loads as many blocks as it can fit in its GPU memory, unless a user limits the number of blocks to utilize the rest of memory for something else.
Before starting, each server calculates the values of  $t_i$  – the total throughput of servers currently holding the  $i$ -th block or loading it (to start holding it in a few minutes). Then, to find the best segment of blocks to serve, the server looks for the most narrow bottleneck in the network. Formally, if the model has  $L$  blocks and the server can hold  $K$  of them in its GPU memory, we calculate:
$$
\operatorname {s t a r t} = \underset {i = 1} {\arg \min } ^ {L - K + 1} \quad \operatorname {s o r t e d} ([ t _ {i}, t _ {i + 1}, \dots , t _ {i + K - 1} ]) \tag {1}
$$
Here, arg min compares the sorted arrays lexicographically and chooses the leftmost start in case of multiple minimums.
This way, the next joining server would always cover a block with the smallest  $t_i$ . If there are multiple bottlenecks like this, the server will try to cover as many of them as possible (we choose to cover the minimums first because the overall throughput is the minimum of throughputs among model blocks). Among the remaining options, we choose a segment covering as many second minimums as possible, and so on.
Quality of block assignment. While we are not aware of the exact polynomial-time solution for the problem of assigning the segments optimally, we have conducted computational experiments and found out that this greedy algorithm (running in polynomial time) usually finds an assignment with total throughput of  $90 - 100\%$  of the optimal one (found by trying out all possible assignments in exponential time), given that the values of throughput are realistic to our setup.
Rebalancing. Since servers may leave at any time, each server also periodically checks if the current assignment is "good enough" compared to the throughput estimated by running the greedy solution for servers currently present in the network.
Formally, each server periodically looks for a segment of blocks that is more appropriate than the currently loaded blocks with respect to the arg min rule (1). If it finds one, it simulates how the rest of the servers would behave if we replace the current blocks with the new ones (how other servers would change their blocks afterwards). If the eventual throughput is at least  $p\%$  better, the server commits to the change and announces that it changes the blocks, then other servers do the rest of the changes (eventually increasing the total throughput).
We use  $p = 20\%$  since it gives a reasonable trade-off between the swarm throughput and the frequency of block replacements in our experiments (see Appendix E). Specifically, a lower value of  $p$  leads to block replacements happening too often, which negatively affects the inference latency since each block replacement resets attention caches for this block.
Stability of the greedy algorithm. The rebalancing algorithm does not cause oscillations since a series of block replacements is executed only if it leads to eventually increasing throughput by at least  $p\%$ . Once a "good enough" throughput is achieved, servers do not change their blocks anymore (unless an essential number of servers join or leave). We verified this behavior computationally, simulating a network with thousands of servers with different throughputs.