Distributed Inference and Fine-tuning of Large Language Models Over The Internet
Authors: Alexander Borzunov
Max Ryabinin
Artem Chumachenko
Dmitry Baranchuk
Tim Dettmers
Younes Belkada
Pavel Samygin
Colin Raffel
Venue: NeurIPS 2023 poster (2023)
Keywords: volunteer computing
distributed deep learning
distributed inference
efficient inference
large language models
TL;DR: We propose a practical algorithm for running large language models by pooling together weak geographically distributed devices. Our system can inference Llama 2 (70B) and BLOOM (176B) over the Internet more than 10x faster compared to RAM offloading.

Abstract: Large language models (LLMs) are useful in many NLP tasks and become more capable with size, with the best open-source models having over 50 billion parameters. However, using these 50B+ models requires high-end hardware, making them inaccessible to most researchers. In this work, we investigate methods for cost-efficient inference and fine-tuning of LLMs, comparing local and distributed strategies. We observe that a large enough model (50B+) can run efficiently even on geodistributed devices in a consumer-grade network. This could allow running LLM efficiently by pooling together idle compute resources of multiple research groups and volunteers. We address two open problems: (1) how to perform inference and fine-tuning reliably if any device can disconnect abruptly and (2) how to partition LLMs between devices with uneven hardware, joining and leaving at will. In order to do that, we develop special fault-tolerant inference algorithms and load-balancing protocols that automatically assign devices to maximize the total system throughput. We showcase these algorithms in Petals — a decentralized system that runs Llama 2 (70B) and BLOOM (176B) over the Internet up to $10\times$ faster than offloading for interactive generation. We evaluate the performance of our system in simulated conditions and a real-world setup spanning two continents.

Distributed Inference and Fine-tuning of Large Language Models Over The Internet
Alexander Borzunov* HSE Univesity, Yandex
Max Ryabinin HSE Univesity, Yandex
Artem Chumachenko Neiro.ai
Dmitry Baranchuk Yandex
Tim Dettmers University of Washington
Younes Belkada
Hugging Face
Pavel Samygin
Colin Raffel
Yandex School of Data Analysis
Hugging Face
Abstract
Large language models (LLMs) are useful in many NLP tasks and become more capable with size, with the best open-source models having over 50 billion param- eters. However, using these 50B+ models requires high-end hardware, making them inaccessible to most researchers. In this work, we investigate methods for cost-efficient inference and fine-tun tne-tuning of LLMs, comparing local and distributed strategies. We observe that large enough model (50B+) can run efficiently even on geodistributed devices LLM efficiently by pooling together idle compute resources of multiple research groups and volunteers. We address two open problems: (1) how to perform infer- ence and fine-tuning reliably if any device can disconnect abruptly and (2) how to partition LLMs between devices with uneven hardware, joining and leaving at will. In order to do that, we develop special fault-tolerant inference algorithms and load-balancing protocols that automatically assign devices to maximize the total system throughput. We showcase these algorithms in PETALS system that runs Llama (70B) and BLOOM (176B) over the Internet up to 10x faster than offloading for interactive generation. We evaluate the performance of our system in simulated conditions and a real-world setup spanning two continents.
consumer-grade network. This could allow running
decentralized
Introduction
In recent years, the NLP community has found that pretrained language models greatly accelcrated progress many research problems through either fine-tuning (Radford et al. 2018) simple
prompting [Brown et al.)
2020). Their quality tends to improve as we increase model scale (Radford
al. 2019 Kaplan al.
2020 Following this trend, modern language models often have hundreds
of billions of parameters Brown et al. 2020 Rae et al. 2021 Zeng et al. 2021 Kim et al.) 2021).
Most recently, several research groups open-sourced their pretrained L.MSs with over 50B parame- ters (Zhang et al. 2022 BigScience] [022a Touvron al. (023a(b). However, they are still difficult to use due the sheer size in terms of parameters. For example, OPT-175B and BLOOM-1762 need over 350 GB accelerator memory for inference and even more for fine-tuning, As a result, even basic inference for these LLMs requires multiple high-end GPUs multi-node clusters. Recent studies
"Correspondence to: borzunov. alexander@gmeil.cm com PeTTaS source code and documentation are available at httpss:/pptals..o
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

BLOOM layers, part
BLOOM layers, part 2/3
BLOOM layers, part 3/3
Clients
French
Hindi
C++
You
Figure 1: A high-level overview of our system design. Servers store pretrained L.MM layers and temporarily hold attention caches for inferencing. Clients hold embedding layers and learned prompts/adapters (if used). Arrows denote temporary chains formed for inference.
propose algorithms for running large models with more affordable hardware (Pudipeddi et al. 2020 Ren et al. 202T), e.g. by offloading parameters to RAM. However, as we show in Section 3.1 these techniques are inefficient in many use cases, such as LLM-based chatbots and search engines.
In this work, we search for a more cost-effective way of running pretrained LLMs their main use cases: inference, in-context learning, and fine-tuning, We analyze latency and throughput for these use cases and determine which factors become dominant for very large models. Notably, for models with over 50B parameters, communicating activations over slow network can be faster than swapping layers from local RAM or SSD. Based on these observations, it should be possible to run LLMs cost-effectively by pooling together commodity hardware over the Internet.
However, existing LM algorithms are not designed to run inference with unreliable devices or high- latency networks. To bridge this gap, we formulate novel algorithm for fault-tolerant distributed autoregressive inference of very large models. Using dual attention caches, this algorithm can quickly recover from failed server and reassign the load to or make sure that there are enough servers for every part of the model, we develop a decentralzied load-balan boad-balancing algorithm that assigns transformer blocks to every server maximize the total system throughput. The fully decentralized nature of these protocols allows participants to add or remove their devices any point, making optimal use of GPU idle time.
replacement servers. Finally, to
We summarize the main contributions of this work as such:
We analyze the problem of cost-efficient LLM inference and propose a novel algorithm that can inference large (50B+) language models on distributed unreliable devices. To the best of our knowledge, this is the first algorithm that can inference LLMs with 50B+ parameters in this setup.
Using this algorithm, LLMs over the Internet. The system allows users to run inference and fine-tuning over a swarm of unreliable devices with the same correctness guarantees when running locally. The system runs persistently with the help of volunteers.
develop PETALS
decentralized system for inferencing and fine-tuning
We benchmark the performance of the proposed algorithms on Llama 2 (70B) (Touvron et al. 20236) and BLOOM (176B) (BigScience (022a). We run experiments in controlled conditions, with simulated network latency and server failures, and in the actual geo-distributed system spanning two continents. With realistic network speeds, our distributed algorithms perform autoregressive generation > 10x faster than local offloading.
2 Background: efficient training and inference
There a wide variety of methods optimizing training and inference for most deep learning workloads. Here, we focus on two areas relevant for our analysis: model parallelism and parameter offloading.
2.1 Model parallelism
Model parallelism a family distributed training algorithms that assigns each device to hold a subset of model parameters, run subset of computations and communicate output activations. Tensor parallelism assigns each device to compute a subset of each model layer (e.g., subset of neurons), then communicate results between each other and proceed to the next layer (Krizhevsky et al. 2012

---

Ben-Nun & Hoefler, 2019; Tang et al. 2020). Each device performs a symmetric computation, applied to a different slice of model weights, which makes tensor parallelism compatible with MPI- based communication. In turn, the main performance overhead of this strategy comes from all-to-all communication (and synchronization) after each layer (Krizhevsky 2014).
Pipeline parallelism reduces the communication overhead by assigning each device with one or several full layers (Huang et al. 2019 Narayanan et al. 2019 Yang et al. 2019). During the forward pass, each stage applies its subset of layers to the inputs supplied by the previous stage, then sends the outputs of the last layer to the next stage. For the backward pass, this process is reversed, with each pipeline stage passing the gradients to the same device that previously supplied it with input activations. To better utilize the available devices, the pipeline must process multiple microbatches per step, allowing each stage to run in parallel on a different batch of inputs. Even with optimal execution, some of the pipeline stages will remain idle some of the time (Huang et al., 2019).
Both of these strategies are actively used for training LLMs. Real-world distributed training systems usually combine multiple forms of parallelism depending on hardware and network type (Narayanan et al. 2021; Rajbhandari et al.) 2020 Jia et al. 2019). Tensor parallelism is typically used within single multi-GPU server closely interconnected TPU cores (Narayanan al.] [202T] Shazeer
et al. 2018). In turn, pipeline parallelism is used to connect multiple servers (Narayanan et al. 2021)
Recent works demonstrate that model parallelism can be used for cost-efficient pre-training LLMs
by pooling together idle GPU devices (Athlur et al. 2022
Wang et al. 2022
Kuszmaul 2022 Yuan
al. 2022 Ryabinin et al., 2021).
2.2
Offloading
Parameter offloading relegates model parameters from accelerator memory to a slower but cheaper storage: typically RAM SSD (Pudipeddi et al., 2020 Ren et al., [2021] Rajbhandari et al. (002)) When using the model, parameters are loaded to the accelerator just-in-time for computation, one or few layers at a time. In principle, this method allows running large models with single low-end accelerator as long as there is enough RAM (or SSD) to store the model.
The main drawback of this strategy is having load and unload through all model parameters for each forward and backward pass, which can be time-consuming This extra time can be amortized in workloads where model can do lot of useful computations for each time a parameter is loaded. In practice, using offloading to run a single token through the OPT-175B on one GPU in the best-case scenario of hardware and bandwidth would require 11 seconds per forward pass, or twice that for training. As we show in Section 4 real-world performance is significantly slower.
Pudipeddi et al. (2200) circumvents this by training with very large batches, and hence, increasing the computation. In turn, Ren et al. (202T); Rajbhandari et al. (202T) reduce the overhead by overlapping communication and computation, that is, doing useful computation for the current layer while waiting for the transfer of the next layer to finish. Some of these systems Ren et al. (202I) also partition offloaded parameters between devices. However, unlike model podel-parallel training, distributed offloading still requires each device to compute the full model.
3 Method
Using pretrained large language models for NLP tasks consists of two main workloads: inference and The inference workload typically consists of encoding an input text, then generating tokens autoregressively. In turn, fine-tuning requires updating either all of the model's parameters or (more commonly for large models) small set of trainable weights (e.g., adapters or soft prompts) by backpropagation. These two workloads also cover more advanced use cases:
Manually engineering prompts for a given task, then deploying the model with these prompts.
Fine-tuning with adapters (Hu et al.) 2021 Houlsby et al. 2019 Liu et al. 2022b) or "soft"
prompts (Liu al. 202lb Lester et al. 2021
Liu al. 202 and inferencing fine-tuned models.
Distillation into a smaller task-specific model for faster inference (Schick & Schütze 2021
"Specifically, 16- bit parameters, PCLe gen. 4 31.5 GB/s (16 lanes), infinite compute and memory bandwidth.
3

---

Counter-intuitvelly, we found that inference is more challenging than fine-tuning for cost-efficient setups. To that end. we dedicate most of this section to anference-speciife problems. As for fine-tuning, we describe a way to support arbitrary parameter-effiient fine-tuning in Section 3.4
3.1 Performance bottlenecks of LLM inference
Unlike training, autoregressive LLM inference cannot be done with single pass through the model. Instead, the model needs to process one token at a time, pass it through the entire model, then generate the next token and repeat the process. In case of model parallelism, training an n-laye n--ayeffmodd model sequence of / tokens needs (n) communication rounds, while generating the same sequence needs O(n t) rounds, making it more susceptible network latency. Similarly with parameter offloading, generating a sequence of t tokens needs loading every layer times, which also takes Om t) time.
The other problem of autoregressive generation is dealing with attention for past tokens Vaswani iet al., 2017). During an inference step t, each layer needs to attend to f f- 1 previous attention keys and values. Existing inference algorithms store past entries in accelerator memory. Caching half- celf-precision activations of 2048-token sequence for large models like GPT-3 (Brown et al. 2020) or OPT-175B (Zhang et al. (2022) (with 96 layers of 12288 units each) takes up 9.6 GB GPU memory for each sequence. Offloading these cached values faces the same problems as offloading in general.
An alternative solution is to recompute all previous tokens every inference step, storing only one set of keys values at time. Naturally, this approach needs increasingly more computation with sequence length f, for total of (t²) time for transformer- model [3] Surrrisingly, this approach is often more efficient than offloaded caching, especially for shorter sequences due to the overhead from loading and storing cache from RAM or SSD.
Parameter offloading can still be efficient when generating large amounts of short sequences in bulk. Each individual sequence still takes a long time to generate, but the system maintains high throughput by running many samples in parallel. Unfortunately, this scenario does not cover many important LLM use cases, For instance, it is incompatible with in-context learning or prompt engineering, where the model needs to process long sequences of training examples (Brown et al.) 2020). More importantly, it does not support "interactive" applications where LLM needs to quickly respond to a user input. This rules out many LLM applications such as conversation systems or input completion (e.g. ChatGPT Smart Compose).
Hence, we explore a new solution based on appline-parallelissi A related line of work (Aminabadi et al., (022)) investigates model parallelism to inference LLMs in GPU clusters. However, their approach does not apply to our more affordable setups: cheap "preemptible" instances or connecting existing resources over the Internet. To operate in these conditions, an inference algorithm needs to deal with node preemption, network errors, and high latency.
3.2 Distributed generation with fault tolerance
In this section, we formulate an algorithm for inferencing LLMs in fleet of unreliable geographically distributed devices connected over the Internet. Each device act a server, client, or both. client is a node operated by the user, which runs inference or fine- seeettnnig jobs through the swarm of servers. A client only holds input and output embeddings (< 3% of model weights for BLOOM- 176B) and delegates running transformer blocks (the most expensive computations) to remote servers. A server is GPU-enabled node holding set of consecutive transformer blocks and processing requests coming from client nodes.
For simplicity, we assume that every block is hosted several servers and examine this assumption in the next section. Following this notation, a fault-tolerant algorithm should allow each client to complete an inference job with reproducible results even if some remote servers fail during inference.
As we discuss in Section 3.1 autoregressive generation requires many sequential communication rounds, making it sensitive to network latency. However, if every device stores its past attention
3 'Here and below, the term model layer (or block) refers to one transformer block that typically combines self-attention, a food-forward network, normalization layers, and residual connection (Vaswani et al. 2017). All public LLMs with 100B+ parameters use standard attention that scales (( 2) for sequence length n.

---

cache, every round only transfers activations for single token, i.e. several kilobytes of date We use this model to directly minimize the inference time over possible pipeline configurations. As we show later in Section 4.2 this allows efficient inference over a low-bandwidth Internet connection.
A more challenging problem is how recover from node and network failures. If a remote server shuts down, any cached attention keys stored on that server will be lost with it. There are two naive solutions to this problem: restarting inference from scratch or recomputing past embeddings on every step. Restarting might be enough at a small scale. However, running 50B+ models may involve many unreliable devices, making it unlikely to generate long sequence without at least one failure. In turn recomputing past attention caches requires communicating past tokens on every communication round, resulting in On -t)) total data transferred, where the number of pipeline layers and is the sequence length. In other words, both these solutions struggle to generate long sequences.
We address this problem by maintaining two types of cache: server-side cache holds past attention keys and values for their layers, like in existing inference algorithms, while client-side cache holds past imputs sent to a given pipeline stage] If server disconnects, client can find another server with that pipeline stage and use client-side cache to restore the server state.
The resulting procedure is described in Algorithm II For every pipeline stage, the client maintains heap (priority queue) of servers that hold this stage (and may hold additional stages). The servers in queue are ordered by the network latency, measured from past communication. These queues are maintained through the lifetime of a client. To begin generation, the client runs a beam-searche search- like procedure find sequence of servers that results in the least total inference time under our performance model. When running inference steps, a client keeps track of intermediate activations sent between pipeline stages. If remote server fails or leaves, the client retrieves the next best server (or multiple servers) and requests it to restore the attention state from the client's ' cached activations.
When servers fail, the algorithm needs to send O(t) data (in one round) for each failed server and compute only the stages held by the failed servers. This can be seen as an interpolation between naive and cached inference, depending on the server failure rate. If none of the servers fail, we recover On t) communication, similarly to Aminabadi et al. (2022). In turn, if all servers fail after one step, the algorithm effectively performs non-caching generation, which is the best option in that scenario.
In the basic formulation, all communication between pipeline stages is routed through the client, i.e. the client receives the outputs of every pipeline stage, caches it and sends it to the subsequent stage. In practice, it is more efficient to let pipeline stages communicate directly: once the server obtains output activations, it sends them to both client and the subsequent stage. This reduces the total step time since both messages are a few kilobytes in size an can be sent in parallel. To verify that both client and the next pipeline stage received the same set of activations, they can verify the checksums (i.e. hash values) of the received activations asynchronously. without blocking computation.
Algorithm| I can support greedy inference or any sampling variants (including Holtzman et al. (2020)) However, requires one more step to support search-basse algorithms such as beam search: cache reordering. This allows client to generate multiple continuations of the same input prefix by cloning its attention cache and dropping less likely hypotheses. We describe beam search in Appendix C
Shortest path routing, In the Algorithm[] I the of servers that can run the required layers the least amount of time. To estimate this time we add up two factors: computation time, determined by server's compute throughput ("GPU speed") and the network latency between the client and that server. Servers measure their own compute throughput and share this information with the clients. In turn, clients measure the network latency between them and a given server by "pinging" the candidate servers during routing, If a server runs multiple consecutive blocks, we multiply the computation time by the number of blocks.
function (line 4) selects sequence
To find the best chain of servers, clients find the shortest path between the first and last block, using graph where edge weights correspond to server inference time, as described in the previous paragraph. To minimize overhead, we do not run pathfinding from scratch each call to find herd_bee__hhaaalee _chaccchhn. Instead, clients run lifelong pathfinding in the background and reuse it between inference calls. More specifically, we use the D* Lite (Koenig & Likhachev 2005) algorithm because it allows clients to quickly adjust paths after server is banned or leaves the network.
$ FFor GPT-3 and OP5-175B, one 12288-dimansoonal token embedding in 16-bit precision takes up 24 KiB. ' Heer,, pipeline stage is set of consecutive model layers hosted server (as in pipeline parallelism).
5

---

Algorithm Generating sequence, client-side code
Algorithm 2 @rpp_pnferenneeee (pp_inference(sevee))
Input: prefix_tokens, embeddings, known_servers
Input: local_layers, stream
I: generated_sequentt
list(
1: cache = dictionary()
2: cache = dictionary()
2: for layer E layers_ do
3: streams dictionary()
3: cacheflayer] make @aakk_eepty[
4: chain = find best_ chain(known_server)
4: end for
5: for server chain do
5: while not stream do
6: streams[server] rpc____t__ infereence(server)
6: inputs stream.receive
7: cache[server] = list()
7: for layer E local local_layers do
8: end for
8: past_kv = cache[layer]
9:
9: inputs, new forward
10: inputs =embeddiigg//rreiiittoknnn
10:
layer, inputs, past_kv) Saaeelllayrrapppnd((eeee kv)
11: while thbouid_cooiimeeegeeerrreeeeeeed
do
11:
12:
tail. servers copy(chain)
12: end for
13:
while not cmpty(tail servers) do server = tail_ wtt_ssseeeeeppeeeee..
13:
stream.send(inputs)
14:
14:
end while
15:
try:
16:
> Attempt normal inference
Algorithm 3 replace ceeplac_faille___ee _eiver_server(..))
17:
outputs streams senu(t)
Input: server, inputs, cache, streams, smown_serveet
18:
cache[server). aapperrreer..pppeednnntttttt
19:
inputs = outputs
1: known smown_serrvrs.bor banterr)
20:
catch ServerFailed:
2: missing get la
21:
> Replace the failed server
3: chains = select_by_laae
22:
streams. cettaansspoppsseee.lllooooo
4: Known_servers missing_ layers)
23:
past_ post__nputs= = cache, )op(serer))))))
5: chain find btseebestthhhheleeeeeeeeeeeeeb cosssschinnchhins)
24:
new_servers = replace_faille failed siiiled_srreert
6: replacements list()
25:
server, pass_iinpppaa pesst_nputt, cache,
7: while not empty(chain) do
26:
streams, known_ known_servers)
8: s= c=cain.pop_leftt)
27:
chain. maaii.replace(server, new_serveen tail fail_servers.pusi left(new_servee)
9: try:
28:
10: streams[s] rre__pppinii infereenc(ss)
29:
end while
11: outputs = streams[s] senddisssssssssssssss
30:
12: replacements.appendds
31: logits = cooppueelogits(ouuue
embeddings)
13:
cache[s] inputs
32:
next soxt_token- = choose _nctsslgiss(((tss)) {e.g. greedy)
14:
33: generated_sequeenee sequeencc..ppeennnnett (
15:
inputs= = outputs
34: inputs = embeeddnngs(neextooknn
16:
catch FailedRPC:
35: end while
17:
known_servers.ban
36:
18: chains = select_by_layee
37: for server chain do
19: chains, missing_layers)
38:
seeamms(srvvrr..lossses
20:
chain find find_best_chee_eeeeeeeeeeeeeeeee best_chain(chains)
39: end for
21:
end while
40: return generated_ seequee
22:
return chain
3.3 Automatic load balancing
In order to run inference or fine-tuning, each server needs to be assigned to a pipeline stage, then reassigned if other servers join leave the network. For example, if we deploy an LLM on idle compute resources from several data centers or labs, the number of participants may change over time based on the demand. Moreover, servers may have different compute throughput, network bandwidth, and geographical location. To operate in these conditions efficiently, servers should automatically choose which model layers they should serve in given situation.
To that end, servers periodically run a load balancing procedure and switch to new blocks necessary. Formally, servers choose blocks so to maximize the total system throughput (tokens per second), Each server periodically announces its blocks and empirically measured throughput to a distributed hash table (Maymounkov & Mazieres 2002). When identify a contiguous interva of blocks that would increase the total system throughput the most.
server joins, it uses this information to
This interval is always contiguous, since splitting it would harm the inference latency.

---

Since peers may leave or fail at any time, all nodes periodically check if launching rebalancing procedure would significantly improve the overall throughput. If it is the case, they switch layers until the throughput becomes near-optimal. In particular, if all peers serving certain blocks suddenly leave the system, this procedure quickly redistributes the remaining resources to close the emerged gaps.
We provide a detailed description of the load balancing algorithms in Appendix D and validate their properties in experiments reported in Appendix E
3.4 Parameter-efficient fine-tuning
While LLMs achieve high quality on many problems with simple prompt engineering (Brown et al., 2020), they often need training to achieve the best results. Traditionally, this is done by fine-tuning all model parameters the downstream task. However, for extremely large models, this strategy becomes impractical due to hardware requirements. For example, fine-tuning BLOOM-1761 with Adam would require almost 3 TB of GPU memory to store the model, gradients, and optimizer states.
Fortunately, parameter-efffcient fine-tuning methods have been developed that keep most of the pretrained model intact. Some of them choose a subset of existing parameters to update (Sung et al. 2021: Guo et al. 202T) while others augment the model with additional trainable weights (Hu et al., 2021 Houlsby et al. 2019 Liu et al. 2021b Lester et al. 2021 [Lu et al.) 2221a 2022b). Despite crrrammttr--fffcien approaches are often competitive with full model fine-tuning (Hu et al. [2021] Liu et al. [021a] Yong & Nikoulina 2022) and even outperform in low-data regimes (Liu et al. 2022a). Another appealing property of these approaches for our use-case is that they allow rapidly switching a pretrained LLM between adapters.
their lower memory requirements,
By focusing on parameter-efficient fine-tuning, we are able to simplify the system design by making clients responsible for storing their trainable parameters (see Figure[] Servers can run backpropa- gation through their layers and return gradients with respect to activations, but they do not update the server-side parameters. Even when client communicates learned values (e.g, soft prompts) to a server, the server treats these values same as input activations. Thus, a server can simultaneously run different fine-tuning tasks without them interfering with one another. This design choice also allows users to define custom adapters in simple PyTorch without having network engineering expertise.
Unlike inference, fine-tuning forward and backward passes process the entire batch at one go and do not need store past attention caches between successive client requests. Thus, in case of a failure, we can discard the incomplete forward/backwaar pass and just repeat the previous forward/backwward pass request. This algorithm behaves similarly to the cache-less bascline from Section
3.5 Implementation details
Since main intended use-case is rumning on inexpensive low-end devices, we need work around their capabilities. In terms of raw FLOPs, even consumer- gaaaddd GPUs like GeForce RTX 3070 could complete inference step of BL0OM-176B in less than second (VVIDIA 2020). However, the GPU memory can only hold a small fraction of model layers: running naively would require 44 RTX 3070 GPUs and 44 communication rounds. To make this more efficient, we quantization to store more parameters per GPU, reducing the number of consecutive devices and communication rounds.
One option for quantization is to use 8-bit mixed matrix decomposition for matrix multiplication to quantize the weights to 8-bit precision and reduce the memory footprint compared to 16-bit weights, as suggested in Dettmers et al. (2022a). This decomposition separates hidden states and weights into two portions: about 0.1 1 of 16-bit outlier and 99.9% of 8-bit regular values, which roughly halves the memory footprint with negligible effect on the model quality (see evaluations in Appendix AA) Another option is to use the 4-bit NormalFloat format (Dettmers et al. 2023).
To send less data between subsequent pipeline stages, we apply dynamic blockwise quantization (Dettmers et al.) (2022)) to the hidden states before hipeline-parallel communication, which halves the bandwidth requirements without any noticeable effect on generation quality (Ryabinin et al./ (021)) During fine-tuning, we also take advantage of gradient checkpointing (Griewank & Walther 2000 Chen et al. 2016) and half precision to reduce VRAM usage - both are standard practice for large language models (Narayanan et al., 2021 Brown et al. 2020 Athlur et al. 2022). In experiments, we apply the same optimizations to baseline systems for a fair comparison.
7

---

Table 1: Sequential inference speed (steps/second) of BLOOM (7.1B) with varying failure rates. A failure rate means that sending any set of activations to the next stage of the pipeline fails with probability p. Missing values mean that the algorithm did not finish within I hour.
128 tokens, failure rate:
1024 tokens, failure rate:
Inference Algorithm
le-4
le-3
le-2
0
le-4
le-3
le-2
Caching with restarts
17.1
16.7
12
0.18
15.5
11.8
0.48
Cache-less inference
3.44
3.44
3.44
3.44
0.89
0.89
0.89
0.89
Algorithm (ours)
11.4
11.4
10.6
3.38
10.7
10.7
7.76
2.17
4 Experiments
4.1 Inference with unreliable servers
First, we conduct small-scale preliminary experiments to test the fault-tolerant generation algorithm described in Section 3.2 For these experiments, we use a smaller BLOOM model with 7.1 billion parameters (BigScience) 2022b). This model contains 30 transformer blocks with hidden size 4096. We compare our algorithm with baselines when generating a single sequence of length 512. For simplicity, we run all computations and communications in single precision and disregard word embeddings and logits for this set of experiments. We measure the time to run a certain number of tokens through all blocks and simulate failures by resetting pipeline stages at a certain rate.
We compare three inference strategies:
1. Caching with restarts, which refers to standard inference with servers storing attention caches. On failure, it restarts the entire generation from scratch since the failed server's 's caches are lost.
2. Cache-less inference, which reruns past tokens on every step. On failure, it restarts only the last generation step.
3. Algorithm
L which is specifically designed for hoult-tolerant inference.
All runs use four pipeline stages with (8. 7, 8, 7) model layers per pipeline stage. Each pipeline stage is served by a single GeForce 1080 Ti GPU; the four GPUs are running in a single system with dual Xeon Gold 6148 CPU, 12 DDR4 LRDIMM sticks with 64 GB each. The system has 16 dedicated PCIe Gen. lanes per GPU in dual root configuration, without using PCIe switches. Each stage runs in an isolated Docker containers with virtual network interfaces, but there is no limit to communication bandwidth for this experiment. We repeat all experiments 50 times and report the average time. The adjusted standard deviation never exceeds O. 0.2%. We use the pipeline parallelism implementation from Megatron- -eeeppee- (BigScience et al., 2022) for the cache-less baseline.
We report performance measurements in Table Unlike baselines, our algorithm provides reasonable performance in all tested conditions, especially for higher failure rates (common for communicating over the Internet, using ppot/preemptibe instances or unreliable hardware). Caching with restarts is most efficient for inference without failures, with our algorithm being somewhat slower due less mature implementation. Finally, the cache-less inference can be competitive for short sequences (128 tokens), but slows down considerably on 1024 tokens, which agrees with our intuition from [..]
We provide plots showing additional evaluations for a wider range of failure rates (up to 5%) and sequence lengths (up 2048 tokens) in Appendix F (Figure 3).
4.2 Experiments for Llama (70B) and BLOOM (176B)
In this section, we evaluate our system on more practical tasks of running Llama (70B) (Touvron
et al. (023b) and BLOOM (176B) (BigScience) 2022a). First, consider servers rumning in a network with controlled bandwidth and latency
3 We measure performance for (a) Llama
distributed across 3 servers with a T4 GPU each, (b) BLOOM distributed across 3 servers with
1000 (80 GB) GPU each, and (c) BLOOM distributed across 10 servers with an RTX 3090 GPU each. We use 4-bit NormalFloa quantization (Dettmers al. (2023) for Llama and 8-bit matrix decomposition (Dettmers et al. 2022a) for BLOOM in all evaluations including the baselines below.
We simulate network conditions using tc qdisc.
8

---

Table 2: Performance of Llama (70B) sequential inference steps and parallel forward passes. The network parameters refer to bidirectional bandwidth and round-trip latency (RTT).
Sequential inference (steps/s, each client)
Parallel forward (tokens/s, each client)
GPUs
Clients Bandwidth
RTT
Sequence length
Batch size
128
2048
1x128
64 128
Gbit/s
<5 ms
2.29
2.02
45.4
155.
100 Mbit/s
< 5 ms
2.29
2.01
37.5
140.2
3x (16 GB)
100 Mbit/s
100 ms
1.57
1.44
23.7
128.7
Gbit/s
< 5 ms
2.02
1.74
21.2
124.2
Offloading
0.139
0.139
18.0
139.9
Table 3: Performance of BLOOM (176B) sequential inference steps and parallel forward passes.
Sequential inference (steps/s, each client)
Parallel forward (tokens/s, each client)
GPUs
Clients Bandwidth
RTT
Sequence length
Batch size
128
2048
1x 128
64x 128
Gbit/s
<5 ms
1.71
1.54
70.0
253.6
1
100 Mbit/s <5 ms
1.66
1.49
56.4
182.0
100 Mbit/s
100 ms
1.23
1.11
19.7
112.2
3x A100 (80 GB)
3
1 Gbit/s
< 5 ms
1.65
1.49
Offloading
0.0495
0.0495
2.5
152.4
Local PP (NVLink)
2.46
2.28
98.4
279.5
Gbit/s
<5 ms
1.65
1.54
59.1
230.1
3
Gbit/s
<5 ms
1.65
1.54
54.7
221.4
10x RTX 3090
Gbit/s
<5 ms
1.17
1.01
31.0
131.0
(24 GB)
10
100 Mbit/s
<5 ms
1.05
0.99
20.1
28.1
100 Mbit/s
100 ms
0.34
0.33
6.5
16.8
Offloading
0.0427
0.0427
2.2
109.3
Gbit/s
< ms
1.24
1.06
37.9
180.0
12x heterogeneous (virtual servers)
100 Mbit/s
< ms
1.24
1.05
25.6
66.0
100 Mbit/s
100 ms
0.57
0.53
5.8
44.3
Gbit/s
<5 ms
0.90
0.86
14x heterogeneous
Real-world setup
0.83
0.79
32.6
179.4
Theoretical-best
Offloading
0.18
0.18
2.7
170.3
We report performance of:
Sequential (autoregressive) inference for batch size (i.e., each step generates 1 token). It is measured generation steps per second a client can do and shows the generation latency.
Parallel forward passes for batches of 128-token sequences" It is measured in tokens per second a client can process. This shows the system's throughpur during batch processing and fine-tuning.
Since the backward pass performance depends on set of trainable weights, batch size, and other hyperparameters, report its performance in different setups separately in Appendix G
Concurrent clients. We also investigate the effect of having concurrent clients. We assume that each server belongs to a different person, and multiple people (possibly, all of them) are interested in running inference fine-tuning at the same time. In order to do that, they run the client interacting with our distributed system. The client runs on the same machine, uses 8 CPU cores and no GPU, We report the speed of sequential inference and parallel forward passes that each client gets average.
IItenaalyy, large batches split into micro- batches of 1024 tokens each minimize pipeline bubbles.

---

Offloading baseline. We also evaluate parameter offloading, where each user runs independently on single GPU, swapping parameters from CPU memory. First, we report the actual throughput of RAM offloading in case of DeepSpeed with default recommended parameters and enabled pin_memory (gives 1.2-2x 2x speedup). Next, we report the theoretical-bet throughput the offloading baseline can reach for BLOOM. It is calculated as a maximal throughput in the best hardware setup possible (CPU RAM offloading via PCIe 4.0 with 16 PCIe lanes), assuming infinite GPU performance. The calculations are detailed in Appendix B
Local pipeline parallelism (NVLink). Next, report performance for BLOOM running on server with x A100 (80 GB) GPUs. In this setup, a single server has enough GPU memory to load the entire model, which provides an upper bound for performance reachable with these GPUs. This setup rums pipeline-paraaleinn parallelism from DeepSpeed v0.7.7.
Heterogeneous servers. To validate that our system works on heterogeneous hardware, we simulate 12 heterogeneous devices by partitioning each A100 (80 GB) into several virtual servers (3 large and 1 small). We get 9 servers hosting 7 blocks each, server with 3 blocks and two more servers with 2 blocks (70 blocks in total, as required for BLOOM). Additionally, we benchmark the system on real heterogeneous GPUs with diverse compute capabilities in the "Real-world setup" below.
Real-world setup. Finally, we benchmark BLOOM in a real-world setup with 14 smaller servers holding 2xRTX 3060, 4x 40800T,, 2x3090, 2x A4000, and 4xA5000 GPUs. These are personal servers and servers from university labs, spread across Europe and North America and connected to the Internet at speeds of 100-1000 Mbit/s. Four of the servers operate from behind firewalls 10
Analysis. We report the results for Llama 2 in Table 2 and for BLOOM in Table 3 For inference, performance does not depend much on bandwidth or sequence length but degrades with higher latency. In turn, fine-tuning forward passes for large batches are affected by both bandwidth and latency.
We can see that the offloading baseline is about an order of magnitude slower than our system for inference, both in practice and in the theoretical-best setup assuming an infinite GPU performance. For parallel forward passes, offloading is competitive if networking is limited to 100 Mbit/s or has high latency. In other cases, algorithm offers higher throughput than offloading for training.
Crucially, our system significantly outperforms offloading even when each GPU node runs its own client doing single-batch inference at the same time. Thus, given the same hardware, group of researchers will get much better inference speed by collaborating over the Internet using our system compared to each of them running offloading independently.
Finally, the real-world setup turns out be slower than the A100 benchmarks due slower hardware. Still, our algorithm outperforms offloading even when communicating between different continents.
Additional experiments. We conduct two additional experiments to test individual components of our system. We evaluate the load balancing from 3.3 in isolation in Appendix [E] We also evaluate the performance of model compression from Section 3.5 in Appendix A To reiterate, for each model, use the same compression strategy in our system and all baselines. Finally, we perform qualitative evaluation of fault tolerance by shutting down random servers during inference and fine-tuning to verify that the algorithm produces correct outputs and gradients.
Conclusion
In this paper, we introduced novel fault-tolerant algorithm for inferencing large language models. On top of it, we introduced a decentralized system for running L.MMs on distributed unreliable devices connected over the Internet, which significantly outperforms other approaches to running inference on consumer-gres gradde hardware. We demonstrated that the proposed system can scale to the largest publicly available language model with hundreds of billions of trainable parameters.
While our work is focused on technical aspects, it important consider limitations of our approach, such as privacy of data processed by outside peers, as well as broader impact of making LLMs more accessible. We discuss these issues and outline directions for future work in Appendix
We use the Circuit Relay protocol from libp2p [Iip2pp 2022)
traverse NATs and firewalls.
10

---

References
A121.
Jurassic-1 language models.
https://studio.aaie com/docs/
wwrassic--Lanuage--mee
Accessed: 2022-06-22.
Aminabadi, R. Y., Rajbhandari, S., Zhang, M., Awan, A. A., Li, C., Li, D., Zheng, E., Rasley, J., Smith, S., Ruwase, O., et al. Deepspeed inference: Enabling efficient inference of transformer 2022.
models at unprecedented scale. arXiv preprint arXiv:2207.000.
Athlur, S., Saran, N., Sivathanu, M., Ramjee, R., and Kwatra, N. Varuna: scalable, low-cost training of massive deep learning models. In Proceedings of the Seventeenth European Conference Computer Systems, pp. 472-487, 2022.
Ben-Nun, T. and Hoefler, T. Demystifying parallel and distributed deep learning: An in- concurrency analysis. ACM Comput. Surv., 52(4), aug 2019. ISSN 0360-0300. doi: 10.1145/ 3320060. URL
BigScience. BLOOM: 176B-parameter open-access multilingual language model. ArXiv, abs/2211.05100, 2022a.
BigScience. A version of BLOOM with 7.1 billion parameters. bigscience/bloom-// 7b1 2022b,
https://huggingface.com
BigScience, Microsoft, and NVIDIA. The fork of LMM and Megatron-DeeppSpecd by htg.gggieeccceooooo.oogggggtrraa
BigScience. bttpps://ght thub.
2022.
Black, S., Biderman, S., Hallahan, E., Anthony, Q.. Gao, L.., Golding, L.., He, H., Leahy, C.,
McDonell, K., Phang. J., Pieler, M., Prashanth, U. S., Purohit, S., Reynolds, L., Tow, J., Wang, B., and Weinbach, S. 2pt-neox-200: An open-source autoregressive language model, 2022. URL https://art
Brown, T. B., Mann, B., Ryder. N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. arXiv preprint arXiv:2005. 14165, 2020.
Chen, T., Xu, B., Zhang, C., and Guestrin, C. Training deep nets with sublinear memory cost. arXiv preprint arXiv: 1604. 04774, 2016.
Dettmers, T., Lewis, M., Belkada, Y., and Zettlemoyer, L. LLM.int8(): 8-bit matrix multiplication for transformers at scale. ArXiv, abs/2208.07339, 2022a.
Dettmers, T., Lewis, M., Shleifer, S., and Zettlemoyer, I.. 8-bit optimizers via block-wise quantization. International Conference on Learning Representations (ICLR), 2022b.
Dettmers, T., Pagnoni, A., Holtzman, A., and Zettlemoyer, L. Qlora: Efficient finetuning of quantized 1lms. arXiv preprint arXiv: 14314, 2023.
Du, N., Huang, Y., Dai, A. M., Tong, S., Lepikhin, D., Xu, Y., Krikun, M., Zhou, Y., Yu, A. W., Firat, O., Zoph, B., Fedus, L., Bosma, M., Zhou, Z., Wang, T., Wang, Y. E., Webster, K., Pellat, M., Robinson, K., Meier--ellstern, K., Duke, T., Dixon, L.., Zhang, Le, Q. V., Wu, Y., Chen, Z., and Cui, C. Glam: Efficient scaling of language models with mixture-of-experts CoRR, abs/21 12.06905, 2021. URL /ttttss////aiivv htttt:::/aarxi..or////////111111 26960
Evans, D., Kolesnikov, V., Rosulek, M., et al. pragmatic introduction to secure multi-party computation. Foundations and Trends in Privacy and Security, 2(2-3):70-246, 2018.
Face, H. and contributors. Accelerate: Run your raw pytorch training script on any kind of device. GitHub. Note: https://github.coom// htwp://ww...oooouggggggeeeaatat. 2020.
Fedus, W., Zoph, B., and Shazeer, N. Switch transformers: Scaling to trillion parameter models with
simple and efficient sparsity, 2021.
Gao, L., Tow, J., Biderman, S., Black, S., DiPofi, A., Foster, C., Golding, L., Hsu, J., McDonell, K., Muennighoff, N., Phang, J., Reynolds, L., Tang, E., Thite, A., Wang, B., Wang. K., and Zou, A. framework for few-shot language model evaluation, September 2021. URL https //doi.org// 10. #281/zenodo 5371168
11

---

Griewank, A. and Walther, A. Algorithm 799: revolve: an implementation of checkpointing for the reverse or adjoint mode of computational differentiation. ACM Transactions on Mathematical Software (TOMS), 26(1):19 2000.
Guo, D., Rush, A. M., and Kim, Y. Parameter--effiient transfer learning with diff pruning. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics, 2021.
Holtzman, A., Buys, J., Du, L., Forbes, M., and Choi, Y. The curious case of neural text degeneration. In International Conference Learning Representations, 2020. URL https://openrevieew net/forum?iddrrg@@yyyvv
Houlsby, N., Giurgiu, A., Jastrzebski, S., Morrone, B., De Laroussilhe, Q. Gesmundo, A., Attariyan, M., and Gelly, S. Parameter-efficien transfer learning for nlp. In International Conference Machine Learning, pp. 2790-2799. PMLR, 2019.
Hu, E., Shen, Y., Wallis, P.. Allen- Z., Li, Y., Wang, L., and Chen, W. Lora: Low-rank adaptation of large language models, 2021.
Huang, Y., Cheng, Y., Bapna, A., Firat, O., Chen, D., Chen, M., Lee, H., Ngiam, J., Le, Q. V., Wu, Y., et al. Gpipe: Efficient training of giant neural networks using pipeline parallelism. In Advances in Neural Information Processing Systems, pp. 103-112, 2019.
Jia, Z., Zaharia, M., and Aiken, A. Beyond data and model parallelism for deep neural networks. In Talwalkar, A., Smith, V., and Zaharia, M. (eds.), Proceedings of Machine Learning and Systems, volume pp. 1-13, 2019. URL httpt:t:/ppocceeeinnggeeo///// mlsys.sssss.........mmm..mm....s orgoorg//aaao hwtgg/ppp///19/9/ File/c74497001 htwp://0000.....eee4444eaaaaaaa. Paper pdf
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A.,
Wu, J., and Amodei, D. Scaling laws for neural language models, 2020.
Khrushchev, M., Vasilev, R., Zinov, N., Petrov, A., and Yandex. Yalm 100b, 2022. http:///ggaae..oooo/////////a/mm
"https
Kim, B., Kim, H., Lee, S., Lee, G., Kwak, D., Jeon, D. H., Park, S., Kim, S., Kim, S., Seo, D., Lee, H., Jeong. M.. Lee, S., Kim, M., Ko, S., Kim, S., Park, T., Kim. J., Kang, S., Ryu, N., Yoo, K. M., Chang, M., Suh, S., In, S., Park, J., Kim, K., Kim, H., Jeong, J., Yeo, Y. G., Ham, D., Park, D., Lee, M. Y., Kang. J., Kang, I., Ha, J., Park, W., and Sung, N. What changes can large-scale language models bring? intensive study on hyperclova: Billions-scale korean generative pretrained transformers. CoRR, abs/2109 09.00000, 2021. URL
atge//bbb/1009 04650
Koenig, S. and Likhachev, M. Fast replanning for navigation in unknown terrain. IEEE Transactions on Robotics, 21(3):354 363, 2005. doi:
Krizhevsky, A. One weird trick for parallelizing convolutional neural networks. CoRR, abs/1 404.59997 2014. URL http:://aaxw..org//aaaaa /ttp///bbb/14449999 5997
Krizhevsky, A., Sutskever, I., and Hinton, G. E. Imagenet classification with deep convolutional neural networks. In Pereira, F., Burges, C. J. C., Bottou, L., and Weinberger, K. Q. (eds.), Advances in Neural Information Processing Systems 25, pp. 1097-1105. Curran Associates, Inc., 2012.
Kuszmaul, J. Bamboo trimming revisited: Simple algorithms do well too. arXiv preprint arXiv:2201.07350, 2022.
Lepikhin, D., Lee, H., Xu, Y., Chen, D., Firat, O.,, Huang, Y., Krikun, M., Shazeer, N., and Chen, Z. Gshard: Scaling giant models with conditional computation and automatic sharding. ArXiv, abs/2006. 16668, 2020.
Lester, B., Al-Rfou, R., and Constant, N. The power of scale for pframeter-efficient prompt tuning. Proceedings of the 202/ Conference on Empirical Methods Natural Language Processing, pp. 3045-3059, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10.18653/1120211emmmp.. www.llatthoooy..or/groooo emnlp-main 243
URLhhttps
libp2p. libp2p circuit relay. https: http:///s...lioooooc/oc/eeeee/rr 2022.
12

---

Liu, H.. Tam, D., Mugeeth, M., Mohta, J., Huang, T., Bansal, M., and Raffel, C. Few-shot parameter- efficient fine-tuning is better and cheaper than in-context learning, 2022a. URL https://arxiv org/abs/22055 05638
Liu, H., Tam, D., Muqeeth, M., Mohta, J., Huang. T., Bansal, M., and Raffel, C. Few-shot parameter- efficient fine-tuning is better and cheaper than in-context learning, 2022b. URL https://arxiv org/ ab// 2205. 65638
Liu, X., Ji, K., Fu, Y., Du, Z., Yang, Z., and Tang, J. P-tuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks. arXiv preprint arXw::211000700,, 2021a.
Liu, X., Zheng, Y., Du, Z., Ding, M., Qian, Y., Yang, Z., and Tang, Gpt understands, too. arXiv:2103. 10385, 2021b.
Maymounkov, P. and Mazieres, D. Kademlia: A peer-to-peer information system based on the xor metric. In International Workshop on Peer-to-Peer Systems, pp. 53-65 65- Springer, 2002.
Narayanan, D., Harlap, A., Phanishayee, A., Seshadri, V., Devanur, N. R., Ganger, G. R., Gibbons, P. B., and Zaharia, M. Pipedream: Generalized pipeline parallelism for dnn training. In Proceedings of the 27th ACM Symposium on Operating Systems Principles. SOSP '19, pp. 1-15, New York, NY, USA, 2019. Association for Computing Machinery. ISBN 9781450368777 doi: 10.1145/ 3341301. 33359666... URL https://///o...oon org 11..111433444 114533413011 3359646
Narayanan, D., Shoeybi, M., Casper. J., LeGreslle. P., Patwary, M., Korthikanti, V., Vainbrand, D., Kashinkunti, P., Bernauer, J., Catanzaro, B., al. Efficient large-scale language model training gpu clusters. arXiv preprint arXiv:2104.04473, 2021.
NVIDIA. NVIDIA Ampere GA102 GPU architecture, 2020.
URL
https
//images, //images.nvidia.comm WWDD=-ppweeeA4222cooco.cccccceee
NVIDIA. Nvidia confidential confidential oompuinng http://www..vvddaa.cccomornrntta
collutions/comf.dettal---oom
2022.
Pudipeddi, B., MrMMmmakoossohhhhi, M., Xi, J., and Bharadwaj. S. Training large neural networks with constant memory using a new execution algorithm. arXiv preprint arXiv:2002.05645 2020.
Radford, A., Narasimhan, K., Salimans, T., and Sutskever, I. Improving language understand-
ing by generative pre-training. 2018. URL https: http:///cddoooonii..co
openai. com.r//eeeeeeeearccooesssss
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, Language models are
unsupervised multitask learners. 2019.
Rae, J. W., Borgeaud, S., Cai, T., Millican, K., Hoffmann, J., Song. H. F., Aslanides, J., Henderson, S., Ring, R., Young. S., and et al. Scaling language models: Methods, analysis & insights from training gopher. CoRR, abs/2 12. 1444,, 2021. htttps://arxww.or///bbbb.
11446
Rajbhandari, S., Rasley, J., Ruwase, O., and He, Y. Zero: Memory optimization towards training a trillion parameter models. In SC, 2020.
Rajbhandari, S., Ruwase, O., Rasley, J., Smith, S., and He, Y. Zero-infinity: Breaking the gpu memory wall for extreme scale deep learning. arXiv preprint arXiv: 104.:22044.087.., 2021.
Ren, J., Rajbhandari, S., Aminabadi. R. Y., Ruwase, O., Yang. S., Zhang, M., Li, D., and He, Zero-offload: Democratizing billion-scale model training, 2021.
Ryabinin, M., Dettmers. T., Diskin, M., and Borzunov, A. Swarm parallelism: Training large models can be surprisingly communication-effiieent. 2021.
Schick, T. and Schütze, H. Generating datasets with pretrained language models. pp. 6943-6951, November 2021. doi: 10. 18653/v1/2021 http://///222...el--mmem emwlp-main.55... URL htRp:ttps:://// htttp:://aclanthology org/
2021. emnlp-main..e 555
13

---

Shazeer, N.. Cheng, Y., Parmar, N., Tran. D., Vaswani, A., Koanantakool, P., Hawkins, P., Lee, H., Hong, M., Young. C., Sepassi, R., and Hechtman, B. A. Mesh-tensoorfow: Deep learning for supercomputers. CoRR, abs/181 11//1811.0000 1.00844, 2018. htllLtttpp//aaii.orgg/////
.02084)
Sung. Y.-L., Nair, V., and Raffel, C. Training neural networks with fixed sparse masks. Advances in Neural Information Processing Systems, 2021.
Tang, Z., Shi, S., Chu, X., Wang, W., and Li, B. Communication-efficaeee distributed deep learning: A comprehensive survey, 2020.
Taylor, R., Kardas, M., Cucurull, G., Scialom, T., Hartshomn, A., Saravia, E., Poulton, A., Kerkez, V., and Stojnic, R. Galactica: A large language model for science. 2022.
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. Ilama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13971, 2023a.
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine-tuned chat models. arXiv 2023b.
preprint ar arXiv:2307.09222
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L. u., and Polosukhin, I. Attention is all you need. In Guyon, I., Luxburg, U. V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., and Garnett, R. (eds.), Advances in Neural Information Processing nips. cc/
Systems 30, pp. 5998-6008. Curran Associates, Inc., 2017. URL faper//181-attentioon all all--o--lll need. pdf
Wang. J. Yuan, B., Rimanic, L.. He, Y., Dao, T., Chen, B., Re, C., and Zhang, C. Fine-tuning language models over slow networks using activation compression with guarantees, 2022. URL
wrg./bb.///........ https://aarxv.orrr/b/2222.. 22222
West, P., Bhagavatula, C., Hessel, J., Hwang, J. D., Jiang, L., Bras, R. L., Lu, X., Welleck, S., and Choi, Y. Symbolic knowledge distillation: from general language models to commonsense models. arXiv preprint wxxi:2110.001120 2021.
Yang. B., Zhang. J., Li, J., Ré, C., Aberger, C. R., and Sa, D. Pipemare: Asynchronous pipeline parallel dnn training. ArXiv, abs/1910.05124, 2019.
Yong, Z.-X. and Nikoulina, V. Adapting bigscience multilingual model to unseen languages, 2022.
htLphttps://art htapp::///arate wrw.org///aaoo abs/ 2atps//224.000 04873
Yuan, B., He, Y., Davis, J., Zhang, T., Dao, T., Chen, B., Liang, P. S., Re, C., and Zhang, C. Decentralized training foundation models in heterogeneous environments. Advances Neural Information Processing Systems, 35:25464 25477, 2022.
Zeng, A., Liu, X., Du, Z., Ding, M., Zheng, Q., Lai, H., Wang, Z., Yang. Z., Yu, J., Zhang, X., Zheng, W., Xia, X., Xu, Y., Tam, W. L., Dong, Y., Ma, Z., He, J., Sun, Z., Zhai, J., Chen, W., Zeng, G., Han, X., Zhao, W. Liu, Z.. Xue, Y., Wang, S., Shan, J., Jiang, H., Guo, Z., Zhang, P., and Tang, J. GLM-130B. An open bilingual pre-trained model, 2022. URL http: 7/keg. cs. tsinghua. www.gngua..euu...oo www. wn/.een///mmm// 13bb/potss/gmmm
Zeng, W., Ren, X., Su, T., Wang, H., Liao, Y., Wang, Z., Jiang, X., Yang, 7.,, Wang, K., Zhang,
X. Li, C., Gong, Z., Yao, Y.. Huang, X., Wang. J., Yu. J. Guo, Q., Yu, Y., Zhang, Y.. Wang, J. Tao, H., Yan, D., Yi. Z.. Peng, F., Jiang, F., Zhang, H., Deng, L., Zhang. Y., Lin, Z., Zhang, C. Zhang, S., Guo, M., Gu, S., Fan, G., Wang, Y., Jin, X., Liu, Q. and Tian, Y. Pangu-a: Large-scale autoregressive pretrained chinese language models with auto- parallla computation. CoRR, abs/2104. 12369, 2021. htRLhttts://// / //arxiv...viio org/ argg/bbs/1144 12369
Zhang, S., Roller, S., Goyal, N., Artetxe, M., Chen, M., Chen, S., Dewan, C., Diab, M., Li, X., Lin, X. V., Mihaylov, T.. Ott, M., Shleifer, S., Shuster, K., Simig, D., Koura, P. S., Sridhar, A., Wang, T.. and Zettlemoyer, L. OPT: open pre-trained transformer language models, 2022. URL http:://arx.voorr/bb//00.... 01068
14

---

Appendix
A Quality and efficiency of BLOOM with 8-bit quantization
As shown in Table this method has little effect on LLM quality for major benchmarks. In terms of inference time, Table demonstrates that quantization has about 5% of overhead with batch size (20 tokens), but becomes negligible for larger batches.
B Estimating theoretical best throughput with RAM offloading
In this estimate, we use the best possible hardware setup for offloading: CPU RAM offloading via PCIe 4.0 with 16 PCIe lanes per GPU. In 8-bit, the model uses GB of memory per billion parameters, and PCIe 4.0 with 16 lanes has a throughput of 256 Gbit/s. We assume offloading latency of zero in the upper bound estimation. As such, offloading 176B parameters takes at least:
176 GB 8
5.5 seconds
256 Gbit/s
This gives the upper bound of 1/5.5 = 0.18 tokens/s for the inference speed.
C Extension to beam search algorithms
There several variations of beam-search algorithm used for language model inference, including standard beam search, diverse beam search, constrained beam search, and more. A common thread between those algorithms is that they maintain fixed number of candidate sequences between steps. These sequences are informally referred to as the "beam". On every step, these algorithms generate possible continuations of sequences the previous beam, then use some fitness criterion to select & of these continuations for the next beam.
From computational point of view, this procedure is similar to simple "greedy" inference with batch of l: sequences. However, there is one important difference: unlike batched inference, beam "shuffle" candidate sequences hetween steps. In other words, 3rd best sequence from time step can produce 1st or 2nd (or any other) sequence on the next step. Furthermore, a single sequence on time step can produce multiple sequences selected for step 1.
search algorithms
Since different beam search variantions use different criteria for selecting top sequences, we need generic algorithm that can fit any criterion. In our system, we implement this by allowing clients to reorder server-side attention cache after each step. Formally, a client can send list at most k integers in range [1, kt), where i-th index specifies which previous attention cache should be used when generating i-th sequence of the next beam.
For instance, when given indices [2, 2, 1, 2). server will use 2nd best sequence from step to produce the new 1st, 3rd and 5th best sequences. Previous 1st and 3rd best sequences go to 3rd and 4th places, respectively. Finally, previous 4th and 5th sequences discarded. From technical point of view, servers implement this reordering by reordering attention cache with the specified indices (torch. gather operation) immediately before performing inference step.
Table 4: Zero-shot accuracy for BLOOM-1766 and OPT-175H with 8-bit and 16-bit weights.
Table 5: Generation throughput (tokens/s) for BLOOM-176B with 8-bit and 16-bit weights on 8x A100 GPUs.
Model
HellaSwag
LAMBADA
WinoGrande
Avg
Batch size
Weights
73.0
67.2
70.1
70.1
BLOOM
72.8
68.1
70.1
70.3
32
16
78.5
74.7
72.6
75.3
16-bit
4.18
31.3
100.6
OPT
78.5
74.6
71.7
74.9
8-hit
3.95
29.4
95.8
15

---

D Details of the server load balancing algorithms
Measuring throughput. Before joining for the first time, each server measures its Internet connec- tion throughput (in tokens/second, using one of public web APIs for doing that) and GPU throughput (in tokens/second, using small benchmark running several forward passes). The minimum of these values becomes the overall server throughput, which is then cached for future runs.
Initial block assignment. We assume that each server holds segment of consecutive transformer blocks to minimize inference latency. Clients may request to perform a forward or backward pass for the whole segment of blocks or its subsegment, if necessary. Normally, each server loads as many blocks as it can fit in its GPU memory, unless a user limits the number of blocks to utilize the rest of memory for something else.
Before starting, each server calculates the values of 4 - the total throughput of servers currently holding the i-th block or loading it (to start holding it in a few minutes). Then, to find the best segment of blocks to serve, the server looks for the most narrow bottleneck in the network. Formally, if the model has L blocks and the server can hold K of them in its GPU memory, calculate:
L-K+1 start arg min sorted 4411, .... (++ ==1
Here, arg min compares the sorted arrays lexicographically and chooses the leftmost start in case of multiple minimums.
This way, the next joining server would always cover a block with the smallest 1,. there are multiple bottlenecks like this, the server will try to cover as many of them as possible (we choose to cover the minimums first because the overall throughput the minimum throughputs among model blocks). Among the remaining options, choose segment covering many second minimums possible, and so on.
Quality of block assignment. While we are not aware of the exact polynomial-time solution for the problem of assigning the segments optimally, we have conducted computational experiments and found out that this greedy algorithm (rumning in polynomial time) usually finds assignment with total throughput of 90-100% of the optimal one (found by trying out all possible assignments in exponential time), given that the values of throughput are realistic to our setup.
Rebalancing. Since servers may leave at any time, each server also periodically checks if the current assignment is "good enough" compared the throughput estimated by running the greedy solution for servers currently present in the network.
Formally, each server periodically looks for segment of blocks that is more appropriate than the currently loaded blocks with respect to the arg min rule (T. If it finds one, it simulates how the rest of the servers would behave if replace the current blocks with the would change their blocks afterwards). If the eventual throughput is at least p% better, the server commits the change and announces that it changes the blocks, then other servers do the rest of the changes (eventually increasing the total throughput).
ones (how other servers
We usey 20% since it gives reasonable trade-off between the swarm throughput and the frequency of block replacements in our experiments (see Appendix [Fp. Specifically, lower value of p leads to block replacements happening too often, which negatively affects the inference latency since each block replacement resets attention caches for this block.
Stability of the greedy algorithm. The rebalancing algorithm does not cause oscillations since series of block replacements is executed only if it leads to eventually increasing throughput by at least p%. Once "good enough" throughput is achieved, servers do not change their blocks anymore (unless an essential number of servers join or leave). We verified this behavior computationally, simulating network with thousands of servers with different throughputs.
To conclude, this greedy heuristic allows servers to quickly close the gaps a substantial share (up to 100%) of servers holding certain blocks leave, but avoids excess block replacements otherwise.
16

---

Pipeline throughput
Throughput (lokenxis)
No load balancing
300
New servers only
Full, p 20%
200
Full, 10%
p
100
Upper baund
300
400
600
700
Number of block replacements per minute min average)
Replacements minule
100
75
0-
0
100
200
300
400
500
600
700
Time (minutes)
Figure 2: Behavior of the load balancing algorithms evaluated in Appendix
E
Evaluation of the server load balancing algorithms
In this section, we measure the effectiveness of the load balancing algorithm used in our system. We run all experiments using a fleet of 206 virtual instances that simulate participants. To keep experiment costs manageable, we do not use GPUs for this evaluation, instead simulating uneven server throughput programmatically. For each server, sample its throughput from the uniform distribution L ~ UO 100] tokens/second, then sample its memory size so can hold b U[1, 10] blocks (out of 70 blocks in total, in BLOOM-176B).
Each server follows certain availability schedule, i.c. turns on and shuts down the same predefined time across all experiments. We assign these schedules such that the number of active servers follows a sine wave, simulating daily activity cycles. The schedule approximately 100-110 active servers during peak activity and 15-25 servers at its lowest points. Note that each peak contains a different subset of 100-110 10 active servers out of 206 instances in total.
We evaluate the following approaches to load balancing:
1. No load balancing a bascline system where servers load a random contiguous interval of model blocks.
Balancing new servers only - simplified load balancing where servers choose the optimal blocks when joining the swarm (using the rule (1) from Appendix DD but never change them.
3. Full load balancing- the full algorithm, where every minute each server checks if they need to replace their blocks. We use the efficiency threshold (as described in Appendix D to avoid excess block replacements.
Upper bound - the best-case throughput estimate that reassigns contiguous block segments to servers optimally every minute.
We report their behavior in Figure 2 The full load balancing maintains connectivity throughout the experiment and achieves throughput close to the upper bound (staying within the 10-15% range most of the time). Higher thresholds p perform slightly worse during peak times but require only relatively infrequent block replacements, unlike the case with p = 1%. Note that using the assignment leading to the upper bound is not possible in practice since it requires each server to load different set of layers every minute, on top of solving the computationally expensive optimization problem.
Curiously, the baseline running load balancing for new servers only achieves reasonable throughput during periods where servers are actively joining. However, quickly loses throughput when random servers leave, since this creates "bottlenecks" in the pipeline that require rebalancing of existing peers. Finally, the naive baseline with random layer assignment has zero throughput most of the time because it is unable to form complete pipeline.
17

---

F Experiments with a wider range of failure rates
In this section, we follow the setup from Section and provide additional evaluations for wider range of failure rates (up to 5%) and sequence lengths (up to 2048 tokens). The results are shown in Figure [3] Unlike baselines, our algorithm provides reasonable performance in all tested conditions, especially for higher failure rates common for communicating over the Internet, using spot/preemptible instances or unreliable hardware).
P(faiture) 0.05
P(failure) 0.02
Cache-less inference
Cache-less inference
3.0
Caching with restarts
Caching with restarts
Ours
Ours
2.5
2.5
Average stepesee
2.0
2,0
1.5
1.5
1.0
1.0
0.5
0.0
128.256
1024
2048
128256
1024
2048
Soquence length
Sequenee length
P(failure) 0.01
P(failure) 0.001
Cache-less inference
12
Cache-less inference
Caching with restarts
Caching with restarts
Ours
10-
Ours
Average stepeisec
123
512
1024
2048
128256
1024
2048
Sequence length
Sequence length
P(failure) = 0.0001
P(failure) = 0.0
17.5
16
Cache-less inference
Caching with restarts
15.0
14
Ours
12
12.5
10
10.0
Cache-less inference
Caching with restarts
Average
8
7.5
Ouns
5.0
2.5
0.0
128:266
512
1024
2018
125256
512
1024
20418
Sequence length
Sequence length
Sequential inference speed (steps/s) for BLOOM (7.1B) with varying failure rates. The setup is the same as in Section 4.1 A failure rate p means that sending set of activations to the next pipeline stage fails with probability p. Zero speed means that the baseline did not finish within I hour.
Figure 3:
18

---

Table 6: Throughput (tokens/sec) of forward and backward passes for different tasks, batch sizes, prefix lengths.
Forward pass throughput
Backward pass throughput
Mode
Batch size
Prompt length
8
16
195.6
57.4
213.2
60.8
Prompt tuning
32
16
272.6
82.8
293.1
84.7
111.0
42.0
Prefix tuning
4
178.7
57.8
(i.e., "deep" prompt tuning)
32
164.1
64.4
32
4
255.8
84.8
G Performance of training-time forward and backward passes
In this section, evaluate throughput of training-timn time forward and backward passes and study factors that affect their performance. We will only consider BLOOM-I766 176B and the "3x A100, I Gbit/s" setup from Section 4.2 and focus on pneetuiing-speccinn hyperparameters, since the influence of network bandwidth and latency has already been discussed in the main paper.
Sequence classification. First, we consider fine-tuning the model on a binary classification task. We take BLOOM-1766 replace the logit layer with trainable classification head (similar to transformers. Sammers.Blomme wowoooorreuueneellassssse www.sssssiiiiiiinnnomoooooooooic and add trainable prompts before the input sequence, then train the model on batches of 128-token sequences. We try (a) both prompt tuning and prefix tuning (involving "deep" prompts), (b) two batch sizes (8 and 32), and (c) two prompt lengths (16 and 4). The client shares 8 CPU cores with of the servers and does not the GPU.
The results are provided in Table 6 The prefix tuning turns out to be slower, since adds several times more trainable parameters. Increasing prompt length and decreasing batch size also make training slower. Notably, we observe that moving client-side computations to GPU does not visibly improve performance, since the client does not perform any heavy operations in this setup
Language modeling. Next, consider fine-tuning the model causal language modeling task. We take BLOOM-176B, keep the logit layer, and add trainable prompts before the input sequence. We explore the same hyperparameters with sequence classification.
We observe that the throughput of the GPU-enabled client similar (within 10% difference) to the throughput in case of sequence classification, reported in Table Indeed, the client performs only small share of GPU computations in the forward and backward passes, and a particular model head and loss function do not have decisive influence the performance. However, performance of the CPU-only client turns out to be 5-10 times worse in this setup, since the client has to multiply the output embedding matrix to the hidden states of all tokens in the batch. This operation is too large to be efficiently computed on CPU 12
H Limitations and broader impact
Privacy. A key limitation of our approach is that servers hosting the first model blocks may their inputs to recover client data. Thus, users working with sensitive data should limit their clients only use trusted servers or, alternatively, For example, multiple research labs or small companies have access to a specific private dataset
up their own isolated network using our software.
"In case of sequence classification, the heaviest operation the client does is multiplying 2 x h and h x b matrices, where is the hidden dimension (14336 in BLOOM-176B) and b the batch size.
In case of language modeling, the client has to multiply x h and f x matrices, where d is the token vocabulary size (250880 in BLO0MM-776). This is z 10 times more FLOPS than used in case of sequence classification.
19

---

and want to process it with large language model, they may set up an isolated distributed network hosting this model get a better inference speed, compared to running the model independently.
In the future, this limitation may be addressed in future work using secure multi-party comput- ing (Evans et al. 2018) privacy-preservie hardware (NVIDIA 2022).
Motivating contributors. Since people using the client are not required to run a server, our system may experience imbalance between supply (peers who dedicate GPUs to serve model layers) and demand (peers using the servers to perform inference or fine-tuning for their own needs).
to serve model blocks would be to introduce system of incentives: peers running servers would carn reward points, which can be spent on high-priority inference and fine-tuning exchanged for other rewards. To implement this, we can run few validator peers that periodically traverse all available servers and issue reward points to their owners.
One way encourage
Security. We assume that servers in system are run by many independent parties. In practice, some them may turn out to be faulty and return incorrect outputs instead of the actual results of forward and backward passes. This may happen due to a malicious intent to influence other people's outputs or, when rewards are introduced (as described above), to earn a reward for serving layers without actually performing the calculations.
To address this issue, we can extend the validator peers, so that they periodically test servers with random requests of different types and ban them if they respond with incorrect outputs (possibly, revoking their rewards). The validator requests should be difficult to distinguish from requests of typical users, so that malicious servers cannot pretend to be honest to the validators but send wrong outputs to other peers. While this approach still leaves a chance of receiving wrong outputs, allows to eventually expose and penalize the faulty servers.
Finally, clients may reduce the probability getting faulty outputs by running their data through multiple disjoint chains of servers simultaneously and comparing the outputs against each other.
Broader impact. This work introduces a general-purper algorithm for decentralized inference and fine-tuning of large models. aiming to simplify access to the latest research in deep learning and provide an alternative way to efficiently run L.LMs without high-end hardware. We do not envision any direct negative impacts from our research, since models that can be hosted with our system are already widely available and may be used via APIs, offloading, or other means.
20


Reviews:
[{"review_id": "hanBIdcvfY", "reviewer": "Reviewer_DRjo", "summary": "This paper presents a system designed for decentralized inference and fine-tuning of large language models over distributed hardware, which allows users to efficiently run LLMs without requiring high-end hardware. The system leverages pipeline-based model parallelism, distributing model layers across nodes. Additionally, this work proposes fault-tolerant inference algorithms and load-balancing protocols to enable dynamic deployment of models. During the evaluation, experiments were conducted using the BLOOM-176B model, which demonstrated a 10x improvement in performance compared to the offloading method.", "questions": "The paper mentions that autoregressive LLM inference cannot be performed with a single pass through the model, leading to a higher system complexity. Do the proposed algorithms still have advantages for models that are not autoregressive? Was this the primary design point of the system, or is it also applicable to other models?\n\nDuring the evaluation of offloading, how many GPUs were used for benchmarking? Is there a strict basis for the best-case scenario?", "limitations": "(1)To better evaluate the system, it would be beneficial to extend the experiments to include more models.\n(2)Data privacy is a concern, as multiple clients may contribute to data misuse. It would be helpful to propose some solutions to address this issue.\n(3)Furthermore, the innovative aspects of the paper could be further clarified.\n", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The writing of this paper is fluent, with clear and logical reasoning, and the proposed solution is well-aligned with the requirements；\nThe design is consistent with current trends and can effectively address the problem, making it highly practical；\nThis paper presents a comprehensive system, with reasonable comparisons made against upper bounds and benchmarks.\n\n", "weaknesses": "The innovative aspects of the paper are not sufficiently elaborated upon. For example, I would like to know if there are any outstanding advantages compared to the latest research such as DeepSpeed, aside from differences in application scenarios. Has there been any comparison of similar metrics?\n\nThe results are somewhat unsatisfactory. Caching with restarts appears to be quite competitive, and the algorithm proposed in the paper performs better only under high failure rates. However, for short sequences, Cache-less inference performs better under high failure rate conditions (which may be due to some inherent communication issues in system). It would be helpful to compare several lengths or plot curves to more clearly show the trend and find the optimal point.\n\nAlthough the focus of the research is on the inference process, it would be best to systematically elaborate on the experimental results and methods related to fine-tuning for the sake of experimental completeness.\n\nA minor suggestion is to cite Algorithms 2/3 more specifically in the text to make the writing clearer.\n\nI am not particularly familiar with distributed learning, so please forgive me if I make any inaccuracies in my statements.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for reviewing the paper and leaving valuable feedback. We address the raised concerns below.\n\n**Weaknesses**\n\n> The innovative aspects of the paper are not sufficiently elaborated upon. For example, I would like to know if there are any outstanding advantages compared to the latest research such as DeepSpeed, aside from differences in application scenarios. Has there been any comparison of similar metrics?\n\nOur paper focuses on an algorithm for running LLMs using a swarm of globally distributed GPUs connected over the Internet, addressing challenges of unreliable network and hardware. Standard pipeline parallelism methods like DeepSpeed can't work in this setup since they are designed for high-speed networked GPU clusters. Thus, we offer **a novel, cheap way to efficiently run LLMs for people without high-end hardware** — they can collaborate with other researchers and join their GPUs over the Internet to host the full model. This was **not possible with previously existing methods** and software.\n\nWe do not expect our algorithm to beat existing methods on a local, reliable GPU cluster, since this is not a setup it was designed for. Still, **we provide comparison with local pipeline parallelism that uses DeepSpeed** in Table 2 (see \"Local PP (NVLink)\", details in L309-311) to demonstrate the overheads that the proposed geo-distributed setup has compared to a local GPU cluster (expensive hardware that is not available to everyone).\n\n> Caching with restarts appears to be quite competitive, and the algorithm proposed in the paper performs better only under high failure rates. [...] It would be helpful to compare several lengths or plot curves to more clearly show the trend and find the optimal point.\n\nThe paper proposes the general-purpose system that has to successfully operate on both short and long sequences for various failure rates. The baselines are indeed competitive at some operating points but are highly impractical at others.\n\nWe agree that the plots would be useful and provide them in the general response **PDF**. Unlike the baselines, our algorithm provides **reasonable performance for all tested conditions**, especially for higher failure rates (common for communicating over the Internet, using spot/preemptible instances or unreliable hardware). We will include the plots in the next paper revision.\n\n> it would be best to systematically elaborate on the experimental results and methods related to fine-tuning for the sake of experimental completeness.\n\nPlease see **fine-tuning experiments in Appendix F.1**. Unlike inference, fine-tuning processes the entire batch at one go and does not need to store past attention caches between successive client requests. Thus, in case of a failure, we discard the incomplete forward/backward pass and just repeat the previous forward/backward pass request. We will elaborate on this in the main text in the next revision.\n\n**Questions**\n\n> Do the proposed algorithms still have advantages for models that are not autoregressive?\n\nOur system is applicable to arbitrary models, not only autoregressive ones. In our opinion, using it may be beneficial when: **(1)** the model does not fit into most customer GPUs (i.e., you can't load it locally), **(2)** single-batch computation time is much faster when the entire model is present in the GPU memory, compared to copying model weights to GPU on demand (i.e., you can't use offloading efficiently).\n\nThese points are especially relevant for autoregressive LLMs due to their huge size, and the extra need to maintain past attention caches makes the problem even more pronounced and challenging.\n\n> During the evaluation of offloading, how many GPUs were used for benchmarking? Is there a strict basis for the best-case scenario?\n\nNote that the bottleneck for offloading-based LLM inference is the GPU bus throughput, not the GPU performance (or the number of GPUs). For inference, the **best-case speed estimate** for offloading considers the GPU bus throughput only (for best existing hardware) and **assumes infinite GPU performance — and still turns out to be 5-10x slower** than our system. We provide step-by-step calculations in Appendix B.\n\nThe offloading experiments use 1x A100 (see Table 2 in Section 4.2) or 1x 3090 (see Table 6, Appendix F.2) and fully confirm our theoretical analysis. They show that **(1)** the **real offloading performance is even smaller** than the estimated best-case performance due to compute costs and other overheads (0.0495 < 0.18 steps/sec) and **(2)** the offloading performance **does not depend much on the GPU model**, having similar performance for both A100 and 3090 (0.0495 vs 0.0427 tokens/sec).\n\n**Limitations**\n\n> it would be beneficial to extend the experiments to include more models\n\nWe agree and provide experiments with **Llama 2 (70B)** in the general response. We will include them in the next paper revision.\n\n> Data privacy is a concern, as multiple clients may contribute to data misuse. It would be helpful to propose some solutions to address this issue.\n\nWe acknowledge this limitation in the paper (L346) and discuss potential solutions, such as using privacy-preserving computation methods or setting up a private swarm between trusted parties, in **Appendix G.**"}, {"review_id": "IImbXOIPHm", "reviewer": "Reviewer_2M8A", "summary": "The objective of this study is to facilitate the operation of Large Language Models (LLMs) using commodity hardware over the internet. However, such hardware can often be characterized by high unreliability and latency issues in networks. To mitigate these challenges, the paper introduces a dual attention caches method that backs up intermediate results and supports device failure recovery. Additionally, the authors have developed a decentralized load-balancing algorithm to optimally assign transformer blocks to each server in a bid to maximize the system's overall throughput. The authors have provided a robust implementation of the proposed system and demonstrated impressive performance on the largest publicly available open-source LLM.", "questions": "* For the comparsion between distributed inference and local offloading in table 2, does the local offloading also use the qunatized version of BLOOM? A clarification on this will help understand where the improvement of distributed inference comes from. ", "limitations": "* Precise and explicit definitions of client and server are missing. \n* Missing references on offloading inference compute across multiple devices [1,2, 3]. Discussing these would provide a more comprehensive context and enhance the depth of analysis.\n\n[1] Kang, Yiping, et al. \"Neurosurgeon: Collaborative intelligence between the cloud and mobile edge.\" ACM SIGARCH Computer Architecture News 45.1 (2017): 615-629.\n\n[2] Matsubara, Yoshitomo, et al. \"Head network distillation: Splitting distilled deep neural networks for resource-constrained edge computing systems.\" IEEE Access 8 (2020): 212177-212193.\n\n[3] Dong, Xin, et al. \"Splitnets: Designing neural architectures for efficient distributed computing on head-mounted systems.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* Offloading parameters of LLM to remote devices instead of local local storage (e.g. SSD) make senses. Although the former could have higher bandwidth, the IO amount could be much higher than the later. Emprical results also support this analysis. \n* The analysis on inference chanllenges is interesting. For example, the communication cost, past token storage. It helps the community understand the chanllenges of serving a large language model with billions of parameters. \n* client/server caching, shorest path routing, and automatic load balancing improve robustness, effiiency and soundness of the system. ", "weaknesses": "* I found that Table 2 is a bit vague to follow. Please eborate the metrics steps/sec and tokens/sec per user. For example, the difference between step and tokens and between clients and users. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for reviewing the paper and leaving valuable feedback. We address the raised concerns below.\n\n> I found that Table 2 is a bit vague to follow. Please eborate the metrics steps/sec and tokens/sec per user. For example, the difference between step and tokens and between clients and users.\n\nTable 2 reports the speed of autoregressive inference and parallel forward passes that **each client gets** on average. We assume that each user runs one client, so there is no difference between \"clients\" and \"users\" in this context.\n\nFor inference, the speed is measured in generation steps per second each client can do (we use batch size 1, so each step generates 1 token), showing **generation latency**. For parallel forward, the speed is measured in tokens per second each client can process, showing the swarm's throughput during **batch processing and/or fine-tuning**.\n\nWe will update the table caption and titles to clarify this in the next revision.\n\n> For the comparsion between distributed inference and local offloading in table 2, does the local offloading also use the qunatized version of BLOOM?\n\nYes, we mention that in L267 after the paragraphs about quantization.\n\n> Precise and explicit definitions of client and server are missing.\n\nWe provide short definitions in L150-152 and will expand them to the more explicit definitions provided below in the next revision.\n\nA **client** is a node operated by the user, which runs inference or fine-tuning jobs through the swarm of servers. A client only holds input and output embeddings (< 3% of model weights for BLOOM-176B) and delegates running transformer blocks (the most expensive computations) to remote servers.\n\nA **server** is a GPU-enabled node holding a set of consecutive transformer blocks and processing requests coming from client nodes.\n\n> Missing references on offloading inference compute across multiple devices [1, 2, 3].\n\nThank you for making us aware of this work. We focused on optimizing distributed computation of common model architectures widely used in practice today, such as BLOOM/Falcon, OPT, and LLaMA. We think the work that you mentioned provides an interesting approach that highlights how new architectures can be used in distributed computing. We will add discussion of these papers to the related work section.\n"}, {"review_id": "0nCPXlgxrA", "reviewer": "Reviewer_zN15", "summary": "The paper discuss about an important application problem of distributed inference for large language models. Given the size and inference requirement for large language models, and the constraint of hardware resources, the authors put forward utilizing idle GPUs in network to sped up the inference, providing detailed algorithm implementation and real world experimentation. Overall the work is solid and the presentation is good, and the application has important real world use cases. I have a few minor comments below.", "questions": "see above comments", "limitations": "see above comments", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. real world problem of focus - the target problem to be resolved is important\n2. solid experimentation - across continents experiment to demonstrate the distributed accelerator system's performance is amazing\n3. detailed implementation - algorithm detail, result comparison, and analysis are solid", "weaknesses": "Overall I feel given the topic of distributed inference for LLM, the work is solid and clear. Some aspects to improve the work includes\n\n1. server utilization discussion - the paper lacks coverage regarding the resource utilization of different GPU servers, given the distributed and heterogeneous computing setting\n2. distributed accelerator infra requirement for LLMs - depending on different LLMs, there should be some basic requirement on the hardware. For example TPU is not covered, high-end CPUs are also not explored\n3. expansion to LLM training - this is not the weakness of work, but would be great if the direction can be explored\n4. consumer-grade network constraint - similar to 2, there are other real world constraints given the topic for the \"consumer-grade network\". To push the work beyond lab setting, network constraints like different firewalls, fault-tolerance mechanism considering failure rate beyond 1% (the upper bound of experimentation setting), etc", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for reviewing the paper and leaving valuable feedback. We address the raised concerns below.\n\n> server utilization discussion - the paper lacks coverage regarding the resource utilization of different GPU servers, given the distributed and heterogeneous computing setting\n\nResource utilization is indeed an important concern with two main aspects: memory and compute. The **memory utilization was above 90%** for all GPUs used in Section 4.2. Most of this memory is allocated for storing LLM parameters, the rest is used for activations and attention caches.\n\nIn turn, the compute utilization rate depends on many factors including network bandwidth, client activity, and the exact workload. To that end, we evaluate several different server configurations with varying GPU and bandwidth: **(1)** RTX 3060 with 100 Mbit/s bandwidth, **(2)** RTX 3090 with 500 Mbit/s bandwidth and **(3)** RTX 3090 with 100 Mbit/s bandwidth. We assign each server to run forward passes for batches of 128 tokens sent by multiple concurrent clients (up to saturation), using the same workload as in Appendix F.1. We measure volatile GPU utilization (see [1]) averaged over 100 consecutive samples after a 1 minute warmup. We observe an average of **91% utilization for RTX 3060 and 100 Mbit/s, 94% utilization for RTX 3090 and 500 Mbit/s, and 66% utilization with 100 Mbit/s bandwidth.**\n\n[1] https://developer.download.nvidia.com/compute/DCGM/docs/nvidia-smi-367.38.pdf\n\n> distributed accelerator infra requirement for LLMs - depending on different LLMs, there should be some basic requirement on the hardware. For example TPU is not covered, high-end CPUs are also not explored\n\n**Server requirements.** The only GPU requirement for a server node is to have enough memory for one pipeline stage (= one transformer block). This is not an issue for most GPUs — only 3 GiB are needed for the largest open LLMs available (BLOOM-176B, OPT-175B), given that we use 8-bit quantization.\n\nRunning on CPUs is possible but most CPUs are an order of magnitude slower. TPUs should work in principle, but are not supported by our software stack.\n\n**Client requirements.** The client node only computes input and output embeddings and does not require an accelerator, unless we perform fine-tuning with a computationally expensive loss function. The only requirement is to have enough memory for the embeddings (8 GiB are needed for BLOOM-176B embeddings in bfloat16).\n\n> expansion to LLM training\n\nOur system does support fine-tuning (see Section 3.4, experiments in Appendix F.1). As for pre-training from scratch, a similar setup was explored in other recent work [1, 2]. However, in contrast to our system, these methods are not designed to **(a)** run autoregressive inference with low latency and fault tolerance and **(b)** allow users to fine-tune the distributed model for multiple different tasks simultaneously.\n\n[1] Yuan, Binhang, et al. \"Decentralized training of foundation models in heterogeneous environments.\" Advances in Neural Information Processing Systems 35 (2022): 25464-25477.\n\n[2] Ryabinin, Max, et al. \"Swarm parallelism: Training large models can be surprisingly communication-efficient.\" arXiv preprint arXiv:2301.11913 (2023).\n\n> To push the work beyond lab setting, network constraints like different firewalls, fault-tolerance mechanism considering failure rate beyond 1% (the upper bound of experimentation setting), etc\n\nWe explore other challenges, such as NATs, firewalls, and using heterogeneous hardware, in the \"Real-world setup\" experiments (L318-321). In particular, we show that **the servers are able to traverse NATs and firewalls** using the Circuit Relay protocol from libp2p, by opening a long-living connection to another directly available peer and asking it to become a relay.\n\nWe also report plots showing the behavior of our algorithm **for failure rates 2% and 5%** in the general response PDF. We can see that it still has competitive performance in these cases. We will include these plots in the next paper revision."}, {"review_id": "VKRXliCO8U", "reviewer": "Reviewer_8b6R", "summary": "The paper proposed cost-efficient inference and fine-tuning methods for LLMs on geodistributed devices in a consumer-grade network. The motivation is that, by pooling together idle compute resources of multiple research groups and volunteers, we could make LLM research and applications accessible to broader communities. Technically. this paper comes up with an algorithm to address two challenges: 1) how to conduct the computing reliably if any device can disconnect abruptly; 2) how to to partition LLMs between devices with uneven hardware. According to their simulations and real-world experiments, the proposed method can outperforms other approaches to running inference on consumer-grade hardware.", "questions": "Please refer to the Weaknesses", "limitations": "The potential limitations include privacy of data processed by outside peers, as well as broader impact of making LLMs more accessible.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The motivation of this paper is realistic. As we all know that playing with LLM is costly regarding the computing resources. Using the idle resources to do LLM inference and fine-tuning is socially and environmentally good.\n- The idea of this paper is clear and practical.\n- The paper applied multiple optimizations from different dimensions regarding the training/inference of LLM under low-resource, e.g., quantization both weights and activations between pipeline stages, efficient fine-tuning and so on. Although each of these methods are not really new, it is still inspiring to put them altogether and show that they work well.\n- The experiments are reasonable, especially the real-world setup.\n", "weaknesses": "1. How to do fine-tuning under the proposed setting is not that clear, although the authors wrote one paragraph to explain the fine-tune part. Inference is clear and relatively simple, but how to recover from the failure during the training is not described and verified.\n2. The communication cost from the tensor parallel is kind of missing. It seems that authors assume each server/client is able to hold a pipeline stage. However, in a more realistic scenario, each stage is further divided into multiple parts and one server/client could only hold a part of a stage. Then the intensive communication of tensor parallel may dominate the inference speed, because usually the tensor parallel only happen within one server, instead of distributed devices.\n3. In Table 1, why the performance of Cache-less is irrelevant to the failure rate?\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for reviewing the paper and leaving valuable feedback. We address the raised concerns below.\n\n> How to do fine-tuning under the proposed setting is not that clear, although the authors wrote one paragraph to explain the fine-tune part. Inference is clear and relatively simple, but how to recover from the failure during the training is not described and verified.\n\nUnlike inference, fine-tuning forward and backward passes process the entire batch at one go and **do not need to store past attention caches** between successive client requests. Thus, in case of a failure, we discard the incomplete forward/backward pass and just repeat the previous forward/backward pass request. This algorithm behaves similarly to the cache-less inference baseline in Table 1. We will extend the paper to clarify that in the next update.\n\nWe also provide **fine-tuning experiments in Appendix F.1**, verifying our algorithm experimentally.\n\n> The communication cost from the tensor parallel is kind of missing. It seems that authors assume each server/client is able to hold a pipeline stage. However, in a more realistic scenario, each stage is further divided into multiple parts and one server/client could only hold a part of a stage. Then the intensive communication of tensor parallel may dominate the inference speed\n\nWe run our experiments without tensor parallelism, since one transformer block of the largest open LLMs available at the moment (such as BLOOM-176B and OPT-175B) fits into 3 GiB of memory with 8-bit quantization. Thus, most consumer GPUs compatible with deep learning software **can hold a pipeline stage** with at least 1-2 transformer blocks.\n\nWe do not use tensor parallelism over the Internet since its communication overhead is indeed too large. However, **our algorithm allows to use tensor parallelism across GPUs on the same machine** to speed up pipeline stages hosted on machines with multiple GPUs, since communication cost between machine's GPUs is much lower compared to the Internet. Our algorithm can treat such machines as a single \"virtual GPU\" with higher performance due to parallel computations. In this case, the system will take intra-host communication overheads into account while evaluating the machine's total compute throughput (later used for load balancing and fastest-inference routing).\n\n> In Table 1, why the performance of Cache-less is irrelevant to the failure rate?\n\nIn case of failure, the cache-less baseline only needs to retry the last generation step and does not need to recover anything. For failure rate $p$, the number of retries for each step follows the geometric distribution with success probability $1 - p$ with the expected value of $1 / (1 - p)$.\n\nThus, the entire process slows down by $1 / (1 - p)$ times, which turns out to be not significant for the setup from Table 1 with failure rates $p \\le 10^{-2}$. However, cache-less inference is highly inefficient for longer sequences, which is not appropriate for most LLM use cases.\n"}]

Author rebuttal:
We thank all reviewers for taking the time to study our paper and leave valuable feedback. We are glad that the reviewers appreciated the motivation behind our work (*8b6R, zN15*), our analysis of LLM inference challenges (*2M8A*), the soundness of the proposed system for geo-distributed inference (*2M8A, 8b6R*), and our experimental work (*zN15, 8b6R*).

**Experiments with Llama 2 (70B).** Following the request of reviewer *DRjo*, we provide experiments with Llama 2 (70B) below. We run our system on 3 machines with one T4 GPU (16 GB memory) each and compare its performance to offloading running on each of these machines independently. The model is quantized to the NF4 format [1] in both cases. All other setup details are the same as in Section 4.2 and Table 2.

| GPUs | Parallel clients | Bandwidth, RTT     | Single-batch inference |         | Parallel forward |                 |
|------|------------------|--------------------|------------------------|---------|------------------|-----------------|
|      |                  |                    | steps/s for each client |        | tokens/s for each client |         |
|      |                  |                    | **128 tokens** | **2048 tokens** | **Batch 1x128** | **Batch 64x128** |
|      | 1                | 1 Gbit/s, < 5 ms   | 2.29           | 2.02            | 45.4            | 155.1            |
|      | 1                | 100 Mbit/s, < 5 ms | 2.29           | 2.01            | 37.5            | 140.2            |
| 3x T4 | 1                | 100 Mbit/s, 100 ms | 1.57           | 1.44            | 23.7            | 128.7            |
|      | 3                | 1 Gbit/s, < 5 ms   | 2.02           | 1.74            | 21.2            | 124.2            |
|      | -                | Offloading         | 0.139          | 0.139           | 18              | 139.9            |

We can see that our system still **beats offloading by more than 10x for inference**, even when all clients run inference simultaneously. Our system is also faster at fine-tuning in case of smaller batches or good network bandwidth. Thus, **all conclusions from Section 4.2 hold** for this setup.

[1] Dettmers, Tim, et al. "QLoRA: Efficient finetuning of quantized LLMs." arXiv preprint arXiv:2305.14314 (2023).

**Experiments with more failure rates.** Following the feedback from reviewers *DRjo* and *zN15*, we attach a **PDF with plots** that report our system's performance for a wider range of failure rates (including > 1%) and sequence lengths. Unlike baselines, our algorithm provides **reasonable performance in all tested conditions**, especially for higher failure rates (common for communicating over the Internet, using spot/preemptible instances or unreliable hardware).

**Limitations.** Finally, reviewers *8b6R* and *DRjo* noted data privacy as a limitation of our work. We acknowledge it in the paper (L346) and discuss potential solutions, such as setting up a private swarm between trusted parties or using privacy-preserving computation methods, in **Appendix G**.

Meta-review: The reviewers and AC discussed this paper and read the authors’ response carefully. All the reviewers unanimously agreed to accept this paper. 

The authors are encouraged to address the concerns from the reviewers in the camera ready, such as:

1.	Adding communication costs

2.	Expansion to LLM training

3.	Improve the presentation of Table 2.
Decision: Accept (poster)