Distributed Inference and Fine-tuning of Large Language Models Over The Internet
Authors: Alexander Borzunov
Dmitry Baranchuk
Tim Dettmers
Max Ryabinin
Younes Belkada
Artem Chumachenko
Pavel Samygin
Colin Raffel
Venue: Submitted to ICLR 2023 (2023)
Keywords: volunteer computing
distributed deep learning
distributed inference
efficient inference
large language models
gpt-3

Abstract: Large language models (LLMs) are useful in many NLP tasks and become more capable with size, scaling to over 100 billion parameters. With the release of BLOOM-176B and OPT-175B, everyone can download pretrained models of this scale. Still, using a pre-trained 100B+ model requires high-end hardware, making it inaccessible to most researchers. Recent studies in memory-efficient training (e.g. offloading) could alleviate these costs, but they do not cover important use cases of LLMs, such as autoregressive inference. In this work, we investigate methods for cost-efficient inference of large language models, comparing local and distributed strategies. We observe that a large enough model (100B+) could run efficiently on geodistributed devices in a consumer-grade network, for example by connecting existing compute resources of multiple research groups or pooling under-utilized compute from multiple cloud regions. To run LLMs in this unconventional setting, we develop a fault-tolerant algorithm for inferencing language models. We propose Petals - a decentralized system for running LLMs - and show that it can run BLOOM-176B over the Internet over $10\times$ faster than offloading for sequential generation. We evaluate the performance of our system in both simulated conditions and an actual distributed system spanning two continents. The design of Petals allows participants to inference, and fine-tune, or inference fine-tuned models simultaneously without affecting each other's results.

Under review as a conference paper at ICLR 2023
DISTRIBUTED INFERENCE AND FINE- OF LARGE LANGUAGE MODELS OVER THE INTERNET
Anonymous authors Paper under double-blind review
ABSTRACT
Large language models (LLMs) useful in many NLP tasks and become capable with size, scaling to over 100 billion parameters. With the release of BLOOM-176B and OPT-175B, everyone can download pretrained models of this scale. Still, using a pre-trained 100B+ model requires high-end hardware, making it inaccessible to most researchers. Recent studies in memory-efficien training (e.g. offloading) could alleviate these costs, but they do not cover important use cases of LLMs,, such as autoregressive inference. In this work, we investigate methods for cost-efficient inference of large language models, comparing local and distributed strategies. We observe that a large enough model (100B+) could run efficiently on geodistributed devices in a consumer- grade network, for example by connecting existing compute resources multiple research groups or pooling under-utilized compute from multiple cloud regions. To run LLMs in this unconventional setting, we develop a fault-tolerant algorithm for inferencing language models. We propose PETALS a decentralized system for running LLMs- and show that it can BLOOM-176B over the Internet over 10x faster than offloading for sequential generation. We evaluate the performance of our system in both simulated conditions and an actual distributed system spanning two continents. The design of PETALS allows participants inference, and fine-tune, or inference fine-tuned models simultaneously without affecting each other's 's results.
INTRODUCTION
In recent years, the NLP community has found that pretrained language models greatly accelerated progress many rescarch problems through either fine-tuning (Radford et al.) (018) simple prompting (Brown et al. 2020). Furthermore, their quality tends to improve as we increase model scale (Radford et al. 2019 Kaplan al. 2020). Following this trend, modern language models often have hundreds of billions of parameters (Brown et al.) 2020) Rae et al.) 2021 Zeng et al. 2021 Kim
al. 2021). Most recently, several research groups open-sources their pretrained LLMS with over 100B parameters for everyone to use (Zhang et al. 2022 Khrushchev et al. 2022 Zeng et al.) (002)).
Even though these models are publicly available, they still difficult use due the sheer size in terms of parameters. For example, OPT-175B and BLOOM- 176B need over 350GB accelerator memory for inference and significantly more for fine-tuning. As result, even basic inference for large language models requires multiple high-end GPUs or multi-node clusters. Several recent studies propose algorithms for running large models with more affordable hardware (Pudipeddi al. 2020, Ren et al.) 2021), e.g. by offloading parameters to RAM. However, these techniques are inefficient in many practical LLM usage scenarios as we show Section 3.1
In this work, we search for a cost-effective way of running large pre-trained language models in their main use cases: inference, in-context learning, fine-tuning. We systematically analyze latency and throughput for training and inference tasks for these cases and determine which factors become dominant for very large models. Notably, for models with over 100B parameters, communicating activations over slow network can be faster than swapping layers from local RAM SSD. Based on these observations, it should be possible to run LLMs cost-effectively by pooling together commodity hardware over the Internet.
However, existing algorithms are not designed to run inference with unreliable devices or high-latency networks. To bridge this gap, formulate novel algorithm for fault-tolerant distributed inference

---

Under review as a conference paper at ICLR 2023
BLOOM layers, part 1/3
BLOOM layers, part 2/3
BLOOM layers, 3/3
Clients
French
Hinndi
$
Figure 1: An overview of PETALS system. Servers store pre-trained LLM layers and temporarily hold attention caches for inferencing. Clients hold embedding layers, well as learned prompts/adapters when inferencing fine-tuned models. Arrows denote temporary chains formed for inference.
of very large models. This algorithm takes care of inference-specific problems: keeping distributed attention caches between inference steps and recovering when some of remote devices fail leave abruptly. This algorithm allows for several cost-effective ways of using LLMs, such as combining under-utilized GPUs in multiple cloud regions, forming collaboration of multiple research groups and connecting their existing infrastructure to run large models together.
The contributions of this work be summarized as follows:
We analyze the performance model for training very large language models and identify bottlenecks that are not addressed by existing algorithms. Notably, both local offloading and existing distributed algorithms struggle with sequential generation in different ways.
We propose novel distributed algorithm that can inference large (100B+) language models on distributed unreliable devices. To the best of our knowledge, this is the first algorithm that can inference LLMs with 100B+ parameters over the Internet.
Using this algorithm, we design PETALS a decentralized system for inferencing and fine-tuning LLMs over the Internet. When inferencing BLOOM-I76B, PETALS outperforms offloading by roughly 10x for autoregressive generation. The implementation of our algorithms and PETALS system is available online
2
BACKGROUND. EFFICIENT TRAINING AND INFERENCE
There is wide variety of methods that can optimize training and inference for most deep learning workloads. this section, focus two lines of research that relevant for analysis: model parallelism and parameter offloading.
2.1 MODEL PARALLELISM
Model parallelism a family distributed training algorithms that assigns each device to hold subset of model parameters, run subset of computations and communicate output activations, There are two main types of model parallelism: "Traditional" model parallelism and pipeline parallelism. Traditional model parallelism (or tensor parallelism) assigns each device compute subset of each model layer (e.g., a subset of neurons), then communicate results between each other and proceed to the next layer (Krizhevsky al., 2012) Ben-Nun & Hoefler 2019 Tang al. (020). Each device performs symmetric computation, applied to a different slice of model weights, which makes traditional model parallelism compatible with MPI-based communication. In turn, the main performance overhead of this strategy comes from all-to-all communication (and synchronization) after each layer (Krizhevsky) 2014).
Pipeline parallelism reduces the communication overhead by assigning each device with one or several full layers (Huang et al.) 2019) Narayanan al. 2019 Yang al. 2019). During the forward pass, each stage applies its subset of layers to the inputs supplied by the previous stage, then sends the outputs of the last layer to the next stage. For the backward pass, this process is reversed, with each pipeline stage passing the gradients to the same device that previously supplied it with input
See gi tht.. wwt..coc.ccccc...2 htt.://m//cccr22220oooooooouu/ee anonymous/petall
2

---

Under review as a conference paper at ICLR 2023
activations. To better utilize the available devices, the pipeline must process multiple microbatches per step, allowing each stage to run in parallel on a different batch of inputs. Even with optimal execution, some of the pipeline stages will remain idle some of the time (Huang et al. 2019).
Both of these model parallel strategies are still actively used for training LLMs. However, world distributed training systems usually combine multiple forms of parallelism depending on hardware and network (Narayanan et al. 2021] Rajbhandari et al.) 2020 Jia et al.) 2019). In particular, "traditional" model parallelism is typically used within single multi-GPU server or closely interconnected TPU cores (Narayanan et al., [02 /Saazeer et al. 2018). In turn, pipeline parallelism is used to connect multiple Cervers(Narayanan et | 2021]. Several recent works demonstrate that pipeline parallelism can be used for cost-efficient pre-training of LLMs by pooling together cheap under-utilized GPU
devices (Athlur et al. 2022
Kuszmaul 2022
Wang et 2022).
2.2 OFFLOADING
Parameter offloading relegates model parameters from accelerator memory to slower but cheaper storage: typically RAM or SSD (Pudipeddi et al. 2020. Ren et al., 2021 Rajbhandari et al. 2021). When using the model, parameters are loaded to the accelerator just-in-time for computation, one or few layers at a time. In principle, this method allows running large models with single low-end accelerator long as there is enough RAM (or SSD) to store full model parameters.
The main drawback of this strategy is having to load and unload through all model parameters for each forward and backward pass, which can be time-consuming, This extra time can be amortized in workloads where model can do a lot of useful computations for each time a parameter is loaded. In practice, using offloading scenario of hardware and bandwidut would require 11 seconds per forward pass, or twice that for training. As we show in Section 4 real-world offloading performance is significantly worse.
single token through the OPT- 175B one GPU in the best-case
Pudipeddi et al. (2020) circumvents this by training with very large batches, and hence, increasing the computation. In turn, Ren et al. (202T): Rajbhandari al. (2021) reduce the overhead by overlapping communication and computation, that is, doing useful computation for the current layer while waiting for the transfer of the next layer to finish. However, unlike in model-parallel training, Ren al. (202I) still requires each device to compute the full model after gathering parameters. From the efficiency perspective. the main difference with model parallelism is that, during forward pass, model-paralee algorithms need to transfer layer activations between devices, whereas offloading algorithms transfer model parameters instead.
METHOD
Using pretrained large language models for NLP tasks consists of two main workloads: inference and fine-tuning. The inference workload typically consists of encoding an input text, then generating tokens autoregressively. In turn, fine-tuning requires updating either all of the model's parameters or (more commonly for large models) a small set of trainable weights (e.g., adapters or soft prompts) by backpropagation. These two workloads also cover more advanced use cases, such as:
Manually engineering prompts for given task, then deploying the model with these prompts.
Learning and inference with adapters (Hu et al. 2021 1 Houlsby et al. 2019 Liu et al.
2022b)
or "soft" prompts (Liu et al. 2021b] Lester et al. 2021 Liu et al. 2021a).
Distillation into smaller task task-specific model for faster inference (Schick & Schützz 2021
Counter-i we found that inference more challenging than fine-tuning for cost-efficient setups. To that end, we dedicate most of this section to snference-specific problems. As for fine-tuning, we describe a generic way to support arbitrary parameter-efficient fine-tuning in Section 3.3
3.1 PERFORMANCE BOTTLENECKS OF LLM INFERENCE
Unlike training, autoregressive LLM inference cannot be done with single pass through the model. Instead, the model needs to process one token at a time, pass it through the entire model, then generate
2 bit parameters, 350GB total, PCIe gen, 4 31.5G//s (16 lanes), infinite compute and memory bandwidth.
3

---

Under review as a conference paper at ICLR 2023
the next token and repeat the process. In case of model parallelism, training on sequence of tokens needs ((n) communication rounds, but generating the same sequence would require O(n . t) rounds, making it more susceptible to network latency. Similarly with parameter offloading, generating sequence of l tokens with an n-laye model would require loading every layer i times (O(n 1) too).
The other problem of autoregressive generation is dealing with attention for past tokens (aawwan et al.] 2017). During an inference step t, each layer needs to attend to - past keys & values from previously generated tokens. Existing inference algorithms store past entries in accelerator memory. Caching half-precision activations of 2048-token sequence for large models like GPT-3 (Brown iet al. 2020) or OPT-175B (Zhang et al.) 2022) (with 96 layers of 12288 units each) will take up 9.6GB GPU memory for each sequence. Offloading these cached values faces the same problems as offloading in general.
An alternative solution is to recompute all previous tokens on every inference step, storing only one set of keys & values time. Naturally, this approach needs quadratically more computation with sequence length i, for total of O(t²) time for transformer-based models since all public LLMs with 100B+ parameters use standard attention, which has a quadratic complexity in terms of sequence length. Surprisingly, this approach is often more efficient than offloaded caching, especially for shorter sequences due to the sheer inefficiency of the latter.
Parameter offloading can still be efficient when generating large amounts very short sequences. Each individual sequence still takes a long time to generate, but the system maintains high throughput by running many samples in parallel. Unfortunately, this is niche scenario that does not cover most LLM use cases. For instance, it is incompatible with in-context learning or prompt engineering, where the model needs process long sequences of training examples (Brown et 2020). More importantly, it does not support "interactive" applications where LLM needs to quickly respond to a user input. This rules out many LLM application S such as conversation systems, question answering. machine translation or input completion (e.g. "Smart Compose"). To the best of our knowledge, this a fundamental limitation of offloading that cannot be circumvented with incremental changes.
Hence, we explore a new solution based on 2022) uses model parallelism to inference LLMs in GPU clusters. However, their approach does not apply to our more affordable setups: cheap "preemptible" instances or connecting existing resources over the Internet. To operate these conditions, an inference algorithm needs to deal with node preemption, network errors, and high networks latency.
Concurrent work (Aminabadi et al.
3.2 DISTRIBUTED GENERATION WITH FAULT TOLERANCE
In this section, we formulate algorithm for inferencing LLMs in fleet of unreliable geographically distributed devices connected over the internet. Each device can act as a server, a client, or both. Each server holds one several consecutive transformer blocks, while clients hold token embeddings and run inference jobs. For simplicity, we assume that every model layer is hosted on several servers and examine this assumption in the next section. Following this notation, fault-tolerant algorithm should allow each client to complete their inference job with reproducible results even if any remote servers fail during inference.
As we discuss in Section 3.1 autoregressive generation requires many sequential communication rounds, making it sensitive to network latency. However, if every device stores its past attention cache. every communication round only transfers activations for single token, i.c. several kilobytes of data In our algorithm, we use this model directly minimize the inference time over all possible pipeline configurations. As we show later in Section 4.2 this allows efficient inference over a now-bandwidth Internet connection.
A more challenging problem is how to recover from node and network failures. If a remote server shuts down, any cached attention keys stored on that server will be lost with it. There are two naive solutions to this problem: restarting inference from scratch or recomputing past embeddings on every step. Restarting might be enough at small scale. However, running 100B+ models will involve
'In this paper, the term model layer (or block) refers to one transformer block that typically combines self-attention, foed-forward network, normalization layers, and residual connection (Vaswani et al. 2017 *See e.g. hhtpp:: / /ggptdmmo......mm/mm///////m/m/m com Foor GPT-3 and OP1-175B, one d2288-dima token embedding takes up exactly 24KB.
4

---

Under review as a conference paper at ICLR 2023
many unreliable devices, making it unlikely to generate long sequence without at least one failure. In turn recomputing past attention caches requires communicating past tokens on every communication round, resulting in On . t?) total data transferred, where is the number of pipeline layers and t is the sequence length. In other words, both these solutions struggle to generate long sequences.
We address this problem by maintaining two types of cache: server-side cache holds past attention keys and values for their layers, while client-side cache holds past inputs sent to a given pipeline stage If a server disconnects, client can find another server with that pipeline stage and use client-side cache to restore the server state.
The resulting procedure is described in Algorithm T For every pipeline stage, the client maintains heap (priority queue) of servers that hold this stage (and may hold additional stages). The servers in queue are ordered by the network latency, measured by ping. Crucially, the queues are maintained through the lifetime of a client. To begin generation, the client runs a beam-search- find a sequence of servers that results the least total inference time under our performance model. When running inference steps, a client keeps track of intermediate activations sent between pipeline stages, one of remote servers fails or leaves, the client retrieves the next best server (or multiple servers) from the heap and directs it to restore the attention state from client's cached activations.
procedure to
Algorithm 1 Generating sequence, client-side code
Algorithm 2 (pe_inference(servee))
Input: prefix_tokens embeddings, known_servers
Input: local layers, stream
local_layers,
1: g = list()
1: cache = dictionary()
2: cache dictionary()
2: for layer E local Laaalll_arrssaa do
3: streams dictionary()
3: cache[layer] make_empty(
4: chain find_best_chanee best_chain(knowwn_srvers
4: while not stream.closed() do
5: for server E charnn do
5: inputs = wtw.esttttaammeeieelllol
6: streams[server]] = rpc inferencc(server)
6: for layer E
do 7: past_ kv = ccachellaayrrr
7:
cache[server] = list()
8:
8: inputs, new kv forward
9: inputs = =ebeeddings(ppefiitoooee
9:
layer, inputs, past kv)
10:
while should continuegeenerateed_squuueeee do tail servers = copy(chain)
10:
Saghelllayrr_appeed(eeeee stream..end(inputs)
11:
11:
12:
while not cmpty(tail_serven servers) do
13:
server = tail_ pail_servers.pppleftg try:
Algorithm 3 replace replace_failed____
14:
Input: server, inputs, cache, streams, known_servers
15:
b Attempt normal inference outputs .appeviiitappeennnnpus))))
1: known_ cerorns_sveerseeee beevers.ban(server)
17:
cache[server)
2: Dossing_layer get layers(server)
inputs = outputs
3: candidates = seleeeebelleeeeeeeeeeeeeee
19:
catch FailedRPC:
4: known_servern missing missing_layers)
20:
P Replace the failed server (rraams.pop(server).
5: chain find find_best_____ www._eehhaincaanddateeeeeee
21:
6: replacements list()
22:
past_inputs = cache. (oochee.pp(server) new_servers= haplace_failedd [
7: while not empty(chain) do
23:
8: s chain.pop_leff.. left()
24:
server, paas_inpuut cache, streams, known __everss)
9: try:
25:
10: streams[s] rpc rec_rpppeeeeeeeeeeeeeeeeeeeeeeee j_iifeeffeee(s))
26:
chain.replace(server,, new new_servers) geil_seeverr.puss_eeedeeee.....
11: outputs = streams (sseeaamsssssseednnnnpr
27:
12: replacementss.ppenddg
28:
13:
cache[s] inputs
29:
logits www.omputellgiitouuttttoooooeeei embeddings) next_token = choose_ choose_next(loiiiss e.g. greedy
14:
pttpaggstssssssppptttt___. pop(get_layers(5)
30:
15: inputs = outputs
31:
generated_ sequencce.pppedd(ent token)
catch FailedRPC:
32:
inputs = embeddings(next (oken)
17:
known_servers.bon
33:
18:
candidates = select_by by_layer(
34: for server E chain do streams((erver))cooses
19:
candidates, missing layers)
35:
20: chain = find Lt___bett__hhe wat.eccccciiddttttt..
36: return
21:
return chain
Here, pipeline stage is of consecutive model layers hosted one server in pipeline parallelism).
5

---

Under review as a conference paper at ICLR 2023
When a remote server fails, the algorithm needs to send (tt) data (in one round) for each failed server and compute only the stages held by the failed server. This can be seen as an interpolation between naive and cached inference, depending on the server failure rate. If none of the servers fail, we recover O(n 1) communication, similar to Aminabadi et al. (002)). In turn, if all servers fail after one step, the algorithm effectively performs non-caching generation, which is the best option in that scenario.
In practice, it is possible to modify the algorithm to further reduce the number of network hops. In the basic formulation, all communication between pipeline stages is routed through the client, i.e. the client receives the outputs of every pipeline stage, caches and sends to the subsequent stage. In practice, it is more efficient to let pipeline stages communicate directly: once the server obtains output activations, it sends them to both client and the subsequent stage. This reduces the total step time since both messages are a few kilobytes in size an can be sent in parallel. To verify that both client and the next pipeline stage received the same set of activations, they can verify the checksums (i.e. hash values) of the received activations asynchronously, without blocking computation.
Algorithm| I can support greedy inference or any sampling variants (including Holtzman et al. (2020). However, it requires one more step to support search-based algorithms such as beam search: cache reordering. During a beam search step. a client can generate multiple continuations of the same input prefix by cloning its attention cache and dropping less likely hypotheses.
3.3 SYSTEM DESIGN
So far, we have focused on whether is possible to run LLM inference on geo-distributed unreliable devices. However, this does not mean that it is practical for researcher to do so. For example, before using Algorithm[] [ each server must be assigned to a pipeline stage - and then reassigned each time another server joins or leaves the network. Here, we describe the design and implementation details that make the system practical. Our description centers around PETALS- a decentralized system that allows geographically distributed devices to run 100B+ parameter language models.
We design PETALS to run persistently across multiple projects, using fleet of servers that can join or leave the system at any time. To operate in these conditions, a system should be able to automatically reassign layers as new servers join or leave. Furthermore, the system should support multiple clients using the model simultaneously for different tasks, e.g. when one client is inferencing the base model, another could be fine-tuning it to different task or inferencing a previously fine-tuned task. Last but not least, it should be easy to use for popular workloads, i.e. run clients and servers.
Server load balancing. As we state earlier, the algorithm relies on having multiple servers per pipeline stage. To that end, each server runs a load balancing procedure to select which layers needs to serve. Formally, servers maximize the total model throughput by choosing the blocks with the worst throughput and eliminating potential bottlenecks. Each server periodically announces its active blocks to a distributed hash table (Maymounkov & Mazieres, 2002). When a new server joins, uses this information to identify an interval of blocks that contains most blocks with the worst throughput. This interval is always contiguous, since splitting it would harm the inference latency. Once the server has selected its layers, it measures its own throughput (both network and compute) and announces it to the distributed hash table.
Since peers may leave or fail at any time, all nodes periodically check if launching a rebalancing procedure would significantly improve the overall throughput. If it is the case, they switch layers until the throughput becomes near-optimal. In particular, if all peers serving certain blocks suddenly leave the system, this procedure quickly redistributes the remaining resources to close the emerged gaps. We provide a detailed description of the load balancing algorithms in Appendix [ and validate their properties in experiments reported in Appendix
B
Parameter efficient fine-tuning. While LLMs achieve high quality on many problems with sim- ple prompt engineering (Brown al. 2020), they often need training to achieve the best results. Traditionally, this is done by fine-tume tinettunigg&e all model parameters on the downstream task. However, for extremely large models, this strategy becomes impractical due to hardware requirements. For example, fine-tuning BLOOM-I 176B with Adam would require almost 3 TB of GPU memory to store the model, gradients, and optimizer states.

---

Under review as a conference paper at ICLR 2023
Fortunately, parameter-efffcient fine-tuning methods have been developed that keep most of the pretrained model intact. Some of them choose a subset of existing parameters to update (Sung et al. 2021:
Guo et al.] 202I) while others augment the model with additional trainable weights (Hu et al. 2021 Houlsby et al. 2019 Liu et al., 2021b Lester et al. 2021 Liu et al. 2021a 2022b). Despite their lower memory requirements, parameter-effiient approaches are often competitive with full model fine-tuning (Hu et al. [2021; Liu et al. 2 Yong & Nikoulina (022) and even outperform it in low-data regimes (Liu al. 2022a). Another appealing property of these approaches for our use-case is that they allow rapidly switching a pretrained LLM between different adapters.
By focusing on carameter-efficien fine-tuning, we are able to simplify the design of PETALS by making clients hold all learned parameters (see Figure Servers can run backpropagation through their layers and return gradients with respect to activations, but they do not update the server-side parameters. Even when client communicates learned values (e.g. soft prompts) to server, the server treats these values same as input activations. Thus, a server can simultaneously run different fine- tuning tasks without them interfering with one another. This design choice also allows PETALS users to define custom adapters in simple PyTorch without the need for network engineering expertise. We discuss the design of the side API in more detail for inference and fine-tuning in Appendix A
Memory efficiency. Since our main intended use-case is running on inexpensive low-end devices, need to work around their capabilities. In terms of raw FLOPs, even consumer-grasd grade GPUs like GeForce RTX 3070 could run a complete inference step of BLOOM- in less than a second (NVIDIA 2020). However, the GPU memory can only hold small fraction of model layers: running naively would require 44 RTX 3070 GPUs and 44 communication rounds. To make this more efficient, we use quantization store more parameters per GPU, reducing the number of consecutive devices and communication rounds.
More specifically, we 8-bit mixed matrix decomposition for matrix multiplication to quantize the weights to 8-bit precision and reduce the memory footprint compared to 16-bit weights, as suggested in Dettmers al. (2022a). This decomposition separates hidden states and weights into two portions: about 0.1 % of 16-bit outlier and 99. 99.9% of 8-bit regular values, which roughly halves the memory footprint. We verify that compressing weights does not affect model quality in Appendix [
To send less data between subsequent pipeline stages, we use dynamic blockwise quantization (Dettmers et al., 2022b). We apply it to the hidden states before hipeline-paralle communication. Dynamic blockwise quantization halves the bandwidth requirements without any noticeable effect on generation quality. When fine-tuning, we also take advantage of gradient checkpointing (Griewank & Walther 2000; Chen al. 2016) and half precision reduce VRAM usage- both of which are standard practice for large language models (Narayanan et al., 2021 Brown et al. 2020 Athlur et al. 2022b. In experiments, we also apply these optimizations baseline systems for a fair comparison.
4 EXPERIMENTS
4.1 INFERENCE WITH UNRELIABLE SERVERS
First, we conduct small-scale preliminary experiments to test the fault-tolerant generation algorithm described in Section 3.2 For these experiments, we use smaller BLOOM model with 7.1 billion parameters (BigScience) 2022). This model consists of 30 transformer blocks with hidden size 4096. We compare our algorithm with baselines when generating a single sequence of length 512 autoregressively. For simplicity, we run all computations and communications in single precision and disregard word embeddings / logits for this set of experiments. We measure the time to certain number of tokens through all transformer layers. We simulate network failures by resetting pipeline stages a certain rate.
We compare three different inference strategies:
1. Standard inference where servers store attention caches. On failure, restarts the entire generation from scratch since the failed server's caches are lost.
2. Cache-less inference that reruns past tokens every step. On failure, restarts only the last generation step.
3. Fault-tolerant inference, as described in Algorithm 1
7

---

Under review as a conference paper at ICLR 2023
Table 1: Sequential inference speed (steps/second) of BLOOM-7BI with varying failure rates. failure rate p means that sending each set of activations the next stage of the pipeline fails with probability The missing value means that the algorithm did not finish in reasonable time.
Inference
128
tokens, failure rate:
1024 tokens, failure rate:
Algorithm
0
le-4
le-3
le-2
0
le-4
1e-3
le-2
Caching with restarts
17.1
16.7
12
0.18
15.5
11.8
0,48
No caching (recompute)
3.44
3.44
3.44
3.44
0.89
0.89
0.89
0.89
Algorithm|
11.4
11.4
10.6
3.38
10.7
10.7
7.76
2.17
All runs use four pipeline stages with (8, 7, 8. 7) model layers per pipeline stage. Each pipeline stage served by single GeForce 1080Ti GPU; the four GPUs are running in single system with dual Xeon Gold 6148 CPU. 12 DDR4 LRDIMM sticks with 64GB each. The system has 16 dedicated PCIe Gen.3 lanes per GPU in dual root configuration, without using PCIe switches. Each stage runs in an isolated docker containers with virtual network interfaces, but there is no limit to communication bandwidth for this experiment. We repeat all experiments 50 times and report the average time. The adjusted standard deviation never exceeds 0.2%. We use the pipeline parallelism implementation from Megatron-Deepppeece DeepSpeed (BigScience et al. 2022) for the stateless inference baseline.
We report all performance measurements in Table I Our algorithm outperforms both baselines for setups with high failure rate. Caching with restarts IS most efficient for inferencing with no failures. We also note that our algorithm is somewhat slower than this baseline when there no failures. We attribute this to a more technically advanced implementation of that baseline. Finally, non-caching inference can be competitive for short sequences (128 tokens), but slows down considerably on 1024 tokens, which agrees with out intuition from 3.1
4.2 EXPERIMENTS FOR BLOOM- 176B
Next, we evaluate PETALS on a more practical task of running BLOOM-I76H|| Transformer language model containing 70 layers with a hidden size of 14336. We evaluate three server configu- rations running a network with controlled bandwidth. Our first setup consists of 3 servers, each running on an A 100 80GB GPU. This is an optimistic scenario that requires the least amount of communication. In the second setup, we simulate 12 weaker devices by partitioning each A100-80GB into several virtual servers (3 large and small). In total, there are 9 virtual servers running blocks each, one server with 3 blocks and two more servers with 2 blocks.
We also evaluate parameter offloading, where each user runs independently on single GPU, swapping parameters from CPU memory. We report two offloading values: the real- throughput with DeepSpeed with the default recommended parameters, but enable pin_meennnnpp sin__emory, since we found that this settings runs 1.2-2x x faster in all setups. For theoretical best throughput, we calculate the maximum inference and forward training throughput to receive an upper bound offloading performance. See technical details in Appendix C Finally, we report performance of a server with 3 A10000 enough GPU memory to load the entire model (the upper bound PETALS could reach). This setup runs tensor maasalllrrllle code (TP=3) based on MegatronnDeepSpeer
(BigScience et al. 2022).
We evaluate PETALS with three network configurations: Gbit/s with < 5 ms latency, 100 Mbit/s with <5 ms latency and 100 Mbit/s with 100 ms latency [ The client-side nodes have 8 CPU cores and no GPU. We also test the effect of having multiple clients. For 12 servers with 100 Mbit/s bandwidth and 100 ms latency, if 8 clients run inference concurrently, each of them gets ~ 20% slowdown compared to the case when runs inference alone.
In Table 2 we report the performance of sequential inference and parallel forward passes. For inference, performance does not depend much on bandwidth or sequence length but degrades in high-latency settings, especially for 12 virtual servers. In turn, training-time time forward passes for large batches are affected by both bandwidth and latency. These results are shown in Table [ We can see that offloading is about an order of magnitude slower for inference compared to PETALS. For the training-time forward pass, offloading is competitive multiple GPUs are used and the networking for PETALS is limited 100 Mbit/s or has high latency. In other cases, PETALS offers higher throughput than offloading for training.
See tps: htt///uuggnngffccccce co co/bigsciience/bloom
We set network conditions with /ttt:s///giiiio...mm//gnnff/////
based tc qdisc
8

---

Under review as a conference paper at ICLR 2023
Table Throughput and latency for BL.0M-1176 generation and forward pass
Inference (steps/s)
Forward (tokens/s)
Network
Sequence length
Batch size
Bandwidth
Latency
128
2048
64
Offlooding Ix A100 (best-case theoretical estimates and actual)
128 Ghit/s
0.09
0.09
2.4
152.8
256 Gbit/s
0.18
0.18
2.7
170.3
Actual
0.0485
0.0495
2.5
152.4
PETALS on 3 physical servers, with one A100 each
Gbit/s
< 5 ms
1.22
1.11
70.0
253.6
100 Mbit/s
5 ms
1.19
1.08
56.4
182.0
100 Mbit/s
100 ms
0.89
0.8
19.7
112.2
PETALS on 12 virtual servers, simulated on 3x A100
Gbit/s
<5 5 ms
0.97
0.86
37.9
180.0
100 Mbit/s
<5 ms
0.97
0.86
25.6
66.6
100 Mbit/s
100 ms
0.44
0.41
5.8
44.3
Same, but with 8 clients running simultaneously
Gbit/s
< 5 ms
0.75
0.72
100 Mbit/s
5 ms
0.74
0.7
100 Mbit/s
100 ms
0.36
0.35
Entire model in GPU memory on a server with 3x A100 (upper bound)
1.35
1.23
90.6
286.6
PETALS on 14 real servers in Europe and North America
Real world
0.68
0.61
32.6
179.4
4.3 RUNNING BLOOM-176B OVER THE INTERNET
Finally, we benchmark BLOOM in real-world distributed setting with 14 smaller servers holding 2xRTX 3060, 2x2080Ti 30900, 2 A4000, and 4x GPUs. These are personal servers and servers from university labs, spread across Europe and North America and connected to the Internet at speeds of 100-1000 Mbit/s. Four the servers operate from behind firewall.
We evaluate this configuration for the same tasks in 4.2 and report results at the bottom of Table 2 This setup is marginally slower than A100 benchmarks, which expected due to slower hardware. However, even when communicating between different continents, PETALS maintains its efficiency, still outperforming offloading by a large margin.
5 CONCLUSION
In this paper, we introduced a novel fault-tolerant algorithm for inferencing large language models. On top of it, introduced PETALS decentralized system for running LLMs distributed unreliable devices connected over the Internet, which significantly outperforms other approaches to running inference on consumer-grade hardware. We demonstrated that the proposed system can scale to the largest publicly available langauge model with hundreds of billions of trainable parameters.
While work is focused on technical aspects, running large language models raises broad range of related questions. One important consideration is privacy: making sure that using PETALS does leak private data outside peers. Last but not least, need ensure that participants can benefit from this system equitably. i.e. in proportion to their contribution. We discuss future problems such as privacy, security, and incentive structures in Appendix F
the Internet
WWe use the Circuit Relay protocol from lihp2p to traverse NATs and firewalls, see hechttss:://ee pettps://docs..ee libp2p io/ btc.connepts/iirrutt-eel....

---

Under review as a conference paper at ICLR 2023
ETHICS STATEMENT
This work introduces general-purpose algorithm for decentralized inference of large models, aiming simplify access to the latest research in deep learning. Consequently, we do not envision any direct negative impacts from our research aside from granting the broader public an ability to interact with LLMs trained on uncurated web-crawled data. However, all models that can be served with PETALS are already in open access and thus can be exposed via APIs or other means: we do not release any artifacts beside the code.
One important limitation of our work in its current state is data privacy: the intermediate activations of the model for given inputs are sent to the servers without any encryption. As such, it might be possible for people hosting the servers to recover the user's input data. We discuss this limitation in more detail in Appendix [ and acknowledge that the development methods for sivacc-preseervng decentralized inference without performance penalties is open research problem.
REFERENCES
A121.
Jurassic-1 language models.
https://studio..e cot.:uuddoo.ii...cm///o./c..
jurassic1 Lang--gg-agggggeeles Accessed: 2022-06-22.
Reza Yazdani Aminabadi, Samyam Rajbhandari, Minjia Zhang, Ammar Ahmad Awan, Cheng Li, Du Li, Elton Zheng, Jeff Rasley, Shaden Smith, Olatunji Ruwase, et al. Deepspeed infer- ence: Enabling efficient inference of transformer models at unprecedented scale. arXiv preprint arXiv:2207.00032. 2022.
Sanjith Athlur, Nitika Saran, Muthian Sivathanu, Ramachandran Ramjee, and Nipun Kwatra. Varuna: scalable, low-cost training of massive deep learning models. In Proceedings of the Seventeenth European Conference on Computer Systems, pp. 472-442--4 177--87,11 2022.
Tal Ben-Nun and Torsten Hoefler. Demystifying parallel and distributed deep learning: An in- depth concurrency analysis. ACM Comput. Surv, 52(4), aug 2019. ISSN 0360-0300, doi: 10.1145/3320060. URL https://doi. /ttp:://dooooo////// 1145/3322060
BigScience. A version of BLOOM with 7.1 billion parameters. bigscience/blooc 7b1 2022.
https://huggingfacc.co
BigScience, Microsoft, and NVIDIA. The fork of Megatron-LM and Megatron-DeeppSpedd by 2022.
BigScience. https://gii thub.
Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George van den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, Diego de Las Casas, Aurelia Guy, Jacob Menick, Roman Ring, Tom Hennigan, Saffron Huang, Loren Maggiore, Chris Jones, Albin Cassirer, Andy Brock, Michela Paganini, Geoffrey Irving, Oriol Vinyals, Simon Osindero, Karen Simonyan, Jack W. Rae, Erich Elsen, and Laurent Sifre. Improving language models by retrieving from trillions tokens, 2021. URL
04426
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005.
2020.
Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin. Training deep nets with sublinear memory cost. arXiv preprint arXiv: 01211604006621 2016.
Tim Dettmers Mike Lewis, Younes Belkada, and Luke Zettlemoyer. LLM.int80) 8-bit matrix multiplication for transformers scale. ArXiv, abs/2208.073339 2022a.
Tim Dettmers, Mike Lewis, Sam Shleifer, and Luke Zettlemoyer. 8-bit optimizers via biock-wise quantization. International Conference Learning Representations (ICLR), 2022b.
Nan Du, Yanping Huang, Andrew M. Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, Barret Zoph, Liam Fedus, Maarten Bosma, Zongwei Zhou, Tao Wang, Yu Emma Wang, Kellie Webster, Marie Pellat, Kevin Robinson, Kathy
10

---

Under review as a conference paper at ICLR 2023
Meier-Hellsterm, Toju Duke, Lucas Dixon, Kun Zhang, Quoe V. Le, Yonghui Wu, Zhifeng Chen, and Claire Cui. Glam: Efficient scaling of language models with mixture-of-experts. CoRR, abs/2112.06905,, 2021. URL httphhhts///aatittt org
06905
David Evans, Vladimir Kolesnikov, Mike Rosulek, et al. A pragmatic introduction to secure multi- party computation. Foundations and Trends in Privacy and Security, 2(2-3):70-246, 2018.
Hugging Face and contributors. Accelerate: Run your raw pytorch training script on any kind of device. GitHub. Note: http:///hwwbbbbooo///ggggggg///s 2020.
William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter
models with simple and efficient sparsity, 2021.
Leo Gao, Jonathan Tow, Stella Biderman. Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, Jason Phang, Laria Reynolds, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, September 2021. URL https: httt:::::ddooorrgrgee.
5281 2221//eoodd..... 5371628
Sebastian Gehrmann, Abhik Bhattacharjee, Abinaya Mahendiran, Alex Wang. Alexandros Papangelis, Aman Madaan, Angelima McMillan-Majjol Anna Shvets, Ashish Upadhyay, Bingsheng Yao, Bryan Wilie, Chandra Bhagavatula, Chaobin You, Craig Thomson, Cristina Garbacea, Dakuo Wang, Daniel Deutsch, Deyi Xiong. Di Jin, Dimitra Gkatzia, Dragomir Radev, Elizabeth Clark, Esin Durmus, Faisal Ladhak, Filipp Ginter, Genta Indra Winata, Hendrik Strobelt, Hiroaki Hayashi, Jekaterina Novikova, Jenna Kanerva, Jenny Chim, Jiawei Zhou, Jordan Clive, Joshua Maynez, João Sedoc, Juraj Juraska, Kaustubh Dhole, Khyathi Raghavi Chandu, Laura Perez-Beltrachin! Leonardo F. R. Ribeiro, Lewis Tunstall, Li Zhang, Mahima Pushkarna, Mathias Creutz, Michael White, Mihir Sanjay Kale, Moussa Kamal Eddine, Nico Daheim, Nishant Subramani, Ondrej Dusek, Paul Pu Liang, Pawan Sasanka Ammanamanchi, Qi Zhu, Ratish Puduppully, Reno Kriz, Rifat Shahriyar, Ronald Cardenas, Saad Mahamood, Salomey Osei, Samuel Cahyawijaya, Sanja Stajner, Sebastien Montella, Shailza, Shailza Jolly, Simon Mille, Tahmid Hasan, Tianhao Shen, Tosin Adewumi, Vikas Raunak, Vipul Raheja, Vitaly Nikolaev, Vivian Tsai, Yacine Jernite. Ying Xu, Yisi Sang, Yixin Liu, and Yufang Hou. Gemv2: Multilingual nig benchmarking in a single line of code, 2022.
Andreas Griewank and Andrea Walther. Algorithm 799: revolve: an implementation of check- pointing for the reverse adjoint mode of computational differentiation. ACM Transactions Mathematical Software (TOMS),
2000.
Demi Guo, Alexander M Rush, and Yoon Kim. Parameter-efficient transfer learning with diff pruming. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics, 2021.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. In International Conference on Learning Representations, 2020. URL https //openreview. www..nttooormmiddrryooeeo
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. In International Conference on Machine Learning, pp. 2790-2799, PMLR, 2019.
Edward Hu, Yelong Shen, Phil Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models, 2021.
Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Chen, HyoukJoong Lee, Jiquan Ngiam, Quoe V Le, Yonghui Wu, et al. Gpipe: Efficient training of giant neural networks using pipeline parallelism. In Advances in Neural Information Prucessing Systems, pp. 12,-112,20 2019.
Zhihao Jia, Matei Zaharia, and Alex Aiken. Beyond data and model parallelism for deep neural networks. In A. Talwalkar, V. Smith, and M. Zaharia (eds.), Proceedings of Machine Learning and Systems, volume 1, pp. 1-13, 2019. URL
mlsys. org/ Ttgg/aper/20119

---

Under review as a conference paper at ICLR 2023
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, 2020.
Michael Khrushchev, Ruslan Vasilev, Nikolay Zinov, Alexey Petrov, and Yandex. Yalm 100b, 2022.
"https://huggiingfaace yoapeeeee/yaaaaaaaaaeeaeee waaaandee/lllm10000000 100b
Douwe Kiela, Max Bartolo, Yixin Nie, Divyansh Kaushik, Atticus Geiger, Zhengxuan Wu, Bertie Vidgen, Grusha Prasad, Amanpreet Singh, Pratik Ringshia, Zhiyi Ma, Tristan Thrush, Sebastian Riedel, Zeerak Waseem, Pontus Stenetorp. Robin Jia, Mohit Bansal, Christopher Potts, and Adina Williams. Dynabench: Rethinking benchmarking in nlp, 2021. httth: www.:orgggbs.//./. 2104. 14337
Boseop Kim, HyoungSeok Kim, Sang-Wor Woo Lee, Gichang Lee, Dong-Hyun Kwak, Dong Hyeon Jeon, Sunghyun Park, Sungju Kim, Seonhoon Kim, Dongpil Seo, Heungsub Lee, Minyoung Jeong, Sungjae Lee, Minsub Kim, SukHyun Ko, Seokhun Kim, Taeyong Park, Jinuk Kim, Soyoung Kang. Na-Hyeon Ryu, Kang Min Yoo, Minsuk Chang, Soobin Suh, Sookyo In, Jinseong Park, Kyungduk Kim, Hiun Kim, Jisu Jeong, Yong Goo Yeo, Donghoon Ham, Dongju Park, Min Young Lee, Jaewook Kang, Inho Kang, Jung- Woo Ha, Woo-Myoung Park, and Nako Sung. What changes can large-scale language models bring? intensive study on hyperclova: Billions-scale korean generative pretrained transformers. CoRR, abs/2109.044) 04650, 2021. URL httpp::/aaxii.orr/bbb/// 04650
Alex Krizhevsky. One weird trick for parallelizing convolutional neural networks. CoRR, abs/1404.5997, 2014. URL http://////a http://rxxx....g/bbbbb orggobs// 400./4444.5555 5997)
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In F. Pereira, C. J. C. Burges, L. Bottou, and K. Q. Weinberger (eds.), Advances in Neural Information Processing Systems 25, pp. 1097-1105. Curran Associates, Inc., 2012. URL http: haips.cc/papertt 4824 caapggnnt--lassificcat weeee-eeeeeeee-----eeeeeeeeeeeee con.oollttoooal--uuuaal--eeettee
John Kuszmaul. Bamboo trimming revisited: Simple algorithms can do well too. arXiv preprint
arXiv:2201.07330, 2022.
Dmitry Lepikhin, H. Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Y. Huang, M. Krikun, Noam Shazeer, and Z. Chen. Gshard: Scaling giant models with conditional computation and automatic sharding, ArXiv, abs/2006. 16668, 2020.
Brian Lester, Rami AI-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. In Proceedings of the 2021 Conference Empirical Methods in Natural Language Processing, pp. 3045-3059 Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 22.18653/1120011emmlmm.. 243. http/ttps www.e--maaill. main-mai.... 243
baaclanthology.org//
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval-augmeeteed generation for knowledge-intennt inneggge-nneniivv nlp tasks. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 9459 9474, Curran Associates, Inc., 2020. URL htttsttts::// http:////rorcceennnnneeeeeeeeeee Paper .pdf
cc/paper 2ttp
Xiang Lisa Li and Percy Liang. Prefix-tuning Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume I: Long Papers). pp. 4582 4597, Online, August 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021 lol-.o.g URL https: h/aps///oocannhooogo.y.. .ww.llooy.oooo..ooooooooo 2021 /ra.://..........oooog 353
Haokun Liu, Derek Tam, Mohammed Mugeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin Raffel. Few-shot parameter-efficaan fine- fine-ttnnin tuning is better and cheaper than in-context learning, 2022a. httphhtts:///rxttt http://///aaaa..ora////////5//
05638
12

---

Under review as a conference paper at ICLR 2023
Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin fine-tumi tuning is better and cheaper than in-context learning, 2022b. URL https://arxiv..rrrt abs/ 2205. 05638
Raffel. Few-shot
Xiao Liu, Kaixuan Ji, Yicheng Fu, Zhengxiao Du, Zhilin Yang, and Jie Tang. P-tuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks. arXiv preprint arXiv:2110.07602, 2021a.
Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. Gpt understands, too. arXiv:2103.10385, 2021b.
Petar Maymounkov and David Mazieres. Kademlia: A peer-to-peer information system based on the xor metric. In International Workshop on Peer-to-Peer Systems, pp. 53-65. Springer, 2002.
Deepak Narayanan, Aaron Harlap, Amar Phanishayee, Vivek Seshadri, Nikhil R. Devanur, Gregory Ganger, Phillip B. Gibbons, and Matei Zaharia. Pipedream: Generalized pipeline parallelism for dnn training. In Proceedings of the 27th ACM Symposium on Operating Systems Principles, SOSP 19, pp. 1-15, New York, NY, USA, 2019. Association for Computing Machinery. ISBN 978 1781443368755. doi: 10.1 1411144//3413313335566 URL 3359646
org 1145/333111
Deepak Narayanan, Mohammad Shoeybi, Jared Casper, Patrick LeGresley, Mostofa Patwary, Vijay Korthikanti, Dmitri Vainbrand, Prethvi Kashinkunti, Julie Bernauer, Bryan Catanzaro, et al. Efficient large-scale language model training on gpu elusters. arXiv preprint arXiv:2104.04472 2021.
NVIDIA. Nvidia ampere gal02 gpu architecture, 2020.
URL
https
simmages.nvidinn www./////..m/////a/aaaaoo http:////mmmmeeesssssuooto//rrre pttp:///ffrrceemme////////////
NVIDIA- httpD//eeerreeae/ooo/ooooeeeeeee
NVIDIA. Nvidia confidential computing. https//www.wwddda.cc//mmm//mtttt 2022.
Jonas Pfeiffer, Andreas Rücklé, Clifton Poth, Aishwarya Kamath, Ivan Vulić, Sebastian Ruder, Kyunghyun Cho, and Iryna Gurevych. Adapterhub: A framework for adapting transformers. arXiv preprint arXiv:2007.077 077799, 2020.
Bharadwaj Pudipeddi, Maral Mesmakhosroshahi, Jinwen Xi, and Sujeeth Bharadwaj. Training large neural networks with constant memory using new execution algorithm. arXiv preprint arXiv:2002.05645. 2020.
PyTorch Hub. PyTorch Hub. http:s::/ppttrc..orr//////
Accessed: 2021-10-04.
Alec Radford, Karthik Narasimhan, Tim Salimans, and llya Sutskever. Improving language under- standing by generative pre-training. 2018. URL htttt:://cch openai com//reseeacchhcovrrss Baaguugge-ussueerviii////ggggge
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and llya Sutskever. Language models are unsupervised multitask learners. 2019.
Jack W. Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, H. Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, Eliza Rutherford, Tom Hennigan, Jacob Menick, Albin Cassirer, Richard Powell, George van den Driessche, Lisa Anne Hendricks, Maribeth Rauh. Po-Sen Huang. Amelia Glaese, Johannes Welbl, Sumanth Dathathri, Saffron Huang, Jonathan Uesato, John Mellor, Irina Higgins, Antonia Creswell, Nat McAleese, Amy Wu, Erich Elsen, Siddhant M. Jayakumar, Elena Buchatskaya, David Budden, Esme Sutherland, Karen Simonyan, Michela Paganini, Laurent Sifre, Lena Martens, Xiang Lorraine Li, Adhiguna Kuncoro, Aida Nematzadeh, Elena Gribovskaya, Domenic Donato, Angeliki Lazaridou, Arthur Mensch, Jean- Baptiste Lespiau, Maria Tsimpoukelli, Nikolai Grigorev, Doug Fritz, Thibault Sottiaux, Mantas Pajarskas, Toby Pohlen, Zhitao Gong, Daniel Toyama, Cyprien de Masson Autume, Yujia Li, Tayfun Terzi, Vladimir Mikulik, Igor Babuschkin, Aidan Clark, Diego de Las Casas, Aurelia Guy, Chris Jones, James Bradbury, Matthew Johnson, Blake A. Hechtman, Laura Weidinger, Iason Gabriel, William S. Isaac, Edward Lockhart, Simon Osindero, Laura Rimell, Chris Dyer, Oriol
13

---

Under review as a conference paper at ICLR 2023
Vinyals, Kareem Ayoub, Jeff Stanway, Lorrayne Bennett, Demis Hassabis, Koray Kavukcuoglu, and Geoffrey Irving. Scaling language models: Methods, analysis & insights from training gopher.
CoRR, abs/2112.1114pp 2021. /ttps://s//// www.gr//ga//////// 11446
Colin Raffel
A call to
build URL
models
like
we
build
open-
source software, soslll-to-bulddmmodd
2021. wiik--e--blle seeel-ppnn---ssee source sof tware. html
Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimization towards training a trillion parameter models. SC, 2020.
Samyam Rajbhandari, Olatunji Ruwase, Jeff Rasley, Shaden Smith, and Yuxiong He. Zero-infinity: Breaking the gpu memory wall for extreme scale deep learning. Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC '21, New York, NY, USA, 2021, Association for Computing Machinery. ISBN 978 9781450384 doi: 10.1145/3558117334762222
WRL/http:://dol org 0145/4458177 3476205
Jie Ren, Samyam Rajbhandari, Reza Yazdani Aminahadi, Olatunji Ruwase, Shuangyan Yang. Minjia Zhang, Dong Li, and Yuxiong He. Zero-offload: Democratizing billion-scale model training, 2021.
Timo Schick and Hinrich Schütze. Generating datasets with pretrained language models. pp. 6943- 6951, November 2021. doi: 10. 10.18653/v1/202 1666:///222...mmmmmmii...5555 URL htt.httts///caaathholloe
org/2021. maa.lp-mannn.mmmm 555.
Noam Shazeer, Youlong Cheng, Niki Parmar, Dustin Tran, Ashish Vaswani, Penporn Koanantakool, Peter Hawkins, HyoukJoong Lee, Mingsheng Hong. Cliff Young, Ryan Sepassi, and Blake A. Hechtman. Mesh-tensorflow: Deep learning for supercomputers. CoRR, abs/1811.0200s 2018.
URL http:://axxv..rrgeeee// wwt.:rrrg/bb///1111/ 02084
Yi-Lin Sung, Varun Nair, and Colin Raffel Training neural networks with fixed sparse masks. Advances in Neural Information Processing Systems, 2021.
Zhenheng Tang, Shaohuai Shi, Xiaowen Chu, Wei Wang, and Bo Li. Communication-eficiienn distributed deep learning: A comprehensive survey, 2020.
TensorFlow Hub. TensorFlow Hub. https::/ww..eenssrrfoow.. org.hubb..gghbbbbb Accessed: 2021-10-04.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. Attention is all you need. I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems 30, pp. //papers. poppers.nips.oo wwtpcccppppppecceeee///p/e httpapppeeeeeeiattteettttooooooa is- all-you-need pdf]
Curran Associates, Inc., 2017. URL http:
Jue Wang, Binhang Yuan, Luka Rimanic, Yongjun He, Tri Dao, Beidi Chen, Christopher Re, and Ce Zhang. Fine-tuning language models slow networks using activation compression with guarantees, 2022. URL htttp:://aaxiii wwwp:xwwwwwwgg.////ebeee 01299
Peter West, Chandra Bhagavatula, Jack Hessel, Jena D Hwang, Liwei Jiang, Ronan Le Bras, Ximing Lu, Sean Welleck, and Yejin Choi. Symbolic knowledge distillation: from general language models 2021.
to commonsense models. arXiv preprint
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State- -hee----tthh-rr natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 38 38-45, Online, October 2020. Association for Computational Linguistics. URI. https: //ww//
www. ac web. org wwwg/anntologyy222222 emnlp-deman demos
Bowen Yang, Jian Zhang, Jonathan Li, Christopher Ré, Christopher R. Aberger, and Christopher De Sa. Pipemare: Asynchronous pipeline parallel dnn training. ArXiv, abs/1910.05124, 2019.
14

---

Under review as a conference paper at ICLR 2023
Zheng-Xin Xin Yong and Vassilina Nikoulina. Adapting bigscience multilingual model to unseen languages, 2022. URL http::://aaxiv.orr//aaaaee httpg//bbs222.....m 04873
Aohan Zeng, Xiao Liu, Zhengxiao Du, Ming Ding, Qinkai Zheng, Hanyu Lai, Zihan Wang, Zhuoyi Yang. Jifan Yu, Xiaohan Zhang, Wendi Zheng, Xiao Xia, Yifan Xu, Weng Lam Tam, Yuxiao Dong, Zixuan Ma, Jiaao He, Zhenbo Sun, Jidong Zhai, Wenguang Chen, Guoyang Zeng, Xu Han, Weilin Zhao, Zhiyuan Liu, Yufei Xue, Shan Wang. Jiecai Shan, Haohan Jiang. Zhengang Guo, Peng Zhang, and Jie Tang. BLM-130B: An open bilingual pre-trained model, 2022. URL http://keg.cs cs. tsinghua. www.ghhaaedu.....mm edu. wn/ waannglmm-11m
1mmm1300//
Wei Zeng, Xiaozhe Ren, Teng Su, Hui Wang. Yi Liao, Zhiwei Wang. Xin Jiang, ZhenZhang Yang, Kaisheng Wang, Xiaoda Zhang, Chen Li, Ziyan Gong, Yifan Yao, Xinjing Huang, Jun Wang, Jianfeng Yu, Qi Guo, Yue Yu, Yan Zhang, Jin Wang. Hengtao Tao, Dasen Yan, Zexuan Yi, Fang Peng, Fangqing Jiang, Han Zhang, Lingfeng Deng, Yehong Zhang, Zhe Lin, Chao Zhang, Shaojie Zhang, Mingyue Guo, Shanzhi Gu, Gaojun Fan, Yaowei Wang, Xuefeng Jin, Qun Liu, and Yonghong Tian. Pangu-a: Large-scale autoregressive pretrained chinese language models with auto-parallel computation. CoRR, abs/2104.12365 2021. URL 12369
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zeutlemoyer. Opt: Open pre-trained transformer language models, 2022. URL https: //axxiv org/abs/2205 01068
15

---

Under review as a conference paper at ICLR 2023
A APPENDIX
A DESIGN AND USE CASES
Practical usage of large language models can be broadly divided into two main scenarios: inference and parameter-efficient adaptation to downstream tasks. In this section, we outline the design of PETALS, showing how it handles both scenarios and also allows easily sharing trained adapters between the users of the system.
A.1 INFERENCE OF BILLION-SCALE MODELS
When generating tokens, a client stores the model's token embeddings (which typically comprise small fraction of the total parameter count and can fit in RAM in most modern laptops, servers, and workstations) locally and relies on servers to run Transformer blocks. Each server holds several consecutive blocks, the number of which depends the server's available GPU memory. Before each inference session, the client finds a chain of servers that collectively hold all model layers.
Once the chain is formed, the client uses the local embedding layer look up embedding vectors for prefix tokens, then sends those vectors to servers and receives new representations. Once the client obtains the outputs of the final block, it computes next token probabilities and repeats this process.
While the session is active, servers store attention keys and values from past client inputs and use them for subsequent inference steps. Clients also store past inputs to each server so that if any server fails or goes offline. another one can quickly take its place.
Client-side API. To generate tokens with PETALS, one first creates an inference session. An inference session iteratively takes inputs PyTorrc tensors, runs them through all Transformer blocks and returns final representations as PyTorch tensors. Under the hood, sessions form server chains, hold cache, and recover from server failures in way that is transparent to the user. An example using an inference session is shown in Figure
System requirements. For BLOOM-1788 inference, clients need least 12 GB RAM, most of which is used to store 3.6B embedding parameters. We recommend at least 25 Mbit/s bidirectional bandwidth to avoid bottlenecks network transfers. Simple greedy inference can use any CPU that runs Py Torch, but more advanced algorithms (e.g., beam search) may require a GPU.
In turn, servers need least GB CPU RAM, 100 Mbit/s bandwidth and GPU of Turing generation or newer with at least 8 GB of memory.
A.2 TRAINING FOR DOWNSTREAM TASKS
While LLMs achieve high quality on many problems with simple prompt engineering (Brown et al. 2020 they often need training achieve the best results. Traditionally, this is done by fine-tuning all model parameters on the downstream task. However, for extremely large models, this strategy
Ini initialize distributed BLOOM model model = AutoModeeFoorCuusallM wwworroooooooeeeeeeeleeeeeeeeeee input_ids (okenizer
with itt.iinnceeeeeesseeeeeeee
as session:
# Session maintains a list of servers that remember attention KV from previous steps for in range (sequence_eengeeeeieeeeeeeeeeeee
# Compute the word embeddings locally
hidden model gord_eembediingssiippee
# Run distributed Transformer blocks, store attention KV for future steps hidden = session. session.step(hidden stepdhidden)
# Generate the next token locally
probs model. hn_head(hiddenn input_ids sample_next_tooen (proo
Figure 2: A basic Py Torch code snippet for sequence generation with distributed BLOOM-176B.
16

---

Under review as a conference paper at ICLR 2023
becomes impractical due to hardware requirements. For example, fine-tuning BLOOM-1766 with
Adam would require almost 3 TB of GPÚ memory to store model, gradients, and optimizer states.
To combat this issue, the NLP community has developed parameter-efficient fine- fine-tuning methods that
keep most of the pretrained model intact. Some them choose a subset existing parameters (Sung
al.
2021 Guo al.
2021), others augment the model with extra trainable weights (Hu et al.) 2021
Houlsby et al. 2019
Liu et al. 2021b Lester et al." 2021
et al. 2021a 2022b).
Despite their lower memory requirements, parameter-efficien approaches often competitive with full model fine-tuning (Hu et al., 2021 Liu et al. 2021a; Yong & Nikoulina 2022) and even outperform it in low-data regimes (Liu et al. (022a). Another appealing property of these approaches for our use-case is that they allow rapidly switching pretrained LLM between different uses.
Distributed fine-tuning, The core principle of fine-tuning in distributed network is that clients "own" trained parameters while servers host original pretrained layers. Servers can run backpropaga- tion through their layers and return gradients with respect to activations, but they do not update the server-side parameters. Thus, clients can simultancously run different training tasks on the same set of servers without interfering with one another.
To illustrate this principle, we first review an example of soft prompt-tuning for text classification and then generalize it to other methods and tasks. Similarly to Section [A.]] clients store the embedding layers locally and rely on servers to compute the activations of Transformer blocks. In this fine-tuning scenario, client needs store trainable soft prompts (task-specifie input embeddings) and linear classification head.
For each training batch, the client routes its data through a chain remote servers compute sentence representations, then obtains predictions with the classifier head and computes the cross- entropy loss. During backpropagation, the client runs its data through the same chain of servers in reverse order to compute gradients for the learned prompt vectors. Having obtained those gradients, the client can use regular PyTorch optimizer to update the parameters of both the head and the prompts, then proceed to the next minibatch.
User interface. To allow users greater flexibility in their training workloads, made distributed backpropagation module compatible with the Py Torch Autograd engine. Like in the inference stage, this module handles fault tolerance and load balancing transparently the user while allowing them to access intermediate activations and insert custom PyTorch modules. shoows an example training code snippet.
This interface can also support other popular parameter-efffcient fine-tuning algorithms, such as LoRA (Hu al. (02T)) or prefix tuning (Li & Liang, 202T). Finally, users can insert custom local modules after some of the existing blocks, which could allow use-cases like
generation (Borgeaud al. 2021 Lewis et al. 2020).
# Initialize distributed BLOOM with soft prompts model = PorooodelForPrompttTnnng
# Define optimizer for prompts and linear head optimizer torch.optin AdamlM Adanl((model. pal...aaamm parameters())
for imput_ids, labels in data_loader:
# Forward with local and remote layers outputs model. forward((inutt loss cross_entroppyouututts logi ts, labels)
# Distributed backward w.r.t. local params loss. backward() Compute optimizer.. step() Update local params only optimizer. Notimizer.zero_grreee (ero_grad()
grad
Figure 3: basic PyTorch code snippet of soft prompt tuning for sequence classification with PETALS.
17

---

Under review as a conference paper at ICLR 2023
Table 3: Zero-shot accuracy for OPT-175B and BL00M-1776 with 8-bit and 16-bit weights.
Model
Bits
HellaSwag
LAMBADA
WinoGrande
Avg
16
78.5
74.7
72.6
75.3
OPT-175B
8
78.5
74.6
71.7
74.9
16
73.0
67.2
70.1
70.1
BLOOM
8
72.8
68.1
70.1
70.3
Table 4: Generation throughput (tokens/s) for BLOOM-176B with 8-bit and 16-bit weights on
8x A100 GPUs
Batch size
Weights
1
8
32
16-bit
4.18
31.3
100.6
8-bit
3.95
29.4
95.8
A.3 SITARING AND RBUSING TRAINED MODULES
Although most fine-tuned extensions for pretrained models can be easily shared as-is, simplifying the workflow for sharing these extensions enables users to more easily adapt the model to their target scenario. Indeed, existing model hubs (Wolf et al. 2020 TensorFlow Hub PyTorch Hub) have gained immense popularity due to many supported models and ease of use, especially when vetting different pretrained models for a given problem. One particularly relevant project AdapterHub (Pfciffer et al.) 2020), repository of trained adapters accompanied by library with implementations of different adaptation methods. While PETALS does not depend on AdapterHub, is possible to leverage this library for training adapters in the distributed setting. Instead, support sharing modules trained by users via the Hugging Face Hub (also used as a backend by AdapterHub). Its infrastructure and the corresponding open source library simplify the learning process for users already familiar with the ecosystem. Because the primary navigation mechanism on the Hugging Face Hub are tags that have been applied uploaded modules, a user only needs to the task was trained on and the model upon which the adapter was built. Uploading the weights and the code of the fine-tuned module is done by committing them Git repository. When navigating the Hub, users can choose the most suitable adapters by filtering the list of all available modules by the required tags.
B QUALITY AND EFFICIENCY OF BLOOM WITH 8-BIT QUANTIZATION
As shown in Table this method has little effect on LLM quality for major benchmarks. In terms of inference time, Table demonstrates that quantization has about 5% of overhead with batch size (20 tokens), but becomes negligible for larger batches.
C ESTIMATING THEORETICAL BEST THROUGHPUT WITH RAM OFFLOADING
In this estimate, we use the best possible hardware setup for offloading: CPU RAM offloading via PCIe with 16 PCIe lanes per GPU. In 8-bit, the model uses of memory per billion parameters while PCIe 4.0 with 16 lanes has throughput of 256 Gbit/s and 128 Gbit/s for PCIc 3,0. As such, offloading 176B parameters takes 5.5 seconds for PCIe 4.0 and 1l seconds for PCIe 3.0. We assume offloading latency of zero for the upper bound estimation. In reality, many GPU servers use PCle switches which limits the bandwidth. For instance, two GPUs are behind PCIe switch, transfering data to both the same time will halve the bandwidth.
18

---

Under review as a conference paper at ICLR 2023
D DETAILS OF THE SERVER LOAD BALANCING ALGORITHMS
Measuring throughput. Before joining for the first time, each server measures its Internet connec- tion throughput (in tokens/second, using one of public web APIs for doing that) and GPU throughput (in tokens/second, using a small benchmark running several forward passes). The minimum of these values becomes the overall server throughput, which is then cached for future runs.
Befonejoiiiin
Initial block assignment. We assume that cach server holds a segment of consecutive transformer blocks to minimize inference latency. Clients may request to perform forward or backward pass for the whole segment of blocks or its subsegment, if necessary. Normally, each server loads as many blocks as it can fit in its GPU memory, unless a user limits the number of blocks to utilize the rest of memory for something else.
Before starting, each server caleulates the values of t, the total throughput of servers currently holding the i-th block or loading it (to start holding it in a few minutes). Then, to find the best segment of blocks serve, the server looks for the most narrow bottleneck in the network. Formally, if the model has L blocks and the server can hold K of them in its GPU memory, we calculate:
start
arg min sorted f441, .... k+))-1))
i=1, ...., L-K+1
Here, arg min compares the sorted arrays lexicographically and chooses the leftmost start case of multiple minimums.
This way, the next joining server would always cover a block with the smallest t,. If there are multiple bottlenecks like this, the server will try to cover as many of them as possible (we choose to cover the minimums first because the overall PETALS throughput is the minimum of throughputs among model blocks). Among the remaining options, we choose a segment covering as many second minimums as possible, and so on.
Quality of block assignment. While we are not aware of the exact polynomial-time solution for the problem of assigning the segments optimally, we have conducted computational experiments and found out that this greedy algorithm (running in polynomial time) usually finds an assignment with total throughput of 90-100% of the optimal one (found by trying out all possible assignments in exponential time), given that the values of throughput are realistic to the PETALS setup.
Rebalancing. Since servers may leave at any time, each server also periodically checks if the current assignment is "good enough" compared to the throughput estimated by running the greedy solution for servers currently present in the network.
Formally, each server periodically looks for a segment of blocks that is more appropriate than the currently loaded blocks with respect to the arg min rule I If it finds one, it simulates how the rest of the servers would behave if we replace the current blocks with the new ones (how other servers would change their blocks afterwards). If the eventual throughput is least p% better, the server commits to the change and announces that it changes the blocks, then other servers do the rest of the changes (eventually increasing the total throughput).
We use p- 20% since it gives a reasonable trade-off' between the swarm throughput and the frequency of block replacements in our experiments (see Appendix (E). Specifically, lower value of p leads to block replacements happening too often, which negatively affects the inference latency since each block replacement resets attention caches for this block.
Stability of the greedy algorithm. The rebalancing algorithm does not cause oscillations since series of block replacements is executed only if it leads to eventually increasing throughput by at least p%. Once "good enough" throughput is achieved, servers do not change their blocks anymore (unless an essential number of servers join or leave). We verified this behavior computationally, simulating network with thousands of servers with different throughputs.
To conclude, this greedy heuristic allows servers to quickly close the gaps if a substantial share (up to 100%) of servers holding certain blocks leave, but avoids excess block replacements otherwise.
19

---

Under review as a conference paper at ICLR 2023
Pipeline throughput
Throughhut (tokensss)
No load balancing
300-
New servers only
Full, 20%
200
p 10%
Full, p 19%
100
Upper baund
100
200
300
400
500
600
700
Number block replacements per minute (10 min averuge)
Replacements per minule
100
50
200
300
400
600
700
Time (minutes)
Figure 4: Behavior of the load balancing algorithms evaluated in Appendix
E
E EVALUATION OF THE SERVER LOAD BALANCING ALGORITHMS
In this section, we measure the effectiveness of the load balancing algorithm used in PETALS. We all experiments using fleet of 206 virtual instances that simulate PETALS participants. To keep experiment costs manageable, we do not use GPUs for this evaluation, instead simulating uneven server throughput programmatically. For each server, we sample its throughput from the uniform distribution t ~ UO, 100] tokens/second, then sample its memory size so it can hold 6 ~ Ui], 10] blocks (out of 70 blocks total, as in BLO0M-176B).
Each server follows certain availability schedule, i.e. turns on and shuts down at the same predefined time across all experiments. We assign these schedules such that the number of active servers follows sine wave, simulating daily activity cycles. The schedule has approximately 100-110 active servers during peak activity and 15-25 servers its lowest points. Note that each peak contains a different subset of 100-110 active servers out of 206 instances in total.
We evaluate the following approaches to load balancing:
No load balancing baseline system where servers load random contiguous interval of model blocks.
2. Balancing new servers only simplified load balancing where servers choose the optimal blocks when joining the swarm (using the rule from Appendix D but never change them.
3. Full load balancing the full algorithm, where every minute each server checks if they need to replace their blocks. We use the efficiency threshold p (as described in Appendix D to avoid excess block replacements.
4. Upper bound the best-case throughput estimate that reassigns contiguous block segments to servers optimally every minute.
The full load balancing maintains connectivity throughout the experiment and achieves throughput close the upper bound (staying within the 10-15% range most of the time). Higher thresholds p perform slightly worse during peak times but require only relatively infrequent block replacements, unlike the case with p = 1%. Note that using the assignment leading to the upper bound is not possible in practice since requires each server to load different set of layers every minute, on top of solving the computationally expensive optimization problem.
We report their behavior in Figure 4
Curiously, the baseline running load balancing for new servers only achieves reasonable throughput during periods where servers are actively joining. However, quickly loses throughput when random servers leave, since this creates "bottlenecks" in the pipeline that require rebalancing of existing peers. Finally, the naive baseline with random layer assignment has zero throughput most of the time because it is unable form complete pipeline.
20

---

Under review as a conference paper at ICLR 2023
F DISCUSSION AND FUTURE WORK
Privacy. A key limitation of our approach is that peers serving the first layers of the model can use their imputs recover input tokens. Thus, clients working with sensitive data should only use the servers hosted by trusted institutions that are allowed to process this data. This limitation may be addressed in future work using secure multi-party computing (Evans al., 2018) or privacy- preserving hardware (VIDIAA(002))
Incentives for peers to contribute. In PETALS, peers using the client are not required to run server. Naturally, this may lead to an imbalance between supply (peers who dedicate GPUs to serve model layers) and demand (peers using the servers to perform inference or fine-tuning for their own needs) in the network. One way to encourage users serve model layers would be to introduce a system of incentives: peers running servers would earn special points, which can be spent high-priority inference and fine-tuning or exchanged for other rewards. This system may be implemented using a centralized "accounting" server or in a decentralized way.
Security. We assume that servers in our system are run by many independent parties. practice. some of them may turn out to be faulty and return incorrect outputs instead of the actual results of forward and backward passes. This may happen due to a malicious intent to influence other people's outputs when rewards are introduced (as described above), to earn reward for serving layers without actually performing the calculations.
A possible way to address these issues would be to use economically motivated approach. Some servers may vouch for the correctness of their outputs (e.g., in exchange for increased inference price) by depositing certain number of points pledge. Then, for each request, they announce cryptographic hash the input and output tensors, so anyone having the inputs can check whether the outputs correct.
If someone finds a mismatch confirmed by a trusted third party, they can claim the server's pledge as reward. In practice, it may be client who suspects that they received wrong outputs "bounty hunter" sending requests to different servers in the hope of catching errors. While this approach still leaves chance of receiving wrong outputs, it makes cheating costly and creates an incentive to quickly expose the malicious servers.
Making changes to the main model. As discussed in Section A.2 distributed parameter-efffiient fine-tuning makes it casy for users to apply the base model to new tasks. In Section A..3] also described how these updates can be easily shared and reused by others. This capability provides a meaningful step towards collaborative improvement of machine learning models (Raffel 2021): as more and more users train the base model, it will effectively become more capable over time.
Furthermore, might expect the model parameters that perform best specific task to change over time. Similarly to version control systems for code, would be useful to track versions of fine-tuned model parameters as they change. A system for rapidly testing the performance of set of parameters on "living benchmarks" (Kiela et al.) [021] Gehrmann et al. [202)] Gao et al.) 2021) would be valuable to ensure that subsequent versions improve the desired capabilities.
Apart from adaptation to new tasks, it would also be useful to eventually update the main model. Ideally, such updates could be tracked in principled way. Users of PETALS could specify the versions of the model they want to use, and servers could indicate which versions they support. Introducing a newer version of the model then reduces to adding new group of layers, which then naturally supersedes older parameters. Similarly, fine-tuned model adapters could be annotated with tags denoting the model version they are applicable for. Such fine- -eeinga versioning of models is currently uncommon but would be straightforward to add to PETALS.
21


Reviews:
[{"review_id": "gSXcpmH0i6", "reviewer": "Reviewer_X2vi", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes PETALS, a system that enables inference or finetune processing of large language models (LLM) without the cost of HPC hardware, but rather by sharing a large pool of consumer-grade hardware. PETALS uses pipeline parallelism to partition LLM layers over the geographically distributed devices and proposes a fault tolerant algorithm for inference and fine tuning to address reliability issues. The paper presents some evaluation of the fault tolerance algorithm and performance comparison to offloading. ", "summary_of_the_review": "While I think PETALS would be a useful system to many users, the current evaluation makes it difficult to understand how well it would function and perform in real life scenarios. Moreover, the previously noted evaluation issues makes it difficult to know the scenarios that PETALS would be preferable to caching with restarts and offloading. ", "strength_and_weaknesses": "Although, pre-trained LLMs are now freely available for download the hardware cost of inference and finetuning computation is still quite prohibitive and is a barrier to broad accessibility of AI advancements. Thus, the paper is tackling an important problem. The proposal of a community-approach that involves pooling consumer-grade resources together to inference and finetune LLMs is also reasonable and applicable to a broad class of users.  The presented algorithm and implementation details are useful in understanding some of the technical challenges. \n\nMy main concern is the evaluation, which I feel does not present a complete and fair assessment of PETALS. Below are specific examples. \n\n1. Overhead of fault tolerance (4.1)\n    a) What are the runtime and memory costs of client and serving caching, and how they scale with increasing client and server counts?\n    b) When clients don't have GPUs, does this mean client-server communication involve GPU/CPU copying on the server?\n    c) Are the runtime and memory costs of fault tolerance reflected in the results in 4.2 and 4.3?\n    d) Table 1 only shows fault tolerance evaluation for inference. Is the fault tolerance provided for fine tuning?\n    e) In Table 1, PETALS is the best option in only 2 of the 8 scenarios. Caching with restarts wins 5 times, No caching wins once. This is not very convincing for PETALs. \n\n2. Comparison to Offloading (4.2)\n   a) Using 1 A100 for Offloading and 3 A100 for PETALs does not seem like a fair comparison. Can you elaborate?\n   b) The theoretical latency numbers of Offloading suggest that 16-bit weights are being used rather than 8-bit weights, is this correct?\n   c) Are the A100 GPUs connected by PCIe or NVLINK?\n  d) Depending on the response to the preceding 3 issues, a fair evaluation could mean scaling the Offloading performance by up to 18X: \n    3X for GPU count, 3X for parallelizing weight fetching from CPU, and 2X for 8-bit weight quantization. \n  e) Why are there no forward pass results for the multiple client section of Table 2?\n ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "_IA5vcDWl3", "reviewer": "Reviewer_s5Ah", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors proposed a method to perform inference na fine-tuning of LLMs over in a distributed manner over the internet, based on the fact that activation communication between geo-distributed devices could be faster than offloading to local memory for really large LMs. The authors developed a fault-talent algorithm to handle the error and proposed some system design guidelines. Experimental results show that the proposed method can outperform caching/recomputing under higher failure rates and has a higher throughput than RAM offloading. ", "summary_of_the_review": "Please see the Strength and Weakness section. I am not 100% familiar with the literature on distributed systems. And I am open to raising scores given a convincing reply. ", "strength_and_weaknesses": "Strength:\n1. Firstly, the paper is generally well-written and easy to follow. \n2. The observation that transferring activation is faster than offloading weights in LLMs is interesting. \n3. The proposed method addressed an important question and allowed communities with lower-end GPUs to be able to serve and tune LLMs. There could potentially be a new business model based on collective inference.\n\nWeakness:\n1. I am not very familiar with distributed system research. But from my understanding, fault tolerance in a distributed system has already been well studied. It seems the proposed method only applies the techniques to LLM inference and does not have a significant novelty. \n2. From my understanding, the method can only work when the model is large enough (as the authors mentioned, >100B). Could you please show how it works for smaller LMs? What model size is the turning point?\n3. When comparing the baseline of offloading, did you apply the same quantization technique to reduce bandwidth requirement? \n4. Did you implement pipeline parallelism to further improve the throughput?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "VodjGdLCerf", "reviewer": "Reviewer_JnBu", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents Petals, a system for inference and fine-tuning of large language models on distributed machines over the internet. The system mainly includes the following techniques:\n\n- Pipeline parallelism across geo-distributed devices.\n- Fault tolerance via storing an extra copy of pipeline communication data within the client.\n- A dynamic load-balancing algorithm that eliminates performance bottlenecks.\n- Using adapters for fine-tuning and storing the adapter parameters in the client for fault-tolerant fine-tuning.\n- Quantization and compression to reduce memory footage.\n\nThe authors evaluate the system on a smaller BLOOM model for fault tolerance performance and test on the large BLOOM-175B model with multiple hardware settings to test the latency of the proposed system.", "summary_of_the_review": "The paper proposed a solid system for geo-distributed inference. The work includes solid system engineering but lacks enough novelty for a research paper.", "strength_and_weaknesses": "## Strength\n\n1. The paper proposes a valid combination of multiple techniques for efficient inference and fine-tuning of LLMs, including pipeline parallelism, adapters, quantization and compression.\n2. The engineering behind the system is non-trivial and I appreciate the authors for open-sourcing their work. I believe the work will be beneficial for researchers with access to geo-distributed machines.\n\n## Weakness\n\n1. Pipeline parallelism is a well-studied technique for large language models. The way this paper applies pipeline parallelism is not new compare to previous works.\n2. The dynamic load balancing (i.e., schedule which layers to run on each GPU) are not evaluated in the experiments. How does this algorithm adapts to different hardware configurations?\n3. Comparing the proposed setting with the offloading techniques is not a completely fair comparison: The offloading setting only has a single GPU, but the geo-distributed setting has many more GPUs and thus much higher total computational power. \n4. A more important comparison is to compare the efficiency of this method with the setting where the GPUs are within a single datacenter and the optimal performance in that case.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "OJS3AKKWV2f", "reviewer": "Reviewer_3oHu", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work presents Petals, a system designed for inferencing and fine-tuning large-language models over distributed, commodity hardware.     This enables users to leverage large, pre-trained models without having exclusive access to high-end hardware, and instead being able to leverage the collective inferencing capability of many systems across the internet.   The system uses pipeline-based model parallelism, distributing model layers across nodes, and both server and client caches to restore state during server failures.  They build the system and evaluate its performance through a failure microbenchmark and illustrate performance on larger clusters with different emulated network conditions.  In that scenario, Petals is about 10x faster compared to offloading on a single GPU.", "summary_of_the_review": "Interesting, motivated, impactful problem whose solution was evaluated on both local and wide area networks for wide scale inference of LLMs.  The work could use further details and experiments to validate the claim that the system works well under network and node failures.  \n\n[Post Author Response] I've reviewed the author's responses and paper changes.  They have addressed a number of my concerns, there remain open questions around system operation at scale (vis-a-vis load balancing).   Some could be addressed by describing the limitations of the current load balancing heuristics and analysis.  Even with these drawbacks, the work could be impactful.  I've adjusted the rating from marginally below to marginally above. \n", "strength_and_weaknesses": "Strengths\n* Great problem, high impact, fun read \n* Technically interesting system architecture with a functional implementation  \n* Evaluation in the wide area, compelling numbers wrt an offloading baseline\n\nWeaknesses\n* The description of the fault-tolerant load-balancing algorithm is insufficient.  The stability of the N-player greedy approach needs to be addressed.  The performance model is not described nor its accuracy evaluated.   \n* It isn't clear how Petals initially groups layers to the servers or if layer groupings can change.   \n* The evaluation does not empirically explore actual server failures or network partitions (to the client, between servers).   I.e., there are no results bolstering the core contribution (I comment below on why Table 1 is a great microbenchmark but insufficient in this regard). \n* Explores uniform network bandwidth and latency, but it's unclear what this is really testing in the Petals architecture.   \n* Compares to offloading but should compare to DeepSpeed in the wide area under conditions that illustrate Petals benefits. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject