SlowGuess/NeurIPS_2023_OCR huggingface dataset:
To conclude, this greedy heuristic allows servers to quickly close the gaps if a substantial share (up to  $100\%$ ) of servers holding certain blocks leave, but avoids excess block replacements otherwise.
Figure 2: Behavior of the load balancing algorithms evaluated in Appendix E.
E Evaluation of the server load balancing algorithms
In this section, we measure the effectiveness of the load balancing algorithm used in our system. We run all experiments using a fleet of 206 virtual instances that simulate participants. To keep experiment costs manageable, we do not use GPUs for this evaluation, instead simulating uneven server throughput programmatically. For each server, we sample its throughput from the uniform distribution  $t \sim \mathbb{U}[0,100]$  tokens/second, then sample its memory size so it can hold  $b \sim \mathbb{U}[1,10]$  blocks (out of 70 blocks in total, as in BLOOM-176B).
Each server follows a certain availability schedule, i.e. turns on and shuts down at the same predefined time across all experiments. We assign these schedules such that the number of active servers follows a sine wave, simulating daily activity cycles. The schedule has approximately 100-110 active servers during peak activity and 15-25 servers at its lowest points. Note that each peak contains a different subset of 100-110 active servers out of 206 instances in total.
We evaluate the following approaches to load balancing:
1. No load balancing - a baseline system where servers load a random contiguous interval of model blocks.
2. Balancing new servers only - a simplified load balancing where servers choose the optimal blocks when joining the swarm (using the rule (1) from Appendix D) but never change them.
3. Full load balancing – the full algorithm, where every minute each server checks if they need to replace their blocks. We use the efficiency threshold  $p$  (as described in Appendix D) to avoid excess block replacements.
4. Upper bound — the best-case throughput estimate that reassigns contiguous block segments to servers optimally every minute.
We report their behavior in Figure 2. The full load balancing maintains connectivity throughout the experiment and achieves throughput close to the upper bound (staying within the  $10 - 15\%$  range most of the time). Higher thresholds  $p$  perform slightly worse during peak times but require only relatively infrequent block replacements, unlike the case with  $p = 1\%$ . Note that using the assignment leading to the upper bound is not possible in practice since it requires each server to load a different set of layers every minute, on top of solving the computationally expensive optimization problem.
Curiously, the baseline running load balancing for new servers only achieves reasonable throughput during periods where servers are actively joining. However, it quickly loses throughput when random servers leave, since this creates "bottlenecks" in the pipeline that require rebalancing of existing peers. Finally, the naive baseline with random layer assignment has zero throughput most of the time because it is unable to form a complete pipeline.
F Experiments with a wider range of failure rates
In this section, we follow the setup from Section 4.1 and provide additional evaluations for a wider range of failure rates (up to  $5\%$ ) and sequence lengths (up to 2048 tokens). The results are shown in Figure 3. Unlike baselines, our algorithm provides reasonable performance in all tested conditions, especially for higher failure rates common for communicating over the Internet, using spot/preemptible instances or unreliable hardware).
Figure 3: Sequential inference speed (steps/s) for BLOOM (7.1B) with varying failure rates. The setup is the same as in Section 4.1. A failure rate  $p$  means that sending a set of activations to the next pipeline stage fails with probability  $p$ . Zero speed means that the baseline did not finish within 1 hour.
Table 6: Throughput (tokens/sec) of forward and backward passes for different tasks, batch sizes, prefix lengths.
Mode	Batch size	Prompt length	Forward pass throughput	Backward pass throughput
Prompt tuning	8	16	195.6	57.4
8	4	213.2	60.8
32	16	272.6	82.8
32	4	293.1	84.7
Prefix tuning (i.e., “deep” prompt tuning)	8	16	111.0	42.0
8	4	178.7	57.8
32	16	164.1	64.4
32	4	255.8	84.8
G Performance of training-time forward and backward passes
In this section, we evaluate throughput of training-time forward and backward passes and study factors that affect their performance. We will only consider BLOOM-176B and the  $3 \times$  A100, 1 Gbit/s setup from Section 4.2 and focus on finetuning-specific hyperparameters, since the influence of network bandwidth and latency has already been discussed in the main paper.
Sequence classification. First, we consider fine-tuning the model on a binary classification task. We take BLOOM-176B, replace the logit layer with a trainable classification head (similar to transformers.BloomForSequenceClassification), and add trainable prompts before the input sequence, then train the model on batches of 128-token sequences. We try (a) both prompt tuning and prefix tuning (involving "deep" prompts), (b) two batch sizes (8 and 32), and (c) two prompt lengths (16 and 4). The client shares 8 CPU cores with one of the servers and does not use the GPU.
The results are provided in Table 6. The prefix tuning turns out to be slower, since it adds several times more trainable parameters. Increasing prompt length and decreasing batch size also make training slower. Notably, we observe that moving client-side computations to GPU does not visibly improve performance, since the client does not perform any heavy operations in this setup<sup>11</sup>.
Language modeling. Next, we consider fine-tuning the model on a causal language modeling task. We take BLOOM-176B, keep the logit layer, and add trainable prompts before the input sequence. We explore the same hyperparameters as with sequence classification.
We observe that the throughput of the GPU-enabled client is similar (within  $10\%$  difference) to the throughput in case of sequence classification, reported in Table 6. Indeed, the client performs only a small share of GPU computations in the forward and backward passes, and a particular model head and a loss function do not have decisive influence on the performance. However, performance of the CPU-only client turns out to be 5-10 times worse in this setup, since the client has to multiply the output embedding matrix to the hidden states of all tokens in the batch. This operation is too large to be efficiently computed on  $\mathrm{CPU^{12}}$ .
H Limitations and broader impact
Privacy. A key limitation of our approach is that servers hosting the first model blocks may use their inputs to recover client data. Thus, users working with sensitive data should limit their clients to only use trusted servers or, alternatively, set up their own isolated network using our software. For example, if multiple research labs or small companies have access to a specific private dataset
In case of sequence classification, the heaviest operation the client does is multiplying  $2 \times h$  and  $h \times b$  matrices, where  $h$  is the hidden dimension (14336 in BLOOM-176B) and  $b$  is the batch size.
In case of language modeling, the client has to multiply  $d \times h$  and  $h \times b$  matrices, where  $d$  is the token vocabulary size (250880 in BLOOM-176B). This is  $\approx 10^5$  times more FLOPS than used in case of sequence classification.
and want to process it with a large language model, they may set up an isolated distributed network hosting this model to get a better inference speed, compared to running the model independently.
In the future, this limitation may be addressed in future work using secure multi-party computing (Evans et al., 2018) or privacy-preserving hardware (NVIDIA, 2022).
Motivating contributors. Since people using the client are not required to run a server, our system may experience an imbalance between supply (peers who dedicate GPUs to serve model layers) and demand (peers using the servers to perform inference or fine-tuning for their own needs).
One way to encourage users to serve model blocks would be to introduce a system of incentives: peers running servers would earn reward points, which can be spent on high-priority inference and fine-tuning or exchanged for other rewards. To implement this, we can run a few validator peers that periodically traverse all available servers and issue reward points to their owners.
Security. We assume that servers in our system are run by many independent parties. In practice, some of them may turn out to be faulty and return incorrect outputs instead of the actual results of forward and backward passes. This may happen due to a malicious intent to influence other people's outputs or, when rewards are introduced (as described above), to earn a reward for serving layers without actually performing the calculations.
To address this issue, we can extend the validator peers, so that they periodically test servers with random requests of different types and ban them if they respond with incorrect outputs (possibly, revoking their rewards). The validator requests should be difficult to distinguish from requests of typical users, so that malicious servers cannot pretend to be honest to the validators but send wrong outputs to other peers. While this approach still leaves a chance of receiving wrong outputs, it allows to eventually expose and penalize the faulty servers.
Finally, clients may reduce the probability of getting faulty outputs by running their data through multiple disjoint chains of servers simultaneously and comparing the outputs against each other.
Broader impact. This work introduces a general-purpose algorithm for decentralized inference and fine-tuning of large models, aiming to simplify access to the latest research in deep learning and provide an alternative way to efficiently run LLMs without high-end hardware. We do not envision any direct negative impacts from our research, since models that can be hosted with our system are already widely available and may be used via APIs, offloading, or other means.