TPI-LLM: Serving 70B-scale LLMs Efficiently on Low-resource Edge Devices

## Abstract

Abstract Large model inference is shifting from cloud to edge due to concerns about the privacy of user interaction data. However, edge devices often struggle with limited computing power, memory, and bandwidth, requiring collaboration across multiple devices to run and speed up LLM inference. Pipeline parallelism, the mainstream solution, is inefficient for single-user scenarios, while tensor parallelism struggles with frequent communications. In this paper, we argue that tensor parallelism can be more effective than pipeline on low-resource devices, and present a compute- and memory-efficient tensor parallel inference system, named TPI-LLM, to serve 70B-scale models. TPI-LLM keeps sensitive raw data local in the users’ devices and introduces a sliding window memory scheduler to dynamically manage layer weights during inference, with disk I/O latency overlapped with the computation and communication. This allows larger models to run smoothly on memory-limited devices. We analyze the communication bottleneck and find that link latency, not bandwidth, emerges as the main issue, so a star-based allreduce algorithm is implemented. Through extensive experiments on both emulated and real testbeds, TPI-LLM demonstrated over 80% less time-to-first-token and token latency compared to Accelerate, and over 90% compared to Transformers and Galaxy, while cutting the peak memory footprint of Llama 2-70B by 90%, requiring only 3.1 GB of memory for 70B-scale models.

## 1 Introduction

Recently, Large Language Models (LLMs) have been widely deployed in the cloud for inference. User inputs are uploaded to the cloud, where high-performance GPUs are used to compute output sequences, and then sent back to user devices for display. This process poses privacy risks, as user prompts are exposed to network intermediaries and clouds.
Therefore, there is an increasing need to shift LLM services to the network edge, such as on laptops, hand phones, tablets, and desktop computers. However, edge devices have very limited memory (4-16 GB) and computing power (often CPU-only). Even with quantization, running a Llama 3.1-70B model still requires at least 40 GB of memory, which far exceeds the capacity of most edge devices. Besides, running Bert-L on one Nano-M device results in a latency that is 120$\times$ longer than on one A100 GPU. This gap requires the use of more edge devices to support and speed up LLM inference on the network edge.

While advanced LLM serving systems (Shoeybi et al., ; Rasley et al., ; Li et al., ; Agrawal et al., ; Miao et al., ) have been designed for high-performance GPU clusters, recent efforts (Zhang et al., ; Mei et al., ; Borzunov et al., ) are adapting these systems to edge environments, by adaptively partitioning model between edge devices and optimizing schedulers to boost token throughput. However, in smart home scenarios like smart speaker, edge LLM systems often handle one user request at a time, making them degrade from pipeline to model parallelism and leaving devices idle most of the time. Thus, tensor parallelism is preferred for better efficiency. For instance, Ye et al. () combine tensor and sequence parallelism to reduce token latency and Wei et al. () introduce block parallelism to restructure Transformer layers.

However, even with 8 devices sharing the load, running full-precision Llama 2-70B still requires 35 GB per device, memory remains a shortage. Solutions like memory block paging (Kwon et al., ) and optimized KVCache storage (Jin et al., ; Lee et al., ) help schedule data between GPUs and CPUs, but unfortunately, GPUs are not available on most edge devices. As a popular alternative, Accelerate (Gugger et al., ) can offload model data from a CPU to a disk to run larger models, but its blocking I/O drastically slows inference, with token latency increases to 30 seconds per token on Llama 3.1-8B.

In this work, we analyze why tensor parallelism is more effective than model parallelism on low-resource edge devices and present TPI-LLM, a computing- and memory-efficient tensor parallel inference framework to serve LLM models. Constrained by the high link latency, a star-based allreduce algorithm is implemented. To address the memory shortage, a sliding window memory scheduler is further introduced. We build a prototype of TPI-LLM with 3K LoC and two testbeds using Klonet (Ma et al., ) and 4 laptops. Extensive results on Llama 3.1-8B/70B (Dubey et al., ), Llama 2-3B/7B/13B/70B (Touvron et al., ) and Yi-34B (AI et al., ) demonstrate the significant reduction of the memory footprint and faster inference speed compared to Transformers (Wolf et al., ), Acclerate (Gugger et al., ), and Galaxy (Ye et al., ).

We summarize the main contributions of this work as follows:

- •
We design a TPI-LLM for edge LLM serving, which keeps prompt privacy in mind to allow edge devices with limited computing power collaborate to deliver faster inference.
- •
We find that network bandwidth is no longer an issue. Instead, link latency causes high delays in advanced allreduce algorithms. Thus, a star-based allreduce algorithm is implemented, which greatly outperforms ring- and tree-based methods.
- •
We introduce a sliding window memory scheduler, which asynchronously loads and unloads layer weights and overlaps disk I/O latency with computations and communications, enabling the inference of larger models on low-memory devices.
- •
We prototype TPI-LLM and show that it reduces time-to-first-token and token latency by over 80% compared to Accelerate and over 90% compared to Transformers and Galaxy. It serves Llama 2-70B with a peak memory footprint of 3.1 GB across 8 low-resource devices.

## 2 Observations and Motivations

Before presenting our TPI-LLM system, we address two questions that guide our design:

Q1: On low-resource edge devices, which dominate inference time: computation or communication? Which is more efficient, tensor parallelism or model parallelism?

On the network edge, the balance between computation and communication differs from that in high-performance GPU clusters. To determine whether tensor or model parallelism offers more benefits, it is essential to identify which—computation or communication—takes up more time. For this purpose, we examine the Llama 3.1-8B model on a LAN network with 4 laptops of 8 cores. The network bandwidth between them is 178 Mbps, and the devices implement allreduce communications using a parameter server architecture (Li et al., ).

Figure: (a)
Refer to caption: x1.png

Figures 1(a) and 1(b) show the timeline and computing-communication time ratio for model and tensor parallelism during inference. In model parallelism, communication accounts for only 2% of the time, with most spent on computation. However, when one device is computing, others are idle, creating pipeline bubbles and resource waste. In tensor parallelism, communication rises to 70%, but all devices compute simultaneously, and the speed boost outweighs the communication cost, leading to less overall inference time. This makes tensor parallelism the preferred choice.

Q2: Is tensor parallelism enough for edge LLM serving?

Tensor parallelism does reduce memory footprint each device by sharing model parameters across multiple devices, but it doesn’t fully address the memory shortage. Figure 1(c) shows that even with 4 tensor parallel nodes, memory footprint remains at 12 GB—still too high for most edge devices. This is because memory footprint includes not just model parameters but also intermediate results, key value cache, libraries, etc., causing the actual usage to exceed the theoretical value. Besides, other apps on the device also compete for memory, which worsens the shortage. Thus, even with tensor parallelism, a memory scheduler is still needed to avoid out-of-memory (OOM) issues.

## 3 TPI-LLM Framework with Sliding Window Memory Scheduling

In a typical inference workflow, many users send their prompts to a cloud-based service. These prompts are pooled and scheduled in batches, undergoing dozens of Transformer layers, and converted into probabilities to predict the next token. This process repeats until the generated sequence is finished. While the fundamental workflow on the cloud and edge are similar, key differences arise:

(a) Keep prompts and generated sequences on users’ device.
In a cloud setup, user prompts are sent to remote servers for processing, which result in exposure of private data. Edge LLM serving systems are required to keep prompts and generated sequences in users’ own devices to ensure raw data never get exposed to external unknown environments.

(b) More single-prompt serving. Current LLM serving systems are typically optimized for batched prompts using pipeline scheduling. However, these optimizations lead to resource underutilization in edge scenarios like smart speakers, where only one prompt is processed at a time.

(c) Low-resource devices without CUDA support.
Edge devices, unlike cloud GPUs, have very limited memory and low computing power. Many of them lack CUDA support or do not have GPUs at all, and they often prioritize full precision to ensure faster computations.

### 3.1 The Parallel Framework Design of TPI-LLM System

Figure: Figure 2: Overview of the TPI-LLM parallel framework.
Refer to caption: x4.png

The proposed tensor parallel inference system (TPI-LLM) tackles these challenges by using a tensor parallel framework that distributes attention heads across multiple nodes. As depicted in Figure 2, it involves a master node, typically the user’s device that initiates the prompt, and several worker nodes that share the computational load. Their pseudo codes are given in Algorithms 1 and 2.

Step 1: The master node partitions and distributes model weights.
Before inference begins, the master node partitions the pretrained model weights ${\bm{W}}$, such as attention heads and FFN weights, among the worker nodes. Workers with greater computing power and larger memory are allocated more attention heads and FFN weights. This ensures no single device bears the full burden.

Step 2: The master node initiates prompt and broadcast the input embedding to workers.
The inference process starts at the master node, where a user prompt is tokenized into a list of token indices ${\bm{x}}$ and then transformed into input embeddings ${\bm{H}}^{0}={\bm{x}}{\bm{W}}_{\mathtt{emb}}$. The embedding is then broadcast to all worker nodes ${\bm{H}}^{0}_{\mathtt{ffn}}={\bm{H}}^{0}$ to initiate the tensor parallel workflow.

Step 3: All nodes perform tensor parallel computing.
The tensor parallel computing follows a cycle of four operations: attention computing, allreduce, FFN computing, and allreduce. These operations together constitute a Transformer block. Devices compute attention and FFN with partitioned weights in parallel, reducing the computing delays on low-power devices.

In the attention computation phase of the $l$-th Transformer block, device $h$ processes only a subset of attention heads ${\bm{Q}}^{h,l}={\bm{H}}^{l}_{\mathtt{norm}}{\bm{W}}_{Q}^{h,l},{\bm{K}}^{h,l}={
\bm{H}}^{l}_{\mathtt{norm}}{\bm{W}}_{K}^{h,l},{\bm{V}}^{h,l}={\bm{H}}^{l}_{
\mathtt{norm}}{\bm{W}}_{V}^{h,l}$, where ${\bm{H}}^{l}_{\mathtt{norm}}=\mathtt{norm}({\bm{H}}_{\mathtt{ffn}}^{l-1})$ is the normed hidden state and weight partitions ${\bm{W}}_{Q}^{h,l},{\bm{W}}_{K}^{h,l},{\bm{W}}_{V}^{h,l}$ are downloaded from the master node in Step 1. Once ${\bm{Q}}^{h,l},{\bm{K}}^{h,l},{\bm{V}}^{h,l}$ are computed, we apply the scaled dot-product attention to calculate the attention score, and the result is then synchronized across devices:

$$ ${\bm{H}}_{\mathtt{attn}}^{l}=\mathtt{all_reduce}(\mathtt{softmax}(\frac{{\bm{ Q}}^{h,l}({\bm{K}}^{h,l})^{\top}}{\sqrt{d}}){\bm{V}}^{h,l})+{\bm{H}}_{\mathtt{ ffn}}^{l-1},$ (1) $$

where $d$ is the dimension for attention head. Here, attention is computed in parallel across devices, followed by an allreduce to aggregate their hidden states and a shortcut connection. The key-value pair $({\bm{K}}^{h,l},{\bm{V}}^{h,l})$ is cached locally on device $h$ to reduce redundant computations. This distributed KVCache partitions the cache across devices, so memory cost is reduced on individual device.

After the attention computation and allreduce, the process continues with the FFN computation:

$$ ${\bm{H}}_{\mathtt{ffn}}^{l}=\mathtt{all_reduce}({\bm{W}}_{\mathtt{down}}^{h,l }\cdot(\sigma({\bm{W}}_{\mathtt{gate}}^{h,l}\cdot{\bm{H}}_{\mathtt{norm}}^{l}) \odot({\bm{W}}_{\mathtt{up}}^{h,l}\cdot{\bm{H}}_{\mathtt{norm}}^{l})))+{\bm{H} }_{\mathtt{attn}}^{l},$ (2) $$

where FFN weights ${\bm{W}}_{\mathtt{gate}}^{h,l},{\bm{W}}_{\mathtt{up}}^{h,l},{\bm{W}}_{\mathtt{
down}}^{h,l}$ are also partitioned weights, ${\bm{H}}_{\mathtt{norm}}^{l}=\mathtt{norm}({\bm{H}}_{\mathtt{attn}}^{l})$, $\sigma$ represents the activation function such as SiLU (Elfwing et al., ). Similar to the attention computation stage, the FFN is computed in parallel, followed by an allreduce and a shortcut connection.

Step 4: The master node reduces tensor parallel results and calculates the next token. After each node $h$ completes its part of computation within the backbone network, the result is sent to the master node. The summed results ${\bm{H}}_{\mathtt{ffn}}^{L}$ are then passed through a task head ${\bm{W}}_{\mathtt{head}}$ and softmax to obtain the probability distribution of the next token ${\bm{z}}=\mathtt{softmax}({\bm{H}}^{L}_{\mathtt{ffn}}{\bm{W}}_{\mathtt{head}})$, which is then sampled. Steps 2 to 4 repeat until an EOS token is generated or the length limit is reached.

TPI-LLM provides three benefits: (i) The user prompt $\{{\bm{x}}_{1},{\bm{x}}_{2},\cdots\}$ and generated sequence $\{z_{1}\sim{\bm{z}}_{1},z_{2}\sim{\bm{z}}_{2},\cdots\}$ are processed only on the master node, keeping them hidden from workers. Even if workers reverse-engineer input embeddings ${\bm{H}}^{0}$, they cannot recover the raw prompt ${\bm{x}}$ or next token $z\sim{\bm{z}}$ since the weights of input embedding ${\bm{W}}_{\mathtt{emb}}$ and task head ${\bm{W}}_{\mathtt{head}}$ reside solely on master. (ii) The inference speed is often limited by the computational latency, but in TPI-LLM, it is accelerated via parallel computing. (iii) Unlike other systems that use a mix of communication primitives (reduce & broadcast (Shoeybi et al., ), reducescatter & allgather (Ye et al., ), etc.), TPI-LLM standardizes communications to allreduce. This enhances compatibility with broader communication libraries like PS-LITE (Chen et al., ) and NetStorm (Li et al., ), leveraging their optimized implementations for edge conditions.

Table: Algorithm 1 Master (with rank 0):

### 3.2 Allreduce latency analysis

Given the dynamic and heterogeneous nature of edge networks, we tested NetStorm (Li et al., ) as the communication backend, but unfortunately, it resulted in high token latency. After further validation, we confirmed that this latency was not due to network bandwidth, but due to link latency.

To analyze the impact of network bandwidth and link latency, we make the following assumption.

###### Assumption 1 .

Assume that the edge network adopts a physical topology as shown in Appendix A.7, the network links have the same latency $\tau$, the allreduce algorithm follows a tree-based structure of depth 2 for aggregation, and each device has the same computing power.

The allreduce latency can be expressed as $t_{\mathtt{all_reduce}}=2L(t_{\mathtt{data}}+t_{\mathtt{link}}+t_{\mathtt{
barrier}}+t_{\mathtt{aggr}})$, where $L$ is the number of Transformer layers, $t_{\mathtt{data}}$ is the cumulative data transfer latency, $t_{\mathtt{link}}$ is the cumulative link latency, $t_{\mathtt{barrier}}$ is the cumulative barrier latency during aggregation, and $t_{\mathtt{aggr}}$ is the cumulative latency for aggregation calculation. Here we ignore $t_{\mathtt{aggr}}$ as it takes only 0.1 ms and thus negligible compared to other factors.

###### Proposition 1 .

The bottleneck in allreduce is not network bandwidth, but link latency.

###### Proof.

The data transfer latency $t_{\mathtt{data}}=2\sum_{\{i\rightarrow j\}\in\mathcal{P}_{h}}\frac{32|{\bm{H}
}|}{B_{ij}}$ depends on the size $32|{\bm{H}}|$ of the data being transmitted and the bandwidth $B_{ij}$ of the links in the path $\mathcal{P}_{h}$, here $\mathcal{P}_{h}$ is an index sequence from device $h$ to the master device. For example, in the case of Llama 2-70B with a hidden size $|{\bm{H}}|=8192$ and a network bandwidth of 300 Mbps, the data transfer latency is only $t_{\mathtt{data}}=3.4$ ms, which is negligible compared to other latencies. In addition, experiment results in Figure 5 show that increasing the network bandwidth does not significantly reduce token latency, further confirming that data transfer and network bandwidth is not the bottleneck.

The link latency $t_{\mathtt{link}}$, which is often neglected, emerges as the main issue. For example, the path from device $h_{2}$ to $h_{1}$ via $h_{8}$ follows the route $h_{2}\rightarrow r_{2}\rightarrow r_{9}\rightarrow r_{8}\rightarrow h_{8}
\rightarrow r_{8}\rightarrow r_{9}\rightarrow r_{1}\rightarrow h_{1}$, resulting in a total link latency of $16\tau$, where $\tau$ is the per-hop link latency. To isolate the impact of

Figure: Figure 3: Impact of link latency $\tau$.
Refer to caption: x5.png

delays during data aggregation. Given the assumption that all devices have equal computing power and network links have equal latencies, the barrier latency can be approximated as negligible:

$$ $t_{\mathtt{barrier}}=\max\{\sum_{(i\rightarrow j)\in\mathcal{P}}\frac{32|{\bm{ H}}|}{B_{ij}},\forall\mathcal{P}\}-\min\{\sum_{(i\rightarrow j)\in\mathcal{P}} \frac{32|{\bm{H}}|}{B_{ij}},\forall\mathcal{P}\}\approx 0.$ (3) $$

Thus, link latency $t_{\mathtt{link}}$ emerges as the key factor in allreduce latency.
∎

###### Proposition 2 .

The star-based allreduce is more effective for TPI-LLM in high-latency networks.

Despite past criticism, the star-based allreduce, where workers push data directly to the master for aggregation and pull the result back (Chen et al., ), stands out as the best choice (see Appendix A.1 for a detailed proof). It has minimal hops ($8$), lowest link latency ($8\tau$), zero intermediate barriers, and avoids the single-point issue due to the small data size (256 KB per device), making it the preferred allreduce algorithm for TPI-LLM.

###### Assumption 1 .

###### Proposition 1 .

###### Proof.

###### Proposition 2 .

### 3.3 Sliding Window Memory Scheduling

Quantizations like FP16 and INT8 are common for NVIDIA GPUs with CUDA support, but most edge devices lack CUDA and prefer full precision for faster computation due to their general-purpose CPU design. As a result, while tensor parallelism helps distribute memory costs across devices, the memory load remains high. Thus, memory scheduling is still required to manage these loads.

We introduce a memory scheduler, which manages memory by dynamically loading and unloading model weights during inference, ensuring that only the necessary parts are kept in memory (see Appendix A.2 for potential use). The memory scheduler operates on a daemon thread to asynchronously handle memory operations. To maintain the peak memory footprint, it uses a sliding window and preloads weights for upcoming layers while unloading those that have been processed.

As mentioned in Section 3.1, each Transformer layer is divided into attention computing, allreduce, FFN computing, and allreduce. For simplicity, in Figure 4, we assume the delays for these stages and weight loading to be equal. In each time slot, the memory scheduler asynchronously loads weights for either an attention or FFN block. By overlapping weight loading with ongoing computations and communications, it hides the I/O latency associated with loading weights from disk. For example, in Figure 4, the memory scheduler loads one more block during each allreduce until the sliding window reaches its size. As computations and communications proceed, we ensure weights are always ready when needed, allowing for seamless inference without computational stalls.

Figure: Figure 4: An illustration of the sliding window memory scheduling. Blue blocks indicate the blocks currently executed, with numbered blocks for attention or FFN computing and unnumbered blocks for allreduce communication. Green blocks indicate loaded model weights. The dashed box represents the sliding window, with size 4 in this case.
Refer to caption: x6.png

Next, we provide the conditions for this mechanism to reach a steady state, under which all required weights are loaded before computation starts.

###### Proposition 3 ( Loose Steady Condition ) .

The memory scheduler reaches a steady state when the following condition is met:

$$ $t_{\mathtt{attn}}+t_{\mathtt{ffn}}+2t_{\mathtt{all_reduce}}\geq\tau_{\mathtt{ ffn}}+\tau_{\mathtt{attn}},$ (4) $$

and one of the following conditions is met:

$$ $\displaystyle l\cdot t_{\mathtt{attn}}+(l-1)\cdot t_{\mathtt{ffn}}+(2l-1)\cdot t _{\mathtt{all_reduce}}$ $\displaystyle\geq l\cdot\tau_{\mathtt{ffn}}+(l-1)\cdot\tau_{\mathtt{attn}},~{} \forall l\in\{1,\cdots,L\},$ (5) $\displaystyle(l-1)\cdot t_{\mathtt{attn}}+l\cdot t_{\mathtt{ffn}}+(2l-1)\cdot t _{\mathtt{all_reduce}}$ $\displaystyle\geq(l-1)\cdot\tau_{\mathtt{ffn}}+l\cdot\tau_{\mathtt{attn}},~{} \forall l\in\{1,\cdots,L\},$ (6) $$

where $t_{\mathtt{attn}}$ and $t_{\mathtt{ffn}}$ are times required for attention and FFN computation, $t_{\mathtt{all_reduce}}$ is the allreduce latency, $\tau_{\mathtt{ffn}}$ and $\tau_{\mathtt{attn}}$ are times required to load attention and FFN weights, and $L$ is the number of Transformer layers.

This condition is loose but a bit hard to assess, so we present a tighter, more intuitive condition.

###### Proposition 4 ( Tight Steady Condition ) .

$t_{\mathtt{attn}}+t_{\mathtt{all_reduce}}\geq\tau_{\mathtt{ffn}}$ and $t_{\mathtt{ffn}}+t_{\mathtt{all_reduce}}\geq\tau_{\mathtt{attn}}$.

The proofs can be found in Appendices A.3 and A.4. This conclusion is straightforward. If the previous block’s computation and allreduce time cover the current block’s weight loading time, the memory scheduler can fully hide the disk I/O latency. As an example, in Section 4.4, we use 4 laptops with Llama 2-7B, setting $p_{i}=0.25$ and $w=4$. We measured $t_{\mathtt{attn}}=11$ ms, $t_{\mathtt{ffn}}=17$ ms, $t_{\mathtt{all_reduce}}=14$ ms, $\tau_{\mathtt{attn}}=18$ ms, and $\tau_{\mathtt{ffn}}=30$ ms. While the tight steady condition is not met, the loose steady condition is met, allowing the memory scheduler to achieve steady state.

###### Proposition 5 ( Peak Memory Footprint ) .

If the memory scheduler reaches a steady state, the peak memory footprint of the master and worker can be expressed as

$$ $\displaystyle M_{\mathtt{master}}$ $\displaystyle=\gamma\times\begin{cases}hv+h,&\text{if }w=1\\ 2hv+h,&\text{if }w=2\\ 2hv+h+\left\lfloor\frac{w-2}{2}\right\rfloor\left(2(1+\frac{b}{a})h^{2}p_{i}+h \right)+\left\lfloor\frac{w-1}{2}\right\rfloor(3hsp_{i}+h),&\text{if }w\geq 3 \end{cases}$ (7) $\displaystyle M_{\mathtt{worker}}$ $\displaystyle=\gamma\times\left(\left\lfloor\frac{w}{2}\right\rfloor(2(1+\frac {b}{a})h^{2}p_{i}+h)+\left\lfloor\frac{w+1}{2}\right\rfloor(3hsp_{i}+h)\right),$ (8) $$

where $h$ is the hidden size, $v$ is the vocabulary size, $a$ is the number of attention heads, $b$ is the number of key-value heads, $s$ is the intermediate size, $p_{i}$ is the proportion of parameters handled by device $i$, $w$ is the memory window size, and $\gamma$ is a memory scaling factor.

The proof can be found in Appendix A.5. However, if a slow disk I/O disrupts the steady state, the memory scheduler will retain some FFN blocks in memory to reduce disk access frequency.

###### Proposition 6 ( Loose Steady Condition with Block Retention ) .

Let the memory scheduler retain one FFN block in memory every $T$ FFN blocks, the condition to reach a steady state is then

$$ $\displaystyle l\cdot t_{\mathtt{attn}}+l\cdot t_{\mathtt{ffn}}+2l\cdot t_{ \mathtt{all_reduce}}$ $\displaystyle\geq(l-\left\lceil\frac{l}{T}\right\rceil)\cdot\tau_{\mathtt{ffn} }+l\cdot\tau_{\mathtt{attn}},$ (9) $\displaystyle l\cdot t_{\mathtt{attn}}+(l-1)\cdot t_{\mathtt{ffn}}+(2l-1)\cdot t _{\mathtt{all_reduce}}$ $\displaystyle\geq(l-\left\lceil\frac{l}{T}\right\rceil)\cdot\tau_{\mathtt{ffn} }+(l-1)\cdot\tau_{\mathtt{attn}}.$ (10) $$

The proof can be found in Appendix A.6. By setting an appropriate $T$, idle memory can help the scheduler reach a steady state, thus achieving a tradeoff between memory use and inference speed.

###### Proposition 3 ( Loose Steady Condition ) .

###### Proposition 4 ( Tight Steady Condition ) .

###### Proposition 5 ( Peak Memory Footprint ) .

###### Proposition 6 ( Loose Steady Condition with Block Retention ) .

## 4 Experiments

Prototype and Testbed. We implemented the prototype of TPI-LLM(^1^11Open available at: [https://anonymous.4open.science/r/tpi-llm](https://anonymous.4open.science/r/tpi-llm).) with 3K LoC using PyTorch and Transformers to provide flexible support for various sizes and versions of pretrained LLMs. Our testbed, illustrated in Appendix A.7, was built upon Klonet (Ma et al., ) to create an edge network environment, emulating realistic conditions with configurable properties like network topology, bandwidth, and latency. By default, 8 edge devices were emulated on 2 Intel Xeon Gold 5220R CPUs, each limited to 8 logical cores, 8 GB of memory, and 4 GB of swap. Network bandwidth between devices was set to 300 Mbps with a 1 ms latency.

Models. The inference speed of TPI-LLM is significantly affected by the model architecture. Deeper layers, more parameters, larger hidden sizes, and more attention heads increase the computational latency. Additionally, deeper layers result in more allreduce communications, and a larger hidden size leads to greater traffic. We tested with various models of different sizes, including Llama 2-3B/7B/13B/70B, Llama 3-8B/70B, and Yi-34B. See Appendix A.8 for their configuration details.

### 4.1 Overview of tpi-llm performance

Fit 70B LLMs into edge devices and run in high efficiency. We tested the performance of TPI-LLM with a focus on 3 key metrics: time-to-first-token (TTFT), token latency, and peak memory footprint per device. The memory window size is set to 2 by default. As shown in Table 1, without the memory scheduler, the full weights are loaded into the memory at once. Despite that these weights have been distributed across multiple devices, the memory is still insufficient for larger models like Yi-34B and Llama 2/3/3.1-70B. Instead, enabling our memory scheduler significantly reduces the peak memory footprint, allowing larger models to run efficiently. For example, the Llama 2-70B model requires just 3.1 GB of memory per device, and the Llama 3.1-70B model fits within device limits. The results are summarized in Table 1.

**Table 1: The TTFT, token latency, and peak memory footprint per device of TPI-LLM.**
| Model (FP32) | Memory Scheduler Disabled | Memory Scheduler Enabled |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| TTFT | Latency | Memory | TTFT | Latency | Memory |  |
| Llama 2-3B | 2.3 s | 1.0 s/token | 2.8 GB | 2.0 s | 1.9 s/token | 1.4 GB |
| Llama 2-7B | 3.1 s | 1.2 s/token | 4.5 GB | 3.0 s | 2.6 s/token | 1.7 GB |
| Llama 2-13B | 5.1 s | 1.9 s/token | 8.1 GB | 5.8 s | 2.9 s/token | 2.1 GB |
| Llama 2-70B | OOM | OOM | 34.9 GB | 29.4 s | 26.1 s/token | 3.1 GB |
| Llama 3.1-8B | 4.5 s | 1.5 s/token | 8.5 GB | 4.5 s | 4.3 s/token | 5.4 GB |
| Llama 3.1-70B | OOM | OOM | 42.3 GB | 32.9 s | 29.9 s/token | 11.3 GB |
| Yi-34B | OOM | OOM | 20.4 GB | 15.7 s | 13.7 s/token | 4.9 GB |

**Table 2: Peak memory footprint per device with the memory window size set to 2.**
|  | Memory Scheduler Disabled (GB) | Memory Scheduler Enabled (GB) |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Model (FP32) | $N=2$ | $N=4$ | $N=6$ | $N=8$ | $N=2$ | $N=4$ | $N=6$ | $N=8$ |
| Llama 2-3B | 7.3 | 4.3 | 3.2 | 2.8 | 1.5 | 1.4 | 1.4 | 1.4 |
| Llama 2-7B | 13.7 | 7.7 | 5.5 | 4.5 | 2.0 | 1.8 | 1.7 | 1.7 |
| Llama 2-13B | 25.7 | 13.9 | 9.8 | 8.1 | 2.3 | 2.2 | 2.2 | 2.1 |
| Llama 2-70B | 130 | 66.6 | 46.6 | 34.9 | 3.7 | 3.3 | 3.3 | 3.1 |
| Llama 3.1-8B | 18.4 | 11.8 | 9.4 | 8.5 | 5.9 | 5.6 | 5.5 | 5.4 |
| Llama 3.1-70B | 137.7 | 74.0 | 51.1 | 42.3 | 10.8 | 10.5 | 11.4 | 11.3 |
| Yi-34B | 67 | 36.4 | 23.9 | 20.4 | 5.0 | 5.0 | 5.0 | 4.9 |

No need for dozens of devices, one or two are enough to run 70B models. We used 8 devices by default, but can fewer devices run 70B-scale models? Table 2 gives detailed peak memory footprints with varying number of devices. Without the memory scheduler, full weights are loaded onto the devices, and with fewer devices, the memory load increases. For instance, using only 2 devices limits users to smaller models, like those between 3B and 7B. However, with the memory scheduler enabled, only a few layers’ weights are loaded and distributed across devices. This allows even larger models, such as 70B, to run smoothly on just 2 devices. Appendix A.9 shows the case with a memory window size of 4, which requires slightly more memory but faster speed. The peak memory footprint in TPI-LLM is primarily determined by the product of vocabulary size and hidden size, which is detailed in equation (7) and can be further reduced in our future work.

### 4.2 Scaling over varying edge conditions

Computation remains the bottleneck, not network bandwidth. In this experiment, we examined the token latency of TPI-LLM under different edge conditions, the results are shown in Figure 5. As expected, increasing the number of devices reduces the computing load on each device, significantly lowering token latency, and more CPU cores also contribute to a reduced latency. Instead, a limited network bandwidth was no longer a bottleneck, boosting it from 300 Mbps to 1 Gbps had little effect on latency due to the tiny data size (only 256 KB) during each allreduce. Thus, the main bottleneck remains in the computation, which our future work should focus on.

Figure: Figure 5: Token latency over varying number of devices, CPU cores, and network bandwidth on Llama 2-70B.
Refer to caption: x7.png

### 4.3 Comparison with benchmarks

Figure: Figure 6: Comparison of TPI-LLM with three benchmarks.
Refer to caption: x8.png

We compared the TPI-LLM with 3 benchmarks: (a) Standalone: LLM inference is executed only on a single edge device using Transformers (Wolf et al., ). (b) Model Parallelism (MP): Since only one user is served at a time, the pipeline parallelism (Zhang et al., ; Mei et al., ; Borzunov et al., ) degrades to the model parallelism, where different layer sequences are distributed across multiple devices. Each device computes its layers and passes the result to the next device until the entire inference is complete. (c) Galaxy (Ye et al., ) combines tensor and sequence parallelism and overlaps communication and computation to accelerate inference. They all run in FP32 mode.

Run larger models with lower latency and memory usage. As shown in Figure 6, a limited memory on a single device makes it challenging to run even 3B models in a standalone mode. MP addresses this by the collaboration of 8 devices, allowing models up to 13B, but suffers from high latency due to pipeline bubbles. Galaxy tries to reduce such latency by combining tensor and sequence parallelism. However, in Section 3.2, we concluded that the network bandwidth was no longer the issue, and the real problem is the link latency. Galaxy’s use of a ring algorithm for reducescatter and allgather forces each link to be traversed at least 14 times. This causes high link latency and outweighs the benefits of parallel computing, ultimately resulting in a higher token latency than MP. In contrast, TPI-LLM adopts a star-based allreduce algorithm, minimizing hops and cumulative link latency. Combined with the blocking-free memory scheduler, TPI-LLM delivers significantly lower token latency and memory footprint, even with larger 70B models.

### 4.4 Real case study

In this study, we used 4 laptops with different CPU architectures and memory capacities, connected via a local Wi-Fi router. The testbed and configurations are detailed in Appendix A.10. Macbook Pro was used by default. Due to the lack of CUDA, all computations were performed in full precision. As shown in Table 3, Transformers loaded the entire model into the CPU memory, and when memory was insufficient, the operating system offloaded data to the swap. This frequent swap exchange significantly increased TTFT and token latency, even for smaller 3B models. As the model size grows, the swap space overflowed, finally leading to OOM errors. As a more efficient alternative, Accelerate (Gugger et al., ) instantly loads layer weights only when required for the computation and reduces unnecessary data I/O. While it speeds up inference, due to implementation flaws on disk offloading, it still requires loading full weights before splitting and offloading them to disk. This results in OOM errors when the model size reaches 13B.

TPI-LLM stands out in TTFT, token latency, and model size. Our memory scheduler (Transformers+MS) outperforms Transformers and Accelerate in both TTFT and token latency across all model sizes. This is because our memory scheduler employs a sliding window mechanism, where a daemon thread asynchronously preloads the weights needed for upcoming computations. By overlapping data I/O with computations and communications, the scheduler avoids delays caused by disk I/O blocks, ensuring smoother and faster inference. To further speed up inference, we integrate the computing power of 4 laptops to serve TPI-LLM. By distributing the computational load across 4 laptops, the reduction in computing time far exceeds communication delays, so both TTFT and token latency are further reduced. The results from using 3 laptops are shown in Appendix A.11, indicating a slightly higher latency due to reduced parallelism.

**Table 3: Comparison of Transformers, Accelerate, Transformers+MS, and TPI-LLM on 4 laptops.**
| Model (FP32) | Transformers | Accelerate | Transformers + MS | TPI-LLM |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| TTFT<br>(s) | Latency<br>(s/token) | TTFT<br>(s) | Latency<br>(s/token) | TTFT<br>(s) | Latency<br>(s/token) | TTFT<br>(s) | Latency<br>(s/token) |  |
| Llama 2-3B | 61 | 30 | 24 | 16 | 4 | 3 | 2.5 | 2 |
| Llama 2-7B | 115 | 56 | 30 | 26 | 13 | 8 | 6 | 5 |
| Llama 2-13B | OOM | OOM | OOM | OOM | 22 | 18 | 10 | 9 |
| Llama 3.1-8B | 133 | 65 | 37 | 31 | 20 | 12 | 11 | 8 |
| Yi-34B | OOM | OOM | OOM | OOM | 185 | 55 | 33 | 29 |

## 5 Conclusion

In this paper, we concluded that tensor parallelism can be more effective than pipeline parallelism on low-resource devices, and presented a compute- and memory-efficient tensor parallel inference system, named TPI-LLM, to serve 70B-scale LLMs. TPI-LLM is designed with user prompt and generated sequence privacy in mind, by keeping sensitive raw data local in the users’ devices. It leverages a sliding window memory scheduler to dynamically manage layer weights during inference with disk I/O latency overlapped by onging computations and communications, allowing larger models to run smoothly on devices with very limited memory. Our analysis showed that link latency, not bandwidth, emerges as the main issue, so TPI-LLM implements a star-based allreduce algorithm, rather than the commonly used ring- and tree-based algorithms. Through extensive experiments on emulated and real testbeds, TPI-LLM demonstrated significantly lower TTFT, token latency, and peak memory footprint compared to Transformers, Accelerate, Galaxy, and enabled serving larger-scale LLMs such as Yi-34B and Llama 2/3/3.1-70B on low-memory devices.

## Reproducibility

We have made efforts to ensure reproducibility by providing the source code at [https://anonymous.4open.science/r/tpi-llm](https://anonymous.4open.science/r/tpi-llm), with a detailed README for guidance included. To ease the use, a prebuilt Docker image is also provided. Key experimental setups are given in Section 4 of the paper.

## Appendix A Appendix

### A.1 Proof of proposition 2

In conventional data parallel systems, each device sends several gigabytes of data, putting significant pressure on network bandwidth. This makes data transfer latency a major concern, while link latency becomes negligible. Then, tree and ring-based algorithms are introduced to optimize the data transfer. However, they do not apply to our case. In TPI-LLM, each device only sends a small amount of data, usually just tens of kilobytes. This tiny data size does not strain the network, so data transfer latency is minimal. Instead, in edge networks where wireless communication causes higher transmission delays, link latency becomes more significant than data transfer latency. As a result, the commonly used tree and ring-based allreduce algorithms are less effective.

Let us consider 1 master and 2 workers connected via a router. In Figure 7, we compare the traffic models of star, tree, and ring-based algorithms. In star-based allreduce, worker 1 sends data directly to the master via the router, and the allreduce latency (includes reduce and broadcast) is $t_{\mathtt{star}}=2(t_{\mathtt{data}}+t_{\mathtt{link}})+t_{\mathtt{barrier}}+
t_{\mathtt{aggr}}$. In this model, the router only forwards data packets.

Figure: Figure 7: Comparison of traffic models for star, tree, and ring-based allreduce algorithms.
Refer to caption: x9.png

In tree-based allreduce, data from worker 1 must first go through worker 2 before reaching the master, so there are 2 hops involved. In this process, worker 1 sends its data to worker 2, which aggregates it and forwards the result to the master. Once the global aggregation is complete, the final result is broadcast back to all workers. The total time for this process is $t_{\mathtt{tree}}=3t_{\mathtt{data}}+4t_{\mathtt{link}}+2t_{\mathtt{barrier}}+
2t_{\mathtt{aggr}}$.

In ring-based allreduce, each device communicates directly with its neighbors in a ring topology. Data is divided and sent in a sequence around the ring, with each device receiving, aggregating, and passing the data to the next device. Unlike star or tree-based methods, there is no central device, and data flows continuously between the devices. The total time for the ring-based allreduce is $t_{\mathtt{ring}}=\frac{4}{3}t_{\mathtt{data}}+4t_{\mathtt{link}}+3t_{\mathtt{
barrier}}+\frac{2}{3}t_{\mathtt{aggr}}$.

Assume that all devices are homogeneous, i.e., $t_{\mathtt{barrier}}\approx 0$, and $t_{\mathtt{data}}\approx 0,t_{\mathtt{aggr}}\approx 0$ because the data size is very small. Then we have latencies simplified as follows:

$$ $t_{\mathtt{star}}=2t_{\mathtt{link}}<t_{\mathtt{tree}}=t_{\mathtt{ring}}=4t_{ \mathtt{link}}.$ (11) $$

Thus, the star-based allreduce is the most efficient method because it minimizes link latency.

### A.2 A simple-to-use memory scheduler

In our implementation, a context manager is used to ensure that the required block weights are loaded correctly and unload the used weights to free up memory for subsequent blocks. This simplifies the deployment of large-scale LLMs on low-memory edge devices, requiring just one additional line of code:

### A.3 Proof of proposition 3

We start with the first attention block and end with the final FFN block.

Time slot 1 (attention computation): In this initialization step, ${\bm{W}}_{\mathtt{attn}}^{1}$ must be loaded before computing the first attention block, taking $\tau_{\mathtt{attn}}+t_{\mathtt{attn}}$. During the computation time $t_{\mathtt{attn}}$, the next FFN weights, ${\bm{W}}_{\mathtt{ffn}}^{1}$, are loading in parallel.

Time slot 2 (allreduce): The attention block is followed by allreduce communication, which takes $t_{\mathtt{all_reduce}}$, with the next FFN weights, ${\bm{W}}_{\mathtt{ffn}}^{1}$, loading in parallel.

Time slot 3 (FFN computation): By this time, the FFN weights ${\bm{W}}_{\mathtt{ffn}}^{1}$ should be fully loaded. If not, the computation must wait for loading to complete. Let $t^{\prime}=t_{\mathtt{attn}}+t_{\mathtt{all_reduce}}-\tau_{\mathtt{ffn}}$, if $t^{\prime}\geq 0$, no blocking occurs; otherwise, the computation is delayed by $|t^{\prime}|$. Once loaded, compute the FFN block in $t_{\mathtt{ffn}}$.

During this time slot, the waiting, computation of the current FFN block and the weight loading of the next attention block occur simultaneously. By the time the current FFN block finishes, the next attention block’s weights ${\bm{W}}_{\mathtt{attn}}^{2}$ have been loading for $\max\{0,t_{\mathtt{attn}}+t_{\mathtt{all_reduce}}-\tau_{\mathtt{ffn}}\}+t_{
\mathtt{ffn}}$.

Time slot 4 (allreduce): The FFN block is followed by allreduce communication, which takes $t_{\mathtt{all_reduce}}$, with the next attention weights, ${\bm{W}}_{\mathtt{attn}}^{2}$, loading in parallel.

Time slot 5 (attention computation): Ensure that the attention weights ${\bm{W}}_{\mathtt{attn}}^{2}$ are fully loaded. Let $t^{\prime}=\max\{0,t_{\mathtt{attn}}+t_{\mathtt{all_reduce}}-\tau_{\mathtt{
ffn}}\}+t_{\mathtt{ffn}}+t_{\mathtt{all_reduce}}-\tau_{\mathtt{attn}}$. If $t^{\prime}\geq 0$, the computation proceeds without blocking. Then, ${\bm{W}}_{\mathtt{attn}}^{2}$ is computed in $t_{\mathtt{attn}}$, and the next FFN weights ${\bm{W}}_{\mathtt{ffn}}^{2}$ have been loading for $\max\{0,\max\{0,t_{\mathtt{attn}}+t_{\mathtt{all_reduce}}-\tau_{\mathtt{ffn}}
\}+t_{\mathtt{ffn}}+t_{\mathtt{all_reduce}}-\tau_{\mathtt{attn}}\}+t_{\mathtt
{attn}}$.

Time slot 6 (allreduce): The allreduce communication takes $t_{\mathtt{all_reduce}}$, while the next FFN weights ${\bm{W}}_{\mathtt{ffn}}^{2}$ are loading in parallel.

Time slot 7 (FFN computation): Ensure that the FFN weights ${\bm{W}}_{\mathtt{ffn}}^{2}$ are fully loaded. Let $t^{\prime}=\max\{0,\max\{0,t_{\mathtt{attn}}+t_{\mathtt{all_reduce}}-\tau_{
\mathtt{ffn}}\}+t_{\mathtt{ffn}}+t_{\mathtt{all_reduce}}-\tau_{\mathtt{attn}}
\}+t_{\mathtt{attn}}+t_{\mathtt{all_reduce}}-\tau_{\mathtt{ffn}}$. If $t^{\prime}\geq 0$, the computation proceeds without blocking.

This process repeats, until the generation task is finished.

For the system to reach a steady state where computation is not blocked by weight loading at any time, the following conditions must hold.

Case 1: $t_{\mathtt{attn}}+t_{\mathtt{all_reduce}}-\tau_{\mathtt{ffn}}\geq 0$.

$$ Time slot 3 ( $l=1$ ): $\displaystyle t_{\mathtt{attn}}+t_{\mathtt{all_reduce}}-\tau_{\mathtt{ffn}} \geq 0,$ (12) Time slot 5 ( $l=1$ ): $\displaystyle t_{\mathtt{attn}}+t_{\mathtt{ffn}}+2t_{\mathtt{all_reduce}}- \tau_{\mathtt{ffn}}-\tau_{\mathtt{attn}}\geq 0,$ (13) Time slot 7 ( $l=2$ ): $\displaystyle 2t_{\mathtt{attn}}+t_{\mathtt{ffn}}+3t_{\mathtt{all_reduce}}-2 \tau_{\mathtt{ffn}}-\tau_{\mathtt{attn}}\geq 0,$ (14) Time slot 9 ( $l=2$ ): $\displaystyle 2t_{\mathtt{attn}}+2t_{\mathtt{ffn}}+4t_{\mathtt{all_reduce}}-2 \tau_{\mathtt{ffn}}-2\tau_{\mathtt{attn}}\geq 0.$ (15) $$

We repeat these conditions and derive the following patterns.

$$ $\displaystyle t_{\mathtt{attn}}+t_{\mathtt{ffn}}+2t_{\mathtt{all_reduce}}$ $\displaystyle\geq\tau_{\mathtt{ffn}}+\tau_{\mathtt{attn}},$ (16) $\displaystyle l\cdot t_{\mathtt{attn}}+(l-1)\cdot t_{\mathtt{ffn}}+(2l-1)\cdot t _{\mathtt{all_reduce}}$ $\displaystyle\geq l\cdot\tau_{\mathtt{ffn}}+(l-1)\cdot\tau_{\mathtt{attn}}.$ (17) $$

Case 2: $t_{\mathtt{attn}}+t_{\mathtt{all_reduce}}-\tau_{\mathtt{ffn}}<0$.

$$ Time slot 3 ( $l=1$ ): $\displaystyle t_{\mathtt{attn}}+t_{\mathtt{all_reduce}}-\tau_{\mathtt{ffn}}<0,$ (18) Time slot 5 ( $l=1$ ): $\displaystyle t_{\mathtt{ffn}}+t_{\mathtt{all_reduce}}-\tau_{\mathtt{attn}} \geq 0,$ (19) Time slot 7 ( $l=2$ ): $\displaystyle t_{\mathtt{attn}}+t_{\mathtt{ffn}}+2t_{\mathtt{all_reduce}}- \tau_{\mathtt{attn}}-\tau_{\mathtt{ffn}}\geq 0,$ (20) Time slot 9 ( $l=2$ ): $\displaystyle t_{\mathtt{attn}}+2t_{\mathtt{ffn}}+3t_{\mathtt{all_reduce}}-2 \tau_{\mathtt{attn}}-\tau_{\mathtt{ffn}}\geq 0,$ (21) Time slot 11 ( $l=3$ ): $\displaystyle 2t_{\mathtt{attn}}+2t_{\mathtt{ffn}}+4t_{\mathtt{all_reduce}}-2 \tau_{\mathtt{attn}}-2\tau_{\mathtt{ffn}}\geq 0.$ (22) $$

Similarly, repeat these conditions and derive the following patterns.

$$ $\displaystyle t_{\mathtt{attn}}+t_{\mathtt{ffn}}+2t_{\mathtt{all_reduce}}$ $\displaystyle\geq\tau_{\mathtt{ffn}}+\tau_{\mathtt{attn}},$ (23) $\displaystyle(l-1)\cdot t_{\mathtt{attn}}+l\cdot t_{\mathtt{ffn}}+(2l-1)\cdot t _{\mathtt{all_reduce}}$ $\displaystyle\geq(l-1)\cdot\tau_{\mathtt{ffn}}+l\cdot\tau_{\mathtt{attn}}.$ (24) $$

Thus, the proposition is proved.

### A.4 Proof of proposition 4

Let $\alpha=l\cdot t_{\mathtt{attn}}+(l-1)\cdot t_{\mathtt{ffn}}+(2l-1)\cdot t_{
\mathtt{all_reduce}}-l\cdot\tau_{\mathtt{ffn}}-(l-1)\cdot\tau_{\mathtt{attn}}>0$, and we derive the following inequality from inequality (16):

$$ $l\cdot t_{\mathtt{attn}}+l\cdot t_{\mathtt{ffn}}+2l\cdot t_{\mathtt{all_ reduce}}-l\cdot\tau_{\mathtt{ffn}}-l\cdot\tau_{\mathtt{attn}}>0.$ (25) $$

By substituting $\alpha$ into this inequality, we have $\alpha+t_{\mathtt{ffn}}+t_{\mathtt{all_reduce}}-\tau_{\mathtt{attn}}>0.$
Let $\alpha>0>\tau_{\mathtt{attn}}-t_{\mathtt{ffn}}-t_{\mathtt{all_reduce}}$, we obtain the first condition:

$$ $t_{\mathtt{ffn}}+t_{\mathtt{all_reduce}}>\tau_{\mathtt{attn}}.$ (26) $$

Let $\beta=t_{\mathtt{ffn}}+t_{\mathtt{all_reduce}}-\tau_{\mathtt{attn}}>0$, and substitute $\beta$ into inequality (16), then we have $\beta+t_{\mathtt{attn}}+t_{\mathtt{all_reduce}}-\tau_{\mathtt{ffn}}>0$. Let $\beta>0>\tau_{\mathtt{ffn}}-t_{\mathtt{attn}}-t_{\mathtt{all_reduce}}$, we obtain the second condition:

$$ $t_{\mathtt{attn}}+t_{\mathtt{all_reduce}}>\tau_{\mathtt{ffn}}.$ (27) $$

Thus, the proposition is proved.

### A.5 Proof of proposition 5

In this section, we analyze the peak memory footprint on both the master and worker nodes to estimate the largest model size that our memory scheduler can handle.

Let us use the Llama model as an example, assume the vocabulary size is $v$, hidden size is $h$, number of attention heads is $a$, number of key-value heads is $b$, and intermediate size is $s$. Let ${\bm{p}}=[p_{1},p_{2},\cdots,p_{n}]$ be a vector representing the proportion of parameters handled by $n$ devices, and $w$ be the window size of the memory scheduler. Following the block definition in Figure 2, the parameter counts for each block are detailed in Table 4:

**Table 4: Parameter counts for the main blocks (e.g., $n=4$, $p_{i}=0.25$, Llama 2-7B).**
| Block | Parameters | Block Size |
| --- | --- | --- |
| Preprocess | $hv$ | 500 MB |
| Attention | $2(a+b)h^{2}p_{i}/a+h$ | 64 MB |
| FFN | $3hsp_{i}+h$ | 129 MB |
| Postprocess | $hv+h$ | 500 MB |

The memory footprint is affected by parameters, activation storage, temporary tensors, memory management, and caching, making precise quantification challenging. To estimate peak memory, we apply an empirical rule: multiply the parameter size by a scaling factor $\gamma$.

Figure: Figure 8: Illustration of the memory window at the peak memory footprint.
Refer to caption: x10.png

From the memory window at the peak memory footprint shown in Figure 8, we can derive the following equations.

$$ $M_{\mathtt{master}}=\gamma\times\begin{cases}hv+h,&\text{if }w=1\\ 2hv+h,&\text{if }w=2\\ 2hv+h+\left\lfloor\frac{w-2}{2}\right\rfloor\left(2(1+\frac{b}{a})h^{2}p_{i}+h \right)+\left\lfloor\frac{w-1}{2}\right\rfloor(3hsp_{i}+h),&\text{if }w\geq 3 \end{cases}$ $$

For any worker node, the memory footprint does not include the preprocess and postprocess blocks. Therefore, the peak memory footprint $M_{\mathtt{worker}}$ can be expressed as:

$$ $M_{\mathtt{worker}}=\gamma\times\left(\left\lfloor\frac{w}{2}\right\rfloor(2(1 +\frac{b}{a})h^{2}p_{i}+h)+\left\lfloor\frac{w+1}{2}\right\rfloor(3hsp_{i}+h) \right).$ $$

Thus, the proposition is proved.

### A.6 Proof of proposition 6

When the memory scheduler reaches a steady state, the overlap between computation, communication, and disk I/O is optimized, ensuring that weights are always pre-loaded before they are needed for computations. However, if disk I/O becomes a bottleneck and disrupts the steady state (e.g., due to high disk latency), the scheduler must adapt by selectively retaining certain blocks in memory to reduce disk access frequency.

In our preliminary experiments, we measured $t_{\mathtt{attn}}=11$ ms, $t_{\mathtt{ffn}}=17$ ms, $\tau_{\mathtt{attn}}=18$ ms, $\tau_{\mathtt{ffn}}=30$ ms, and observed that FFN blocks generally exhibit higher computation and weight loading latency. By retaining some FFN blocks in memory, we can reduce the need to reload large weights.

Let the memory scheduler retain one FFN block in memory every $T$ FFN blocks, and

$$ $\mathbb{I}_{\{l=1+kT\}}=\begin{cases}1,&\text{if }l=1+kT\text{ and }k\in \mathbb{Z}{\geq 0},\\ 0,&\text{otherwise}.\end{cases}$ $$

Similar to the analysis in Appendix A.3, we have

$$ Time slot 3 ( $l=1$ ): $\displaystyle t_{\mathtt{attn}}+t_{\mathtt{all_reduce}}-(1-\mathbb{I}_{\{l=1+ kT\}})\tau_{\mathtt{ffn}}\geq 0,$ Time slot 5 ( $l=1$ ): $\displaystyle t_{\mathtt{attn}}+t_{\mathtt{ffn}}+2t_{\mathtt{all_reduce}}-(1- \mathbb{I}_{\{l=1+kT\}})\tau_{\mathtt{ffn}}-\tau_{\mathtt{attn}}\geq 0,$ Time slot 7 ( $l=2$ ): $\displaystyle 2t_{\mathtt{attn}}+t_{\mathtt{ffn}}+3t_{\mathtt{all_reduce}}- \sum_{i=1}^{2}(1-\mathbb{I}_{\{i=1+kT\}})\tau_{\mathtt{ffn}}-\tau_{\mathtt{ attn}}\geq 0,$ Time slot 9 ( $l=2$ ): $\displaystyle 2t_{\mathtt{attn}}+2t_{\mathtt{ffn}}+4t_{\mathtt{all_reduce}}- \sum_{i=1}^{2}(1-\mathbb{I}_{\{i=1+kT\}})\tau_{\mathtt{ffn}}-2\tau_{\mathtt{ attn}}\geq 0,$ Time slot 11 ( $l=3$ ): $\displaystyle 3t_{\mathtt{attn}}+2t_{\mathtt{ffn}}+5t_{\mathtt{all_reduce}}- \sum_{i=1}^{3}(1-\mathbb{I}_{\{i=1+kT\}})\tau_{\mathtt{ffn}}-2\tau_{\mathtt{ attn}}\geq 0.$ $$

By repeating these conditions, we derive the following patterns:

$$ $\displaystyle l\cdot t_{\mathtt{attn}}+l\cdot t_{\mathtt{ffn}}+2l\cdot t_{ \mathtt{all_reduce}}-\sum_{i=1}^{l}(1-\mathbb{I}_{\{i=1+kT\}})\tau_{\mathtt{ ffn}}-l\cdot\tau_{\mathtt{attn}}$ $\displaystyle\geq 0,$ $\displaystyle l\cdot t_{\mathtt{attn}}+(l-1)\cdot t_{\mathtt{ffn}}+(2l-1)\cdot t _{\mathtt{all_reduce}}-\sum_{i=1}^{l}(1-\mathbb{I}_{\{i=1+kT\}})\tau_{\mathtt {ffn}}-(l-1)\cdot\tau_{\mathtt{attn}}$ $\displaystyle\geq 0.$ $$

Since $\sum_{i=1}^{l}\mathbb{I}_{\{i=1+kT\}}=\left\lceil\frac{l}{T}\right\rceil$, we have

$$ $\displaystyle l\cdot t_{\mathtt{attn}}+l\cdot t_{\mathtt{ffn}}+2l\cdot t_{ \mathtt{all_reduce}}$ $\displaystyle\geq(l-\left\lceil\frac{l}{T}\right\rceil)\cdot\tau_{\mathtt{ffn} }+l\cdot\tau_{\mathtt{attn}},$ $\displaystyle l\cdot t_{\mathtt{attn}}+(l-1)\cdot t_{\mathtt{ffn}}+(2l-1)\cdot t _{\mathtt{all_reduce}}$ $\displaystyle\geq(l-\left\lceil\frac{l}{T}\right\rceil)\cdot\tau_{\mathtt{ffn} }+(l-1)\cdot\tau_{\mathtt{attn}}.$ $$

Thus, the proposition is proved.

### A.7 Klonet testbed

One of our testbed, as shown in Figure 9, was built upon Klonet (Ma et al., ) to create an edge network environment. Klonet is a network emulation platform designed to support the development and testing of new network protocols and applications in a realistic environment. It can emulate various network scenarios, such as wireless, mobile, satellite, and optical networks, and provide fine-grained control over the network parameters, such as bandwidth, delay, jitter, and packet loss. It can also integrate with real devices and applications, such as routers, switches, sensors, and smartphones, to create hybrid network experiments. Klonet is based on the Linux operating system and uses virtualization and containerization technologies to create isolated network nodes and links. It provides both GUI and CLI to help users configure and manage their network experiments.

Figure: Figure 9: Testbed built upon Klonet.
Refer to caption: extracted/5892352/figures/testbed.jpg

This testbed includes 8 user devices (devices 1 to 8), 8 home gateways (routers 1 to 8), and 1 core router (router 9). User devices connect to their home gateways via wired or wireless connections, and these home gateways are interconnected through routers or switches in the edge network. This topology reflects real-world household network interconnections. In addition, the CPU cores, memory, swap limits, bandwidth, and latency settings in Section 4 are based on measurements from the authors’ edge network.

### A.8 Configurations of the used models

**Table 5: Configurations of the used Llama models.**
| Model (FP32) | Layers | Params | Hidden Size | Heads | KV Heads | Required Mem |
| --- | --- | --- | --- | --- | --- | --- |
| Llama 2-3B | 26 | 3 billion | 3200 | 32 | – | 14 GB |
| Llama 2-7B | 32 | 7 billion | 4096 | 32 | – | 26 GB |
| Llama 2-13B | 40 | 13 billion | 5120 | 40 | – | 50 GB |
| Llama 2-70B | 80 | 70 billion | 8192 | 64 | 8 | 257 GB |
| Llama 3.1-8B | 32 | 8 billion | 4096 | 32 | 8 | 31 GB |
| Llama 3.1-70B | 80 | 70 billion | 8192 | 64 | 8 | 266 GB |
| Yi-34B | 60 | 34 billion | 7168 | 56 | 8 | 130 GB |

### A.9 Peak memory footprint with memory window size 4

**Table 6: Peak memory footprint per device with the memory window size set to 4.**
|  | Memory Scheduler Disabled (GB) | Memory Scheduler Enabled (GB) |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Model (FP32) | $N=2$ | $N=4$ | $N=6$ | $N=8$ | $N=2$ | $N=4$ | $N=6$ | $N=8$ |
| Llama 2-3B | 7.3 | 4.3 | 3.2 | 2.8 | 1.7 | 1.5 | 1.5 | 1.5 |
| Llama 2-7B | 13.7 | 7.7 | 5.5 | 4.5 | 2.4 | 2.1 | 1.8 | 1.8 |
| Llama 2-13B | 25.8 | 13.9 | 9.7 | 8.0 | 2.8 | 2.5 | 2.3 | 2.2 |
| Llama 2-70B | 129.9 | 66.5 | 46.7 | 35.0 | 4.5 | 3.1 | 3.1 | 3.1 |
| Llama 3.1-8B | 18.4 | 11.8 | 9.4 | 8.5 | 6.3 | 5.8 | 5.6 | 5.5 |
| Llama 3.1-70B | 137.8 | 74.0 | 51.4 | 42.5 | 10.8 | 10.5 | 11.5 | 11.4 |
| Yi-34B | 67 | 36.4 | 23.9 | 20.4 | 6.0 | 5.6 | 5.3 | 5.2 |

### A.10 Real testbed and configurations

The real testbed consists of 4 laptops, all connected via a local Wi-Fi router, as shown in Figure 10. Table 7 details the hardware and network configurations of these laptops. In this case study, the laptop in the lower right serves as the master, while the other three laptops act as workers. The workers are connected to the master, and the master is currently generating the output sequence. The generated sequence is identical to that of single-server inference.

Figure: Figure 10: A real testbed composed of 4 laptops connected via local Wi-Fi.
Refer to caption: extracted/5892352/figures/real-testbed.jpg

**Table 7: Hardware and network configurations of the laptops.**
| Device | CPU Model | Cores | Memory | Bandwidth | Latency | CUDA | Number |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Mac Pro | Apple M1 | 8 | 8 GB | 510 Mbps | 5 ms | No | 1 |
| Mac Air | Intel Core i5 | 4 | 8 GB | 320 Mbps | 7 ms | No | 1 |
| Dell | Intel i7-1165G7 | 8 | 16 GB | 610 Mbps | 3 ms | No | 2 |

### A.11 Case study with 3 laptops

In this case, only 3 out of the 4 laptops are used: one MacBook Pro, one MacBook Air, and one Dell laptop. The results are given in Table 8, indicating a slightly higher latency due to reduced parallelism.

**Table 8: Comparison of Transformers, Accelerate, Transformers+MS, and TPI-LLM on 3 laptops.**
| Model (FP32) | Transformers | Accelerate | Transformers + MS | TPI-LLM |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| TTFT<br>(s) | Latency<br>(s/token) | TTFT<br>(s) | Latency<br>(s/token) | TTFT<br>(s) | Latency<br>(s/token) | TTFT<br>(s) | Latency<br>(s/token) |  |
| Llama 2-3B | 61 | 30 | 24 | 16 | 4 | 3 | 3 | 2 |
| Llama 2-7B | 115 | 56 | 30 | 26 | 13 | 8 | 7 | 6 |
| Llama 2-13B | OOM | OOM | OOM | OOM | 22 | 18 | 14 | 12 |
| Llama 3.1-8B | 133 | 65 | 37 | 31 | 20 | 12 | 13 | 9 |
| Yi-34B | OOM | OOM | OOM | OOM | 185 | 55 | 48 | 41 |

## Peer Reviews

confidence: 5
contribution: 1
initial_rating: 1
presentation: 1
questions: 1. What is the use-case for this work if it will take minutes to hours to generate an output?
2. How do these results change if you have a faster edge device compared to a CPU?
soundness: 1
strengths: The paper aims to solve a very important problem, how to run very large models with billions of parameters on CPUs or machines with no CUDA.
summary: The paper introduces a technique to run 70B LLM on CPU based (edge) devices. The system uses a tensor parallel framework to distribute attention heads across multiple nodes. The authors then perform an All-Reduce latency analysis, claiming that latency, not bandwidth, is the main bottleneck for all-reduce. The authors then describe a sliding memory scheduler, followed by experiments where they show the performance of their system.
weaknesses: Thank you for submitting your work to ICLR. I believe the current version of the paper has many shortcomings which I will describe here in detail:

1. The paper needs thorough proof-reading. Some examples:
- by adaptively partitioning model--->a model or models
- with token latency increases-->increasing
- Write in Active voice to avoid odd sentence structures such as :
- "Constrained by the high link latency, a star-based allreduce algorithm is implemented"
- "We design a TPI-LLM"--->We design TPI-LLM
- "power collaborate"--->to collaborate
- "which dominate inference time"---> "what dominate"


2. I did not really understand the example on p.5 "For example, the path from device h2 to h1 via h8
follows the route h2 → r2 → r9 → r8 → h8 → r8 → r9 → r1 → h1, resulting in a total link latency of 16τ , where τ is the per-hop link latency"


3. The Sliding Window Memory Scheduling is very similar to PipeSwitch (https://www.usenix.org/conference/osdi20/presentation/bai). The only main difference being that you are swapping in/out from Disk to/from device memory. This is in many ways also similar to memory page to disk swapping.

4. Starting with the results for the swapping. Having an OOM is probably better than having 26.1s/token. For a 100 tokens output, you need to wait for roughly 30 minutes. This is an output of about 75 words as per OpenAI (https://help.openai.com/en/articles/4936856-what-are-tokens-and-how-to-count-them), i.e., one paragraph. Why would a user want to tolerate this? If the user chooses to use the fastest model (the Llama2-3B), they will wait for a bit less, about 3.3 minutes. I am not sure if there is a use-case for such a slow running LLM.

5. Moving to the networking results in 4.2, I think the authors are drawing the wrong conclusions. The computations are just super slow in this case that the network is not really a bottleneck in the computations. I think a better experiment would be to try the system with proper edge GPU/TPUs, e.g., Google's Corel, NVidia's Orin, or NVidia's Nano. Right now, I believe that what we are seeing is the result of just a very slow computation. You can already see that in a real world scenario, things are much worse in the real case study where the latency in Table-3 is multiple times compared to Table-1.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: When the author discusses opportunities, I have several points of confusion:

1. Why does the author believe that the communication proportion of tensor parallelism is high, but the overall inference time is reduced due to parallel computation? Is this conclusion drawn from the results in Figure 1-(b)? Has the author considered the synchronization issues among multiple edge devices in tensor parallelism?
  
2. Why is the total time of tensor parallelism less than 100% in Figure 1-(b)? Does the author want to use the pipeline parallelism as a baseline to illustrate the superiority of tensor parallelism among edge devices?
  
3. In Figure 1-(c), why is the memory footprint of each device in the TPI-LLM framework proposed in this paper the same? If tensor parallelism is used, it should decrease as the number of devices increases.
soundness: 2
strengths: This paper addresses the emerging challenge of deploying large language models (LLMs) on edge devices, which is a novel and increasingly relevant problem in the era of edge computing and privacy concerns. It proposes a new approach to tensor parallelism that is specifically tailored for low-resource environments, which is an original contribution to the field.

The authors combine concepts from distributed computing, memory management, and parallelism to create a system that is both memory and compute-efficient. The sliding window memory scheduler and the star-based allreduce algorithm are creative solutions that address specific pain points in edge device inference.

The work has significant implications for the future of edge computing, as it enables the deployment of LLMs on devices that were previously considered incapable due to resource constraints.
summary: The paper presents a novel tensor parallel inference system named TPI-LLM, designed to efficiently serve large-scale language models (LLMs) on low-resource edge devices. The system addresses the challenges of limited computing power, memory, and bandwidth on edge devices by introducing a sliding window memory scheduler that dynamically manages layer weights during inference, overlapping disk I/O latency with computation and communication. This approach allows larger models to run smoothly on memory-limited devices while keeping sensitive raw data local to the user's devices, enhancing privacy.

The key contributions of the paper are as follows:

1. Tensor Parallelism on Edge Devices: The paper argues for the effectiveness of tensor parallelism over pipeline parallelism on low-resource edge devices and presents TPI-LLM as a compute- and memory-efficient solution for serving 70B-scale models.
  
2. Sliding Window Memory Scheduler: A memory scheduler is introduced to asynchronously load and unload layer weights, which enables the inference of larger models on devices with limited memory by overlapping disk I/O with computations and communications.
  
3. Star-Based AllReduce Algorithm: The paper identifies link latency as the main issue in communication and implements a star-based allreduce algorithm to reduce latency, outperforming ring- and tree-based methods commonly used in high-latency networks.
weaknesses: I think that although this paper is a bold attempt at the edge of LLM serving, most of the solutions provided are based on the application of past works. Firstly, whether it is tensor parallelism, sliding window, or the star-based algorithm, they are all proposed in existing and relatively mature works. The author's approach to using these methods to optimize edge LLM serving is similar to theirs, hence the paper's contribution lacks a certain level of innovation.

Furthermore, I think there are some flaws in the author's logic when explaining the motivation and opportunities for the research. I only learned from the first sentence of the abstract that the significance of serving large models at the network edge lies in data privacy-preserving. In the first paragraph of the introduction, I learned that if edge devices must be used for LLM serving to ensure user privacy and security, then more edge devices will have to be used in a distributed collaborative serving due to the limitations of computing and storage resources. However, the title of the second paragraph is "Observations and Motivations," but the content only contains observations (which should only contain them without motivations). Therefore, I suggest that the author optimize the logic of explaining the research motivation and opportunities. The scope of privacy security issues is too large, so that the research motivation seems slightly pale. Can it be combined with some more specific downstream tasks or application scenarios?

Finally, the experimental results shown in Table 1 shows that the latency of serving large models on edge devices is much higher than that in the cloud, with TTFT reaching above the second level, and the throughput is far from comparable to that of the cloud. Does this indicate that the research motivation of the paper only considered privacy protection and neglected performance issues? Although the experimental results are already much better than Galaxy. This issue is also a huge challenge that all edge LLM serving inevitably faces.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 2
questions: - This technique might also be useful even in the cloud server settings, especially when there is not enough GPUs, can the sliding window memory management can help avoid the OOMs. Any thoughts in that direction or future recommendations maybe?

However, following are the concerns/questions.

### Major concerns
- The proposed framework has two major operational changes that are applied on the pre-trained LLM, that are 1) distribution of attention heads across nodes, ii) the memory management through the sliding window approach. Given these two changes, the paper does not discuss the performance implications with and without the proposed framework. It is important to guarantee that the performance remains same. 
  - Given that, it is recommended to show that the performance (atleast in the best case settings on atleast one model) remains unchanged with or without the TPI-LLM.
- Figure 5 is more concerning in the following perspectives.
  - No impact on bandwidth: There is no surprise on the lack of impact of increasing the bandwidth since the sliding window size is defaulted to 2. However, we can only learn about the impact by increasing the window size. There is no such ablation in the paper that shows a best combination of the number of devices, maximum possible sliding window size, bandwidth, available memory on each of the devices. 
    - Recommendation is to provide a thorough feasibility study on the combination of the above variables to clearly understand the impact of the bandwidth. In fact on that note, there is no clear analysis on the maximum possible sliding window size for a given hardware configuration of the master/worker nodes. Therefore, the feasibility study can be preceded by maximum window size in order to limit the number of combinations to be studied.
  - Increasing the number of devices/cores reduces the token latency (from first two sub-figures in Figure5) is not a true statement and is really vague. Those plots are shown simply for 8 devices, if the number of devices are kept increasing, after a point we see diminishing returns of parallelism. That is where the communication dominates and hence to diminishing returns. Without having proper study `increasing the devices reduces the latency` are not valid claims. 
    - The recommendation is to conduct a quick roofline analysis to substantiate the claims or remove the controversial parts. 
- There are a few limitations on what the proposed framework can offer. They are as follows.
 - There is a security and privacy concern, this framework should not run any device connected in the same Wi-Fi network unless there is a prior consent. It is not stated in addressed in the paper and hence please clearly state the limitation or the constraints under which the framework can operate.
 - The star configuration can not scale to large number of nodes/devices. It probably can be extended to scale in a hierarchical-star configuration etc, but that is not the scope of the paper and hence this needs to be state clearly as a limitation. There are real-time use-cases in the resource constrained edge scenarios where the number of devices is high, which leads to failures of a single master node in star config.
 - There is probably an unstated assumption in the paper that the data gets generated on the master or stays centrally on the master node/device. However, there is a high chance of the worker nodes/devices having user specific data. It appears that the proposed framework does not handle data that is generated on all the other devices. If that is not the case, please clarify how that data is handled on each of the workers? Otherwise, this limitation should be stated. 

### Minor concerns
- In Figure 4, Time steps 7 and 8 have the same memory window, why is it the case, it is a bit confusing to understand. Is the memory peaked and hence the window does not slide or something else? Please provide clarifications or amend the figure to make it clear.
- Tables 1 & 2 provide comparisons for TTFT, latency and memory usage, etc between with and without the use of memory scheduler. However, when the memory scheduler is disabled, it is not clear how those numbers are attained. How were they measured? by using accelerate, vanilla Klonet or Galaxy or llama.cpp or others?
  - Please add those details on how the stats were measured and the underlying frameworks. Ideally benchmark comparisons against the best possible frameworks/methods is a common practice?
- Section 4.3 states the `comparison with benchmarks` Ideally those (a. Standalone, b. Model Parallelism (MP) and c. Galaxy) are SOTA methods/frameworks (in this case). Are they not? Why are they called benchmarks? Clearly there is benchmarking of TPI-LLM against those other things. 
 - Recommendation is to please rephrase in order to convey the message so that the confusing the reader can be avoided.
soundness: 2
strengths: Following are the strengths:
- The paper is well-written, easy to follow and understand the concepts presented.
- The paper tries to addresses the critical problem of LLM inference on the edge devices.
- The paper discusses the existing frameworks in this space and positions within the body of knowledge.
- Achieving the faster TTFT latency using 3k LOC and for large models is always interesting and of use for broader audience.
- Although the paper uses multiple devices connected in the same network (Wi-Fi/cable connected), it is understandable that there are use-cases where a house will have multiple edge devices all of which might operate in-tandem, there maybe business orgs that have edge devices that don't have permission issues to run LLMs on multiple of such devices. For such use-cases this is a significant contribution.
- The paper proposed an easy to use technique of overlaying the data fetch during communication step in LLMs in the proposed sliding window strategy.
summary: The paper proposes a new framework called TPI-LLM for model serving on low resource edge devices using tensor parallelism and memory scheduling. The proposed frameworks shows better performance over the SOTA frameworks in terms of latency to predict the first token and the overall token latency.
weaknesses: Please follow the questions section, there is a cohesive list of weaknesses and the corresponding questions that are to be addressed.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: Please see questions from weaknesses.
soundness: 3
strengths: 1 - The paper provides extensive theoretical analysis.

2 - The proposed approach is evaluated in both emulations and real-world testbeds against state-of-the-art baselines.

3 - The performance is significant, especially on edge devices with limited resources.
summary: The paper proposes TPI-LLM, a tensor parallelism-based distributed LLM inference system tailored to low-resource edge devices. Unlike inference tasks on servers, TPI-LLM keeps user data securely on user (edge) devices during inference. By analyzing the network latency bottleneck and memory limits, TPI-LLM leverages a star-based all-reduce algorithm to facilitate distributed inference and employs a sliding window-based memory scheduler to reduce inference memory footprints. Experiments on both emulations and real-world testbeds show that TPI-LLM outperforms existing baselines by providing lower TTFT, token latency, and peak memory footprints.
weaknesses: 1 - In Section 2, Q1 is somewhat ambiguous. First, isn't tensor parallelism already a form of model parallelism? And second, even on low-resource edge devices, can we combine and use both parallelism techniques instead of simply abandoning one?

2 - In Section 3.2, the example network topology here is star-based (Appendix A.7). Given this star-based topology, a star-based all-reduce scheme indeed would be most efficient. Is this a common network topology for all edge scenarios?

3 - Figure 4 may need further detailed descriptions. The sliding window is not very clear in the figure. For example, in Time 7 and 8, why would you prefetch Blocks 6, 7, and 8 so early when it's still far away from actually using them? Isn't that prefetching too early causing memory waste?

4 - Note that the layer-wise parameter offloading is not new. Many popular frameworks, such as DeepSpeed-Inference, Mixtral Offload, and Llama.cpp, support this offloading scheme. How does the proposed TPI-LLM differ from existing offloading techniques?

5 - Evaluation lacks sensitivity analysis on the memory sliding window size. Why would you pick a window size of 2?