Prima.cpp: Fast 30-70B LLM Inference on Heterogeneous and Low-Resource Home Clusters
Authors: Zonghang Li
Tao Li
Wenjiao Feng
Rongxing Xiao
Jianshu She
Hong Huang
Mohsen Guizani
Hongfang Yu
Qirong Ho
Wei Xiang
Xue Liu
Venue: ICLR 2026 Poster (2026)
Keywords: Distributed LLM system
on-device inference
low-resource and heterogeneous devices
home AI
Primary area: infrastructure, software libraries, hardware, systems, etc.
TL;DR: Prima.cpp is the first on-device distributed system to deliver practical performance for 30-70B LLMs on consumer-grade home devices, with mixed CPUs/GPUs, insufficient RAM/VRAM, slow disks, Wi-Fi links, and heterogeneous OSs.

Abstract: On-device inference offers privacy, offline use, and instant response, but consumer hardware restricts large language models (LLMs) to low throughput and capability. To overcome this challenge, we present prima.cpp, a distributed on-device inference system that runs 30-70B LLMs on consumer home clusters with mixed CPUs/GPUs, insufficient RAM/VRAM, slow disks, Wi-Fi links, and heterogeneous OSs. We introduce pipelined-ring parallelism (PRP) to overlap disk I/O with compute and communication, and address the prefetch-release conflict in mmap-based offloading. We further propose Halda, a heterogeneity-aware scheduler that co-optimizes per-device CPU/GPU workloads and device selection under RAM/VRAM constraints. On four consumer home devices, a 70B model reaches 674 ms/token TPOT with <6% memory pressure, and a 32B model with speculative decoding achieves 26 tokens/s. Compared with llama.cpp, exo, and dllama, our proposed prima.cpp achieves 5-17× lower TPOT, supports fine-grained model sizes from 8B to 70B, ensures broader cross-OS and quantization compatibility, and remains OOM-free, while also being Wi-Fi tolerant, privacy-preserving, and hardware-independent. The code is available at https://gitee.com/zonghang-li/prima.cpp.

Published as a conference paper at ICLR 2026
PRIMA.CPP: FAST 30-70B LLM INFERENCE ON HET- EROGENEOUS AND LOW-RESOURCE HOME CLUSTERS
Zonghang Li Tao Li
Wenjiao Feng² Rongxing Xiao²
Jianshu She'
Hong Huang?
Mohsen Guizani' Hongfang Yu² Qirong Ho Wei Xiang'
Xue Liu
MBZUAI UESTC
*City University of Hong Kong
"La Trobe University
ABSTRACT
On-device inference offers privacy, offline use, and instant response, but con- sumer hardware restricts large language models (LLMs) to low throughput and capability. To overcome this challenge. we present prima.cpp, distributed on-device inference system that runs 30-70B LLMs on consumer home clus- ters with mixed CPUs/GPUs, insufficient RAM/VRAM, slow disks, Wi-Fi links, and heterogeneous OSs. We introduce pipelined ring parallelism (PRP) to over- lap disk I/O with compute and communication, and address the prefetch-release conflict in mmap-based offloading. We further propose Halda, a heterogeneity- aware scheduler that co-optimizes per-device CPU/GPU workloads and device selection under RAM/VRAM constraints. On four consumer home devices, a 70B model reaches 674 ms/token TPOT with <6% memory pressure, and a 32B model with speculative decoding achieves 26 tokens/s. Compared with llama.cpp. exo, and dllama, our proposed prima.cpp achieves 5-17x lower TPOT, supports fine-graines model sizes from 8B to 70B, ensures broader cross-OS and quantization compatibility, and remains OOM-free, while also being Wi-Fi tol- erant, privacy-preservingg and hardwarc-inddependent The code is available at attps::/giteee com/ zong/angggagggo wi/prriiiiiiie httpppriaaa.pppp.
INTRODUCTION
As representative of next-generation AI, embodied (Gupta et al., 202T) acutely sensitive to privacy protection, stable connectivity, and long-term financial costs. Home surveillance, always- listening assistants, and companion robots cannot send raw interactions to the cloud. Cloud services also suffer from frequent network failures, queuing, timeouts, and their costs scale linearly with token usage and service time. Consequently, embodied AI is moving from the cloud to the device
However, on-device inference is constrained by modest chips and small RAM, struggling to run models beyond 8B (MLC) 2025 Lugaresi et al. 2019 Ghorbani] 2025), yet reliable long-term planning and tool use require 32B more. To support larger models, disk offloading helps capacity but at the cost of speed (Gerganov] 2025 Li 2023), e.g.. Qwen 14B Q4K on an 8 GiB Mac MI laptop takes 10 s/token with llama.cpp. Recent efforts use dedicated devices (c.g.. Jetson, Mac Studio) for larger models and faster speed (Ye et al.) 2024 2025), but they are too expensive for most households. This motivates our goal: Can we deliver fast 30- 707 inference on user-owned consumer devices that meet embodied Al's sss demands summarized in Table
Distributed inference may be the only solution to improve model size and speed while preserving the original outputs. Users usually own multiple devices, e.g., laptops, PCs, phones, and tablets. Some laptops and PCs have low-end GPUs like NVIDIA 20-/30-/40-series. and some Maes have Apple Silicon GPUs By pooling the computing and memory of home devices, we run larger models.
Despite recent progress, four limitations remain: (a) Existing systems require sufficient aggregated RAM/VRAM to hold the full model Ye et al. 2024 Exo 2025 Tadych 2025 Zhang al. 2025
Lee et al.) 2024 Zhao et al., 2023 Zhang et al.) 2024). driving up hardware costs and restricting
(TPOT) of tens of seconds (Li et al. 2025). (c) Layer partitioning relies on the strong assumption in (a) and overlooks
model size. (b) Disk offloading slows inference, causing time per-output-too
heterogeneity in OS-specific memory reclamation and disk access. (d) All devices assumed to

---

Published as a conference paper at ICLR 2026
Table 1: Comparison of cloud and local LLM deployments. TPS1 indicates the per-request token
rate, TPS² denotes TPS values measured across 32-70B models. OA denotes open-access.
Cost ($)
Privacy
Speed (TPS)
Queue
Net
Support Model
App
100
K
10-70'
Specific models Specific models Any OA model
Cloud
API
100
10-70"
<
Dedicated
1,000
X
10-35²
x
Dedicated
1,000
V
4-17a
X
X
Any OA model (< (<B)) Any OA model (=70B)
Local
Consumer
Free
2-26a
be necessary, even though removing slow ones could improve speed. These limitations raise two questions that motivate design: (Q1) How memory constraints be relaxed run larger models? If disk offloading is required, how can disk latency be hidden? (Q2) Given QI, how can we design heterogeneity-awan layer partitioning and identify bottleneck devices?
We propose prima.cpp, the first distributed inference system for consumer home clusters, with mixed CPUs/GPUs, insufficient RAM//RAM, slow disks, Wi-Fi links, heterogeneous OSs, and capable of running 30-70B models at practical speed. To address Q1, Section 3.1 proposes pipelined-ring parallelism with prefetching overlap disk latency and resolve the prefetch-release conflict in mmap. To address Q2, Section 3.2 models heterogeneity in compute, communication, memory, and OS-specific memory reclamation and disk optimizations. Section [3.3] develops Halda to find the optimal layer partitioning and select the best best-performing devices. Section 4 presents experiments on real home clusters and shows that, even under limited memory, 70B model runs locally 674 ms/token with <6% memory pressure. With speculative decoding (Leviathan et al.) (023), 32B model achieves 26 tokens/s, which is fast enough and paves the way for household embodied AI.
As summarized in Table I prima.cpp runs existing, free devices; all data stays local; no queueing or timeouts; offline; and supports models up to 70B. To the best of our knowledge, prima.cpp is the first on on-device system deliver practical performance for 30-70B models in such constrained environments, without relying on specialized hardware (e.g., Jetson, NPUs) or altering model outputs.
RELATED WORK
On-device LLM systems. Most of them run on a resource-constrriieed device and can only handle small models. MLC-LLM (MLC, 2025) and MediaPipe (Lugaresi al. 2019) bring 7B models to mobile phones and browsers. PocketPal AI (Ghorbami, 2025). which runs on Android using Ilama.cpp GGrggnovv 2025). supports models up 3.8B. Some efforts push for larger models, like AirLLM (Li) (023), which loads only the needed layers to save memory but at the cost of speed. Others turn high-end hardware, such the Apple M2 Ultra (192 GiB RAM) for 65B model or k &Transformers (kvcache ai, (0255) (382 GiB RAM) for a 67 IB model (~75 GiB for 70B). Like KTransformers, HeteGen (Zhao al. 2024 also collaborates with the CPU and GPU on one device to accelerate inference. These setups (large RAM, advanced CPUs with specific instruction sets, dedicated hardware) go far beyond common home devices and are inaccessible to most households.
Distributed on-device LLM systems. Such distributed systems follow two main parallelism paradigms: tensor parallelism and pipeline parallelism.
Tensor Parallelism (TP). TP splits tensors across devices to share the load (Shocybi et al. 2019). To speed up all-reduce, dilama (Tadych) 2025) uses USB4 and Thunderbolt. for fast connections, and Airlnfer (Zhang et al. 2025) uses wireless analog superposition to perform all-reduce over the air. Due to device heterogeneity, Hepti (Lee et al. 2024) optimizes workload partitioning with three slicing strategies for different memory budgets, and Galaxy (Ye et al. 2024) prioritizes compute power first, then memory, to maximize speed and avoid OOM. These systems reside the full model in run. TPI-LLM (Li
the aggregate memory. With limited aggregate memory, only small models et al., 2025) loads model layers on demand and hides disk loading with prefetching. This enables low-end devices with only 4 GiB of RAM to 70B model, but at 30 s/token, making impractical.
2

---

Published as a conference paper at ICLR 2026
Table 2: Comparison of distributed on-device LLM systems. (Abbr.: Quantization, Heterogeneity)
Type
Backends
Mem
Quant.
Mem Stress
Speed
Hete.
dllama
TP
CPU
RAM
Q4
Critical
Slow
x
Airlnfer
TP
CPU
RAM
FP32
Critical
Slow
Hepti
TP
CPU
RAM
FP32
Critical
Slow
Galaxy
TP4SP
CPU/GPU
RAM/VRAM
FP32
Critical
Slow
TPI-LLM
TP
CPU
RAM
FP32
Medium
Slow
x
CXO
PP
CPU/GPU
RAM/VRAM
Q4+FP32
Critical
Slow
/
Lingual.inked EdgeShard
PP
CPU
RAM
Q8/FP32
Critical
Slow
/
PP
CPU/GPU
RAM/VRAM
FP32
Critical
Fast
/
prima.cpp
PRP
CPU&GPU
RAM&VRAM
Q4/QQ1
Low
Fast
Pipeline Parallelism (PP). Due to Wi-Fi's high latency, pipeline parallelism becomes more suitable for home clusters as it requires less P2P communication. Exo 2025): Zhao et al. (202)) Zhang et al. (2024) split the model into segments and assign them to devices based memory, compute, and network conditions. Each device computes its segment and passes the result to the next, until the last device outputs the
token. Exo (2025) partitions model segments based memory ratio; LinguaLinked (Zhao et al., 2023) uses linear optimization to solve the device assignment problem; and Shard (Zhang et 2024) uses dynamic programming.
However, these systems either require dedicated hardware (c.g., Jetson AGX/Nano) or sufficient ag- gregate memory to reside the full model, support only CPU or GPU backends, overlook heterogeneity (especially in disk offloading), and impose high memory pressure. Table [ Iuummarizes their features. These limitations raise hardware costs, restrict use to small models, cause slow inference and device freezes, and discourage users from deploying LLMs at home.
Instead, prima.cpp runs on consumer devices, supports disk offloading to run larger models, and uses pipelined-ring parallelism (PRP) to overlap disk latency. It runs on both GPUs and CPUs, combines RAM and VRAM, and models system heterogeneity to optimize GPU-CPU layer partitioning per device. Moreover, it stores model weights in the OS page cache, allowing the OS to reclaim memory as needed to preserve user experience. These features distinguish prima.cpp from existing systems.
PRIMA.CPP: USE PIPELINED-RING PARALLELISM IN LLAMA.CPP
3.1 PIPELINED-RE RING PARALLELISM WITH PREFETCHING
PP is effective for batched inference when aggregated memory is abundant; however, this prerequisite rarely holds in households. In such cases, on-device LLMs typically serve very few users at low frequency, often single request at a time which prevents mini-batching and leaves significant pipeline bubbles. Besides, home devices are few and have limited available memory making difficult for users to build a cluster that can hold the full model in memory.
To relax memory constraints, extend PP with mmap, building Ilama.cpp Gerganov 2025) when computation requires the layers, mmap loads them from external storage on demand, and lets the OS evict them under memory pressure, small-ment memory cluster can run larger models. To hide disk loading latency, we employ prefetching, allowing devices to preload the next model segment.
This naive design supports larger models with low memory pressure but suffers from slow speed, because prefetching will fail due to prefetch-release conflict: if disk reads are fast, later-loaded layers will evict earlier prefetched layers from page cache, so when computation begins, the needed layers are no longer cached (see Appendix A. for details). This triggers page faults and reloads, negating the benefit of prefetching, and pipeline bubbles remain significant (see Fig. d in Appendix (A.2). providdd empirical evidence for this conflict and performance breakdown: even after prefetching completes, required model weights still need be reloaded during computation.
Appendix A. provides
WWe target a single request, but this can he extended to batches via dynamic batching (see Appendix A.11 We cannot occupy all memory, this may disrupt other apps (e.g., Tok and cause kill the T.I.M.
3

---

Published as a conference paper at ICLR 2026
Pipelined-ring parallelism (PRP) with prefetching. To address the prefetch-release conflict, further propose PRP, which connects devices end-to-end in a ring and runs multiple rounds to predict one token. In each round, devices prefetch different layer segments from the disk, which overlap with other devices' ongoing operations, such as computing, communication, and disk loading. Since only small segment (whose size is referred memory overflow can be avoided, and prefetched layers are less likely to be evicted, thus mitigating the prefetch-releat conflict (see Figs. and 5 in Appendix [..2] for examples). In addition, PRP processes both input and output on the head device, providing enhanced interaction privacy.
the layer window size) is loaded per round,
In Fig. | all devices share layer window size of 2. Each device processes two model layers per round and takes 3 rounds to pre-
Model Segieets (total window size
10
15
1R
dict one token. Fig. 6 b in Ap-
Device
Device
Device
pendix A.2 show the PRP timeline on homogeneous devices with fast and slow disks. On fast disks, prefetching latency is fully over- lapped; on slow disks, CPU and disk operate alternately, remov- ing bubbles but leaving residual page-fault loading latency, since required layers may not be fully loaded when computation begins. Fig. evaluatess this multi-round design: for large models, PRP re- duces PP's TPOT by about 50% (PP is equivalent to PRP k = 1), while for small models, PRP con-
Raund
Round
Hello
Device 6
Device
Device
Figure 1: Pipelined-ring parallelism. In this case, 6 devices handle a 36-layer model. With a layer window size of 2, the model is split into 18 segments and assigned to 6 devices in ring order, so each device needs 3 rounds to predict one token.
verges to PP with similar TPOT. Appendix A.12 presents the overall design.
In practice, however, home devices are heterogeneous. Using a uniform layer window size still produces pipeline bubbles (see Fig. 6 in Appendix(A.2), whereas assigning different window sizes per device can reduce them (see Fig. 6 in (A.2) In general, stronger devices should be assigned larger window sizes. However, determining which devices are stronger and how large their window sizes should be is challenging. Beyond common heterogeneity in computing, communication, and memory, disk loading is heavily affected by OS-specific memory reclamation and disk read throughput. Such complex heterogeneity makes disk latency difficult to quantify.
3.2 LAYER-TO-DEVICE ASSIGNMENT PROBLEM
As mentioned in Table prima.cpp is designed to multiple backend Within a layer window, some layers reside in VRAM and execute on the GPU, while others are offloaded to RAM and executed on the CPU. If the layers assigned to the CPU exceed available RAM, the overflow is further offloaded to external storage and reloaded into RAM via mmar" set the layer window size for each device? Which layers run on the GPU and which on the CPU?
E This leads to two challenges: How
Prior work assumes sufficient aggregated VRAM and proposes heuristic partitioning strategies. For example, Exo (2025) partitions model layers based on memory ratio, devices with more RAM/VRAM handle more layers. Ye et al. (2024) partition by compute power and migrate layers from OOM devices to those with free memory. These heuristics work on some testbeds but are not always optimal (see Fig. 10 in Appendix Ao since memory size does not guarantee compute power, and under disk offloading, devices with larger memory but weaker CPUs/GPUs can perform worse than those with less memory but faster CPUs and disks.
OOu prototype supports CUDA, Metal, and CPU backends, but with minor adaptation, it can also work with others such Vulkan and ROCm.
'We pin model layers in VRAM and avoid paging them as in disk RAM: otherwise, disk RAM and RAM- VRAM transfers would contend for PCIc (or shared) bandwidth, increase disk reload volume, and amplify disk loading latency, which offsets the gain from GPU acceleration.
4

---

Published as a conference paper at ICLR 2026
To achieve optimal partitioning, TPOT must be quantified through an analytical model. Building such a model is challenging: beyond device heterogeneity, we must account for CPU-GPU coordination, memory contention, OS-specific memory reclamation, disk optimizations, and quantization. For instance, some PCs have dedicated GPUs with separate VRAM; others (e.g.. Mac M-series) adopt a UMA architecture where CPU and GPU share memory, and OS reclamation will be more aggressive; some are NUMA systems without GPUs, and they differ in reclamation thresholds across OSs; even on the same device, OS reclamation differs with or without Metal; Linux optimizes sequential disk reads, making reloads faster; quantization influences compute latency, memory access, disk loading, and RAM//RAM constraints, ultimately shaping the partitioning strategy.
After extensive performance analysis in Appendix A.3 and preliminary experiments, we formalize this as the layer-to-device assignment (L.AA) problem as follows.
Definition 1 (Layer-to-device assignment). Assume there M devices, Wm, is the layer window size on device dm and 7.m is the number of GPU layers within Wm- Let the decision variables be w [wm,, W2, ... was] and n [mi, "ly, , Our objective is find the optimal w and n.
that minimizes the TPOT:
L a w DT +
e
min w.n
+ tK,
(1)
eT w
s.t.
Wm E Z 11m E Z2o, fm S um S L,
(2)
L -- -(e" w) = 0, k E Z>0;
(3)
P.. w' Pa n' + e
0,
(4)
Pspu sP e- w papu S 0.
(5)
where L is the number of model layers; a. b, are latency coefficient vectors determined by compute latency, memory access latency, disk loading latency, and communication latency on each device; is constant latency offset; k is the number of rounds to predict one token; w' and n' are the extended vectors of w and n,; z and RP are constraint vectors for RAM and VRAM; Pw, Pa, P diagonal matrices that activate or deactivate the decision variables; and is an all-ones vector.
Constraint 2 ensures the layers do not exceed the limit. Constraint 3 enforces that all devices are assigned an equal number of windows and all windows are filled) Constraints (4) and (5) ensure that RAM and VRAM usage stay within limits. summariess the key symbols. To construct a, b, C, K, wp, n', ; z, Pw:; P.. we categorize devices into four sets: (a) Set Mi: macOS with Metal disabled and insufficient RAM; (b) Sei M2: macOS with Metal enabled and insufficient RAM; (c) Ser Ms: Linux and Android with insufficient RAM: (d) Set M, devices with sufficient RAM or slow disk. For Sets Mj-M3 where devices should overload, RAM usage should stay above available RAM; for Set M4. where overloading is not allowed, RAM usage should stay below available RAM. set and adjust variable dimensions.
We can extend other OSs to these sets, or create
This problem is an NP-hard integer linear fractional program (ILFP): both the numerator and denominator of the objective are linear in the decision variables, and all constraints are linear inequalities. Whether a device is overloaded depends on w and n, e.g., a large wm. 2mm will overload RAM. However, cannot determine device's 's set before solving the LDA problem, and without the set assignment, we cannot solve the LDA problem. This traps us in circular dependency.
3.3 HALDA: AUTOMATIC LAYER PARTITIONING AND DEVICE SELECTION
To solve this non-standard, NP-hard LDA problem, our core ideas include: (i) transform the original problem into a set of standard integer linear programs (ILPs) by enumerating over all valid & that divide L, and (ii) search for optimal set assignments Mi-M4 by iterative optimization.
Transform into standard IIPs. Given that the number of layers I in LLMs is typically less than 100, the integer i has limited range of values: at most 11 valid factors for any L. S 100. By
This is not mandatory in implementation, but it simplify the problem model.
5

---

Published as a conference paper at ICLR 2026
Algorithm 1: Heterogeneity-Aware Day Allocation (HALDA)
Initialize layer windows un proportionally to devices' memory budgets, and GPU layers n t- 0: Calculate platform-specificc coefficients O'm> Bin Sm for each device m;
Calculate valid factors 1c, of L (excluding L);
while true do
Calculate W = e and k- = L/W:
Reassign devices sets M1, Ma, M4 based on the latest k and M the assignment sets M1, M1,M2,44 Ma,
I Ma remain unchanged then
break;
Calculate the objective coefficients b, the RAM upper bound z, and the VRAM/shared memory upper bound z 2P according to the updated assignment sets; foreach k XC, do Solve the ILP for fixed using a solver; Update best solution (w", n²) ) if the current objective is smaller; if any device has free VRAM but another device is overloaded then Force the device m from (MM1M2, Mz} with slowest disk reads into M continue:
11
12
13
14
15
Update <- = and n n²;
17 return w", ,n";
enumerating these factors, we treat is and W as constants, and the problem becomes:
min
k(aa T w b n n TT. c) c)+ +r,
(6)
w.n
s.t.
Wm E Σ>o; 10m Zoo 7m S Wm S L,
(7)
e w W,
(8)
Pw w' Pm n' 1 W 0,
(9)
papa n W pa 2p < O.
(10)
Hence, for each fixed k; the objective and constraints boil down to linear functions/inequaliiie, and the problem becomes an ILP. Then, we can run a standard ILP solver (e.g.. HiGHS (Huangfu Hall 2018)) to obtain the optimal w. n2.
Iterative optimization for set assignment. The problem remains unsolvable because the sets Mi- M4 unknown. To break this circular dependency, we adopt iterative optimization procedure. We start by setting w proportional to available memory and initializing n = O, which gives an initial division of devices into Mi-M4. We then solve the ILPs update w and n. reassign devices, and iterate until the sets converge.
However, this approach still has defects. For example, if device (M1, MM, Ms) is assigned 30 layers, although the GPU can host 20 layers, memory constraints may end up assigning only 10 to the GPU and 20 the CPU, underutilizing the GPU. This issue arises from an improper set initialization: if the device was initialized in M4. its GPU could be fully utilized. Therefore, Algorithm I iincludes a calibration step: if a GPU is underutilized (VRAM not full) while another device is overloaded (VRAM full with layers offloaded to CPU, or RAM exceeded). we move the device with slowest disk reads from the set again. This ensures convergence to the optimal set assignment and the corresponding optimal values of un n. and k.
into Afforce (Mifirce c M44 and solve
Complexity analysis. The main loop alternates between set assignment and LDA solving, repeated for T' iterations with T' ((M) in the worst case. The set assignment takes O(M). M. For LDA, we solve tiny ILP for each valid factor of L, giving K O(log L) factors in total. Although ILPs are NP-hard, our instances are small and sparse, allowing modern solvers to finish quickly. Empirically, on 4-32 devices, the global scheduling latency is 10- 12 ms. The overhead of rerunning Halda is
gaail for macOS without Metal and I Linux, meral for macOS with Metal, and avail ponpout for Android.
d

---

Published as a conference paper at ICLR 2026
Table 3: Testbed configuration. Termux is used on D4 and DS.
DI
D2
D3
D4
D5
D6
Device
Mac M1
Laptop
Desktop
Mato HarmonyOS
Honor Pad Android
Mac Air
OS
macOS
Linux
Linux
macOS
CPU
Apple M1 8
Intel i9
Intel i9
Kirin 9000
Dimensity 8100
Intel i5
CPU Cores
8
16
8
8 5.1 GiB 2.0 GB/s
4 6.8 GiB
RAM (avail.)
2.4 GiB
4.1 GiB
9.7 GiB
1.9 GiB 1.4 GB/s
Disk Read
0.7 GB/s
3.0 GB/s
3.0 GB/s
0.4 GB/s
GPU Type
Apple Silicon
3070 8 GiB
2080TI
VRAM (avail.)
11 GiB
negligible, and since no cross-device layer migration is required, workloads can be repartitioned whenever the task queue is empty to adapt to environmental changes.
Device selection. Weak devices assigned only one layer are removed, since Halda indicates that excluding them would improve speed. Appendix A..] A.7 illlustras this selection process, For small models, Halda prefers to allocate all layers to one most powerful device, reducing prima.cpp to llama.cpp. However, if removing a device would break the communication ring (e.g., it is the only reachable hop due to network policy), Halda keeps it in the ring as relay node but assigns no workload. This design frees users from device selection: we only need to discover more available devices, and Halda will select the optimal subset for us.
4 EXPERIMENTS
We implement prima.cpp with 20K lines of code based on llama.cpp and build real testbed of heterogeneous, low-end home devices (see Table These devices connect via a local Wi-Fi router, with inter-device bandwidth ranging from 320-610 Mbps and link latency from 3-7 ms. By default, four devices (DI (DDD4) with aggregated RAM+VRAM of 37 GiB (insufficient for a Q4K-quantized 70B model) are used. We evaluated Llama models from 8B to 70B (Q4K) in terms of time-per-output- token (TPOT). time-to Lime-to-first-eie fo-fiirtttoken (TTFT), and memory pressure. We select llama.cpp. exo, and dllama as baselines, as they are the most popular open- on- on-device (distributed) LLM inference systems" 7 Since llama.cpp is a standalone on-device system, we run it on D3, which offers the largest RAM/VRAM and highest decoding efficiency Exo a PP system, which we run on D1-D4, but D4 failed because it requires root access on the phone; dllama and prima.cpp successfully run on D1-D4.
4.1 FAST INFERENCE ON LARGE MODELS
Table 4 presents TPOT and TTFT across model sizes of 8-70 As result, prima.cpp achieves substantially lower TPOT and TTFT than all baselines, and matches llama.cpp on small models (<14B). Compared with llama.cpp. it reduces TPOT by up to 17x and TTFT by up to 8x; against exo and dilama, it achieves at least 18x lower TPOT and 42x lower TTFT, without OOMs.
For small models (<14B), Halda finds that D3 can hold the entire model in VRAM, so removes the other devices and runs only on D3. This reduces prima.cpp to the single-device case (equivalent to llama.cpp). with both achieving the same TPOT and TTFT. At 30B, as shown in Fig. 9a in Appendix A.8. D3-GPU runs out of VRAM and forces layers onto the CPU, causing llama.cpp to deteriorate rapidly. In contrast, Fig. 9d shows that Halda offloads the overflow from D3-GPU to D2-GPU, keeping TPOT and TTFT stable. At 45B, both RAM and VRAM on D3 are exhausted, and mmap in llama.cpp begins frequent reloads, incurring disk latency. At this stage, only a few pages are reloaded, so the efficiency loss is minor. At 60B, high memory stress causes more active pages to be evicted earlier, sharply increasing TPOT and TTFT. This indicates that llama.cpp is ill ill-suited for large models on consumer-grade devices. In contrast, Halda balances workloads across devices according their
7 At submission, llama.cpp had 87K stars, exo 31K stars, and dllama 2.7K stars. We also tried deploy VLL.M and SGLang, but as server-oriented systems, they failed to run on our home cluster.
At small batch sizes, decoding is handwidth- bound, and 2080TI offers higher memory handwidth than 3070. 2 Exo supports Ilama 8/70B and dllama supports only Llama so mark unavailable values with "-".
7

---

Published as a conference paper at ICLR 2026
Table TPOT (ms/token) and TTFT (ms) Llama models for Ilama.cpp. exo, dllama, prima.cpp.
prima.cpp (w/o halda)
prima.cpp (w/o prefetch)
Size
llama.cpp
exo
dllama
prima.cpp
TPOT
TTFT
TPOT
TTFT
TPOT
TTFT
TPOT
TPOT
TPOT
TTFT
8B
15
18
263
960
1150
4409
78
15
15
18
14B
20
2201
10279
131
20
25
30B
202
611
258
79
72
214
45B
328
712
6235
18532
409
263
233
440
60B
7965
8350
7053
532
468
990
65B
8807
9662
OOM
OOM
12253
688
569
1770
70B
10120
10806
OOM
OOM
OOM
OOM
20848
755
674
1793
capabilities. Even when disk offloading is required, it assigns workloads to devices with strong CPU and fast disk, while PRP with prefetching hides disk latency. As a result. prima.cpp maintains stable TPOT and TTFT growth and achieves sub-second TPOT even at 70B. To the best of our knowledge, prima.cpp the first system to achieve this speed on real mansuerr-grade low-end device 10
For exo and dilama, even with only few valid samples, their limitations are already clear. Llama 3-8B (Q4K) requires 5.3 GiB of VRAM, so the entire model could fit into D3-GPU. However, exo allocates layers proportional to each device's 's memory: D1 (8 GiB RAM), D2 (8 GiB VRAM), and D3 (11 GiB VRAM) are assigned 9, 10, 13 layers, respectively. While D1 has an Apple Silicon GPU, its efficiency is much lower than the 2080TI on D3-GPU, making it a bottleneck. Dilama uses TP. An 8B model needs 64 all-reduce operations, leading to heavy communication delays on high-latency Wi-Fi and much higher TPOT than prima.cpp. As model depth grows with size, the communication bottleneck worsens. At 70B, dllama runs out of memory, but our independent tests show ~ 150 ms per all-reduce, so 160 all-reduces add 24 s/token, resulting in much higher TPOT than prima.cpp's 674 ms/token (442 ms/token with speculative decoding). These results highlight that Halda and PRP better suited to heterogencous, high-latency home environments than heuristic partitioning or TP.
on more models (Qwen 2.5, QwQ and DeepSeek R1) (Appendix (A.6). and on larger heterogenous testbed (Appendix 19) support the strong generalization of prima.cpp. Appendix A.7 explains the underly- ing rationale for device selection: more not always better, and aggregated memory need not match the model's 'ss needs. It also shows how Halda selects subset of devices to build best-perfoor pest-performing cluster. Appendix [A.9]shows that with speculative decoding, 32B inference achieves 26 tokens/s, offering both the speed and intelligence needed for LLM agents.
More experiments on heterogenous and homogeneous testbeds (Appendices A.4 A.5
4.2 ABLATION STUDY ON PREFETCHING, HALDA, AND PIPELINED-RING RING PARALLELISM
Table 4 also presents ablations on Halda and prefetching. Exo is also a PP system, so we migrate exo's layer partitioning (based memory ratio) to prima.cpp (w/o halda), making it exo variant. Unlike exo, this variant uses available instead of total memory, adds broader range of models, adds cross-platform quantization and prefetching, and supports CPU/disk offloading to prevent OOMs.
Prefetching. To evaluate prefetching, we compare prima.cpp with and without it. It has little effect on small models that fit in RAM/VRAM, but for larger models, evicted layers cause frequent page faults and reloads. Without prefetching, all reloads are triggered on demand by page faults. With prefetching, upcoming layers are loaded in advance and their latencies overlapped, reducing page fault reloads and lowering TPOT by 9-17%.
Halda. proper layer partitioning critical for fast inference. As shown in Table prima.cpp with Halda reduces TPOT by up to 31 x over prima.cpp (w/o halda). For small models, Halda selects the most powerful device (D3) to run all layers, while exo partitions by memory ratio, assigning more layers weak devices. For larger models, where disk offloading is unavoidable. Halda balances workloads to reduce I/O pressure or prioritizes devices with strong CPU and fast disk, whereas exo
10We exclude sparse inference systems, as they do not meet the conditions in Table they require dedicated hardware such NPUs, support only sparse models, and may degrade output quality.
8

---

Published as a conference paper at ICLR 2026
Table Memory pressure for each device on Llama models.
Size
llama.cpp
exo
dllama
prima.cpp
D3
DI
D2
D3
D1
D2
D3
D4
DI
D2
D3
D4
8B
2.0%
20.0%
51.3%
42.5%
17.3%
17.3%
19.0%
13.7%
5.3%
5.4%
2.7%
<1.0%
14B
2.5%
29.2%
27.3%
24.4%
24.9%
5.3%
4.3%
2.2%
<1.0%
30B
8.0%
3.0%
5.7%
2.9%
$1.0%
45B
3.9%
61.0%
73.9%
62.5%
73.8%
4.9%
$1.0%
6.0%
<1.0%
60B
5.5%
6.3%
4.7%
4.7%
31.0%
6SB
15.6%
OOM OOM
OOM OOM
OOM 0OM
OOM OOM
3.9%
$1.0%
$1.0%
$1.0%
70B
6.0%
OOM
OOM
OOM
4.7%
4.8%
4.8%
$1.0%
often overloads slow-disk devices, causing TPOT to spike. Appendix AA.10 compares Halda with two heuristic baselines, further showing its effectiveness and novelty.
PRP with multiple per-token rounds. To evalu- ate PRP in isolation, we built a CPU cluster with 4 devices, each with 8 cores, 8 GiB RAM, and an SSD of 2 GB/s. We tested models from 8-72B,
180
Liama 3-88
Liama
Liama 3-148
DeepSeek -08
DeepSeek R1-329
Uama 700
and assigned layers evenly across devices. For
240
Liarna 3-450
Owen 2.5-728
example, for models with 80 layers, layers were
Normalized TPOT
120
Liama 3-608
split 20:20:20:20 at k = 1. 10: 10:10:10
2. and so With large models (>60B) and k 1. PRP degrades to PP and provides no benefit due
100
80
60
to the prefetch-releas conffict. In contrast, with
2 2, Fig. shows that PRP halves TPOT by resolving the conflict and allowing finer-grained overlap of disk loading. draw the same results With small models (<45B) and sufficient memory, the model fits in the cluster without disk offload- ing. Then, increasing k has minimal impact on TPOT, aside from minor overhead from additional communication and kernel launches. This shows that PRP is more effective than PP for 70B-scale models.
20
(.1enddiee//114.]]] A. heterogenous testbed.
0
0.0
2.5
5.0
7.5
10.0
12.5
15.0
17.5
20.0
Figure 2: TPOT of PRP under different rounds k.
4.3 Low MEMORY PRESSURE TO PRESERVE USER EXPERIENCE
Memory pressure is critical for user experience, as high pressure can slow apps or crash the device. For example, if a phone runs LLM service while the user is browsing TikTok, memory competition may cause Tok to lag or crash, prompting the user to terminate the LLM. An implicit advantage of prima.cpp is that it accounts for this by running at lower memory priority than other apps: it reduces RAM usage when apps start and uses more when they stop. By giving up RAM to keep other apps responsive, prima.cpp preserves user experience and protects itself from being killed by the user.
We define memory pressure from the LLM as Amem _aamm__aiiiii lable //mem_mmmmmmmm htt///mmmmtaaaaaeeeeeeeeeeeeeaee total 1 (e.g., 2 GiB used Table 5 shows that exo and dllama pin model weights in RAM, raising memory pressure by over 50% on some devices. This can force the OS to reclaim memory from other apps (e.g., compression swapping), causing lag or crashes, and can still lead OOM for larger models. The root cause is that they prioritize the LLM at the cost of other apps. which users do not want. In contrast, llama.cpp and prima.cpp exhibit low pressure: both pin only minimal KV caches and compute buffers, while prima.cpp further distributes them across devices. Model weights are loaded via mmap and cached pages that can released instantly for other apps. These properties make prima.cpp a practical choice for consumer devices, where LLMs must run alongside user apps.
8 GiB total 25% %
"mee__aee avai lable includes free and reclaimable pages, so its reduction reflects only non-reclaimeble pages, i.c., true memory stress. Appendix A.8 shows each device's 's memory footprint, but not memory pressure, since reclaimable memory (c.g.. page cache) included and can be freed instantly.

---

Published as a conference paper at ICLR 2026
5 CONCLUSION
This paper proposes prima.cpp. the first on-device distributed system to deliver practical speed for 30- 70B LLMs on consumer- grade home devices, achieving 26 tokens/s for 32B models and tokens/s for 70B models with speculative decoding. We introduce PRP to run models exceeding the memory limit and resolve the prefetch-release conflict to reactivate prefetching. reduce bubbles in PRP, we develop the LDA model, which captures the heterogeneity in computing, communication, memory, and OS- speecific reclamation, disk optimizations. We solve via Halda, which performs smart layer partitioning for minimal TPOT and automatically selects the best- performing devices. The system is meticulously designed, and this level of detail surpasses prior work that often relies simplified heuristics. It offers features desired by home users: no extra hardware cost, no specialized hardware, privacy, offline, no queueing or timeouts, models beyond memory limit, practical intelligence and speed, low memory pressure, cross-platform, heterogeneity awareness, automatic workload allocation and device selection, and Wi-Fi ready. These make prima.cpp a compelling choice for home deployment and drive broader adoption of embodied Al home.
handle device heterogeneity and
REPRODUCIBILITY STATEMENT
We release the full prima.cpp implementation with detailed documentation to help readers reproduce the results on their own devices. The repository includes step-by-step instructions and pinned dependencies. A Docker image also provided for quick validation. We document hardware and software requirements, expected runtimes, and common pitfalls. These artifacts enable faithful, independent reproduction of our results with prima.cpp.
ACKNOWLEDGEMENT
We thank Robert Wen from Pinclr Co. for his technical and device support for this study.
REFERENCES
Exo. exo: Run your own cluster home with everyday devices. exo-explore/exo 2025.
https://github.co
Georgi Gerganov. llama.cpp: Llm inference in c/c++. https://github.ccmm/gg/ wtt.:/omo//o///// 1lama. cpp. 2025.
Asghar Ghorbani. Pocketpal ai: An app that brings language models directly to your phone. ail 2025.
https
Agrim Gupta, Silvio Savarese, Surya Ganguli, and Li Fei-Fei. Embodied intelligence via learning and evolution. Nature Communicattions, 12(1))5721, 2021.
Qi Huangfu and JA Julian Hall. Parallelizing the dual revised simplex method. Mathematical Programming Computation, 10(1):119-143) 2018
kvcache ai. ktransformers: A flexible framework for experiencing cutting-edge Ilm inference optimizations, 2025. URL http:///ggttbbbbbo/oeeeeeeeeeeee wwtteeeaaaaaaaaaaaaaaseeeeo
Juhyeon Lee, Insung Bahk, Hoseung Kim, Sinjin Jeong, Suyeon Lee, and Donghyun Min. An autonomous parallelization of transformer model inference on heterogeneous edge devices. In Proceedings of the 38th ACM International Conference on Supercomputing, pp. 50-61, 2024.
Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. In International Conference on Machine Learning, pp. 19274-19286. PMLR, 2023.
Gavin Li. Airllm: scaling large language models on low-end commodity computers, 2023, URL https:// https://github.coo wtt.thi..cmm/ygggai.i./// htt.i///arllm/m/mmm
Zonghang Li, Wenjiao Feng, Mohsen Guizani, and Hongfang Yu. Tpi-llm: Serving 70b-scale Ilms efficiently on low-resource mobile devices. IEEE Transactions on Services Computing, 2025.
10

---

Published as a conference paper at ICLR 2026
Camillo Lugaresi, Jiuqiang Tang, Hadon Nash, et al. Mediapipe: A framework for perceiving and processing reality. 3rd Workshop on Computer Vision for AR/VR at CVPR, 2019.
MLC. Mlc-llm: Universal 1lm deployment engine with ml compilation. mlc- ai m1mmlc-1lmm 2025.
com/
Mohammad Shoeybi, Mostofa Patwary, Raul Puri, et al. Megatron-Im: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08055. 2019.
Bartlomiej Tadych. distributed 1lama, 2025.
Distributed llama.
bttps://gttauub.com//atttttt
Shengyuan Ye, Jiangsu Du, Liekang Zeng, Wenzhong Ou, Xiaowen Chu, Yutong Lu, and Xu Chen. Galaxy: resource-efficient collaborative edge system for in-situ transformer inference. In IEEE INFOCOM 2024-IEEE Conference on Computer Communications, pp. 1001-1010, 2024.
Shengyuan Ye, Bei Ouyang, Liekang Zeng. Tianyi Qian, Xiaowen Chu, Jian Tang, and Xu Chen. Jupiter: Fast and resource-efficient collaborative inference of generative Ilms on edge devices. In IEEE INFOCOM 2025-IEEE Conference on Computer Communications, pp. 1-10, 2025.
Kai Zhang, Hengtao He, Shenghui Song, Jun Zhang, and Khaled B Letaief. Distributed on-device Ilm inference with over-the the-air computation. arXiv preprint arXiv:2502.12552 2025.
Mingjin Zhang, Xiaoming Shen, Jiannong Cao, Zeyang Cui, and Shan Jiang. Edgeshard: Efficient IIm inference via collaborative edge computing. IEEE Internet Things Journal, 2024.
Junchen Zhao, Yurun Song, Simeng Liu, Ian G Harris, and Sangeetha Abdu Jyothi. Lingualinked: A distributed large language model inference system for mobile devices. arXiv preprint arXiv:2312.00388, 2023.
Xuanlci Zhao, Bin Jia, Haotian Zhou, Ziming Liu, Shenggan Cheng, and Yang You. Hetegen: Efficient heterogeneous parallel inference for large language models resource-constrained devices. 6th Proceedings of Machine Learning and Systems, pp. 162-172, 2024.

---

Published as a conference paper at ICLR 2026
A APPENDIX
A.1 PREFETCH-RELEASE CONFLICT
As illustrated in Fig. before computation starts, the OS prefetches 3 model layers to the available memory limit. However, does not stop and continues to load the 4th layer, causing the 1st layer to be released. This prefetch-release cycle repeats, by the end, the last 3 layers are in memory, while the first 3 are not. Then, when computation begins, the 1st layer, which not in memory, triggers page fault, prompting the OS to reload it and the 4th layer to be released. Finally, all layers loaded twice, incurring unnecessary disk I/O without any benefit from prefetching.
Prefetch
Prefetch
Prefetch
Prefetch
Prefetching:
Page Fault
Page Fault:
Page Fault
Page Fault
Computing:
OS Release
OS Release
OS Release
OS Release
Figure 3: Illustration of model layers loaded into memory in PP with prefetching. In this case, the device handles 6 model layers, but its available memory can only hold 3. The green blocks show the layers loaded into memory, while white blocks indicate those not yet loaded.
A.2 How PIPELINED-RINS PARALLELISM SOLVES THE PREFETCH- -EEEEAE CONFLICT
Fig- illustrates a fast-disk device where prefetching is fast enough to complete before computation begins. In this case, with fast disk and layer window size of 2. ! prefetching is fast enough to load layers before computation begins, then 2 computation runs without page faults. Then, 3 the next round of layers is prefetched, replacing the used layers. Steps 2-0 repeat until inference is complete. Prefetching overlaps with other devices' operations, so its latency does not contribute to TPOT. Here, with page faults, TPOT comes only from computation. In other words, disk loading latency fully overlapped.
Prefetch
Compute
Prefetch
Compute
OS Release
Compute
Jump to ?
Figure 4: Illustration of model layers loaded into memory in PRP with a fast disk.
Fig. 5 shows common case with slow disk. In this case, ! prefetching loads only one layer, then 2 computation begins, 3 a page fault is triggered upon reaching the 2nd layer, blocking until it loads. After computation, 4 the device prefetches the next round of 2 layers. but only one layer loads due to the slow disk, and the OS releases the oldest layer. Then, 5 the next round of computation begins, and 6 the 6th layer, another page fault occurs. This cycle of "loading (prefetch) computing
12

---

Published as a conference paper at ICLR 2026
Prefetch
Compute
Compute (Page Fault)
Prefetch
OS Release
Compute
Compute (Page Fault)
Prefetch
Compute
OS Release
OS Release
Compute (Page Fault)
Prefetch
Compute
Compute (Page Fault)
19
10
Release
OS Release
OS Release
Figure Illustration of model layers loaded into memory in PRP with slow disk.
loading (page fault) computing" repeats until inference completes. While page fault-induced loading blocks computation, prefetching helps overlap some latency.
We use a timeline to visualize this overlap. In Fig. green blocks show prefetching that is overlapped, and orange blocks show page fault-induced loading that is not overlapped. In Fig. [a, with a fast disk, disk loading is fully overlapped. In Figs. [6 and 6 with device that has a slow disk. only part of the disk loading overlapped, while other devices are fully overlapped. In Figs. 60 and 66.. although disk loading is not fully hidden, PRP significantly reduces TPOT compared with vanilla PP. In Fig. [6. while prefetching is used, exceeds memory limits and triggers prefetch-relean where the OS releases earlier prefetched layers new ones are loaded, adding disk I/O cost without benefit. This underscores the need to combine PRP with prefetching for higher efficiency.
A.3 LAYER-TO- TEVIICE ASSIGNMENT: FROM LATENCY ANALYSIS VECTORIZED MODEL
Assume there are M devices, where the layer window size for device dm Wm- On device dm, the number of GPU layers 7m is defined as follows: within a layer window of size Ums 7. layers run on the GPU, while the remaining Wim 7ym layers run on the CPU (wm and nm can vary across devices). Our objective is to find vector minimize the TPOT T, which is the sum of latencies from computation Trump, memory access memm
W1,,-.. WMr] }o and a vector
(",,".. nas } to
disk loading Trisk, and communication T comm on each device.
M
T
Tromp + T Tnemm + Trisk T
(11)
DM T,, instead of T (a.(TT.) because Fig. is an idealized illustration. In practice, the OS does not start prefetching immediately after computation ends, and the timing unknown. As a result, device may experience more bubbles and higher page fault-induced latency than expected. This uncertainty prevents us from solving T' = max(Tm before deployment (which is also hard to measure), and historical data is useless due to fluctuating device conditions. Thus, we take worst-case approach, assuming the OS has not started prefetching when computation begins, leading to our objective -MM Tm. Next, we analyze these latencies detail.
Here, we minimize T
Estimation of computation lateney 7.amp The computation latency device d,, is defined as the time taken to process In model layers and the output layer (if d, is the head device), where 1gu layers the CPU. Here, we have in & Wm + min Wm. max(0, 0, Σn - min(u,, R))). gro
the GPU, and the remaining L.n nn layers and output layer run
1e nm + min(n max(0, R Σm min(w;, R))), where n. Wms W >M Wm., and
13

---

Published as a conference paper at ICLR 2026
Init
Round
Round
Round
Init
Round
Round 2
Round
Device
Device
Device
Device
Pipelined parallelism: Homogenesus devices
(b) Pipelined-ring parallelisme Homogeneeus
Init
Round'1 1
Round
Round 3
Round
Device 2*
Device
Device
(c)
Pipelined ring parallellsm: Heteregeneour devices with slow disk on device
Init
Round 1-3
Device
Device 5
Device
Vanilla pipeline parallelism (wlo prefetching):: Heterogeneous devices
device
Round
Device
Device 2t
Prefetch-Reiee ReleechhRRelese
Device
Prefetch Release
Device
Prefetch Release
(c)
Vanilla pipeline parallelism prefetching): Heterogeneous devices with slow disk device
Init
Round
Round
Round 3
Round
Round
Round 6
Device 3
Device
Device
(f) Pipelined ring parallelism: Heterogeneous devices with different layer window sizes
Computing
Prefetching
Page Fault Loading
Device idle
Figure 6: Timeline of (a.b) PRP on homogeneous devices with fast/slow disks; (c.c) PRP on heterogeneous devices with the same/different window sizes; and (d,e) vanilla PP heterogeneous devices with/without prefetching.
R= = mod W. Since the input layer uses a lookup table, does not contribute to computation latency.
To estimate the computation time, we develop model profiler to count the floating Doating-point operations (FLOPs) for each model layer and device profiler measure the floating- point throughput (FLOPS) of each device. Taking Q4K as an example, the model weights are primarily quantized in the Q4K format, though some weights _6_K, Q8 0, FP16, FP32) and three types of backends: CPU, CUDA, and Metal. The FLOPs for each layer Fm and output layer Fout consist of values: F., {f
other formats. Specifically,
consider Q (Q4 K, Q5_K,
pq4k æ95k =aōk =980 <fpl6 sfp32 J.
Jm
Fost (oak 195k Jmnouus fook 1980 fIp16 Jun.ou sip32 each representing the FLOPS under specific quantization format. The FLOPS Sm consists of 3 sets: (S Spuuda S Smetal)) J. with each set consisting spn.o4k epn.q5k _pp..osk .ppp.gon spu.fpl6 spu.fp32
of values (c.g., for CPU, S { Sm .mm Stm Stn sm }) representing
the floating-point throughput for specific bockend-quantizzttin pair. Fm. Juul and Sm can be easily extended for more backends and quantization formats.
With these profilers,
estimate the computation time as follows:
Temmp (I..
7SPU
cpu.g
29
HPU
gpu.g
m
m=
mu,out
cpu.g
(12)
8m
8m
9
14

---

Published as a conference paper at ICLR 2026
Here, "9 refers to GPU FLOPS. If prima.cpp is compiled with CUDA support, sm corresponds to squliun If it runs on an Apple device with Metal enabled, sm. sppu corresponds to metaal. In our
sm implementation, the output layer is executed on the CPU by the master node (m 1).
Estimation of memory access latency Tm This latency consists of three components: (a) KV cache copy time T the time taken to copy the new token's 's cache to the KV cache storage on device m; (b) device copy time m Tele_sPp: the time taken to copy hidden states between the CPU and the GPU (e.g.. CUDA and Metal); (c) device loading time ] the time taken to load data from RAM or VRAM into the processing cores of the CPU or GPU.
For the KV cache copy time, in each token step, new key and value caches are generated with dimensions (hge,, 1) and (h,e,, 1), respectively. Here. h2 and h2 are the number of attention heads for the key and value caches, ej and ey are the embedding size per head for the key and value vectors, respectively. Thus, for generating one token, each layer needs to copy hge; + C. values to the KV cache storage. If values are stored in FP16, each takes bytes, so the total number of bytes to be copied is 2(h + h.) bytes. In the device profiler module, we measure the time of copying 2(hxee + ex) bytes CPU, CUDA, and Metal to obtain en exy_spycpu and ( kvycepsepu Then, the KV cache copy time - can be estimated by (Im- - (5t (m ky-spp + 150h km _y_-spp-
For the device copy time, this latency arises when the GPU is enabled, as it involves copying the input from RAM to VRAM and then copying the output from VRAM back to RAM. Both input and output have shape (e. 1), where is the embedding size. These values are typically stored in the FP32 format. In the device profiler module, we measure the latency for two operations: the time taken to copy de bytes of data from RAM to VRAM, denoted yram vran, and the time taken to copy 4e bytes of data from VRAM to RAM, denoted as tyram-ram, For a sequence of layers within a window, one RAM-to-VRAN so the device copy time for one window is pram-vaa + Im
needed, (yram-ram For device dm, it was assigned Wm = w + min(1, ma R - D==1 min(wy, R))) windows. Thus, the device copy time for dm is 7rdev_ppy = Wm ( (tram-vram + m (yram-rm) (1 m a unified memory architecture (UMA, e.g., Apple M-series) and the CPU and GPU share memory, so no explicit RAM- copy is needed.
copy and one VRAM-to-RAN
copy
m
-E
-TUMA), where NUMA = 1 indicates that device dm uses
For the device loading time, processing cores must load data from RAM/VRAM into registers before executing instructions, which incurs latency. However, the theoretical RAM/VRAM bandwidth does not determine this latency because applications often fail to fully utilize the bandwidth, and multi-level caching also has significant influence. To capture these effects, our device profiler implements an operator to read data from RAM/VRAM into registers. By measuring its latency at data volumes similar to the tensor sizes, we obtain practical throughputs {T Traulaa T.metal }. Next, we calculate the data volume that needs to be loaded into registers during each token step, which typically consists of the weight data and the KV cache. In the model profiler, we record the total bytes of weight data for the input and output layers as b;, by. and for each layer as b. Additionally, the KV cache size for each layer is 2(hbex + hh.c,) where Niuv is the number of tokens for which the cache is stored. Then, the device loading time Tyly_ load can be expressed as Tedeelood Im
T (b- 2(h + ho + b./V+bb.
T.l"" Im==1, where T. depends on the
m
hardware: it equals Timetaal for Metal and T for CUDA, and V is the vocabulary size.
Now we can combine the three latency components and give the formal definition of the memory access latency Immm:
Tmmem
= (I
1sPu
) upycpu
+ 1ggu cpy-gpu + Wm (1r STaIn
+ Ly
prram-ram) )(1 TUMA)
(13)
1m
1gpu
ggu
(b 2(hbex + hocr)mko)) b /V bo
+
-m==-
(14)
T.epu
gpp
cpu
Estimation of disk loading latency Tdisk Prima.cpp is designed to run on memory-constrained devices, so it cannot load the entire model into RAM. To address this, prima.cpp uses mmap to manage model weights. By using mmap, model weights are loaded into memory from disk only when needed for computation, and the OS will release inactive memory-mapper (mmapped) pages when memory pressure is high. This prevents OOM risks but incurs significant disk I/O latency, because evicted pages must be reloaded from disk the next time they are needed. To estimate this disk loading latency, it is necessary to determine the data volume that mmap needs to reload in each token
15

---

Published as a conference paper at ICLR 2026
step. This is a challenging task because different OSs exhibit very different memory management behaviors and high variability.
On macOS (without Metal) and Linux, the OS gradually reclaims memory. When memory pressure is (8% ) c >> davail mmmaped pages are released incrementally until the pressure is alleviated. As result, some weight data remain in the page cache, and the amount of data that mmap needs to reload is max(bio + 2(hg,C + Mo (Im 1 copu davail 0). where blo (lm _ }b + (b:/V + bo) Im= I, 2(bge h, (
moderate, i.c, when bis + 2fyge + thpee
1 )
( is the KV cache size, and d is the compute buffer size. If CUDA is enabled on Linux, the model weights in private VRAM are locked by the CUDA driver, keeping them resident so no disk I/O
occurs. Therefore, the disk loading latency for macOS (without Metal) and Linux can be estimated as
max (biso + 2(hxex + ho (e
+ cPP"
de by b./V)
Tdlsk mScOS(no Metal)
T Linux
gdisk
(15) 15p and sdisk is the random-read throughput of the disk. On Linux, mmap is configured for sequential access, so gdisk is now the sequential-res throughput of the disk.
For Tdisk
m.maoUSino Metaly
When Metal is enabled on macOS, the behavior changes. Metal loads mmapped pages into shared memory, and the OS prioritizes retaining these pages. That is, the OS is more inclined to swap out compress active pages while keeping mmapped model weight pages in shared memory intact. However, when memory is exhausted (with free and inactive pages exhausted, the compression pool nearing saturation, and heavy swap usage), macOS will release these mmapped pages more aggressively. This may cause the entire model weights to be repeatedly reloaded and released. As a result, when the required memory exceeds the total available memory, i.e., when Lm (b./V + ba) 1 2(h + Mukum + c + chPu > davail
mmmmetal device dm needs to reload bmb (b./V + b.)- Im=1 bytes in each token step. Here, duvail mm.metal denotes the maximum working set size recommended
by Metal. By measuring the random-reat throughput gdisk of the disk, we can calculate the disk loading latency for macOS (with Metal) as:
Lmb - (b,/V + bb) Tm==
1 disk m. mS (with Metal) = max
1(mmb b + (b,/V + bb) 1m==1
m
+ 2(hxex + Mepe)mmnnn + su + cP
davail m.metal
(16)
diiss
m
When running on Android devices, the OS prioritizes swapping out inactive pages to disk, such as memory used by background applications, to ensure that the active application runs smoothly. As result, the available RAM for prima.cpp can be higher than expected because the OS swaps cold pages to disk, freeing some memory. Thus, on Android, the number of bytes that mmap needs to
reload is max(bii + 2(hxex + h,ye))au (Im 1 + epu favail
m dm Jewapout 0), where dm dewapout
min(max( (0, bbio + 2(hEek + heex)nkoo(m (9(h)) + c devail), ), min(min)) byles_cam can_aaaanaaaa swap drmap_awail represents the data bytes that are swapped disk, poyee_ccmm, _aa___aaapppp_____ is the data bytes of currently used
memory that can be swapped out, and d avwal_vvill the total available swap space. Then we have:
max(boio + 2(14ex + 1e Me Om
1811 + cpu
gavail
ewapout bv/V)
Tulis Android
(17)
disk
By aggregating them, we obtain a unified expression compatible with cross- ploss-platfomm devices:
Trdik
T disk
mac0S (no Metal)
ImmeOS (no Metal)
T
m. m. (with Metal) ImacOS (with Metal)
+ I Triikinux
IIiuux + Tus IAndroid
(18)
where UmacOS (no Metal), -macOS (with Metal), Linux AAndrod are indicator functions. This expression can be easily extended to include new OSs.
Estimation of network communication latency Tromm, In prima.cpp. devices are connected in a ring, and each device receives inputs from its predecessor, processes them, and forwards the outputs to the next device. After a device completes the computation for one layer window, it transmits the result (e values in the FP32 format, totaling 4e bytes) to the next device for further computation in the next
16

---

Published as a conference paper at ICLR 2026
layer window. During each token step, the number of network communications device dm, equals the number of layer windows, which is Win v + min(1, max(0, R دド= min(wj. R)))
By measuring the latency mm of transmitting 4e bytes between adjacent devices, we estimate the network communication latency on device dm as:
(
L W
T
+ min(1, max((, R
min(wy, R)))) youmm
(19)
By aggregating these latencies, our objective becomes:
M
f g
f out
T
(1m
m cpu.g
+ Im=1
cpu.y
m=1
Q
977
9E
9m
+
(m
18pu
cpy.epu
egpuuky_cpyeppp opy.gpu
+ Wm (tm
(tram-vram vram +
pram--ram (1
m
UMAA
m
mm
im
gepa
(b + 2(hEek + mm + b + bo
Im=1
T
T.
Tm
max + (b./V + bo) ====1 + 2(1xek to + copu
davail bz/V)
+
ImacOS (no Metal)
stick
bmb (b./V + bo) 11
-Ilmbb (b./V + bo) 1m-1 + 2(bge hn ) + cPu + ara duvail metal
+ max
ddisk m
bi
TmacOS (with Metal)
mm <diskT/
max (Im
180 bb + (b./V + bb) Tm== + 2(hx heee)mkol 158 copu
d bi VV)
II
sdisk m
max (Im 1E bb + (b./V + bo) 1m-=1 + 2(1EEE
h Cr )n
ggp + c
davail m
dm
pwaapout
bb/V)
Android
m
+ min(1, ma R
min(wy, R)))
ccomm m
W
Ø=1
M
mut cpu.q Stn
bv/V+ + Bo)
mm= + (mm
18pu
fm cpe.g
+ jky oppcppp
b b 2(hxek cx + ho )
T.
Tma
m=1
gE Q
9EQ
fy
b+ 2(hrex + moeeeven Joke
+
spy.ggg m
Wm
(Lm
(Fram
grram ram
(1 UMA
Joumm
spu.y
T.
q
5m
max + (b./V + bb) I1==1 + 2(hxex + hoe e, mauum + pp
davail by/V) m
Immeos (no Metal)
ødisk
Umb + (b./V + bo) Um
+ max
-(mmb + (b./V + bo) T==1 + 2(heek + hoee Mxolm + Spu + 800
avail mettal
sdisk
be gdisk
UmnacOS (with Metal)
(m
b 2(huet + heev)nku
(b./V + bo) Im 1 + 6P0
pavail
ewapout
LAndroid
+ max
disk m
disk
by gdisk
(ILimux + Lnndrod))
To remove the max operator, we decompose the disk loading latency into multiple terms, sepa- rately accounting for whether memory is sufficient not. Let M be the set of all devices, and M,, M3. M, be the subsets of devices that satisfy the respective conditions Cases 1-4, where Mi n.Ma MM A M4 0 and Mi UM2 M M4 M.
17

---

Published as a conference paper at ICLR 2026
Case 1 (macOS with Metal disabled and insufficient RAM): If Lnb Im=1 + 2(h Mkke + cu davail
(b,/V + bo) =
and diisk
ddisk threshold
then
T
In bmb+(b//++b,. +2(haes teeesthueee that tn ma m t m E M1-
Case 2 (macOS with Metal enabled and insufficient RAM); If Lmb + (b./V + bo) 1m + 2(1xex M c capu M2.
davail
and sdisk
disk threshold
then Trisk im b+ (b,6/ -Vo.//Vboooooooo Amm= m
Case 3 (Linux and Android with insufficient RAMJ: If (Im + damappu
( [6+ 2(hbek + h,cr)mku + +b.:/V
bo) Im=1 c dgvail
Amdroid and sdisk Lüreshold"
then T
(Im
1 )b
=m
2(hxek + he )rke) + (b./V + bo) -Im=1 + cspu dawail
IAndroid
m MM3
Case 4 (OS with sufficient RAM or low disk speed): In these cases, the physical RAM is large enough hold the model weights or the disk speed slow (i.e., sdisk < Sthesshold) As a result, disk loading expected, except for the latency incurred during lookup table access, thus T V m E .M4-
With these cases, we can rewrite the objective function as follows:
b + bo b ddisk
bo
T
IIg.M
(20)
T1
disk
BE
4 b+ 2(hbee + Mbee)mkk
(Im
+
jvy spy.pu m
cpu.g
T
mEM
QEQ
fy gpu.q 8m
b + 2(hxc: + hoev )
18pu
spy-guu
Wm
pram (pram-vram
ram
TUMA )
ypu
qEQ
b
Σ
(m
gpu m
[b 2(hREk + )unol
c
avviil
ewappuu
LAnrroid
gtisk
gdisk
mE.2
M.MMMUM
To further simplify the objective function, we make the following assumption.
L be an integer (i.e., R = 0), where W wy EmEe Wm. Then all devices are Assumption 1. Let W assigned an equal number of windows, and all windows are filled.
Thus, we have Im W. I
1 n.L
Wm f 2कात
L W.
Let V'
b 2(hnek + h O' kvy_epsepu
f jy spycpu गाग EgEo
T. , >
1 Swtin
kpy_ppyg tm
N
b/
T
Sm (1 (ram-vram + prram-ramy (1
TUMA) + ccomm m
K Eqe
b./V T,
b./V
he S
Tig
E EmEM,UMs
where Omyy Bm, Em are platform-specific constants and k is global constant. Then, we add the first general term to the three platform-specific terms and obtain:
2500 genil prrapost
LAnined
T
Σ mEM:
(am + Jum
B'
+ Em
+
Σ
[amm
+
1u + Bm + Em
W
ddisk
W
mE.22
m
L
Σ mEMS
(am
b Wm
(8m
b'
)mm + 5m
L
Σ
am Wm Bm + Em
W
s
sdisk
W
mE.44
This objective a sum over device sets M1, M2, Mg. Each summand involves expressions linear in Wm and n., plus platform-specife constant terms. To clarify the form, define linear function [a, b, c) = awn + bimn + where the platform-specific constants a, b, are independent of decision variables Wmn; Nmn- Consequently, the objective can be rearranged combination of linear functions:
L W.
b' ydisk Y7s
b edisk MyL
T
Σ
f(am +
Emx
Σ flam +
Bms Emr.
mEM:
MEM2
b'
Σ ME
f(am +
8m
disk Sm + Σ m
f(amm Bm, 5m) + K
disk m
meMs
18

---

Published as a conference paper at ICLR 2026
Note that the objective T' is nonlinear because W Emem Wm depends on the decision variables, and the term w introduces nonlinearity. Now, put everything together:
min
T
(21)
s.t.
Wm E Z, Am E Z2o nm S Wm S I.,
(22)
L. = kW, k E Z,00
(23)
W Σ Wmm
(24)
mEM
b(a,b,c) awm + bmm +
(25)
4
4
n Mr = 0. M. i=1
(26)
i=1
W
Wm
(dmmail
) m MI,
(27)
Lb
W
Wm
(mmmmetal
b - E M2.
(28)
Lb
W
Wm
7m
Lb
(drvvil
pawapout
m
IAndroid - -bol)), m E Ma.
(29)
W
Wm
ImacOS (no Metal)
(d
b ) E Mr.
(30)
Lb
Wm ImmoOS (with Metal)
T. mm
(d Javail
ccio - ) m E EM,
(31)
W
(wm mm (I + IAndroia)
<
(d + dssapout IAAndroid - b6hn) m E M4,
(32)
LV
m
Bekko (b./V + bo) [mm== + cou
(33)
W
Rem Teanaa S
(devail - CEPU) Teudaa
(34)
LV
W
Tem Tmmetal S
(duvail
capu bo Im==)
T
(35)
LV
Tom 0, Leuda = O and Iemetal
(36)
Constraint (2 requires that the window size Wm must be a positive integer, the number of GPU
layers 7m must be a non-negative integer, and n'm cannot exceed Wm- Constraint (23) requires that all devices be assigned an equal number of windows and all windows be filled. Constraints (27 29
ensure that devices categorized into sets M1; M2, M2, M3 meet the memory condition outlined in Cases
1-3. Similarly, Constraints (30 (32) ensure that devices assigned to set M4 meet the memory condition outlined in Case 4. boco in Eq. (33 is flatform-independeer
constant. Constraints (34 35 ensure that the VRAM used by CUDA or the shared memory used by Metal does not exceed the available capacity. Here, daviil denotes the available GPU private memory for CUDA, and davaill dys.mettal denotes
the maximum working set size recommended by Metal.
This is integer linear fractional program (ILFP) because the numerator is linear function of the decision variables Wm, Phes and the denominator W is also a linear function of Wm. Moreover, the constraints are linear inequalities. The platform indicators TmacOs, T Jinuux I Android, Meuua Timeat J are known priori, and they activate/deactivate corresponding linear constraints for each device.
Next, we transform the model into vectorized form. Let the decision variables be TT [wi, W2, Wrr, n2" [ni, 122, .. AM,, and the coefficients a, b, c be:
B' E M1
am +
0 m EM1
Em m E Mi
Qm
+
m E M2
Bm m E M2
Em m E M2
mm
Bm
m E EM3
Em m E M3
O'm
E M3
3m m E MM
Em m E Mr
Om m E M4
19

---

Published as a conference paper at ICLR 2026
To apply constraints to the subset of and corresponding to M3, we define diagonal matrices P = diag(-IM::- -IM.. -I,,, PM. Pit May) Pn = where IM IM,, LMs are identity matrices and M. M2 Mss are zero matrices corresponding
diag(0 0a MMs MM -
the subsets MI, M2, and PMa PM.: PM are diagonal binary matricies (i.e., selection matricies) corresponding to the three constraints 32 within the subset M4. To construct PM MM, P M, we define bi- nary vector PmacOs. where a value of indicates that the current device is running on macOS and value of 0 indicates otherwise. The number of elements in PmacOS matches the number of de- vices in the set M4. Similarly, we define binary vectors PLinux PAnduidd Pinetal- Thus, we have PM. diag(Pmmcss (PmacOs (1 Pmmui)), P. diag(Pmmeet Pmmeta)); ) P PLimux PAndroid-
To handle constraints 34 35
define P similar diagonal binary matrix, with elements set to one for devices with CUDA or Metal support. Specifically, we let P -peuda + panetal where
pemla ddiag(,,,, 0. Poula P and pametall diag(0M, IM2i OMs Pmota)).
Let the decision variables be whi, [wm m E Ma). n!, -[m | m E MMa,, w. [wT whe w wh,hn" n = [ n n.gg the RAM upper bound be nn
davail
beio
m Mı
pavail m.
baio m
cupu m E M2
metal
Javail qewapout
Android
ieto m Ms
1
2 Lb
daviil m
+ beio m M4
Avail metal
+ b + c m E Mr
devail
swapour
LAndroid + brio m E M4
m
and the VRAMishared-memory upper bound be zEPU
[=]
Epu) "M / where
0,
leuuaa = and Lettal = 0,
1
mn,cuda dwval mmmmetal 8pu pavail m. metal
eSpu
leuda 1,
Lb
if Tmmetal and + 1, Umettll and m
epu
boz
The problem model can then be reformatted as:
L a. w + bT
n c TT.
min w.n
+ K,
(37)
eT 1
s.t.
Wm E Zz 1m Z20, 0m
S Wm < L,
(38)
k(e w) = 0, k E Z000
(39)
P - w' + Pa n' + e T
w z < 0, pepu zap e m papa S 0.
(40)
(41)
Table Juumaa/sss the key symbols used in this paper.
A.4 GENERALIZATION TO A NEW HETEROGENEOUS TESTBED
To show the generalizability of prima.cpp, we repeated the experiments of Table on another testbed. This testbed includes: host PC (5 GB RAM, 1080 TI GPU with 11 GB VRAM), Mac Mini (10 GB UMA RAM), a laptop (23 GB RAM, 3060 GPU with 6 GB VRAM), and a Redmi phone (7 GB RAM). Here, all RAM/VRAM values refer to available memory rather than total capacity. We ran llama.cpp on the laptop (with a 3060 GPU), and exo on the host PC, Mac Mini, and laptop (since exo not supported on the phone). Meanwhile, dllama and prima.cpp on all four devices. Their TPOT
20

---

Published as a conference paper at ICLR 2026
Table 6: Summary of key symbols and their explanations.
Symbol
Explanation
M
Number of devices.
Wm
Layer window size on device dm-
Pm
Number of GPU layers on device dm-
T
The optimization objective (i.e., TPOT).
bm
Total model layers processed by device dm
Total GPU layers processed by device dm.
Total number model layers.
W
Total layer window size across all devices (W mmm Wm)).
lu, Ity
Number attention heads for keys and values.
Ck Po
Embedding size per attention head.
Embedding size.
by, bn
Bytes weight tensors per layer, and of input/output tensors.
Number of tokens stored in the KV cache.
Vocabulary size.
Available memory on device dm-
Buffer sizes for CPU/GPU computations.
dink
Disk read throughput for device dm.
iiheehold
A threshold for disk read throughput. the throughput is below this threshold, the disk is considered slow.
Ms.
Set assignments, corresponding cases 1-4.
a, b,
Coefficient vectors for objective function.
Pw, Pa
Diagonal binary selection matrices for constraints on w and n.
P
Diagonal binary matrix that indicates whether device uses GPU.
w
Extended vectors w and n.
2, 2
Vectors of RAM/VRAM upper bounds for constraints.
Table 7: TPOT (ms/token) on the new testhed.
Model (4-bit)
llama.cpp
dllama
prima.cpp
Llama 3-8B
27
OOM
875
27
Llama 3-14B
199
67
Llama 1-30B
469
308
Llama 3-45B
623
328
Llama 3-60B
12762
671
Llama 1-65B
20073
703
Llama 3-70B
23834
OOM
OOM
718
values are listed in Table 7 following a trend similar to Table In this case, Ilama.cpp encounters VRAM bottleneck at 14B, and prima.cpp matches (only at 8B) or achieves the fastest speeds across 8-70B. Additionally, due to the limited RAM of the host PC, the weight files. This supplementary experiment supports the generalizability of prima.cpp.
out of memory while loading
A.5 GENERALIZATION TO A HOMOGENEOUS TESTBED
We also compared llama.cpp, exo, dilama, and prima.cpp on homogeneous, low-resource cluster. Since we do not have identical home devices, we used Docker to create four identical Linux containers on server with an RTX 4090. Each container had 8 CPU cores, 8 GiB RAM (4 GiB available), 5 GiB VRAM, and 2 GB/s disk read throughput. Table reports their TPOT results.
Prima.cpp achieves substantial speedups over llama.cpp in this low-resource testbed, delivering more than 100x improvement on the 14B model. Exo, however, crashes even on the 8B model due to its full-precision GPU backend, making unsuitable for low-end devices. Although exo's TPOT at 8B cannot be measured directly, we can infer its lower bound based on the performance of llama.cpp. For example, llama.cpp runs the full 8B (Q4K) model on 5 GiB of VRAM at 15 ms/token, implying that exo's 's TPOT would be at least this high. Furthermore, prima.cpp matches the speed of llama.cpp this scale, suggesting that it is least fast exo. Finally, despite the near-zero communication
21

---

Published as a conference paper at ICLR 2026
Table TPOT (ms/token) on a homogeneous testbed.
Model (4-bit)
llama.cpp
dllama
prima.cpp
Llama 3-8B
15
OOM
495
15
Llama 3-14B
2243
888
21
Llama 1-30B
6870
52
Llama 3-45B
10563
OOM
195
Llama 3-60B
14652
OOM
391
Llama 1-65B
15798
502
Llama 3-70B
17590
OOM
OOM
1128
delay from the Docker bridge, dllama remains slow, indicating that TP can still be inefficient even on homogencous, high-bandwidth clusters.
A.6 RUN PRIMA.CPP ON LLAMA 1&3, QWEN 2.5, QWQ AND DEEPSEEK R1
Fig. plots TPOT and TTFT for Llama models as curves. The prima.cpp curves (solid and dashed diamonds) consistently lie at the bottom, confirming that prima.cpp achieves the lowest TPOT and TTFT across 8-70B. Disabling prefetching or Halda increases latency, but the performance drop is substantially larger when Halda disabled. This suggests that Halda contributes critically the speedup than prefetching.
Lazeficy
TToken Latency (milliecond)
Caloncy -ery--eeenne prima.co
Latency twto halda)
arima prefetch)
TTFT (millisecond)
10a
10a
10a
10a
TIFT 488
TTPT cllmm
Llama 3-80
Uama 3-148
Liama 1-308
Uama 3-458
Liama 3-608
Uama 1-658
Liama 3-708
Figure 7: TPOT and TTFT for Llama models across 8-70B.
Table 9: TPOT (ms/token) for Qwen 2.5, QwQ, and DeepSeek R1 models across 7-72B.
Model (4-bit)
llama.cpp
cxo
dllama
prima.cpp
Qwen-2.5-7B
14
86
1314?
14
DeepSeck-RII 1- Distiill-Qwen-11
14
68
DeepSeck-R1 1 Ri---stlll---amm---
14
77'
1169
14
Qwen-2.5-14B
23
31710
15655
23
DeepScck-RY RR--istill-Qwenn 14B
24
23475
Qwen-2.5-32P and QwQ-32B
224
OOM
28322
89
DeepSeek-RII --iit--Rl-------l----- Distill-Qwen-322
232
OOM
93
DeepSeek-RI ---tss-----iiiil--aaaaaa
10978
OOM
OOM
Qwen-2.5-72B
12227
OOM
00M²
867
TPOT is lower because exo provides full-precision Qwen models, whereas the
DeetSeek-distilled models quantized 3-bit.
As dllama doesn't support Qwen-2.5, we use a Qwen-3 model of same size comparison.
22

---

Published as a conference paper at ICLR 2026
extends the evaluation to more models, including Qwen 2.5, QwQ, and DeepSeek R1 (distilled versions) across 7-72B. The results are consistent with Table 4 For small models (<14B), Halda places prima.cpp D3, so TPOT and TTFT match those of llama.cpp, and 68- 98-94 times faster than dllama. For larger models, Halda distributes the workload across GPUs and CPUs in a smart way, enabling prima.cpp maintain sub-second TPOT even 70B, whereas lpp can only offload locally, triggering disk offloading much earlier and causing TPOT to explode.
special case arises for exo, which only provides an MLX backend for these models. Among the devices in Table D2-D6 failed because Tinygrad only supports Llama-series models and exo can only use D1. For small models, since D1 is less powerful than D3, exo is 6x slower than prima.cpp. At 14B, DI runs out of memory, and TPOT spikes due to disk swapping. As the model size increases further, exo fails with OOM. This highlights that pinning model weights in memory can induce uncontrolled swapping, whose overhead is often much more severe than proactively managing disk offloading via mmap.
A.7 SELECT DEVICES TO BUILD THE MOST POWERFUI. CLUSTER
Existing systems require clusters with sufficient aggregated RAM/VRAM, often forcing users to add more devices to support larger models. However, assembling enough devices is difficult for households. This raises two questions: (a) Should we collect enough devices to meet the model's ss needs? (b) Do more devices always lead to better performance? ? The answer is Fig. g iillsttrates how the TPOT of prima.cpp on Llama 3-70B (Q4K) is affected by the device set. For each device set, the layer partitioning across device-backent pairs is determined by Halda.
80
D1-CPU
D3-CPU
DS-CPU
10000
D2-CPU
D3-GPU
D6-CPU
70-
D2-GPU
D4-CPU
TPOT
Number of Assigned Layers
60
56
TPOT (ms/ttoken)
50
6000
40
38
38
38
30-
4000
25
23
23
23
20-
13
2000
10
3
Number Devices
Figure 8: Layer partitioning and TPOT over different device sets.
For question (a), with only devices (D2, D3, and D5), the aggregated RAM+VRAM is 38 GiB, which is insufficient to hold the 40 GiB Llama 3-70B (Q4K) model. However, thanks to the fast SSDs on D2 and D3, mmap can reload model layers quickly, and prima.cpp achieves the lowest latency. Thus, prima.cpp does not require memory sufficient to hold the entire model.
For question (b), when we increase the number of devices 50 GiB, which is enough to hold the entire model. However, the TPOT is lower with just 3 devices, because the devices D4 and D6 have weak CPUs and slow disks, creating bottlenecks. This shows that more devices do not always result in faster inference.
the aggregated RAM+VRAM reaches
This raises a new question: (c) If the user has a device with a weak CPU or a slow disk, should it be removed from the cluster? Intuitively, such a weak device would be bottleneck. However, in cases of severe memory shortage, it does help. For example, D2 and D3 are devices with a GPU, a strong CPU, and fast disk, while D5 is weak device. As shown in Fig. 8 adding D5 reduced TPOT by roughly half because D3's s disk loading latency (which was heavily overloaded) dominated the impact of D5's weaker CPU.
23

---

Published as a conference paper at ICLR 2026
This raises more questions: if users have some weak devices, which ones should be removed? More generally: (d) Given a set of heterogeneous devices, how can we select a subset to build the best- performing cluster? This is challenging due to the uncertain number of devices to be selected, the highly heterogeneous cluster, and the various factors like CPU, GPU, RAM, VRAM, disk, network, and even OS that significantly affect inference speed. Fortunately, Halda offers an easy solution: include all available devices, then remove those with only one assigned layer, since Halda indicates that excluding them would improve speed. This procedure made automatically in prima.cpp.
A.8 MEMORY FOOTPRINT ON DEVICE-BACKEND PAIRS
Fig. 9 shows each device's 's RAM and VRAM usage to illustrate why prima.cpp achieves faster speed and prevents OOM. As exo and dllama don't It support Llama 14B-65B and encounter OOM at 70B, the memory usage for 14B-70B in Figs. 9b and 9c is estimated based on system behavior and memory load at 8B. Fig. 9a was discussed in Section
Figs. 9b and 9c show that exo and dllama consume high memory. Exo mixes multiple backends, using MLX on macOS for 4-bit computation and Tinygrad on Linux, where model weights are loaded in 16-bit on the CPU and decoded to 32-bit on the GPU. In case, D1(8 GiB UMA RAM) and D2 (8 GiB VRAM) get the same number of model layers, yet D2-CPU uses 4x more RAM and D2-GPU 8x more VRAM than DI-GPU, This results in high memory usage on Linux devices, increasing the risk of OOM.
For dllama, uses TP and Q40 quantization to distribute and compress memory usage, but lacks GPU support, so all memory load is on RAM, and inference speed is limited. It has similar memory usage across devices due to uniform tensor splitting, which causes problems on low-memory devices. In Fig. 19c] for 30B model, D22 and D3 have more available RAM, while D1 and D4 have less. To allocate enough memory, D1 and D4 must free more active pages or swap out application data, which can slow user apps even crash the system. In such cases, OOM may be the safer outcome. Additionally, D3 (the head device) loads the entire model before slicing and distributing it, taking significant RAM and making it more prone to OOM.
In contrast, prima.cpp optimizes workload distribution with Halda and prevents OOMs with mmap. Although the solution to the problem Eqs. -1(5)) is hard understand, we can observe Halda's 's preference from Fig. dd powerful GPUs > weak GPUs > powerful CPUs > fast disks. For example, 8B-30B, Halda first fills D2-GPU and D3-GPU. At 45-65B, it fills D1-CPU to D4-CPU. Lastly, the remaining layers are placed on D2-CPU and D3-CPU because they have fast disks. This assignment prevents weak CPUs and slow disks from being used. Finally, only D2-CPU and D3-CPU experience RAM overload, but this does not cause OOM because the OS will free inactive mmapped pages instantly and prefetch model layers in advance. With fast disk reads, disk loading latency stays low, ensuring minimal TPOT, which is exactly the result of our optimization goal (Eq. I.
Beyond the advanced workload distribution via Halda, prima.cpp also prevents memory waste. With mmap, it loads only the required model layers instead of the full model, climinating the need for model slicing (which may cause OOMs like in dllama). Additionally, it supports model inference in Q4K format across heterogeneous platforms, eliminating the need to decode back to 16-bit or 32-bit, thereby further reducing RAM/VRAM usage.
A.9 RUN PRIMA.CPP WITH SPECULATIVE DECODING
Prima.cpp can be further accelerated with speculative decoding. In prima.cpp, we add support for this technique. Since the draft model is small (0.5-3B), we run it as standalone process on the head device and set the most powerful device as the head. The draft model predicts 5 tokens per step, which are then verified in batch by the larger target model. The testbed consists of 4 Linux devices, each with an 8-core CPU, 8 GiB RAM, and a sequential disk read throughput of 600 MB/s. Two nodes are equipped with 4090 GPU, but each is limited to 11 GiB VRAM.
Table 10 compares the TPOT of llama.cpp and prima.cpp with and without speculative decoding. With speculative decoding, prima.cpp delivers an additional 25-45% latency reduction across 14-70B models. For example, Qwen 2.5-32B speeds up from 18 to 26 tokens/s, and Llama 3.B-70B from 1.2 to 2.3 tokens/s. At this throughput, a 32B model meets the 20-50 tokens/s throughput commonly required by LLM agents, facilitating broader deployment of frontier LLM agents on home devices.
24

---

Published as a conference paper at ICLR 2026
120
120
D1-GPU
D1-GPU Bound
D3-GPU Bound
D2-CPU
D2-CPU Bound
100
D3-CPU
100
D2-GPU
D3-CPU
D3:CPU Bound
Memory Usage (GB8)
Memory Usage (Gi8)
D3- GPU
D3-GPU Bound
80
20
Llama 3-80
Liama 3-148
Liama 1.308
Liama 3-458
Liama 3-608
Liama 1-658
Liama 3-708
Liama 3-88
Llama 3-148
Lama 1-308
Liama 3-708
(a) llama.cpp
(b) exo
120
D1-CPU
D1-CPU Bound
DZ-CPU
D2-CPU Bound
100
D3-CPU
D3-CPU Bound
D4-CPU
D4-CPU Bound
Memory Usage (GiB)
80
60
20
Llam 3-8B
Liama 3-148
Liama 1-30B
Liama 3-45B
uama 3-60B
Liama 1-650
Liama 3-708
(c) dllama
120
D1-CPU
D3-GPU
D2-GPU Bound
DZ-CPU
D4-CPU
D3-CPU Bound
100
D2-GPU
D1-CPU Bound
D3-GPU Bound
D3-CPU
D2-CPU Bound
D4-CPU Bound
Memory Usage (GiB)
80
60
20
Liama 3-88
Liama 3-148
Uama 1-308
Liama 3-458
Liama 3-60B
Liama 1-658
Liama 3-708
(d) prima.cpp
Figure 9: Memory footprint on each device-backend pair.
25

---

Published as a conference paper at ICLR 2026
Table 10: TPOT (ms/token) for llama.cpp and prima.cpp with and without speculative decoding.
Model (4-bit)
llama.cpp
prima.cpp
prima.cpp (with speculative)
Qwen-2.5-7B
20 + 0
20 + 0 ms
18 + ms (draft 0.5B GPU)
Llama 3.2-8B
20 1 0
20 1 ms
20 + ms (draft 1B on GPU)
Qwen-2.5-14B
ms
36 ms
27 ms (draft 1.5B on GPU)
32 1 ms
32 : 0 ms
22 1: ms (draft 1.5B on GPU)
Qwen-2.5-32B
6551 + 38 ms
55 1 0 ms
38 4: 5 ms (draft 0.5B on GPU)
DeepSeck-RI BarekkRR--iiiee D-stiillLlamman
20118 + 69
859 = 40
593 + 65 (draft 8B CPU)
Llama 3.3-70B
20083 ms
803 = ms
442
(draft
GPU)
Qwen-2.5 2.5-22
21600 + 80 ms
963 = 18 ms
544 + 19 ms (draft 3B on CPU)
A.10 ABLATION STUDY OF HALDA WITH HEURISTIC SCHEDULER BASELINES
While prior work has studied layer partitioning across heterogeneous devices, these approaches often rely strong and unrealistic assumptions that limit their in real home clusters: (a) They only support workload scheduling within GPU clusters; (b) They require the cluster's aggregated VRAM to the model's needs; (c) They assume all devices are necessary. For (a) and (b), most households cannot afford an expensive machine with sufficient VRAM, let alone a GPU cluster. For (c), since home devices can vary widely, it could be better to drop weak devices than keep them.
The novelty in Halda lies in supporting GPU/CPU-mixed clusters while relaxing memory require- ments, and in explicitly accounting for disk latency and OS-specific memory behaviors. Moreover, Halda can automatically select the optimal subset of devices from candidate pool to serve an inference engine that runs at maximum speed. We call this ability "device selection". No previous work has considered these features, but they are necessary in home clusters.
To better demonstrate Halda's s effectiveness and novelty, we conduct a supplementary experiment comparing it with two heuristic scheduler baselines:
MemSched: Partitions layers according to RAM/VRAM ratios. This method originates from and is also the default strategy in prima.cpp (w/o halda).
PerfSched: This method originates from Galaxy. First, partition layers by devices' compute power, then migrate OOM layers to other devices. This can still hit OOM, so we add greedy fallback: If OOM persists, offload to CPUs (in proportion to CPU compute power).
The testbed uses Mac Mini (16GB RAM), a host PC (16GB RAM and 11GB 1080TI GPU), a laptop (32GB RAM and 6GB 3060 GPU), and a Redmi phone (16GB RAM). In this setup, aggregated VRAM is insufficient, but aggregated RAM+VRAM is sufficient, so OOM does not occur.
2.5
MemSched
PerfSched
TPOT (s/token)
Haida
1.0
0.5
8B
14B
30B
458
60B
65B
70B
Llama Models
Figure 10: TPOT of MemSched, PerfSched, and Halda on Llama models across 8-B.B.
As shown Fig. 10 on Llama 3-70B (Q4K), Halda runs 3.3 x faster than MemSched and 1.6x faster than PerfSched because it learns drop the 4th device for faster speed. We also tested models of smaller sizes, and Halda always outperforms heuristic schedulers. For 8-45B models, Halda is 1.1-1.5x faster than PerfSched, 4.5-8.8x faster than MemSched. For 60B models, 1.6- 1.6-1.8x faster than PerfSched, 1.9-3.3x faster than MemSched. This demonstrates the effectiveness and novelty of the proposed Halda scheduler.
26

---

Published as a conference paper at ICLR 2026
A.11 ABLATION STUDY ON MULTI-REQUER WORKLOADS
This paper focuses on single-request inference because: (a) home/mobile deployments typically have low concurrency, and (b) continuous batching not our contribution. However, prima.cpp does implement continuous batching and can handle concurrent requests.
In our implementation, continuous batching merges active requests the token step. Since PRP also operates on token-step batches, PRP and Halda work unchanged in multi-user/requee settings. Each request maintains independent KV cache. We run Llama-3-14B and 70B models with 1-40 concurrent requests on 2 heterogeneous testbeds:
Low-mem testbed: including host PC (5 GB RAM, 1080 TI GPU with 11 GB VRAM), Mac Mini (10 GB UMA RAM), a laptop (23 GB RAM, 3060 GPU with 6 GB VRAM), and Redmi phone (7 GB RAM);
Medium-ment mem testbed: including four Linux PCs, one with a 1080 Ti GPU (11 GiB VRAM) and 24 GiB RAM; one with 2080 Ti (11 GiB) and 32 GiB RAM; one with 3090 (24 GiB) and 16 GiB RAM; one with a 4060 Ti (8 GiB) and 32 GiB RAM; and a Mac mini with 16 GB of unified memory.
100
Throughput (tokens/s)
80
14B on medium mem testbed
14B on low- mem testbed
70B on medlum-mem testbed
40
20
0
10
20
30
40
Num of requests
Figure 11: Token throughput over concurrent requests low-/nediunn-memrry testheds.
As shown in Fig. when the model is large (70B on the medium- testbed) or memory is tight (14B on the low-mem testbed), throughput grows sublinearly, peaks around 8-16 concurrent requests, then declines. This behavior is expected: higher concurrency consumes more memory for KV caches and compute buffers, leading to more frequent model weight eviction and reloads. When resources are more abundant (14B on the medium-ment testbed), throughput scales nearly linearly at first, saturates around 32 concurrent requests, and then drops. This range covers typical houschold usage, which is often 1-8 concurrent requests.
A.12 SYSTEM DESIGN AND IMPLEMENTATION
illustrates the overall framework design of prima.cpp. Our goal is to enable efficient 70B-level LLM inference on heterogeneous, coontrycconttrained home devices while minimizing per-token latency. In prima.cpp design, achieve this through:
Fig. 12
PRP:
new mmap- and prefetch-based parallelism paradigm that supports low-cost disk offloading to overcome limited memory and resolves prefetch-release conflicts to maximize kernel efficiency.
Halda: An adaptive scheduler that determines the optimal layer partitioning rule according device heterogeneity.
The system should also support:
Multi-request batching: dynamic batching for concurrent tetti-uuer//uuiirrqqueee infer- ence.
27

---

Published as a conference paper at ICLR 2026
mulli-mquunt
IBB
Crall model
Tokens/ activations
Hodel layero nunning on CPU
Proffling meto data / Portition rulos
Model lagune nurning
bokiens
T:b
Contirumue
Hanee
Misappedd weight londing
carche BAMA/VAA ViAM
batching
KV coche
External storago
Request p00s
Arte
Schoduior
Matariule
BLonn
GPU
GPU
CPU
GPU
CPU
GPU
GPU
GPU
Modier layers
Dound
Mode. layers
Hound
Acta
Model
Model layoru
Round Round
Found
10:00
Round
layers
Round
2
mmmag
KV
Metariue
KV
Motairule
KY
Motahrue
mmag-
Device
Device
Figure 12: Overall system design of prima.cpp.
Speculative decoding: uses draft model to transform per-token generation into batched token evaluation, thus improving throughput further.
Device join, dropout, and selection: adds or removes devices runtime to enable fault recovery and automatic device selection.
Cross-LAN connectivity: enables distributed inference across multiple local networks.
Following the system design illustrated in Fig. 12 we describe how each component is implemented:
Mmap- and prefetch-based PRP. PRP executes the model in striped segments, dividing all layers into small chunks that are distributed across devices in a rotating, multi-round schedule. For example, an 80-layer model, during the first round, devices 0-3 process layers 1-10, 11-20, 21-30, and 31-40, respectively; then in the second round, they process 41 41-50, 51 51-60, 61-70, and 71-80, respectively. Within each window, some layers run on the CPU and the rest on the GPU.
During inference, only KV caches, compute buffers, and GPU-resident layer weights remain in RAM or VRAM. CPU-resident weights are lazily loaded via mmap and automatically released when memory becomes scarce, allowing prima.cpp to maintain low memory pressure even for very large models. KV caches are sharded across devices/backends based on layer partitions. Throughout execution, only activations exchanged between devices backends, and no migration of KV caches or model weights occurs.
Profiling and heterogeneity-awwe evenly across devices and backends in Fig. 12 but this is suboptimal in heterogeneous settings. To solve this, we introduce the Halda scheduler, which determines both the size of each device's 's model segment and the split between CPU and GPU layers, so as to adapt to diverse compute, memory, communication, and 1/O speed conditions.
layer partitioning. To ease description, we divided model layers
We implement a profiler on each device to profile both model and hardware characteristics, including model FLOPs and size, device FLOPS, RAM capacity and bandwidth, network bandwidth and latency, disk I/O speed, OS, and execution backend. These metadata are propagated along the ring topology and gathered at the head device, where the Halda module will use them construct the LDA optimization model and solve it using our proposed solver to obtain a layer partitioning rule that satisfies RAM and VRAM constraints while minimizing TPOT. The resulting rules are then broadcast back along the ring topology to all devices. Profiling and Halda scheduling typically occur during system initialization or when the task queue is idle, introducing only 10-12 10-12 ms overhead, which negligible compared to inference time.
Multi-request batching. This paper focuses on improving single-request inference, but the proposed PRP and Halda are compatible with batch inference. On the head device, job queue is maintained, where multiple jobs are batched into nicro-bbacher using continuous batching and then executed by prima.cpp to produce batched outputs. Notably, unlike PP, which may process multiple micro- batches concurrently. PRP handles only one micro-batch at a time to avoid contention for memory and disk across micro-batches..
28

---

Published as a conference paper at ICLR 2026
Speculative decoding. We support speculative decoding to further improve throughput. When enabled, a draft model runs to generate draft tokens quickly. Since the draft model small (0.5-8B), it rums solely on the head device, and the draft tokens are consumed by the target model's input layer on the same device. The larger target model runs on multiple devices using prima.cpp and verifies the draft tokens. Notably, speculative decoding keeps model outputs unchanged and therefore does not impact output quality or perplexity.
Node join, dropout, and automatic device selection. Devices communicate through unidirectional ring throughout the workflow, which provides additional flexibility;
Node joining: a new device can be added by linking it between its predecessor and successor in the communication ring.
Node dropout: when a device leaves, its neighboring devices reconnect directly.
This design enables joining and removal of devices during runtime without disrupting other devices' communication.
Combined with Halda's layer partitioning, prima.cpp can automatically exclude low-load idle devices (e.g., those assigned 0-1 layers) from the ring. so we can achieve automatic device selection and reduce ring size further accelerate inference.
Cross-LAN collaboration. By adding a proxy device in the ring with zero load, prima.cpp can form cross-LAN ring topology to involve more devices. For example, one device can reach another LAN via a VPN connection. it can serve as a proxy, linking the two LANs into single closed communication ring. This enables prima.cpp to scale inference across multiple local networks.
A.13 ENERGY CONSUMPTION: FROM CLOUD TO LOCAL & STANDALONE TO DISTRIBUTED
Energy consumption is key metric for both cloud and edge serving. In this section, we quantify energy per 1K output tokens and break it down into compute kernels, memory/PCle, disk I/O, communication, and non-zero wait power. On single device with tight memory, prefetch-release conflicts cause frequent page-fault reloads, stretching I/O waits while hardware still draws non-zero idle power. In prima.cpp, pool memory across devices and run faster, so reloads and wait energy drop substantially.
To compare the energy consumption of local single-/multi-device inference and cloud inference, we set local testbed with 4 mobile devices:
Mac MI Pro (8 cores, 8 GiB UMA RAM);
Mac M3 Pro (12 cores, 18 GiB UMA RAM);
Linux laptop (8 cores, 4 GiB available RAM, and 4090 GPU with 16 GiB VRAM);
Honor Pad (8 cores, 5 GiB available RAM).
datacenter server with 8 RTX 6000 Ada GPUs. Power consumption was recorded via PowerMetrics on Mac devices, NVIDIA Power Counters and powercap on Linux devices, and Ludashi on the Honor Pad. Fig. 13 compares the energy-cons corgyycoonsumption baseline of running llama.cpp on single device with the per-device and total energy consumption when using prima.epp.
For cloud inference,
Local single-device vs. prima.cpp distributed. Across the four local devices, switching from single-device llama.cpp to distributed prima.cpp reduces per-device energy by 91-99% and total energy by 57-90% for KK tokens. Take the GPU laptop an example, its average power reduces by 37% (less disk I/O and wait time), time-to- K-tokens reduces by 86%, hence the total energy reduces by 91%. In the distributed run, the GPU laptop contributes 86% of total energy, while others account for only 14%. This indicates that assigning a few layers to low-power helpers can shorten the critical path at low energy cost.
Local serving vs. cloud serving. On the cloud GPU, abundant VRAM eliminates disk/network power and greatly shortens running time, despite higher average power, the total energy is slightly lower (28%) than a local cluster. However, if we take into account the cooling power consumption of data centers, the energy gap narrows. Regarding energy cost, many countries/regions use tiered
29

---

Published as a conference paper at ICLR 2026
125
120.2
llama.c llama.cpp
prima.cpp cpp
Wh /K tokens
100
prima.cpp (total)
75
50
40.7
42.9
29.3
25
10.8
9.1
1.0
0.5
0.3
GPU Laptop Honor Pad
Mac M3
Mac M1
Total
Cloud
Figure 13: Energy consumption for local single-/multi-device inference vs cloud inference.
electricity pricing: residential prices about 30% cheaper than commercial prices, which offsets the energy-cost gap between cloud and local serving.
To up, prima.cpp significantly reduces energy consumption compared to local single-device serving, while maintaining comparable energy use and cost to cloud serving. If we consider token pricing, bandwidth fees, cloud failure, and cloud costs, local serving with prima.cpp would be more cost-effective.
A.14 PERFORMANCE BREAKDOWN FOR PREFETCH -EEEETTH-RELEASE RELEASE CONFLICTS, PRP, AND HALDA
In this section, we include a performance breakdown to show the existence and impact of prefetch- release conflicts, well as the latency composition of disk, compute, communication before and after applying PRP/Halda.
A.14.1 AN INTUITIVE LOOK AT THE PREFETCH-RELFFASE CONFLICTS
To show that prefetch-release conflicts exist, we should analyze the latency composition of compute time and prefetch time. Compute time refers to the execution time of the computation graph. During kernel exccution, the model weights required by a kernel may have already been reclaimed by the OS, triggering page faults and reloading from disk. Thus, measured compute time will include this reloading overhead, which belongs disk I/O and cannot be separated. In addition, when cache hit rates are low, kernels incur frequent page faults and off-chip memory access penalties, so even if disk 1/O time is excluded, kernel efficiency drops.
Once we clarify that compute time includes disk reloads, we can measure both compute time and in-compute reload bytes to observe how reloading affects its time. The following experiment was conducted an RTX 4090 laptop with 16 GiB VRAM, rumning Llama-3-70B Q4K, with available memory limited 32 GiB, 24 GiB, 16 GiB, and GiB. To prove the existence of prefetch-relest conflicts, only need to show that after prefetching, in-compute reload bytes still occur during the compute phase, indicating that some prefetched weights were later reclaimed due to memory shortage and must be reloaded via page faults. The results in Fig. 14 clearly confirm its existence.
Fig. 14 also reveals several key details:
With sufficient memory (32 GiB RAM): the model fits entirely in the page cache and VRAM. No prefetching is needed, kernels have full cache hits, make compute time minimal.
24 GiB RAM: most model weights remain cached, but about 0.6 GiB of them are re- claimed and reloaded, slowing computation, while disk I/O from prefetching still dominates overall latency.
At 16 GiB RAM: memory becomes scarce. During prefetching, the last 5 layers loaded evict the first 5 layers. When computation begins, those early layers must be reloaded, which in turn evicts the next few layers (6-10), repeating throughout the process. As a result, although with prefetching, the entire model has be reloaded during computation. Our results confirm this: even with prefetching completes, the in-compute reload bytes approaches 25 GiB (roughly the total model weights offloaded to CPUs), indicating that
30

---

Published as a conference paper at ICLR 2026
35
Prefetch time
Compute time
30
In-compute pyles
20
10 reload bytes (GB)
Time (seconds)
25
15
20
10
15
32
24
Physical RAM (GiB)
Performance breakdown (per token step) under different RAM capacities and the corre- sponding in-compute reload bytes.
Figure 14:
nearly all prefetched weights reclaimed before use, which is clear evidence of prefetch-release conflicts. The sharp drop in cache hit rate causes high-frequent page faults and off-chip memory access, making compute time surge and dominate latency.
GiB RAM: since conflicts have reached the worst case 16 GiB RAM (i.e., almost all model weights must be reclaimed and reloaded), further reducing RAM no longer changes the reload bytes cache hit rate, so the compute time and prefetch time remains stable.
These results demonstrate that in mmap-based disk offloading, prefetch-release conflicts are real and can severely slow computing, which in turn overwhelms the overlap gains brought by prefetching.
A. 14.2 LATENCY COMPOSITION BEFORE AND AFTER APPLYING PRP
The root cause of prefetch-release conflicts is excessive prefetching under limited RAM: prefetched pages are reclaimed by the OS, and their eviction lowers cache rates and triggers frequent reloads, which slow down computation. To address this, prima.cpp introduces the pipelined-ring parallelism (PRP) mechanism, which divides model layers into multiple prefetch-compute rounds to limit the number of model layers prefetched in each round. For example, if device has enough RAM hold 5 layers but is assigned 20 layers, PRP sets 4 rounds, prefetching only 5 layers per round. This design alleviates automatic reclamation and improves cache hit rates during kernel execution.
To validate its effectiveness, we build a heterogeneous testbed consisting of four mobile devices:
DI: Mac MB Pro (12 cores, 18 GiB UMA RAM);
D2: Mac M1 Pro (8 cores, GiB UMA RAM);
D3: Linux laptop (8 cores, 4 GiB available RAM);
D4: Honor Pad (8 cores, 5 GiB available RAM).
To isolate PRP's effect, Halda is disabled. We measured prefetch, compute, and communication time compositions, well in-compute reload sizes and the overlap between prefetching and other devices' runtime. Note that when h: = 1, PRP degrades to PP. For clarity, Figure reports results on device D3.
From the in-compute reload size curve, PRP clearly reduces the amount prefetched data reclaimed by the OS. This raises cache hit rates and sharply shortens compute time. The efficiency gain comes from two factors:
Prefetching loads kernel weights in advance, avoiding frequent disruptions during comput- ing.
PRP prevents early eviction of required weights under low memory.
31

---

Published as a conference paper at ICLR 2026
Profetch time (D3)
Compute time (D3)
In-compute releed size (GiB)
Communication time (03)
Normaliza Time
(D1+D2-1 (D)+D2)D4)
In-compute mioad 6ize (D33
(k=1)
(k=2)
PRP (km4)
Figure 15: Performance breakdown (per token step) for PP and PRP and the corresponding in- compute reload bytes.
Together, these two factors improve cache hit rate and ensure faster computation, showing that PRP effectively resolves prefetch-release conflicts and improves kernel efficiency PP.
In addition, since prima.cpp uses disk offloading, PRP also reduces exposed disk I/O more effectively. Comparing PP
and PRP (k 2), we can see that:
With PP, other devices can overlap D3's prefetch time, but reload-indun induced disk I/O during compute remains significant and uncovered.
With PRP, prefetch time is still fully overlapped, and reload I/O during compute becomes negligible thanks to higher cache hits.
Overall, PRP reduces D3's per-token latency to just 20-23% of that under PP.
1.14.3 LATENCY COMPOSITION BEFORE AND AFTER APPLYING HALDA
Compute
12
Prefetch time
Comm time
Normalized time
10
14
TPOT
D2 D4 MemSched
D1 D2 D3 PerfSched
D1 D2 D3
Halda
Figure 16: Performance breakdown (per token step) for MemSched, PerfSched, and Halda.
In this experiment, we fix PRP (A: = 2) and compare Halda with two heuristic layer partitioning rules introduced in Appendix A. MemSched and PerfSched. Among them, MemSched follows the same rule as prima.cpp (w/o halda) and thus serves as our Halda-disabled baseline. As shown in Fig. 16 the results of these heuristic rules are highly random:
MemSched: devices with slow CPUs may be assigned many layers; while prefetch time remains small, compute time can become the bottleneck.
PerfSched: devices with small RAM may be assigned many layers, causing frequent kernel disruptions and again making compute time the bottleneck.
In contrast, Halda performs much smarter scheduling. It jointly considers compute capacity, memory size, communication cost, OS/hackend heterogeneity, and automatic reclamation behavior, solving an
32

---

Published as a conference paper at ICLR 2026
optimization problem to find the optimal partitioning rule. In this case, Halda assigns most layers to DI, the device with the fastest compute and disk, while slower devices (D2-D4) act as helpers to offload part of DI's memory load. This increases DI's 's cache hit rate and minimizes overall compute time while keeping global prefetch time low. As a result, on this heterogeneous testhed, Halda reduces TPOT by 50% compared the baseline (MemSched).
It's important to note that this smart scheduling is not hard-coded but derived automatically from our LDA formulation and Halda's 's solver. The resulting strategy adapts to device characteristies, generalizes across diverse testbeds by design.
A.15 ABLATION STUDY ON CONTEXT LENGTH
A longer context length increases the memory demand of the KV cache. Under fixed memory budget, allocating more space to the KV cache leaves less cache available for model weights, making prefetch-release conflicts more likely. This, in turn, causes more frequent in-compute reloads and lower kernel efficiency.
To study how context length affects TPOT, set up two heterogeneous testbeds: one with lower memory and another with larger memory:
Low-mem testbed: including Mac M1 Pro (8 cores, 8 GiB UMA RAM), Mac M3 Pro (12 cores, 18 GiB UMA RAM), Linux laptop (8 cores, GiB available RAM, and 4090 GPU with 16 GiB VRAM), Honor Pad (8 cores, 5 GiB available RAM).
Medium-ment testbed: including four Linux PCs, and 24 GiB RAM; one with a 2080 Ti (11 GiB) and 32 GiB RAM; one with a 3090 (24 GiB) and 16 GiB RAM; with 4060 Ti (8 GiB) and 32 GiB RAM; and Mac mini with 16 GB unified memory.
with 1080 Ti GPU (11 GiB VRAM)
35
Low-mem testbed
30
Medium-mem testbed
TPOT (sitoken)
25
00M
20
15-
10
5
0
M
5
10
15
20
25
30
Context length (x10*)
Figure 17: Performance breakdown (per token step) for MemSched, PerfSched. and Halda.
We ran the 3lama-3-70B Q4K model on both testbeds and measured its TPOT under context lengths ranging from 1K to 32K tokens. Fig. shows that increasing the context length has safe region where TPOT remains flat. Larger memory provides a wider flat region, e.g., about 1-4K tokens on the low-memory testbed and 1-16K tokens the medium-memory testbed. Beyond this range, TPOT rises sharply or causes OOM, as the expanding KV cache exhausts available memory.
A. ABLATION STUDY PIPELINED-RING PARALLELISM ON HETEROGENOUS TESTBED
We reproduced the experiment from Fig. 2 the heterogeneous testbed described in Appendix 1.14 comparing PP and PRP in terms of TPOT across models from 8B to 72B. As shown in Fig. 18 the results consistent with the findings reported in the main text. When l 1 or the model is small (i.e., each device's 's memory can fully hold its assigned model layers), PRP degenerates to PP. For smaller models, using PRP introduces additional kernel launch overhead, slightly increasing
33

---

Published as a conference paper at ICLR 2026
Llama 3-8B
Liama 638
200
DeepSeek
DeeoSeek R1-328
Liama 3-708
Llama 3-458
Qwen 2.5-728
Normalized TPOT
Llama 3-608
Liama 3-30B
100
50
0
2 3
4
9
10
Rounds per token k
Figure 18: TPOT vs. per-token rounds (k) for 8-72B models on the heterogeneous testbed.
TPOT. However, for larger models (> 45B), PRP with k effectively reduces compute time, leading to a significant TPOT drop by 51%-84%.
A.17 ABLATION STUDY ON NETWORK SENSITIVITY
To analyze prima.cpp's sensitivity to network bandwidth and link latency, we built four-device heterogeneous testbed:
Mac M1 Pro (8 cores, 8 GiB UMA RAM);
Mac M3 Pro (12 cores, 18 GiB UMA RAM);
Linux laptop cores, 4 GiB available RAM, and 4090 GPU with GiB VRAM);
Honor Pad (8 cores, 5 GiB available RAM).
All devices were connected via Wi-Fi. configured bandwidth and link latency using Network Link Conditioner the Macs and to the Linux laptop and tablet (simulated via Termux), The bascline RTT was 94 ms, and we incrementally added 100 ms latency to observe TPOT variation. Each experiment repeated 5 times plot the shaded area. On this testbed, ran Llama-3-70B (Q4K) for inference.
TPOT (sitoken)
TPOT (s/taken)
prima.cpp
prima.cpp llama.cpp
llama.cpp
200
400
600
800
1000
100
120
140
160
180
Bandwidth (Mbps)
RTT (ms)
(a) TPOT vs. network bandwidth
(b) TPOT RTT
Figure 19: Sensitivity of TPOT to (a) bandwidth and (b) RTT on the heterogeneous testbed.
From Fig.
19
draw insights:
34

---

Published as a conference paper at ICLR 2026
Prima.cpp is insensitive to bandwidth changes, except under extremely low bandwidth (e.g.. lower than 50 Mbps).
Prima.cpp will be affected by link latency, but the impact modest: when RTT increases from 100 ms to 200 ms. TPOT rises by 68%, but still far outperforms the baseline.
Therefore, from the network aspect, bandwidth is longer the primary bottleneck, but link latency has become the main limiting factor.
A.18 RUN PRIMA.CPP ON BUSY SYSTEMS
By default, all experiments were run on an idle system with no background apps running. In this section, we evaluate prima.cpp's 's performance on a busy system by running 6 types of background apps on the head device:
TikTok: TikTok stresses multiple resources: network (audio/video fetch, bandwidth con- tention, added latency); CPU/GPU (video decoding, UI compositing, ABR bitrate control); and memory (A/V buffers, decoded- frame queues, prefetch caches).
Work: This setup includes several commonly used productivity apps running simultaneously in the background: Chrome, Teams, Slack, ChatGPT, VSCode, Outlook, and WhatsApp.
YouTube: Similar to TikTok, but YouTube consumes more CPU/GPU/memory/netwo/ resources due to long-form 1080p/60fps (or 4K) streams.
Zoom: We co-ran Zoom with screen share, camera video, audio, and live captions enabled. It stresses CPU/GPU (real-time screen capture and HW encode, camera capture, UI com- positing. noise suppression and echo cancellation). network (sustained uplink + download streams increase RTT/jitter), and memory (large frame buffers and capture/encode queues reduce effective page cache residency for model weights, causing more in-compute reloads).
Download: Download large game from App Store while prima.cpp is rumning in the background. It stresses network (sustained downlink saturates Wi-Fi bandwidth and uplink ACKs CDN retries add jitter; queue buildup raises RTT and variance). CPU (TLS/QUIC encryption, checksum verification, and package integrity), disk (continuous writes con- tend with prima.cpp's 's read-only mmap of model weights), and memory (App Store and decompression buffers add moderate RAM pressure, shrinking available cache for model weights).
Game: Play large realtime game Asphalt (always ranked 1st in each race) while prima.cpp running in the background. It stresses GPU (3D rendering at high FPS), CPU (game logie/phystess/ppuueenee thread scheduling preempts background work, real-time threads and frame pacing reduce CPU headroom), RAM/VRAM (large textures and geometry increase shared memory usage, shrinking page-cache residency for model weights), disk (asset/texture streaming can add read bursts that interfere with mmap readahead), and network (multiplayer/telemetty traffic raises RTT/variance).
The experimental results are shown in Fig. 20 Each run was repeated 5 times. Everyday apps such TikTok, YouTuue,, downloads, work suites, and video conferencing increase TPOT by only 5-18%, showing good resilience under normal workloads. The sharpest slowdown appears with large realtime 3D games, which heavily compete for GPU time and increase TPOT by 66%.
Although prima.cpp shows a slight slowdown, this behavior is expected. For home users, the smooth operation of everyday apps matters more than perfectly consistent LLM speed. For example, an LLM service causes a game to lag or crash, users will likely terminate it to preserve gameplay. Our prima.cpp addresses this by adopting a lower RAM priority: it reduces memory use when other apps start and reclaims it when they stop, thanks to mmap-based on on-demand loading and OS reclamation. By yielding RAM to keep other apps responsive, prima.cpp both protects the user experience and avoids being killed by the user.
A. RUN PRIMA.CPP ON LARGER DEVICE POOL.
In this section. we compare the performance of llama.cpp, exo, dllama, and prima.cpp on a larger 10-node heterogeneous testbed. The hardware configuration of this testbed is shown in Table 11
35

---

Published as a conference paper at ICLR 2026
2.00
1.75
1.50
Normalized TPOT
1.25
1.00
0.75
0.50
0.25
0.00
Idle
TikTok
Work
YouTube
Zoom
Download
Game
Background App
Figure 20: Normalized TPOT of prima.cpp under different types of background apps.
Table 11: Testhed hardware summary
ID
Type
OS
Cores
RAM
GPU
VRAM
Disk Read
DO
PC
Linux
12
32 GiB
4060T:
8 GiB
0.56 GB/s
D1
Linux
16
32
N/A
1.89 GB/s
D2
PC
Linux
8
GiB
3090
24 GiB
1.23 GB/s
D3
PC
Linux
12
16 GiB
N/A
2.64 GB/s
D4
PC
Linux
24
16 GiB
1080TЛ
11 GiB
0.32 GB/s
D5
Linux
12
24
1080TЛ
11
1.57 GB/s
D6
Linux
12
32
2080Ti
11
GB/s
D7
PC
macOS
10
16 GiB
Metal
11 GiB
2.36 GB/s
D8
Laptop
Linux
18
32 GiB
N/A
3.33 GB/s
D9
Laptop
macOS
8
16 GiB
Metal
11 GiB
3.88 GB/s
covering both GPU and non-GPU devices, CPU/CUDA/Metal backends, heterogeneous OSs. RAM capacities from 8-32 GiB, VRAM capacities from 8-24GiB, and disk read speeds ranging from 0.32-3.88 GB/s.
A.19.1 COMPARISON OF TPOT FOR LLAMA.CPP, Exo, DLLAMA, AND PRIMA.CPP
In this experiment, compared the TPOT of llama.cpp. exo, dllama, and prima.cpp on models ranging from 7B to 70B. For Qwen models, exo supports only Qwen-2.5, but dllama only Qwen-3, so we compared their same-size models; llama.cpp and prima.cpp default to Qwen-2.5.
Llama.cpp ran on the most powerful device (D2), while exo (Llama) and prima.cpp ran on DO-D9, We also tested exo (Qwen) on DO-D9, but its Linux backend supports only Llama models, could run only on two macOS devices D)7 and D9. Dllama splits tensors by KV heads and supports up to 8 devices, so we removed D1 and D8. two CPU-only devices with lowest FLOPS. To run the 70B model, we upgraded D2 to 128 GiB RAM for exo and dllama (64 GiB still caused OOM to dllama). Even so, exo still ran out memory, as it requires to upgrade all Linux devices.
The results in Table 12 are consistent with those in Table 4
Prima.cpp reduces to llama.cpp small models. For models smaller than 32B, Halda identifies D2-GPU as the most capable device, with 24 GiB VRAM sufficient to hold the full 4-bit model. therefore assigns all layers D2-GPU and removes other devices, reducing prima.cpp to llama.cpp.
Llama.cpp faces a sharp spike in TPOT due to prefetch-release conflicts on larger models. When scaling to 70B, llama.cpp must offload excess layers to the CPU, but its
36

---

Published as a conference paper at ICLR 2026
Table 12: TPOT (ms/token) for Ilama.cpp. exo, dllama, and prima.cpp.
Model (4-bit)
llama.cpp
dllama
prima.cpp
Llama-3.1 3--8B
35
2500
2408
35
Qwen-2.5-7B
34
116'
2177²
34
Swvnn22.5144B
52
169
26082
52
Owen-2.5-32D1
66
313'
33992
66
Llama-3.3-70B
24037
OOM
8415
146
Qwen models have lower TPOTs/TIFTss because devices DO-
D6, D8 failed in exo. As dllama doesn't 't support Qwen-2.5, we Qwen-3 model same size for comparison.
GiB RAM cannot hold the 16 GiB overftow, leading to prefetch-release conflicts. Frequent slow disk reloads cause TPOT to spike to an unusable 23 s/token.
Prima.cpp's TPOT scales smoothly and remains highly efficient. With Halda's smart layer partitioning and device selection (see Appendix A.19.2), prima.cpp remains efficient at the 70B scale with TPOT of only 146 ms/token.
Exo suffers major slowdown due to its preporypropoortonal layer partitioning. D1/D8 CPUs are the weakest, and D3's CPU, though faster, is still far slower than Metal GPUs. However, with exo's 's memory propoooooppoo layer partitioning, D1/D3/D8 received half of the model layers, causing severe computation bottlenecks.
Exo's backend limits its OS compatibility. For Qwen models, we run them on DO-D9, but DO-D6 and D8 failed because exo's 's Linux backend supports only Llama models, and exo provides only 8B/70B Llama variants. This limitation actually favors exo, as it excludes the slow CPU-only devices (D1/D3/D8). The remaining D7/D9 macOS devices have 22 GiB of shared Metal memory in total, enough for 4-bit models under 32B, and benefit from Apple Silicon GPU acceleration, exo's 's TPOTs on 8-32B models significantly lower for Qwen models than for Llama.
TP doesn't I work well over high-latency networks. In dllama, TP requires two all-reduce operations for each model layer, so an 80-layer model performs 160 communications per to- ken step. With more devices, the overhead grows, leading to seconds of pure communication delay per token on typical home network, even for small 7B/8B models.
A.19.2 HALDA DEVICE SELECTION VS. HEURISTIC LAYER PARTITIONING
Halda performs better than human intuition and heuristic methods. Fig. 21 compares Halda with MemSched (used by exo) on Llama-3-70B (Q4K). For MemSched, its memory cecmoor-propoottonn partitioning assigns nearly half the layers to CPU-only nodes (DI/D3/D8). On pure CPU backends, 4-bit weights are decoded into FP16/FP32 and stored in limited RAM, causing these nodes of memory. Although we couldn't 't measure its TPOT, it is evidently higher than the 8B model's 2.5 s/token, which is already unacceptably slow.
Since the aggregate VRAM can hold the full model, we also propose an intuitive "Fil--astest-GPUs First" heuristic that partitions layers by GPU compute rank (D2 DO > D6 D5 D4 > D7 D9 D3 > DI D8), yielding a 13/46/19/2 split across D2/DO/D6/DS (80 layers total).
Halda adopts a slightly different plan: it removes D5 and moves its two layers to D2's CPU. This is counterintuitive because D5's GTX 1080Ti GPU has about 30x the FLOPS of D2's CPU. However, Halda's adjustment lowers TPOT from 1.56 + ms/token to 146 + ms/token. The reason behind extra RTT per token on the high-latenes Wi-Fi ring. which costs more than D5 GPU's gain. In contrast, offloading to local CPUs avoids the additional RTT and keeps the communication ring shorter.
this counterintuitive result is that adding D5 introduces
More devices do not mean better performance. In this case, Halda achieves the fastest speed with just 3 devices, and adding more weak devices offers benefit because Halda will remove them automatically. practice, home users prefer fewer devices for lower power use and simpler setup. so candidate pool of up to 5 devices is enough (as users may not have more).
37

---

Published as a conference paper at ICLR 2026
GPU Layers
CPU Layers
TPOT
50
40
100
Layer partitioning
TPOT (sitoken)
30
20
OOM
OOM
10
OOM
146 ms
156 MS
DOD1 D2 DDDD0222DDDD
D001D203D4DSD6DDDBBB FillFaastettGPU--irss
DO D
Halda
MemSched
Figure 21: Layer partitioning distribution of prima.cpp and exo Llama-3-70B (Q4K).
38


Reviews:
[{"review_id": "fWTGc3vQh4", "reviewer": "Reviewer_7c8W", "summary": "The paper introduces prima.cpp which is a distributed on-device inference system to run LLMs on home clusters connected via Wi-Fi. It enables fast, private, offline-inference using heterogeneous CPUs/GPUs and limited memory and slow disks.\n\nThe paper extends pipeline parallelism to pipelined-ring parallelism where the devices form a ring, each processing a layer window of the model across multiple rounds per token. The work also includes Halda which partitions the model layers across devices considering CPU/GPU spec. It formulates the problem as a Integer Linear Fractional Program to minimize TPOT.", "questions": "* How well do the experiments generalize beyond the few tested devices?\n* How does the system handle multi-user or multi-request workloads common in real applications?\n* Why are comparisons to frameworks like vLLM and SGLang missing?\n* What is the power consumption or energy efficiency compared to cloud inference costs?\n* Can the authors provide a detailed performance breakdown showing compute, communication, and disk I/O contributions?\n* Can more system design and implementation details be provided to give readers better insight?\n* How does the proposed approach relate to optimization techniques such as quantization and pruning?\n\nI like the paper overall. However, it has a lot of missing parts hence the score (4). I would be open to reevaluating.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* On-device AI is an important topic considering the privacy concerns with regard to LLM inference that may handle personal sensitive data.\n* Formulation of resource mapping seems to be neat and seems to provide a nice performance.\n* Pipelined Ring Parallelism seems to be a little extension that seems to be effective.\n* Code is open-sourced https://anonymous.4open.science/r/prima-cpp", "weaknesses": "* Experiments seem to be limited to a few devices and it does not really show the generalizability of the work.\n* Only single inference request is presented whereas real use-cases may really want multi-user (multi-request) workloads. In other words, it seems that the overall serving related discussions are missing in the paper.\n* Extending from previous point, it seems to lack comparisons to vLLM, SGLang, ... which are very widely used frameworks in both academia and industry.\n* Power consumption and energy efficiency OR cost vs cloud inferences seem to be missing.\n* More analysis of the performance breakdown would really help.\n* Details about how it is designed is not really presented in the paper, really limiting the amount of insight that the readers can get from reading the paper.\n* Discussion about how the work relates to various optimization approaches such as quantization, pruning are missing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "e2LZN5FEmd", "reviewer": "Reviewer_6ZVN", "summary": "This paper presents prima.cpp, a distributed on-device inference framework enabling large language models (30–70B) to run efficiently on heterogeneous consumer clusters with limited resources. It introduces pipelined-ring parallelism (PRP) to overlap disk I/O, computation, and communication, and Halda, a heterogeneity-aware scheduler optimizing CPU/GPU utilization under RAM/VRAM constraints. Experiments on four home devices demonstrate strong performance—674 ms/token for 70B and 26 tokens/s for 32B models.", "questions": "As those in weakness.\n\n1. Could you provide insights on energy consumption when running prima.cpp?\n2. How is the framework implemented for different hardware backends?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Efficient large-model deployment under limited VRAM/RAM.  \n2. Clear, well-structured, and easy-to-follow paper.  \n3. Solid, industrial-grade implementation with strong reproducibility.", "weaknesses": "1. Not show comparison with capable baselines, like SpecExec [1], which reaches 100-250ms/token TPOT. \n\n2. Leveraging the Disk to offload (because of frequent writing and erasing) will cause damage to the hardware, which is not discussed in the limitation.\n\n3. Lack of an ablation study of the impact of workload, such as context length, frequency of requests, etc.\n\n\n\n\n[1] https://github.com/Infini-AI-Lab/UMbreLLa", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VbKGfIRgwq", "reviewer": "Reviewer_kAiH", "summary": "The paper introduces prima.cpp, a distributed inference framework designed to run 30–70B parameter language models on heterogeneous and memory-constrained “home clusters.” The core ideas are (i) pipelined-ring parallelism (PRP), a variant of pipeline parallelism that repeatedly circulates activations around a ring of devices to overlap disk I/O with computation, and (ii) Halda, a heterogeneity-aware scheduler formulated as an integer linear program (ILP) that assigns per-device layer windows and CPU/GPU splits while enforcing RAM/VRAM constraints. Experiments on a Wi-Fi connected cluster of four to six consumer devices compare prima.cpp against llama.cpp, exo, and dllama, reporting large speedups in time-per-output-token (TPOT) and time-to-first-token (TTFT). The authors also present ablations on Halda and prefetching and discuss memory pressure considerations.", "questions": "PRP vs. standard PP: Can you quantify the incremental benefit of multi-round PRP over single-round pipeline parallelism on the heterogeneous cluster, with and without Halda? A per-stage latency breakdown (compute, disk, communication) would help attribute the gains to specific mechanisms.\n\nBaseline configuration fairness: Why is llama.cpp restricted to a single device while prima.cpp uses all available devices? Have you evaluated llama.cpp with manual layer partitioning (e.g., via tensor or pipeline parallel modes) to assess whether the comparison is fair?\n\nQuality metrics under quantization: Do the aggressive quantization formats (e.g., Q4K, IQ1) and speculative decoding degrade output quality? Please report perplexity or downstream task accuracy to demonstrate that prima.cpp preserves model performance relative to the baselines.\n\nDevice selection robustness: Halda is claimed to automatically drop slow devices. Could you provide experiments on a larger heterogeneous pool (≥10 devices) showing how many devices are selected, how often the assignment changes, and the resulting impact on TPOT?\n\nConvergence and overhead of Halda: What is the typical number of iterations before Halda converges, and how long does the scheduling phase take relative to inference time? Are there scenarios where the ILP fails to find a feasible solution or oscillates between assignments?\n\nNetwork sensitivity: How does prima.cpp perform under higher-latency or lower-bandwidth Wi-Fi (e.g., 50–100 ms RTT or 100 Mbps throughput)? Since home networks can be noisy, understanding the tolerance to adverse conditions is important for practical deployment.\n\nMemory pressure measurement methodology: Table 5 reports “memory pressure” as percentage reductions in mem_available. Can you detail the measurement interval (sampling frequency), the baseline state (idle system vs. with other apps running), and whether fluctuations during inference were observed?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "Originality: The paper tackles a timely and underexplored setting—distributed on-device inference across heterogeneous, low-cost consumer hardware. Integrating mmap-based offloading with pipeline parallelism and heterogeneous scheduling is an interesting direction. The attempt to capture OS-level memory reclamation in the scheduling formulation is a thoughtful touch.\n\nQuality: The implementation effort is substantial (∼20k LOC) and the evaluation spans a range of model sizes up to 70B parameters, which is rare for “home cluster” work. The ablation that removes Halda highlights the importance of scheduling in this regime, lending credibility to the design.\n\nClarity: The writing is generally clear, with useful figures (e.g., Fig. 1 showing PRP) and tabular summaries of baselines and experimental results. The high-level algorithmic flow of Halda (Algorithm 1) is easy to follow.\n\nSignificance: Demonstrating sub-second TPOT for 70B models on commodity hardware, if validated, would be impactful for privacy-preserving and offline LLM deployment. The emphasis on low memory pressure addresses an important practical concern for end users.", "weaknesses": "Soundness of PRP claims: PRP is positioned as a key innovation, yet the mechanistic explanation for resolving the “prefetch-release conflict” is largely qualitative. There is no formal or empirical breakdown of the time saved per component (disk vs. compute vs. communication), and the homogeneous-cluster experiment in Fig. 2 lacks realism (uniform 8-core CPUs and SSDs) relative to the heterogeneous setting the paper targets. It remains unclear how much of the reported TPOT gains stem from PRP versus simply better scheduling or quantization choices.\n\nBaseline fairness and completeness: The experimental section focuses exclusively on latency metrics. There is no evaluation of model quality (perplexity, task accuracy) under the aggressive quantization and speculative decoding used. Comparisons with exo and dllama are limited: exo “runs on D1–D3” but the authors disable D4 due to root access requirements, while prima.cpp does leverage D4; dllama is measured only on the 8B model and declared OOM later without empirical numbers. Additionally, llama.cpp is constrained to a single device (D3), even though its distributed variants or multiple instances could also use the other devices, making the comparison potentially unfair.\n\nAblations insufficient for causal attribution: The ablation “prima.cpp (w/o halda)” simply reuses exo’s partitioning rule but retains the rest of the system. There is no attempt to isolate Halda’s scheduling quality from the PRP machinery, nor to test alternative schedulers in a controlled manner. Similarly, the prefetching ablation shows 9–17% improvements, yet TPOT reductions of up to 17× over llama.cpp are claimed; the gap between these numbers is not reconciled.\n\nScalability and generality concerns: The entire evaluation uses a single anonymized cluster with modest Wi-Fi bandwidth. There is no analysis of how prima.cpp scales with more than six devices, higher-latency networks, or weaker hardware (e.g., devices without SSDs). The claim that Halda “selects” devices is not backed by experiments on a large pool where many must be excluded. Furthermore, the scheduler assumes access to per-device disk throughput and available memory, but it is unclear how those are measured or kept up to date in deployment.\n\nClarity issues and missing details: Several important implementation choices are omitted: handling of activation checkpointing or KV cache distribution, the cost of coordinating speculative decoding across devices, and how OS-specific behaviors (e.g., macOS page cache policies) are detected at runtime. The paper also references Appendix sections for critical arguments (e.g., Appendix A.1 for the prefetch-release conflict) rather than summarizing key empirical evidence in the main text.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)