DynamicInfer: Runtime-Aware Sparse Offloading for LLMs Inference on a Consumer-Grade GPU
Authors: Zhui Zhu
Weichen Zhang
Zhenghan Zhou
Yunhao Liu
Fan Dang
Venue: ICLR 2026 Poster (2026)
Keywords: Artificial Intelligence
Offloading
LLM inference
Primary area: foundation or frontier models, including LLMs
TL;DR: The paper proposes DynamicInfer, a runtime inference framework that dynamically schedules and offloads neurons between the CPU and GPU. And the system speed up the LLM inference speed on consumer-grade GPUs

Abstract: Large Language Models (LLMs) have achieved remarkable success in various NLP tasks, but their enormous memory footprints pose significant challenges for deployment on consumer-grade GPUs.
Prior solutions, such as PowerInfer, combine offloading and sparse activation to reduce memory and computational overhead, but suffer from static neuron partitioning, leading to suboptimal GPU utilization and increased latency.
In this work, we present DynamicInfer, a runtime neuron offloading framework that dynamically adapts neuron scheduling based on input-dependent activation patterns. DynamicInfer introduces (1) a hierarchical neural caching strategies, (2) a load-aware neuron activation mechanism tailored to heterogeneous hardware, and (3) an activation-aware prefetching pipeline that overlaps data transfer with computation.
Extensive experiments on ReluLLaMA and Prosparse models across multiple hardware platforms demonstrate that DynamicInfer achieves up to 253\% speedup over llama.cpp and 59\% over PowerInfer, while retaining model accuracy. Our approach offers a practical and scalable solution for high-performance LLM inference on resource-constrained devices.

Published as a conference paper at ICLR 2026
DYNAMICINFER: RUNTIME-AWARE SPARSE OF- FLOADING FOR LLMS INFERENCE ON A CONSUMER- GRADE GPU
Zhui Zhu'; Weichen Zhang"; Zhenghan Zhou', Yunhao Liu Fan Dang*
"Department of Automation and BNRist, Tsinghua University
Global Innovation Exchange, Tsinghua University
School of Software Engineering, Beijing Jiaotong University
fz-zhu22, weic.zhang23, (rouzh23)@@alll tsinghua, eduuuaaeedd... edu. en yurhao@tsinghua edu. adu.cn. en, dangfaneb edu. on
ABSTRACT
Large Language Models (LLMs) have achieved remarkable success in various NLP tasks, but their enormous memory footprints pose significant challenges for deployment consumer-grade GPUs. Prior solutions, such as PowerInfer, com- bine offloading and sparse activation to reduce memory and computational over- head, but suffer from static neuron partitioning, leading suboptimal GPU uti- lization and increased latency. In this work, we present DynamicInfer, a runtime neuron offloading framework that dynamically adapts neuron scheduling based on input-dependent activation patterns. DynamicInfer introduces (1) a hierarchical neural caching strategies, (2) load-aware neuron activation mechanism tailored to heterogeneous hardware, and (3) an activation-aware prefetching pipeline that overlaps data transfer with computation. Extensive experiments on ReluLLaMA and Prosparse models across multiple hardware platforms demonstrate that Dy- IamicInfer achieves up to 253% speedup over Ilama. cpp and 59% over PowerIn- fer, while retaining model accuracy. Our approach offers practical and scalable solution for nigh-performance LLM inference on resource-constraing devices.
INTRODUCTION
Large language models (LLMs) have revolutionized natural language processing tasks such as text summarization (Zhang et al. 2024 Shakil et al. 2024), natural language understanding (Yu et al.) 2024
Zhao et al. 2023), and dialogue systems Yi et al. 2024 Guan al. 2025). While tradition-
ally deployed in data centers, there a rapidly growing interest in deploying LLM on personal de-
vices, driven by increasing concerns over data privacy (by PrivateGPT 2023). customization needs
(Liu et al. 2024c Xu et al. 2024), and inference cost reduction (Lyu et al. (023))
However, deploying LLMs consumer-grade hardware faces fundamental challenges. Modern L.MMs contain tens or even hundreds of billions of parameters, requiring memory footprints far beyond the capacity of mainstream consumer-graad grade GPUs (e.g., RTX 4090 with 24GB memory).
Model offloading (Song et al. 2023 Gerganov [025 Hugging 2025 Sheng et al. 2023) and
sparse activation (Liu al. 2023 2024a Wang et 2024 Shin al. 2025) have emerged as
promising solutions to address the memory constraints of LLM inference on consumer devices. Offloading stores part of the model weights in the relatively large CPU memory and incrementally prefetches them into GPU memory (Sheng et al., 2023 HuggingFace (225) or directly computes in CPU during inference (Gerganov
2025
Song al. (003)).
PowerInfer (Song et al. 2023 pioneers the idea of combining model offloading with sparse acti- vation, enabling efficient LLM inference on personal devices. However, despite its improvements, we found that PowerInfer's static hot-cold neuron partitioning often leads to suboptimal inference
Equual contribution.
Corresponning author.

---

Published as a conference paper at ICLR 2026
speed-ups (3 This is because its one-time assignment is based offline profiling of general acti- vation trends, while actual active neurons can vary dynamically depending on the input prompt and decoding tokens. As result, many active neurons end up being processed on the CPU, despite the GPU's higher computational capabilities, leading to increased latency and GPU under-utilization.
In this paper, we present Dynamiclnfer, run-time neuron offloading framework for efficient LLM inference on consumer-gradee grade GPUs. DynamicInfer introduces a lightweight online neuron scheduler that tracks the sparsity of activation during runtime and dynamically reallocates neurons between CPU and GPU based on their predicted actívity. This online adaptation enables DynamicInfer to achieve better GPU utilization and lower latency compared to static offloading baselines such as Powerinfer.
However, building such a system entails overcoming two major challenges: (1) How design an online scheduling strategy that maximizes GPU utilization without compromising inference accu- racy? (2) How to construct a computation and data transfer pipeline tailored the large weight footprint of FFN layers, enabling efficient scheduling such that dynamic adjustments incur minimal additional latency?
To address the aforementioned challenges, we propose the following three core mechanisms:
To accommodate the dynamic nature of sparse neuron activation during inference, we introduce hierarchical neural caching framework. The system determines the allocation of neurons between GPU and CPU by utilizing predictors to forecast neuron activations (micro-level) in conjunction with historical activation data (macro-level).
Static sparsity thresholds often result in underutilized GPU neurons. address this, we propose load-aware neuron activation mechanism grounded in the principles of the sparsity predictor. This mechanism dynamically adjusts the activation thresholds for neurons on the GPÚ and CPU while adapting the varying workload and preserving theoretical inference accuracy.
To mitigate the I/O bottlenecks caused by cross-device neuron migration, we propose activate neu- prefetching. Based early-stage neuron activation predictions, the system preemptively plans the neuron migration required for the upcoming inference steps. By leveraging asynchronous trans- fer mechanisms, pipelines the transfer of neuron weights with the execution of feedforward com- putations on both GPU and CPU.
Building on the aforementioned three core mechanisms, we propose DynamicInfer, a pioncering large-scale model inference framework that integrates online neuron scheduling with dynamic spar- sity optimization. Compared Ilama.cpp (Gerganov) 2025) and PowerInfer (Song al. 2023), our method improves a generation speed (token/s) by 60%-2539% and 11 11%-59% respectively in real- world testing, while simultaneously improving GPU utilization and overall efficiency.
The main contributions of this paper are as follows:
We conduct systematic study the input-dependent and dynamic characteristics of sparse activation in the FFN layers of LLMs. We identify the fundamental limitations of offline scheduling under diverse input scenarios and propose corresponding online neuron scheduling system.
We design a hierarchical neural caching strategy and a load-aware neuron activation meeh- anism to significantly accelerate inference.
By prefetching active neurons and introducing pipelined scheduling mechanism that over- laps computation with neuron transfer, we further mitigate I/O bottlenecks and enable pipelined acceleration of the inference process.
We implement a prototype system, DynamicInfer, and evaluate it on RTX 4090 and RTX 2080Ti GPUs. Experimental results demonstrate that, while maintaining comparable in- ference accuracy, our system increases the generation speed by an average of 143.4% and 25.3% compared to existing solutions such as Llama.cpp (Gerganov 2025) and PowerInfer (Song et al. 2023), respectively, and achieves up to 253.3% and 59.4% speedup specific scenarios.
2

---

Published as a conference paper at ICLR 2026
GPU-CPU Hybrid
Powerinfer: Sparsity-Aware Hybrid
GPU-Centric Offloading
Offloading
Offloading
Memory
Memory
Memory
GPU
Weight
CPU Results
CPU
GPU Risultts
Weight
Weight
CPU
Memory
Memory
Core
Memory
Core
Modal Weight
Activate Neurons
Transfer
Cores
Neurone
Computation
Figure 1: Offloading System
2 BACKGROUND
2.1 LLM OFFLOADING
Large language models typically consist of multiple transformer layers, with each layer composed of attention component and Feed-Forwant Network (FFN) component (Touvron et al.) 2023 Zhao et al. 2023). The majority of parameters in these large models are contributed by the FFN portion (Touvron al. 2023).
As the size LLMs continues to grow, deploying them on resource-constrained devices presents significant challenges, mainly because the consumer-grade GPUs these personal devices lack sufficient memory capacity to accommodate the entire model (Song et al. 2023 2022).
Aminabadi et al.,
Model offloading techniques offer a feasible solution to these challenges. GPU-centric offloading methods, such as DeepSpeed- DeepSpeed-Inferene Inffreecn- (Aminabadi al. (2022) and FlexGen (Sheng al. [023), utilize huge CPU memory to store model parameters that exceed the GPU memory's 's capacity. Dur- ing each inference iteration, computations are performed on the GPU, and parameters are transferred from the CPU memory as needed. These methods are well-suited for throughput-oricet oriented scenar- ios, but the frequent data transfers over the PCIe bus between CPU and GPU introduce substantial latency overhead in consumer-grade GPU settings or under small-batch inference workloads.
GPU-CPU hybrid offloading, on the other hand, not only distributes model parameters across CPU and GPU memory. but also leverages both CPU and GPU processors to compute. For instance, in llama.cpp (Gerganov] 2025 Song et al. 2023), different model layers are allocated to either the CPU or GPU. The CPU handles computations for the first layers and passes intermediate activation values to the GPU for subsequent inference. This layered computation strategy effectively reduces both the frequency and volume of cross-device data transfers, thereby lowering inference latency. However, introduces new bottleneck: Since the computational performance (FL.OPS) of CPUs significantly lower than that of GPUs, the overall inference latency remains constrained.
OBSERVATIONS
Prior studies Liu et al. 2023 2024a Wang et al. 2024: Song et al. 2023) show that many neurons in the self-attention and feed-ford forward network (FFN) modules are near-zer zero and have little impact on the final output. For instance, in beLed models such OPT and ProSparse, the FFN layers can achieve up to 90% sparsity (Liu et al. 2023 Song et al. 2023).
Powerinfer (Song et al. 2023) leverages this property by integrating sparse activation with model offloading to address memory constraints on personal devices. It starts with offline profiling of an entire training dataset to estimate each neuron's activation frequency. At runtime, PowerInfer
3

---

Published as a conference paper at ICLR 2026
200%
700
138
1.3
738
133
70
Juccarts Disarce
Jeccant
73
Neurons Proportion
Edinot Vanes
(a) CDF of Activated Neu- rons
(b) Sentence-level Dynamic
(c) Token-level Dynamic
Figure 2: Dynamics of Activated Neurons
Powerinfer:PP CPU
Oracie CPU GPU
Powerinter GPU
Others
FFN, CPU
Runtime Breakdown
6.8
70
FC-LoW
30
1:0 HC.High
105
0.0
FC-Low
PC-High
203
(a) Activated Neuron Proportion
(b) Computation Breakdown
Figure 3: Computation breakdown and neuron placement
sparse activation prediction to identify and compute only active neurons, skipping the inactive ones. Note that those infrequently (cold) neurons are still executed on the CPU once activated.
While PowerInfer's cold-hot neuron partitioning alleviates GPU memory constraints and reduces computational overhead, our observations reveal significant sentence-level and token-level variabil- ity in neuron activations, which undermines its effectiveness.
Observation One: sentence-level neuron activation dynamics. Similar PowerInfer, when feed- ing LLMs different sentences (a.k.a,, prompts), we observed that the activated neurons in FFN layers often exhibit locality where small subset of neurons tends to activated with high frequency fig- ure 2a However, this locality is highly input- depttddeendent. Even if set of high-frequency neurons identified through offline profiling on the training data, the actual activation distribution can vary significantly for new input.
Taking the ReluLLaMA models as examples, we randomly sampled 400 instances from the C4 dataset (Raffel et al., 2020). As illustrated in figure 2a we observed that in specific layers, the top 40% neurons account for approximately 80% the activations. We define these as "hot neurons". However, considerable variation exists in hot neurons under different inputs. As shown in figure the Jaccard Distance between hot neuron sets for different inputs ranges from 0.53 0.57, indicating over 50% of hot neurons differ between inputs.
Observation Two: token-level neuron activation dynamics. In addition to the variation of sentence-level neuron locality, also observed high variability of activated neurons between ad- jacent tokens within the same input sentence. As shown in figure 2c we observed that the Jaccard Distance (Wikipadia) between the activated neurons of different tokens reaches values between 0.70 and 0.75, indicating substantial variation. Additionally, the Jaccard Distance between token-specific activated neurons and the globally defined hot neurons ranges from 0.64 0.76. This token-level dynamism exacerbates the inefficiency of global or offline neuron allocation strategies, as they are unable to adapt to the real-time computational demands of each inference step.
As result, in PowerInfer, many hot neurons preloaded onto the GPU remain unused during infer- ence, occupying valuable memory without contributing computation. figure 33a illustrates that considerable number of neurons are still computed the CPU. In the figure, the Oracle represents the ratio activated neurons to the number of neurons placed on the GPU, defined as #Activated Neurons / #GPU Neurons. Meanwhile, a bunch of cold neurons occasionally activated and ex- ecuted on the CPU (instead of the GPU), leading to considerable latency. For example, our bench- marks (see figure 3b) show that when running 70B ReluLLaMA model (with 24GB memory), nearly 86% the forward pass time is spent executing FFN computations on the CPU.
RTX 4090 GPU

---

Published as a conference paper at ICLR 2026
Offline Stage
Online Stage: Initialization
Online Stage: Inference
Menwyy
Mocel
Schokkrr
Hitrral
GPU NCV
CPU
Peod
Nurthd iolinn
miit
Figure 4: System Workflow
Aariicn Compun
Otee Compuc
Neght Thurake
mo
Coann Prodit
Deaa Transer
Tranffomers
Hete
Wepe
Unmus
Figure 5: Compute Pipeline
SYSTEM DESIGN
4.1 DYNAMICINFER WORKFLOW
Our system operates in offline and online stages as figure
During the offline stage, we train the model's 's sparse predictor and simultaneously collect statistics on neuron activation frequencies. The sparse predictor is trained on the C4 dataset RRffel et al. 2020) using 2,000 randomly sampled examples, generating approximately 700,000
Offiine Stage:
output tokens.
Online Stage: During inference, based on the activation frequencies collected offline, we pre-load subset of the most frequently activated (hot) neurons into GPU memory. This process is formulated as an optimization problem that aims to maximize the GPU cache hit rate while adhering to memory constraints.
During layer-by-layer inference, our framework uses the hidden representation of the current layer to asynchronously predict the activated neurons for the future layer via the sparse predictor and may prefetch the CPU neuron to the GPU. Activated neurons preloaded into GPU memory are processed the GPU, while the remaining activated neurons computed the CPU, with their results transferred to the GPU for integration. Detailed allocation and loading implementation is provided in Appendix C
4.2
NEURON I/O PIPELINE DESIGN
4.2.1
CROSS-LAYER SPARSITY PREDICTION
Dynamiclnfer reduces computation by leveraging sparse activation prediction. An MLP-based pre- dictor takes the hidden state output of each attention layer as input and estimates which FFN neurons should be activated. During inference, the engine computes only those FFN neurons predicted be active.
Previous studies have demonstrated that hidden states evolve slowly during inference, exhibiting high similarity between adjacent and even non-adjacent layers (Liu al. 2023 Zhong al. 2025 This observation motivates our design of cross-layer sparsity prediction. Specifically, for the sparse predictor (ML+++ corresponding the (i+k)) k) layer, use the output hidden state h, of the ius attention layer as input to the predictor. The predictor then outputs a sparsity prediction 2;- E Rd where is the fixed number of neurons in the FFN layer, determined by the model architecture.
5

---

Published as a conference paper at ICLR 2026
States
Predictor
Sparse Prediction
Activation History
Scheduler
Constrant
Policy
Figure 6: Hierarchical Neural Caching Strategies
4.2.2 I/O PIPELINE WITH OVERLAPPED COMPUTATION AND DATA TRANSFER
FFN layers account for the majority of parameters in large language models, and frequent neuron scheduling can result significant I/O bottlenecks. Even when scheduling is limited only the neurons predicted to be active, the overhead introduced by data transfer may exceed the perfor- mance gains from offloading computation to the CPU. Cross-layer sparsity prediction provides temporal window between prediction and execution, allowing sufficient time to transfer the required neurons from CPU host memory to GPU memory. Building upon this observation, we propose an 1/O pipeline that overlaps computation and transmission.
Specifically, after the attention module of the in layer completes its computation, a new sub-thread spawned to perform sparsity prediction and neuron scheduling. While the main thread proceeds with the FFN computation of layer i, the sub-thread concurrently predicts the sparsity pattern of layer + k. This thread is synchronized with the CPU-GPU execution of the FFN computation and utilizes CUDA asynchronous transfer mechanisms to prefetch the parameters the activated neurons the upcoming layer during ongoing computation. Details regarding data transmission and 1/O implementation are provided in Appendix C.3
4.3 HIRRARCHICAL NEURAL. CACHING STRATEGIES
Prior systems like PowerInfer (Song et al. 2023 statically allocate GPU memory based on offline neuron activation statistics. However, as shown in 3 neuron activations are highly input-dependent and dynamic, making static strategies insufficient for efficient inference. To address this, we propose a hierarchical neural caching mechanism that dynamically adapts neuron placement during runtime based on both sentence-level and token-level activation patterns, thereby improving both resource utilization and inference performance.
4.3.1 SCHEDULING METHOD
Macro-level Scheduling: Neuron activation patterns exhibit strong locality when process- ing the same input sentence. Our system continuously monitors activation frequencies and persistently caches these high-frequeng (hot) neurons in GPU memory. This strategy min- imizes the overhead from redundant scheduling and data transfers.
Micro-level Scheduling: Even within single sentence, the set of activated neurons varies significantly across tokens. address this fine- fine-grained dynamism, we employ an online activation prediction mechanism that leverages the current token's 's context to estimate the neurons likely to be activated in the next inference step. The system then prioritizes loading these predicted neurons into GPU memory.
4.3.2 SCHEDULING STRATEGY MODELING
To implement this hierarchical strategy, we formulate the neuron scheduling task as an optimization problem. The objective is maximize the total importance of neurons assigned the GPU, where

---

Published as a conference paper at ICLR 2026
0 =
MEM
MEM
0.580.72
0.580.72
0.41
0.5
CPU
3
CPU
C
MEM
MEM
P=
0.35
O. 0.840.530.67
0g 0.55
0.35
0.840.530.67
GPU
GPU
CPU
CPU
Latency
Latency
Figure Load-awar aware Neuron Activation Mechanism
a neuron's importance is jointly determined by its predicted activation for the current step (micro- level) and its historical activation frequency (macro-level).
This optimization is subject to several key constraints:
Memory Constraint:
The total memory consumed by neurons allocated to the GPU must not exceed the available memory budget.
Communication Constraint: The transfer of neurons from CPU to GPU must complete
within the available computation-I/C overlap window to prevent 1/O from becoming a bot- tleneck.
Macro-residency Constraint: High-frequert neurons pinned to the GPU to enforce residency and avoid repeated transfers.
Since solving this integer program exactly is computationally prohibitive for real-time inference, we adopt an efficient layer-wise greedy algorithm. This algorithm first guarantees GPU residency for high-frequency neurons, then sorts the remaining neurons by importance, and iteratively loads the highest-ranking ones until the communication constraint is met. This approach strikes practical balance between overhead and performance. Detailed mathematical modeling for this optimization problem is provided in Appendix E.3
4.4 LOAD-AWARE NEURON ACTIVATION MECHANISM
Existing sparse activation methods often adopt unified threshold to determine neuron activation (Song et al. 2023 Liu et al., 2023). However, in heterogeneous CPU-GPU systems, this setting is suboptimal due to differing compute capabilities by the heterogeneous computing capabilities.
Our approach dynamically adjusts the threshold based on the workload distribution between CPU and GPU, specifically the number of activated neurons. This dynamic adjustment aims to minimize the overall FFN layer latency, which is dictated by the slower the two processors figure
To ensure model accuracy is not compromised, we introduce accuracy-presse preservaioon constraint during this adjustment. This constraint guarantees that the prediction error rate does not exceed that of a bascline static threshold. In this way, the system achieves dynamic load balancing between the CPU and GPU, further reducing inference latency without sacrificing accuracy. The detailed modeling and solving method for dynamic threshold adjustment can be found in Appendix E.4
EVALUATION
5.1 EXPERIMENT SETTING
Hardware Configuration. To evaluate the performance and adaptability of DynamicInfer under heterogeneous and consumer-grade hardware environments, we conduct experiments on two differ- ent PC configurations:
PC-High: A pegforeeformance workstation featuring NVIDIA RTX 4090 (24GB) and an Intel Xeon Platinum 8358P CPU. It utilizes a PCLe 4.0 interface (64GB/s bandwidth) and is equipped with 450GB of host memory.

---

Published as a conference paper at ICLR 2026
lame
Speed (takents)
PatulLaMAA-13
ProSperse-136
PehoLLaMA- 708
20
16
4-
(a) PC- Hiih
Transformers
lomm CDP
Poweriteer
OuTs
Spocd (tolenta)
PeluLLaMA-78
30
ProSparse-78
ReloLLLMM-133
5-
(64.128)
(128.64)
(128.128)
164.128)
(328.664
(328.128)
164.128 28)
(128.64)
(1228.28)
(b) PC-Low
Figure 8: Generation Speed of various models on two hardware. The axis indicates the input length and output length.
FC:ligh
Peaernte ePu
Powerideerry cPu
Dursinn
Aestagg
330
155
PC
PC
PC Low
PCHigh
(a) Neuron Transfer from CPU to GPU
(b) Neuron Proportion
(c) FFN Breakdown
Figure 9: Neuron proportion and computation breakdown on different conditions
PC-Low: A standard configuration equipped with an NVIDIA RTX 2080 Ti (12GB) and AMD Ryzen 9 5950X CPU. It operates PCIe 3.0 interface (32GB/s bandwidth) with 120GB of host memory.
Models. We evaluate two series of models: ReluLLaMA (Song et al. 2023), Prosparse (Song test the 7B and 13B model variants. On the PC-
al. 2024). On the PC-Low configuration, High configuration, we test the 13B and 70B variants. Due to memory limitations, the 70B model quantized to int4, while all other models use FP16 to balance inference speed and accuracy. Intermediate activations are consistently stored in FP32 format.
Evaluation Metrics. For load and latency evaluation, we use the Alpaca (Taori al. 2023) dataset as input. Each latency test is conducted with 100 samples. Unless otherwise specified, we use an input of 64 tokens and generate an output of 128 tokens. The primary metric for latency evaluation is Tokens Per Second (TPS) (NVIDIA), calculated by dividing the total number of tokens generated during the decoding phase by the total response time. This metric provides a quantitative measure of response generation efficiency.
Baselines. We compare Dynamiclnfer against three state-of-the-art baselines:
Transformers (Wof et al.) 2020):
The widely-used framework for large-scale model load-
ing and inference, and integrated with the Accelerate toolkit, supports dynamic offloading CPU and GPU-centric inference.
llama.cpp (Gerganov) 2025): An efficient and popular inference engine optimized for local deployment of LLMs.
PowerInfer (Song al. 2023): A framework that integrates offloading and sparse activa- tion techniques, achieving latency optimizations in local inference scenarios.
5.2 MAIN RESULT
We first measured the end-to-end inference latency of our system in comparison with other systems. Inputs of varying lengths sampled from the Alpaca dataset to evaluate their TPS performance.
8

---

Published as a conference paper at ICLR 2026
Speed (tokervs)
Speed (tokervs)
1.4
4-
Powerinfer
Mico
DyDaamic
Powerinfer
Dynamk
(a) ReluLLaMA 13B
(b) ReluLLaMA 70B
Figure 10: Ablation Study
We tested three configurations: input lengths of approximately 64 or 128 tokens, and output lengths of 64 and 128 tokens.
8a DynamicInfer achieves 18%-26% speedup 60%-97% over llama.pp for the 13B models, and 53%-59% and 148%-253% improvements re- spectively for the 70B model. The most notable gains are observed with ReluLLaMA-70B,, where the longer latency of large models amplifies the benefits of our online scheduling.
On PC-High (figure
PowerInfer and
On PC-Low (figure 8b), DynamicInfer delivers consistent improvements across smaller models: achieving 20%-35% speedup over Powerinfer and 176%-2022 over llama.cpp for 7B models, and 11%-13% and 156%-191% gains respectively for the 13B model. The relatively modest improve- ment on the 13B model is mainly due to the limited IGB GPU memory, which restricts neuron placement on the GPU, increases CPU reliance, and reduces scheduling efficiency.
5.3 NEURON PROPORTION
During inference, our system dynamically offloads neurons from the CPU to the GPU. figure 9a presents the average number of neurons dynamically transferred per layer under various ReluL- LaMA models and hardware configurations. This mechanism enables our system to activate more neurons.
figure Ubjillsstrates the proportion of active neurons between the CPU and GPU for the ReluLLaMA model series. The active neuron ratio refers to the proportion of active neurons executed by each processing unit. Notably, on PC-High, DynamicInfer significantly increases the proportion of active neurons handled by the GPU from average of 51% to 68%. This indicates that, compared to PowerInfer, DynamicInfer substantially enhances GPU utilization while reducing CPU load.
This result is further corroborated by figure 9c which shows reduction in the latency breakdown attributed to CPU computation in the FFN. Under Dynamiclnfer, the average CPU contribution to feedforward latency decreases from 72% to 57%, highlighting key factor in the overall speedup achieved by our system.
5.4 ACCURACY
Since our approach selectively omits neurons predicted to be inactive, we investigated whether such method affects the accuracy of LLMs. We conducted experiments four datasets: RTE (Gi- ampiccolo et al., 2007). PIQA (Bisk et al.) 2020), COPA (Roemmele et al., 201T), and Winogrande (Sakaguchi et (021). The result shows that the sparsity method does affect the accuracy of the model, and the detailed result is in Appendix F.
5.5 ABLATION STUDY
figure 10 illustrates the contribution each component to the overall performance acceleration of DynamicInfer. We evaluated both the Relul.aMM--13 and 70B models the PC-HIGH platform using a stepwise integration approach, progressively incorporating Dynamiclnfer features into the Powerinfer baseline.
The Micro-level neuron scheduling yielded the most significant speed improvement, achieving 11.3% and 27.6% acceleration for Ral-13L and 70B, respectively. When combined with

---

Published as a conference paper at ICLR 2026
Macro-level neuron scheduling, the overall inference speed was further improved by 13.6% and 34.0%. This improvement is primarily attributed to the migration of more neuron computations to the GPU, thereby reducing the computational load of active neurons on the CPU. Finally, by incor- porating dynamic thresholding, we achieved an overall inference speedup of 23. 1% and 57.1 17.1% % for the models.
CONCLUSION
In this paper, we present DynamicInfer, a novel runtime neuron scheduling and offloading frame- work for efficient LLM inference consumer-grade GPUs. Dynamiclnfer addresses the limita- tions of static neuron partitioning by introducing an online, input-aware scheduling strategy that dynamically adapts to token-level and sentence-level sparsity patterns. Through combination of hierarchical neural caching strategies, load-aware neuron activation mechanism, and proactive neu- ron prefetching. DynamicInfer outperforms llama.cpp and PowerInfer by up 253.3% and 59.4% respectively in real-world scenarios.
ACKNOWLEDGMENT
This work is supported in part by the National Key R&D Program of China under grant No. 2024YFC2607400, and the Natural Science Foundation of China under Grant No. 62441233.
REFERENCES
Amey Agrawal, Nitin Kedia, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav S. Gulavani, Alexey Tumanov, and Ramachandran Ramjee. Taming throughput-latency tradeoff in Ilm infer- with sarathi-serve. In Proceedings of the 18th USENIX Conference Operating Systems Design and Implementation, 2SD1'2,, USA, 2024. USENIX Association. ISBN 978-1-939133- 40-3.
Keivan Alizadeh, Seyed Iman Mirzadeh, Dmitry Belenko, S. Khatamifard, Minsik Cho, Carlo C Del Mundo, Mohammad Rastegari, and Mehrdad Farajtabar. LLM in a flash: Efficient large language model inference with limited memory. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguisticy (Volume 1: Long Pupers). pp. 12562-12584, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10. 16.18/65// 12t.//.ac1--ooo../177777 URL https://aclathologyy wtt.or///22..../o/. long. 6787
Reza Yazdani Aminabadi, Samyam Rajbhandari, Ammar Ahmad Awan, Cheng Li, Du Li, Elton Zheng, Olatunji Ruwase, Shaden Smith, Minjia Zhang, Jeff Rasley, al. deepspeed-inferent enabling efficient inference of transformer models at unprecedented scale. In SC22: International Conference for High Performance Computing, Networking, Storage and Analysis, pp. 1-15. IEEE, 2022.
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, al. Piqa: Reasoning about physical com- monsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 1432-7439, 2020.
Zylon by PrivateGPT. PrivateGPT, May 2023. URL ivate-gpt
Gongfan Fang, Xinyin Ma, Mingli Song, Michael Bi Mi, and Xinchao Wang. Depgraph: Towards any structural pruning. The IEEF/CVF Conference on Computer Vision and Puttern Recognition, 2023.
Yao Fu, Leyang Xue, Yeqi Huang, Andrei-Octavee Brabete, Dmitrii Ustiugov, Yuvraj Patel, and Luo Mai. Serverlessllm: Low-latency serverless inference for large language models. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24), pp. 135-153,
2024.
10

---

Published as a conference paper at ICLR 2026
Georgi Gerganov. llama.cpp, May 2025. /tl.htttp////gthhb.comm
p9
Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and William B Dolan. The third pascal rec- ognizing textual entailment challenge. In Proceedings of the ACL-PASCAL workshop on textual entailment and paraphrasing, pp. 1-9, 2007.
Shengyue Guan, Haoyi Xiong, Jindong Wang, Jiang Bian, Bin Zhu, and Jian-guang Lou, Evaluating Ilm-based agents for multi-turn conversations: A survey, arXiv preprint AKiv:2503.2245: 2025.
Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural networks. In Proceedings of the 29th International Conference on Neural Information Processing Systems- Volume 1, NIPS 15, pp. 1135-1143, Cambridge, MA, USA,
2015. MIT Press.
Song Han, Huizi Mao, and William Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint 1rX1v:1510.00199. 2016.
HuggingFace. Working with large models, May 2025. URL Utt/htttee////gggiiafccc.oo////t s/accelerate/vl. wwwp:............./..// w/t. wac.kaaa reference/big http:///ii.oooeeeeaaaeeeeeeeeeee
epu offload with hook
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
Donghyun Lee, Kangwook Lee, Kyunghyun Cho, and Jason D Lee. Cats: Context-aware threshold- ing for sparsity in large language models. In First Conference on Language Modeling. 2024a.
Wonbeom Lee, Jungi Lee, Junghwan Seo, and Jaewoong Sim. InfiniGen: Efficient generative infer- ence of large language models with dynamic kv cache management. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24), 2024b.
James Liu, Pragaash Ponnusamy. Tianle Cai, Han Guo, Yoon Kim, and Ben Athiwaratkun. Training- free activation sparsity in large language models. arXiv preprint arXiv:2408. /4690, 2024a.
Yuhan Liu, Chengeheng Wan, Kuntai Du, Henry Hoffmann, Junchen Jiang, Shan Lu, and Michael Maire. Chamelconapi: automatic and efficient customization of neural networks for ml applica- tions. In Proceedings of the 18th USENIX Conference Operating Systems Design and Imple- mentation, OSDI'24, USA, 2024b. USENIX Association. ISBN 978-1-939133-40-
Zechun Liu, Changsheng Zhao, Forrest landola, Chen Lai. Yuandong Tian, Igor Fedorov, Yunyang Xiong, Ernie Chang, Yangyang Shi, Raghuraman Krishnamoorthi, et al. Mobilellm: Optimiz- ing sub-billion parameter language models for on-device use cases. In Forty-first International Conference on Machine Learning, 2024c.
Zichang Liu, Jue Wang. Tri Dao. Tianyi Zhou, Binhang Yuan, Zhao Song, Anshumali Shrivastava, Ce Zhang, Yuandong Tian, Christopher Re, and Beidi Chen. Deja vu: Contextual sparsity for efficient LLMs at inference time. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 22137-22176. PMLR, 23-29 Jul 2023. URL ptcpp:://proceednngg....... v202/ ht///luu3300 1u23am.mmmmmmmmm.. html
Cheng Luo, Zefan Cai, Hanshi Sun, Jinqi Xiao, Bo Yuan, Wen Xiao, Junjie Hu, Jiawei Zhao, Beidi Chen, and Anima Anandkumar. Headinfer: Memory-efficient Ilm inference by head-wise offload- ing. arXiv preprint arXiv:2502. /2574, 2025.
Hanjia Lyu, Song Jiang, Hanqing Zeng. Yinglong Xia, Qifan Wang, Si Zhang, Ren Chen, Christo- pher Leung, Jiajie Tang, and Jiebo Luo. LIm-rec: Personalized recommendation via prompting large language models. arXiv preprint arXiv:2307. 15780, 2023.

---

Published as a conference paper at ICLR 2026
Xinyin Ma, Gongfan Fang, and Xinchao Wang. LIm-pruner: On the structural pruning of large language models. In Advances in Neural Information Processing Systems, 2023.
NVIDIA. Nvidia nim Ilms benchmarking: Metries. URL htt.ttttp//csss..// m/ //t///ccaarrrln//////aatts/eseee
com/mt
html
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1-61 2020.
Zebin Ren, Krijn Doekemeijer, Tiziano De Matteis, Christian Pinto, Radu Stoica, and Animesh Trivedi. An i/o characterizing study offloading Ilm models and kv caches to nvme ssd. In Proceedings of the 5th Workshop Challenges and Opportunities of Efficient and Perfor- mant Storage Systems, CHEOPS '25, pp. 23-33, New York, NY. USA, 2025. Association for URL
Computing Machinery. ISBN 09984007112977, doi: (0.1145/711333033222 3721230
/ 00..0.114533777 https: /tt.//...o//1.111111111113131111
Melissa Roemmele, Cosmin Adrian Bejan, and Andrew Gordon. Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In AAAI spring symposium: logical formaliza- tions of commonsense reasoning, pp. 90-95, 2011.
Keisuke Sakaguchi, Ronan Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adver- sarial winograd schema challenge at scale. Communications of the ACM, 64(9):99-106, 2021.
Hassan Shakil, Ahmad Farooq, and Jugal Kalita. Abstractive text summarization: State of the art, challenges, and improvements. Neurocomputing, pp. 128255, 2024.
Ying Sheng, Lianmin Zheng. Binhang Yuan, Zhuohan Li. Max Ryabinin, Beidi Chen, Percy Liang, Christopher Ré, Ion Stoica, and Ce Zhang. Flexgen: high-throughput generative inference of large language models with single gpu. Proceedings of the 40th International Conference on Machine Learning, ICML'23, JMLR.org, 2023.
Jiho Shin, Hoeseok Yang, and Youngmin Yi. Sparseinfer: Training free prediction of activation sparsity for fast IIm inference. arXiv preprint arXiv:2411.12692, 2025.
Chenyang Song, Xu Han, Zhengyan Zhang, Shengding Hu, Xiyu Shi, Kuai Li, Chen Chen, Zhiyuan Liu, Guangli Li. Tao Yang, et al. Prosparse: Introducing and enhancing intrinsic activation spar- sity within large language models. arXiv preprint arXiv:2402. 135/6, 2024.
Yixin Song, Zeyu Mi, Haotong Xie, and Haibo Chen, Powerinfer: Fast large language model serving with a consumer-gren grade gpu. arXiv preprint arXiv:2312.12456,, 2023.
Biao Sun, Ziming Huang, Hanyu Zhao, Wencong Xiao, Xinyi Zhang, Yong Li, and Wei Lin. Llum- nix: dynamic scheduling for large language model serving. In Proceedings of the 18th USENIX Conference on Operating Systems Design and Implementation, OSDI'24, USA, 2024. USENIX Association. ISBN 10-1--99933--4--4
Rohan Taori, Ishaan Guirajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An cestruction-follovipi llama model. coom/atte tat tatttatsuast 1ab/ stanford alpaca] 2023.
htttps://githie com/
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13977 2023.
Qinsi Wang, Saeed Vahidian, Hancheng Ye, Jianyang Gu, Jianyi Zhang, and Yiran Chen. Coreinfer: Accelerating large language model inference with semantics-inspired adaptive sparse activation. arXiv preprint 2r01v22400.18311, 2024.
Wikipadia, Jaccard index. URL https: / /en. wiki http://tpss..........aaaaaaaa//a
12

---

Published as a conference paper at ICLR 2026
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jemite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin I Lhoest, and Alexander M. Rush. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 38 4844,, Online, October 2020. Association for Computational Linguistics. URL htt.http///// www. acl.allwwwe org/ wathhoooghoo/200000 emn.
p-demos.
Jiajun Xu, Zhiyuan Li, Wei Chen, Qun Wang. Xin Gao, Qi Cai, and Ziyuan Ling. On-device language models: A comprchensive review. arXiv preprint arXiv:2409.00088, 2024.
Zihao Yi, Jiarui Ouyang, Yuwen Liu, Tianhao Liao, Zhe Xu, and Ying Shen. A survey on recent advances in Ilm-based multi-turn dialogue systems. arXiv preprini arXiv:2402. 18013, 2024.
Fei Yu, Hongho Zhang, Prayag Tiwari, and Benyou Wang. Natural language reasoning, survey. ACM Computing Surveys, 56(12):1-39, 2024.
Gyeong-In In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, and Byung- Chun. Orca: A distributed serving system for Transformer Based generative models. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22), pp. 521-538, Carlsbad, CA, July 2022. USENIX Association. ISBN 978-1-939133-222 URL https:://///
www.:usenix..o org
Haopeng Zhang, Philip S Yu, and Jiawei Zhang. A systematic survey of text summarization: From statistical methods to large language models. ACM Computing Surveys, 2024.
Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song. Yuandong Tian, Christopher Ré, Clark Barrett, Zhangyang Wang, and Beidi Chen. H2o: heavy- hitter oracle for efficient generative inference of large language models. Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS '23, Red Hook, NY, USA, 2023. Curran Associates Inc.
Zhenyu Zhang, Zechun Liu, Yuandong Tian, Harshit Khaitan, Zhangyang Wang, and Steven Li. R-sparse: Rank-aware activation sparsity for efficient Ilm inference. arXiv preprint arXiv:2504. 19449, 2025.
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. A survey of large language models. arXiv preprint arXiv:2303.18222 1(2), 2023.
Shuzhang Zhong, Yanfan Sun, Ling Liang, Runsheng Wang, Ru Huang, and Meng Li. Hybrimoe: Hybrid cpu-gpu scheduling and cache management for efficient moe inference. arXiv preprint arXiv:2504.05897, 2025.
Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, and Hao Zhang. Distserve: disaggregating prefill and decoding for goodput-optimized large language model serving. Proceedings of the 18th USENIX Conference on Operating Systems Design and Implementation, OS,I'24, USA, 2024. USENIX Association. ISBN 978 17771-999133--4--3
A APPENDIX AND LLM STATEMENT
The contents below are appendix.
In this paper, Large Language Models (LLMs) were employed to enhance the article's quality. Specifically, LLMs were utilized to optimize the contents in the introduction, observation and sys- tem design sections. All ideas, data, and conclusions presented in this work were developed solely by the authors, without any involvement of large language models.
B RELATED WORK
In this section, we will briefty introduce the related works that motivate DynamicInfer.
13

---

Published as a conference paper at ICLR 2026
B.1 SPARSITY TECHNIQUES FOR LLMS
The computational demands of large language models (LLMs) have spurred research into sparsity on model weights (Han et al. 2016; 2015 Ma et al., 2023 Fang et al. 2023), tokens (Zhang et al.)
2023). and activations (Liu et al. 2024a Zhang al. 2025 aiming at reducing inference latency and memory usage.
Deja Vu (Liu et al. 2023) introduces concept of contextual sparsity and predicts it on the fly to speed up the LLM inference with an asynchronous and hardware-awe aware implementation. CoreIn- fer (Wang et al. 2024 collects activation statistics during the prefilling stage and selectively activate only a small subset of neurons during decoding. SparseInfer (Shin et al. (2025) offers a training-free predictor for activation sparsity in ReLU-activated LLMs, which also enables efficient inference with negligible accuracy.
B.2 OFFLOADING STRATEGIES FOR LLMS' INFERENCE
The large memory requirements of LLMs have led to growing body of work on offloading tech- niques that move parts of the model or computation other system components, such CPUs, GPUs, and secondary storage, to enable efficient inference under hardware constraints (Gerganov) 2025
Sheng al. 2023 Ren al. 2025 Luo al. 2025).
LLM in flash(Alizader, et al. 2024 solves the challenge of effectively operating LLMs that exceed available capacity of DRAM by storing model parameters flash memory but bringing them into DRAM demand. PowerInfer (Song et al. (223)) is high-speed LLM inference engine designed for consumer-gee grade GPUs. exploits the power-law distribution in neuron activation to prioritize the loading of more frequently activated neurons, enabling efficient inference hardware with limited resources.
B.3 OPTIMIZATIONS IN LLM SERVING SYSTEMS
As demand for high-throughhee LLM applications grows, recent work has also focused on optimiz- ing serving systems through better scheduling (Sun et al. 2024 Agrawal et al., 2024, Zhong al.) 2024). memory management (Fu et al.
2024
Lee et al. 2024b), and model execution strategies (Liu
et al. 2024b).
In this field, vLLM (Ko et al. 2023) is high-throughput LLM serving engine that employs Page- dAttention, memory management technique that reduces the fragmentation of key-value caches. Orca(Yu Yu et al. 2022) is distributed serving system that employs iteration-level scheduling and selective batching to improve scalability for transformer-basee models.
These work typically focuses on improving throughput under large-batch computation scenarios, and therefore cannot be directly applied to aatency-sensitive tasks on local platforms.
C IMPLEMENTATION DETAILS
C.I CODE AND DEVELOPMENT
Our Dynamiclnfer system developed based on two major open-source frameworks: PowerInfer (Song et al. 2023) and llama.cpp (Gerganov) (222).. The core inference pipeline is implemented in C++ and CUDA, enabling efficient tensor computation and effective sparsity exploitation. The sparse predictor is implemented in Python.
The sparse predictor is implemented as a lightweight MLP, accounting for less than 10% of the total model parameters. We deploy the sparse predictor the GPU to fully leverage the GPU's parallel computation capabilities and reduce prediction latency. In our experiments, the inference latency introduced by the predictor accounts for less than 2% of the total inference time.
14

---

Published as a conference paper at ICLR 2026
Table 1: Information of Mobile Devices
Baseline
Commit Id
llama.cpp Powerinfer @4155eaasSssss//oooooooeooo2hhh2
C.2 STORAGE AND PLACEMENT
FFN layers dominate the parameter count in large language models and exhibit significant sparsity. For instance, in ReluLlama 7B and ReluLlama 70B, FFN accounts for 70% of the model parameters. Therefore, during the model loading phase, we prioritize allocating non-FFN components as attention layers, output layers, and sparse predictors to the GPU.
We store the KV cache in CPU memory, thereby frecing up more GPU memory for active neu- rons. Under small batch sizes, the latency incurred by accessing the KV cache is relatively low and negligible compared to the compute and I/O bottleneck introduced by other components.
Before inference begins, we load the model and determine the initial GPU neuron cache. We em- ploy Integer Linear Programming (ILP) solver to address this optimization problem. The solver takes the offline neuron activation frequencies as input and, given GPU memory budget, allocates memory for each FFN module and determines which neurons should be prioritized for loading onto the GPU. The objective to maximize the expected cache hit rate. This method is similar to the strategy used in PowerInfer (Song et al. 2023
For the FFN layers, we store the full set FFN weight parameters on the CPU and preload them into pinned memory accelerate data transfer from CPU to GPU. On the GPU, maintain one- dimensional array. which serves as an index table. During neuron computation, the system utilizes this array to map the input hidden states from the original dimensional space to the submatrix di- mensions on the GPU. On the GPU, we maintain a one-dimensional array called gpu buckel, which serves index table. Specifically, spukkeckee indicates that the igs row of the GPU submatrix corresponds to the gpu_bucket (1 neuron in the original FFN layer. During neuron computation, the system utilizes this array map the input hidden states from the original dimensional space to the submatrix dimensions on the GPU.
C.3 TRANSFER AND SYNCHRONIZATION
The data transfer threads concurrently with the main thread. While the main thread is computing layer L, this thread uses the hidden representations from layer L-1 to perform sparse prediction and neuron scheduling for layer L+k.
The scheduling thread leverages CUDA's asynchronous data transfer mechanisms (e.g., cudaMem- epyAsync) move data. When "cold" neuron predicted to be active, its computation is per- formed on the CPU, and the results are then transferred back to the GPU. To ensure data consistency, we place a synchronization barrier before the FFN computation of each layer begins. This guarantees that all necessary neuron data transfers and result integrations are complete before the main thread starts processing the next layer, thereby maximizing the hiding of I/O latency while maintaining correctness.
D EXPERIMENT SETTING DETAILS
D.1 COMPARISON VERSIONS
The versions of llama.cpp Gerganov 2025) and PowerInfer (Song et al., 2023) used in our experi- ments are listed in Table The open-source version of PowerInfer differs from the implementation described in its original paper. Specifically, the open-source code does not include optimizations for the prefill phase, and its handling of the KV cache also deviates from the paper's description. Therefore, although we did not modify the PowerInfer codebase, the experimental results may differ from those reported in the original publication.
15

---

Published as a conference paper at ICLR 2026
LLaMA-2
ReluLLaMA
1.00
0.8
Cosine Similarity
Cosine Similarity
0.95
0.6
0.0
70B
Model Size
Layers
(a) Model Comparison
(b) Across Layer
Figure 11: Hidden State Similarity Across Layers
D.2 INFERENCE PARAMETERS
All models are evaluated under consistent inference parameters: random seed set to 42, greedy sam- pling (temperature-O, top_k=1), and batch size of to simulate real-world interactive scenarios. For llama.cpp. PowerInfer, and our system, the number of threads is set to 8 for CPU-side prepro- cessing and scheduling, ensuring fair comparison.
E METHOD DETAILS
E. SPARSITY PREDICTION DETAILS
In the implementation PowerInfer, sparse prediction executed immediately after FFN compu- tation. attempt schedule neurons, it will inevitably introduce significant I/O latency that cannot be ignored.
figure 1ia illustrates the cosine similarity of hidden states between adjacent layers across different models, with all models exhibiting an average similarity exceeding 88%. figure T1b presents the cosine similarity between both adjacent and non-adjacent layers in the ReluLLaMA 70B model.
This observation motivates our design of cross-layer sparsity prediction.
E.2 SINGLE INPUT EXAMPLE
figure illustrates how DynamicInfer operates when processing specific input, using the predic- tion of activation states for the next layer as an example. Our system is capable of predicting sparse activations across multiple layers.
Specifically, after the Attention computation of a given layer is completed, the hidden states are for- warded to the sparsity predictor. All neurons are distributed across the GPU and CPU-for example, in the current state, neurons 1, 3, 6, and 7 reside on the GPU, while the remaining neurons are on the CPU, The predictor identifies the neurons that are likely be activated in future layers; for instance, it may predict that neurons 1, 2, 4, 5, and 8 will be activated. Based on the activation predictions and historical activation probabilities, the scheduler migrates neuron from the CPU to the GPU and evicts neuron 7 from the GPU. The dynamic threshold analysis module then analyzes the current configuration and identifies the CPU as the potential bottleneck for the upcoming computation. As result, lowers the activation threshold for the GPU and raises it for the CPU, thereby activating neurons and 6 and suppressing neuron 8.
During this process, the main thread concurrently executes the current layer's FFN computation and the next layer's Attention computation. Once both the main thread's ' computation and the sub- thread's scheduling are complete, the FFN computation for the next layer begins.
16

---

Published as a conference paper at ICLR 2026
GPU
7
9
CPU MEM
Sparse Predictor
Prediction
Scheduler
GPU
MEM
Dynamic Threshold Setting
GPU MEM
CPU MEM
Figure 12: Single Input Example
CPU Memory
GPU Memory
Gate
Down
Mamory Address
PCIE
Figure 13: Memory Arrange
Prior to scheduling, the GPU is responsible for computing one neuron and the CPU for four neurons. After scheduling, the GPU computes four neurons while the CPU handles two, thereby achieving dynamic load balancing.
E.2.1 CONTIGUOUS MEMORY LAYOUT FOR FFN NEURONS
To further optimize the speed of neuron scheduling and data transfer, we redesign the storage layout of FFN neurons. In the I.LaMA family of models, the FFN is composed of three submodules: gate, up, and down. Similar upsampling and downsampling modules are also present in other models. Notably, the same set of neurons is activated across all three sublayers.
To streamline this process, we store the gate, up, and down weights of each neuron in a contiguous memory block, reducing the number of reads and improving transfer efficiency. Specifically, on the CPU side, the weights for gate, up, and down layers are stored contiguous layout. On the GPU, the sparse matrix representation of these weights is also maintained in contiguous memory space. During neuron scheduling, we utilize CUDA asynchronous data transfer (c.g.. "cudaMem- cpy2DAsync') to move the relevant memory blocks.
17

---

Published as a conference paper at ICLR 2026
E.3 SCHEDULING STRATEGY DETAILS
E.3.1 NEURON IMPORTANCE METRIC
The neuron importance metric quantifies the contribution of each neuron, when placed on the GPU, the overall inference performance of large language model. We define the importance of neuron i in layer / as:
Vt, = ai, + ASi.
where d1.s {0, 1} indicates whether the neuron active at the current inference step (micro-level), and ft, is its historical activation frequency (macro-level). À is a tunable parameter balancing the two.
E.3.2 OPTIMIZATION OBJECTIVE
We aim to maximize the total importance of neurons assigned to the GPU:
max Σ
where gt, E {0,1} is binary variable indicating whether neuron in layer is allocated to GPU memory.
E.3.3 CONSTRAINTS
The placement of neurons is subject to the memory capacity of the processing unit. Specifically, the total memory consumed by neurons allocated to the GPU must not exceed the available GPU memory budget:
Memory Constraint:
Σ 91, Maingle Mbudget,
where Maingle denotes the memory footprint of single neuron and Moudget refers to the GPU memory budget available for the FFN layer under the current hardware conditions.
Communication Constraint:
The number of neurons loaded onto the GPU is also constrained by intra-layer communication overhead, which is limited by the PCIe bandwidth. To avoid blocking computation, neuron transfers must complete within the available overlap window:
to + Σ 91. * (1 11-gh -(,.)) btrans
(Lother + (PENLL),
where gh. indicates whether neuron has already transferred to GPU, to represents the fixed over- head of the scheduling system, such as function invocations and frequency tracking. ttrens denotes the time it takes to transfer one neuron from CPU to GPU, tather is the computation time of non- FFN components Transformer layer, and LFENE the FFN computation time of layer I. and modeled as
tFFN, = max (typu Ngpu,Ls topu * Nepu.L + teyne);
where 9pp and tepu denote the per-neuron computation time GPU and CPU, respectively. N Napp, and Nepu, represent the number of activated neurons in layer I. computed on GPU and CPU, respectively. tsyne is the synchronization time required transfer FFN results from CPU to GPU.
In practice, during the Prefill and early Decode stages of inference, we adopt conservative upper bound on the number of neurons transferred per step and measure the values of lg. 4tans lothers lypus lepu. Layne- These empirical measurements are then used for solving the optimization objective in subsequent inference steps.
Macro-level Residency Constraint: addition, high-frequency neurons are pinned to the GPU to avoid repeated transfers:
[1, f 2 thight
gi. 2 h4g
0,
otherwise,
18

---

Published as a conference paper at ICLR 2026
0.790
GPU threshold =
Better
Accuracy
(-1, 0.83)
0.785
Origin 0 (0. 0)
0.780
10
12
14
16
18
20
Generation Speed (token/sec)
Figure 14: Dynamic Sparsity Profile
where &high is a predefined frequency threshold to enforce residency for high-frequency neurons This ensures that neurons identified high-importance from Macro-level perspective remain in GPU memory.
E.3.4 PROBLEM SOLVING
Solving the full integer program computationally expensive for real-time inference. We adopt layer-wise greedy strategy using pre-allocated memory budgets Mbudget, (from offline IIP in PowerInfer):
Σ Mgingeiggte < Mbudget,l
where Mbndget.J represents the memory allocated to layer I, which determined offline using the ILP solver in PowerInfer based on neuron activation probabilities.
This formulation enables an efficient greedy scheduling strategy that allows selective migration of neurons from CPU to GPU without blocking the main computation thread. In practice, first identify neurons designated as GPU-resident based on the macro-level residency constraint. Then, sort the remaining neurons by their importance scores and iteratively load the highest-ranking neurons into GPU memory. Simultaneously, neurons with lower importance scores are evicted, continuing this process until the communication constraint is met.
Moreover, since the per-layer memory budget Moudget,1 is pre-allocated, there is no need for dy- namic GPU memory allocation during each scheduling step. This further reduces runtime overhead and enhances scheduling efficiency.
E.4 DYNAMIC SPARSITY THRESHOLD DETAILS
Existing sparse activation methods often adopt unified threshold determine neuron activation (Song et al. 2023 Liu et al., 2023). However, in heterogeneous CPU-GPU systems, this setting is suboptimal due to differing compute capabilities.
We observe that applying aggressive thresholds on the GPU and conservative ones on the CPU can reduce latency without sacrificing accuracy. For instance, on PIQA (Bisk al.) (2020), setting 0, -1 and 0g 0.83 achieves comparable accuracy to the unified threshold while reducing
latency notably (figure 14).
E.4.1 MODELING
We treat activation prediction as binary classification. Let & MLP((z)) be the output of the predictor, trained with cross cross-entropy loss:
(y, log (2,) (1 yi) log =(2)))),
19

---

Published as a conference paper at ICLR 2026
where denotes the sigmoid function. Assuming the neuron activation follows Bernuulii distri- bution, Le., ~ Bernoullipp), maximizing the oog-likeihood is equivalent to minimizing the cross- entropy loss. Through maximum likelihood estimation, the result ((2)) asymptotically converges to the true activation probability = P(y = [z:).
A neuron is activated if p; 0. The approximation from skipping neurons under fixed threshold is:
Err
(1- P:<0
Err Errsingle:
where singee is the average error from omitting single neuron. This formulation captures the trade-off between computational savings and the accuracy degradation introduced by sparsity.
E.4.2 DYNAMIC THRESHOLD ADJUSTMENT
To balance CPU and GPU workloads, we optimize thresholds to minimize FFN layer latency:
min max (ga tep (0c) ) (syne). 8.
where Nopa(Op and Nepu (0.) denote the number of neurons activated GPU and CPU under thresholds fg and te. respectively. And because the attention layers are computed on the GPU, CPU-side computations also incur an additional synchronization overhead.
To ensure that inference accuracy is not degraded relative to a baseline static threshold 6, we intro- duce an accuracy constraint based on the error model:
Err(Og. S Err(O).
This constraint can be equivalently expressed as:
Σ (-P;) + Σ (1-P))
9 9y=1,p,<00 <0
9,=0,pf <be
Σ
(1 (1-P)) + Σ (1-P)) pT).
$
9, g, <0
9; g,=,, <0
This is an NP-hard MINLP problem. We adopt greedy strategy: starting from a static threshold 0, we iteratively decrease 0g to shift more computation to the GPU, and increase Og to reduce CPU load, maintaining the error constraint.
The adjustment strategy is GPU load-aware, favoring higher GPU utilization. And the algorithm sacrifices some sparsity enhance CPU throughput, achieving more balanced and efficient exe- cution.
E.5
SYSTEM PARAMETER MEASUREMENT
The constants introduced in our optimization constraints (Section E.3) are determined through a brief offline profiling process. These parameters, including lns Ltrans- lothers 'ypus lepus and symnes are hardware-dependent and reflect the specific system's CPU-GPU I/O bandwidth and computational speeds.
For given hardware configuration, these values can be treated constants. The profiling is per- formed during the initial decoding steps, where we measure the actual timing characteristics of the system. In practice, adopt conservative estimates for transfer time (ftrans) prioritize avoiding computational stalls over maximizing neuron transfer throughput.
We acknowledge that this approach assumes dedicated execution environment. In scenarios with resource contention from concurrent applications, more sophisticated online measurement mech- anism would be beneficial, which represents direction for future work.
20

---

Published as a conference paper at ICLR 2026
Table Accuracy
Model
RTE
PIQA
CОРA
Winogrande
ReluLLaMA 7B (Dense)
68.23
77.42
87.00
66.22
ReluLLaMA 7B (Sparse)
68.59
77.47
87.00
63.98
ReluLLaMA 13B (Dense)
59.92
78.02
84.00
67.56
ReluLLaMA 13B (Sparse)
59.57
78.07
85.00
66.93
ReluI.LaMA 70B (Dense)
74.36
80.14
93.00
75.85
ReluLLaMA 70B (Sparse)
73.28
80.41
92.00
75.06
ProSparse 7B (Dense)
70.76
75.51
67.00
67.72
ProSparse 7B (Sparse)
70.40
75.13
66.00
66.53
ProSparse 13B (Dense)
74.01
76.39
90.00
70.40
ProSparse 13B (Sparse)
73.64
74.10
89.00
70.33
10
20
lama.cpp
lama. apo
howeltffar
Noworlineer
TPS (token/sec)
TPS (token/sec)
12
92GB
16GB
20GB
24GB
926B
16GB
20GB
24GB
Vram Budget
Vram Budget
(a) ReluLLaMA-13B
(b) RelulLLaMA-70B
Figure 15: Performance under Different Vram Budget
MORE EXPERIMENT RESULTS
F.1 ACCURACY EXPERIMENTS DETAILS
We conducted experiments on four datasets: RTE Giampiccolo
(2007). Bisk al. 2020
COPA Roemmele et al. (201T), and Winogrande Sakaguchi et al. (2021). RTE Giampiccolo et al.
2007 evaluates logical reasoning capabilities in language understanding. Bisk al. (2020)
tests physical commonsense reasoning, COPA Roemmele et al. (20IT) assesses causal reasoning,
and Winogrande SSkaguchi al. (2021) measures commonsense- based language understanding and disambiguation.
Table 2 compares the accuracy of the ReluLLaMA and ProSparse model series. The results demon- strate that across all model sizes and task types, the impact of sparse activation reasoning capa- bilities is negligible.
F.2 PERFORMANCE IN DYNAMIC SCENARIO
extended our experiments across various scenarios to demonstrate the effectiveness of our ap- proach under different conditions.
figure 15a and figure 15b illustrate the variation in TPS when dynamically managing the GPU VRAM budget on the PC-High configuration. As shown, method consistently outperforms PowerInfer across a range of GPU VRAM capacities from 12 GB to 24 GB. When the available VRAM decreases, the GPU accommodate fewer FFN neurons, resulting in relatively smaller performance gains. Conversely, as the VRAM increases, more neurons can scheduled on the GPU, leading to greater performance improvements.
By setting different initial thresholds, we can obtain varying trade-offs between inference speed and accuracy. Therefore, in systems based sparse activation prediction, there exists inherent
21

---

Published as a conference paper at ICLR 2026
80%
Powerinfer
15.5
llama cpp
Ours 19,1
PIQA Accuracy
78%
Speed (token/s)
Powerinfer Ours
76%
74%
72%
Powerinfer
Durs
70%
Origin
0
10
12
14
24 Speed (token/s)
16
18
20
22
26
28
2
4
8
Thread
(a) Speur-Accuracy Tradeoff
(b) Dynamic Thread
Figure 16: Speed-Accuracy Tradeoff and Dynamic Thread
Table 3: Peak Memory Usage Comparison (GB)
GPU Memory
CPU Memory
Model
Hardware
PowerInfer
DynamicInfer
PowerInfer
Dynamiclnfer
ReluLlama 7B
2080Ti
11.3
11.2
7.4
11.8
ReluLlama 13B
2080Ti
11.4
11.6
21.3
36.6
ReluLlama 13B
4090
23.5
23.7
36.6
39.8
ReluLlama 70B (int4)
4090
23.6
23.5
24.2
28.4
balance between performance and accuracy. figure 16a illustrates the speed-accuracy curve in the PIQA Bisk al. (2020) dataset for the ReluLLaMA 13B model running on the PC-HIGH platform under different initial threshold configurations. As observed, DynamicInfer achieves nearly the same accuracy as the dense model while delivering significantly higher inference speed compared the PowerInfer system.
figure 16b presents the results of the ReluLLaMA-7000 model on the PC-high machine under varying thread confi gurations. Different thread counts correspond to different levels of parallelism and CPU workload. Under various thread configurations, our system achieves a speedup of 166%-211 % compared to llama.cpp, and 27.6%-57.3%% improvement over PowerInfer.
F.3 MEMORY USAGE ANALYSIS
detaeSs the peak memory usage for both GPU and CPU across different model configurations. Dynamielnfer introduces minimal memory overhead compared to PowerInfer.
The additional CPU memory consumption in DynamicInfer is primarily attributed to the sparsity predictor and buffers required for dynamic neuron transfer. The GPU memory footprint remains nearly identical between the two systems, demonstrating that our dynamic scheduling approach does not impose additional GPU memory overhead.
F.4 GENERALIZATION TO NON-RELU ACTIVATION MODELS
While primary experiments focus on ReLU-activated models that exhibit explicit sparsity, our framework generalizes to models with other activation functions. To demonstrate this, we evaluate Dynamiclnfer Llama-2 models, which utilize the SiLU (Sigmoid Linear Unit) activation function.
For non-Rea Ron- activations where exact zero values are rare, we adapt the strategy from CATS Lee al. (2024a), predicting and pruning neurons with the lowest activation magnitudes. Table presents the results on Llama-2 models.
The results demonstrate that Dynamiclnfer achieves consistent speedups over baselines ReLU models while maintaining comparable accuracy. This validates that our core contribution- the dynamic scheduling framework-is not limited to ReL.Ubaaedd models but can be effec- tively applied to various architectures with appropriate sparsity prediction strategies.
22

---

Published as a conference paper at ICLR 2026
Table 4: Performance Non-ReLU Models (Llama-2 with SiLU Activation)
Model
Method
RTE
PIQA
COPA
Winogrande
Speed
Speedup
llama.cpp
63.90
78.07
89.00
69.38
7.9
1.00
Llama2-7B (2080Ti)
PowerInfer
63.17
76.71
88.00
67.48
14.2
1.79x
Dynamiclnfer
63.54
76.55
88.00
67.56
16.5
2.09x
llama.cpp
67.15
79.43
91.00
72.45
2.5
1.00
Llama2-13B 13B (2080Ti)
PowerInfer
66.45
77.80
89.00
71.50
5.27
2.11x
Dynamielnfer
66.01
77.69
90.00
71.43
5.81
2,32x
llama.cpp
67.15
79.43
91.00
72.45
9.8
1,00x
Llama2-133 13B (4090)
PowerInfer
66.45
77.80
89.00
71.50
12.4
1.27x
Dynamiclnter
66.01
77.69
90.00
71.43
13.8
1.41x
23


Reviews:
[{"review_id": "Uqc7l683mu", "reviewer": "Reviewer_uAnA", "summary": "This paper presents DynamicInfer, a runtime neuron offloading framework for efficient LLM inference on consumer-grade GPUs. The key innovation is dynamic neuron scheduling that adapts to input-dependent activation patterns, in contrast to PowerInfer's static offline partitioning. The system achieves significant speedups (up to 253% over llama.cpp and 59% over PowerInfer) through three main mechanisms: hierarchical neural caching, load-aware neuron activation, and activation-aware prefetching.", "questions": "-   While the comparison with PowerInfer is reasonable, can you justify comparing your method against llama.cpp? It supports a much larger class of models (non-ReLU, quantization, etc) and was not tailored for efficient inference. It seems unfair.\n-   How sensitive is the method w.r.t to λ, the neuron importance metric.\n-   Line 361, why did you mention “SparseInfer” ?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "**Well motivated Problem**\n- Figure 2,3 underlines the importance of dynamic neuron activations.\n- The two observations clearly clarify the drawbacks of static partitioning that PowerInfer uses.\n\n**Comprehensive Engineering efforts**\n- Authors implement the core inference engine in C++ and CUDA for efficiency. The three component approach is well integrated to minimize overhead between various parts.\n- Constraint design shows a thorough understanding of latency profiling.\n- Contiguous memory layout (Appendix E.2.1) for FFN neurons is interesting. \n- Benchmarked on two different generations of NVIDIA GPUs with different memory bandwidths.", "weaknesses": "**Need Additional Comparisons.**\n-   As authors note, this is not the first work to dynamically detect activations across the layers. SparseInfer (Shin et al., 2025) (cited by authors) also provides training-free prediction for ReLU-activated LLMs. A direct comparison is essential since both papers target the same models and claim similar benefits. At minimum, discuss architectural differences and when each approach is preferable.\n-   Section D.1 states that the open source implementation of PowerInfer differs from the main paper. The results are hence unfair to compare. Authors should ideally contact the authors to implement and/or discuss the implications to the results.\n-   Latency evaluation on at least one more dataset can help show robustness.\n  \n**Insufficient details on the proposed approach.**\n-   How accurate is the trained sparse predictor for new samples? across layers? across models?\n-   A number of constants are introduced w.r.t the optimization problem and constraints in E.3. How are they measured? How much do they depend on the hardware?\n-   E.4.1 is overly simplified. Lacks justification for Bernoulli distribution. “average error” can vary across layers and models. How is k decided for the sparsity predictor beforehand? Is it data/hardware/model agnostic? what is the procedure?\n-   Can you explain the difference between your Macro-level residency constraint and PowerInfer? Although it makes sense to introduce, it does not seem very important from Fig10.\n-   Line 1048 abruptly switches from the optimization problem to the greedy strategy. Can you discuss the differences in optimal solution and how much it affects? At the very least, an empirical comparison on some instances would be helpful.\n-   Can you provide a table showing peak memory usage for GPU and CPU for PowerInfer and DynamicInfer?\n    \n**Writing needs polish.**\n-   The references are not clickable.\n-   The visual and text layer of the document are mismatched, making it hard to select text.\n-   The majority of details about the method are in the appendix.\n-   Captions should contain more details.\n-   A Grammar check would be helpful.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zZtszWmkgy", "reviewer": "Reviewer_CLC1", "summary": "The paper proposes DynamicInfer, a runtime-aware system for efficient LLM inference on resource-constrained devices. It enhances efficiency by dynamically and finely scheduling FFN neurons between CPU and GPU. Its core mechanisms involve using a lightweight sparse predictor to forecast activation patterns online and dynamically adjusting the sparsity threshold based on the overlapping window of computation and data transfer, thereby optimizing hardware utilization. Experiments show that this method significantly reduces end-to-end latency compared to baseline approaches.", "questions": "1. Does cross-layer sparsity prediction still work effectively for layers that have low I/O similarity?\n2. Moreover, do you have any additional mechanisms or fallback strategies specifically designed for those exceptional layers with very low similarity? Besides, can you provide some data to show cross-layer prediction has mere impact on prediction accuracy?\n3. The data transfer between CPU and GPU is rather slow compared with GPU memory access. In your article, does the cross layer prediction fully cover the time of weight transfer? It'll be helpful if you provide detailed data.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The work addresses the critical challenge of deploying massive LLMs under tight memory constraints by combining model offloading with sparse activation. Unlike prior systems such as PowerInfer that rely on static neuron partitioning based on offline profiling, DynamicInfer introduces a dynamic, input-aware scheduling mechanism that adapts neuron placement between CPU and GPU at runtime. The method dynamically adjust the activated neuron with optimization from system level to minimize the latency. Practical and Timely Problem: The focus on enabling high-performance LLM inference on consumer-grade hardware is highly relevant and impactful. With growing interest in on-device AI, this work provides a compelling solution to a real-world deployment bottleneck.\n2. Strong Technical Innovation: The shift from static to runtime-adaptive scheduling is a key conceptual advance. The design of cross-layer sparsity prediction, combined with a greedy yet effective optimization framework, enables fine-grained, input-dependent control over neuron placement—a clear improvement over fixed partitioning schemes.\n3. Well-Rounded System Design: The integration of three synergistic components—dynamic caching, load balancing via threshold tuning, and I/O pipelining—forms a cohesive and robust system. The attention to engineering details (e.g., contiguous memory layout, asynchronous transfers) enhances practicality and performance. The method constrains the memory, communication and macro residency to balance dynamic adaptability", "weaknesses": "1. As far as I know, the I/O similarity between the initial layers and deep layers of LLMs is quite low—in some other models, this phenomenon may be even more pronounced. In such cases, does cross-layer sparsity prediction still work effectively for these layers or hidden states?\n2. Moreover, do you have any additional mechanisms or fallback strategies specifically designed for those exceptional layers with very low similarity?\n3. The data transfer between CPU and GPU is rather slow compared with GPU memory access. In your article, does the cross layer prediction fully cover the time of weight transfer? It'll be helpful if you provide detailed data.\n4. There are some typos and mismatch of your illustrations with your pictures that may cause misunderstanding. The \"Figure3 B\" in Observation one seems to be Figure2 B?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tpBtbUtraP", "reviewer": "Reviewer_N69j", "summary": "DynamicInfer improves LLM inference throughput served with memory-constrained GPUs via sparsification. It does this by sparsifying the neurons in the Feed Forward Network (FFN) stages of transformers. The key contribution is a scheme for 'online sparsification', where the sparsity map of the next layer is estimated with the previous layer activations, giving enough time for the CPU-GPU memory transfers of the FFN weights to happen. DynamicInfer Is tested with LLM architecture that utilize ReLU activations and does not markedly drop LLM accuracy on four datasets.", "questions": "I am willing to increase my score if the authors can either provide reasonable answers to, or provide empirical evidence for the following questions:\n* Can DynamicInfer improve the throughput of modern LLMs, e.g., Llama3 family or gpt-oss?\n* How does DynamicInfer compare against vLLM with quantized models (throughput and accuracy)?\n* How do DynamicInfer's improvements change at ISL/OSL 2K/128?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "Thank you for submitting your work. I overall like the paper. In terms of strengths:\n* I like the idea of online sparsity. It's a complicated technique, but there are few alternatives when memory restrictions are aggressive (except for quantization, which should be compared to).\n* The paper is mostly well-presented. §2 lays the groundwork for the rest of the discussion, and §3 conveys the general idea (though I have suggestions for improving §3).\n* DynamicInfer is evaluated on two machines, 5 models, and compared to 3 other schemes.", "weaknesses": "I find 3 major flaws in the paper:\n* From what I know, most modern LLMs do not use ReLU activation functions. Older LLMs used GeLU. The previous generation, such as Llama3, Qwen2 used SiLU, and the newest generation such as gpt-oss use SwiGLU. These families of activations always contribute to the next layer neurons, regardless of their value. ReLU, on the other hand, is akin to a on/off switch; some neurons do not affect the next layer at all, which is why sparsification would be a prime optimization candidate. This makes me worry if DynamicInfer would incur more substantial accuracy losses with commonly used LLMs, such as the original Llama family.\n* The problem setup is a memory constrained GPU, and model compression is allowed (sparsification that skips neurons is technically model compression). So a natural contender is quantization. The PC-low setup with 7B models and PC-high setup with 13B models can be compared to int8 quantized versions (done properly, as simply casting would break the model) of the original models. I understand that quantization cannot work with 70B models, but they are still simpler alternatives than sparsification.\n* The input/output sequence lengths are too small compared to real workloads. Even the relatively old ShareGPT dataset has input/output sequence lengths in the 300/200 range, and more modern datasets have even higher input lengths (due to longer prompts) and output lengths (due to reasoning chains). I wonder if the benefits of sparsification still hold at longer sequence lengths, since the attention runtime will increase and become a bigger share of the full inference pass.\n\nSome minor comments:\n* I found the explanation of the figures in §3 to be confusing. For example, why are there two bars in every group in Figures 2b and 2c? I expected a single bar to show the Jaccard distance between global hot neurons and per sentence hot neurons (Also, please cite a reference for Jaccard distance, or use the more common IoU metric which is quite similar). As another example, the oracle is defined to be $\\frac{\\text{\\\\# activated neurons}}{\\text{\\\\# GPU neurons}}$. If this is the definition, then what is \"Oracle-GPU\" and \"Oracle-CPU\", and why do they sum to 1?\n* The paper has formatting issues (figures have inconsistent reference formatting) and needs proofreading (typos).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tNTnByu7jz", "reviewer": "Reviewer_h9mu", "summary": "This work proposes a CPU–GPU collaborative inference method for large language models on a consumer-grade GPU by leveraging activation sparsity. Specifically, it introduces a hierarchical neural caching strategy that determines neuron placement based on sentence-level and token-level activation patterns. This differs from the baseline, PowerInfer, which also performs CPU–GPU collaborative inference but relies on offline profiling and a static hot–cold neuron partitioning. Given a layer’s hidden state, the predictor forecasts the sparsity of a single later layer located several layers deeper in the network. To hide host-to-device weight-transfer overhead, the paper applies optimization techniques such as I/O pipelining with overlapping of computation and data transfer. To maintain load balancing between GPU and CPU, it adaptively adjusts the threshold based on the observed loads of each device. The neuron-placement strategy must satisfy memory, communication, and macro-residency constraints; the paper formulates this as an integer linear programming (ILP) problem. Evaluated with ReluLLaMA and Prosparse on the Alpaca dataset, the method achieves a 253% speedup over llama.cpp and a 59% speedup over PowerInfer.", "questions": "- Accuracy/generalization: Beyond ReluLLaMA and ProSparse, can you report accuracy on additional architectures (e.g., Qwen, DeepSeek) to demonstrate that accuracy is preserved more generally?\n\n- Activation functions: Do the proposed gains (accuracy and speedup) hold for models that use non-ReLU activations such as ReGLU or SwiGLU?\n\n- Experimental setup clarity: Could you clarify the “low-end PC” configuration—especially the 120 GB host memory claim—and provide the exact memory specifications to justify this designation?\n\n- Sensitivity/robustness: How sensitive is performance to the hyperparameter k? And how do speedups vary with input length, output length, and batch size?\n\n- Baselines: Can you include stronger baselines (e.g., vLLM, SpecInfer with speculative decoding) and compare throughput/latency directly to these systems?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- This paper targets an important research problem: democratizing LLM inference and preserving users’ data privacy by enabling inference on a consumer-grade GPU.\n\n- The plots and figures are very helpful for understanding the proposed method, when each component operates, and where the speedup comes from.\n\n- The achieved speedup over the baseline is substantial, and the results show the method works across several model sizes (7B, 13B, 70B).\n\n- Figure 1, which describes the offloading system, is very helpful for understanding the baselines, prior work, and how data transfer occurs in each case.\n\n- The rationale behind the method is sound—both in terms of systems optimization (I/O pipelining, overlapping, and balancing the load) and algorithmic design (dynamic, runtime prediction versus offline, static prediction).", "weaknesses": "- The evidence supporting the claim that the method preserves accuracy is insufficient. The paper evaluates only two model architectures (ReluLLaMA and ProSparse). To demonstrate generality, additional experiments on other architectures (e.g., Qwen, DeepSeek) are needed to show that accuracy is maintained.\n\n- The method targets models whose activation functions induce high sparsity by design (e.g., ReLU). The paper should evaluate whether comparable accuracy and speedups hold for architectures using other activations such as ReGLU and SwiGLU.\n\n- The experimental setup needs clearer description and stronger justification. The paper states that a “low-end PC” uses 120 GB of host memory, but 120 GB is arguably not low-end. Clearly specifying the exact memory configuration would improve transparency.\n\n- A sensitivity study is needed for the hyperparameter k and for how speedup varies with input length, output length, and batch size, to establish robustness across diverse settings. Since k controls how far ahead the predictor targets and influences predictor accuracy, its impact should be quantified.\n\n- Stronger baselines are needed. Although “Transformers” is included, it typically shows slower performance than highly optimized inference frameworks (e.g., vLLM) and baselines like SpecInfer that employ speculative decoding.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)