VLM in a flash: I/O-Efficient Sparsification of Vision-Language Model via Neuron Chunking
Authors: Kichang Yang
Seonjun Kim
Minjae Kim
Nairan Zhang
Chi Zhang
Youngki Lee
Venue: NeurIPS 2025 poster ()
Keywords: Vision Language Model
Activation Sparsification
Flash Storage
Primary area: infrastructure
TL;DR: Neuron Chunking is a hardware-aware sparsification framework that abstracts access patterns into contiguity distributions to couple neuron selection with flash I/O behavior and improve I/O efficiency in VLM inference.

Abstract: Edge deployment of large Vision-Language Models (VLMs) increasingly relies on flash-based weight offloading, where activation sparsification is used to reduce I/O overhead. However, conventional sparsification remains model-centric, selecting neurons solely by activation magnitude and neglecting how access patterns influence flash performance. We present Neuron Chunking, an I/O-efficient sparsification strategy that operates on *chunks*—groups of contiguous neurons in memory—and couples neuron importance with storage access cost. The method models I/O latency through a lightweight abstraction of access contiguity and selects chunks with high utility, defined as neuron importance normalized by estimated latency. By aligning sparsification decisions with the underlying storage behavior, Neuron Chunking improves I/O efficiency by up to 4.65× and 5.76× on Jetson Orin Nano and Jetson AGX Orin, respectively. The code is available at https://github.com/snuhcs/vlm-flash.

VLM in a flash:
I/O-Efficient Sparsification of Vision-Language Model via NEURON CHUNKING
Kichang Yang* Seoul National University kichang960snu.c ac kr
Sconjun Kim Scoul National University
Minjae Kim
Seoul National University aingo033040snu.cm ac. kr
cyanide170snu.ee ac. kr.
Nairan Zhang Meta nairanzhang@meta.co
Chi Zhang Amazon zhanbchi Bwabbchi@mmzzo.ccom com
Youngki Lee
Seoul National University youngkilee@dnu. ac kr
Abstract
Edge deployment of large Vision- LanguaLLa Models (VLMs) increasingly relies on flash-based weight offloading, where activation sparsification used to reduce I/O overhead. However, conventional sparsification remains model-centric, select- ing neurons solely by activation magnitude and neglecting how access patterns influence flash performance. We present NEURON CHUNKINGG, an V/O-efficient sparsification strategy that operates on groups of contiguous neurons memory-ane and couples neuron importance with storage access cost. The method models I/O latency through a lightweight abstraction of access contiguity and selects chunks with high utility, defined as neuron importance normalized by timated latency. By aligning sparsification decisions with the underlying storage behavior, Neuron Chunking improves I/O efficiency by up 4.65x and 5.76x Jetson Orin Nano and Jetson AGX Orin, respectively. The code available at btttps://gitaie com snuhcs, Bariicssvllmeee flm-flash
Introduction
Recent vision- language models (VLMs) demonstrate strong multimodal reasoning and real-time language interaction with visual scenes. Deploying these models on edge devices is becoming essential for applications such as augmented reality (AR) and autonomous robotics that require on-device inference for robustness to limited connectivity and privacy (5) [53) These systems must process video frames continuously without frame drops while maintaining interactive latency.
The scalability of on-device inference is fundamentally constrained by memory capacity. Edge platforms provide far less memory than what modern VLMs require. Jetson Orin Nano, for example, offers only 8 GB of memory, while VA-nnee
[18] requires 16 GB (fp16) for weights and inference engines [1 9 address this mismatch through weight offloading, which stores model parameters in external flash memory and loads them on demand during inference. This method allows large models to execute on small devices but introduces substantial I/O latency, which often dominates total inference time.
One Vision--2 7B
alone. Recent systems 2 3
Activation sparsification has been widely explored to mitigate this latency loads only the weights corresponding to neurons with high importance (e.g., magnitude of activation
The approach
"quual contribution
t Corresponding author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Neuron importance
Latency per chunk
Total latency
Total importance
Baseline
5
5
4
6
16
23
Ours
4
5
5
4
6
11
26
Latency Model
Chunk-based
5
4
4
6
Pareto Improvement through Higher Contiguity
(Section 3.1]
Utility guided Chunk Selection (Section 3.2)
Figure 1: Illustration of conventional sparsification vs. our approach. Existing methods select neurons solely based on activation importance, which often leads scattered, irregular access patterns with poor I/O efficiency. In contrast, our method explicitly accounts for actual I/O latency, favoring contiguous chunks that achieve better importance-latecyy trade-offs.
value), reducing total data transfer and improving input adaptivity. Despite its effectiveness, existing sparsification remains menel-centric. It selects channels solely based on activation importance while assuming that I/O latency scales linearly with data size. Flash storage does not follow this assumption; its latency depends strongly on access contiguity, and scattered reads severely degrade throughput. Farlier methods were less affected because they targeted highly sparse L.LMs or GPU memory transfers, where locality plays smaller role. Modern VL.Ms exhibit smoother activation distributions and lower sparsity (Figure ( leading to fragmented reads and high I/O overhead Abb.
We propose NEURON CHUNKING, sparsification framework that improves the I/O efficiency of flash-offloaded inference by coupling activation sparsity with storage access behavior. The key idea is to jointly optimize neuron importance and flash I/O latency by selecting contiguous channel groups that provide better trade-off between accuracy and latency. Contiguous reads provide higher flash throughput, allowing moderately important neighboring channels to be loaded more efficiently than distant but highly important ones (Figure . This design forms compact chunks that enhance access locality, leading to significant performance gains during VI.M inference.
Efficient realization of this strategy requires capturing hardware I/O behavior in form that the runtime can readily exploit. NEURON CHUNKING achieves this by reducing complex access patterns into compact structural representation, termed the contiguity distribution. It summarizes how memory accesses cluster into contiguous groups (i.e., chunks) while omitting their exact spatial layout. This abstraction underlies two key components of our system. chunk-based latency model profiles load latency for each chunk size and efficiently estimates the I/O latency of arbitrary access patterns from their contiguity distributions. A seility--uided chunk selection algorithm formulates neuron selection as constrained optimization problem and iteratively selects chunks that maximize the importance- lattenc utility. Evaluation on Jetson Orin AGX and Nano using open-source VLMS and standard benchmarks shows consistent improvement in the accuracy-latency trade-off. At comparable accuracy, NEURON CHUNKING reduces I/O latency by an average of 2.19x on Nano and 2.89x on AGX, with maximum gains of 4.65x and 5.76x, respectively.
Our contributions are summarized follows.
We identify and characterize the hardware inefficiencies that arise when conventional activation sparsification techniques are applied to flash offh-oofloaded VLM inference, revealing their mismatch with underlying storage access patterns.
We propose NEURON CHUNKING, sparsification framework that enhances flash I/O efficiency by coupling activation sparsity with storage access patterns. It jointly optimizes neuron importance and flash latency by selecting contiguous channel groups that balance accuracy and latency.
We introduce the contiguity distribution as compact representation of flash access behavior. Building on this abstraction, we develop a chunk-based latency model and a utility-guided chunk selection algorithm that together enable latency-aware sparsification by balancing model quality and I/O efficiency.
We evaluate NEURON CHUNKING on Jetson Orin AGX and Nano with open-source VLMSs and standard benchmarks. Results show consistent improvement in the accuracy -atency trade-off,

---

OPF66,7 70 Defode Phase
Throughput (GB(6)
2.5
Magnitude
0.3
LLaV--OmVaain OniVa Apponding
120
0.2
0.1
16
Dlods
25
50
100
150
200
250
300
460
Activation index
Number Requeett
Figure 2: Activation-magnitudee plot for workloads: (teal) a ReLUUbased LLM in the decode phase and (magenta) a gated-activation- based VLM in the frame appending phase. VLM exhibits a smoother distribution, with much less variation between high and low activation values.
Figure 3: Read throughput function of block size and number of requests, profiled on Jet- son AGX Orin with Samsung 990 Pro SSD, Throughput quickly saturates and remains stable once the request count exceeds minimal thresh- olds.
reducing I/O latency by up to 5.76x on AGX and 4.65x on Nano while maintaining comparable accuracy.
2 Background and Motivation
2.1 Overview of LLM/VLM Inference Pipeline
LLM inference pipeline consists of two stages: (i) the prefill stage, where the input prompt. consisting of multiple tokens, is processed to generate key-value (KV) cache, and (ii) the decoding stage, where the model generates tokens one time autoregressively using KV cache.
When processing an online video stream with VLMs, an additional duced between the prefill and decoding stages. In this stage, incoming video frames processed sequentially as they arrive. VLMs integrate vision encoder alongside the backbone LLM. Each frame is divided into patches and passed through the vision encoder, which converts them into sequence of visual tokens. These tokens are then fed into the language model, augmenting the existing KV cache generated from the language prompt (see Appendix B. for details).
stage is intro-
2.2 Model-side Observation: Smooth Activation Profiles in VLMs
Modern VLMs exhibit smooth activation distributions as shown in Figure 2 This smoothness is general property of the architecture rather than model-specific artifact. arises from key factors: (i) gated activation functions such as SwiGLU (35) and GeLU 12 produce continuous rather than sparse activation values, and (ii) averaging these values multiple visual tokens, in LLaVA- Vision with 14 x 14 tokens per frame, further reduces the variation in importance scores. As shown in Appendix C this phenomenon consistently appears across diverse models.
This observation suggests that when activation values are less distinct, selection policies can balance model quality with system performance rather than focusing solely the largest activations. Further- more, the lack of sharply separated activations makes difficult to depend on only a few dominant neurons, making moderate sparsity a more favorable operating point.
2.3 System-side Observation: Flash I/O Sensitivity to Access Contiguity
Flash read performance depends on memory access patterns, and access contiguity is the dominant factor. As shown in Figure 4x larger contiguous reads improve throughput until reaching the bandwidth limit, marking shift from overhead bound to bandwidth-boun bound operation. This behavior reveals counterintuitive aspect of sparsification: while higher sparsity reduces data transfer, fragments memory accesses and can increase latency (Figure
Notably, throughput stabilizes once the request count exceeds minimal threshold (Figure 3 This stability enables latency estimation from access contiguity using one-time throughput profile across block sizes, making latency-awe aware sparsification practical during inference.

---

Throughpun Throughput (GBis)
300
Seattered (Nano)
Lalency (ms)
Contigueus (Nano)
4
200
Scattared (AGI)
Contirucus
Nano
100
AGX
0
0
100
200
300
400
500
20
40
60
80
Block Size (KB)
Sparsity (%)
(a) Block Size vs. Flash Read Throughput
(b) Sparsity vs. Flash Read Latency
Figure 4: Flash read performance under varying access patterns. Left: Throughput vs. block size when reading 128 MB (MLP weight sizes in Qwen2-7B [50]). Right: Latency vs. sparsity across two modes-ssctterre (random) and contiguous (sufficiently block-aligned saturate throughput: 328 KB on AGX, 236 KB on Nano). Error bars show + std; dashed lines indicate saturate throughput and full-load latency. Experiments use Linux direct I/O [3 with 6-thread thread-poo -poll in C++.
NEURON CHUNKING
Motivated by the aforementioned considerations, we introduce NEURON CHUNKING, an I/O efficiency- aware activation sparsification method. Unlike traditional sparsification methods, NEURON CHUNK- ING jointly considers neuron importance and the memory access costs associated with retrieving selected neurons from flash memory.
To enable such a joint optimization, two key requirements must be satisfied: (i) latency estimation for given memory access pattern, and (ii) neuron selection that balances costs and benefits. Both tasks must be executed frequently at runtime, once per weight matrix (e.g.. approximately 200 times per frame for LLaVA-One Visionniiin-eeee Miiion-Qeen2-BB [18). Each must therefore complete within very short time frame (i.e., about a few milliseconds). Previous approaches avoided this complexity by assuming that latency scales monotonically with the number of selected neurons-an assumption that does hold in our target workloads.
Our key idea is abstract memory access pattern into concise representation called the contiguity distribution, defined as the frequency distribution of the contiguity of the selected neurons. Concretely, group consecutively selected neurons into chunks, each representing maximal contiguous range of neuron indices within the selected set. For example, selecting neurons with indices {1, 2, 4. 6, 7} function of
yields three chunks: (1, 2), {4), and {6, 7]. We then model the total read latency this contiguity distribution-ee e.g., in the above case, one chunk of size and two of size while discarding global structural cues such the exact spatial arrangement of chunks. This abstraction both simplifies hardware modeling and substantially reduces the search space the neuron selection.
In Section 4.1, present chunk-based latency model. It builds lookup table of per-chunk-size latencies via offline profiling and estimates total latency directly from the contiguity distribution.
Section 4.2, we introduce utility-guided chunk selection algorithm. Given list of activation importance, this algorithm generates candidate neuron chunks of varying sizes and greedily selects the chunks with high utility (i.e., sypportance-pe-laatene ratio).
In Section 4.3, we apply lightweight offline reordering step that groups neurons based on activation statistics improve I/O contiguity, and find that this simple approach suffices without relying on co-activation information.
3.1 Chunk-based Latency Model
We first build a lightweight latency estimation model that predicts the I/O latency arbitrary neuron- access patterns on flash storage. Even after abstracting access pattern into contiguity distribution, the space possible distributions remains combinatorially large (the number of combinations grows exponentially with the number of channels), making exhaustive profiling infeasible. Meanwhile, prior SSD modeling frameworks [ 4 provide device-level fidelity but require low-level hardware configurations or full-system simulations, which impractical during inference.

---

To address this, we propose simple and scalable latency model tailored for inference- sparsifi- cation. We approximate the total read latency of the access pattern as the sum of the latencies of its constituent chunks. Let the selected chunks be C; (i = 1, ... n)), each of size The total lateney is where [(s) denotes the profiled read latency for chunk of size 8. estimated as Tuoaal >x=1 [(
Profiling T[s]. We build a lookup table for [ss via offline microbenchmarks: for each chunk size s, place a throughput-saturann read latency (See Appendix [D] for details). Fixed overheads such as command setup or metadata access during flash read initiation amortized and become negligible in T[s]. This procedure yields stable per-size latencies with low measurement variance.
number of chunks of size fixed strides and measure steady-state
Empirical Validation. We evaluate the effectiveness our latency model across different devices and models, shown in Figure Each plot shows actual and estimated latency values of loading selected chunks from our selection algorithm (Section 3.2)
We observe a near- linear relation between estimated and measured latency, indicating consistent proportional bias] This bias arises from our The lookup table is built under idcalized conditions, where each chunk size is measured in isolation with uniform strides. In contrast, real access patterns interleave diverse chunk sizes and strides, invoking naternnddepndent controller and queue behaviors. These effects accumulate and average out to proportional lift in actual latency. The near-linear correlation weakens for smaller models or lower-end devices, where I/O concurrency is lower and controller dynamics amplify tail latency, reducing the averaging effect. Importantly, the error remains near-linear, leaving the greedy chunk selection algorithm unaffected (see Section 3.2
distribution abstraction and profile setup.
Dvile-lite-2b
60
Measured Leiencc
20-
0.929
0.832
20
10
10 Estimated Latency type up
Estimated Latency (ms)
Estimated Latency (ms)
Estimated Latency (ms)
type wyppyyee.o gale
typu
Ispe:
teppes
type.
Figure 5: Comparison between real and estimated flash access latency across models and devices.
3.2 Utility-Gidded Chunk Selection
Given the latency model, our goal is to jointly consider latency and neuron importance when selecting neurons. The selection problem is intractable, as the number of possible neuron com- binations grows exponentially with the model size. To make the problem tractable, we leverage the contiguity-distributon distribution abstraction together with the additive latency assumption, where total latency is approximated as the sum of per-chunk latencies. Under this scheme, latency becomes decomposable across chunks, which naturally motivates a greedy, chunk-level selection strategy.
3.2.1 Problem Formulation
Given activation magnitudes V IR (where N the number of neurons) and selection budget R (the number of channels to load), we aim to output a binary selection mask M E {0, 1 that maximizes importance per latency:
END V3 M;
N-1 Σ T=0
max Me(0.1)
s.t.
M
Lateney(M)
where Latency MM) is the estimated cost of loading the selected channels, modeled via the contiguity distribution of M.
'The latency gap between two points that differ by one additional chunk remains roughly proportional across the entire plot.
5

---

3.2.2 Algorithmic Procedure
Our algorithm consists of three main stages:
Candidate chunk generation. We construct candidate chunks by sliding windows of multiple sizes over the linear space of neuron indices, and each window position yields one candidate chunk. The maximum chunk size is set the hardware-spees -peaare point where throughput saturates (Section 2.3)
The minimum chunk size and the stride between windows are tunable hyperparameters that control the trade-off between search granularity and computational overhead of the algorithm.
2. Chunk evaluation. Each chunk is assigned a utility score, defined as the sum of its neuron importance values divided by its estimated latency, Latency obtained efficiently from a pre-profiled lookup table based on our latency model.
3. Greedy selection. Chunks sorted by utility, and the algorithm iteratively selects the highest- ranked ones while excluding those that overlap with previously selected chunks, continuing until the selection budget is met.
Because the latency model error is approximately linear, merely scales utility scores by constant factor without changing their relative order. Therefore, the output of the greedy selection remains unaffected.
One limitation of this approach is that it does not account for potential synergies among adjacent low-scoring chunks, which could collectively form more latency-efficient region. Even so, observe that the algorithm performs robustly in practice and effectively identifies high-importance subsets within its runtime constraints (see Section 4.2).
complete implementation, including mask updates, stride scheduling, and latency lookup logic, is provided in Appendix
3.3 Additional Optimization: Hot-cold Reordering
Previous works 2 44 have observed neuron co-activation patterns and improved I/O efficiency in ReI.U-based I.LMs through offline reordering based on these statistics. Motivated by these findings, explore simpler reordering scheme that leverages hot-cold activation patterns observed in prior studies 38 [49). Specifically, we reorder neurons according to their activation frequency. which yields comparable I/O efficiency improvements to co-activation-base methods without the need for complex optimization (See Appendix F G for details). Thus, we adopt this hot-cold reordering as a preprocessing step during the offline profllngg stage.
The procedure as follows. We first count how frequently each neuron activated (designating the top 50% by importance as active) using calibration dataset. Then we sort the neurons in decreasing order of activation frequency. Based on this ordering, we permute the corresponding rows of the weight matrix so that frequently activated neurons are placed together. At runtime, the same permutation is applied to the activation vector, aligning it with the reordered weights. The runtime permutation operation incurs negligible overhead. For example, profiling the down-projection layer of -nneeeiiion on Jetson Orin Nano over 100 trials showed a mean overhead 1.5 ms with 95% confidence bound of 1.8 ms, representing less than 0.02% of total inference latency.
4 Evaluation
4.1 Experimental Setup
Hardware. All experiments were performed on two different embedded device setups, representing low-end and high-end hardware environments:
Jetson Orin Nano (8 GB memory) with SK Hynix Gold P31 SSD (peak sequential read: 3500 MB/s)
Jetson Orin AGX (32 GB memory) with Samsung 990 Pro SSD (peak equential read: 7450 MB/s)
We cache the vision encoder and KV cache in memory. All weights of the backbone LLM are loaded from flash on demand. Unless specified, we use Jetson Orin Nano as our default device, reflecting the memory-constrained setting.

---

HaVAA-OneVision-28 Wiasn-neeiisoon-...8
LongVA-7B
VILA1.5-8B
NVILA-Lite:2B
Ours
3.2
Video DC199 TempCompan
Labency'e
0.45
4.5
2.4
0.30
3.0
0.15
1.5
0.8
'22
56
60
64
0.00
47.5 50.0 52.5
50
52
54
56
0.0
51
34
57
60
0.0
51.0
52.5 54.0
3.2
0.45
241
0.30
3.0
0.15
1.5
0.8
2.502.753.033.25
0.0g
2.
1.8
2.1
0.0
2.1
3.0
0.0
1.752.0022 252.50
NexIQA -N pa on
0.45
4.5
244
0.30
1.6
0.15
1.5
0.00
0.0
0.0
72
76
50
54
65.0
60.062.85 565.6 Accuracy
Accuracy
Accuragy
Accurory
Acouracy
Figure 6: End-to-end performance on Jetson Orin Nano. Latency reported with enror bar 1 std.
Models and Datasets. We use VLM models that operate frame-by-frame, as models that process entire videos once do not fit real-time streaming scenario. We evaluate five models with various sizes: LLaVA-One Vision Owen2-7B, LLaVA-Ori VAATVA--Oeiision--veee Vision-Qwen2-0.35
(18), Llama-3- Llama-3-VILA1.5 VILA1.5- two multiple-choiee video
8B [22] NVILA-LLte-2D (271. and LA-TV [54] For datasets,
QA benchmarks benchnarks-TTeppCoore Compaospass (25] and NExT-QA 47 (randomly sampled 3000 examples)
video description dataset, VideoDetaailCaptio
29 Unless otherwise specified, use LLaVA-
One MieVisson-OOe Dession-wwen2-8B and the TempCompass dataset by default.
Comparison Setup. As a baseline, we implement top-k activation sparsification that selects the most important channels based on activation magnitude, following prior works [2] 16 24). We apply TEAL's 24 profiling-basse
method to determine layer-wise sparsity levels for both the baseline and our method, using 25 out of 410 videos from the TempCompass dataset (excluded from the main evaluation). See Appendix for hyperparameter details.
Metric. We report accuracy and I/O latency. Accuracy is defined as the ratio of correct answers on multiple-choice QA datasets, and as a 0-5 score from ChatGPT-based evaluation on video description dataset, following prior works 4 Due to the large dataset size, accuracy is measured using Supermicro A+ Server 4124GS-TNE with RTX A6000 GPUs. We measure accuracy under sparsity level from 0% to 70% in 10% increments. Each latency experiment was repeated 30 times under identical conditions. We report the median latency together with 95% confidence intervals computed by 10000-sample parammertiic bootstrap (bias-corrected and accelerated, BCa). We enable jetson_ccoooeo clocks and disable swap to ensure stable measurement.
4.2 Experimental Results
Accuracy-Latency Trade-off. Olgure/o/shows the accuracy-latency curves for the baseline and our method across all models and datasets. Our method consistently achieves better accuracy-latency trade-off, with an average I/O speedup of 2. 191 and up to 4.65x at comparable accuracy levels based on linear interpolation. The baseline often suffers from poor latency, especially low medium sparsity levels, occasionally increasing total latency-consisten consistent with our observations in Figure This issue is pronounced in smaller models, where channels are smaller, leading to more fragmented method decisively addresses these inefficiencies by tailoring selection to storage
I/0. In contrast, behavior, enabling consistently faster inference. Note that the slight accuracy gain higher sparsity appear when weak noisy activations are removed. Similar regularization effects have been reported in pruning- based model compression work [5].
Pross-Device Evaluation. Figure 7 presents results Jetson Orin AGX (the full results provided in Appendix | show consistent trends). Our method delivers similar relative improvements, achieving
'For VideoDetailCaption, we queried the OpenAl endpoint gpt-4o-mini stronger version than used in earlier work; thus, results not directly comparable.
mioommin--- 1ani--20 2--22000-772 more recent and
7

---

HaaVA-OneVsioon-E
NV
Ours
Desellne
(Ous))
Temp CompCompass
2.0
11
Laieacy
1.5
1.0
XVIA 2B (Durs)
0.5
A-28
Gerpun
P0
0.0
Chirkkk exlection
56
60
64
52.5
54.0
Accurnoy
Ancurway
Latency (9)
Figure 7: End-to-end performance on AGX.
Figure 8: Latency breakdown.
6000
Mon
OneVsion
Besoline
+Roonder
Boudder Chunking
Froquesry
2300
Mox
30
100
Miax
03
52
20
100
150
Figure 9: Ablation of IWO components: Reordering and Chunk selection.
Figure 10: Contiguity distribution before/after our techniques applied.
average 2.89x speedup and up to 5.76x I/O lateney reduction, computed over the full set of models and datasets. The larger speedup on AGX reflects its wider throughput gap between contiguous and scattered access.
Latency Breakdown. Figure shows the latency breakdown a 5% accuracy drop. The end-to- end speedup is smaller than the 1/O-only gain because compute time remains nearly constant, so its relative share increases as I/O time decreases. This gap could narrow with optimized kernels or 1/O-compute overlap 66. [O], which we do not apply in evaluation. Our method substantially reduces I/O latency while incurring a slight compute increase, as maintaining the same accuracy requires loading marginally more channels. Nevertheless, the overall latency still decreases because data is fetched contiguously rather than through scattered accesses. The chunk selection overhead is 2 ms per weight matrix, totaling roughly 400 ms for the full relative to total inference time.
which is small
Ablation Study. Figure 9 shows the accuracy-latency trade-off as each component is incrementally added: baseline, with hot hot-cold reordering, and with both reordering and chunk selection. For the LLaVA--B model, hot-cold reordering yields up to 1.23x speedup, which increases to to 2.55x when chunk selection is additionally applied. Notably, online chunk selection plays a critical role, as the optimal subset of neurons is input-dependent and cannot be determined offline.
Visualization of Utility-Guided Chunk Selection Figure visualizes selected channels for three variants-baseline. baseline with reordering, and baseline with both reordering and chunk selection along with the contiguity distributions at matched accuracy. Reordering yields only modest gains by loosely clustering frequently activated channels. In contrast, chunk-based selection drives the dominant improvement: targets high-utility contiguous regions, raising the average chunk size from roughly 1-2 to nearly 50. See Appendix for results across broader range of settings.
Discussion and Future Work
Generalization to other models and workloads. The proposed framework extends beyond vision- language models to a broader class of architectures and inference settings. The same principle of hardware-awart structured sparsification applies naturally to multi-token LLM inference scenarios such as speculative decoding, parallel sampling, and batched interactive serving, where activations aggregated across tokens yield smoother neuron- curoonimmpotancce distributions. This property enables

---

latency-awee sparsification to maintain responsiveness in real-time, user-facing applications includ- ing chat assistants and copilots that operate under tight latency constraints.
The approach also generalizes to plain L.Ms and ViT-hased models that exhibit smooth activation magnitudes and operate under I/O-hound conditions. Recent L.LMs increasingly employ non-Rel.U activations such as SwiGLU or GeLU, making them amenable to our chunking formulation. Similarly, ViT-based models on edge devices benefit from reduced access fragmentation across smaller channel dimensions. Overall, these characteristics indicate that the proposed framework provides a general foundation for coupling structured sparsity with hardware-awe aware optimization across diverse model families. Additional details and preliminary results on LLMs and ViTs are presented in Appendix
Impact of Emerging I/O Mechanisms. Emerging I/O frameworks such as io_ jo_rrin 4 offer improved support for asynchronous and scattered reads, which may reduce the performance gap between random and contiguous access. However, advances in storage hardware (e.g., internal prefetching, read coalescing) will likely continue to favor contiguity, suggesting that structured access optimization will remain beneficial.
Leveraging Additional Memory Budget for Caching. While our method assumes minimal memory availability, additional latency reduction is possible when the device has sufficient memory to cache frequently accessed weights. Caching strategies (e.g., hot-neuron caching) proposed in prior can be applied in a complementary manner by simply assigning zero importance to cached neurons. Once hot weights are cached, the remaining uncached accesses become more scattered (even after reordering). making our chunk- selection more critical for sustaining I/O efficiency. However, if the device has sufficient memory to cache large portion of the model weights, the overall flash I/O volume becomes negligible, reducing the benefit of our method.
works 2 38 49
6 Related Work
6.1 Activation Sparsification
Activation sparsification, which selectively loads weight channels corresponding to large activations, has been widely studied (See Appendix [.2 for details). Deja Vu (28) observed that MLP layers in LLMs exhibit significant dynamic sparsity, while CATS [16] extended this insight to modern LLMs. which utilize gated MLPs with non-ReLU activations. TEAL (24) further explored sparsification by applying it to attention layers. However, these methods are model-centric-they sparsify solely based on activation magnitude without considering hardware-level access patterns. This design choice was reasonable in their settings, where all weights reside in GPU VRAM and the bandwidth between device and shared memory saturates quickly even with limited access contiguity. In contrast, in flash- offloaded settings, such model-centric sparsification leads significant performance degradation. Other approaches [31 37 (39] seek to apply ReLU-ification to non-ReL R-beedd LLMs, fine-tuning them to enhance sparsity. Although effective, this method demands extensive retraining on a minimum of 50 billion tokens.
6.2 LLM Weight Offloading
LLM weights often exceed GPU VRAM capacity, prompting various offloading strategies. Some approaches [3] 36]38 38) offload weights to CPU memory. This is impractical on edge SoCs with unified memory, where the CPU and GPU draw from the same DRAM pool and no additional capacity is gained [32].
Several recent works 2 44 49 adopt flash-based offloading and propose techniques to reduce I/O latency. LLM in Flash 2 and PowerInfer-2 49 improve I/O efficiency by bundling channels across projection layers, but the resulting gains in access contiguity are limited and rely on large memory budgets for caching (see Appendix D.. Ripple 4 enhances access locality through offline neuron reordering, yet its reliance on ReLU-based sparsity and lack of hardware-and aware runtime mechanisms limit its effectiveness to modern VLMs (see Section 4.2). These approaches were sufficiently effective for ReL.Ubasedd L.Mss with high activation sparsity, where the total I/O volume was low enough to offset efficiency degradation. In contrast, VL.Ms exhibit smoother activation distributions and higher I/O demand, where such techniques fail to sustain efficiency under realistic I/O constraints.

---

6.3 LLM Compression
Various LLM compression techniques, including quantization 7 46], weight pruning 30 [88] and distillation [33 40 [1], have been proposed to reduce computational and memory overhead. In contrast to these static compression approaches, activation sparsification is inherently input-adaptive, providing distinct advantage in exploiting runtime activation dynamics. These methods are orthogonal to activation sparsification and can be combined to effectively mitigate I/O
latency from storage devices [24]
7 Conclusion
We presented NEURON CHUNKING, a latency-aware activation sparsification approach tailored for flash-oflloaded VLM inference. Unlike prior methods that treat I/O latency as a function of volume alone, our method models the performance implications of access contiguity and aligns neuron selection with storage behavior. We show that contiguity- guided chunk selection algorithm consistently improve the securacy-latency trade-off. These results underscore the importance of co-designing sparsification with hardware characteristies for efficient edge inference.
latency model and utility-
Acknowledgments and Disclosure of F Funding
We sincerely thank our anonymous reviewers for their valuable comments. This work was supported by National Research Foundation (NRF) funded by the Korean government (MSIT) (No. RS-2024- 00463802).
References
[1] Ilama.cpp. https://github. ctw. caoom/ggmlorrr hetg//laaaa... cpp ggml-orge 1lama. cpp Accessed: 2025-02-27
URL httpp::/gitthb..co
[2] Keivan Alizadeh, Seyed Iman Mirzadeh, Dmitry Belenko, S Khatamifard, Minsik Cho, Carlo C Del Mundo, Mohammad Rastegari, and Mehrdad Farajtabar. Llm in flash: Efficient large language model inference with limited memory. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 22562-1256 2024.
[3] Reza Yazdani Aminabadi, Samyam Rajbhandari. Ammar Ahmad Awan, Cheng Li, Du Li, Elton Zheng, Olatunji Ruwase, Shaden Smith, Minjia Zhang, Jeff Rasley, et al. Deepspeed- inference: enabling efficient inference of transformer models at unprecedented scale. In SC22: International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1-15. IEEE, 2022.
[4] Jens Axboe. Efficient i/o with io uring. https://kernel. wttii_urkin uring pdf 2019. Ac- cessed: 2025-05-15.
[5] Joya Chen, Zhaoyang Lv, Shiwei Wu, Kevin Qinghong Lin, Chenan Song, Difei Gao, Jia- Wei Liu, Ziteng Gao, Dongxing Mao, and Mike Zheng Shou. Videollm-online: Online video large language model for streaming video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pallern Recognition, pages 18407-18418, 2024.
[6] Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-efficient exact attention with io-awareness. Advances in neural information processing systems, 35: 35:16444- 2022.
[7] Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Gpt3. int8 (): 8-bit matrix multiplication for transformers at scale. Advances in neural information processing systems, 35: 1333332, 2022.
[8] NVIDIA Developer Forum. Gen 3 PCIe NVMe SSD with Lanes Gets Higher IOPS on Nano Compared to the Xavier NX. https://forums. developer nvidia. com/t/
10

---

gen- wwwe---ee-sssddeeeeee witt- lanel-nness-e 228818 2022. [Accessed 21-10-2025].
-iee-xaveer-nn
[9] Sylvain Gugger, Lysandre Debut, Thomas Wolf, Philipp Schmid, Zachary Mueller, Sourab Mangrulkar, Marc Sun, and Benjamin Bossan. Accelerate: Training and inference at scale made simple, efficient and adaptable. https:/ hwwpa//.ww./uggggggggc/////eeeee Accessed: 2025-02-27.
2022.
[10] Liwei Guo, Wonkyo Choe, and Felix Xiaozhu Lin. Sti: Turbocharge nlp inference the edge via elastic pipelining. In Proceedings of the 28th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2, pages 791-803, 2023.
[11] Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. In NeurlPS, 2015.
[12] Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint 1606.0660060811. 2016.
[13] Joo Kim, Donghyuk Lee, Sam Choi, and Myoungsoo Jung. Simplessd: Modeling sigh-fidliity modern ssds for holistic system simulation. IEEE Computer Architecture Letters, 15(1):31-34 2016.
[14] Eldar Kurtic, Daniel Campos, Tuan Nguyen, Elias Frantar, Mark Kurtz, Benjamin Fineran, Michael Goin, and Dan Alistarh. The optimal bert surgeon: Scalable and accurate second-order pruning for large language models. arXiv preprint vrXiv:2203.07254 2022.
[15] Yann LeCun, John Denker, and Sara Solla. Optimal brain damage. In NeurIPS, 1990.
[16] Donghyun Lee, Je-Yon Yong Lee, Genghan Zhang, Mo Tiwari, and Azalia Mirhoseini. Cats: Contextually-aware thresholding for sparsity in large language models. arXiv preprint arXiv:2404.0866, 2024.
[17] Jingyu Lee, Hyunsoo Kim, Minjae Kim, Byung-Gor Chun, and Youngki Lee. Maestro: The analysis-simulation integrated framework for mixed reality. In Proceedings the 22nd Annual International Conference on Mobile Systems, Applications and Services, pages 99-112, 2024.
[18] Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer. arXiv preprint MrXiv:2408.03326 2024.
[19] Mengtian Li, Yu-Xiong Wang, and Deva Ramanan. Towards streaming perception. In Com- puter Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Purt II 16, pages 473-488. Springer, 2020.
[20] Zonglin Li, Chong You, Srinadh Bhojanapalli, Daliang Li, Ankit Singh Rawat, Sashank J Reddi, Ke Ye, Felix Chern, Felix Yu, Ruiqi Guo, et al. The lazy neuron phenomenon: On emergence of activation sparsity in transformers. arXiv preprint arXiv:2210.06311 2022.
[21] Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. Awq: Activation-aware weight quantization for on-device Ilm compression and acceleration. Proceedings of Machine Learning and Systems, 6:87-100, 2024.
[22] Ji Lin, Hongxu Yin, Wei Ping, Pavlo Molchanov, Mohammad Shoeybi, and Song Han. Vila: On pre-training for visual language models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 26689-26699, 2024.
[23] Linux. llama.cpp. html URLhtttpa Accessed: 2025-03-05.
kernel. http////ww..eeeeeeeerr/////////a
[24] James Liu, Pragaash Ponnusamy, Tianle Cai, Han Guo, Yoon Kim, and Ben Athiwaratkun. Training-free activation sparsity in large language models. arXiv preprint arXiv:2408. 14690, 2024.

---

[25] Yuanxin Liu, Shicheng Li, Yi Liu, Yuxiang Wang, Shuhuai Ren, Lei Li, Sishuo Chen, Xu Sun, and Lu Hou. Tempcompass: Do video Ilms really understand videos? arXiv preprint arXiv:2403.00477 2024.
[26] Zejian Liu, Fanrong Li, Gang Li, and Jian Cheng. Ebert: Efficient bert inference with dynamic structured pruning. In Findings of the Association for Computaaional Linguistics: ACL-IJCNLP 2021, pages 4814-4823, 2021.
[27] Zhijian Liu, Ligeng Zhu, Baifeng Shi, Zhuoyang Zhang, Yuming Lou, Shang Yang, Haocheng Xi, Shiyi Cao, Yuxian Gu, Dacheng Li, et al. Nvila: Efficient frontier visual language models. arXiv preprint 24X1:2241..44468 2024.
[28] Zichang Liu, Jue Wang. Tri Dao. Tianyi Zhou, Binhang Yuan, Zhao Song, Anshumali Shrivas- tava, Ce Zhang, Yuandong Tian, Christopher Re, et al. Deja vu: Contextual sparsity for efficient Ilms at inference time. In International Conference on Machine Learning, pages 22137-22176. PMLR, 2023.
[29] LMMs-Lab. Videodetailcaption dataset. https://mnngnnaacccc////aattttts VideoDetailCCatioon 2024. Accessed: 2025-05-15.
[30] Xinyin Ma, Gongfan Fang, and Xinchao Wang. LIm-pruner: On the structural pruning of large language models. Advances in neural information processing systems, (6:211002-2172)) 2023.
[31] Iman Mirzadeh, Keivan Alizadeh, Sachin Mehta, Carlo C Del Mundo, Oncel Tuzel, Golnoosh Samei, Mohammad Rastegari, and Mehrdad Farajtabar, Relu strikes back: Exploiting activation sparsity in large language models. arXiv preprini arXiv:2310.04564 2023.
[32] Jetson AGX Orin Series Technical Reference Manual. NVIDIA Corporation, 2022. URL petps://developoet nvidia. com/ bebbbeddddddoomooloe Unified memory architecture specification.
[33] Haojie Pan, Chengyu Wang, Minghui Qiu, Yichang Zhang, Yaliang Li, and Jun Huang. Meta-kd: A meta knowledge distillation framework for language model compression across domains. arXiv preprint arXiv:2012.01226. 2020.
[34] PyTorch Core Team. Sort.cu: Cuda sorting kernels in pytorch. https://githubb com/pytorch/ http://torrhpyytccc/bbbbeoee http//yytttc/hooooe/o wwtpoo//aanaattttnea/aeeeeeeeeee httt:///eee//matee////a////tttte cu, 2025. Ac- cessed: 2025-05-23.
[35] Noam Shazeer. Glu variants improve transformer. arXiv preprint arXiv2200.05202, 2020.
[36] Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Beidi Chen, Percy Liang, Christopher Ré, lon Stoica, and Ce Zhang. Flexgen: High-throughput generative inference of large language models with a single gpu. In International Conference on Machine Learning, pages 31094-31116 11.. PMLR, 2023.
[37] Chenyang Song, Xu Han, Zhengyan Zhang, Shengding Hu, Xiyu Shi, Kuai Li, Chen Chen, Zhiyuan Liu, Guangli Li, Tao Yang, et al. Prosparse: Introducing and enhancing intrinsic activation sparsity within large language models. arXiv preprint arXiv:2402. 13516. 2024.
[38] Yixin Song, Zeyu Mi, Haotong Xie, and Haibo Chen. Powerinfer: Fast large language model serving with a consumer-grade gpu. In Proceedings of the ACM SIGOPS 30th Symposium on Operating Systems Principles, pages 590-606, 2024.
[39] Yixin Song, Haotong Xie, Zhengyan Zhang, Bo Wen, Li Ma, Zeyu Mi, and Haibo Chen. Turbo sparse: Achieving I1m sota performance with minimal activated parameters. arXiv preprint arXiv:2406.05955. 2024.
[40] Siqi Sun, Yu Cheng, Zhe Gan, and Jingjing Liu. Patient knowledge distillation for bert model compression. arXiv preprint arXiv: 09355, 2019.
[41] Siqi Sun, Zhe Gan, Yu Cheng, Yuwei Fang. Shuohang Wang, and Jingjing Liu. Contrastive distillation on intermediate representations for language model compression, arXiv preprint axXiv:2009.14167 2020.
12

---

[42] Amy Tai, Igor Smolyar. Michael Wei, and Dan Tsafrir. Optimizing storage performance with calibrated interrupts. ACM Transactions on Storage (TOS), 18(1):1-32, 2022.
[43] Arash Tavakkol, Amirali Boroumand, and Onur Mutlu. Mqsim: A framework for enabling realistic studies of modern multi-queue ssd devices. In FAST, 2018.
[44] Tuowei Wang, Ruwen Fan, Minxing Huang, Zixu Hao, Kun Li, Ting Cao, Youyou Lu, Yaoxue Zhang, and Ju Ren. Ripple: Accelerating I1m inference on smartphones with correlation-awan neuron management. arXiv preprint (.div:2410.1922222 2024.
[45] Yuctian Weng, Mingfei Han, Haoyu He, Xiaojun Chang, and Bohan Zhuang. Longvlm: Efficient long video understanding via large language models. In European Conference on Computer Vision, pages 453-470. Springer, 2024.
[46] Guangxuan Xiao, Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models. In International Conference on Machine Learning, pages 38087-38099, PMLR, 2023.
[47] Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. Next-qa: Next phase of question- answering to explaining temporal actions. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 9777-9786, 2021.
[48] Dongkuan Xu, Ian EH Yen, Jinxi Zhao, and Zhibin Xiao. Rethinking network pruning-under the pre-train and fine-tune paradigm. arXiv preprini wrXiv:2104.08682 2021.
[49] Zhenliang Xue, Yixin Song, Zeyu Mi, Xinrui Zheng, Yubin Xia, and Haibo Chen. Powerinfer-22 Fast large language model inference on smartphone. arXiv preprint PrXiv:2406.06282 2024.
[50] An Yang, Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115. 2024.
[51] Kichang Yang. Juheon Yi, Kyungjin Lee, and Youngki Lec. Flexpatch: Fast and accurate object detection for on-device high-resolution live video analytics. In IEEE INFOCOM 2022-IEEE Conference Computer Communications, pages 1898-1907. IEEE, 2022.
[52] Kichang Yang, Minkyung Jeong, Juheon Yi, Jingyu Lee, KyoungSoo Park, and Youngki Lee. Logan: Loss-tolerant live video analytics system. In Proceedings of the 30th Annual International Conference on Mobile Computing and Networking, pages 1314-1329, 2024.
[53] Shengyuan Ye, Jiangsu Du, Lickang Zeng, Wenzhong Ou, Xiaowen Chu, Yutong Lu, and Xu Chen. Galaxy: A resource-efficiet collaborative edge ai system for in-situ transformer inference. In IEEE INFOCOM 2024-IEEE Conference on Computer Communications, pages 1001-1010. IEEE, 2024.
[54] Peiyuan Zhang. Kaichen Zhang. Bo Li, Guangtao Zeng, Jingkang Yang, Yuanhan Zhang, Ziyue Wang, Haoran Tan, Chunyuan Li, and Ziwei Liu. Long context transfer from language to vision. arXiv preprint 24f.::2006.685222 2024. http:: org/
16852
[55] Yuanhan Zhang, Bo Li, haotian Liu, Yong jae Lee, Liangke Gui, Di Fu, Jiashi Feng, Ziwei Liu, and Chunyuan Li. Llava-next: A strong zero-shot video understanding model, April 2024. URL https://llava- wwt.tb.iitotiii http:///w..ioooooo///000000----0
13

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Yes, the main claims the abstract and introduction are well-aligned with the paper's contributions.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Yes, the paper discusses its limitations, including the greedy algorithm's 's inability to capture interactions between adjacent low-scoring chunks, and the latency model's exclusion of overheads, particularly in low-volume www-performance settings.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Liimitations section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and prool's
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
14

---

Answer: [NA]
Justification: N/A - the paper does not present formal theoretical results or proofs, as the focus is empirical modeling and algorithm design for practical sparsification under I/O constraints.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: Yes, the paper provides sufficient detail to reproduce the main experimental results, including latency modeling procedures, benchmark setup, hardware configurations, and algorithmic design. While full code and data availability may vary, all key components affecting the main claims are clearly described.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided not.
If the contribution a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, if the contribution is specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
Depending the contribution, reproducibility
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, may be that access to the model is limited in some
15

---

way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: have open-sourced the code and provided with sufficient instructions to reproduce the main experimental results.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
for more details.
While encourage the release of code and data, understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https //nips. http//ullc/guuddese//eeee
PPolicyy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reprodueible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized ver- sions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Yes, although the paper does involve training models from scratch, provides detailed information on the experimental setup used for evaluation, including sparsity levels, chunk size configurations, latency measurement procedures, and hardware settings. These details are sufficient to understand and interpret the results.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Yes, the paper reports medians with 95% bias-corrected and accelerated (BCa) bootstrap confidence intervals across 30 independent runs for key latency experiments, providing an appropriate indication of variability and robustness in the results.
Guidelines:
16

---

The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the emor bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report I-sigma error bars, but one should state it. The authors should preferably report 2-sigma error bar than state that they have 96% CI. if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: The paper provides hardware details for all experiments, including device models (Jetson AGX Orin, Jetson Orin Nano), SSD types, and use of Linux direct I/O with multi-threaded C++ implementations. While exact memory usage and execution time are not reported for each experiment, the provided information sufficient approximate the computational setup required for reproduction.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics http://///eearepp.ccpccppb//////
Answer: [Yes]
Justification: Yes, the research conforms to the NeurIPS Code of Ethics. It does not involve human subjects, sensitive data, deceptive practices, and it promotes efficient computation by aligning sparsification with hardware constrainss-conttibbtigg contrbbuting positively to sustainability and resource-aware machine learning.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consideration due to laws regulations in their jurisdiction).
17

---

10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: N/A the paper focuses on algorithmic and system- levell contributions and does not discuss societal impacts.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release models, providing defenses addition to attacks, mecha- nisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: N/A - the paper does not release new pretrained models or datasets that pose high risk of misuse.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have a high risk for misuse dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions access the model implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
18

---

Justification: Yes, the paper properly credits the creators of all models and datasets used, such as Qwen2-7B and LLaVA-One Vision, and uses them in accordance with their respective licenses and terms of use.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include a URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset.
For scraped data from particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets. paperswi thcode com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
The paper introduces a new algorithm and latency modeling framework. While these are clearly described in the main text and appendix, the associated code and assets are not yet released. We are planning to open-source the implementation, which is expected to include accompanying documentation.
Justification:
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mooel as part of their sub- missions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: N/A the paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage the country of the data collector.
15. Institutional review board (TRB) approvals or equivalent for research with human subjects
19

---

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: N/A the paper does not involve human subjects and therefore does not require IRB approval.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [Yes]
Justification: Yes, the paper clearly describes the usage of LLMs (e.g.. Qwen2-7B, LLaVA- One Vision) as part of the experimental setup for evaluating sparsification strategies during inference. Their role central to the method and analysis, and they are properly credited.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy hbtree///eerrpppcc//oo/dererrrr/ cetpp//epsscpecooocccccccc/s//// 2025/ /LMM) for what should or should not described.
20

---

A Appendix Overview
This appendix provides additional details supporting the main paper.
Default Setup. Unless otherwise specified, we adopt the same default configuration as used the main paper's s evaluation: Llava-One Vaseei-iiiinnnnn
[8] as the model and the TempCompass [25]
dataset as the input source. For end-to-end evaluation on Jetson Orin AGX (Appendix []. we use the full set of evaluation datasets following the setup in the main paper. For analysis-oriented experiments-inclinne
visualization, reordering, and activation statistics we use a subset of 25 videos from TempCompass that were excluded from the main evaluation (as described in Section 5.1). In the reordering experiment, 20 videos are used for calibration and 5 for validation (i.e., result visualization). When analyzing attention and MLP modules, projections, and omit k, v, and up since they share input activations with q and gate, respectively. Additionally, include three representative layers- early (0), middle (13), and late (27) capture variation across layer depths (LLaVA-One Vision-Qwen22-7 has 28 layers). We restate this setup where relevant throughout the appendix.
focus the q. o, gate, and down
B Extended Background on VLM Sparsification
B.1 Vision-Language Model Inference Process
A vasion-language model consists of vision encoder fuision projector fyroj. and a backbone LLM fum.
Given a language prompt {t1, . t of m tokens, the model first performs a prefill step that processes all tokens once generate key-value (KV) caches for each transformer layer:
fm+1, KV <m+11
fumm (p)
In the frame appending stage, each video frame F; (i = 1,..., N) is processed upon arrival. Each frame is encoded into n visual tokens via the vision encoder and projector:
(
{uj U2 ..... o foros (fesston (Fi))
(i) (6)
These tokens are then fed into the LLM to produce additional KV pairs, which are appended to the existing cache: imini+1. KV <m+ni+1 fum (v((), KV <m=n(i-1)44)
During the prefill and frame appending stage, the token output of fum ignored; only the KV cache is used in subsequent decoding.
In the decoding stage, new token generated one time autoregressively. At decoding step the model takes the previous token Em+nN4j and the current KV cache to generate the next token:
fm+nN+++LL KV <mmtn+++411 = fusm (fm+ +j: KV <m=+N+5))
The decoding stage may begin in one of two ways: either via an explicit query provided by the user (e.g., a natural language instruction, which is appended in a similar way to visual tokens), or by designing the model emit special control token at important frames, signaling that decoding should commence. In the former case, decoding starts after appending the query tokens; in the latter, the final generated token from the last frame preserved and used as the first input of the decoding stage.
B.2 Activation Sparsification
Notation. Let the activation vector (also referred to as hidden states) for single layer be E R and the corresponding weight matrix be W E Rm where m is the number of neurons (also referred to channels) and is the output dimension. Each row W, E R" of W corresponds to single neuron, contributing to the output through the dot product a, W,. Thus, the output E R" is computed as:
m
y y=u Tw
a.W.
i=1
21

---

which can be interpreted as a weighted sum over neurons, where the activation values act as per-sample dynamic weights.
Saliency via Magnitude. In modern LLMs where non-ReLU activation functions (e.g.. SwiGLU, GeGLU) are standard, activation values are not exactly zero as opposed ReLU-based activa- tion functions, which produce exact zeros for inactive neurons. As a result, identifying salient neurons-thow most critical to output quality-is nontrivial. Prior works such as TEAL [24 and CATS [16] propose using the magnitude of activations as proxy for saliency. This approach assumes that neurons with higher |ay] contribute more significantly to the output.
Magnitude- Sparsification. Given sparsity target E [0, 1), the goal is to retain only the top- 1 s)m neurons per input based on their importance. The process as follows:
1. Compute importance scores U;
|ay| for ,,...,
2. Select binary mask M {0, 1) such that M, if [a is among the top- s)m entries of v, and M;- = 0 otherwise.
3. Construct the sparsified output:
Ma,WW
1=1
This technique is input-dependent and requires re-evaluation of M at each inference step. While simple and effective, it does not consider the memory access cost retrieving weight rows from flash storage, which becomes critical in selection to use fixed activation threshold to filter out low- -ww-imprttance neurons.
inference settings. An alternative to top- 1op-(1= s)m
In vision-language models (VLMs), where a single input (e.g., image) corresponds to multiple tokens, extend this method by computing the importance of each neuron the average absolute activation magnitude across tokens. This yields a single importance vector per input, allowing sparsification to proceed as in the single-token case.
C Additional Evidence of Activation Smoothness Across VLMs
To further validate that the smoothing effect is general architectural property of VLMs rather than model-specific behavior, we measured the coefficient variation (CV) of neuron importance before the down-projection layer-where conventional sparsification is typically applied in ReLU-based across
LLMs across multiple VLM architectures and ReLU-based baseline (OPT-6.7B).
Table 1: Coefficient of variation (CV) of neuron importance before the down-projection layer across
multiple models.
Layer LLaVA-7B
LLaVA-0.5B
VILA--BB
NVILA-2B
LongVA
OPT-6.7B
First 1.44
1.31
1.25
1.07
1.20
11.65
Mid
1.25
1.33
1.38
1.32
1.34
8.63
Last
3.30
3.58
2.48
4.55
3.01
9.19
Across all VLM models, the CV values (1.07-4555) are dramatically lower than those of the ReLU- based baseline (8.63-1 1.65), demonstrating that smooth activation distributions consistent property of modern VLM architectures. This smoothing effect makes contiguity-aware selection particularly beneficial for VLMs: when importance differences between neurons small, I/O efficiency becomes the decisive factor for overall performance.
D Benchmark Details
profiled read throughput as a function chunk size on two devices: Jetson AGX Orin (Samsung 990 Pro SSD) and Jetson Orin Nano (SK Hynix Gold P31 SSD), Each device reaches 99% of its peak
22

---

throughput at approximately 236 KB (AGX) and 348 KB (Nano). Measurements were taken in KB increments up the saturation point, with all runs completing within 20 minutes per device.
Profiling Setup.
Prepare large dummy file (e.g. 128MB) on flash-backed storage.
Issue sequential reads of size 2 2KB... , where Smax is the smallest size reaching 99% of peak throughput.
Record average throughput over multiple trials.
Throughput variance was negligible (standard deviation <1 of the mean) across all sizes.
E
Algorithm Implementation Details
Algorithm provides pseudocode of multi-scale chunk selection method. Below, we describe the corresponding implementation in detail, which designed for runtime efficiency and integrates both CPU and GPU components.
Inputs.
The inputs to Algorithm D
are:
E RV; Activation magnitudes.
R: Total number of rows to select.
row_size_KBi Size each row in kilobytes, used to convert kilobyte-hased parameters to row units.
[minn Smax] and As (in KB): Define the chunk size range and the granularity of sizes considered.
jump_cap (in KB): Limits the maximum stride between starting indices of candidate chunks for efficiency.
- By default, stride equals the chunk size (i.e., non-overlapping).
the chunk size exceeds jump_cap, stride is clipped to the cap. allowing overlapping candidates.
L(+): pevice-specifie latency lookup function mapping chunk size (in rows) to access cost.
the actual implementation, a device flag selects the appropriate lookup table (AGX or Nano). In the pseudocode, this is simplified by directly passing L-)).
Prefix Sum. To enable constant time computation of the sum of importance for any contiguous chunk, CPU-side prefix sum of the activation magnituees is first computed.
Chunk Candidate Generation. For each chunk size (converted to row count), the algorithm slides window across the activation vector steps of min(s, sump_cap)) Each candidate chunk is scored using the ratio of summed importance to estimated latency, with latency values retrieved from a pre-profiled lookup table L(s) based on hardware throughput (see Appendix D).
GPU Sorting. The importance-to-ee -otsotsst scores of all candidate chunks transferred to GPU memory and sorted in descending order using PyTorch's GPU-accelerated sort. This step enables scalable candidate prioritization with minimal overhead.
Greedy Selection. Candidates are selected greedily based on the sorted scores. Each selected chunk is added to the output mask if it does not overlap with already selected rows and does not exceed the remaining budget R. Overlaps are checked with early termination, and the mask is updated in in-place.
The algorithm design reflects the trade-off between optimality and runtime feasibility: by limiting the chunk search space and leveraging GPU sorting, it enables input-dependent sparsification inference time within few milliseconds latency.
23

---

Algorithm Multi-scale Chunk Selection
Require: Activation magnitudes V TR-V number of rows select R, row size in KB, chunk size range (8min: Smax] in KB, step size A in KB, jump cap in KB, latency lookup function I.())
Ensure: Binary mask indicating selected rows
1: Convert chunk-related parameters to row units:
Tmin f- max(1, [mmin/rrn Stwowwize_Kee KBJ)
Tmax + max(1, /mmxx/row___ wtwowwize___aae KB__
Ar f- max (1, As/row size _i_____eeeee
Sump_cpprrww rop_rowwooo <- max(1, |jump_cup cap/row_siii size__ieeeeeee si___izee_BB]
2: Compute prefix sum array: cumsum[0 N k prefix prefix_sum(V)
3: Initialize empty candidate list C
4: for from rmin to I "max with step Ar do
5: stride +- min(m,,,uuum jumppp_jupppee _ap__capp___ss
6: for = 0 to .
with step stride do 7: benefit +- cumsumli + r| cumsum[i]
8: cost + Lrr)
9: Append candidate (benefit/cost, i, to C
10: end for
11: end for
12: Sort by score descendingly (GPU-accelerated)
13: Initialize mask [0 N] 0, selected 4- 0
14: for candidate r) in sorted do
15: if chunk overlaps selected rows > R selected then
16:
continue
end if
18: Set mask(i i + r| +-
selected selected
20: if selected 2 R then break 22: end if
end for
24:
return mask
Neuron Activation Frequency Analysis
Figure 11 illustrates the distribution of neuron activation frequency across different layers when the effective sparsity is 40%. The plot is structured as a 3 x grid, where each row corresponds to layer and each column to activation type. Many neurons are neither always-on nor always- off, confirming the presence of input-dependent sparsity in VLMs, consistent with prior findings in LLMs (20] 28). This suggests that input-aware sparsification remains effective in our setting, although such dynamic sparsity inevitably leads to fragmented access patterns.
Additionally. TEAL profiling introduces sparsity variation across layers, resulting in some layers with very high or low sparsity (e.g.. q projection layer has 94% sparsity). These layers exhibit high proportion of hot or cold neurons, suggesting that simple offline hot-cold reordering can be effective for improving contiguity in these cases.
G Impact of Offline Reordering Schemes
Although offline reordering is not our primary focus we target online Dolicies-Figure compares the contiguity of selected neurons before and after applying offline reordering, using either hot-cold method. Both methods yield modest improvements over the original ordering, with comparable gains across most layers. While Ripple performs better in one case (the o projection of layer 0), the overall difference minor, suggesting that hot-cold reordering offers lightweight and effective alternative.
reordering or Ripple's 44 coactivation-base

---

gate sparsity=0.35
spersity=0.94
sparsity=0.26
oparsity=0.48
Layer 0
hatácold ratio: 70%
hot&cold ratio: 22%
hot&cold ratio: 28%
hot&cold ratio: 38%
0.05
0.05
0.05
0.05
0.00
0.00
0.00
0.00
400
400
sparsity=0.60
sparsity=0.09
sparsity=0.41
sparsity=0.52
Layer 13
0.10
hatkcold ratio: 57%
hot&cold ratio: 62%
hot&cold rati:: 9%
hot&cold ratio: 53
0.05
0.05
0.05
0.05
0.00
0.00
0.00
0.00
400
600
200
400
400
400
600
aparsity=0.88
sparsity cerssity-O.11. 0.13
sparsity=0.33
aparsity=0.70
Layer 27
density
0.10
hot&cold ratio: 73%
hot&cood ratio: 33%
hot&cold ratio: 9%
hotkkcold ratio: 65%
0.05
0.05
0.05
0.05
0.00
0.00
0.00
0.00
200
100
600
200
400
600
200
100
600
200
400
600
Activation Frequency
Activation Frequency
Activation Frequency
Activation Frequency
Figure 11: Activation frequency of neurons across layers, with effective sparsity set to 40% (layer- wise sparsity determined by TEAL [24 profiling). The text in the center of each plot indicates the proportion of hot neurons (activated >99% of the time) and cold neurons (<1%).
Original
Hot-cold
Ripple
gate
down
1.0
1.0
1.0
1.0
Layer
CDF
0.5
0.5
0.5
0.5
0.0
0.0
0.0
0.0
50
100
0
50
100
50
100
50
100
1.0
1.0
1.0
1.0
Layer 13
CDF
0.5
0.5
0.5
0.5
0.0
0.0
50
100
100
50
100
50
100
1.0
1.0
1.0
1.0
Layer 27
CDF
0.5
0.5
0.5
0.0
0.0
0.0
0.0
0
50
100
0
50 Contiguity
100
50
100
50 Contiguity
100
Contiguity
Contiguity
Figure 12: CDF of contiguity of selected neurons before and after reordering, with sparsity=0.4.
25

---

H Hyperparameter Selection
n0
60
Jano DND in J..
Hanp ONp in 2b
RD
Jonp pop in
June cup
60-
20
2
ctunk
CbNniE
cbunk,
Shape: 3084x1 5964
Shape: 6000x4096
Shape: 12944x3584
Shape: L530x1 536
Jung 355
JuNe Gap 36
60-
80
20-
20
40
60
20
40
60
20
40
60
20
4.0
60
chunk, Mart in kb
chank, ar slact in, kb
chank XX, start in, kb
chunk XX. start, in. kb
Shape: L530x256
Shape: 8PGx128
Shape: 1433886006
4864x886
Nino 03 83
.000
Jimp cap kbb
60
Jump DED in Abb
Jump aN la Ab
50
80-
20
20
0-
0
Shape: 53
Shape: 2954ttis
Shape: 40189444221
Shape: 25
June can 10
janp cop in. kh
Jung CNO in kb
Jonp ONN in 3b
20
20
20
20
60
40
60
chunk startttttt kh
chank start Jbb
obunk in kbb
ctunk 17, www.ttttttt...... In
(a) Jetson Orin AGX
Stape: 2586x2584
Shape: 8860s118 155
Shape: 880xt861
Shape 4036x1000 024
ALED cap 5b
40
40
40
40
Larger scorch spare
20-
20
20
20
40
60
20
40
60
20
40
60
20
40
60
chunk sz start in kb
chunk 22 stact In. Ab
cbunk S2. start la. lb
cbunk 2z start In kb
Shaps: 385441
Shaps: Grutietove
Shaps: 1RM4Aee0e
Shape: 15.6e1
60-
jump ppp Wb
60
Jump DND Abb
Jump spp in lbb
20
0-
60
Shape: 15309256
Shaps Balke 23
Shaps: 16330ee006
Shege: Anreetion
Junp ONN in 3b
Jum in.
Jung cup in
20
60
40
60
40
60
40 60
chunk start Jn, Xb
chunk start 3t, 1bb
cbunkk start In, Lib
cbunk S2. start In. lbb
Shage 35M45512
Shape: BRGxton
Shapo- 4096<14336
Shape: 15W6neen
86
Jano Dap in 26
80
60
40
60
40 60
start
start
ovarhead Zno
overhead 2na
(b) Jetson Orin Nano
Figure 13: Runtime overhead of chunk selection across hyperparameter configurations Jetson Orin AGX (top) and Jetson Orin Nano (bottom). Each point represents a configuration defined by starting chunk size (chunk_ taackkss_sttrr_..
kb, .z-axis) and jump cap (jump- (jump_cap___ in bbb,,, y-axis). Step size set equal to the start size; the chunk size end is fixed from I/O profiling (236 KB for AGX, 348 KB for Nano). Circle size is inversely proportional to runtime, and color indicates whether the 2 ms latency threshold is exceeded.
26

---

The hyperparameters of our chunk selection algorithm are chosen with two objectives in mind: (i) the runtime overhead must remain within a practical lateney threshold (under ms), and (ii) the compromise in selection quality for computational efficiency should be minimal.
We adopt two-stage selection strategy. First, filter out configurations that exceed ms of runtime overhead. Since the overhead depends on the shape of the weight matrix, we benchmark each configuration across representative matrix shapes drawn from the models used in our evaluation. Measurements are conducted at sparsity 0.1 to conservatively capture the worst-case overhead.
Among the remaining feasible configurations, of the search space-where wheeee chunk sizes grow fine-grained manner and the chunk stride is small. These settings allow for broader search coverage while maintaining overhead within budget.
heuristically select those near the lower-left region
Figure 13 shows the measured overhead for Jetson Orin AGX (top) and Jetson Orin Nano (bottom). For each device, we sweep over the starting chunk size (chunk_sze ss_sstttt___i__ee in_kb, z>axis) and the jump cap (jump_cap. campp_aapiinn.. in in_kb, y-axis), where the hyperparameter space spans from 4 KB increments. For simplicity, the step size is set equal to the start size, and the end size is fixed from I/O profiling-2236 KB for AGX and 348 KB for Nano. Each configuration is evaluated 30 times using randomly generated activation magnitudes. This provides a reliable estimate, over 80% of the total runtime is dominated by GPU sorting via indeeendeddd radix sort [34] allowing random inputs to be used for measuring overhead.
64 KB in
We observe two clear trends: (i) configurations involving large weight matrices (e.g.. 18944x3584)
tend to incur higher overhead, making some configurations infeasible; (ii) AGX supports more configurations due its higher compute capacity compared Nano.
Final hyperparameters are selected near the boundary between feasible (green) and infeasible (red) regions, with small margin for safety. The selected settings summarized in Table
While we have not conducted a full sensitivity analysis, our empirical findings suggest that the method performs consistently well across range of hyperparameter settings. A more thorough investigation into the effects of different configurations remains an interesting direction for future work.
Table 2: Selected hyperparameters per weight matrix shape on Jetson Orin AGX and Nano
AGX
Nano
Shape (Rows x Cols)
chunk_ sz
jump_cap
jump_cap
(3584, 3584)
20
20
24
36
(8960, 1536)
16
16
20
20
(896, 4864)
8
8
8
(4096, 1024)
12
12
16
16
(3584, 18944)
8
8
8
8
(4096, 4096)
20
20
24
24
(18944, 3584)
32
32
36
36
(1536, 1536)
16
12
12
(1536, 256)
8
8
8
8
(896, 128)
8
8
8
(14336, 4096)
32
32
40
36
(4864, 896)
12
16
16
(3584, 512)
8
12
12
(896, 896)
8
8
8
(4096, 14336)
8
8
8
(1536, 8960)
8
8
8
I Full Evaluation Results on Jetson Orin AGX
Figure 14 presents full results on Jetson Orin AGX. Due to powerful SSD and compute capability, overall latency is lower compared to Jetson Orin Nano.
27

---

HaVVA-OneVision-28 LLaV.-nneiision-..BB
LongVA-7B
VILA1.5-8B
NVILA-Lite:2B
0.32
2.0
Video DC199 TempCompan
0.24
1.5
0.16
1.0
0.08
0.5
0.00
0.0
56
60
64
45.0
47.5
50.0
52.5
50
52
54
56
51
54
57
60
52.5 34.0
0.32
2.0
0.24
1.0
0.08
2.75 3.00 3.20 3.25
0.00
2.55
1.8
2.1
2.7
3.0
0.0
1.752. 0155.002..20
NextQA 1 1o No
2.0
0.24
1.5
0.16
1.0
0.08
0.5
0.00
0.0
76
50
64
60
66
67.5 5 70.0
Accuracy
Accuracy
Acourory
Accuracy
Acouracy
Figure 14: End-to-end performance Jetson Orin AGX.
J Extended Visualization of Mask Patterns and Contiguity
gate
down
Deg: LE
Aeg: 537
Dvp
Arg 3.1E
Layer
Mar
Avg. 60.27
Aig 46-79
Aog 13:35
Aug 325-30
1.7T 10T 307 X0P
30t 10 10'
2Pf 12! 19
10 30
Mach Pattwrrs (60%))
Clsck Dies
Mark Pasterns (tfi))
Coec Diess
Mask Potterns (81%)
Coals Bise
Mask Patforne (522))
Coers Blst
Avgs 1.tt
25.00
Hep 2:46
1.02
Lays Saye 13
Avg 5.
13.04
Nep de
50
Dop
Mak
Nearvaa
22 2ar Do*
1.32 Mat
Mox at
Layer 27
Aug 2.04
Berenlertigg
Hpp Nax
Aug:
Peig:
Hep d.
Aug
Figure 15: Mask patterns and corresponding contiguity distributions before and after applying method, shown across different layers (0, 13, 27) and activation types (q, o, gate, down) sparsity=0.4.
Figure 10 presented a case study on the effect of our method on mask patterns and contiguity distributions for layer 0, activation type q, under effective sparsity 0.3 and profiled sparsity 0.9.
Here, we provide an extended visualization across a broader range of settings in Figure T5] The visualization is structured as a 3 x 4 grid, where each row corresponds layer and each column
28

---

to an activation type. Each cell contains two subfigures: the left shows the binary mask patterns for three configurations- chucking-stacked
baseline with reordering, and baseline with reordering plus stacked vertically. The z-axis represents neuron index and the y-axis represents different input samples. The right subfigure presents the corresponding contiguity distribution, with the z-axis (log-scaled) denoting chunk size and the y-axis showing density. We annotate both the average and the mode (i.e., the most frequent chunk size) for each distribution.
These visualizations highlight that our method consistently promotes contiguity across layers and activation types. Qualitatively, the mask patterns become visibly less fragmented, particularly in high-sparsity regimes such as q and down. Quantitatively. both the average and mode of the contiguity distribution shift toward larger chunk sizes. While offline reordering provides marginal improvements, the majority of the contiguity gain arises from our online chunk selection policy, which adapts to input-dependent activation patterns (see Appendix
K
Effect of Visual Token Density
Tokens per
Tokens per 9x9 Ours
per Baseline
Tokens per
Labeory (e)
35
63
55
60
65
60
65
05
Accuraty
Accuracy
Accuracy
Accurary
Figure 16: End-to-end performance Jetson Orin Nano under different token counts per frame.
large number of tokens per frame limits how many frames can fit within the model's s context window. To address this, various token reduction techniques have been proposed, ranging from simple spatial pooling (55) to more advanced clustering [45]. We evaluate the impact fer-frame token count in Figure T6 using spatial pooling to control the number of tokens. As the token count decreases, we observe a modest drop in accuracy across all methods. Nonetheless, our method consistently outperforms the baseline, indicating that the I/O benefits of our approach are robust to changes in visual token density.
L Extended Comparison with Related Methods
We provide extended comparisons with three major classes of related methods: (i) LLM in a Flash (2), which also addresses flash-offloaded inference; (ii) TEAL (24), upon which our baseline implemen- tation is built; and (iii) regularization based pruning methods (c.g.. LI and group-Lasso), which promote sparsity through training- trining-time time penalties.
LLM in a Flash. VLMs' smooth activation distributions require fundamentally different approaches. Our work identifies critical inefficiencies in flash-offloaded VLMM inference unaddressed by LLM in a Flash (hereafter LLMFlash). LLMFlash targets ReLU- based LLMs with >90% sparsity; we target non- ReLU VLMs with smoother activation distributions requiring 40-60% sparsity. This difference creates counterintuitive phenomenon: in VLMs, higher sparsity can increase latency due to fragmentation- induced throughput degradation outweighing data savings. This workload difference motivates our methodological divergence. While L.L simply reduces total I/O volume using sparsification, we explicitly model I/O latency through contiguity- cost modeling and jointly consider neuron importance and I/O efficiency during sparsification. Given these workload differences, we now analyze why LLMFlash's core techniques cannot effectively address our contiguity challenges.
Neuron bundling proves insufficient without explicit contiguity optimization. LLMFlash employs col-colum bundling, grouping weights corresponding to the same activation (up- projectoooon columns with down-projectin projection rows). However, bundling alone is insufficient in our setting due to both hardware and methodological differences. Hardware-wise. LLMFllah was evaluated on MacBooks, where throughput saturates at chunk sizes <100 KB, whereas our Jetson devices require 236-348 KB for peak throughput (Figure 4a). This discrepancy likely arises from Jetson boards routing NVMe interrupts single CPU core, causing IOPS saturation, in contrast to MacBooks' multi-core
29

---

Methodologically. LLMFlash's 's up/down- projjectton bundling conflicts with our predictor-free free approach. where each matrix is sparsified based on its own activations. Even when adapted to bundle matrices sharing input activations (e.g., Q/K/V or up/gate), the largest bundled weights (~74 KB) achieve only half the optimal bandwidth on our hardware. Thus, bundling alone cannot achieve peak 1/O performance explicitly targeting contiguity as a design objective is essential.
interrupt distribution 8 42
Furthermore, when techniques such as TEAL's sparsification are combined with bundling, they introduce preprocessing and postprocessing overheads and can yield paradoxical I/O behavior. For instance, bundling Q/K/V matrices based on overlapping neurons improves locality for bundled weights but scatters remaining unbundled neurons across matrices, leading fragmented reads. The net performance depends on whether contiguity gains outweigh fragmentation penalties, making bundling's s effectiveness highly pattern-dependet.
Table 3: Comparison between our method and bundling based implementations across models and datasets. Each cell shows two average speedup ratios: (1) ours vs. baseline and (2) ours vs. basdline+bundling
Dataset Model
LLaVA-7B
LLaVA-0.5B
VILA-8B
NVILA-2B
LongVA
TempCompass
2.06/2.41
2.05/1.94
1.60/1.83
3,24/3.76
2.15/2.50
Video DC499
2.11/2.45
2.06/2.02
1.60/1.78
3.22/3.70
2.25/2.59
NexIQA
1.76/1.98
2.12/1.99
1.50/1.70
3.44/3.96
2.04/2.34
Table reports the experimental results comparing our method with bundling-based implementations. Our method achieves consistent speedups of 1.5-3.4x over the baseline and 1.7-4.0x over bundling- based implementations across all models and datasets. Bundling degrades performance in most cases except LLaVA-0.5B, confirming that its benefits are unpredictable and pattern-dependent, whereas our contiguity-aware approach consistently improves efficiency.
Sliding window caching vs. offfine reordering trades memory for adaptability. LLMFlash's 's sliding- window caching maintains recently activated neurons' weights in memory, trading memory for reduced latency-an an infeasible approach in our memory-constrained edge deployments. Both their caching and PowerInfer [38)ss hot neuron caching leverage additional memory to reduce flash accesses by exploiting statistical access patterns. Instead, we employ offline reordering for comparable benefits: both methods make frequently accessed weights cheaper load, but caching is auntiee-adaptive while consuming memory, whereas reordering imposes zero memory overhead while being less adaptive. In practice, we keep only essential weights-vision encoder, LM head, and KV cache-in device memory, representing the truly "hottest" components. Our chunk selection algorithm naturally accommodates any caching strategy by assigning zero importance to cached neurons, making flexible and complementary to memory optimization approaches.
TEAL. Our method builds upon TEAL's fine- mne-grained sparsity allocation across matrices rather than applying uniform sparsity. However, our focus differs fundamentally: TEAL determines how much to sparsify each layer under uniform access cost assumptions, whereas determine which neurons to load at runtime by jointly considering activation importance and I/O efficiency in flash-based systems. This introduces latency-aware chunk selection that restructures selected neurons into contiguous memory layouts, aligning model-level sparsification with system-level latency behavior.
Regularization-basee pruning. L1 regularization typically operates at the individual-went -wiight level and rarely climinates entire rows or columns of weights. As a result, does not reduce the number of rows that must be loaded from flash, limiting its effecct activation sparsity and overall latency.
To meaningfully impact latency, sparsity must be structured (e.g., at the row or column level). This can be achieved by replacing L1 with group-Lasso regularization that applies L2-norm penalties to entire rows or columns. Column- wise regularization (e.g., applied gate or up-projection matrices) encourages certain output activation channels to become zero, effectively deactivating the corresponding rows in the down-projection matrix and increasing activation sparsity. Row- wise regularization can also promote sparsity when the neuron-importance metric incorporates both activation magnitude and weight norm, lowering the importance of neurons associated with low-norm rows.
30

---

These regularization-basee approaches are inherently input-agnostic: they prune the same weight regardless of the imput context. This limits their achievable sparsity before severe accuracy degradation occurs- as shown in TEAL 30 where even 20% pruning causes noticeable performance drops.
M
On Tradeoffs Between Accuracy and Latency
Our method is designed not to preserve accuracy at all costs, but to enable more favorable tradeoff between accuracy and latency. In latency-sensitive deployments, it is often preferable accept a modest accuracy degradation in exchange for significantly faster responses. The objective is not merely to match the performance of dense inference, but shift the accuracy-latency Pareto coontier-achevviie lower latency for comparable accuracy, or improved accuracy within a fixed latency budget. This tradeoff is particularly valuable practical vision-language applications where the input is video and the output is natural language response. In many such scenarios, the user can quickly validate or refine the system's s output, making responsiveness more critical than marginal gains in precision. Examples include:
Object or person retrieval. When the user asks, e.g., "Where is the person in a red shirt?" or "Is the car still visible in this scene?", delivering fast candidate answers enables immediate visual verification and iteration.
Temporal localization. In tasks like "When does the object fall?" or "At what time does the person enter the room?", baars-ggriined temporal answers that arrive quickly often more useful than delayed fine-grained ones.
In these streaming input scenarios, delayed responses can themselves degrade performance. This phenomenon- often referred to as streaming accuracy-has been observed in streaming perception literature, where the timeliness of model outputs directly influences their correctness I7
In such use cases, VLMs are part of interactive systems where user experience benefits more from fast responses than from exact answers. Our method supports this objective by enabling structured sparsification that reduces latency while maintaining actionable utility in response quality.
N Generalization to Other Use Cases
Extension to multi-token LLM inference. Although our method is evaluated the context of vision- language models, the core Slea--hardware-ae domain. In particular, it is well-suited for multi-token L.LM inference scenarios such as speculative decoding, parallel sampling for reasoning, and batched inference in interactive applications. In such settings, activations from multiple tokens are aggregated, leading to smoother neuron- -iuuoon-mppotance distributions similar those observed in VLMs.
structured cossifiiaatineeetennti beyond this
These multi-token inference workloads also underpin latency-critical user-facing systems such as chat assistants, copilots, and dialogue agents, which must maintain responsiveness under real-time constraints while often serving multiple concurrent requests. Minimizing end-to-end latency in these deployments is therefore essential, and extending framework to such applications may offer broader benefits in practical LLM serving environments.
Extension to plain LLM / VIT inference. Our method can also be directly applied to plain LLMs and ViT-based models. The system relies on two key conditions: (i) the model exhibits smooth activation magnitude distributions (e.g., due to non-ReLU activations or multi-token inputs), and (ii) the handware-model pair operates below 1/O saturation when loading a single weight row. Recent LLMs increasingly employ smooth activation functions such as SwiGLU or GeLU, yielding moderate sparsity levels. While their activations are typically less smooth than those of multi-token scenarios, method remains applicable, albeit with slightly smaller gains.
For ViT models, the transformer backbone remains largely compatible with our framework. As long as activation sparsity exists to a measurable degree, our approach can be applied without modification. Although ViTs are generally smaller (hundreds of millions of parameters), they still
'The sparsity mask generated from aggregated activations is shared across tokens, ensuring uniform inference latency across them (e.g., all samples in a batch finish simultaneously).
31

---

face I/O bottlenecks on assurce-llmitedd devices. In such cases, our chunking method becomes particularly valuable, smaller weight channels make access fragmentation proportionally more severe.
To assess applicability plain LLMs,, we conducted preliminary experiment LLaMA3-8B and Qwen2-7B using the GSM8K dataset. We used the sum of selected neuron importance as a proxy for accuracy rather than full-dataset evaluation. We measured the importance-latency tradeoff in the first, middle, and last layers, observing average speedups of 1.22x and x.09 for LLaMA3-8B and Qwen2-78 respectively. These initial results suggest that our method generalizes LLMs, though further work is needed to validate accuracy -ecurcyylatennyy tradeoffs at scale.
32


Reviews:
[{"review_id": "hy9SvSRecl", "reviewer": "Reviewer_3ayJ", "summary": "This paper proposes ​​an activation sparsification method named Neuron Chunking​​, which is designed to accelerate vision-language models on edge devices. The method improves flash-based weight offloading with a contiguity-based latency model to predict the cost of access patterns and a multi-scale chunk selection algorithm to explicitly consider actual I/O latency by selecting contiguous chunks. The method is verified based on various VLMs on edge devices, including Jetson Orin Nano and Jetson AGX Orin.", "questions": "Please refer to my comments above.", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n- The paper is well-written with a clearly stated motivation and a reasonable and easy-to-understand method. \n\n- Extensive studies are conducted to show the access patterns and model performance. The proposed method achieves decent improvements over the baseline activation sparsification method.\n\nWeaknesses:\n\n- The method proposed in the paper is largely based on existing work on acceleration for flash-offloaded LLM inference. The idea of latency analysis is closely related to the observation in [2]. The paper can be viewed as an extension of [2] for VLM acceleration.\n\n- The paper focuses on studying the problem of accelerating video understanding inference, where a frame appending stage is considered between the prefill and decoding stage in inference, which I believe may not be the most common case for VLM inference. Will the observation (access pattern, speed-up results) be the same if we consider a simple image understanding task? The method can be more general if the solution is also valid for general vision-language understanding scenarios.", "was_revised": true, "final_justification": "Thanks for the feedback. My concerns are mostly addressed. After reading the rebuttal as well as other reviews, I would like to keep my initial rating and lean towards accepting this paper.", "rebuttal": "We thank the reviewer for their thoughtful feedback and detailed evaluation of our work.\n\n\n### Overview of the Review & Our Responses:\n\n\n\n* **S1 (Clear motivation and method):** We appreciate the reviewer’s positive feedback on the clarity and presentation of our paper.\n* **S2 (Extensive evaluation):** We are glad the reviewer recognized our extensive experimental study, which demonstrates substantial speedups on real hardware and consistent improvements across diverse models.\n* **W1 (Distinction from LLM in a Flash):** We position our work by analyzing how VLMs create fundamentally different I/O challenges than prior ReLU-based LLMs and demonstrate why LLM in a Flash's techniques cannot address our problem.\n* **W2 (VLM scenario generality):** We highlight the growing importance of online video understanding for practical applications with supporting references. We also explain that our system’s underlying conditions hold for single-image tasks, though with smaller end-to-end impact.\n\nWe now provide detailed responses to each concern.\n\n*Note: The proposed additions will be included in the main paper, with supplementary material used if necessary due to space constraints.*\n\n\n### W1: “The paper can be viewed as an extension of LLM in a Flash[1] for VLM acceleration.”\n\nWhile both our work and LLM in a Flash(hereafter LLMFlash) address I/O bottlenecks in flash-offloaded inference, our key contributions and technical focus diverge significantly:\n\n**VLMs' smooth activation distributions require fundamentally different approaches** : Our work identifies critical inefficiencies in flash-offloaded VLM inference unaddressed by LLM in a Flash (hereafter LLMFlash). LLMFlash targets ReLU-based LLMs with >90% sparsity; we target non-ReLU VLMs with smoother activation distributions requiring 40-60% sparsity. This difference creates a counterintuitive phenomenon: in VLMs, higher sparsity can increase latency due to fragmentation-induced throughput degradation outweighing data savings.\n\nThis workload difference motivates our methodological divergence. While LLMFlash simply reduces total I/O volume using sparsification, we explicitly model I/O latency through contiguity-aware cost modeling and propose sparsification jointly considering neuron importance and I/O efficiency.\n\nGiven these workload differences, we now analyze why LLMFlash's core techniques cannot effectively address our contiguity challenges.\n\n**Neuron bundling proves insufficient without explicit contiguity optimization:** LLMFlash uses row-column bundling, grouping weights corresponding to the same activation (up-projection columns with down-projection rows). However, bundling proves insufficient in our setting due to hardware and methodological differences. Hardware-wise, LLMFlash was evaluated on MacBooks where throughput plateaus at &lt;100KB, while our Jetson devices require 236-348KB for peak throughput (Figure 4a)—likely due to Jetson boards routing NVMe interrupts to single CPU cores causing IOPS saturation versus MacBooks' multi-core interrupt distribution [6, 7]. Methodologically, their up/down projection bundling conflicts with our predictor-free approach, where different matrices are sparsified based on respective input activations. Even when the method is adapted by bundling matrices sharing input activations (Q/K/V or up/gate), the largest bundled weights (~74KB) achieve only half the optimal bandwidth on our hardware. This demonstrates that bundling alone is insufficient—peak I/O performance requires explicitly targeting contiguity as a design objective.\n\nTEAL's sparsification [2] introduces preprocessing/postprocessing overhead and creates paradoxical I/O effects when bundling is adopted. For example, when Q, K, V matrices are bundled based on overlapping neurons, bundled weights load efficiently but remaining unbundled neurons become scattered across matrices, forcing fragmented reads. The net performance depends on whether contiguity gains from bundling outweigh fragmentation penalties from unbundled neurons, making bundling unpredictably beneficial or detrimental depending on sparsity patterns.\n\nOur evaluation compares baseline sparsification with neuron bundling against our approach across five VLM models and three datasets:\n\n|Dataset/Model|LLaVA-7B|LLaVA-0.5B|VILA-8B|NVILA-2B|LongVA|\n|-|-|-|-|-|-|\n|**TempCompass**|2.06/2.41|2.05/1.94|1.60/1.83|3.24/3.76|2.15/2.50|\n|**Video DC499** |2.11/2.45|2.06/2.02|1.60/1.78|3.22/3.70|2.25/2.59|\n|**NextQA**|1.76/1.98|2.12/1.99|1.50/1.70|3.44/3.96|2.04/2.34|\n\n*Each cell shows two average speedup ratios: (1) our method vs. baseline, (2) our method vs. baseline with bundling*\n\nOur method achieves consistent average speedups of 1.5-3.4× over baseline and 1.7-4.0× over bundling implementations across all models and datasets. Bundling degrades performance in most cases except LLaVA-0.5B, confirming that bundling effects are unpredictable and pattern-dependent, while our contiguity-aware approach delivers reliable improvements.\n\n**Sliding window caching vs. offline reordering trades memory for adaptability:** LLMFlash's sliding window caching—maintaining recently activated neurons' weights in memory—represents a memory-for-performance tradeoff potentially infeasible in our memory-constrained edge deployment. Both their caching and PowerInfer's hot neuron caching [8] leverage additional memory to reduce flash accesses by exploiting statistical access patterns. Instead, we employ offline reordering for similar benefits: both techniques make frequently accessed weights cheaper to load, but caching is runtime-adaptive while consuming memory, whereas reordering imposes zero memory overhead while being less adaptive. We keep only essential weights in memory—vision encoder, LM head, and KV cache—accessed every inference iteration and representing truly \"hottest\" components. Our chunk selection naturally accommodates any caching strategy by assigning zero importance to cached neurons, making it flexible and complementary to memory optimization approaches.\n\nTo address these comparisons, we will restructure the paper: split the current Section 2 \"Background and Related Work,\" with Section 2 focusing solely on \"Background\" and adding a new dedicated \"Related Work\" section before \"Discussion and Future Work.\" This new section will provide detailed comparisons with LLMFlash as outlined above, mapping the broader landscape of related approaches to clarify our method's positioning and contributions.\n\n\n### W2: “The paper focuses on studying the problem of accelerating video understanding inference ... Will the observation (access pattern, speed-up results) be the same if we consider a simple image understanding task?”\n\nWhile early VLMs primarily focused on single-image understanding, the field is rapidly shifting toward online video understanding[3, 4, 5], enabled by models trained on large-scale video datasets. This transition is especially critical for enabling practical, real-world systems such as AR assistants with smart glasses and autonomous agents like robots. In these scenarios, the model continuously processes incoming video frames, with user queries arriving intermittently. As a result, the frame appending phase dominates inference time, while prefill (initial prompt setup) and decoding (answer generation) occur less frequently. We formally described this inference pipeline in Appendix D.1.\n\nRegarding single-image understanding tasks, the core conditions that our system targets—namely, (i) models with smooth activation magnitude distributions (e.g., due to non-ReLU activation functions and multi-token inputs), and (ii) hardware paired with models small enough that reading a single weight row does not saturate I/O throughput—remain the same. Therefore, the speed-up trends we observe should hold similarly in that setting. However, in single-image tasks, the total latency is often dominated by decoding rather than image processing. Thus, while our chunking method may still reduce image processing latency, its relative impact on end-to-end latency may be less significant.\n\nWe will revise the Introduction and Related Works sections to explicitly define our target scenario and emphasize its importance for practical real-world applications. We will also add a brief discussion in the Discussion section clarifying how our system’s benefits translate to single-image tasks.\n\n[1] \"Llm in a flash: Efficient large language model inference with limited memory.\" ACL’24.\n\n[2] \"Training-free activation sparsity in large language models.\"ICLR’25.\n\n[3] \"Videollm-online: Online video large language model for streaming video.\" CVPR’24.\n\n[4] \"OVO-Bench: How Far is Your Video-LLMs from Real-World Online Video Understanding?.\" CVPR’25.\n\n[5] ChatGPT voice mode. [https://help.openai.com/en/articles/8400625-voice-mode-faq#h_fa3ca78ac7](https://help.openai.com/en/articles/8400625-voice-mode-faq#h_fa3ca78ac7)\n\n[6] \"Optimizing storage performance with calibrated interrupts.\" TOS’22 \\\n\n[7] NVIDIA Developer Forum. “Gen 3 PCIe NVMe SSD with x4 Lanes Gets Higher IOPS on Nano Compared to the Xavier NX.” 6 Sept. 2022\n\n[8] \"Powerinfer: Fast large language model serving with a consumer-grade gpu.\" SOSP’24"}, {"review_id": "aHZVauNW9d", "reviewer": "Reviewer_AaFb", "summary": "Large VLMs exceed edge device memory, so weights are offloaded to external storage and loaded as needed, causing latency. The authors propose selection of contiguous, moderately important channels, rather than scattered, highly important ones, to improve memory access efficiency and reduces this latency.\nMain contribution include I/O efficiency-aware activation sparsification approach and efficient chunk-based neuron selection algorithm for improving the efficiency in VLM inference.", "questions": "- What could be the effect of L1 regularization, which is known to encourage weight sparsity, on the accuracy-latency tradeoff? Can it be adapted to support chunk-level sparsity instead of neuron-level sparsity?\n- Does this approach affect the efficiency of batch processing? For instance, there might be cases where different samples in a batch have different inference times after applying this method.\n- It will be interesting to see how this method can be extended to plain LLMs and plain ViT-based models.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "- The paper presents a technically sound approach to making VLMs more efficient using hardware-aware sparsification. While the experimental evaluation is generally good, the paper would benefit from additional analysis on cases where the latency-accuracy tradeoff drops significantly. \n- The manuscript is clearly written and logically organized.\n- The paper addresses an important problem of translating model acceleration techniques to real-world latency gains on hardware. \n- Although the proposed method is similar to structured pruning approaches, the incorporating I/O latency and approximating it using a model is novel.", "was_revised": true, "final_justification": "The authors have provided well-reasoned responses to the concerns raised in my initial review. \nThe paper addresses a practically important problem with hardware-aware approach. While related to structured pruning, the explicit modeling of I/O latency and chunk-based selection adds novelty. Given the quality of the work, its relevance to real-world deployment, and the comments of other reviewers, I am maintaining my rating.", "rebuttal": "We thank the reviewer for their thoughtful feedback and detailed evaluation of our work.\n\n### Overview of the Review & Our Responses:\n\n* **S&W1 (Latency–accuracy tradeoff behavior analysis):** We analyze the latency–accuracy curves in detail and explain the key factors that influence their behavior.\n* **S&W2 - S&W4 (Problem importance, Method novelty, Writing quality):** We appreciate the reviewer’s recognition of the technical soundness of our work and, in particular, the importance of bridging the gap between algorithmic optimizations and real-world latency—a direction we believe is critical for practical and deployable VLM systems.\n* **Q1 (Effect of L1 regularization):** We explain why standard L1 regularization does not directly benefit activation sparsity and propose structured regularization as a potential complementary approach.\n* **Q2 (Effect on batch processing):** We clarify our system’s behavior under batch processing, discuss its accuracy implications, and explain why our method becomes increasingly important.\n* **Q3 (Extension to LLMs and ViTs):** We confirm that our method applies to non-VLM models with preliminary quantitative results.\n\nWe now provide detailed responses to each concern.\n\n*Note: The proposed additions will be included in the main paper, with supplementary material used if necessary due to space constraints.*\n\n\n### W1: “the paper would benefit from additional analysis on cases where the latency-accuracy tradeoff drops significantly.”\n\nSince the reviewer’s comment on “cases where the latency–accuracy tradeoff drops significantly” may refer to different situations, we provide a comprehensive response covering multiple possible interpretations: (i) baseline latency increase at medium sparsity, (ii) accuracy increase in the low-sparsity regime despite higher sparsity, and (iii) potential limitations of our method under extreme conditions, such as scenarios involving severe sparsity or large memory budgets.\n\n(i) As noted in Figure 4b, naive magnitude-based sparsification can cause non-monotonic latency behavior, especially at medium sparsity levels, due to fragmented memory accesses. This results in counterintuitive regions in the baseline latency–accuracy curves (Figure 6) where latency increases even as more weights are sparsified. The effect is particularly pronounced for smaller models (LLaVA-0.5B, NVILA-2B), where each weight row is smaller and access fragmentation becomes more severe. In contrast, our method consistently yields monotonic latency behavior across models.\n\n(ii) A noticeable irregularity in the tradeoff curve appears in the low-sparsity regime, where we occasionally observe a slight increase in accuracy despite higher sparsity. We attribute this to a regularization effect from sparsification suppressing noisy activations, as stated in Line 281.\n\n(iii) At very high sparsity (e.g., >90% of ReLU-based LLMs), the opportunity for chunking diminishes since so few neurons remain active that there are insufficient contiguous regions to form meaningful chunks. However, such high sparsity often comes at the cost of significant accuracy degradation, which can limit the model's practical applicability.\n\nAlternatively, if the device has sufficient memory to cache a large portion of the model weights, the overall flash I/O volume becomes negligible, reducing the benefit of our method. Our work explicitly targets memory-constrained environments, where such caching is infeasible and flash I/O efficiency is critical for low latency.\n\nWe will expand the Discussion section with a concise discussion clarifying the applicability boundaries of our method under extremely high sparsity or in memory-rich environments.\n\n\n### Q1: “What could be the effect of L1 regularization, which is known to encourage weight sparsity, on the accuracy-latency tradeoff? Can it be adapted to support chunk-level sparsity instead of neuron-level sparsity?”\n\nWe appreciate this insightful question, which prompted us to examine the impact of L1 regularization on our system more closely.\n\nL1 regularization operates at the individual weight level and rarely eliminates entire weight rows. Thus, it does not reduce the number of rows to be loaded, limiting its effect on activation sparsity and latency.\n\nTo meaningfully impact latency, sparsity must be structured (e.g., at the weight row or column level). This can be achieved by replacing L1 with group Lasso regularization, which applies L2 norm penalties over entire rows or columns of weight matrices.\n\n\n* Column-wise regularization (e.g., on gate or up-projection matrix) would encourage certain output activation values to become zero. This, in turn, deactivates the corresponding rows in the down-projection matrix, increasing sparsity.\n* Row-wise regularization could also promote sparsity if the neuron importance metric incorporates weight norm in addition to the activation magnitude. This would lower the importance of the neurons that correspond to weight rows with low norm, increasing sparsity.\n\nThe higher sparsity induced by these methods can potentially reduce the opportunity for chunking, thereby narrowing the performance gap between our method and the baseline.\n\nHowever, since these methods are input-agnostic, meaning they prune the same structure regardless of context. This limits their achievable sparsity before accuracy drops sharply—as shown in [2], where even 20% pruning causes noticeable degradation.\n\nWe will add a discussion in the Related Works section introducing L1 regularization as a potential method to increase sparsity and explaining how structured variants such as group lasso could interact with our method, while also noting their practical limitations.\n\n\n### Q2: “Does this approach affect the efficiency of batch processing? For instance, there might be cases where different samples in a batch have different inference times after applying this method.”\n\nWe appreciate this question, as batch processing is an important consideration for real-world deployment.\n\nOur method does not negatively impact batch processing efficiency. In typical batched inference, all samples in a batch share the same model weights during computation. Thus, we apply a shared sparsity mask across the batch, loading the same subset of weights for every sample. This ensures uniform inference time across the batch.\n\nInterestingly, this shared-mask setting further smooths the neuron-importance distribution by aggregating importance across samples. This makes contiguity-aware sparsification even more critical, as adjacent neurons are more likely to have moderately high values, creating greater opportunities for chunking. However, sharing a single mask across a large batch may sharply degrade accuracy, requiring a lower sparsity level to maintain accuracy.\n\nThat said, our target deployment scenario is memory-constrained edge devices, where single-sample inference is more common than large-batch processing. Nonetheless, our method remains compatible with both regimes and can provide benefits in batched settings as well.\n\nWe will explain in the Discussion section how batching changes the sparsification behavior and the benefit of our chunking method under such settings.\n\n\n### Q3: “It will be interesting to see how this method can be extended to plain LLMs and plain ViT-based models.”\n\nOur method can be directly applied to both LLMs and ViT models. Our system relies on two conditions: (i) models with smooth activation magnitude distributions (e.g., due to non-ReLU activation functions and multi-token inputs), and (ii) hardware paired with models small enough that reading a single weight row does not saturate I/O throughput.\n\n\n\n* Recent LLMs increasingly adopt smooth activation functions such as SwiGLU or GeLU, resulting in medium-level sparsity[3]. While the activation distribution may be less smooth than in multi-token VLMs, our method remains applicable, albeit with slightly reduced benefit.\n* Moreover, LLMs are often used with multi-token inputs in settings such as the verification stage of speculative decoding or tree-based parallel generation. These scenarios further smooth out the activation distribution, making them closely aligned with the VLM use cases we evaluated.\n* For ViT models, the transformer architecture is largely similar to that of LLMs and VLMs. As long as some degree of activation sparsity is present—which needs verification—our method can be applied without modification. That said, ViTs are typically smaller in size (on the order of hundreds of millions of parameters [4,5]), which reduces the overall demand for sparsification. However, on extremely resource-constrained devices where even ViTs cannot be fully cached in memory, the smaller per-row weight size makes our chunking method even more valuable, as the access fragmentation becomes more severe.\n\nWe conducted a preliminary experiment to assess the applicability to LLMs (Llama3-8B, Qwen2-7B with GSM8k dataset). Due to time constraints, we used the sum of selected neuron importance as a proxy for accuracy instead of running full-dataset accuracy evaluations. We measured the importance–latency tradeoff in the first, middle, and last layers, observing speedup of 1.22x/2.09x on average for Llama3-8B, Qwen2-7B.\n\nWe will update Section 5 to explicitly demonstrate that our approach is applicable beyond VLMs by including these quantitative results, and we will add further detailed experiments in the revision to strengthen this analysis.\n\n[1] \"Learning structured sparsity in deep neural networks.\" NIPS’16\n\n[2] \"Llm-pruner: On the structural pruning of large language models.\" NeurIPS’23\n\n[3] \"Training-free activation sparsity in large language models.\" ICLR’25\n\n[4] \"Learning transferable visual models from natural language supervision.\" International conference on machine learning. ICML’21.\n\n[5] \"An image is worth 16x16 words: Transformers for image recognition at scale.\" ICLR’21"}, {"review_id": "CUfUYusg5Y", "reviewer": "Reviewer_4gFS", "summary": "This paper considers improving the latency of VLM inference on edge devices. Authors notice that while sparsely loading a fraction of neurons based on their importance can theoretically reduce the I/O overhead, this strategy ignores the flash storage behavior and in practice doesn’t lead to much latency gain. The authors propose to jointly consider the importance of neurons and the loading latency and select neurons by co-optimizing the two criteria. Experiments show that the authors’ method achieves lower latency compared to natively loading neurons solely based on their importance.", "questions": "- Please address the confusion as listed in the first point of the weaknesses.\n- More comparisons with recent methods, including LLM in a Flash, would be helpful for better validating the effectiveness of the method.", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n\n- The proposed co-optimization is intuitive and easy to understand, and the experimental results are promising for its effectiveness.\n- The observation that relative ordering of latency is sufficient and crucial to the latency estimation is novel. In Figure 5, chunking-based latency estimation is more accurate than the conventional top-$k$ sparsification.\n- Experiments demonstrate the effectiveness of the method on improving the latency compared to the native top-$k$ importance selection. The method achieves a large reduction of I/O latency in the breakdown.\n\n**Weaknesses:**\n\n- While the method is intuitively easy to understand, some parts of the description in the paper are confusing to me:\n    - In Figure 1, how is the total latency $1.1$ calculated?\n    - The chunking and reordering are not described with sufficient detail. Especially, the reordering is only described with several lines of text, but it is very important to the performance, as shown in the ablation study. More illustrations would be helpful.\n    - In line 175, the authors mentioned that he selected rows are grouped into contiguous chunks. But if selected rows are like Row1, Row3, and Row5, how are they grouped in this case?\n    - How is chunking designed to avoid overlapping from different scales? For example, in Figure 1 on the scale of two neurons, why is the group of 7&4 not selected?\n- Lack of comparison to the recent method. While the authors mentioned LLM in a Flash and claimed it is suboptimal, I don’t find any comparison with it in the main body.", "was_revised": true, "final_justification": "Thank you for the detailed response. It has addressed all my questions and concerns. It would be great to include these clarifications and changes to the revised manuscript. I have adjusted my score accordingly.", "rebuttal": "We thank the reviewer for their thoughtful feedback and detailed evaluation of our work.\n### Overview of the Review & Our Responses:\n* **S1-S3**: We acknowledge the reviewer's recognition of our co-optimization approach's intuitiveness, novel latency estimation insights, and experimental effectiveness.\n* **W1 (Technical clarity)**: We will revise technical descriptions to improve algorithmic clarity and reproducibility.\n* **W2 (Recent methods comparison)**: We position our work by analyzing how VLMs create fundamentally different I/O challenges than prior ReLU-based LLM, demonstrate why LLM in a Flash's techniques cannot address our problem, and clarify TEAL's orthogonal focus on sparsity profiling.\n\nWe now provide detailed responses to each concern.\n\n*Note: The proposed additions will be included in the main paper, with supplementary material used if necessary due to space constraints.*\n### W1: \"While the method is intuitively easy to understand, some parts of the description in the paper are confusing...\"\n\nWe appreciate these clarification requests and address each technical detail below:\n\n**\"In Figure 1, how is the total latency calculated?\"**\n\nThe total latency is the sum of individual chunk latencies, where each chunk's latency = data size ÷ throughput, with throughput values pre-profiled for each chunk size. For the example shown:\n\n\n\n* Chunk size 2 has throughput 4 → latency = 2/4 = 0.5\n* Chunk size 3 has throughput 5 → latency = 3/5 = 0.6\n* Total latency = 0.5 + 0.6 = 1.1\n\nWe will add these formulas to Figure 1 to make the logic easy to follow.\n\n**\"The chunking and reordering are not described with sufficient detail...\"**\n\n**Chunking**: The complete algorithm with pseudocode is provided in Appendix C (Algorithm 1).\n\nWe will make this reference more prominent and provide a more detailed description of algorithmic steps (detailing how we construct multi-scale chunk candidates, score them, and greedily select non-overlapping chunks) in Section 4.2.2 Algorithm Overview.\n\n**Reordering**: The full procedure involves three steps:\n\n\n\n1. Count how often each neuron activates using calibration dataset\n2. Sort neurons in decreasing frequency and permute weight matrix rows\n3. At runtime, apply the same permutation to activation\n\nThe runtime permutation operation incurs negligible overhead. For LLaVA-7B with shape (1, 196, 18944), we profiled this on Jetson Nano over 100 trials: mean overhead was 1.5ms with 95% confidence bound of 1.8ms, representing &lt;0.02% of total latency.\n\nWe will add the above detailed explanations to Section 4.3 Additional Optimization: Hot-cold Reordering.\n\n**\"If selected rows are like Row1, Row3, and Row5, how are they grouped?\"**\n\nOnly adjacent rows form contiguous chunks. In your example, rows 1, 3, and 5 would not be grouped; each would form a separate chunk. We group consecutive selected rows into larger chunks for basic I/O optimization—standard practice for improving memory access efficiency.\n\nWe will revise Line 175 to: \"Selected rows are grouped into sets of contiguous chunks, where only adjacent rows are merged into the same chunk.\"\n\n**\"How is chunking designed to avoid overlapping from different scales?\"**\n\nWe generate multi-scale chunk candidates, sort by importance-per-latency ratio, then greedily select the highest-ranked non-overlapping chunks (Lines 231-232). In Figure 1, chunk 7&4 is excluded because the higher-ranked chunk 7&4&5 has already been selected.\n\nWe will revise Lines 231-232 to: \"Candidates are sorted by score, and the algorithm iteratively selects the highest-ranked chunks while excluding overlapping candidates until the budget is met.\"\n\n\n### W2: \"Lack of comparison to the recent method...\"\n\nWe provide comprehensive comparisons with the two most relevant recent methods: LLM in a Flash [1], which also addresses flash-offloaded inference, and TEAL [2], which our baseline directly builds upon.\n\n**LLM in a Flash**\n\n**VLMs' smooth activation distributions require fundamentally different approaches**: Our work identifies critical inefficiencies in flash-offloaded VLM inference unaddressed by LLM in a Flash (hereafter LLMFlash). LLMFlash targets ReLU-based LLMs with >90% sparsity; we target non-ReLU VLMs with smoother activation distributions requiring 40-60% sparsity. This difference creates a counterintuitive phenomenon: in VLMs, higher sparsity can increase latency due to fragmentation-induced throughput degradation outweighing data savings.\n\nThis workload difference motivates our methodological divergence. While LLMFlash simply reduces total I/O volume using sparsification, we explicitly model I/O latency through contiguity-aware cost modeling and propose sparsification jointly considering neuron importance and I/O efficiency.\n\nGiven these workload differences, we now analyze why LLMFlash's core techniques cannot effectively address our contiguity challenges.\n\n**Neuron bundling proves insufficient without explicit contiguity optimization:** LLMFlash uses row-column bundling, grouping weights corresponding to the same activation (up-projection columns with down-projection rows). However, bundling proves insufficient in our setting due to hardware and methodological differences. Hardware-wise, LLMFlash was evaluated on MacBooks where throughput plateaus at &lt;100KB, while our Jetson devices require 236-348KB for peak throughput (Figure 4a)—likely due to Jetson boards routing NVMe interrupts to single CPU cores causing IOPS saturation versus MacBooks' multi-core interrupt distribution [3, 4]. Methodologically, their up/down projection bundling conflicts with our predictor-free approach, where different matrices are sparsified based on respective input activations. Even when the method is adapted by bundling matrices sharing input activations (Q/K/V or up/gate), the largest bundled weights (~74KB) achieve only half the optimal bandwidth on our hardware. This demonstrates that bundling alone is insufficient—peak I/O performance requires explicitly targeting contiguity as a design objective.\n\nTEAL's sparsification introduces preprocessing/postprocessing overhead and creates paradoxical I/O effects when bundling is adopted. For example, when Q, K, V matrices are bundled based on overlapping neurons, bundled weights load efficiently but remaining unbundled neurons become scattered across matrices, forcing fragmented reads. The net performance depends on whether contiguity gains from bundling outweigh fragmentation penalties from unbundled neurons, making bundling unpredictably beneficial or detrimental depending on sparsity patterns.\n\nOur evaluation compares baseline sparsification with neuron bundling against our approach across five VLM models and three datasets:\n\n|Dataset/Model|LLaVA-7B|LLaVA-0.5B|VILA-8B|NVILA-2B|LongVA|\n|-|-|-|-|-|-|\n|**TempCompass**|2.06/2.41|2.05/1.94|1.60/1.83|3.24/3.76|2.15/2.50|\n|**Video DC499** |2.11/2.45|2.06/2.02|1.60/1.78|3.22/3.70|2.25/2.59|\n|**NextQA**|1.76/1.98|2.12/1.99|1.50/1.70|3.44/3.96|2.04/2.34|\n\n*Each cell shows two average speedup ratios: (1) our method vs. baseline, (2) our method vs. baseline with bundling*\n\nOur method achieves consistent average speedups of 1.5-3.4× over baseline and 1.7-4.0× over bundling implementations across all models and datasets. Bundling degrades performance in most cases except LLaVA-0.5B, confirming that bundling effects are unpredictable and pattern-dependent, while our contiguity-aware approach delivers reliable improvements.\n\n**Sliding window caching vs. offline reordering trades memory for adaptability:** LLMFlash's sliding window caching—maintaining recently activated neurons' weights in memory—represents a memory-for-performance tradeoff potentially infeasible in our memory-constrained edge deployment. Both their caching and PowerInfer's hot neuron caching [5] leverage additional memory to reduce flash accesses by exploiting statistical access patterns. Instead, we employ offline reordering for similar benefits: both techniques make frequently accessed weights cheaper to load, but caching is runtime-adaptive while consuming memory, whereas reordering imposes zero memory overhead while being less adaptive. We keep only essential weights in memory—vision encoder, LM head, and KV cache—accessed every inference iteration and representing truly \"hottest\" components. Our chunk selection naturally accommodates any caching strategy by assigning zero importance to cached neurons, making it flexible and complementary to memory optimization approaches.\n\n**TEAL**\n\n**How Much to Sparsify vs. Which Neurons to Select:** Our method builds upon TEAL's fine-grained sparsity allocation across weight matrices rather than uniform sparsity. However, our focus differs fundamentally: TEAL determines **how much to sparsify** each weight assuming uniform access cost, while we determine **which neurons to select **at runtime by jointly considering neuron importance and I/O efficiency on flash-based systems, introducing latency-aware chunk selection that restructures selected neurons into more contiguous memory layouts.\n\nTo address these comparisons, we will restructure the paper: split the current Section 2 \"Background and Related Work,\" with Section 2 focusing solely on \"Background\" and adding a new dedicated \"Related Work\" section before \"Discussion and Future Work.\" This new section will provide detailed comparisons with LLMFlash and TEAL as outlined above, mapping the broader landscape of related approaches to clarify our method's positioning and contributions.\n\n[1] \"Llm in a flash: Efficient large language model inference with limited memory.\" ACL’24 \\\n[2]\"Training-free activation sparsity in large language models.\" ICLR’25 \\\n[3] \"Optimizing storage performance with calibrated interrupts.\" TOS’22 \\\n[4] NVIDIA Developer Forum. “Gen 3 PCIe NVMe SSD with x4 Lanes Gets Higher IOPS on Nano Compared to the Xavier NX.” 6 Sept. 2022 \\\n[5] \"Powerinfer: Fast large language model serving with a consumer-grade gpu.\" SOSP’24"}, {"review_id": "7d9rOkBQen", "reviewer": "Reviewer_EnFY", "summary": "This paper focuses on enhancing the efficiency of inference in vision-language models (VLMs) by addressing flash-based weight offloading and activation sparsification. It critically analyzes the current limitations of sparsification in VLMs and proposes a targeted approach that combines the importance of activation with the optimization of flash I/O efficiency.", "questions": "see weaknesses", "limitations": "yes", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n* Innovative Approach: The introduction of a system-level approach that couples model sparsification with hardware I/O characteristics is a novel contribution. The paper emphasizes the importance of considering the underlying flash storage behavior, which is often overlooked in conventional sparsification methods.\n\n* Comprehensive Evaluation: The paper presents detailed experiments on two different edge platforms, utilizing multiple models and benchmarks, and demonstrates consistent improvements across various setups.\n\n**Weaknesses**\n\n* In section 3.1, the comparison of VLM smoother distributions of activation magnitudes is based on a single model assumption （llava-onevision-7B）, and more models are needed for analysis and validation of the activation issue in VLMs.\n* Limited Discussion on the profile cost and overheads in the latency Model. The latency model does not account for fixed overheads such as metadata lookup or command submission during flash read initiation. While this is mentioned in the paper, a more detailed discussion of how these factors could be incorporated in future work would help in improving the precision of the latency estimation, especially for smaller models or devices with limited throughput.", "was_revised": true, "final_justification": "My concerns are mostly addressed in the rebuttal.", "rebuttal": "We thank the reviewer for their thoughtful feedback and detailed evaluation of our work.\n\n\n### Overview of the Review & Our Responses:\n\n\n\n* **S1 (Novel system-level approach):** We appreciate the reviewer's recognition of our innovative coupling of model sparsification with hardware I/O characteristics.\n* **S2 (Comprehensive evaluation):** We are glad the reviewer acknowledged our thorough experiments across multiple edge platforms, models, and benchmarks.\n* **W1 (Multi-model validation)**: We provide coefficient of variation analysis across 5 VLM models demonstrating universal smoothing phenomenon due to architectural properties, not model-specific behavior.\n* **W2 (Profiling costs and overheads)**: We clarify that profiling takes &lt;20 minutes per device and fixed overheads don't affect our algorithm's correctness due to its greedy ranking nature—designed to operate within strict 2ms runtime budgets. Future global optimization approaches requiring absolute latency precision would benefit from more precise latency modeling.\n\nWe now provide detailed responses to each concern.\n\n*Note: The proposed additions will be included in the main paper, with supplementary material used if necessary due to space constraints.*\n\n\n### W1: \"In section 3.1, the comparison of VLM smoother distributions of activation magnitudes is based on a single model assumption (llava-onevision-7B), and more models are needed for analysis and validation of the activation issue in VLMs.\"\n\nThe smoothing phenomenon is not model-specific but a mechanistic consequence of VLM architecture. This occurs due to two fundamental factors that apply universally to VLMs:\n\n\n\n* **Non-ReLU activation functions**: Modern VLMs universally adopt gated activations (SwiGLU, GeGLU) rather than ReLU, which produce continuous rather than sparse activations.\n* **Multi-token averaging**: VLMs process video frames as multiple patches (e.g., 14×14 = 196 tokens). Neuron importance is computed by averaging activation magnitudes across these tokens, which naturally reduces variation in importance scores.\n\nTo empirically support this explanation, we measured the coefficient of variation (CV) of neuron importance before the down-projection layer—where conventional sparsification is typically applied in ReLU-based LLMs—ensuring consistency when comparing VLMs with a ReLU-based LLM (OPT-6.7B):\n\n| Layer  | LLaVA-7B | LLaVA-0.5B | VILA-8B | NVILA-2B | LongVA | OPT-6.7B |\n|--------|----------|------------|---------|----------|---------|----------|\n| First  | 1.44     | 1.31       | 1.25    | 1.07     | 1.20    | 11.65    |\n| Mid    | 1.25     | 1.33       | 1.38    | 1.32     | 1.34    | 8.63     |\n| Last   | 3.30     | 3.58       | 2.48    | 4.55     | 3.01    | 9.19     |\n\nAll VLM models consistently show dramatically lower CV values (1.07-4.55) compared to the ReLU baseline (8.63-11.65), demonstrating this is an architectural property, not model-specific behavior. This smoothing makes contiguity-aware selection particularly beneficial for VLMs: when importance differences between neurons are small, I/O efficiency becomes the decisive factor for overall performance.\n\nWe will enhance Section 3.1 to provide a more detailed explanation of why activation smoothness emerges in VLMs—specifically due to gated activation functions and multi-token averaging—and include the coefficient of variation measurements across multiple VLM architectures to demonstrate this is a universal architectural property rather than a model-specific phenomenon.\n\n\n### W2: \"Limited Discussion on the profile cost and overheads in the latency Model...\"\n\nOur latency model design intentionally excludes fixed overheads without impacting algorithm effectiveness. Let us clarify both profiling costs and the rationale for this design choice:\n\n**Profiling Cost**: As noted in Line 179, the profiling process is detailed in Appendix B, where we measure throughput in 1KB increments until reaching 99% of peak performance, completing under 20 minutes per device. This one-time cost is negligible compared to deployment benefits.\n\n**Impact of Fixed Overhead and Greedy vs. Global Optimization Trade-offs**: Our greedy chunk selection uses importance-per-latency ratios for ranking candidates. Since fixed costs apply uniformly to all candidates, they don't affect the relative ranking used in our greedy selection. While global optimization algorithms would require absolute latency precision including fixed overheads, we use a greedy approach due to runtime constraints—executing ~196 times per frame for LLaVA-7B within a strict 2ms budget (Appendix H, Figure 4). As described in Lines 142-146, this frequent execution demands lightweight computation, making our current design well-suited for real-time inference.\n\nWe will enhance Section 4.1.1 to discuss profiling overhead considerations, expand Section 4.1.2 to explain how fixed overheads interact with our greedy algorithm and why they don't affect relative ranking decisions, and add to Section 6 a discussion of how future global optimization approaches could benefit from more sophisticated latency models that account for absolute costs, including fixed overheads."}]

Author rebuttal:

Meta-review: This paper proposes an efficiency-aware activation sparsification approach (Neuron Chunking) designed to accelerate vision-language models on edge devices. This is done through an efficient chunk-based neuron selection algorithm for improving the efficiency in VLM inference. While the proposed method is largely based on existing work on acceleration for flash-offloaded LLM inference, reviewers agree that this is a technically sound approach to making VLMs more efficient using hardware-aware sparsification. The paper is well written and the approach is solid and well validated empirically.
Decision: Accept (poster)