Memory-Efficient Visual Autoregressive Modeling with Scale-Aware KV Cache Compression
Authors: Kunjun Li
Zigeng Chen
Cheng-Yen Yang
Jenq-Neng Hwang
Venue: NeurIPS 2025 poster ()
Keywords: Visual Autoregressive Modeling，Image Generation，Efficiency
Primary area: deep_learning
TL;DR: We efficetively address the resolution scalability challenge caused by exponential growth in token sequences in VAR models, while preserving pixel-level fidelity.

Abstract: Visual Autoregressive (VAR) modeling has garnered significant attention for its innovative next-scale prediction approach, which yields substantial improvements in efficiency, scalability, and zero-shot generalization. Nevertheless, the coarse-to-fine methodology inherent in VAR results in exponential growth of the KV cache during inference, causing considerable memory consumption and computational redundancy. To address these bottlenecks, we introduce ScaleKV, a novel KV cache compression framework tailored for VAR architectures. ScaleKV leverages two critical observations: varying cache demands across transformer layers and distinct attention patterns at different scales. Based on these insights, ScaleKV categorizes transformer layers into two functional groups: drafters and refiners. Drafters exhibit dispersed attention across multiple scales, thereby requiring greater cache capacity. Conversely, refiners focus attention on the current token map to process local details, consequently necessitating substantially reduced cache capacity. ScaleKV optimizes the multi-scale inference pipeline by identifying scale-specific drafters and refiners, facilitating differentiated cache management tailored to each scale. Evaluation on the state-of-the-art text-to-image VAR model family, Infinity, demonstrates that our approach effectively reduces the required KV cache memory to 10% while preserving pixel-level fidelity.

Memory-Eficient Visual Autoregressive Modeling with Scale-Aware KV Cache Compression
Kunjun Li
Zigeng Chen
Cheng-Yen Yang*
Jenq-Neng Hwang*
University of Washington *
National University of Singapore
(kunjun, zigeng99) tu. nus nus.edu, {cycyang, hwang) edu
85
ScaleKV
efficient
10% KV Cache
ScaleKV
8.5 GB
Figure 1: We introduce new KV cache compression framework for Visual Autoregressive modeling that preserves pixel-level fidelity. On Infinity-8B, it achieves 10x memory reduction from 85 GB to 8.5 GB with negligible quality degradation (GenEval score remains 0.79 and DPG score marginally decreases from 86.61 to 86.49).
Abstract
Visual Autoregressive (VAR) modeling has garnered significant attention for its innovative next-scale prediction approach, which yields substantial improvements in efficiency, scalability, and zero-shot generalization. Nevertheless, the coarse-to- fine methodology inherent in VAR results in exponential growth of the KV cache during inference, causing considerable memory consumption and computational redundancy. To address these bottlenecks, we introduce ScaleKV, novel KV cache compression framework tailored for VAR architectures. ScaleKV leverages critical observations: varying cache demands across transformer layers and distinct attention patterns at different scales. Based on these insights, ScaleKV categorizes transformer layers into two functional groups: drafters and refiners. Drafters exhibit dispersed attention across multiple scales, thereby requiring greater cache capacity. Conversely, refiners focus attention the current token map to process local details, consequently necessitating substantially reduced cache capacity. ScaleKV optimizes the multi-scale inference pipeline by identifying scale-specific drafters and refiners, facilitating differentiated cache management tailored each scale. Evaluation the state-of-the-art text-to-image VAR model family, Infinity, demonstrates that our approach effectively reduces the required KV cache memory to 10% while preserving pixel-level fidelity. Code available at httts:://// :/gi gigg.......... com/ Swwaatttarrr/Saaoooo SSaleKV
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Traditional: Uniform Layer Budget Scales
Multi-Scale Tokens
Scale K
Scale K
+1
Uniform Layer
2
3
Uniform Layer
5
6
7
Uniform Layer
1
2
3
Uniform Layer
1
7
Uniform Layer
1
2
Uniform Layer
5
6
7
Ours: Scale-Aware Layer Budget Allocation
Reallocated
Scale K
Srale +1
Layer
3
Refiner Layer
6
Drafter Layer
1
2
4
Draftor Layer
3
4
5
7
Drafter Layar
2
Layer
7
Figure 2: By implementing scale-aware layer budget allocation, ScaleKV enables diffcrentiated cache management tailored to each layer's computational demands at every scale.
1 Introduction
68 31 have achieved impressive image Recent advances Autoregressive (AR) models [51
23
58 60 for unified vision understanding and genera- tion. However, the token-by-token generation approach requires numerous decoding steps. Visual Autoregressive (VAR) modeling (33 has revolutionized this process through next-scale prediction, enabling models to decode multiple tokens in parallel. Building upon this framework, several have demonstrated promising results for VAR-based text-to-image generation.
quality and multi-modal capabilities [34 63
61
approaches [6 52
Despite these advancements, VAR models face a fundamental scalability challenge due exponential growth in token sequence across scales. Unlike traditional next-token prediction models, which process one token per step with linear KV cache growth, VAR models must preserve KV states from all previous token maps. Generating a 1024x x 1024 image requires processing over 10K tokens across multiple scales, creating severe memory bottlenecks-will KV cache alone consuming approximately 85 GB of memory when generating 1024x x 1024 images with a batch size of 8 using Infinity-8B [16], text-to-image VAR model. Figure (a) illustrates this complexity, where each scale contributes a new KV cache entry of size h4 X wg. The total cache requirement grows cubically with the number of scales n, while the computational complexity of attention reaches (( These memory constraints and increased inference latency significantly impede practical deployment.
To address these inefficiencies, we analyze the specific properties of VAR's next-scale prediction paradigm. First, we observe that cache demands vary significantly across transformer layers. Next, we find that different scales exhibit distinct attention patterns. These findings indicate that VAR requires both layer-adaptive and scale-specific cache management strategies for optimal performance.
Our Approach. Inspired by these observations, we propose Scale-Aware KV Cache (ScaleKV), simple yet highly effective method that significantly reduces inference memory while maintaining high generation quality. Our approach categorizes transformer layers into two functional groups: Drafters and Refiners. Drafters distribute attention across multiple scales to access global information from preceding tokens, requiring greater cache capacity. In contrast, refiners focus attention current token map to process local details, necessitating substantially reduced cache storage. As illustrated in Figure S optimizes multi-scale inference by identifying scale-specific drafters and refiners, facilitating differentiated cache management to their computational demands at each scale.
Extensive evaluation demonstrates the effectiveness of our method. As shown in Figure I compared to the original Infinity-88 -B model, ScaleKV achieves negligible quality degradation (GenEval score remains 0.79 and DPG score decreases slightly from 86.61 to 86.49) while requiring merely 10% of the original GPU memory consumption. These results validate that ScaleKV effectively addresses the fundamental memory bottlenecks that have constrained the practical deployment of VAR models.
In conclusion, we introduce ScaleK V, a novel KV cache compression framework for VAR. S categorizes transformer layers into drafters and refiners and implements scale-aware layer budget

---

allocation. Through extensive experiments, our method achieves significant memory reduction while preserving pixel-level fidelity, enabling efficient deployment in resource-constraanee environments,
2
Related Works
Autoregressive Visual Generation. Early works 16
54 pioneered pixel-by-pixel image genera-
tion, later enhanced by VQVAE [55] and VQGAN
8 through image patch quantization. Recent
advances include GPT-style models 51 34, mixture-of-experts [23] linear attention [26]. diffusion-
autoregressive hybrids 63 81 and masked approaches [26 38 However, autoregressive
approaches suffer from substantial inference latency due to sequential token generation. VAR [53]
overcomes this limitation through hierarchical parallel decoding. and has been extended to text-to- image synthesis [16
74
52
40
27
31,, audio synthesis
45). and 3D content creation 73
Efficient Visual Generation. For diffusion models, efficiency optimization methods already well- 48
developed. [47 71
37
69] focus on reducing sampling steps while [29] 80 9 72 67
28 49 24
optimize models through quantization [24], pruning [ knowledge distillation
19 Several
approaches 41 59
76
66
50
25
39 79
skip redundant computations during the denoising process.
However, research memory optimization for VAR image generation remains in its early stages. Lite VAR [64] and VAR (i5] enhance inference speed but not address fundamental memory bottlenecks, while CoDe (7] improves memory efficiency through collaborative decoding but requires an additional VAR model. Hack 44 reduces memory via per-head budgets, but its irregular tensor shapes require specialized kernels. In contrast, approach directly reduces memory consumption and integrates with existing techniques to enhance efficiency.
KV Cache Compression. Current KV cache compression techniques for large language models (LLMs) and vision-language models (VLMs) primarily utilize quantization [36] 70
22 17 evic-
tion [78 35, 42 46 30], and merging 77
57 strategies. Quantization reduces precision but faces granularity limitations; eviction removes less important tokens using attention metrics while and merging consolidates redundant KV pairs.
33
56
optimizing allocation within budgets [12 65 10
However, primarily designed for single-sequence processing LLMs and VLMs, existing techniques fail to accommodate the multi-scale operational characteristics of VAR and the varying attention patterns across layers and scales.
3 Methods
3.1
Preliminary
Visual Autoregressive modeling (53] advances traditional AR approaches by shifting the prediction paradigm from "next token" to "next scale." Within this framework, each autoregressive operation produces token map corresponding to specific resolution scale, rather than individual tokens.
Given an image feature map E R VAR quantizes into multi-scale token maps R = (r1, .2..,,,. with increasingly finer resolutions. The joint probability distribution over these multi-scale maps is decomposed autoregressively according to:
K
p(ri, F2, ... TK) II p(rk (1, ....1)....T.)... k=1
where each token map F' E {1, V. V)h,, X,, contains hą x wy discrete tokens, selected from vocabulary of size V scale k. At each autoregressive step k; the model generates all h3 wk tokens comprising F1 in parallel, conditioning on previously generated scales (r1,..., rk-1). While VAR provides substantial improvements in both inference efficiency and generation quality, this coarse-to-fine generation strategy significantly expands the sequence length.
3.2 Key Observations
VAR represents an innovative paradigm that diverges from traditional autoregressive approaches. In this work, we examine the next-scale prediction process identify properties that can leveraged to reduce computational redundancy. Our analysis focuses specifically on attention patterns in VAR.

---

XV Cadio Xanoey (G85)
60.0
04
25.0
002
0.0
Scalie
Scaled Blockn, Head2 Stated Blos KS1
HoedS
Scale5 Blocko Head2 Scales, Böock31, Head3
KV Cache Metoony al diffieenn scales
[b) Allrrln Mtp V Visualizaion
Figure 3: (a) Exponential KV cache growth. (b) Visualization of two distinct attention patterns.
Cache demands vary significantly across transformer layers. Our analysis revealed two distinct at- tention pattern typologies in VAR models, visualized in Figure bb). The left pattern (BlockO_Heake exhibits diagonal structure with dispersed attention that spans preceding scales, indicating broad contextual integration and high cache demand. contrast, the right pattern (Block31 Heaa3)) demon- strates highly concentrated attention focused predominantly within the current token map, suggesting localized processing with minimal cache requirements. Through systematic analysis of these attention maps, we identified that most layers fall into one of these two categories, which we term Drafters and Refiners. Drafters distribute attention broadly across historical context, necessitating substantial KV cache capacity to access multiple scales. Refiners, however, concentrate attention primarily on local information within the current token map, requiring significantly reduced cache storage.
Different scales exhibit distinct attention patterns. As VAR progresses through its hierarchy, both groups display scale-dependent evolution. Drafter layers exhibit increasingly dispersed attention patterns at higher scales to integrate broader contextual information. Conversely, refiners grow progressively more concentrated, as evidenced by the heightened focus in Scale5 Block31 Head3 compared to its Scale4 counterpart (Figure ())). This bidirectional evolution reveals specialized hierarchical process where drafters gather global context while refiners perform localized processing.
These findings challenge both uniform cache allocation 62 and position- -baiid cache reduction S employed by current methods, suggesting that VAR models would benefit from adaptive allocation strategies accounting for both layer-specific requirements and scale-dependent characteristics.
3.3
Scale-Aware KV Cache
Based on our observations, we propose simple yet highly effective KV cache compression frame- work for next-scale prediction called Scale-Awar KV Cache, As illustrated in Figure ScaleK categorizes transformer layers into two functional groups termed Drafters and Refiners, implement- ing adaptive cache management strategies based on these roles. This approach optimizes multi-scale inference by identifying each layer's function at every scale, enabling adaptive cache allocation that aligns with specific computational demands of each layer.
Identifying Drafter and Refiner Layers. To systematically distinguish between drafter and refiner layers across different scales, we introduce the Attention Selectivity Index (ASI). This metric quantifies each layer's attention patterns by considering two critical factors: (1) the proportion of attention directed to current token map and (2) the concentration of attention in history sequence.
(L,k) Let a; represent the normalized attention score from query position in the current map l'k to key/value position j in layer when processing scale k. The token indices can be partitioned into history indices PR-1 (from previously generated maps PI....,Tk-1 and current map indices Cj (from Tk). The ASI for layer at scale & is defined as:
(L.)) E(NU(H))
Σ
(1,)) Ou E Co) Exee(HH) TTpKSum' ((a i PE-1
(1.)
Current Attention Ratio
History Top K Ratio
where E denotes expectation over query positions sampled uniformly from the current map Tk. and TopKSum'() computes the sum of the top- K attention scores directed toward the history tokens. The parameter K' controls the number of top scores included in the selectivity term.

---

Multi Scale Tokens Evicted
K
Drafte Layar
Refiner Layer
Layer
2
3
4
5
1
6
8
19
2
3
X
16
:
2
3
1 2
X
5
16
Laya
2
3
4
5
3
14
15
16
Overview of ScaleKV. Our method categorizes transformer layers into drafters (require extensive cache for global context) or refiners (process local details with minimal cache). This scale- wise identification enables adaptive cache allocation based on each layer's computational demands.
Figure 4:
Intuitively, high ASI(LK) value indicates that the layer either focuses strongly on the current map or exhibits high selectivity toward specific history tokens, or both. This suggests the layer is functioning refiner. Conversely, low AST(I.k) value indicates the layer distributes attention more broadly across the prefix context, characteristic of drafter behavior.
Since raw ASTU.K) values vary significantly across scales due to differences in token counts and attention patterns, we normalize these values within each scale using Z-scores. Let S = {(, k) 1 + L, S k K} be the set of all layer-scale pairs. We rank these pairs by their Z-soores and define the set of drafters D as the Na pairs with the lowest Z-scores:
ASI(L.E) MK
D = ((,,k) ESS (((A) S Z(Na))
Z(.A)
(3)
+
where Z(Na) represents the Na-th smallest Z-score. The remaining constitute the refiners R
S
The identification process occurs prior to inference using minimal calibration data. Our experiments demonstrate that set of 10 prompts is sufficient to accurately determine the drafters and refiners.
Cache Budget Allocation. After identifying drafters and refiners, we establish an efficient budget allocation strategy that satisfies the same total memory consumption uniform budget allocation Buniform while implementing scale-dependent reduction for refiners:
K (NN B2(k) + N B,())) Buniform L. = 8 k. 1=1
(4)
Here, Nf and N represent drafter/refiner layer counts at scale k. while Ba(k) and (,kk) denote their respective cache budgets. The parameter 8 controls the refiner budget decay rate. By leveraging the second observation that refiner attention exhibits increasing concentration at higher scales, refiner cache budgets are linearly reduced from the initial refiner budget (0) as scale increases. The saved memory is subsequently reallocated to drafters, ensuring Ba(k) > (k) to align with the scale-specific computational demands each layer.
KV Cache Selection. After establishing cache budgets for drafter and refiner layers, we implement an efficient token selection strategy to determine which specific KV states should be preserved.
For each token map Tk. we first partition the map into N patches and select the centroid token from each patch to form observation window W. This sampling approach ensures spatial coverage across the token map while maintaining a minimal memory footprint. We then evaluate the relative importance of the remaining tokens based their attention interactions with the observation window.
Similar to (30,, for each attention head h, we compute an importance score s for each token by measuring the cumulative attention it receives from the observation window tokens;
5

---

lighthouse origami style.
Freshly tea kottle
Breecktakling product
Creaee on imaae More
Ar. asdrenase t/e masn
Bravette pile girl Outdeors
Welcimn Puburd
WELCOME LFVTURE
EXPLORE MORE
Orignal
Budget - 20%
WELCO WELCOME FUUUREE
EXPLORE MORE
r1 1,45
Budget 30% H10 1:12
COME 4FVTURE
EXPLORE MORE
WELCOME LEYTURE
EXPLORE MORE
Figure 5: Qualitative comparison between the original Infinity-BB model and our proposed ScaleK V.
Σ JEW
Ah
A = softmax (Q* (K*)T/ /vd)).
(5)
jar
Here, A%, represents the normalized attention weight from query token in the observation window to key token and dy denotes the dimension of the key vectors. This formulation effectively quantifies each token's contribution to the contextual representation of the observation window.
After calculating cumulative attention, we aggregate these scores through average pooling to produce unified importance metric. For each layer I, we then select the top-k' tokens with the highest aggregated importance scores, where ke corresponds to the layer-specific cache budget determined by our allocation strategy. Only the KV states of these selected tokens, along with the observation window tokens, are preserved in the cache for subsequent steps, while others are efficiently pruned.
The observation window is compact (typically comprising only 16 tokens), enabling efficient attention score computation between this window and the remaining tokens with negligible computational overhead. Rather than hindering performance, our experimental results show this approach yields notable speedups of up to 1. 1,25x through significantly reduced KV cache memory requirements.
4 Experiments
4.1
Experimental Setup
Base Models. We evaluated S on two VAR-based text-to-image models of different capacities: Infinity-2B and Infinity-SB [16), to validate our method's generalizability across model scales. To represent practical deployment scenarios with varying resource constraints, we analyzed performance under three memory budget constraints: 4%. 10%, and 20% of the original KV cache size.
Dataset and Metrics. We assessed output consistency with the original models using the MS-COCO 2017 (32) validation set, which comprises 5,000 images and captions. Consistency was measured by the Fréchet Inception Distance (FID) [8] Learned Perceptual Image Patch Similarity (LPIPS) 75 and Peak Signal-to- -to-isee Ratio (PSNR). We also used two established image generation benchmarks GenEval [13] and DPG 20 for perceptual quality and semantic alignment with input prompts. For memory efficiency, we report the KV cache memory usage measured with a batch size of We use GPT-40 [21] to generate 10 prompts for calibrating drafters and refiners. Our drafterr/efiner identification method effective across different calibration prompt sources, with results converging after analyzing only small sample of prompts, as detailed in the appendix.

---

Table 1: Quantitative comparisons of output consistency MS-COCO 2017 dataset.
Infinity-29 2B
Infinity
Method
Budget
KV Cache
FID
LPIPS
PSNR
KV Cache
FID 1
LPIPS 4 PSNR
Full Cache
100%
38550 MB
84328 MB
Sliding Window
20%
7800 MB
5.63
0.17
20.71
17062 MB
4.82
0.14
20.99
Streamingl
SnapKV [30
20%
7800 MB
3.25
0.12
22.51
17062 MB
3.10
0.10
22.65
PyramidKV
20%
7800 MB
3.23
0.11
22.62
17062 MB
3.03
0.10
22.76
ScaleKV
20%
7800 MB
1.82
0.08
24.84
17062
1.45
25.60
Sliding Window
10%
3900 MB
8.58
0.24
18.99
8531 MB
8.71
0.20
19.02
StreamingLLM 62
MB
5.49
0.19
8531 MB
6.29
0.17
19.97
SnapKV 30
10%
3900 MB
4.66
0.16
20.83
8531 MB
4.68
0.15
20.60
PyramidKV
10%
3900 MB
4.52
0.16
20.92
8531 MB
4.69
0.14
20.79
ScaleK
2.12
23.25
Sliding Window
4%
1590 MB
16.68
0.30
17.49
3478 MB
19.23
0.27
17.50
StreamingL_MM 62
4%
1590 MB
8.71
0.25
18.31
3478 MB
8.54
0.22
18.63
SnapKV 3D
4%
1590 MB
5,10
0.24
18.23
3478 MB
6.68
0.19
19.15
PyramidKV
4%
1590 MB
5.51
0.223
18.65
3478 MB
6.55
0.19
19.26
ScaleK
4%
3.51
0.16
3478 MB
3.37
21.41
Table 2: Quantitative comparisons of perceptual quality on GenEval and DPG Benchmarks.
GenEval
DPG
Methods
Params
Two Obj.
Position
Color Attri
Overall +
Global
Relation
Overall
SDXL 11
2.6B
0.74
0.15
0.23
83.27
86.76
74.65
LlamaGie
0.8B
0.34
0.07
0.04
0.32
65.16
Show-o 61
1.3B
0.80
0.31
0.50
0.68
67.48
PixAri-Signa Sigma
5
0.6B
0.62
0,14
0,27
0,55
86.89
86.59
80.54
HART [52
0.7B
0.62
0.13
0.18
0.51
80.89
DALL-E3
90.97
90.58
83,50
Emu3 [88
8.5B
0.81
0.49
0.45
0.66
81.60
Infinity-2B 16
2.0B
0.84
0.43
0.57
0.725
89.01
90.03
83.06
ScalekV (10%)
2.0B
0.84
0.44
0.55
0.730
82.45
90.48
83.01
Infinity-8B 16
8.0B
0.89
0.61
0.68
0.792
89.51
93.08
86.61
SV (10%)
8.0B
0.89
0.61
0.67
0.790
92.49
88.92
86.49
Compression Baselines. We assessed ScaleKV's performance mainly against four representative
KV cache compression baselines: Sliding Window Attention [ (retains local window of most recent tokens), StreamingL.LM [62] (keeps attention sinks (initial tokens) and recent tokens), SnapKV (30) (clusters tokens based on attention scores) and PyramidKV (employs a fixed pyramid-shaped allocation strategy across transformer layers).
4.2 Main Results
Comparison with Compression Baselines. Table Presentsss presents evaluation the MS-COCO 2017 validation set [32], focusing on pixel-level consistency with the original outputs. ScaleKV consistently outperforms all baselines across different memory budgets, with significant improvements in FID, LPIPS, and PSNR metrics.
At the most constrained budget (4%), ScaleKV achieves FID reductions of 31.2% and 48.5% compared to the next best bascline for Infinity-2B and Infinity-8B The performance gap widens at higher budgets, with ScaleKV achieving FID scores of 1.82 and L.45 at 20% budget. representing substantial improvements over all competitors. The LPIPS results further validate these findings, with ScaleKV achieving scores of 0.08 and 0.06 at 20% budget for the two models, compared to PyramidKY V's 0.11 and 0.10, indicating better perceptual similarity to the original outputs.
Each baseline exhibits specific limitations in the VAR context: Sliding Window Attention and StreamingLLMM employ statie policies that lose information carried by middle tokens, resulting in poor performance at low budgets. SnapK clustering strategy helps preserve some image coherence cannot effectively prioritize critical tokens across different scales. Notably, PyramidK V's fixed allocation pattern offers limited improvement over SnapKV and sometimes produces worse results

---

Small Scoles
4.13
Large Scaales
1.4
2.54
0.6
2.5
2.3
04
24-
2.58
2.50
200 Nomallzod Cumoon tirntion Distihution
ABC
Nankon
[bo Rinftse Identificaiion Mebiic
[4) Rafiner Rodget Decay Rata
Figure 6: (a) Kernel Density Estimation of normalized current attention scores at small scales (T2, T3, ra) and large scales (r10: '1, T12). (b) Ablation experiments on using different drafter identification metrics. (c) Ablation experiments on the impact of refiner budget decay rate.
(e.g., 4% 4% budget on Infinity-2B with FID 5.51 vs. SnapKV's 5.10), confirming that predetermined allocation strategies do not generalize well to VAR's scale-dependent attention behaviors.
Results GenEval and DPG. To further validate the perceptual quality and semantic understanding capabilities of our compressed models, we conducted evaluations on two established benchmarks: GenEval [K] and DPG (20). Table presents these results, comparing our ScaleK V-compressed
models against state-of-the-an image generation models, including diffusion models [43] and
autoregressive models [51 63 58 52]. The results demonstrate that ScaleKV preserves semantic
understanding remarkably well despite substantial KV cache reduction. For Infinity-2B. our method delivers exceptional performance that matches the full model (83.01 vs. 83.06 on DPG), with the GenEval score even showing a slight improvement from 0.725 to 0.730, while using only 10% of the original KV cache. Similarly, for Infinity-8B, ScaleKV maintains nearly identical performance (0.790 vs. 0.792 on GenEval, and 86.49 vs. 86.61 on DPG). This minimal performance degradation particularly noteworthy given that Infinity models already outperform most existing approaches on these benchmarks, including larger models like DALL-E 3 and Emu3-8.51 ScaleK V-compressed Infinity-8B maintains this superior performance while requiring only 8.5 GB of KV cache memory, a dramatic reduction from the original 85 GB.
Qualitative Results. We provide an extensive qualitative comparison between the Infinity-8B model with full KV cache and our proposed ScaleKV, with varying budgets of 4%, 10%, and 20%. As illustrated Figure 5 our approach achieves significant memory optimization, with only minimal quality degradation that is nearly imperceptible to the human eye. Even at compression rate of 25 times, the generated images maintain exceptionally high quality and accurate semantic information.
4.3 Analytical Experiments
Attention Distributions Across Scales. We quantify attention pattern variations throughout the multi- scale generation using normalized current scale attention, defined as the average attention per token within the current scale to the average attention across the entire sequence, to capture contribution of the cached tokens to generation. Figure (()) demonstrates the kernel density estimation (KDE) of normalized current attention, revealing distinct distributions across small scales (rz, F', r44 and large scales (F1o: F11, P12). Small scales exhibit an approximately uniform distribution, indicating broad context utilization without strong selectivity. In contrast, large scales concentrate around higher attention values, suggesting focused information selection. This progression reflects an evolution from global information aggregation in early scales selective attention refinement in later scales.
Impact of Drafter Identification Metric. Figure b(b) demonstrates the comparative effectiveness of different metrics for drafter identification. Our proposed Attention Selectivity Index (ASI) (Equa- tion 2) combines the attention score ratio in the current token map and the Top- K ratio in history sequence. When evaluated on Infinity-2B with 10% cache budget constraint, ASI achieves an FID score of 2.53, representing a substantial 42.5% improvement over using only the Top-K ratio (4.60). This significant performance gap validates that our comprehensive attention pattern analysis effectively identifies layers requiring larger cache allocation, enabling optimal resource distribution.
Impact of Refiner Budget Decay Rate. Figure ()c) shows the effectiveness of refiner budget decay strategy (Equation 4 under 10% budget constraint (650 tokens per head/layer). With an initial refiner budget of 600 tokens, observe consistent improvement in FID from 3.49 to 2.53 as decay

---

Infinity-28
Infinity-8B
Infinity-88
15.0
StreemlbgLLM
12.5
StreamingL SnapKV
Pull Cache
71
12.5
SnapKV
10.0
Latening (ma)
(10%)
10.0
ScaleKV
7.5-
ScaleKV
7.5-
29
5.0
5.0
12
10%
166
1%
10%
16%
768
1024
KV Cache Budget (%)
KV Cache Dudget (%)
Resolution
(a) Quality Efficiency Trade-off
(b) Inference Latency
Figure 7: (a) FID under different KV cache budgets. (b) Inference latency for different resolutions.
Table 3: Memory usage comparison across different batch sizes
Memory Consumption]
Method
Running
KV Cache
Params
Total
Infinity (bs=1)
LIGB 0.8GB
10.5GB
19,5GB
31.4GB 21.4GB
+ScaleKY (10%)
LIGB
19,5GB
Infinity (bs=8)
85GB
19,5GB
0OM
+ScaleKY (10%)
21JGGB
855B
49,2GB
Infinity (bs=16) +ScaleKV (10%)
170GB
19.5GB
0OM (>10KCB))
42.4GB
17.1GB
19.5GB
78.8GB
rate increases from 0 to 70, confirming our observation that refiner attention becomes increasingly focused higher scales, requiring fewer resources. The monotonic improvement also validates our drafter identification method, reallocating cache capacity from refiners to drafters consistently enhances generation quality, indicating accurate identification of layers with divergent cache demands.
Quality-Efficiency Trade-off. We evaluated the quality-efficiency trade-off of ScaleKV against established compression methods across different cache budgets, shown in Figure a). For both Infinity-2B and Infinity-8B models, ScaleK consistently achieves the lowest FID at all budget constraints, with performance gap widening at more restrictive memory allocations. These results demonstrate that ScaleK' effectively preserves generation quality even under severe memory con- straints, making adaptable to diverse deployment scenarios with varying computational resources.
Memory Efficiency and Time Cost Analysis. In Table we present a comprehensive analysis of memory consumption during the Infinity-8B model inference process. The KV cache of the Infinity model is the largest memory consumer, requiring approximately 10 times the memory needed for the model's decoding operation due to the significantly extended sequence length. Our proposed S drastically reduces the KV cache memory requirements, compressing it to 10% of the original model. Moreover, batch size increases, the memory savings with ScaleK become even more pronounced. We are able to generate images with a large batch size of 16 using less than 80GB total memory, whereas the KV cache alone of the original model requires 770G during inference.
While primarily developed to improve memory efficiency, ScaleKV also delivers notable inference acceleration by reducing tensor access and transfer operations. Figure bb) illustrates how inference latency increases substantially with image resolution due to exponential growth in the token sequence. Our method achieves up to 1.25x speedup becoming more pronounced as resolution increases. These results demonstrate ScaleKV's potential for deployment resource-con couurrce-onstrrieedd environments and scaling VAR models to ultra-high resolutions such as 4K, which would otherwise be limited by memory bottlenecks and inference latency.
single NVIDIA H20 GPU, with performance gains
5
Conclusion
This work introduces ScaleKV, novel KV cache compression framework for Visual Autoregressive modeling that effectively addresses the memory bottlenecks in high-resolution image generation. By implementing scale-aware layer budget allocation, ScaleKV enables adaptive cache management tailored to the specific demands of each layer across scales. Through extensive experimentation, our method demonstrates superior efficiency-quality trade-off, enabling efficient deployment in resource- constrained environments and facilitating the scaling of VAR models to ultra-high resolutions.

---

References
[1] Iz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv preprint arXiv:200..05/54 2020.
[2] James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li. Long Ouyang. Juntang Zhuang. Joyce Lee. Yufei Guo, al. Improving image generation with better captions. Computer Science. hmps://cdn. openai. com/papersidall--- pdf, 2(3):8, 2023.
[3] Zefan Cai, Yichi Zhang, Bofei Gao, Yuliang Liu, Tianyu Liu, Keming Lu, Wayne Xiong, Yue Dong, Baobao Chang, Junjie Hu, and Xiao Wen. Pyramidkv: Dynamic kv cache compression based on pyramidal information funneling. arXiv preprint
Stti::2406.020000 2024.
[4] Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T Freeman. Maskgit: Masked generative image transformer. Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 11315 11315-13222 11325, 2022.
[5] Junsong Chen, Chongjian Ge, Faze Xie, Yue Wu, Lewei Yao, Xiaozhe Ren, Zhongdao Wang. Ping Luo, Huchuan Lu, and Zhenguo Li. PixArt-Sigma: Weak-to-strong training of diffusion transformer for 4k text-to-image generation. In ECCV, pages 74-91. Springer, 2024.
[6] Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever. Generative pretraining from pixels. In International conference on machine learning, pages 1691-1703. PMLR, 2020.
[7] Zigeng Chen, Xinyin Ma, Gongfan Fang, and Xinchao Wang. Collaborative decoding makes visual auto-regressive modeling efficient. arXiv preprint arXiv:2411. 17787, 2024.
[8] Patrick Esser, Rohin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12873-12883, 2021.
[9] Gongfan Fang, Kunjun Li, Xinyin Ma, and Xinchao Wang. Tinyfusion: Diffusion transformers learned shallow. arXiv preprint arXiv:2412.0f 199, 2024.
[10] Yuan Feng, Junlin Lv, Yukun Cao, Xike Xie, and Kevin Zhou. Ada-kv: Optimizing kv cache eviction by adaptive budget allocation for efficient Ilm inference. arXiv preprint arXiv:2407.11550 2024.
[11] Yu Fu, Zefan Cai, Abedelkadir Asi, Wayne Xiong, Yue Dong, Wen Xiao. Not all heads matter:
A head-level kv cache compression method with integrated retrieval and reasoning. arXiv preprint vrXiv:2410.19258 2024.
[12] Suyu Ge, Yunan Zhang, Liyuan Liu, Minjia Zhang, Jiawei Han, and Jianfeng Gao. Model tells you what to discard: Adaptive kv cache compression for llms. arXiv preprint 1100022310001801, 2023.
[13] Dhruba Ghosh, Hannaneh Hajishirzi, and Ludwig Schmidt. Geneval:
An object-focused framework for evaluating text-to-image alignment. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 52132-52152. Curran Associates, Inc., 2023.
[14] Jiatao Gu, Yuyang Wang, Yizhe Zhang, Qihang Zhang, Dinghuai Zhang, Navdeep Jaitly, Josh Susskind, and Shuangfei Zhai. Dart: Denoising autoregressive transformer for scalable text-to-image generation. arXiv preprint arXiv:2410.088
2024.
[15] Hang Guo, Yawei Li. Taolin Zhang. Jiangshan Wang. Tao Dai, Shu-Tao Xia, and Luca Benini. Fastvar: Linear visual autoregressive modeling via cached token pruning. arXiv preprint arXiv:2503.23333 2025.
[16] Jian Han, Jinlai Liu, Yi Jiang, Bin Yan, Yuqi Zhang, Zehuan Yuan, Bingyue Peng, and Xiaobing Liu. Infinity: Scaling bitwisc autoregressive modeling for high-resolution image synthesis, 2024.
[17] Yefei He, Luoming Zhang, Weijia Wu, Jing Liu, Hong Zhou, and Bohan Zhuang. Zipcache: Accurate and efficient kv cache quantization with salient token identification. arXiv preprint arXiv:2405.142255 2024.
[18] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
[19] Geoffrey Hinton, Oriol Vinyals, Jeff Dean, et al. Distilling the knowledge in a neural network. arXiv preprint arvi: 503.02531, 2(7), 2015.
[20] Xiwei Hu, Rui Wang, Yixiao Fang, Bin Fu, Pei Cheng, and Gang Yu. Blla: Equip diffuusion models with Ilm for enhanced semantic alignment. arXiv preprint arXiv:2403.051.55 2024.
[21] Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint 21X1:2241. 2024.
10

---

[22] Hao Kang, Qingru Zhang, Souvik Kundu, Geonhwa Jeong, Zaoxing Liu, Tushar Krishna, and Tuo Zhao. Gear: An efficient kv cache compression recipe for near-lossless generative inference of llm. arXiv preprint arXiv:2403.088 0.55277 2024.
[23] Haopeng Li, Jinyue Yang, Kexin Wang, Xuerui Qiu, Yuhong Chou, Xin Li, and Guoqi Li. Scalable autoregressive image generation with mamba. arXiv preprint arXiv:2408. 12245, 2024.
[24] Muyang Li*, Yujun Lin Zhekai Zhang*, Tianle Cai, Xiuyu Li, Junxian Guo, Enze Xie, Chenlin Meng. Jun Zhu, and Song Han. Svdquant: Absorbing outliers by low-rank components for 4-hit diffusion models. arXiv preprint arXiv:241 1,05007, 2024.
[25] Senmao Li, Taihang Hu, Fahad Shahbaz Khan, Linxuan Li, Shiqi Yang, Yaxing Wang, Ming-Ming Cheng, and Jian Yang. Faster diffusion: Rethinking the role of unet encoder in diffusion models. arXiv preprint arXiv:2312 220233....9000. 2023.
[26] Tianhong Li, Yonglong Tian, Hc Li, Mingyang Deng, and Kaiming Hc. Autoregressive image generation without vector quantization. arXiv preprint arXiv:2406./ 1838, 2024.
[27] Xiang Li, Kai Qiu, Hao Chen, Jason Kuen, Zhe Lin, Rita Singh, and Bhiksha Raj. Controlvar: Exploring controllable visual autoregressive modeling. arXiv preprint arXiv:2406.0977 2024.
[28] Xiuyu Li, Yijiang Liu, Long Lian, Huanrui Yang, Zhen Dong, Daniel Kang, Shanghang Zhang, and Kurt Keutzer. Q-diffusion: Quantizing diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 17535 17545, 2023.
[29] Yanyu Li, Huan Wang. Qing Jin, Ju Hu, Pavlo Chemerys, Yun Fu, Yanzhi Wang. Sergey Tulyakov, and Jian Ren. Snapfusion: Text-to-image diffusion model on mobile devices within two seconds. Advances in Neural Information Processing Systems, 36, 2024.
[30] Yuhong Li, Yinghing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen. Snapkv: Lim knows what you are looking for before generation. arXiv preprint arXi4:2404. 14469, 2024.
[31] Zongming Li, Tianheng Cheng, Shoufa Chen, Peize Sun, Haocheng Shen, Longjin Ran, Xiaoxin Chen, Wenyu Liu, and Xinggang Wang. Controlar: Controllable image generation with autoregressive models. arXiv preprint arXiv:2410.02705, 2024.
[32] Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zätnick. Microsoft coco: Common objects in context. In Computer vision-ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, proceedings, part 13, pages 1400-55..6 Springer. 2014.
[33] Akide Liu, Jing Liu, Zizheng Pan, Yefei He, Gholamreza Haffari, and Bohan Zhuang. Minicache: Ky cache compression in depth dimension for large language models. arXiv preprint arXiv:2405. 2024.
[34] Dongyang Liu, Shitian Zhao, Le Zhuo, Weifeng Lin, Yu Qiao, Hongsheng Li, and Peng Gao. Lumina-mgpt: lllmiintte ficxible photorcalistic text-to-image generation with multimodal generative pretraining. arXiv preprint arXi::2408. 240.02200. 2024.
[35] Zichang Liu, Aditya Desai, Fangshuo Liao, Weitao Wang, Victor Xie, Zhaozhuo Xu, Anastasios Kyrillidis, and Anshumali Shrivastava. Scissorhands: Exploiting the persistence of importance hypothesis for Ilm kv cache compression test time. Advances in Neural Information Processing Systems, 36, 2024.
[36] Zirui Liu, Jiayi Yuan, Hongye Jin, Shaochen Zhong, Zhaozhuo Xu, Vladimir Braverman, Beidi Chen, and Xia Hu. Kivi: A tuning- free asymmetric 2bit quantization for cache. arXiv preprint arXiv:2402.02750 2024.
[37] Simian Luo. Yigin Tan, Longbo Huang. Jian Li, and Hang Zhao. Latent consistency models: Synthesizing high-resolution images with few-step inference. arXiv preprint arXiv:2310.04376 2023.
[38] Zhuoyan Luo. Fengyuan Shi, Yixiao Ge, Yujiu Yang, Limin Wang, and Ying Shan. source project toward democratizing auto-regressive visual generation. arXiv preprint vrXiv:2409.04411 2024.
An open-
[39] Zhaoyang Lyu, Xudong Xu, Ceyuan Yang, Dahua Lin, and Bo Dai. Accelerating diffusion models via early stop of the diffusion process. arXiv preprint arXiv:2205.125:: 125244 2022.
[40] Xiaoxiao Ma, Mohan Zhou, Tao Liang, Yalong Bai, Tiejun Zhao, Huaian Chen, and Yi Jin. Star: Scale-wise text-to-image generation via auto-regressive representations. arXiv preprint 2406. 10797, 2024.
[41] Xinyin Ma. Gongfan Fang, and Xinchao Wang. Deepeache: Accelerating diffusion models for free. arXiv preprint
2023.
[42] Matanel Oren, Michael Hassid, Yossi Adi, and Roy Schwartz. Transformers are multi state mns. arXiv preprint 000000200000 2024.

---

[43] Dustin Podell, Zion English. Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. SDXL: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint wwXXiv2307.019955 2023.
[44] Ziran Qin. Youru Lv, Mingbao Lin, Zeren Zhang, Danping Zou, and Weiyao Lin. Head-aware ky cache compression for efficient visual autoregressive modeling. arXiv preprint arXivv2504.0925) 2025.
[45] Kai Qiu, Xiang Li, Hao Chen, Jie Sun, Jinglu Wang, Zhe Lin, Marios Savvides, and Bhiksha Raj. Efficient autoregressive audio modeling via next-scale prediction. arXiv preprint arXiv:2408. 09007, 2024.
[46] Siyu Ren and Kenny Q Zhu. On the efficacy of eviction policy for key-value constrained generative language model inference. arXiv preprint arXiv:2402. 06262, 2024.
[47] Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. arXiv preprint 220i::2002. 005 2022.
[48] Axel Sauer, Dominik Lorenz, Andreas Blattmann, and Robin Rombach. Adversarial diffusion distillation. arxiv preprint arXiv:2311.17042, 2023.
[49] Yuzhang Shang, Zhihang Yuan, Bin Xie, Bingzhe Wu, and Yan Yan. Post-training quantization on diffusion models. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1972-1981, 2023.
[50] Junhyuk So. Jungwon Lee, and Eunhyeok Park. Frdiff: Feature reuse for exquisite zero-shot accelcration of diffusion models. arXiv preprint arXiv:23/2.03517, 2023.
[51] Peize Sun, Yi Jiang, Shoufa Chen, Shilong Zhang. Bingyue Peng, Ping Luo, and Zehuan Yuan. Autore- gressive model beats diffusion: Llama for scalable image generation. arXiv preprint arX15:2406.06555 2024.
[52] Haotian Tang, Yecheng Wu, Shang Yang, Enze Xie, Junsong Chen, Junyu Chen, Zhuoyang Zhang, Han Cai, Yao Lu, and Song Han. HART: Efficient visual generation with hybrid autoregressive transformer. arXiv preprint arXiv:2410. 20.000022, 2024.
[53] Keyu Tian, Jiang, Zehuan Yuan, Bingyue Peng, and Liwei Wang. Visual autoregressive modeling: Scalable image generation via next-scale prediction. Advances in neural information processing systems, 37:84839-84865, 2024.
[54] Aaron Van den Oord, Nal Kalchbrenner, Lasse Espeholt, Oriol Vinyals, Alex Graves, et al. Conditional image generation with pixelenn decoders. Advances in neural information processing systems, 29, 2016.
[55] Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning, Advances in neural information processing systems, 30, 2017.
[56] Zhongwei Wan, Xinjian Wu, Yu Zhang, Yi Xin, Chaofan Tao, Zhihong Zhu, Xin Wang, Siqi Luo, Jing Xiong, and Mi Zhang. D2o: Dynamic discriminative operations for efficient generative inference of large language models. arXiv preprini arXiv:2406.1 13035, 2024.
[57] Zhongwei Wan, Ziang Wu, Che Liu, Jinfa Huang. Zhihong Zhu, Peng Jin, Longyue Wang. and Li Yuan. Look-m: Look-once optimization in ky cache for efficient multimodal long-contex! inference. arXiv preprint arXiv:2406.18112
2024.
[58] Xinlong Wang, Xiaosong Zhang, Zhengxiong Luo, Quan Sun, Yufeng Cui, Jinsheng Wang. Fan Zhang, Yueze Wang, Zhen Ii, Qiying Yu, et al. Emuu3 Next-token prediction is all you need. arXiv preprint 18880 2024.
[59] Felix Wimbauer, Bichen Wu, Edgar Schoenfeld, Xiaoliang Dai, Ji Hou, Zijian He, Artsiom Sanakoyeu, Peizhao Zhang, Sam Tsai, Jonas Kohler, et al. Cache me if you can: Accelerating diffusion models through block caching. arXiv preprint arXiv:2312.033: 0200999 2023.
[60] Chengyue Wa, Xiaokang Chen, Zhiyu Wa, Yiyang Ma, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, Chong Ruan, et al. Janus: Decoupling visual encoding for unified multimodal understanding and generation. arXiv preprint arXiv:2410. 138488 2024.
[61] Yecheng Wa, Zhuoyang Zhang, Junyu Chen, Haotian Tang, Dacheng Li, Yunhao Fang, Ligeng Zhu, Enze Xie, Hongxu Yin, Li Yi. al. VILA-U: A unified foundation model integrating visual understanding and generation. arXiv preprint ar arXiv:2409.044 (4000,, 2024.
[62] Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. arXiv, 2023.
[63] Jinheng Xie, Woijia Mao, Zechen Bai, David Junhao Zhang, Weihao Wang, Kevin Qinghong Lin, Yuchao Gu. Zhijie Chen. Zhenheng Yang, and Mike Zheng Shou. Show-o: One single transformer to unify multimodal understanding and generation. arXiv preprint arXiv:2408. 0204408..222..... 12528, 2024.
[64] Rui Xie. Tianchen Zhao, Zhihang Yuan, Rui Wan, Wenxi Gao, Zhenhua Zhu, Xuefei Ning, and Yu Wang. Litevar: Compressing visual autoregressive modelling with efficient attention and quantization. arXiv preprint 21111.2411.17178. 2024.
12

---

[65] Dongjie Yang. XiaoDong Han, Yan Gao. Yao Hu, Shilin Zhang, and Hai Zhao. Pyramidinfer: Pyramid kv cache compression for high-throughput I1m inference. arXiv preprint arXiv:2405.12532 2024.
[66] Xingyi Yang and Xinchao Wang. Hash3d: Training-free acceleration for 3d generation. arXiv preprint arXiv:2404.06099 2024.
[67] Xingyi Yang, Daquan Zhou, Jiashi Feng, and Xinchao Wang. Diffussion probabilistic model made slim. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22552- 22562, 2023.
[68] Ziyu Yao, Jialin Li, Yifeng Zhou, Yong Liu, Jiang, Chengjie Wang, Feng Zheng, Yuexian Zou, and Lei Li. Car: Controllable autoregressive modeling for visual generation. arXiv preprint arXiv:2410.04671, 2024.
[69] Tianwei Yin, Michaäl Gharbi, Richard Zhang, Eli Shechtman, Fredo Durand, William Freeman, and Taesung Park. One-step diffusion with distrihution matching distillation. arXiv preprint arXiv:231 188228 2023.
[70] Yuxuan Yue, Zhihang Yuan, Haojie Duammu, Sifan Zhou, Jianlong Wu, and Liqiang Nie. Wkvquant: Quan- tizing weight and key/value cache for large language models gains more, arXiv preprint arXiv:2402. 12065, 2024.
[71] Zongsheng Yue, Jianyi Wang. and Chen Change Loy. Resshift: Efficient diffusion model for image super-resolution by residual shifting, Advances Neural Information Processing Systems, 36, 2024.
[72] Dingkun Zhang, Sijia Li. Chen Chen, Qingsong Xie. and Haonan Lu. Laptop-diff: Layer pruning and normalized distillation for compressing diffusion models. arXiv preprint 2r 2024.
[73] Jinzhi Zhang, Feng Xiong, and Mu Xu. Gåpt: Unleash the power of autoregressive modeling in 3d generation via cross-scale querying transformer. arXiv preprint arXiv:2409, 06322, 2024.
[74] Qian Zhang, Xiangzi Dai, Ninghua Yang, Xiang An, Ziyong Feng, and Xingyu Ren. Var-clip: Text-to- image generator with visual auto-regressive modeling. arXiv preprint arXiv:2408.0I 181, 2024.
[75] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as perceptual metric. Proceedings of the IEEE conference on computer vision and pattern recognition, pages 586-595, 2018.
[76] Wentian Zhang, Haozhe Liu, Jinheng Xie, Francesco Faccio, Mike Zheng Shou, and Jürgen Schmid- huber. Cross-attention makes inference cumbersome in text-to-image diffusion models. arXiv preprint arXiv:2404.022 022447, 2024.
[77] Yuxin Zhang, Yuxuan Du, Gen Luo, Yunshan Zhong, Zhenyu Zhang, Shiwei Liu, and Rongrong Ji. Cam Cache merging for memory-efficient Ilms inference. In Forty-first International Conference on Machine Learning, 2024.
[78] Zhenyu Zhang, Ying Sheng. Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song. Yuandong
Tian, Christopher Ré, Clark Barrett, et al. H2o: Heavy-hitter hitter oracle for efficient generative inference of large language models. Advances Neural Information Processing Systems, 36:34661- -347100, 2023.
[79] Xuanlei Zhao, Xiaolong Jin, Kai Wang, and Yang You. Real-time video generation with pyramid attention broadcast. arXiv preprint arXiv:248.. /2588, 2024.
[80] Yang Zhao, Yanwu Xu, Zhisheng Xiao, and Tingbo Hou. Mobilediffusion: Subsecond ext-to--maae generation on mobile devices. arXiv preprint (tfi:2311.16660 2023.
[81] Chunting 7hou, Lili Yu, Arun Babu, Kushal Tirumala, Michihiro Yasunaga, Leonid Shamis, Jacob Kahn, Xuezhe Ma, Lake Zettlemoyer, and Omer Levy. Transfusion: Predict the next token and diffuse images with one multi-modal model. arXiv preprint 2r0iv:2408. 1039, 2024.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The main claims made in abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitation of our work in the Appendix.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
14

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide mental results.
a detailed description of our method along with extensive experi-
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
15

---

Answer: [Yes]
Justification:
We offer the full code along with relevant instructions.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ /pipps.cc/pullc//uuiii wwwticccguie//eeeeeeeeeii CoodeSubeiiieeee ssi wswonnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We provide all the details about the experiment in our paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We provide the details about initialization and dataset split.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
16

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative eтror rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide the details about the computation resources we used in the experi- ments.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: htttps://euurpp...ttt httprppp....ppbbcc/cccceeeeeeeee
Answer: [Yes]
Justification:
We strictly adhere to the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations their jurisdiction..
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss in Appendix.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
17

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
The creators or original owners assets (e.g., code, data, models) used the paper are properly credited, and the license and term of use are explicitly mentioned and properly adhered to.
Justification:
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version the asset used and, possible, include URI.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswithcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
18

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: New assets introduced in the paper are well documented, and the documenta- tion is provided alongside the assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
19

---

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: We do LLMs methods in this research.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non standtdddard components.
Please refer to our LLM policy for what should or should not be described.
20

---

Technical Appendices and Supplementary Material
In this document, we provide supplementary material that could not fit into the main manuscript due to the page limit. It includes detailed explanations, visualization results, and quantitative experiments.
A Robustness to Calibration Data
To evaluate the stability of our drafter/refiner identification method across varying calibration set sizes, we conducted an ablation study using prompts sampled from the LAION-Art dataset. We systematically evaluated ScaleKV's performance using calibration sets ranging from single prompt to 128 prompts, measuring the resulting FID scores on the MS-COCO validation set. As demonstrated in Figure the FID score remains stable at 2.53 with zero standard deviation across the entire range of calibration set sizes. This exceptional consistency indicates that the attention patterns distinguishing drafters from refiners represent fundamental architectural properties of VAR models rather than dataset-specific characteristics. The immediate convergence with even a single calibration sample demonstrates that Attention Selectivity Index effectively captures the intrinsic scale-dependent attention Sitaviors-disperrse attention for drafters and concentrated attention for teiners-withhee requiring extensive statistical sampling. These findings validate that ScaleKY's layer categorization is both theoretically sound and practically robust.
FID Score Consistency Across Sample Sizes
2.58
Nean 2.53
2.56
2.53
2.53
2.53
2.53
2.53
2.53
2.53
2.53
2.53
FID
2.50
2.48
16
64
80
96
112
128
Samples
Figure 8: FID score consistency across calibration set sizes. ScaleKV maintains stable performance (FID 2.53, 0.0) from I to 128 calibration samples, demonstrating the robustness of our drafter/refiner identification method.
B Attention Map Visualization
Figures 9 and I0 present representative attention maps from transformer layers identified as drafters and retiners at scale 7, providing empirical validation for our layer categorization framework. The drafter layer (Block 23) exhibits distinctly dispersed attention patterns across multiple attention heads, with weights distributed broadly across the spatial dimensions capture global contextual information from preceding scales. This dispersed mechanism enables comprehensive integration of hierarchical features, justifying the larger cache allocation for such layers. contrast, the refiner layer (Block 29) demonstrates highly concentrated attention patterns, with attention heads focusing predominantly on localized spatial regions within the current token map. These contrasting attention behaviors provide strong empirical evidence for our differentiated cache management strategy: drafters require
21

---

substantial cache capacity to maintain broad contextual access while refiners can operate effectively with significantly reduced cache allocation due to their localized processing nature.
C
Additional Qualitative Results
Figures and 12 present comprehensive galleries of images generated by ScaleK ScaleKV-comprresee Infinity-8B 8B and Infinity-2B models, respectively, demonstrating the practical effectiveness of our compression framework across diverse visual content. These compressed models operate with merely 10% of the original KV cache memory requirement, yet maintain exceptional generation quality across various image categories including natural scenes, objects, portraits, and artistic compositions. These results demonstrate that ScaleKV V achieves substantial memory reduction without compromising the generative capabilities of VAR models, making high-qualic, image synthesis feasible in memory- constrained deployment scenarios.
D Limitations
In this analysis, we critically examine the constraints of our methodology.
First, while ScaleKV demonstrates robust compression performance across models of varying capacities, evaluation larger VAR models would provide additional insights into method's scalability. Due to the limited availability of large-scale models, our evaluation was restricted to Infinity-8B, currently the largest available VAR model. Testing models with greater capacity, such as those with 20B parameters, would enable more comprehensive assessment of ScaleK V's scalability. Second, ScaleKV functions post-training KV cache compression solution that relies on pre-trained VAR models and mirrors the original model's outputs. Therefore, if the baseline quality of the original VAR models is unsatisfactory, achieving high-quality results with our method could be challenging.
E Societal impacts
This work introduces new KV cache compression framework for VAR models that addresses critical memory bottlenecks in high-resolution image generation. By reducing memory requirements to 10% of the original capacity while maintaining generation quality, our method enhances the accessibility of advanced image synthesis technologies and carries several important societal implications. First, democratizes access to high-quality image generation by enabling deployment on consumer-gre gradd hardware and edge devices, thereby benefiting creative industries and educational institutions that previously lacked the computational resources for such applications. Second, the reduced memory footprint results in lower energy consumption during inference, contributing to Al deployment practices. Third, by enabling ultra-high resolution generation at scales up to 4K, framework creates opportunities for professional content creation, medical imaging, and scientific visualization applications.
sustainable
22

---

0.5
0.30
0.200
0.20
0.25
0.4
0.175
0.150
0.20
0.3
0.125
0.15
0.100
0.10
0.2
-010
0.1
0.050
0.05
0.05
0.025
Scale7
Scale?_ Block23 Head1
Scale7_Blocka) Block33 Head2
Scale7 Block23 Head3
0.35
0.25
0.175
0.5
0.30
0.150
0.20
0.125
0.15
0.100
0.3
0.2
0.050
0.1
0.025
Scale? Block23 Headd
Scale7
Scale7
Scale? Head7
0.1
0.5
0.30
0.4
0.3
0.25
0.3
0.3
0.2
0.15
0.2
0.2
0.10
0.1
Scale7 Heed8
Scale7 BBlcc23__Bkk3333 Head9
Scale?_Blann Black23kk333 Head10
Scale7 Block23 Heed11
0.40
0,30
0.30
0.35
0.30
0.25
0.30
0.25
0.20
0.20
0.15
0.15
0.15
0.10
0.10
0.10
0.05
0.05
0.05
0.05
Scale7 Head12
Scale7 Block23 Head13
Scale7 Block23 Head14
Scale7 Heed15
Figure 9: Visualization of Drafter Layer Attention Maps.
23

---

0.8
0:35
0.8
0.4
0.30
0.6
0.25
0.6
0.3
0.20
0.4
-0.4
0.2
0.3
0.10
0.1
0.2
0.05
0.0
Scale7 Headd
Scale7 Block29 Head1
Scale7 Blockllo_oaaaaee Head2
Scale7 Block29 Head3
0.35
0.14
0.8
0.8
0.30
0.12
0.7
0.25
0.10
0.6
0.6
0.20
0.08
0.5
0.4
0.15
0.08
0.3
0.10
0.04
0.2
0.2
0.05
0.1
0.1
Scale7 Block29 Head4
Scale7 Block29 Head5
Scale7 Block29 Head6
Scale7 Block29 Heed7
0.35
0.7
0.5
0.30
0.5
0.6
0.4
0.25
0.4
0.4
0.3
0.3
0.2
0.2
0.2
0.1
0.05
Scale7 Block29 Head8
Scale7_ Scale7_Block29 Head9
Scale7_ Block29
Scale7 Headl1
0.8
0.8
0.7
0.4
0.6
0.3
0.4
0.4
0.4
0.3
0.3
0.2
0.2
0.2
0.2
0.1
Scale? Block29 Head12
Scale7, Block29 Head13
Scale7, Block29 Headid
Scale7 Block29 Head15
Figure 10: Visualization of Refiner Layer Attention Maps.
24

---

Choigunee
Figure 11: Generated Images from ScaleKV.Compressed Infinity-8B.
25

---

Figure 12: Generated Images from ScaleK VcolesV-Compressee Infinity-2B.
26


Reviews:
[{"review_id": "9Tk6XDf29t", "reviewer": "Reviewer_74F6", "summary": "The paper introduces ScaleKV, a novel KV cache compression framework for Visual Autoregressive (VAR) modeling, addressing the exponential memory growth issue in multi-scale inference. Key insights reveal varying cache demands across transformer layers and distinct attention patterns at different scales, leading to the classification of layers into \"Drafters\" (requiring large cache for global context) and \"Refiners\" (needing minimal cache for local detail processing).\n\nScaleKV implements scale-aware layer budget allocation, using the Attention Selectivity Index (ASI) to distinguish layer types. Refiner budgets decrease linearly with scale, reallocating saved memory to Drafters. A token selection strategy preserves critical KV states based on attention importance, ensuring spatial coverage with minimal overhead.\n\nExperiments on Infinity-2B/8B models show ScaleKV reduces KV cache memory to 10% (e.g., 85GB to 8.5GB for Infinity-8B) with negligible quality loss (GenEval score remains 0.79, DPG drops marginally from 86.61 to 86.49). It outperforms baselines in FID, LPIPS, and PSNR across memory budgets, achieving up to 1.25× inference speedup.", "questions": "refer to weakness", "limitations": "none", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n1/ After carefully reading the paper, I find the proposed ScaleKV method an interesting training-free acceleration work, and the paper is clearly written. The paper identifies several obvious issues in VAR, among which the token sequence and KV cache become heavy at high resolutions and subsequent scales, which is an urgent problem to solve. I have seen many related works optimizing this problem, indicating the importance of this direction. Based on this, the work discovers that different scales exhibit distinct attention patterns. Through this interesting phenomenon, the authors propose a scale-aware KV cache to optimize FLOPs. The overall motivation and experiments are reasonable.\n\n2/ The experimental part is also straightforward. The authors conduct sufficient experiments on GenEval and DPG benchmarks based on the VAR T2I works Infinity 2B & 8B. The experimental results are sufficient and detailed. The authors also compare with classic methods such as Sliding Window, Streaming LLM, and PyramidKV. From the indicators, this optimization basically causes no performance drop.\n\nWeakness:\n\n1/ The Attention Selectivity Index (ASI) requires minimal calibration data (10 prompts) to distinguish Drafters and Refiners. However, the method’s sensitivity to calibration prompt diversity and content is not fully explored. For example, if prompts are biased toward specific visual domains, the layer classification might not generalize to broader scenarios, potentially affecting cache allocation accuracy. Additionally, I am curious whether the experimental conclusions are consistent with different amounts of calibration data. ScaleKV only tested on 10 prompts to determine the drafters and refiners—are there more conclusions supporting this?\n\n2/ Regarding actual running speed rather than computational load or GPU usage, I noticed in Fig7(b) that the latency for 1024×1024 resolution is only 52ms vs. 71ms compared to full cache. Are there any optimization potentials or observations here? What other drawbacks does the current KV cache method have?", "was_revised": true, "final_justification": "The author has basically solved my problem, and I will upgrade my rating to \"accept\".", "rebuttal": "We sincerely appreciate the valuable feedback and constructive suggestions. Thanks so much for taking time and effort to review our paper.\n\n> **Q1:** The Attention Selectivity Index (ASI) requires minimal calibration data (10 prompts) to distinguish Drafters and Refiners. However, the method's sensitivity to calibration prompt diversity and content is not fully explored. For example, if prompts are biased toward specific visual domains, the layer classification might not generalize to broader scenarios, potentially affecting cache allocation accuracy.\n\n**A1:** Thanks for the valuable feedback. We evaluated ASI robustness across three distinct visual domains using 128 calibration prompts each from HuggingFace datasets: flickr30k_captions_simCSE (human faces), architecture_house_building_prompts_SDXL (architecture details) and nature-dataset (natural landscapes). Table 1 shows consistent performance across all domains, confirming that drafter-refiner distinctions reflect intrinsic VAR model characteristics rather than domain-specific patterns.\n\n| Dataset | FID ↓ | LPIPS ↓ | PSNR ↑ |\n|:---|:---|:---|:---|\n| Human Face | 2.58 | 0.11 | 22.77 |\n| Architecture | 2.57 | 0.11 | 22.81 |\n| Landscape | 2.57 | 0.11 | 22.79 |\n\n*Table 1: Ablation of calibration data diversity and content*\n\n> **Q2:** Additionally, I am curious whether the experimental conclusions are consistent with different amounts of calibration data. ScaleKV only tested on 10 prompts to determine the drafters and refiners—are there more conclusions supporting this?\n\n**A2:** Thanks for the valuable question. We tested calibration sizes from 1 to 128 prompts using GPT-4o generated data. Table 2 demonstrates stable performance regardless of calibration size, validating that minimal data suffices to identify layer roles due to fundamental attention pattern differences and cache requirements.\n\n| Size | FID ↓ | LPIPS ↓ | PSNR ↑ |\n|:---|:---|:---|:---|\n| 1 | 2.58 | 0.11 | 22.78 |\n| 4 | 2.57 | 0.11 | 22.79 |\n| 16 | 2.57 | 0.11 | 22.81 |\n| 64 | 2.57 | 0.11 | 22.81 |\n| 128 | 2.57 | 0.11 | 22.81 |\n\n*Table 2: Ablation of calibration data size*\n\n> **Q3:** Regarding actual running speed rather than computational load or GPU usage, I noticed in Fig7(b) that the latency for 1024×1024 resolution is only 52ms vs. 71ms compared to full cache. Are there any optimization potentials or observations here?\n\n**A3:** Thanks for the valuable observation. ScaleKV prioritizes memory efficiency over raw speed, as VAR's main bottleneck is exponential KV cache growth at high resolutions. VAR is already significantly faster than diffusion transformers or traditional AR models of comparable size. Nevertheless, our compression method can be integrated with FlashAttention [1] to further optimize inference speed, achieving combined 2.66× speedup (77ms → 29ms) on Infinity-8B with NVIDIA A100-SXM4-80GB.\n\n| Method | Budget | Latency (ms) |\n|:---|:---|:---|\n| Full KV | 100% | 77 |\n| ScaleKV | 10% | 46 |\n| ScaleKV + FlashAttention | 10% | 29 |\n\n*Table 3: Inference latency of Infinity-8B*\n\n[1] FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness\n\n> **Q4:** What other drawbacks does the current KV cache method have?\n\n**A4:** Thanks for raising this important concern. As a post-training compression framework, ScaleKV's output quality is fundamentally bounded by the original VAR model's capabilities. Therefore, if the baseline quality of the original VAR models is unsatisfactory, achieving high-quality results with our method could be challenging."}, {"review_id": "vRRMo6QRXy", "reviewer": "Reviewer_VBDw", "summary": "The paper introduces ScaleKV, a novel KV cache compression framework for Visual Autoregressive (VAR) models, addressing memory bottlenecks arising from exponential KV cache growth during inference. Based on the attention patterns, the proposed method categorizes transformer layers into two functional groups (Drafters and Refiners). Drafters exhibit dispersed attention across multiple scales, requiring substantial cache storage, whereas Refiners have localized attention, requiring minimal storage. The framework implements an adaptive, scale-aware budget allocation strategy, reducing KV cache memory usage to approximately 10% of the original size while maintaining image quality. Experimental validation on Infinity-2B and Infinity-8B VAR models using standard benchmarks (MS-COCO, GenEval, DPG) shows ScaleKV significantly outperforms other state-of-the-art cache compression methods.", "questions": "## Questions\n1. Architecture variations: The paper's evaluation focuses exclusively on Infinity models. How does ScaleKV perform on VAR models with architectural variations? Specifically, I'm curious about models using cross-attention mechanisms between different scales rather than just self-attention within scales, as these might exhibit fundamentally different attention patterns. \n2. Integration with other optimizations: Can ScaleKV be effectively combined with quantization methods (e.g., LiteVAR [1])? Cache compression and reduced precision can either help each other or get in each other’s way.\n\n## Minor Issues:\n1. Citation [51] [52] repeats. \n2. The number of decimal places in Table 2 are not uniform. Some numbers have 2 decimal places and some numbers have 3 decimal places.\n3. Typo in a reference title. There is a missing space in ‘recipefor’ in the title of [22].\n4. In Eq. (2), the variable name ASI should use mathrm or text to distinguish with the math symbols.\n5. \\citet and \\citep are not properly used. For example, in Line 163, it should not directly refer to the reference [30]. There should be a noun before it.\n6. It is better to have more explanation in the caption of Figure 2 and Figure 3. Otherwise, the information the figures are intended to convey are unclear.\n\n[1] Xie, Rui, et al. \"LiteVAR: Compressing Visual Autoregressive Modelling with Efficient Attention and Quantization.\" arXiv preprint 2024.", "limitations": "yes", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "## Strengths:\n\n1. The visualization is clear and understandable. It is easy for readers to grasp the core idea without diving into formulas. The progressive visualization from Scale 4 to Scale 5 demonstrates how refiners become increasingly concentrated at higher scales, supporting the core hypothesis.\n2. The experiments are solid and comprehensive. The paper evaluates the proposed method on multiple popular benchmarks, combining quantitative metrics with side-by-side images, which lends credibility to the claims. Two model scales (Infinity-2B and 8B) are both put through full ablations, comparing against static quotas, random dropping, and learned baselines, which evidences that the authors did extensive groundwork rather than cherry-pick results.\n3. It achieves 10x reduction in KV cache memory requirements. Cutting the 85 GB cache of an 8-B model down to 8.5 GB turns “needs multi-GPU offload” into “fits on one high-end card”. Besides raw memory, smaller caches can reduce PCIe traffic and hidden latency spikes, so throughput may rise in practice.\n\n\n## Weaknesses:\n1. Lack of discussion of generalization of the proposed method: While the paper thoroughly evaluates on Infinity models, it provides insufficient discussion of transferability. The evaluation does not include VAR variants with different attention mechanisms. It remains unclear whether the drafter/refiner would hold for architectures with different layer configurations or attention head designs. Additionally, the paper neither analyzes if the ASI metric (Eq. 2) requires adjustment for models with varying attention mechanisms nor discusses its adaptability to future VAR architectures with novel designs.\n2. Evaluation on larger models: As acknowledged in Appendix D, the evaluation covers models up to 8B parameters, with Infinity-8B as the largest tested. Thus, empirical validation is missing for truly large-scale (e.g., 20B) models, where attention patterns and drafter/refiner ratios might differ. Additionally, the linear scaling assumption for memory savings remains unverified at larger scales.\n3. Trade-off analysis: While Figure 7 (a) provides a trade-off curve showing FID scores across five compression ratios (1%, 4%, 10%, 16%, 20%), the analysis could be more comprehensive. The paper focuses primarily on three ratios (4%, 10%, 20%) in most experiments and does not provide guidance on choosing appropriate compression levels for specific use cases. There is no principled method offered for practitioners to determine optimal compression ratios given their quality constraints. Besides, the paper lacks analysis of which image types or generation tasks are most sensitive to compression. For example, whether architectural details, human faces, or natural landscapes show different degradation patterns.", "was_revised": true, "final_justification": "The authors have fully addressed my concerns, and I believe my concerns have been adequately addressed. I am satisfied with the answers and will maintain my positive score to reflect my support for acceptance.", "rebuttal": "Thank you for your valuable feedback on our work. Your constructive comments on our work are invaluable.\n\n> **Q1:** Lack of discussion of generalization of the proposed method: While the paper thoroughly evaluates on Infinity models, it provides insufficient discussion of transferability. The evaluation does not include VAR variants with different attention mechanisms. It remains unclear whether the drafter/refiner would hold for architectures with different layer configurations or attention head designs. \n\n**A1:** Thank you for this valuable feedback. Our method generalizes beyond VAR architectures. We validate this by applying ScaleKV to LlamaGen [1], an autoregressive image model with fundamentally different attention mechanisms. Table 1 shows consistent performance with minimal degradation while achieving significant memory reduction. We also tested across different model scales (Infinity-2B/8B), confirming robustness across layer configurations in the paper.\n\n| Method | Budget | Global | Entity | Attribute | Relation | Other | Overall ↑ |\n|:---|:---|:---|:---|:---|:---|:---|:---|\n| Full KV | 100% | 77.85 | 76.61 | 76.12 | 79.52 | 78.23 | 65.00 |\n| ScaleKV | 50% | 77.50 | 75.50 | 73.91 | 77.11 | 79.35 | 62.50 |\n\n*Table 1: DPG score of ScaleKV-compressed LlamaGen model*\n\n[1] Autoregressive model beats diffusion: Llama for scalable image generation\n\n> **Q2:** Additionally, the paper neither analyzes if the ASI metric (Eq. 2) requires adjustment for models with varying attention mechanisms nor discusses its adaptability to future VAR architectures with novel designs.\n\n**A2:** We appreciate this important concern. For future VAR architectures that use Grouped-Query Attention (GQA) for efficient computation, ASI should be adjusted to measure attention diversity between entire query groups rather than between individual heads. This involves shifting the core unit of analysis from the head to the group level, as heads within a group are no longer independent.\n\n> **Q3:** Evaluation on larger models: As acknowledged in Appendix D, the evaluation covers models up to 8B parameters, with Infinity-8B as the largest tested. Thus, empirical validation is missing for truly large-scale (e.g., 20B) models, where attention patterns and drafter/refiner ratios might differ. Additionally, the linear scaling assumption for memory savings remains unverified at larger scales.\n\n**A3:** Thanks for the valuable comment. We tested our method on Infinity-8B, currently the largest available VAR model. Results demonstrate improved scaling behavior—performance increases from Infinity-2B to 8B (FID: 2.53→2.12 at 10% compression), suggesting larger models benefit more from ScaleKV due to increased attention redundancy. The linear scaling assumption holds because memory reduction is directly proportional to pruned KV cache tokens. We will evaluate on larger models when available.\n\n> **Q4:** Trade-off analysis: While Figure 7 (a) provides a trade-off curve showing FID scores across five compression ratios (1%, 4%, 10%, 16%, 20%), the analysis could be more comprehensive. The paper focuses primarily on three ratios (4%, 10%, 20%) in most experiments and does not provide guidance on choosing appropriate compression levels for specific use cases. There is no principled method offered for practitioners to determine optimal compression ratios given their quality constraints.\n\n**A4:** Thank you for your constructive suggestion. We provide comprehensive trade-off analysis with clear guidance for practitioners. Table 2 demonstrates ScaleKV's effectiveness across five compression ratios on Infinity-2B, showing a clear quality-efficiency trade-off. Extreme compression levels (1-4%) achieve dramatic memory reduction suitable for resource-constrained environments like mobile devices, while moderate compression (10-16%) maintains high visual quality for standard deployments. Conservative compression (20%) preserves exceptional quality for professional applications requiring high fidelity.\n\n| Budget | KV Cache | FID ↓ | Use Case |\n|:---|:---|:---|:---|\n| 1% | 390MB | 5.91 | Mobile/embedded devices |\n| 4% | 1590MB | 3.51 | Edge computing |\n| 10% | 3900MB | 2.53 | Desktop GPUs |\n| 16% | 6200MB | 2.14 | Workstations |\n| 20% | 7800MB | 1.82 | Cloud deployment |\n\n*Table 2: Quality & efficiency trade-off (Infinity-2B)*\n\n> **Q5:** Besides, the paper lacks analysis of which image types or generation tasks are most sensitive to compression. For example, whether architectural details, human faces, or natural landscapes show different degradation patterns.\n\n**A5:** Thank you for raising this important point. We evaluated ScaleKV across three distinct visual domains using 5,000 prompts each from HuggingFace datasets: flickr30k_captions_simCSE (human faces), architecture_house_building_prompts_SDXL (architectural details) and nature-dataset (natural landscapes). Table 3-5 demonstrate consistent performance across all domains, confirming ScaleKV's robustness for diverse visual content.\n\n| Method | Budget | FID ↓ | LPIPS ↓ | PSNR ↑ |\n|:---|:---|:---|:---|:---|\n| Full KV | 100% | - | - | - |\n| ScaleKV | 10% | 2.02 | 0.08 | 27.41 |\n\n*Table 3: Output consistency on human faces*\n\n| Method | Budget | FID ↓ | LPIPS ↓ | PSNR ↑ |\n|:---|:---|:---|:---|:---|\n| Full KV | 100% | - | - | - |\n| ScaleKV | 10% | 2.34 | 0.18 | 19.80 |\n\n*Table 4: Output consistency on architecture details*\n\n| Method | Budget | FID ↓ | LPIPS ↓ | PSNR ↑ |\n|:---|:---|:---|:---|:---|\n| Full KV | 100% | - | - | - |\n| ScaleKV | 10% | 2.72 | 0.15 | 22.52 |\n\n*Table 5: Output consistency on nature landscapes*\n\n> **Q6:** Architecture variations: The paper's evaluation focuses exclusively on Infinity models. How does ScaleKV perform on VAR models with architectural variations? Specifically, I'm curious about models using cross-attention mechanisms between different scales rather than just self-attention within scales, as these might exhibit fundamentally different attention patterns.\n\n**A6:** Thank you for the question. VAR models employ complex attention mechanisms beyond simple self-attention: causal cross-attention between current scale tokens and previous prefix tokens in KV cache, combined with bidirectional attention within current scale tokens. Our successful evaluation on both VAR and LlamaGen with different mechanisms (discussed in A1 and Table 1) demonstrates broad applicability across diverse attention patterns.\n\n> **Q7:** Integration with other optimizations: Can ScaleKV be effectively combined with quantization methods (e.g., LiteVAR [1])? Cache compression and reduced precision can either help each other or get in each other’s way.\n\n**A7:** Thank you for the valuable suggestion. ScaleKV is designed to be complementary to other optimization techniques. Both ScaleKV and LiteVAR can optimize inference efficiency with negligible quality degradation, so combining them could further enhance overall efficiency. However, since LiteVAR has not been open-sourced yet, we could not test this integration experimentally.\n\nBeyond quantization, ScaleKV could be integrated effectively with FlashAttention [1] to further optimize computational efficiency. We report inference latency of Infinity-8B measured on a single NVIDIA A100-SXM4-80GB GPU in Table 6, showing that the combined approach achieves 2.66× speedup while maintaining the same memory compression benefits. \n\n| Method | Budget | Latency (ms) |\n|:---|:---|:---|\n| Full KV | 100% | 77 |\n| ScaleKV | 10% | 46 |\n| ScaleKV + FlashAttention | 10% | 29 |\n\n*Table 6: Inference latency of Infinity-8B*\n\n[1] FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness\n\n> **Q8:** Minor Issues\n\n**A8:** Thank you for the careful review. We will address all minor issues including typos, formatting inconsistencies, and notation clarifications in the revised manuscript."}, {"review_id": "IV9Npti55o", "reviewer": "Reviewer_KGmY", "summary": "The paper highlights that in Visual Autoregressive (VAR) models, the KV cache experiences exponential growth during inference due to the multi-scale generation approach. To address this problem, the paper introduces a KV cache allocation method called ScaleKV, which classifies Transformer layers into \"Drafters\" and \"Refiners\". Experimental results on the Infinity-8B and 2B models show that this KV cache compression effectively resolves memory pressure while maintaining accuracy.", "questions": "1. Why not perform the drafter/refiner identification on a per-sample basis? I am concerned about the method's adaptability to other models. Also, when the calibration set size varies, do other metrics besides FID also remain consistent?\n2. What is the dominant component of the Attention Selectivity Index (ASI)—the 'current' attention ratio or the 'history' Top-K ratio, and how are they balanced? How is the Top-K' parameter selected, and what is the basis for choosing the observation window size (mentioned as typically 16 tokens)?\n3. The baselines for comparison might not be recent enough; could you compare against more current cross-layer KV allocation methods? I'm also wondering whether a more granular, per-layer management strategy would be better than the current simple distinction between 'drafters' and 'refiners'.\n4. I would like to know the specific details of the inference latency measurements in Figure 7b. Can ScaleKV be seamlessly integrated with standard optimizations like Flash Attention? Additionally, what was the image resolution used for the main results presented in the tables?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strengths\n\n* The paper focused on a meaningful problem of reducing the exponential KV cache growth of a popular generation paradigm, Visual Autoregressive (VAR) models. I like the problem and the scope.\n* The proposed Drafter/Refiner is interesting and novel.\n* The experimental results are promising and improvements are significant (if reproducible). I appreciate the authors' attachment of code in the supplementary materials.\n\n### Weaknesses\n\n* How can this method generalize to other datasets? Currently, the method's static, binary classification of layers has only been evaluated on the Infinity model family and requires further validation for its generalization across diverse inputs and other model architectures. I would like to see some discussions or empirical results. \n* It is still unclear about selecting core hyperparameters, such as the K' in the ASI formula and the observation window size. Some key experimental setup details should be complete.\n* The paper did not discuss the method's integration and compatibility with critical performance optimizations, such as Flash Attention.", "was_revised": true, "final_justification": "I appreciate the authors' solid rebuttal, which has addressed most of my concerns. I have also read other reviews and the authors' responses to them. I agree with other reviewers on accepting the paper. Therefore, my final rating is an Accept. Congrats!", "rebuttal": "Thank you very much for your review of our manuscript and attachment. We appreciate your time and effort in evaluating our work.\n\n> **Q1:** How can this method generalize to other datasets? Currently, the method's static, binary classification of layers has only been evaluated on the Infinity model family and requires further validation for its generalization across diverse inputs and other model architectures. I would like to see some discussions or empirical results.\n\n**A1:** Thank you for this valuable feedback. We evaluated ASI robustness across three distinct visual domains using 128 calibration prompts each from HuggingFace datasets: flickr30k_captions_simCSE (human faces), architecture_house_building_prompts_SDXL (architecture details) and nature-dataset (natural landscapes). Results show consistent performance across domains, confirming that drafter-refiner distinctions reflect model characteristics rather than domain-specific patterns. We also demonstrate applicability to traditional AR architectures like LlamaGen (Table 5).\n\n| Dataset | FID ↓ | LPIPS ↓ | PSNR ↑ |\n|:---|:---|:---|:---|\n| Human Face | 2.58 | 0.11 | 22.77 |\n| Architecture | 2.57 | 0.11 | 22.81 |\n| Landscape | 2.57 | 0.11 | 22.79 |\n\n*Table 1: Cross-domain validation results*\n\n> **Q2:** It is still unclear about selecting core hyperparameters, such as the K' in the ASI formula and the observation window size. Some key experimental setup details should be complete.\n\n**A2:** We appreciate your suggestions on clarifying these details. Our method demonstrates robust performance across varying hyperparameters, requiring minimal tuning. Tables 2-3 show stable results across different Top-K values (1%-20%) and window sizes (4-32 tokens).\n\n| Top-K | FID ↓ | LPIPS ↓ | PSNR ↑ |\n|:---|:---|:---|:---|\n| 1% | 2.58 | 0.11 | 22.79 |\n| 5% | 2.58 | 0.11 | 22.79 |\n| 10% | 2.58 | 0.11 | 22.79 |\n| 20% | 2.59 | 0.11 | 22.77 |\n\n*Table 2: Top-K parameter ablation*\n\n| Window | FID ↓ | LPIPS ↓ | PSNR ↑ |\n|:---|:---|:---|:---|\n| 4 | 2.58 | 0.11 | 22.81 |\n| 8 | 2.61 | 0.11 | 22.78 |\n| 16 | 2.61 | 0.11 | 22.79 |\n| 32 | 2.58 | 0.11 | 22.79 |\n\n*Table 3: Observation window size ablation*\n\n> **Q3:** The paper did not discuss the method's integration and compatibility with critical performance optimizations, such as Flash Attention.\n\n**A3:** Thank you for pointing this out. ScaleKV is fully compatible with FlashAttention. We report the inference latency of Infinity-8B measured on a single NVIDIA A100-SXM4-80GB GPU. Table 4 shows combined optimization achieves 2.7× speedup while maintaining compression benefits.\n\n| Method | Budget | Latency (ms) |\n|:---|:---|:---|\n| Full KV | 100% | 77 |\n| ScaleKV | 10% | 46 |\n| ScaleKV + FlashAttention | 10% | 29 |\n\n*Table 4: Inference latency comparison (Infinity-8B)*\n\n> **Q4:** Why not perform the drafter/refiner identification on a per-sample basis? I am concerned about the method's adaptability to other models.\n\n**A4:** Thanks for raising this important concern. Our drafter/refiner identification process is conducted before model deployment. Performing calibration on a per-sample basis would require capturing full attention maps for each sample, which would eliminate the computational benefits of KV cache compression.\n\nOur compression method is also applicable to traditional AR image generation frameworks, including LlamaGen [1].\n\n| Method | Budget | Global | Entity | Attribute | Relation | Other | Overall ↑ |\n|:---|:---|:---|:---|:---|:---|:---|:---|\n| Full KV | 100% | 77.85 | 76.61 | 76.12 | 79.52 | 78.23 | 65.00 |\n| ScaleKV | 50% | 77.50 | 75.50 | 73.91 | 77.11 | 79.35 | 62.50 |\n\n*Table 5: DPG scores of ScaleKV-compressed LlamaGen model*\n\n[1] Autoregressive model beats diffusion: Llama for scalable image generation.\n\n> **Q5:** Also, when the calibration set size varies, do other metrics besides FID also remain consistent?\n\n**A5:** Thank you for this question. We tested calibration sizes from 1 to 128 prompts using GPT-4o generated data. Table 6 demonstrates stable performance regardless of calibration size, confirming that minimal data suffices due to fundamental attention pattern differences and cache requirements between layer types.\n\n| Size | FID ↓ | LPIPS ↓ | PSNR ↑ |\n|:---|:---|:---|:---|\n| 1 | 2.58 | 0.11 | 22.78 |\n| 4 | 2.57 | 0.11 | 22.79 |\n| 16 | 2.57 | 0.11 | 22.81 |\n| 64 | 2.57 | 0.11 | 22.81 |\n| 128 | 2.57 | 0.11 | 22.81 |\n\n*Table 6: Calibration data size ablation*\n\n> **Q6:** What is the dominant component of the Attention Selectivity Index (ASI)—the 'current' attention ratio or the 'history' Top-K ratio, and how are they balanced? How is the Top-K' parameter selected, and what is the basis for choosing the observation window size (mentioned as typically 16 tokens)?\n\n**A6:** Thanks for the valuable comment. The Attention Selectivity Index (ASI) multiplies the current attention ratio and history Top-K ratio, ensuring both components contribute equally to layer classification. The robustness experiments above demonstrate minimal parameter tuning requirements (see A2 and Tables 2-3). We chose 16 tokens as the default window size to ensure fair comparison with existing baselines, as this is the standard setting used in most KV cache compression methods.\n\n> **Q7:** The baselines for comparison might not be recent enough; could you compare against more current cross-layer KV allocation methods?\n\n**A7:** Thanks for the suggestion. We compared ScaleKV against two recent cross-layer KV cache allocation methods: PyramidKV [1] (employs pyramid-shaped budget allocation) and SimLayerKV [2] (attention-based selective cache allocation). ScaleKV significantly outperforms both methods across all metrics.\n\n| Method | FID ↓ | LPIPS ↓ | PSNR ↑ |\n|:---|:---|:---|:---|\n| PyramidKV | 4.52 | 0.16 | 20.92 |\n| SimLayerKV | 3.55 | 0.13 | 22.39 |\n| ScaleKV | 2.57 | 0.11 | 22.81 |\n\n*Table 7: Comparison with cross-layer KV cache allocation baselines*\n\n[1] PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling\n\n[2] SimLayerKV: A Simple Framework for Layer-Level KV Cache Reduction\n\n> **Q8:** I'm also wondering whether a more granular, per-layer management strategy would be better than the current simple distinction between 'drafters' and 'refiners'.\n\n**A8:** We appreciate this constructive feedback. We implemented a per-layer granular management strategy that allocates individual budgets based on the proposed Attention Selectivity Index (ASI). As shown in Table 8, granular allocation does not provide additional benefits over our current method, supporting our hypothesis that cache demands are similar within groups but different between drafter/refiner groups.\n\n| Method | FID ↓ | LPIPS ↓ | PSNR ↑ |\n|:---|:---|:---|:---|\n| Granular | 2.70 | 0.12 | 22.51 |\n| ScaleKV | 2.57 | 0.11 | 22.81 |\n\n*Table 8: Comparison with per-layer management strategy*\n\n> **Q9:** Additionally, what was the image resolution used for the main results presented in the tables?\n\n**A9:** Thanks for the question. All experiments use 1024×1024 resolution to validate ScaleKV's effectiveness for high-resolution generation, where memory bottlenecks are most critical."}, {"review_id": "pTl5RqTQ9C", "reviewer": "Reviewer_KeK9", "summary": "This paper addresses the KV cache compression problem in Visual Autoregressive (VAR) models, proposing ScaleKV, a framework that categorizes transformer layers into \"Drafters\" and \"Refiners\" for scale-aware cache budget allocation. It achieves a 10× reduction of KV cache from 85GB to 8.5GB for the Infinity-8B model with negligible quality degradation (GenEval score remains 0.79, DPG score drops marginally from 86.61 to 86.49). The technical approach is clear and experimentally validated, but its impact is limited by the niche nature of VAR models in the visual generation landscape.", "questions": "Please refer to the weakness part", "limitations": "yes", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths: \n1. Technical Insights：The framework leverages two critical insights: varying cache demands across transformer layers and scale-dependent attention patterns in VAR. \n2. The paper is well writen and the authors conduct rigorous experiment to verify the effectiveness.\n\n\nWeakness\n1. Niche Application Scope: VAR models (e.g., Infinity) are not the mainstream in text-to-image generation, which is currently dominated by diffusion models (e.g., SDXL, SD3, SANA, ...). The paper does not demonstrate whether ScaleKV’s core strategies (e.g., layer classification, scale-aware allocation) can generalize to non-VAR frameworks, limiting practical impact.", "was_revised": true, "final_justification": "Thanks for the authors' response. I am happy to Borderline accept this paper.", "rebuttal": "We're grateful for the feedback you've provided.\n\n> **Q1:** Niche Application Scope: VAR models (e.g., Infinity) are not the mainstream in text-to-image generation, which is currently dominated by diffusion models (e.g., SDXL, SD3, SANA, ...). The paper does not demonstrate whether ScaleKV’s core strategies (e.g., layer classification, scale-aware allocation) can generalize to non-VAR frameworks, limiting practical impact\n\n**A1:** Thank you for the valuable feedback. We acknowledge that diffusion models currently dominate text-to-image generation, but we believe ScaleKV addresses fundamental challenges that extend beyond VAR models:\n\n**Broader Applicability Beyond VAR:** Our method generalizes beyond VAR to traditional AR frameworks. We validate this by applying ScaleKV to LlamaGen [1], an image autoregressive model, demonstrating robust performance with minimal quality loss while achieving substantial memory savings:\n\n| Method | Budget | Global | Entity | Attribute | Relation | Other | Overall ↑ |\n|:---|:---|:---|:---|:---|:---|:---|:---|\n| Full KV | 100% | 77.85 | 76.61 | 76.12 | 79.52 | 78.23 | 65.00 |\n| ScaleKV | 50% | 77.50 | 75.50 | 73.91 | 77.11 | 79.35 | 62.50 |\n\n*Table 1: DPG score of ScaleKV-compressed LlamaGen model*\n\n**Strategic Focus on VAR:** VAR models are gaining remarkable attention due to their efficiency, scalability, and potential for unified vision understanding and generation. ScaleKV effectively solves the memory bottleneck that makes high-resolution modeling challenging, enabling these promising architectures to scale to ultra-high resolutions such as 4K.\n\n**Future Extensibility:** Our drafter/refiner classification and scale-aware allocation principles provide a methodology that could adapt to future architectures, including potential hybrid approaches combining visual autoregressive and diffusion techniques.\n\n[1] Autoregressive model beats diffusion: Llama for scalable image generation."}, {"review_id": "fq7fhWKM2n", "reviewer": "Reviewer_PZnV", "summary": "This paper focuses on improving the inference efficiency for Visual Autoregressive (VAR) modeling, proposing ScaleKV, a novel KV cache compression framework tailored for VAR architecture. Specifically, the proposed ScaleKV categorizes transformer layers into two different groups: drafters and refiners, designing different cache management for each group. Experiments validate its effiectiveness of reducing the required KV cache memory for VAR.", "questions": "Except for the weaknesses, the following questions are encouraged to answer:\n\n- Some works share very similar ideas of this paper for improving VAR's generation efficiency, for example [1]. I recommend the authors to discuss with these works, if considering they are concurent works.\n- As the rapaid development for AR image models, it is necessary to include more representative AR image models like [2][3].\n- Including the original un-conpressed VAR model's performance in Table 1 and Table 2 will help authors to a more comprehensive understanding. \n- Add the difference map between generation results of compressed and un-compressed models will be helpful, such as in Figure 1.\n- The identification of Drafter and Refiner Layers are decided by the value of Attention Selectivity Index, will there exist a hybrid layer arounding the identification threshold?\n\n[1] Head-Aware KV Cache Compression for Efficient Visual Autoregressive Modeling\n[2] Towards Accurate Image Coding: Improved Autoregressive Image Generation With Dynamic Vector Quantization\n[3] An Image is Worth 32 Tokens for Reconstruction and Generation", "limitations": "See weaknesses and questions.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- Good analysis on the attention pattern of VAR models;\n- Strong results on reducing the KV cache memory for VAR models;\n\nWeaknesses:\n- The proposed method is specially designed for VAR. Therefore, considering the rapid development of AR image generation, the effectiveness for future AR models maybe limited.\n- Too many hyper-parameters making the proposed method much complex and less easy to extend and generalize.", "was_revised": true, "final_justification": "I appreciate the authors’ responses. After reading the authors’ responses and other reviews, I’ve decided to keep my positive score. However, the result on LlamaGen shows that the proposed method exhibit notiable performance drop when generalize the other kinds of AR models (2.5 drop on the overall score). Though I agree the proposed method is valuable for the VAR models, as also pointed by other reviewers that its generalization remain a concern. Therefore I keep my original score, and the generalization limit it from higher scores.", "rebuttal": "We extend our gratitude for your insightful feedback and suggestions, which have helped us improve the clarity and completeness of our work.\n\n> **Q1:** The proposed method is specially designed for VAR. Therefore, considering the rapid development of AR image generation, the effectiveness for future AR models maybe limited.\n\n**A1:** Thank you for the question. Our method generalizes beyond VAR to AR architectures. We demonstrate this by evaluating ScaleKV on LlamaGen [1], an image autoregressive model, showing consistent performance with minimal degradation while achieving significant memory reduction:\n\n| Method | Budget | Global | Entity | Attribute | Relation | Other | Overall ↑ |\n|:---|:---|:---|:---|:---|:---|:---|:---|\n| Full KV | 100% | 77.85 | 76.61 | 76.12 | 79.52 | 78.23 | 65.00 |\n| ScaleKV | 50% | 77.50 | 75.50 | 73.91 | 77.11 | 79.35 | 62.50 |\n\n*Table 1: DPG score of ScaleKV-compressed LlamaGen model*\n\n[1] Autoregressive model beats diffusion: Llama for scalable image generation\n\n> **Q2:** Too many hyper-parameters making the proposed method much complex and less easy to extend and generalize.\n\n**A2:** Thank you for the valuable feedback. Our method has two tunable hyper-parameters: Top-K scores in drafter/refiner classification and observation window size in KV cache selection. Both show robust performance across wide ranges:\n\n| Top-K | FID ↓ | LPIPS ↓ | PSNR ↑ |\n|:---|:---|:---|:---|\n| 1% | 2.58 | 0.11 | 22.79 |\n| 5% | 2.58 | 0.11 | 22.79 |\n| 10% | 2.58 | 0.11 | 22.79 |\n| 20% | 2.59 | 0.11 | 22.77 |\n\n*Table 2: Ablation of Top-K score*\n\n| Window | FID ↓ | LPIPS ↓ | PSNR ↑ |\n|:---|:---|:---|:---|\n| 4 | 2.58 | 0.11 | 22.81 |\n| 8 | 2.61 | 0.11 | 22.78 |\n| 16 | 2.61 | 0.11 | 22.79 |\n| 32 | 2.58 | 0.11 | 22.79 |\n\n*Table 3: Ablation of observation window size*\n\nThis stability demonstrates good generalizability and practical usability.\n\n> **Q3:** Some works share very similar ideas of this paper for improving VAR's generation efficiency, for example [1]. I recommend the authors to discuss with these works, if considering they are concurrent works?\n\n**A3:** Thank you for raising this important point. While both methods optimize the KV cache for VAR models, HACK [1] allocates fixed budgets by classifying individual attention heads, whereas ScaleKV dynamically categorizes entire layers as \"drafters\" or \"refiners\" to adaptively manage memory at each generation scale. Since HACK hasn't been open-sourced yet, we directly report their DPG score from the original paper.\n\n| Method | Budget | Global | Relation | Overall ↑ |\n|:---|:---|:---|:---|:---|\n| Infinity(8B) | 100% | 89.51 | 93.08 | 86.61 |\n| HACK | 10% | - | - | 85.74 |\n| ScaleKV | 10% | 92.49 | 88.92 | 86.49 |\n\n*Table 4: Comparison of ScaleKV and HACK on DPG benchmark*\n\n[1] Head-Aware KV Cache Compression for Efficient Visual Autoregressive Modeling.\n\n> **Q4:** As the rapid development for AR image models, it is necessary to include more representative AR image models like [2][3].\n\n**A4:** We appreciate this constructive feedback. We have evaluated ScaleKV on LlamaGen in A1 and Table 1, demonstrating effectiveness on large-scale text-to-image AR models beyond VAR framework. This validates our method's broad applicability to the AR family.\n\n> **Q5:** Including the original uncompressed VAR model's performance in Table 1 and Table 2 will help authors to a more comprehensive understanding.\n\n**A5:** We appreciate this observation. We have included performance of the uncompressed models (Infinity-2B and 8B) in the paper's Table 2. Table 1 specifically measures consistency between compressed and original uncompressed models, so only compressed model metrics (FID, LPIPS and PSNR) are relevant there.\n\n> **Q6:** Add the difference map between generation results of compressed and uncompressed models will be helpful, such as in Figure 1.\n\n**A6:** Thank you for this helpful suggestion. We will include difference maps in the revision to provide visual insights into generation quality preservation.\n\n> **Q7:** The identification of Drafter and Refiner Layers are decided by the value of Attention Selectivity Index, will there exist a hybrid layer around the identification threshold?\n\n**A7:** Thank you for the valuable comment, this is indeed a question that needs further explanation. We use a strict threshold ensuring layers are classified as drafters only when cache requirements are definitively large. This conservative approach eliminates ambiguous hybrid cases, ensuring robust layer categorization."}]

Author rebuttal:

Meta-review: The paper presents ScaleKV, a scale-aware KV cache compression framework for visual autoregressive (VAR) models that reduces memory usage by up to 90% while maintaining generation quality. Its main strengths lie in the clear identification of drafter/refiner layer patterns, extensive experiments on Infinity-2B/8B models, and robustness analyses showing stable performance across calibration sizes, domains, and architectural variations. While reviewers raised concerns about generalization to non-VAR frameworks and practical impact beyond Infinity-style models, the rebuttal addressed these points with additional experiments and discussions, leading to consensus that the work merits acceptance.
Decision: Accept (poster)