SSD: Spatial-Semantic Head Decoupling for Efficient Autoregressive Image Generation

## Abstract

Abstract Autoregressive image generation models like Janus-Pro produce high-quality images, but at the significant cost of high memory and ever-growing computational demands due to the large number of visual tokens. While KV cache compression has been extensively studied in language modeling, it still remains largely unexplored for the image generation domain. In this work, we begin by identifying a distinct and prominent attention phenomenon, which we term spatial locality and emergent semantic sink. To leverage this key insight, we introduce a novel KV cache compression framework. Specifically, we compress the KV cache for all visual tokens by adaptively decoupling attention heads into two separate types: for spatial-locality heads, our method maintains a short recent token window; for semantic-sink heads, it strategically preserves a compact set of highly-attended tokens. Our extensive experiments demonstrate that the proposed method achieves a 5 × \times reduction in memory usage and a notable 6.6 × \times speedup in overall throughput with only minimal visual quality loss, thereby enabling highly efficient native autoregressive image generation on resource-constrained hardware.

## 1 Introduction

Recent advancements in autoregressive (AR) multimodal models (Tian et al., 2024; Sun et al., 2024; Chern et al., 2024; Wang et al., 2025; Ma et al., 2025), especially unified AR models (Hurst et al., 2024; Team, 2024; Wang et al., 2024; Chen et al., 2025) have revolutionized high-fidelity image generation from textual descriptions.
Models like Janus-Pro (Chen et al., 2025) demonstrate remarkable capabilities in producing photorealistic images by treating image generation as a sequence prediction problem within a unified decoder-only architecture.
This approach simplifies the generation pipeline by eliminating the need for separate encoders or decoders, while maintaining strong performance across diverse visual domains (Zhang et al., 2025; Mu et al., 2025).

However, this remarkable success comes at a significant computational cost (Fan et al., 2024; Xiong et al., 2024). The autoregressive generation of high-resolution images and large batches requires processing an enormous number of visual tokens, which often leads to substantial memory demands and prohibitively prolonged inference times, especially for long sequences (Liu et al., 2024a).

This linear scaling of memory usage with the number of tokens, which grows quadratically with resolution, poses a critical bottleneck and severely limits the practical deployment of these models, especially at large batch sizes. While KV cache compression has been extensively studied for language modeling tasks (Xiao et al., 2023; Zhang et al., 2023; Li et al., 2024), these techniques remain largely underexplored within the visual token generation landscape.

Language-oriented approaches typically exploit the sparse attention patterns and token redundancy found in text, while visual tokens exhibit fundamentally different characteristics—they maintain strong spatial relationships and display structured attention patterns that existing methods fail to capture.
Consequently, the visual token KV cache, which constitutes the majority of the total cache during image decoding, remains a primary uncompressed bottleneck.

In this work, we address this limitation by identifying a novel attention phenomenon unique to visual token generation: the coexistence of *spatial locality* and *semantic sink*.
Through careful analysis of attention patterns in native visual decoder-only autoregressive models, we observe that certain attention heads specialize in processing local spatial relationships (spatial-locality heads), while others focus on preserving globally significant semantic information (semantic-sink heads). This structural dichotomy presents a unique opportunity for efficient cache compression.

Leveraging this insight, we introduce a novel KV cache compression framework that dynamically differentiates between spatial-locality and semantic-sink attention heads.
For spatial-locality heads, we maintain a compact sliding window of recent tokens to preserve local structural information.
For semantic-sink heads, we preserve a minimal set of highly-attended tokens that serve as semantic anchors throughout the generation process.
This dual-strategy approach achieves substantial memory reduction without compromising image quality.

Our comprehensive experiments demonstrate that the proposed method reduces KV cache memory usage by 5$\times$ and improves decoding throughput by 6.6$\times$ at the batch size of 128, while maintaining negligible performance degradation on standard image generation benchmarks.
These advancements enable practical high-resolution image generation on hardware with limited memory resources.
The main contributions of this work are as follows:

- •
We identify and characterize a novel dual attention phenomenon in autoregressive image generation models, termed *spatial locality* and *semantic sink*, which reveals fundamental differences between visual and linguistic attention patterns.
- •
We propose a specialized KV cache compression framework that exploits the structural properties of visual attention by applying differentiated compression strategies to spatial-locality and semantic-sink attention heads.
- •
Extensive evaluation shows our method achieves 5$\times$ memory reduction and 6.6$\times$ speedup with minimal quality loss, advancing the practicality and accessibility of autoregressive image generation thereby paving the way for its wider adoption.

## 2 Related Work

Figure: Figure 1: Visualization of the average attention map and reshaped attention across different generation steps in Janus-Pro-1B over 100 prompts, randomly sampled from Geneval (Ghosh et al., 2023). Our observation reveals: (1) Attention in the vision modality is sparse. (2) There are two types of sparse attention: one termed spatial locality attention, the other termed semantic sink attention. See Appendix [A](https://arxiv.org/html/2510.18716v1#A1) for more details.
Refer to caption: x1.png

#### Autoregressive Visual Generation and Unified Models

Recent advancements in autoregressive modeling have increasingly extended the proven scaling laws of large language models (LLMs) to diverse visual generation domains (El-Nouby et al., 2024).
These approaches typically discretize images into visual tokens via a tokenizer, enabling sequential prediction using decoder-only transformers in a manner similar to text generation, thereby leveraging established architectures.

Pioneering works such as DALL·E (Ramesh et al., 2021), Parti (Yu et al., 2022) and LlamaGen (Sun et al., 2024) have demonstrated increasingly impressive capabilities in high-fidelity image synthesis and instruction following, with SimpleAR (Wang et al., 2025) further demonstrating leveraging post-training refinement through reinforcement learning.
A prominent research direction extends this paradigm by unifying multiple modalities within a single autoregressive framework.
Models such as Chameleon (Team, 2024), Emu3 (Wang et al., 2024), and Janus-Pro (Chen et al., 2025) adopt a purely decoder-only architecture to process interleaved multimodal sequences (e.g., image-text-text or image-text).
Unlike alternative strategies that employ diffusion models, continuous representations, or multi-stage cascaded systems, these increasingly popular native autoregressive unifiers emphasize end-to-end training on discretized token sequences, leading to stronger multimodal synergy and more coherent conditional generation.

Our work aligns with and extends this native autoregressive research line, with a specific focus on improving the inference efficiency image generation—a key challenge for real-world deployment of these unified models.

#### KV Cache Compression

Autoregressive decoding in transformer models is inherently memory-bound due to the linear expansion of the key-value (KV) cache during inference.
Prior research has explored various strategies, including eviction (Zhang et al., 2023; Li et al., 2024), merging (Zhang et al., 2024; Yao et al., 2025; Li et al., 2025), and quantization (Liu et al., 2024b; Dong et al., 2024) to mitigate this memory footprint.

Early methods, such as window attention (Beltagy et al., 2020), H2O (Zhang et al., 2023), and StreamingLLM (Xiao et al., 2023) primarily focused on decoding-stage cache compression in language models.
More recent approaches, including SnapKV (Li et al., 2024), Ada-KV (Feng et al., 2024), PyramidKV (Cai et al., 2024), and DuoAttention (Xiao et al., 2024), have targeted long-context scenarios, emphasizing improvements in prefilling efficiency for large language models (LLMs).
Furthermore, while existing methods leverage sparsity patterns inherent in linguistic data, their effectiveness in the visual domain remains underexplored, primarily due to the differing underlying mechanics and distinct token distribution.
Other related efforts, such as HACK (Qin et al., 2025) focus on next-scale prediction in visual generation models (Tian et al., 2024), representing a specialized approach to KV cache optimization in hierarchical autoregressive frameworks.

To our knowledge, we present the first novel eviction framework specifically designed to exploit these visual-specific structural properties, enabling efficient and high-fidelity native unified visual autoregressive generation, thereby paving the way for its broader adoption.

## 3 Methodology

This section presents our methodology for efficient KV cache compression in visual autoregressive generation.
We begin by formalizing the problem setup and reviewing fundamental concepts of visual autoregressive image generation with Classifier-Free Guidance (CFG), highlighting the computational bottleneck that motivates our work.

Subsequently, we introduce a key empirical observation: semantic information from textual prompts is preferentially injected into specific spatial regions—particularly the margin columns of the raster-scanned image token sequence.
This finding naturally leads to the identification of two distinct types of attention heads with specialized roles: *semantic heads* that capture global context and *spatial heads* that handle local dependencies.
Leveraging this structural dichotomy, we then present the SSD framework, which applies asymmetric compression policies tailored to each head type, yielding substantial memory and computational savings while preserving generation quality.

### 3.1 Preliminaries: Native Autoregressive Visual Generation

Native autoregressive models for visual synthesis, as exemplified by Janus-Pro (Chen et al., 2025), reconceptualize image generation as a sequential token prediction task analogous to text generation.
Unlike diffusion-based approaches that employ iterative denoising procedures, AR models generate images token-by-token following a raster scanning order.

Formally, given an input textual prompt tokenized into tokens $\mathbf{T}\in\mathbb{R}^{L}$, the model autoregressively generates a flattened sequence of visual tokens $\mathbf{Z}=(z_{1},z_{2},\dots,z_{N})$, where $N=h\times w$ corresponds to the target image resolution in token space (e.g., $N=576$ for a $24\times 24$ grid).
Each token $z_{t}$ is predicted conditional on all preceding visual tokens $z_{<t}$ and the text conditioning $\mathbf{T}$.
Finally, visual tokens $\mathbf{Z}$ are decoded by a visual decoder (Yu et al., 2021; Esser et al., 2021) to image $\mathbf{X}\in\mathbb{R}^{H\times W\times 3}$, where $p=H/h$ denotes the compression ratio.

A critical enhancement for improving output fidelity and prompt alignment is Classifier-Free Guidance (CFG), which operates through a dual-forward-pass mechanism at each decoding step:

- 1.
Conditional pass: Computes the probability distribution $\mathbf{p}_{t}^{c}=p(z_{t}|\mathbf{T},z_{<t})$ when conditioned on the text prompt.
- 2.
Unconditional pass: Computes the distribution $\mathbf{p}_{t}^{u}=p(z_{t}|z_{<t})$ without text conditioning.

The guided distribution is obtained by amplifying the difference between these distributions:

$$ $\mathbf{p}_{t}^{\text{cfg}}=\mathbf{p}_{t}^{c}+\gamma(\mathbf{p}_{t}^{c}-\mathbf{p}_{t}^{u}),\quad\gamma>1,$ (1) $$

where $\gamma$ modulates the guidance strength, effectively enhancing alignment with the textual prompt.
The autoregressive decoding process necessitates maintaining a growing Key-Value (KV) cache for all previous tokens to avoid redundant recomputation.

The memory and computational complexity of this cache scales linearly with sequence length in memory ($\mathcal{O}(N)$) but quadratically in attention computation ($\mathcal{O}(N^{2})$), creating a fundamental bottleneck for high-resolution image generation.
Our work directly addresses this limitation by exploiting the structured attention patterns that emerge in visual generation.

### 3.2 Margin Columns as Semantic Anchors

Figure: (a) Sparsity across heads
Refer to caption: x2.png

Our investigation reveals a crucial characteristic of AR visual generation: semantic information from the textual prompt $\mathbf{T}$ distributes non-uniformly across the image token sequence, with pronounced concentration in tokens corresponding to margin columns of the raster-scanned grid.
This spatial asymmetry creates natural semantic anchors that orchestrate global image coherence, which informs our compression strategy.

Semantic Concentration via KV Cache Analysis.
To directly quantify semantic injection patterns, we analyze the divergence between KV caches generated under CFG guidance branch versus non-CFG guidance native branch. Specifically, we compute the Mean Squared Error (MSE) per token position $j$:

$$ $\text{MSE}_{\text{token}}(j)=\|\mathbf{K}^{\text{cfg}}[:,j]-\mathbf{K}^{\text{native}}[:,j]\|^{2}_{2}+\|\mathbf{V}^{\text{cfg}}[:,j]-\mathbf{V}^{\text{native}}[:,j]\|^{2}_{2}.$ (2) $$

As illustrated in Figure [2](https://arxiv.org/html/2510.18716v1#S3.F2)(b), empirical analysis reveals that MSE peaks occur at periodic positions corresponding precisely to margin columns in the $24\times 24$ token grid.
These elevated MSE values indicate heightened semantic perturbation in these tokens, confirming their role as privileged conduits for prompt-derived concepts.

This emergent anchoring behavior parallels engineered approaches in models like Lumina-mGPT (Liu et al., 2024a), which explicitly augment margin tokens for adaptive resolution generation. In native AR models, however, this phenomenon arises organically under CFG, revealing an exploitable asymmetry: margin tokens aggregate global semantic information while interior tokens prioritize spatial relationships.

### 3.3 Spatial and Semantic Head Dichotomy and the SSD Framework

Guided by our empirical observations, we propose a systematic classification of attention heads into two functionally distinct archetypes:

Figure: Algorithm 1 SSD KV Cache Compression

- •
Semantic Heads: Characterized by dense, heavy-tailed attention distributions that preferentially attend to global anchors, particularly margin column tokens. These heads maintain overarching semantic coherence and ensure prompt alignment throughout the generation process.
- •
Spatial Heads: Exhibit sparse, localized attention patterns focused primarily on recent neighboring tokens. These heads process spatial relationships and refine local visual details through neighborhood interactions.

Spatial Sparsity Analysis. We quantify spatial sparsity patterns by analyzing attention distributions across different layers and heads. For each attention head at layer $l$ and head index $h$, we compute the sparsity metric averaged over multiple generation steps and prompts:

$$ $s_{l,h}=\frac{1}{P}\sum_{p=1}^{P}\frac{1}{T}\sum_{t=1}^{T}\frac{\sum_{i=0}^{t-1-w}a_{l,h,p,t}(i)}{\sum_{i=0}^{t-1}a_{l,h,p,t}(i)},$ $$

where $P$ is the number of prompts, $T$ is the max visual token length, $w=32$ is a recency window, and $a_{l,h,p,t}$ is the attention distribution for specific layer, head, prompt, and timestep.
This ratio measures the proportion of attention allocated to tokens beyond the immediate neighborhood, effectively distinguishing globally-attentive heads (high $s$) from locally-focused ones (low $s$).
As illustrated in Figure [2](https://arxiv.org/html/2510.18716v1#S3.F2)(a), Janus-Pro exhibits a structured pyramidal sparsity progression across layers. Lower layers, responsible for processing fundamental visual features (e.g., textures, contours), demonstrate broader attentional spans that establish structural frameworks.
In contrast, higher layers progressively focus on local neighborhoods, refining spatial details and fine-grained patterns.

We operationalize this classification using the sparsity metric $s_{i}$ computed for each head $i$ across diverse prompts and generation steps.
The bimodal distribution of $s_{i}$ values enables robust classification via a threshold $\tau$:

$$ $\text{HeadType}(i)=\begin{cases}\text{Semantic},&s_{i}>\tau\quad\text{(dense, global attention)}\\ \text{Spatial},&s_{i}\leq\tau\quad\text{(sparse, local attention)}\end{cases}$ (3) $$

This functional partitioning transcends mere descriptive categorization, enabling precisely tailored compression strategies that respect the distinct roles of different heads types—a significant advancement over uniform compression approaches derived from language model optimizations.

SSD Framework. The Spatial-Semantic (SSD) framework implements head-specific compression policies that leverage the identified functional specialization:

- •
Spatial Heads: Employ *sliding window compression*, retaining only the most recent $W$ tokens along with an initial sink token to anchor global context, inspired by StreamingLLM (Xiao et al., 2023). This strategy capitalizes on the strong locality bias exhibited by these heads while preserving essential long-range dependencies, safely discarding distant tokens with minimal impact on generation quality.
- •
Semantic Heads: Utilize *heavy-hitter retention*, preserving the top $M$ tokens ranked by accumulated attention scores across generation steps, augmented by a reserved recent window of $R$ tokens. This approach safeguards critical semantic anchors and maintains recency awareness while aggressively compressing less influential tokens.

This asymmetric approach achieves significant efficiency gains—up to 5$\times$ memory reduction and 6.6$\times$ decoding throughput acceleration—with minimal quality degradation. Algorithm [1](https://arxiv.org/html/2510.18716v1#alg1) provides the complete implementation.

## 4 Experimental Results

This section presents a comprehensive evaluation of the SSD framework, assessing its effectiveness in preserving multimodal generation quality under aggressive Key-Value (KV) cache compression. We detail the experimental setup, including datasets, comparison methods, and implementation details, followed by quantitative and qualitative results, efficiency metrics, ablation studies, and a discussion of broader implications and future directions.

### 4.1 Experiment Settings

Datasets.
We evaluate SSD on two well-established benchmarks designed to test multimodal generation quality and compositional reasoning under varying context complexities:
(a) GenEval (Ghosh et al., 2023) features 553 curated prompts assessing compositional reasoning, focusing on object attributes, spatial relationships, and complex understanding.
(b) DPG-Bench (Hu et al., 2024) includes 1,065 graph-structured prompts (avg. 84 tokens via CLIP tokenizer) testing prompt adherence and compositional understanding, originally for diffusion but adaptable to autoregressive contexts.
These benchmarks span a range of context lengths, enabling evaluation of compression artifacts across fidelity, compositional accuracy, and scalability.

Comparison Methods.
We compare SSD against two decoding-stage KV cache compression methods and an uncompressed full-cache baseline.
(a) StreamingLLM (Xiao et al., 2023) uses a sliding window with sentinel tokens, effective for streaming but with potential long-range dependency issues.
(b) H2O (Zhang et al., 2023) dynamically retains high-attention tokens via submodular maximization, improving throughput with minimal perplexity loss.
(c) Full Cache: The vanilla transformer decoder with unrestricted KV caching serves as the baseline for fidelity comparisons.

All baseline implementations are sourced from their official repositories and re-tuned for the Janus-Pro model to ensure fair comparisons.

Implementation Details. The SSD framework is implemented on the Janus-Pro models (1B and 7B parameters) (Chen et al., 2025), utilizing a $24\times 24$ token grid (576 tokens) to generate $384\times 384$ images.
For fair comparison across compression methods during quantitative evaluation, we standardize the sliding window size to $W=32$. Classifier-Free Guidance (CFG) is applied with a guidance scale of $\gamma=5.0$. Experiments are conducted on NVIDIA A100 GPUs (80GB), with batch sizes up to 256 to assess scalability and robustness.

### 4.2 Quantitative Evaluation

We demonstrate that combining window-based locality compression with attention-based semantic compression yields better generation quality than either method alone.
We evaluate our framework across two memory scenarios: a low setting (20% token budget) and a high setting (50% token budget).

As shown in Table [1](https://arxiv.org/html/2510.18716v1#S4.T1), SSD achieves better GenEval and DPG-Bench scores than H2O and StreamingLLM under 20% and 50% token budgets, comparable to the performance of the vanilla full-cache baseline. These results match our prior observations. Figure [3](https://arxiv.org/html/2510.18716v1#S4.F3) further illustrates that SSD maintains robust performance across compression ratios, with minimal degradation even at 20% cache size, aligning with our hypothesis that combining window-based and attention-based compression leverages both local and global dependencies effectively.

**Table 1: Performance on GenEval and DPG-Bench under 20% and 50% cache sizes, compared to Full Cache, StreamingLLM, and H2O. Our Method consistently achieves comparable performance to Full Cache with significantly reduced memory.**
| Method | Cache Size | GenEval | DPG |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Two Obj. | Counting | Color Attri. | Overall$\uparrow$ | Entity | Attribute | Relation | Overall$\uparrow$ |  |  |
| Janus-Pro-1B |  |  |  |  |  |  |  |  |  |
| Full | 100% | 0.82 | 0.51 | 0.56 | 0.73 | 89.06 | 89.18 | 89.73 | 82.98 |
| Streaming | 20% | 0.77 | 0.28 | 0.44 | 0.64 | 87.28 | 87.60 | 87.78 | 81.01 |
| H2O | 0.74 | 0.44 | 0.43 | 0.67 | 87.49 | 87.54 | 85.48 | 80.44 |  |
| Ours | 0.81 | 0.48 | 0.57 | 0.73 | 89.23 | 87.94 | 89.48 | 82.82 |  |
| Streaming | 50% | 0.80 | 0.48 | 0.55 | 0.72 | 89.75 | 88.67 | 89.92 | 83.17 |
| H2O | 0.81 | 0.49 | 0.56 | 0.67 | 88.96 | 88.47 | 86.58 | 82.28 |  |
| Ours | 0.81 | 0.52 | 0.58 | 0.74 | 88.54 | 89.31 | 88.90 | 83.21 |  |
| Janus-Pro-7B |  |  |  |  |  |  |  |  |  |
| Full | 100% | 0.89 | 0.59 | 0.66 | 0.80 | 89.35 | 90.24 | 90.45 | 84.77 |
| Streaming | 20% | 0.87 | 0.52 | 0.62 | 0.77 | 86.45 | 87.51 | 89.48 | 83.20 |
| H2O | 0.78 | 0.53 | 0.47 | 0.71 | 81.90 | 83.86 | 83.12 | 75.97 |  |
| Ours | 0.86 | 0.58 | 0.67 | 0.79 | 89.52 | 89.26 | 90.17 | 84.45 |  |
| Streaming | 50% | 0.87 | 0.59 | 0.63 | 0.79 | 88.70 | 88.89 | 92.22 | 84.83 |
| H2O | 0.86 | 0.60 | 0.62 | 0.79 | 87.37 | 87.73 | 89.47 | 83.36 |  |
| Ours | 0.89 | 0.60 | 0.65 | 0.80 | 88.92 | 89.51 | 89.19 | 84.72 |  |

Figure: Figure 3: Performance comparison of SSD, Full Cache, StreamingLLM, and H2O on (a,b) GenEval and (c,d) DPG-Bench across varying compression ratios for Janus-Pro-1B and Janus-Pro-7B. SSD achieves performance comparable to Full Cache while significantly reducing memory usage, outperforming both H2O and StreamingLLM on both benchmarks.
Refer to caption: x4.png

### 4.3 Quality Evaluation

To qualitatively assess the detailed impact of KV cache compression, we carefully visualize generated images from Janus-Pro under various compression ratios (20%, 50%) in Figure [4](https://arxiv.org/html/2510.18716v1#S4.F4).
The examples highlight the ability of SSD to consistently preserve fine details, spatial relationships, and semantic coherence even under aggressive compression.
These visuals conclusively corroborate the quantitative gains, demonstrating the robustness of SSD in real-world generation tasks.

Figure: Figure 4: Qualitative visualization of generated images from Janus-Pro-1B and Janus-Pro-7B under varying compression ratios (20% to 50% cache sizes) for SSD and Full Cache. SSD preserves fine details, spatial relationships, and semantic coherence in complex scene generation.
Refer to caption: x5.png

### 4.4 Efficiency Evaluation

We assess the effectiveness of SSD in reducing memory consumption and enhancing time efficiency during inference by analyzing peak memory usage and throughput across different batch sizes.

Peak Memory Usage.
As depicted in the left panel of Table [3](https://arxiv.org/html/2510.18716v1#S4.T3), SSD exhibits substantial memory savings capabilities, comparable to other KV cache eviction methods, both attention-based (e.g., H2O) and window-based (e.g., StreamingLLM).
All these approaches maintain a fixed-size KV cache. When compared to the full cache implementation, SSD achieves an impressive reduction in peak memory usage of up to 5$\times$ with batch size of 256.

Throughput.
As batch size increases, decoding latency for the full cache method grows significantly due to escalating computational demands and I/O latency bottlenecks.
In contrast, SSD maintains a relatively stable throughput by preserving a fixed amount of KV cache, resulting in significantly lower latency compared to the full cache, particularly for larger batch sizes.
It is noteworthy that SSD demonstrates remarkable efficiency, achieving over 6.6$\times$ speedup in throughput compared to the full cache approach when processing batches of 128.

### 4.5 Ablation Study

To further validate the key components of SSD, we conduct ablation experiments on the Janus-Pro-7B model. Our analysis focuses on two critical aspects: head selection strategy and threshold sensitivity.

**Table 2: Quantitatively comparison between randomly partition and our method.**
| Methods | Geneval | DPG-Bench |
| --- | --- | --- |
| Random | 0.75 | 81.23 |
| Ours | 0.79 | 84.65 |
| Baseline | 0.80 | 84.19 |

Threshold Sensitivity Analysis.
We investigate the impact of threshold selection $\tau$ on dividing the semantic KV cache.
By varying $\tau$ across (0, 0.45, 0.9) as shown in Figure [5](https://arxiv.org/html/2510.18716v1#S4.F5), we observe that higher thresholds progressively filter heads to those with more pronounced semantic concentration.
Specifically, higher threshold $\tau$ exhibits higher spiked MSE patterns at margin columns, confirming that global semantic information predominantly resides in heads whose sparsity $>\tau$, which validates our threshold-based approach for identifying semantic-critical heads.

Figure: Table 3: System performance scaling with batch size. SSD demonstrates superior efficiency compared to full cache, achieving up to 6.6$\times$ throughput improvement and 5$\times$ memory reduction, which highlights its practical advantage in real-world deployment scenarios.
Refer to caption: x6.png

Effectiveness of Head Selection Strategy.
We further evaluate SSD against random head assignment mimicking our asymmetric policy allocation but without semantic justification.
As shown in Table [2](https://arxiv.org/html/2510.18716v1#S4.T2), our sparsity-based dichotomy ($\tau=0.8$) achieves better GenEval and DPG-Bench evaluation scores, demonstrating that distinguishing semantic and spatial heads is crucial for effective asymmetric compression.
The performance gap underscores the importance of our head classification over heuristic alternatives.

### 4.6 Discussion and Future Work

We argue that our contributions establish that effective KV cache compression for AR image generation necessitates a hybrid approach combining window-based locality compression and attention-aware semantic retention.
This strategy adeptly addresses the spatial-semantic dichotomy inherent in visual token generation, outperforming language-derived methods that overlook visual-specific patterns (Zhang et al., 2023; Xiao et al., 2023).

Computational Overhead Mitigation.
The dynamic token identification process in the SSD framework introduces moderate computational and memory allocation overhead due to iterative attention score calculations. To address this, we propose a buffering mechanism that stores candidate tokens in a temporary cache, updated periodically to minimize redundant computations and resource usage. Preliminary experiments, as shown in Table [4](https://arxiv.org/html/2510.18716v1#S4.T4), demonstrate that this optimization reduces computational overhead by 50% while preserving generation quality, with negligible impact on GenEval and DPG-Bench metrics. Further details are available in Appendix [B](https://arxiv.org/html/2510.18716v1#A2).

**Table 4: Throughput and memory efficiency across batch sizes with 20% cache. SSD with buffer method delivers significant performance improvements—achieving up to 10.7$\times$ higher throughput than full cache—while reducing memory consumption by approximately 80%. Notably, these efficiency gains are achieved without substantial degradation in task performance, as evidenced by comparable scores on GenEval and DPG evaluation benchmarks.**
| Method | BS = 32 | BS = 64 | BS = 128 | GenEval | DPG |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Thr. | Mem. | Thr. | Mem. | Thr. | Mem. |  |  |  |
| Full Cache | 897.3 | 18.1 | 543.6 | 32.2 | 289.2 | 60.4 | 0.73 | 82.63 |
| SSD | 1253.0 (1.4$\times$) | 5.9 | 1658.4 (3.1$\times$) | 7.9 | 1911.7 (6.6$\times$) | 11.8 | 0.73 | 82.82 |
| SSD w/buffer | 1754.1 (2.0$\times$) | 5.8 | 2524.6 (4.6$\times$) | 7.7 | 3099.4 (10.7$\times$) | 11.6 | 0.72 | 82.53 |

Integration with Other Compression Techniques.
The text modeling field provides extensive research on prefill-phase compression, including layer-wise (Cai et al., 2024) and head-wise (Feng et al., 2024) allocation strategies. These methods, tailored to enhance the initial prompt processing stage, can be effectively integrated with the decoding-phase optimization of SSD. Future research could investigate hybrid frameworks that merge layer-wise or head-wise budget allocation compression with our adaptive decoding policies to enhance overall efficiency.

## 5 Conclusion

Unified autoregressive models for image generation face significant and increasingly critical memory challenges due to the exceptionally large number of visual tokens processed during inference. While KV cache eviction methods have greatly improved memory efficiency in text-based models, their application to visual autoregressive generation has been largely underexplored.

We address this gap by identifying distinct spatial and semantic attention patterns in visual AR models and proposing SSD, a novel compression framework that applies tailored policies to spatial and semantic attention heads. This approach achieves substantial memory reduction and inference speedup while preserving high-quality output. To our knowledge, SSD is the first comprehensive framework for KV cache compression in unified visual AR models, paving the way for efficient deployment of large-scale multimodal systems and advancing practical inference solutions.

## References

- Beltagy et al. (2020)
Iz Beltagy, Matthew E Peters, and Arman Cohan.
Longformer: The long-document transformer.
*arXiv preprint arXiv:2004.05150*, 2020.
- Cai et al. (2024)
Zefan Cai, Yichi Zhang, Bofei Gao, Yuliang Liu, Tianyu Liu, Keming Lu, Wayne Xiong, Yue Dong, Baobao Chang, Junjie Hu, et al.
Pyramidkv: Dynamic kv cache compression based on pyramidal information funneling.
*arXiv preprint arXiv:2406.02069*, 2024.
- Chen et al. (2025)
Xiaokang Chen, Zhiyu Wu, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, and Chong Ruan.
Janus-pro: Unified multimodal understanding and generation with data and model scaling.
*arXiv preprint arXiv:2501.17811*, 2025.
- Chern et al. (2024)
Ethan Chern, Jiadi Su, Yan Ma, and Pengfei Liu.
Anole: An open, autoregressive, native large multimodal models for interleaved image-text generation.
*arXiv preprint arXiv:2407.06135*, 2024.
- Dong et al. (2024)
Shichen Dong, Wen Cheng, Jiayu Qin, and Wei Wang.
Qaq: Quality adaptive quantization for llm kv cache.
*arXiv preprint arXiv:2403.04643*, 2024.
- El-Nouby et al. (2024)
Alaaeldin El-Nouby, Michal Klein, Shuangfei Zhai, Miguel Angel Bautista, Alexander Toshev, Vaishaal Shankar, Joshua M Susskind, and Armand Joulin.
Scalable pre-training of large autoregressive image models.
*arXiv preprint arXiv:2401.08541*, 2024.
- Esser et al. (2021)
Patrick Esser, Robin Rombach, and Bjorn Ommer.
Taming transformers for high-resolution image synthesis.
In *CVPR*, 2021.
- Fan et al. (2024)
Lijie Fan, Tianhong Li, Siyang Qin, Yuanzhen Li, Chen Sun, Michael Rubinstein, Deqing Sun, Kaiming He, and Yonglong Tian.
Fluid: Scaling autoregressive text-to-image generative models with continuous tokens.
*arXiv preprint arXiv:2410.13863*, 2024.
- Feng et al. (2024)
Yuan Feng, Junlin Lv, Yukun Cao, Xike Xie, and S Kevin Zhou.
Ada-kv: Optimizing kv cache eviction by adaptive budget allocation for efficient llm inference.
*arXiv preprint arXiv:2407.11550*, 2024.
- Ghosh et al. (2023)
Dhruba Ghosh, Hannaneh Hajishirzi, and Ludwig Schmidt.
Geneval: An object-focused framework for evaluating text-to-image alignment.
In *NeurIPS*, 2023.
- Hu et al. (2024)
Xiwei Hu, Rui Wang, Yixiao Fang, Bin Fu, Pei Cheng, and Gang Yu.
Ella: Equip diffusion models with llm for enhanced semantic alignment.
*arXiv preprint arXiv:2403.05135*, 2024.
- Hurst et al. (2024)
Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al.
Gpt-4o system card.
*arXiv preprint arXiv:2410.21276*, 2024.
- Li et al. (2025)
Siyuan Li, Luyuan Zhang, Zedong Wang, Juanxi Tian, Cheng Tan, Zicheng Liu, Chang Yu, Qingsong Xie, Haonan Lu, Haoqian Wang, and Zhen Lei.
Mergevq: A unified framework for visual generation and representation with disentangled token merging and quantization.
In *CVPR*, 2025.
- Li et al. (2024)
Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen.
Snapkv: Llm knows what you are looking for before generation.
In *NeurIPS*, 2024.
- Liu et al. (2024a)
Dongyang Liu, Shitian Zhao, Le Zhuo, Weifeng Lin, Yi Xin, Xinyue Li, Qi Qin, Yu Qiao, Hongsheng Li, and Peng Gao.
Lumina-mgpt: Illuminate flexible photorealistic text-to-image generation with multimodal generative pretraining.
*arXiv preprint arXiv:2408.02657*, 2024a.
- Liu et al. (2024b)
Zirui Liu, Jiayi Yuan, Hongye Jin, Shaochen Zhong, Zhaozhuo Xu, Vladimir Braverman, Beidi Chen, and Xia Hu.
Kivi: A tuning-free asymmetric 2bit quantization for kv cache.
*arXiv preprint arXiv:2402.02750*, 2024b.
- Ma et al. (2025)
Xu Ma, Peize Sun, Haoyu Ma, Hao Tang, Chih-Yao Ma, Jialiang Wang, Kunpeng Li, Xiaoliang Dai, Yujun Shi, Xuan Ju, et al.
Token-shuffle: Towards high-resolution image generation with autoregressive models.
*arXiv preprint arXiv:2504.17789*, 2025.
- Mu et al. (2025)
Jiteng Mu, Nuno Vasconcelos, and Xiaolong Wang.
Editar: Unified conditional generation with autoregressive models.
In *CVPR*, pp. 7899–7909, 2025.
- Qin et al. (2025)
Ziran Qin, Youru Lv, Mingbao Lin, Zeren Zhang, Danping Zou, and Weiyao Lin.
Head-aware kv cache compression for efficient visual autoregressive modeling.
*arXiv preprint arXiv:2504.09261*, 2025.
- Ramesh et al. (2021)
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever.
Zero-shot text-to-image generation.
In *ICML*, 2021.
- Sun et al. (2024)
Peize Sun, Yi Jiang, Shoufa Chen, Shilong Zhang, Bingyue Peng, Ping Luo, and Zehuan Yuan.
Autoregressive model beats diffusion: Llama for scalable image generation.
*arXiv preprint arXiv:2406.06525*, 2024.
- Team (2024)
Chameleon Team.
Chameleon: Mixed-modal early-fusion foundation models.
*arXiv preprint arXiv:2405.09818*, 2024.
- Tian et al. (2024)
Keyu Tian, Yi Jiang, Zehuan Yuan, Bingyue Peng, and Liwei Wang.
Visual autoregressive modeling: Scalable image generation via next-scale prediction.
In *NeurIPS*, 2024.
- Wang et al. (2025)
Junke Wang, Zhi Tian, Xun Wang, Xinyu Zhang, Weilin Huang, Zuxuan Wu, and Yu-Gang Jiang.
Simplear: Pushing the frontier of autoregressive visual generation through pretraining, sft, and rl.
*arXiv preprint arXiv:2504.11455*, 2025.
- Wang et al. (2024)
Xinlong Wang, Xiaosong Zhang, Zhengxiong Luo, Quan Sun, Yufeng Cui, Jinsheng Wang, Fan Zhang, Yueze Wang, Zhen Li, Qiying Yu, et al.
Emu3: Next-token prediction is all you need.
*arXiv preprint arXiv:2409.18869*, 2024.
- Xiao et al. (2023)
Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis.
Efficient streaming language models with attention sinks.
*arXiv preprint arXiv:2309.17453*, 2023.
- Xiao et al. (2024)
Guangxuan Xiao, Jiaming Tang, Jingwei Zuo, Junxian Guo, Shang Yang, Haotian Tang, Yao Fu, and Song Han.
Duoattention: Efficient long-context llm inference with retrieval and streaming heads.
*arXiv preprint arXiv:2410.10819*, 2024.
- Xiong et al. (2024)
Jing Xiong, Gongye Liu, Lun Huang, Chengyue Wu, Taiqiang Wu, Yao Mu, Yuan Yao, Hui Shen, Zhongwei Wan, Jinfa Huang, et al.
Autoregressive models in vision: A survey.
*arXiv preprint arXiv:2411.05902*, 2024.
- Yao et al. (2025)
Jiayi Yao, Hanchen Li, Yuhan Liu, Siddhant Ray, Yihua Cheng, Qizheng Zhang, Kuntai Du, Shan Lu, and Junchen Jiang.
Cacheblend: Fast large language model serving for rag with cached knowledge fusion.
In *ECCV*, 2025.
- Yu et al. (2021)
Jiahui Yu, Xin Li, Jing Yu Koh, Han Zhang, Ruoming Pang, James Qin, Alexander Ku, Yuanzhong Xu, Jason Baldridge, and Yonghui Wu.
Vector-quantized image modeling with improved vqgan.
*arXiv preprint arXiv:2110.04627*, 2021.
- Yu et al. (2022)
Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, et al.
Scaling autoregressive models for content-rich text-to-image generation.
*arXiv preprint arXiv:2206.10789*, 2022.
- Zhang et al. (2025)
Hong Zhang, Zhongjie Duan, Xingjun Wang, Yuze Zhao, Weiyi Lu, Zhipeng Di, Yixuan Xu, Yingda Chen, and Yu Zhang.
Nexus-gen: A unified model for image understanding, generation, and editing.
*arXiv preprint arXiv:2504.21356*, 2025.
- Zhang et al. (2024)
Yuxin Zhang, Yuxuan Du, Gen Luo, Yunshan Zhong, Zhenyu (Allen) Zhang, Shiwei Liu, and Rongrong Ji.
Cam: Cache merging for memory-efficient llms inference.
In *ICLR*, 2024.
- Zhang et al. (2023)
Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, et al.
H2o: Heavy-hitter oracle for efficient generative inference of large language models.
In *NeurIPS*, 2023.

## Appendix A More Attention Visualization

Figure: (a) Janus-Pro-1B
Refer to caption: x7.png

## Appendix B Buffer Mechanism

This appendix evaluates an enhanced buffering mechanism in the SSD framework to boost inference throughput while preserving high-quality image generation in the Janus-Pro-1B model. We compare three configurations—Full Cache, standard SSD, and SSD with buffering (SSD w/ buffer)—under a 20% cache size constraint across batch sizes (BS = 32, 64, 128). The buffering mechanism stores recent tokens in a temporary cache, refreshed every 24 steps, to reduce redundant attention computations.

Table [4](https://arxiv.org/html/2510.18716v1#S4.T4) reports throughput (tokens/s), GenEval, and DPG-Bench metrics, averaged over three runs on NVIDIA A100 GPUs (80GB). At BS = 128, SSD w/ buffer achieves significantly higher throughput (3099.4 tokens/s) than standard SSD (1911.7 tokens/s), with minimal quality degradation (GenEval: 0.72 vs. 0.73; DPG-Bench: 82.53 vs. 82.82). The detailed algorithm is presented in Algorithm [2](https://arxiv.org/html/2510.18716v1#alg2).

Figure: Algorithm 2 SSD KV Cache Compression with Row Buffer

## Appendix C More information about Model and Sparsity

We provide the model architecture configurations and sparsity distribution across heads in Table LABEL:tab:appendix_model_archi and Table LABEL:tabs:appendix_sparsity_accross_heads.

**Table 5: Model specifications for Janus-Pro**
| Model | Image Tokens | Layers | Embedding Size | Attention Heads |
| --- | --- | --- | --- | --- |
| Janus-Pro-1B | 576 | 24 | 2048 | 16 |
| Janus-Pro-7B | 576 | 30 | 4096 | 32 |

**Table 6: Sparsity distribution across heads in Janus-Pro models**
| Sparsity Range | Percentage (%) | Count | Cumulative % | Cumulative Count |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | 1B | 7B | 1B | 7B | 1B | 7B | 1B | 7B |
| 0.0–0.1 | 34.1 | 21.8 | 131 | 209 | 34.1 | 21.8 | 131 | 209 |
| 0.1–0.2 | 16.4 | 16.6 | 63 | 159 | 50.5 | 38.3 | 194 | 368 |
| 0.2–0.3 | 12.2 | 10.2 | 47 | 98 | 62.8 | 48.5 | 241 | 466 |
| 0.3–0.4 | 9.4 | 8.8 | 36 | 84 | 72.1 | 57.3 | 277 | 550 |
| 0.4–0.5 | 7.8 | 11.0 | 30 | 106 | 79.9 | 68.3 | 307 | 656 |
| 0.5–0.6 | 8.9 | 8.6 | 34 | 83 | 88.8 | 77.0 | 341 | 739 |
| 0.6–0.7 | 5.2 | 8.1 | 20 | 78 | 94.0 | 85.1 | 361 | 817 |
| 0.7–0.8 | 2.6 | 8.8 | 10 | 84 | 96.6 | 93.9 | 371 | 901 |
| 0.8–0.9 | 1.6 | 3.6 | 6 | 35 | 98.2 | 97.5 | 377 | 936 |
| 0.9–1.0 | 1.8 | 2.5 | 7 | 24 | 100.0 | 100.0 | 384 | 960 |

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 6
presentation: 3
questions: 1. On Generalizability: Can you comment on the generalizability of your findings? Is there any evidence or strong reason to believe that the "spatial-semantic dichotomy" and "margin column anchoring" phenomena are also present in other visual AR models?

2. On Profiling Cost: Regarding the offline step, "Classify all heads via sparsity profiling": what is the computational cost of this process? Specifically, how many prompts were used to gather the statistics what hardware was used, and approximately how much time did this profiling take? Does this step pose a significant barrier to applying SSD to new models?

3. On Hyperparameters $W$ and $M$: Could you please elaborate on how the spatial window size $W$ and the semantic budget $M$ were selected? For instance, in the 20% cache budget scenario in Table 1, what were the typical values or ratio for $W$ and $M$? How sensitive is the model's performance to this allocation ratio?

4. On CFG Dependence: Your core analysis for identifying semantic injection (Figure 2(b)) is heavily dependent on CFG. Does the SSD framework remain effective, and does the margin anchoring phenomenon persist, during non-CFG sampling ($\gamma=1$) or at very low guidance scales?

5. On Static Classification: The head classification is static. Did you consider or experiment with a dynamic classification strategy, where a head's role could be re-evaluated or adapted based on the generation context?
soundness: 3
strengths: - Originality: The paper's primary strength is its originality. Instead of merely adapting language-based KV compression, it presents a new, empirically-grounded understanding of attention mechanisms in visual AR models. The identification of the "spatial-semantic dichotomy" and the "margin column anchoring" phenomenon is a novel and significant finding.

- Quality: The work is of good quality, with strong empirical validation for its claims.Notably, Figure 2(b) provides exceptionally clear and intuitive evidence for the "semantic anchor" hypothesis. By plotting the MSE between the KV caches of the CFG and non-CFG branches, it accurately visualizes the periodic spikes in semantic content at the margin column positions . The experimental setup is robust, using competitive baselines (H2O, StreamingLLM) and standard benchmarks (GenEval, DPG-Bench).

- Clarity: The paper is well-written, logically structured, and easy to follow. The problem is clearly defined , and the core concepts are introduced intuitively. The figures (especially 2(b)) and Algorithm 1 effectively illustrate the method.

- Significance: This work addresses a critical and practical bottleneck for the deployment of large-scale AR image generators. The reported efficiency gains (5x memory, 6.6x throughput) are substantial and could significantly advance the practical usability of unified multimodal models on resource-constrained hardware.
summary: This paper addresses the significant memory and computational overhead of the KV cache in autoregressive image generation models , noting that existing language-focused compression methods are suboptimal for visual tokens. The authors empirically identify a novel attention phenomenon: a functional dichotomy where some heads focus on spatial locality and others act as emergent semantic sinks . Crucially, they find semantic information is preferentially anchored at the margin columns of the token grid . Based on these insights, the paper proposes SSD, a framework that classifies heads as spatial or semantic and applies distinct, tailored compression policies to each type . Experiments demonstrate that SSD achieves up to a 5x memory reduction and 6.6x speedup with negligible quality degradation.
weaknesses: - Generalizability: The paper's primary weakness is the limited scope of its validation. All analyses and experiments are conducted exclusively on the Janus-Pro model family. It remains unclear whether the core findings—the spatial-semantic dichotomy and margin column anchoring—are fundamental properties of visual AR generation or emergent properties specific to the Janus-Pro architecture. The claim needs to be validated on other visual AR models to be considered general.

- Static Head Classification: The classification of heads as spatial or semantic is performed offline and remains static throughout inference. This approach ignores the possibility that a head's function might be dynamic and context-dependent. A static policy may be suboptimal compared to a dynamic one.

- Hyperparameter Sensitivity: The method introduces several key hyperparameters, including the spatial window size $W$, the semantic budget $M$, and the classification threshold $\tau$. While the paper provides a good sensitivity analysis for $\tau$, it lacks a detailed discussion on the selection and sensitivity of $W$ and $M$. It is unclear how these are balanced to meet a specific cache budget and how performance is affected by their interplay.

---

confidence: 5
contribution: 2
initial_rating: 2
presentation: 2
questions: See weaknesses.
soundness: 3
strengths: 1. The proposed methods are simple and effective.
2. The paper is easy to follow.
summary: The paper proposes ​​SSD​​, a framework for compressing the KV cache in autoregressive image generation models. The method categorizes attention heads into two types: (1) spatial locality heads, which focus on spatially adjacent tokens, and (2) semantic sink heads, which attend to a few critical tokens. The KV cache for each type is then compressed using a dedicated strategy. Experimental results show that SSD effectively reduces the KV cache size and accelerates the decoding process, with only minimal performance degradation.
weaknesses: 1. The experimental evaluation is conducted exclusively using Janus-Pro models. To fully establish the robustness and general applicability of the proposed methods, validation across a broader range of model architectures is necessary.
2. The concept of exploiting spatial locality to accelerate autoregressive (AR) image generation has been widely adopted in methods such as PAR [1], ZipAR [2], and NAR [3]. These works, which also employ parallel decoding by restricting the attention window, are highly relevant yet are not discussed or compared against in the paper.
3. The core motivation of the paper may require reconsideration. While KV cache size presents a significant challenge for LLMs with long contexts, the context length for AR image generation models is typically much shorter, often comprising only hundreds or a few thousand tokens. Furthermore, increasing the batch size does not alter the model's computational intensity (i.e., the compute-to-memory-access ratio). From this perspective, the necessity of compressing the KV cache for AR image generation appears debatable. The paper's primary contribution likely stems instead from the throughput gains achieved via sparse attention during decoding.

[1] Parallelized Autoregressive Visual Generation, CVPR 2025.

[2] ZipAR: Parallel Auto-regressive Image Generation through Spatial Locality, ICML 2025.

[3] Neighboring Autoregressive Modeling for Efficient Visual Generation, ICCV 2025.

---

confidence: 4
contribution: 3
initial_rating: 4
presentation: 2
questions: - **Semantic Concentration Metric** : The definition of the "Semantic Concentration" metric is not clear. Why is the difference between the KV cache values of CFG and "native" (non-CFG) generation representative of semantic concentration?

- **Sliding Window Implementation** : In a flattened 1D token sequence, the local "neighborhood" tokens should include not only tokens immediately to the "left" (preceding in the sequence) but also spatially adjacent tokens from previous rows. Figure 1 seems to confirm this. However, why sliding method of SSD just retain the preceding(left) tokens in the 1D sequence?
soundness: 3
strengths: - The paper is well-written, intuitive, and easy to understand.
- This is the first work that tries to analyze characteristics of KV-cache in AR image models, and found interesting attention patterns (spatial and semantic). This observation aligns well with intuition.
- Also, this paper propose intuitive KV cache compression methods tailored for two distinct attention types.
summary: This paper presents KV cache compression methods for next-token-prediction-based autoregressive (AR) image generation. The authors observe that attention heads in AR image models are highly sparse, and can be broadly categorized into two types: "semantic heads," which focus on periodic anchor tokens, and "spatial heads," which concentrate on spatially-local tokens. Based on this observation, the paper proposes two distinct KV cache compression techniques tailored for each head type: a sliding window approach for spatial heads and a heavy-hitter-retention method for semantic heads. Experimental results on the Janus-Pro model demonstrate a superior Pareto frontier on the DPG-bench compared to existing KV cache compression methods developed for text LLMs.
weaknesses: - **Limited Generalizability** : All experiments were conducted solely on the Janus-Pro model. It is uncertain whether the paper's findings, including the observed attention patterns and the efficacy of the proposed compression methods, generalize to other AR image generation models. Experiments on other AR image models, such as llamaGen, Emu3, Anole, and Lumina-mGPT (1, 2), are necessary. I believe experiments on llamaGen are essential, and additional validation on Lumina-mGPT would be welcome.

- **Insufficient Evaluation** : Performance evaluation was restricted to the DPG-bench, which may not adequately capture the perceptual quality of the generated images. An experiment using standard image generation metrics, such as FID or IS, on datasets like MS-COCO, is required.

- **Novelty** : While tackling this problem for the first time and identifying the sparse attention patterns is novel, the proposed methods lack originality. They appear to be direct applications of existing KV cache compression techniques.

---

confidence: 4
contribution: 2
initial_rating: 4
presentation: 3
questions: see above
soundness: 3
strengths: SSD’s greatest strength lies in connecting a real hardware bottleneck (KV cache bloat) with a novel structural insight (spatial–semantic head specialization), and turning that into a principled, empirically validated framework that achieves major efficiency gains without sacrificing image quality.
summary: The paper introduces SSD (Spatial-Semantic Decoupling) — a framework for Key-Value (KV) cache compression in autoregressive (AR) image generation models like Janus-Pro.
SSD aims to reduce memory and computation costs during image generation without hurting image quality.
weaknesses: 1 The paper identifies “spatial locality” and “semantic sink” as two distinct attention behaviors in autoregressive image generation.
However, the spatial locality aspect is not novel — similar locality patterns have been discussed in NAR, LPD, and ZipAR, which all highlight that visual attention predominantly focuses on nearby spatial tokens. The authors should clarify the difference between their “spatial-locality heads” and previously observed locality mechanisms, and properly cite these works to avoid overclaim. The attention sink is also old-fashioned since the ViT era [4,5], making it expected in AR. If the authors just use this proposed or similar observation for caching, I think the novelty is quite limited since many similar works have been extensively done in LLM [6].

[1] Locality-aware Parallel Decoding for Efficient Autoregressive Image Generation

[2] Neighboring Autoregressive Modeling for Efficient Visual Generation

[3] ZipAR: Parallel Auto-regressive Image Generation through Spatial Locality

[4] See What You Are Told: Visual Attention Sink in Large Multimodal Models

[5] Quantizable Transformers: Removing Outliers by Helping Attention Heads Do Nothing

[6] MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attention

2 The paper’s analysis and method are entirely built on raster-order autoregressive generation. However, this paradigm is now computationally suboptimal and is being replaced by faster alternatives such as MAR, LPD, and VAR. The authors should discuss whether their observed attention patterns — particularly spatial locality and the semantic-sink behavior — hold across different generation orders. More applications on these frameworks are necessary. Besides, if the work is raster-specific, more applications on Llamagen are necessary.


Decision: Reject