ICLR 2025

Published as a conference paper at ICLR 2025
CAKE: CASCADING
AND ADAPTIVE KV CACHE
EVICTION WITH LAYER PREFERENCES
Ziran Qin1,2∗, Yuchen Cao2, Mingbao Lin3†, Wen Hu2, Shixuan Fan2, Ke Cheng2,
Weiyao Lin1†, Jianguo Li2†
1Shanghai Jiao Tong University, 2Ant Group, 3Independent Researcher
ABSTRACT
Large language models (LLMs) excel at processing long sequences, boosting de-
mand for key-value (KV) caching. While recent efforts to evict KV cache have al-
leviated the inference burden, they often fail to allocate resources rationally across
layers with different attention patterns. In this paper, we introduce Cascading and
Adaptive KV cache Eviction (CAKE), a novel approach that frames KV cache
eviction as a “cake-slicing problem.” CAKE assesses layer-specific preferences
by considering attention dynamics in both spatial and temporal dimensions, allo-
cates rational cache size for layers accordingly, and manages memory constraints
in a cascading manner. This approach enables a global view of cache alloca-
tion, adaptively distributing resources across diverse attention mechanisms while
maintaining memory budgets. CAKE also employs a new eviction indicator that
considers the shifting importance of tokens over time, addressing limitations in
existing methods that overlook temporal dynamics. Comprehensive experiments
on LongBench and NeedleBench show that CAKE maintains model performance
with only 3.2% of the KV cache and consistently outperforms current baselines
across various models and memory constraints, particularly in low-memory set-
tings. Additionally, CAKE achieves over 10× speedup in decoding latency com-
pared to full cache when processing contexts of 128K tokens with FlashAttention-
2. Our code is available at https://github.com/antgroup/cakekv.
1
INTRODUCTION
Large language models (LLMs) (Zhao et al., 2023; Achiam et al., 2023; Dubey et al., 2024; An-
thropic, 2024; AI, 2024) have enhanced their long-text processing capabilities, improving perfor-
mance in mul...