Title: GaugeKV: Composable Exact KV Cache Compression
Primary area: foundation or frontier models, including LLMs
Keywords: KV cache, Transformers, Gauge Symmetry, Gauge canonicalization, GQA/MQA/MoE/RoPE

Abstract: The key–value (KV) cache is a dominant memory cost in long-context Transformer inference. We introduce GaugeKV, a training-free method that leverages the head-wise gauge symmetry of attention to reduce KV memory both exactly and with certificates. A one-time gauge canonicalization rewrites weights so that values are orthonormal and queries/keys are scale-balanced; thereafter the model produces K/V in a compression-friendly basis without changing its function or runtime FLOPs. Combined with lossless hot/cold staging, this yields bit-identical outputs (FP32 deterministic) with measurable KV reductions (e.g., 1.11×–1.21× on GPT-2; 1.13× key-side on Qwen2.5-7B).
Crucially, the canonical basis makes these gains multiply with grouped/multi-query attention (GQA/MQA), yielding 4.6× total KV reduction for typical GQA (h=32, g=8) and 36.8× structural (73.6× total KV reduction with FP8, γ=0.5) for aggressive MQA (g=1, h=32). When further combined with FP8 quantization, total KV reductions can exceed 20×. Beyond exact savings, the canonical value basis turns rank-r value caching into a first-class primitive with a priori guarantees: we prove end-to-end bounds and observe 100% compliance at r=32 on GPT-2. Both tracks compose multiplicatively with grouped/multi-query attention, quantized KV, paging, and MoE—substantially extending feasible context on fixed VRAM.

TL;DR: GaugeKV reparameterizes Transformer heads into a canonical basis to losslessly compress the KV cache. It composes with GQA/MQA and FP8, preserves FP32-deterministic outputs, supports RoPE/MoE, and enables 4.6×–73.6× total KV-memory reduction