COSPADI: Compressing LLMs via Calibration-Guided Sparse Dictionary
  Learning
Authors: name: Dmitriy Shopkhoev, name: Denis Makhov, name: Magauiya Zhussip, name: Ammar Ali, name: Stamatios Lefkimmiatis
Organization: MWS AI

Post-training compression of large language models (LLMs) largely relies on
low-rank weight approximation, which represents each column of a weight matrix
in a shared low-dimensional subspace. While this is a computationally efficient
strategy, the imposed structural constraint is rigid and can lead to a
noticeable model accuracy drop. In this work, we propose CoSpaDi (Compression
via Sparse Dictionary Learning), a novel training-free compression framework
that replaces low-rank decomposition with a more flexible structured sparse
factorization in which each weight matrix is represented with a dense
dictionary and a column-sparse coefficient matrix. This formulation enables a
union-of-subspaces representation: different columns of the original weight
matrix are approximated in distinct subspaces spanned by adaptively selected
dictionary atoms, offering greater expressiveness than a single invariant
basis. Crucially, CoSpaDi leverages a small calibration dataset to optimize the
factorization such that the output activations of compressed projection layers
closely match those of the original ones, thereby minimizing functional
reconstruction error rather than mere weight approximation. This data-aware
strategy preserves better model fidelity without any fine-tuning under
reasonable compression ratios. Moreover, the resulting structured sparsity
allows efficient sparse-dense matrix multiplication and is compatible with
post-training quantization for further memory and latency gains. We evaluate
CoSpaDi across multiple Llama and Qwen models under per-layer and per-group
settings at 20-50\% compression ratios, demonstrating consistent superiority
over state-of-the-art data-aware low-rank methods both in accuracy and
perplexity. Our results establish structured sparse dictionary learning as a
powerful alternative to conventional low-rank approaches for efficient LLM
deployment.