70% Size, 100% Accuracy: Lossless LLM Compression for Efficient GPU
  Inference via Dynamic-Length Float
Authors: name: Tianyi Zhang; name: Yang Sui; name: Shaochen Zhong; name: Vipin Chaudhary; name: Xia Hu; name: Anshumali Shrivastava
Published: 2025-04-15 22:38:38

Abstract: Large Language Models (LLMs) have grown rapidly in size, creating significant
challenges for efficient deployment on resource-constrained hardware. In this
paper, we introduce Dynamic-Length Float (DFloat11), a lossless compression
framework that reduces LLM size by 30% while preserving outputs that are
bit-for-bit identical to the original model. DFloat11 is motivated by the low
entropy in the BFloat16 weight representation of LLMs, which reveals
significant inefficiency in existing storage format. By applying entropy
coding, DFloat11 assigns dynamic-length encodings to weights based on
frequency, achieving near information-optimal compression without any loss of
precision. To facilitate efficient inference with dynamic-length encodings, we
develop a custom GPU kernel for fast online decompression. Our design
incorporates the following: (i) decomposition of memory-intensive lookup tables
(LUTs) into compact LUTs that fit in GPU SRAM, (ii) a two-phase kernel for
coordinating thread read/write positions using lightweight auxiliary variables,
and (iii) transformer-block-level decompression to minimize latency.
Experiments on recent models, including Llama-3.1, Qwen-2.5, and Gemma-3,
validates our hypothesis that DFloat11 achieves around 30% model size reduction
while preserving bit-for-bit exact outputs. Compared to a potential alternative
of offloading parts of an uncompressed model to the CPU to meet memory
constraints, DFloat11 achieves 1.9-38.8x higher throughput in token generation.
With a fixed GPU memory budget, DFloat11 enables 5.3-13.17x longer context
lengths than uncompressed models. Notably, our method enables lossless
inference of Llama-3.1-405B, an 810GB model, on a single node equipped with
8x80GB GPUs. Our code and models are available at
https://github.com/LeanModels/DFloat11.