Title: 70% Size, 100% Accuracy: Lossless LLM Compression for Efficient GPU Inference via Dynamic-Length Float (DFloat11)
Venue: NeurIPS 2025
Accepted: yes
Keywords: Compression

Lossless Compression

LLM

Efficiency

Diffusion Models
Abstract: Large-scale AI models, such as Large Language Models (LLMs) and Diffusion Models (DMs), have grown rapidly in size, creating significant challenges for efficient deployment on resource-constrained hardware. In this paper, we introduce Dynamic-Length Float (DFloat11), a lossless compression framework that reduces LLM and DM size by 30\% while preserving outputs that are bit-for-bit identical to the original model. DFloat11 is motivated by the low entropy in the BFloat16 weight representation of LLMs, which reveals significant inefficiency in the existing storage format. By applying entropy coding, DFloat11 assigns dynamic-length encodings to weights based on frequency, achieving near information-optimal compression without any loss of precision. To facilitate efficient inference with dynamic-length encodings, we develop a custom GPU kernel for fast online decompression. Our design incorporates the following: (i) compact, hierarchical lookup tables (LUTs) that fit within GPU SRAM for efficient decoding, (ii) a two-phase GPU kernel for coordinating thread read/write positions using lightweight auxiliary variables, and (iii) transformer-block-level decompression to minimize latency. Experiments on Llama 3.3, Qwen 3, Mistral 3, FLUX.1, and others validate our hypothesis that DFloat11 achieves around 30\% model size reduction while preserving bit-for-bit identical outputs. Compared to a potential alternative of offloading parts of an uncompressed model to the CPU to meet memory constraints, DFloat11 achieves 2.3--46.2$\times$ higher throughput in token generation. With a fixed GPU memory budget, DFloat11 enables 5.7--14.9$\times$ longer generation lengths than uncompressed models. Notably, our method enables lossless inference of Llama 3.1 405B, an 810GB model, on a single node equipped with 8$\times$80GB GPUs.


Peer reviews:
comments: SUMMARY:
This paper introduces Dynamic-Length Float (DFloat11), a lossless compression framework for BFloat16 model weights, achieving ~30% size reduction while preserving bit-for-bit accuracy. The method leverages entropy coding for exponents and employs optimized GPU kernels for efficient decompression during inference. Experiments on models like Llama 3 and Stable Diffusion demonstrate significant memory savings and throughput improvements.

STRENGTHS AND WEAKNESSES:
## Strengths

1. Great insight. The authors identify the redundancy of BF16 and then propose to leverage entropy coding for exponents. The proposed DFloat11 maintains almost 100% model accuracy, which is impressive.

2. The authors implemented fast GPU kernels, which enables practical usages for on different hardwares.

3. Overall, the performance of this method is great. The paper is well-written and in a good shape. 

## Weaknesses

1. My main concern is the evaluation on LLM is not very comprehensive. Specially, recent LLM releases tend to report performance across different tasks, including code, math, multilingual, instruction following, etc. It is not very clear how the proposed DFloat11 affect LLM's performance in these tasks. In my opinion, WikiText and C4 cannot fully represent the capability of today's LLMs.

2. Latency Overhead: Small but non-zero decompression costs may affect latency-sensitive applications, though amortized at larger batches.

QUESTIONS:
Please refer to the weaknesses.

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 4
ORIGINALITY: 4
DATE: 2025-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes Dynamic-Length Float (DFloat11), a lossless compression framework that compresses weights of bfloat16-based LLMs and diffusion models to ~70% of their original size while preserving them bit-for-bit. The authors observe that the 8-bit exponent field in bfloat16 weights carries about less than 3 bits of information. They exploit this redundancy by Huffman-coding the exponent bits and leaving the sign and mantissa untouched, yielding an effective 11-bit representation. The weights are meant to be quickly decompressed with a GPU-based algorithm which stores decomposed LUTs in SRAM of every thread block, rendering the decompression operation fast and memory-bound.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n - The paper is clearly written and carefully checked even for minor errors.\n - It offers a clever solution to a rarely touched problem of lossless model compression.\n - The appendix is well-thought out and complements the paper with insightful details.\n\n**Weaknesses**\n - The paper is missing a clear presentation of the overhead introduced by DFloat11, e.g., showing how large is the latency overhead in relation to BF16 calculation, when weight matrix dimensions and batch size change. This is somewhat visible _in vivo_ on Figure 6, which hints that for small token batch size the latency roughly doubles because DF11 is memory bound, and for large batch sizes it hides behind FLOPs. It would be useful to have a performance benchmark for, e.g., a single linear layer (of different sizes) and not an entire model.\n- Quantization of bfloat16 models to > 8 bits is a niche problem, since proficient methods of quantizing to 8-bit formats or lower exist and the format is widely supported in hardware ; however, the authors well justify the lossless approach, which is a small but nonetheless valuable contribution.\n - Figure 4: The font on the plots is hard to read after printing. Perhaps a shared ylabel would save some space that could be spent there.\n - In general, the figures are hard to read, some fonts are too small.\n - Minor: it seems that the common capitalization (also used by Google which introduced this format) is “bfloat16”, while the paper uses “BFloat16”.\n - Source code absent (but promised to be released upon acceptance). In a paper like this, where the kernel is the central contribution, releasing the code is in my opinion a necessity.\n\nQUESTIONS:\n- Can DFloat11 be enabled on-demand only when the memory is exceeded (e.g., when the context gets too long)?\n - What would it take to fuse DF11 decompression into GEMM/attention kernels? How would this hypothetically lower the cost of decompression?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces DF11, a novel data format to losslessly reduce the size of LLMs by applying entropy coding. DFloat11 assigns dynamic length encoding to weights. The authors also write custom GPU kernels for fast online decompression. This method shows improvement over a large range of models, improving efficiency, generation length and the capability of GPU hosting LLMs.\n\nSTRENGTHS AND WEAKNESSES:\nStrenghs\n1. The idea of losslessly compressing LLMs is interesting and novel.\n2. The experiments are convincing and detailed.\n3. The writings are easy to follow.\n4. The algorithms to support the lossless compress make sense and are hardware-efficient.\n\nWeeknesses\n1. The paper does not show the performance gap between DF11 and AWQ/GPTQ/FP8, even if I acknoledge that these methods are lossy.\n2. The paper does not provide quantitative information on the time consumption of compression.\n3. DFloat11 is less helpful when FP8 becomes the default data format. (e.g., DeepSeek-V3/R1)\n4. Token Batch Size 16384 is not that convincing, since I assume the use case of DF11 is when VRAM is limited. That means this token batch size should refer to prefilling stage. However, except for super-long context, (where KV cache memory will become the major bottleneck), the major overhead is in decoding. Assuming 40GB A100 and Llama3.1-8B, the common serving batch size is hardly beyond 16-32. In this regime, the parameter loading will become the main cost. DF11, to my understanding, will at least cost 70% additional overhead since it will load the 11-bit and 16-bit weights.\n\nQUESTIONS:\n1. Is it possible to apply the dynamic length weight to FP8/FP4 models? What is the room for improvement?\n2. How long does it take to compress LLMs in DF11? For example, 1B/8B/32B/405B.\n3. How is the latency when batch size is 1/2/4/8/16/32?\n\nLIMITATIONS:\nyes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces Dynamic-Length Float (DFloat11), a lossless compression method for storing BF16 model weights by applying Huffman coding only to the exponents. This reduces the original 8-bit exponent representation to an average of 2.6 bits. Additionally, the paper proposes an efficient GPU-friendly implementation of Huffman en/decoding, addressing parallelization challenges to maintain acceptable inference speed.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n- The core idea of compressing exponents—given their lower entropy—is well-motivated and supported by entropy analysis across multiple models, making the approach convincing.\n\n- The authors thoughtfully address implementation challenges, particularly for GPUs, and their proposed two-phase kernel to handle Huffman coding appears sound.\n\n- The paper provides solid experimental results. End-to-end evaluations on large LLMs demonstrate that DFloat11 delivers real-world improvements.\n\n**Weaknesses**\n- Some of the proposed implementation optimizations for Huffman encoding—such as using a gap array and index storage—have already been explored in prior work, which may make these contributions appear incremental.\n\n- The method is BF16-specific, while many production systems are shifting toward INT8, FP8, or mixed-precision formats. Although the paper mentions this limitation in the introduction, it lacks a more detailed discussion comparing BF16 accuracy and throughput with INT8, which would help clarify DFloat11’s relevance in current deployment contexts.\n\nQUESTIONS:\n- Since gap arrays, index storage, and Huffman encoding are not novel techniques, what do you consider the key contribution or novelty of your paper?\n\n- Could the proposed exponent-only compression strategy be extended to other formats such as FP16, FP8, or even 8-bit quantized models? If so, what kind of savings or trade-offs would you expect in those settings?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-09"}