70% Size, 100% Accuracy: Lossless LLM Compression for Efficient GPU Inference via Dynamic-Length Float (DFloat11)
Authors: Tianyi Zhang
Mohsen Hariri
Shaochen Zhong
Vipin Chaudhary
Yang Sui
Xia Hu
Anshumali Shrivastava
Venue: NeurIPS 2025 poster ()
Keywords: Compression
Lossless Compression
LLM
Efficiency
Diffusion Models
Primary area: other

Abstract: Large-scale AI models, such as Large Language Models (LLMs) and Diffusion Models (DMs), have grown rapidly in size, creating significant challenges for efficient deployment on resource-constrained hardware. In this paper, we introduce Dynamic-Length Float (DFloat11), a lossless compression framework that reduces LLM and DM size by 30\% while preserving outputs that are bit-for-bit identical to the original model. DFloat11 is motivated by the low entropy in the BFloat16 weight representation of LLMs, which reveals significant inefficiency in the existing storage format. By applying entropy coding, DFloat11 assigns dynamic-length encodings to weights based on frequency, achieving near information-optimal compression without any loss of precision. To facilitate efficient inference with dynamic-length encodings, we develop a custom GPU kernel for fast online decompression. Our design incorporates the following: (i) compact, hierarchical lookup tables (LUTs) that fit within GPU SRAM for efficient decoding, (ii) a two-phase GPU kernel for coordinating thread read/write positions using lightweight auxiliary variables, and (iii) transformer-block-level decompression to minimize latency. Experiments on Llama 3.3, Qwen 3, Mistral 3, FLUX.1, and others validate our hypothesis that DFloat11 achieves around 30\% model size reduction while preserving bit-for-bit identical outputs. Compared to a potential alternative of offloading parts of an uncompressed model to the CPU to meet memory constraints, DFloat11 achieves 2.3--46.2$\times$ higher throughput in token generation. With a fixed GPU memory budget, DFloat11 enables 5.7--14.9$\times$ longer generation lengths than uncompressed models. Notably, our method enables lossless inference of Llama 3.1 405B, an 810GB model, on a single node equipped with 8$\times$80GB GPUs.

70% Size, 100% Accuracy:
Lossless LLM Compression for Efficient GPU Inference via Dynamic-Length Float (DFloat11)
Tianyi Zhang', Mohsen Harirí', Shaochen (Henry) Zhong', Vipin Chaudhary?, Yang Sui', Xia Hu', and Anshumali Shrivastava'
'Department of Computer Science, Rice University Department of Computer and Data Sciences, Case Western Reserve University Ken Kennedy Institute
(tz21, henry .zong, yang. yang.sui, xia. anshumali arice.aleerrrieeeeeeeeeeee edu, (mohsen. hariri,
vipin)@case. edu
Code: http://ppggggttaaooocc///aaaaaaa
Models: http://pp///gggggaaffoooo///////
Abstract
Large-scale AI models, such as Large Language Models (LLMs) and Diffusion Models (DMs), have grown rapidly in size, creating significant challenges for efficient deployment on resource-constrained hardware. In this paper, we introduce Dynamic-Length Float (DFloatl 1), a lossless compression framework that reduces LLM and DM size by 30% while preserving outputs that are bit-for-bit identical to the original model. D motivated by the low entropy the B6 weight representation of LLMs, which reveals significant inefficiency in the exist- ing storage format. By applying entropy coding, DFloat I assigns lynamic--englee encodings to weights based on frequency, achieving near information-optina compression without any loss precision. To facilitate efficient inference with dynamic-length encodings, pression. Our design incorporates the following: (i) compact, hierarchical lookup tables (LUTss) that fit within GPU SRAM for efficient decoding, (ii) two-phase GPU kernel for coordinating thread read/write positions using lightweight auxiliary variables, and (iii) wunsformer-block-len Experiments on Llama 3.3, Qwen 3, Mistral 3, FLUX.1, and others validate our hypothesís that D 1 achieves around 30% model size reduction while preserv- ing bit-for-bit identical outputs. Compared to a potential alternative of offloading parts of an uncompressed model to the CPU meet memory constraints. DFloatl achieves 2.3-46.2x higher throughput in token generation. With a fixed GPU memory budget, DFloat1 enables 5.7-14.9x longer generation lengths than un- compressed models. Notably, our method enables lossless inference of Llama 3.1 405B, an 810GB model, on single node equipped with 8 80GB GPUs.
develop custom GPU kernel for fast online decom-
decompression
minimize latency.
1 Introduction
Foundation models, such as Large Language Models (LLMs) and Diffusion Models (DMs), have demonstrated remarkable capabilities across wide range of Natural Language Processing (NLP) (56) and Computer Vision (CV) tasks [57]. However, their huge model sizes create substantial obstacles
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

for efficient deployment, especially in meemory-constraie environments. For example, a competitive recent LLM, Llama 3.1 405B [20,, has 405 billion parameters in 16-bit Brain Float (BFloat16) format and requires about 810 GB of memory for full inference, exceeding the capacity of a typical high-end GPU server (e.g., DGX A100/H100 with 8 GPUs). As a result, deploying this model requires multiple nodes, making it expensive and inaccessible. In this work, we present a solution that compresses any BFloat16 model to approximately 70% of its original size while preserving 100% of its accuracy on any task.
Model compression via quantization has limitations. Quantization is type of lossy compression method that lowers the precision of model weights by converting them into lower bit-width repre- sentations [15 37 36 43). Although it can significantly reduce memory usage and often improve inference speed, quantization is not a mnc-size-fits-all solution and presents several key limitations: 0 Accuracy degradation. By design, quantization introduces approximation errors. The degree of accuracy loss depends on multiple factors, including the base model, quantization method, evaluation benchmark, and target bit-width [35] These interactions make it difficult to predict quantify the impact comprehensively, Even mild quantization can noticeably degrade performance. For example, applying 8-bit SmoothQuant [51] to ReepSeek-RR SEI-Distill-Qwert 1.5B [1] results in 9.09% drop in average accuracy across reasoning tasks [39] @ Behavioral shifts. Even when overall accuracy metrics appear roughly unchanged, quantized models may behave differently from their full-precision counterparts. For instance, Dutta et al. [13 observe a phenomenon called flips, where quantized
models produce answers that change from correct to incorrect and vice versa. This indicates that quantization can significantly alter model behavior, even when standard accuracy metrics show minimal change. For example, the W8A16 GPTQ-quanized Qwen2-1.5B 13 54 exhibits only 0.3% drop in GSM8K (8-shot) accuracy [51, yet 6.37% of its answers flip in correctness [33] Compliance and reliability concerns. In domains like finance or healthcare, quantized models may not satisfy regulatory or reliability standards, as their outputs may differ from those of the original We refer readers to Appendix [o a more detailed discussion on quantization.
models 31.
Existing lossless model compression does not support efficient GPU inference. Unlike lossy compression, lossless compression reduces model size while preserving the full precision of the original weights. This ensures the model's output distribution remains identical to that of the uncompressed counterpart. However, most existing lossless methods focus on storage efficiency, such as compressing model checkpoints 122 25), or target specialized hardware like FPGAs [59], rather than accelerating inference on general-purper purpose GPUs. While useful for tasks like checkpoint rollback during large-scale training [47 or reducing download time from model hubs 25 these methods offer little to no benefit for GPU-based inference.
Our proposal, Dynamic-Lengte Float (DFloat 1), is lossless compression framework optimized for efficient GPU inference. We identify key inefficiency in the commonly used BF format: its 8-bit exponent field carries only about 2.6 bits of actual information. This redundancy is consistent across a wide range of LLMs, as shown in Section 2.2 To exploit it, we apply Huffman coding (28) to the exponent bits of BFloat16 weights, while leaving the sign and mantissa bits uncompressed. The resulting exponents have dynamic- encodings: frequent values are assigned shorter codes, while rarer ones use longer codes. However, standard Huffman decoding relies on sequential bit-by-bit tree traversal, which is inefficient on GPUs due to limited parallelism. Assigning one GPU thread per decompression task leads to severe hardware underutilization and high latency. To overcome this, design hardware-awarr algorithm that enables efficient online decompression of dynamic-length floats on GPUs Our solution includes three key components: 1. compact, hierarchical lookup tables (LUTs) that fit in GPU SRAM to support fast, table-based Huffmman decoding, 2. a two-phase GPU kernel that uses lightweight auxiliary variables coordinate thread-level read and write operations, and 3. batched decompression at the transformer-block level to maximize throughput. We summarize our contributions as follows:
1. We propose Dynamic-Length Float (DFloat11), a losslessly compressed floating- -oint format that reduces BFloat16 weights to approximately 11 bits. This yields around 30% model size reduction with bit-for-bit identical outputs.
We develop optimized, hardware-aware algorithms for efficient GPU inference with DFloat 1- compressed models by leveraging GPU memory and compute hierarchies.

---

Entraay
Exponent Enropy
Entropy
1.0
Bge 900
Dagnnen 20 BPg
Martaaa ( Bid
Brain Float (BFloat1 or FF1
T 5.4 of
exporent are
Erpusy
compressjba
Sign (1 Bi))
Exponent (8 Bits)
Mantissa (7 Bits)
62-
Lamu sis
Lamaaaa 7i0 Mairnd final demeng 2.5
Derin 2.8 kis Lamaa a do
Liama Music cht
Gartres an atl
Qena 2.5 1ib
Matze
Unira 708
Figure 1: (Left) The allocation of bits for the components of BFloat16. (Right 3) The Shannon entropy of the components (sign, exponent, mantissa) B6 weights various LLMs.
3. We evaluate D I across popular LLMs and diffusion transformers, including Llama 3, Qwen Mistral DeepSeek RI Distilled, FLUX.11, and Stable Diffusion 3.5 method consistently achieves 30% compression without altering original outputs at all. Notably, it enables running Llama-3.1 1.10BB on single node (8x80GB A100 GPUs), reducing hardware requirements by half without accuracy loss.
(20
46
45
21
32 2 Our
2 Method
In this section, we introduce our proposed foating-point format, Dynamic-Length Float (DFloat 1), along with its custom decompression kernel designed for efficient GPU inference.
2.1
Preliminary
Brain Float (BFloat16) Recent state-of-the-ee LLMs predominantly employ the 16-bit Brain Float format ( BF16) for storing weights, due its balance of numerical precision with memory efficiency. BF16 allocates its 16 bits as follows: sign bit, 8 exponent bits. and 7 mantissa bits. The numerical value represented by BF16 number computed as:
(-1)ssm x 2expoment- 127 x (1 (1.mantissa),
where mantissa is interpreted as binary fractional value.
technique in lossless data compression that leverages statistical redundancy to reduce data size. Several widely used methods fall under this category. in- cluding Huffman coding (28), arithmetic coding (33), and Asymmetric Numeral Systems (ANS) [12). Among these, Huffman coding is one of the most widely adopted, which uses variable-length encod- ing minimize the size of encoded data. It assigns shorter binary codes more frequent symbols and longer codes to less frequent ones. The codes are decoded using a prefix-free binary tree, known as Huffman tree. Due to the prefix-free property of Huffman codes, no code a prefix of any other, which ensures unique decodability of the encoded bitstream without the need for delimiters. The tree constructed based on symbol frequencies and provably optimal for any given frequency distribution. However, decoding Huffman codes in massively parallel manner is challenging due to its inherently sequential nature.
Entropy Coding Entropy coding is
GPU Computation and Memory Paradigm GPUs are designed perform computations in massively parallel manner. A modern GPU consists of thousands of threads, which are organized into blocks and executed on streaming multiprocessors (SMs). Each block has access to small, fast, on-chip memory called shared memory (often referred to as SRAM), which provides much lower latency and higher bandwidth than the off-chip global memory, commonly known as high-bandwidth memory (HBM). The capacity of shared memory is limited, typically having up to 100 KB per block. this work, we leverage the fast access characteristics of SRAM to enable efficient on-the-fly decompression of compressed weights during inference.
3

---

Dynamic-Length Float (DFloat11 or DF11)
Huffman Tree
1st
Encoded Exponents
(Dynamic Widths, Averaging -2.6 Bits)
Code: Exponent: 118
1
0
1
0
Decode
1st Element
2nd Element
Code: 10
Exponent: 121
Sign & Mantissa
(Fixed Bit Widths, Bit & Bits)
1
1
Code: 110
Bit
Exponent:
1at Elemant
2nd Elemant
Figure 2: Our proposed format Dynamic-Length Float for compressing BFloat16 weights of LLMs losslessly down to 11 bits. The exponents are compressed via Huffman coding, while the sign and mantissa bits remain uncompressed.
2.2 Motivation: BFloat16 Representation is Information Inefficient
To motivate the lossless compression of LLM weights, analyze the compressibility of the BF weights of recent LLMs. Specifically, we use Shannon entropy to quantify the information content of 16 components (sign, exponent, and mantissa) for all linear projection matrices of an LLM. The Shannon entropy H-) is defined as:
=(X)= =- Σ p(x) E(()) loga
(2)
ZEX
where X is a discrete random variable with support X. and p X' - [0. denotes its probability mass function. We present the computed entropy values in Figure [I] As shown, the entropy of the sign and mantissa bits is close to their respective bit widths, indicating limited potential for compression. In contrast, the exponent exhibits significantly lower entropy, approximately 2.6 bits versus its allocated 8 bits, suggesting substantial opportunities for lossless compression.
To understand this discrepancy, visualize the frequency distribution of all BFloat16 components in Figure S and the ranked frequency of exponent values in Figure 9 both in the Appendix. The sign and mantissa values relatively uniform across their ranges, but the exponent distribution highly imbalanced: only about 40 of the 256 possible 8-bit values are used, with the rest never appearing. Ranked frequencies also decay rapidly. These observations reveal the low entropy of the exponent and its potential for compression.
2.3 Dynamic-Lengte Float: Lossless LLM Compression for Efficient GPU Inference
To address the substantial information inefficiency in the BFloaa16 representation of LLM weights, we propose lossless compression framework that encodes floating-point parameters using entropy coding. Specifically, we build Huffman tree based on the distribution of exponents in model weights. We then compress the exponents using Huffman coding, while preserving the original signs and mantissas. Exponents are encoded and tightly bit-packed into a byte array. EncodedExponent, while the sign and mantissa are left uncompressed and stored in separate byte array PackedSignMantisaa. Figure illustrates Dynamic-Lenge Float (DFloatl or DF11) 1), our proposed format for compactly representing BFloat16 model parameters.
The Core Challenge: Efficient GPU Inference with Compressed Weights While DFloat enables lossless compression of LLM weights, efficient GPU inference remains a key challenge. Entropy-coded weights use variable-length encoding and cannot be directly used in matrix multiplica- tions. As result, each weight matrix must be decompressed on-the-fly to its original BFloat format

---

Hierarohical
Huffman
(one entry per Bread)
Gaps
3
LUT
LUT
Bybes
ni Bylee
Byles
LUTZ
Encoded Exponents
poca
000311
00001 001 ...
LUT
119
119
118
118
DERI
by Block Thread
Decoded by D, Thread
Block by
1st Exponent
9184-th Exponent
LUT
117
917
121
116
Block Outpuf
Blagy Coie
Eupneet
Eaposcon
Positions
9183
(one onlyy per block)
Figure (Left) The Huffman tree is decomposed into set of non-overlapping subtrees, each corresponding to compact lookup table (LUT). These hierarchical LUTs reside in GPU SRAM to enable efficient Huffman decoding via array lookups. (Right) Each thread decodes n bytes of encoded exponents. The array Gaps stores the bit offset of the first element assigned to each thread, while the array Block Output Positions stores the index of the first element for each thread block.
before matrix multiplication, then discarded immediately after use conserve memory. However, traditional Huffman decoding is inherently sequential, requiring bit-by-bit tree traversal for each element, which ill-suited for GPUs parallel architecture. Naively assigning single thread for decompression leads to poor utilization and high latency. Addressing this bottleneck is essential for practical compressed inference.
In the following paragraphs, we present our solution in detail: a set of hardware- aware algorithmic designs tailored for low-latency decoding of entropy-coded weights in a massively parallel manner. Our approach consists of three key components: leveraging compact lookup tables that fit within GPU SRAM for efficient, lookup-based decoding, introducing two-phase kernel design to coordi- nate read/write operations for all threads using lightweight auxiliary variables, and @ performing decompression the transformer block level minimize latency.
2.3.1 Efficient Decoding with Hicrarchical Lookup Tables
The traditional approach decoding Huffman codes involves reading the encoded bitstream bit by bit and traversing the Huffman tree accordingly. However, this method is inefficient on GPUs due to frequent branching and limited parallelism. To enable efficient decoding on GPUs, we adopt lookup-table-bit basse approach (53)
Assume the maximum Huffman code length is L, and construct lookup table LUT of size 2 At each index i, LUT stores the decoded exponent whose Huffman code matches the prefix of the binary representation of i. To decode the next exponent, bitstream, interpret them as an index into LUT, and retrieve the corresponding value. To determine how many bits to advance in the stream, we use secondary lookup table CodeLengths, which maps each exponent to the length of its Huffman code. A detailed example of this decoding process is provided in Section the Appendix.
read the next L bits from the encoded
In practice, the value of L can be large. For LLMs, typically ranges from 24 to 32, resulting in a LUT with up to 232 entries, which cannot fit within GPU SRAM for fast lookups. To address this, we decompose the monolithic LUT into a hierarchy of compact lookup tables [53]. We first partition the Huffmmnn tree into non-overlapping subtrees of height 8. Each subtree corresponds compact LUT that decodes 8 bits, requiring only 2 256 entries.
Figure 3 shows an example of how a Huffman tree of height 4 can be decomposed into hierarchy of compact LUTs, each with 4 entries. Because the LUTs are organized hierarchically, some entries must serve references to other LUTss lower in the hierarchy. We take advantage of the sparsity in 8-bit exponent usage: although 256 values are available, typically only around 40 are used in LLMs (see Figure 9 in the Appendix). We repurpose unused values (specifically, the range 240 to 255) as pointers to other LUTSs. These values correspond to extremely large magnitudes (+-2113 to +2128 that do not occur in LLM weights, making them safe for use internal markers.
5

---

We use & to denote the number of compact LUTs. In our experiments, we observe that l ranges from 4 to 8 for the Huffman trees built from BFloat16 exponent values. Combined with CodeLengths. these LUT's occupy at most (8 + 1) x 256 bytes of memory, which easily fits within SRAM and allows for fast repeated lookups.
2.3.2 Two-Phase Kernel and Lightweight Auxiliary Variables
To leverage the parallel processing capabilities of GPUs, we assign each thread to a contiguous, non-overlapping block of encoded exponents consisting of n bytes (n = 8 in our experiments), Each thread decodes elements whose Huffman codes begin within its assigned block. Since Huffman codes are variable-length, thread may need to skip some bits at the start before decoding the first element. Similarly, the last element may span beyond the assigned byte range.
This approach introduces two key challenges: 1. The starting bit position for each thread is unclear due to the variable-length nature of Huffman codes. 2. Except for the first thread, the index of decoded elements is unknown, making it difficult to determine their correct output locations.
To address the first issue, array Gaps has one entry per thread, where each entry indicates the offset of the first valid Huffman code relative to the thread's 's assigned starting byte. With maximum code length of 32 bits, each offset lies in [0, 31] and is stored using only 5 bits.
gap array [53] to specify the starting bit offset for each thread. The
For the second issue, maintaining an output position for each thread is straightforward but memory- intensive. Each position requires 32- integer, and with tens of thousands of threads per weight matrix, this leads to significant overhead, undermining DFloatll's compression benefits. To reduce this overhead, we store the output position only for the first element of each thread block rather than for every thread. Since each block typically contains hundreds to thousands of threads, this optimization reduces the overhead from one 32-bit integer per thread to one per block, making the memory cost negligible. Figure 3 illustrates how the gap and block-level output position arrays encode the metadata associated with the encoded exponents.
To support this design, we implement a two-phase kernel. In the first phase, each thread decodes its assigned block and counts the number of elements, without writing to the HBM. Afterward, threads within a block synchronize to compute per-thread output positions via prefix sum over the element counts. We use the Blelloch algorithm [4] for this step. In the second phase, each thread re-decodes the same block, this time writing decoded values to a write buffer in SRAM at the calculated positions. To avoid redundant global memory access, the encoded exponents are loaded into SRAM before the first pass. Once all decoded exponents are written to SRAM, a single batch of coalesced writes is issued to HBM. Pseudocode for the two-phase kernel is provided in Algorithm of the Appendix.
2.3.3 Thansformer-Block-Leeee Decompression
We now have a complete recipe for decompressing entropy-coded exponents in a massively parallel manner. During inference, the LLM weights stored in DFloat format, along with auxiliary variables (the thread-level gap array and block-level output position array), reside entirely in GPÚ memory. When a weight matrix is needed for matrix multiplication, it is decompressed on-the-fly into the original B6 format. Once the matrix multiplication is complete, the BFloat16 matrix is immediately discarded to conserve GPU memory.
In practice, decompressing a single weight matrix often underutilizes GPU resources due to its relatively small size. As the matrix size increases, decompression throughput improves. Figure 7 illustrates this trend, showing how DFloat decompression scales with matrix size. To capitalize on this, we propose batching the decompression of multiple matrices together to improve throughput and hide latency. Specifically, we decompress all 1 weight matrices within a transformer block as a single batch. This batched decompression occurs right before the forward pass of the transformer block, We also compress the token embedding and language modeling head of L.LMs. Since these matrices are large enough to saturate GPU resources, batching their decompression is unnecessary.

---

Table 1: DF11 statistics for various models. Model sizes are shown before and after compression.
Model
Original + DF11 Compressed Compression Ratio
Avg. Bit Width
Large Language Models
Llama 3.1 8B Instruct
16.06 GB - 10.90 GB
67.84%
10.85
Ilama 3.3 70B Instruct
141.11 GB - 95.40 GB
67.61% 6
10.82
Llama 3.1 405B Instruct
811.71 GB 551.22 GB
67.91%
10.87
Qwen 3 14B
29.54 GB - 20.14
68.17%
10,91
QwQ 32B
65,53 GB - 44.65 GB
68.14%
10.90
Mistral Nemo Instruct
24.50 GB + 16.59 GB
67.74%
10.84
Mistral Small 3
47.14 GB - 31.86 GB
67.58%
10.81
Phi 4 Reasoning Plus
29.32 GB + 19.83 GB
67.64%
10.82
DeepSeck RI Distill Llama 8B
16.06 GB + 10.89 GB
67.81%
10.85
Diffusion Trunsformers
FLUX.1 dev FLUX.11 schnell
23.80 GB + 16.33 GB
68.61%
10.98
23.78 GB -
68.58%
10.97
Stable Diffusion 3.5 Large
16.29 GB - 11.33 GB
69.52%
11.12
Table 2: Comparison of accuracy and perplexity for the BF16 and DF11 models different benchmarks. DF11 compression results absolutely no loss in accuracy or perplexity.
Accuracy
Perplexity
Model
Data Type
MMLU
TruthfulQA
WikiText
C4
BF16
68.010 + 0.375
36.965 + 1.690
8.649
21.677
Llama 3.1 8B Instruct
DF11 (Ours)
68.010 + 0.375
36.965 1.690
8.649
21.677
Experiments
We empirically evaluate the effectiveness of DF11 compression and its GPU inference efficiency. A range of recent LLMs and DMs are compressed from their original BFloat16 format into DF1 and report the resulting compression ratios. We then compare the inference performance of DF1 1-compressed models against their uncompressed counterparts across different GPUs, followed by an ablation study analyze the impact of compression.
Software and Hardware We implement the DF1 decompression kernel CUDA and C++, and integrate it into the HuggingFace Transformers 48 inference framework. We evaluate the inference efficiency of our DF1 models against the original BF16 counterparts. We use the HuggingFac Accelerate framework to support CPU offloading and multi-GPU inference. To assess the performance of the DF11 kernel across different hardware configurations, we run experiments on multiple machines with varying GPU and CPU setups. The hardware specifications for all experimental machines provided in Table in the Appendix.
3.1 Results
DF11 compresses models to 70% size. presennt wide selection of recent LLMs and DMs. Specifically, we apply compression to all weight matrices and token embeddings LLMs and all weight matrices in the transformer blocks of DMs. The models we compress include Llama 3. 1/3.3 (20), Qwen 3 [54). Mistral Nemo/Small 44 45), Phi 4 (Л]
presents the compression factors of DF11 for
DeepSeek RI Distilled (21). Stable Diffusion 3.5 (2). FLUX.1 [32]. DF11 I achieves approximately 70% compression across all models, corresponding to an effective bit width of around 11 bits.
Accuracy and perplexity evaluations confirm DF11 compression is lossless. We verify the lossless property of DF11 compression through a series of accuracy and perplexity evaluations on standard benchmarks. Evaluations are conducted using
[18, reporting
accuracy on MMLU (4) and TruthfulQA (38]. and word-level perplexity on WikiText [41 and
C4 42 The results are shown in Table 2 As demonstrated, the compressed model achieves
identical accuracy and perplexity to the original BF16 counterpart. We also present the text-to-image

---

Table 3: Comparison of peak GPU memory usage and text-to-image generation time for diffusion
transformers in BF16 and DF11, using single A5000 GPU.
Peak GPU Memory (GB)
Generation Time (s)
Model
BF16
DFI1 (Ours)
BF16
DFII (Ours)
Stable Diffusion 3.5 Large FLUX.1 dev
16.44
11.78
66.36 + 0.13
69.08 0.11
23.15
16.72
74.41 + 0.15
78.53 0.18
Original Mode/ (BF16) CPU Offoading
Losslessly Compressed Model [DF11, Ourst GPU-orly
Owen 14B (28G)) APU:A5000 (24G0)
Mistral Small (4BGB) AP0: (40GB))
GPU: 8000 (48GD)
200
+00
5
Owen 148 (28GB)
Mintral Small (48GB) GPU:ATTO (40GB)
(54GB) GPU: 8OOO (48GB)
ASPU:AS00O (24GB)
Lakniy (seoodaetan))
Lakony (oconb25kkk))
12A
Laaeny Oscoortttoaa))
100
755
E
8.0
00
Batch Size
Figure 4: Comparison of throughput (top row) and latency (bottom row) for token decoding using the original BF16 models and their DF1 -compressed counterparts. Portions of the BF16 models are offloaded to the CPU due to GPU memory constraints.
generation results of BF16 and DF11 Stable Diffusion 3.5 same random seed and text prompt, the image generated are pixel-wise identical with the original model.
Large model in Appendix J Given the
DF11 outperforms CPU offloading in inference efficiency. We compare the inference perfor- mance of DF11 and BF16 models across various hardware platforms. Due to memory constraints, BF16 models exceed the capacity of a single GPU and require partial CPU offloading, while DF11 models fit entirely within GPU memory. For fair comparison, we retain most computation on the GPU for BF16 models and offload only necessary components. Latency and throughput are measured after a 100-token warm-up run, followed by decoding 100 tokens from an empty prompt across varying batch sizes. Each configuration is run five times, and we report the average results. As shown in Figure 4 DF11 consistently outperforms BF16 with CPU offloading, achieving 2.31-46.244 lower latency or higher throughput. Multi-GPU comparisons are shown in Figure T0 in the Appendix.
DF11 reduces memory usage for diffusion transformers with minimal latency impact. We assess the impact of DF11 compression on diffusion transformer models by measuring peak GPU memory usage and text-to-image generation latency for an 1024 1024 image across five runs. Neither the BF16 nor DF11 1 models employ CPU offloading. As shown in Table 3 DF11 1 reduces memory consumption by 28.3% for Stable Diffusion 3.5 and 27.8 8% for FLUX.11. The relative increase in latency is small: 4.1% for Stable Diffusion and 5.5% for FLUX.1.
DF11 memory savings enable longer generation lengths. DF11 compression not only can reduce the number of GPUs needed for inference but can also support longer generation under the same VRAM budget. During decoding, the KV cache grows linearly with the number of tokens and quickly becomes a memory bottleneck. Figure GPU memory usage for DF11 1 and BF1 models with batch size as token count increases. DF11 allows 5.70 to 14. 86 more tokens to be decoded before reaching memory limits.
8

---

Original, BF16
Lasslessly Compressed, DF11 (Ours)
Mistral-Nemo-nnstruc (24GB)
GPU: Mastaal-maall228BB-nnniuuu414000
GPU: A5000 (24GB)
(24GBx2)
A5000 Memory Capacity W44 D/O.M. M. aller 2624"16kans"" lakanis
A5000x 2 Memory Capacity
25
50
M K0 O.M älfer 1 1564b takans
G.O.M after
afer
20
41976 tolions
89080 takens
Momary
15
30
10
20
0
10000
20000
30000
40000
20000
40000
60000
80000
Number of Decoded Tokens
Number of Decoded Tokens
Figure 5: Comparison of GPU memory consumption between BF16 models and DF11 counterparts. The DFI1 models support 5.70-14.861 longer context lengths by allowing more GPU memory to be used for storing the KV cache. "O.O.M." means out memory.
Inference Latency Breakdown, Llama 3.1 8B Instruct
Token Batch Size
DFou11 (sun]
BFicanté
Token Betch
DFest loun)
BFloune
100
150
200
Taken Batch Size 16384
tours)
BFFeatt
200
400
1000
1200
1400
1800
Latency (ms)
Forward Token Embeedinng
DF11 Decompression Token Embedding
Forward of Transtomor Block
DF11 Decompreesion of Trareformer Block
Language Modeling
Decompression Language Modaiing Head
Figure 6: Comparison of latency breakdown for DFI1 and BF16 Llama 3.1 Instruct during GPU inference for different token batch sizes, using one A100-40GB GPU.
3.2 Ablation Study
Latency breakdown shows decompression overhead is amortized at larger batch sizes. We analyze the latency of Llama 3. 8B Instruct BF16 and DF11 formats across varying token batch sizes on an A100-40GB GPU. For each setting, we measure the average latency of each component over 10 runs, as shown in Figure 6 DF11 introduces additional latency from decompressing the token embedding, transformer blocks, and language modeling head. This overhead is constant and independent of batch size, so increasing the token batch size effectively amortizes the cost.
DF11 decompression is significantly faster than CPU-to-GPU transfer and nyCOMP ANS. We compare DF11 decompression latency and throughput with two baselines: CPU-to-GPU weight transfer and ANS decompression 12 from NVIDIA's nvCOMP (6) using sliced weight matrices from Llama 3.1 Instruct language modeling head. As shown in Figure DF1 achieves up to 34.95x higher throughput than CPU transfer and up 20.97 x faster decompression than nvCOMP. DF11 also offers a better compression ratio (68%) compared nvCOMP (79%). Moreover, DF11 decompression throughput improves with larger matrix sizes due to better GPU utilization.

---

CPU-o-GPU BF16
nvCOMP ANS Decompression (GPU-olll)
Decompession (GPUUonnly curs)
GPU A5050
G100
GPU:AG000
GFU:A100
240
Throighea (DDO
Trrouphh
120
Maanang
Late diae
mone
doae
Matrix Size
Figure 7: Throughput (left two) and latency (right two) comparisons between transferring BF16 matrices from CPU to GPU and decompressing the same matrices on GPU using the NVIDIA nvCOMP ANS library and our proposed DF11 kernel, across matrix sizes and GPU types.
Related Works
Data Formats for Model Weights Full-precision model weights are typically stored in formats such as BF16, FP16, or FP32. Several works have proposed 4-bit compressed formats, including FP4, INT4, NF4 (NormalFloat) 19), AF4 (AbnormalFloat) (58). and SF4 (Student Float) [[]] which represent each parameter with bits. Unlike these lossy formats, the proposed DF11 format compresses weights losslessly.
Lossless Model Compression While lossy compression methods such as pruning 14 and quan- tization [3 [5] are well-studied, lossless compression remains less explored. Four prior works have addressed this area. Deep Compression [22] applied Huffman coding (2 quantized CNNs,
achieving 22% additional compression. ZipNN [25 extended this approach to language models with improved compression over classical methods. However, both techniques target storage efficiency and do not support inference-time gains. NeuZip (23 is the only prior work supporting GPU inference. It uses Asymmetric Numeral Systems (ANS) with layer-wise decompression and relies on NVIDIA's nyCOMP for GPU-based operations. nvCOMP is no longer open source, and its binary-only distribu- tion limits adoption. Moreover, as shown Figure nvCOMP ANS incurs higher latency and lower throughput compared our DFloat 1 kernel. Huff-LLM (59] is designed for FPGA-like hardware and is not applicable to GPUs. Additional discussion of related formats is presented in Appendix B
Conclusion
We introduce Dynamic-Length Float (DFloat]1), I), lossless compression framework designed for efficient GPU inference of BFloat16 models, including both large language models (LLMs) and diffusion models (DMs). DFloat 1 exploits the information redundancy inherent in foundation model weights through entropy-coded, dynamic-length encoding, achieving compression rates close limit. To enable efficient deployment, we develop hardware-aware
to the informationntheereet algorithms that support high-speed inference directly on compressed weights. Extensive experiments demonstrate that DFloat I significantly reduces GPU memory requirements for LLMs and DMs, allowing for longer generation lengths, while maintaining bit-exact accuracy and incurring only negligible decompression overhead.
Acknowledgements
This work was supported by National Science Foundation SHF-2211811 22115 and Ken Kennedy Insti- tute Cluster Grants. Additionally, Henry and Xia are supported by IT2-2429680, IIS-2310260, and US Department of Transportation (USDOT) Tier-1 University Transportation Center (UTC) Transportation Cybersecurity Center for Advanced Research and Education (CYBER-CARE) grant 4355333333333.. Mohsen and Vipin are supported by OAC-2320952, OAC-21 12066, and OAC- 2117439. The views and conclusions this paper are those of the authors and do not represent the views of any funding supporting agencies.
10

---

References
[1] Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J Hewett, Mojan Javaheripi, Piero Kauffmann, et al. Phi-4 technical report. arXiv preprint arXiv:2412.08900. 2024.
[2] Stability AL.
Introducing stable diffusion 3.5. https://stabilityy htw.:iiiitt.....e.aa..aaeae ai/news/ introducing- ttable-diffusien
October 2024. Accessed: May 15, 2025.
[3] Anonymous. FAFO: Lossy KV cache compression for lossless inference acceleration via draftless fumble decoding. In Submitted to The Fourteenth International Conference on Learning Representations, 2025. under review.
[4] Guy E Blelloch. Scans as primitive parallel operations. IEEE Transactions on computers, 38(11):1526-15) 1989.
[5] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jeny Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint 1rXiv:2110.14168, 2021.
[6] NVIDIA Corporation. nvCOMP: Gpu-accelerated compression and decompression library. https://develope: nviiia. com/nvcomp) 2025. Accessed: April 11, 2025.
[7] Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-efficient exact attention with io-awareness. Advances in neural information processing systems, 35:16344-1635, 2022.
[8] Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Gpt3, int8 (): 8-bit matrix multiplication for transformers at scale. Advances in neural information processing systems, 35:30318 1130318--0332, 2022.
[9] Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized 1lms. Advances in neural information processing systems, 36:10088- 10115, 2023.
[10] P. Deutsch and J.-L.. Gailly. Rfc1950: Zlib compressed data format specification version 3.3, 1996.
[11] Jordan Dotzel, Yuzong Chen, Bahaa Kotb, Sushma Prasad, Gang Wu, Sheng Li, Mohamed S Abdelfattah, and Zhiru Zhang. Learning from students: Applying t-distributions to explore accurate and efficient formats for llms. In International Conference on Machine Learning, pages 11573-11599 11591. PMLR, 2024.
[12] Jarek Duda. Asymmetric numeral systems: entropy coding combining speed of huffmman coding with compression rate of arithmetic coding. arXiv preprint arXiv: 2013.
[13] Abhinav Dutta, Sanjeev Krishnan, Nipun Kwatra, and Ramachandran Ramjee. Accuracy is not all you need. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
[14] Elias Frantar and Dan Alistarh. Sparsegpt: Massive language models can be accurately pruned in one-shot. In International Conference on Machine Learning, pages 10323-10337. PMLR, 2023.
[15] Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. Gptq: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210.17323,, 2022.
[16] Yichao Fu, Peter Bailis, Ion Stoica, and Hao Zhang. Break the sequential dependency of LLM inference using lookahead decoding. In Forty-first International Conference on Machine Learning, 2024.
[17] Kazuki Fujii, Taishi Nakamura, and Rio Yokota. Balancing speed and stability: The trade-offs of fp8 vs. bf16 training in llms. arXiv preprint arXiv:2411.08779, 2024.

---

[18]
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang. Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, 07 2024.
[19] Ruihao Gong, Yang Yong. Shiqiao Gu, Yushi Huang, Chengtao Lv, Yunchen Zhang, Xianglong Liu, and Dacheng Tao. Llme: Benchmarking large language model quantization with a versatile compression toolkit. arXiv preprint arXiv:2405.06001, 2024.
[20] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:240..2178. 2024.
[21] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprini arXiv:2501. 2025.
[22] Song Han, Huizi Mao, and William Dally. Deep compression: works with pruning, trained quantization and huffman coding. arXiv preprint 1rXiv:1510.00149, 2015.
Compressing deep neural net-
[23] Yongchang Hao, Yanshuai Cao, and Lili Mou. Neuzip: Memory-efficient training and inference with dynamic compression of neural networks. arXiv preprint arXiv:2410. 2024.
[24] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Mcasuring massive multitask language understanding, In International Conference Learning Representations.
[25] Moshik Hershcovitch, Andrew Wood, Leshem Choshen, Guy Girmonsky, Roy Leibovitz, Ilias Ennmouri, Michal Malka, Peter Chin, Swaminathan Sundararaman, and Danny Harnik. Zipnn: Lossless compression for ai models. arXiv preprint arXiv:2411.052399 2024.
[26] Coleman Richard Charles Hooper, Sehoon Kim, Hiva Mohammadzadeh, Michael W. Mahoney, Sophia Shao, Kurt Keutzer, and Amir Gholami. KVQuant: Towards 10 million context length LLM inference with KV cache quantization. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
[27] Edward Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022.
[28] David A Huffman. A method for the construction of minimum- codes. Proceedings of the IRE, 4099):1088-110,, 1952.
[29] Renren Jin, Jiangeun Du, Wuwei Huang, Wei Liu, Jian Luan, Bin Wang, and Deyi Xiong. A comprehensive evaluation of quantization strategies for large language models. In Findings of the Association for Computational Linguistics ACL 2024, pages 12186-12215, 2024.
[30] Dhiraj Kalamkar, Dheevatsa Mudigere, Naveen Mellempudi, Dipankar Das, Kunal Banerjee, Sasikanth Avancha, Dharma Teja Vooturi, Nataraj Jammalamadaka, Jianyu Huang. Hector Yuen, et al. A study of 6 for deep learning training. arXiv preprint arXiv: 1905. /2322, 2019.
[31] Artyom Kharinaev, Viktor Moskvoretski,, Egor Shvetsov, Kseniia Studenikina, Bykov Mikhail, and Evgeny Burnaev. Investigating the impact of quantization methods the safety and reliability of large language models. arXiv preprint arXiv:2502. 2025.
[32] Black Forest Labs. Flux.
http:///attuuu..cooocaaccc-aaca. labss/fftlabb/flus 2024.
[33] G. G. Langdon. An introduction to arithmetic coding. IBM Journal of Research and Develop- ment, 28(2):135-149, 1984.
[34] Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. In Fortieth International Conference on Machine Learning, 2023.
12

---

[35] Shiyao Li, Xuefei Ning, Luning Wang, Tengxuan Liu, Xiangsheng Shi, Shengen Yan, Guo- hao Dai, Huazhong Yang, and Yu Wang. Evaluating quantized large language models. In International Conference on Machine Learning, pages 28480-28522 PMLR, 2024.
[36] Xiuyu Li, Yijiang Liu, Long Lian, Huanrui Yang. Zhen Dong, Daniel Kang, Shanghang Zhang, and Kurt Keutzer. Q-diffusion: Quantizing diffusion models. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 17535-17545, 2023.
[37] Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang. Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. Awq: Activation-aware weight quantization for on-device IIm compression and acceleration. Proceedings of Machine Learning and Systems, 6:87-100, 2024.
[38] Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Mcasuring how models mimic human falsehoods. In Proceedings of the 60th Annual Meeting of the Association for Computa- tional Linguistics (Volume 1: Long Papers), pages 3214-3252, 2022.
[39] Ruikang Liu, Yuxuan Sun, Manyi Zhang, Haoli Bai, Xianzhi Yu, Tiezheng Yu, Chun Yuan, and Lu Hou. Quantization hurts reasoning? empirical study on quantized reasoning models. arXiv preprint arXiv:2504.04823 2025.
[40] Zirui Liu, Jiayi Yuan, Hongye Jin, Shaochen Zhong, Zhaozhuo Xu, Vladimir Braverman, Beidi Chen, and Xia Hu. KIVI: A tuning-free free asymmetric 2bit quantization for KV cache, In forty-first International Conference on Machine Learning, 2024.
[41] Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. In International Conference on Learning Representations, 2017.
[42] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1667, 2020.
[43] Yang Sui, Yanyu Li, Anil Kag, Yerlan Idelbayev, Junli Cao, Ju Hu, Dhritiman Sagar, Bo Yuan, Sergey Tulyakov, and Jian Ren. Bitsfusion: 1.99 bits weight quantization of diffusion model. arXiv preprini 4xXiv:2466.04333, 2024.
[44] Mistral AI Team. Mistral NeMo. http::/miistral.aa/eee//saaaaaa
July 2024.
[45] Mistral AI Team. Mistral Small 3. http:://mmitttraalaaaa/ee//aaaas 2025.
January
[46] Qwen Team. Qwen3: Think deeper, act faster, April 2025.
[47] Zhuang Wang, Zhen Jia, Shuai Zhang, Zhen Zhang, Mason Fu, T. S. Eugene Ng, and Yida Wang. Gemini: Fast failure recovery in distributed training with in-memory checkpoints. 2023.
[48] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Transformers: State- of-the-art natural language processing. In Proceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations, pages 38 45,45,, 2020.
[49] Heming Xia, Tao Ge, Peiyi Wang, Si-Qing Chen, Furu Wei, and Zhifang Sui. Speculative decoding: Exploiting speculative execution for accelerating seq2seq generation. arXiv preprint arXiv:2203. 1487, 2022.
[50] Heming Xia, Zhe Yang, Qingxiu Dong, Peiyi Wang, Yongqi Li. Tao Ge, Tianyu Liu, Wenjie Li. and Zhifang Sui. Unlocking efficiency in large language model inference: A comprehensive survey of speculative decoding. arXiv preprint arXiv:2401 007851, 2024.
[51] Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wa, Julien Demouth, and Song Han. Smoothquant: Aceurate and efficient post-training quantization for large language models. In International Conference on Machine Learning, pages 38087-38099. PMLR, 2023.
13

---

[52] Mengwei Xu, Wangsong Yin, Dongqi Cai, Rongjie Yi, Daliang Xu, Qipeng Wang, Bingyang Wu, Yihao Zhao, Chen Yang, Shihe Wang, al. A survey of resource-effiient Ilm and multimodal foundation models. arXiv preprint wrXiv22401000992 2024.
[53] Naoya Yamamoto, Koji Nakano, Yasuaki Ito, Daisuke Takafuji, Akihiko Kasagi, and Tsuguchika Tabaru. Huffman coding with gap arrays for gpu acceleration. In Proceedings of the 49th International Conference Parallel Processing, pages 2020.
[54] An Yang. Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint 1.11:24422.5115, 2024.
[55] Ge Yang. Changyi He, Jinyang Guo, Jianyu Wu, Yifu Ding, Aishan Liu, Haotong Qin, Pengliang Ji, and Xianglong Liu. LLMCBench: Benchmarking large language model compression for efficient deployment. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024.
[56] Jingfeng Yang. Haongye Jin, Ruixiang Tang, Xiaotian Han, Qizhang Feng, Haoming Jiang, Shaochen Zhong, Bing Yin, and Xia Hu. Harnessing the power Ilms in practice: A survey on chatgpt and beyond. 2024.
[57] Ling Yang. Zhilong Zhang, Yang Song, Shenda Hong, Runsheng Xu, Yue Zhao, Wentao Zhang, Bin Cui, and Ming-Hsual Yang. Diffusion models: A comprehensive survey of methods and applications. ACM Computing Surveys, 56(4):1-39, 2023.
[58] Davis Yoshida. Nf4 isn't information theoretically optimal (and that's good). arXiv preprint arXiv:2306. 06965, 2023.
[59] Patrick Yubeaton, Tareq Mahmoud, Shehab Naga, Pooria Taheri, Tianhua Xia, Arun George, Yasmein Khalil, Sai Qian Zhang, Siddharth Joshi, Chinmay Hegde, et al. Huff-lIm: End-to-end lossless compression for efficient Ilm inference. arXiv preprint arXiv:2502.0092. 2025.
[60] Haochen Zhang, Junze Yin, Guanchu Wang, Zirui Liu, Lin Yang, Tianyi Zhang, Anshumali Shrivastava, and Vladimir Braverman. Breaking the frozen subspace: Importance sampling for low-rank optimization in LLM pretraining. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025.
[61] Tianyi Zhang and Anshumali Shrivastava. Leanquant: Accurate and scalable large language model quantization with loss-error-aware grid. In The Thirteenth International Conference on Learning Representations, 2025.
[62] Tianyi Zhang, Junda Su, Aditya Desai, Oscar Wu, Zhaozhuo Xu, and Anshumali Shrivas- tava. Sketch to adapt: Fine- Fine-tunable sketches for efficient LLM adaptation. In Forty-second International Conference on Machine Learning, 2025.
[63] Tianyi Zhang, Jonah Wonkyu Yi, Zhaozhuo Xu, and Anshumali Shrivastava. KV cache is I bit per channel: Efficient large language model inference with coupled quantization. In The Thihth-eighth Annual Conference on Neural Information Processing Systems, 2024.
[64] Tianyi Zhang, Jonah Wonkyu Yi, Bowen Yao, Zhaozhuo Xu, and Anshumali Shrivastava. NoMAD-attentonn: Efficient LLM inference on CPUs through multiply-add-fres attention. In The Thihty-eighth Annual Conference on Neural Information Processing Systems, 2024.
[65] Jiawei Zhao, Zhenyu Zhang. Beidi Chen, Zhangyang Wang, Anima Anandkumar, and Yuandong Tian. Galore: Mcmory-effit effficient LLM training by gradient low-rank projection. In Forty-first International Conference on Machine Learning, 2024.
14

---

Appendix
A Discussion: Is Quantization Universal Solution?
Much of the motivation behind our work lies in understanding whether lossless compression of large-scale models such as LLMs, which preserves 100% identical output behavior compared to the original uncompressed model, is a practical direction worthy of further study. Specifically, how does DFloatl I, which compresses LLMs to approximately 11 bits, compare to widely used lossy quantization techniques [15 37]. where models are typically reduced to even lower bit-widths (e.g.. 8-bit or 4-bit)?
The answer is far more nuanced than a simple "Yes/No" or one-size-fits fits-all judgment about which approach is better. For instance, existing benchmark studies like [19 55 29] often suggest that 8-bit (weight-only or not) quantization is relatively "safe" compression scheme. Although technically lossy, 8-bit models can often maintain strong task performance across a range of standard benchmarks. However, we must note these benchmarks typically focus on narrow set of tasks (e.g., WikiText2 perplexity, MMLU, Commonsense Reasoning), and thus fail to offer comprehensive view of real-world LLM usage, especially from the perspective of end-users.
That being said, the argument that "current benchmarks fail to capture the performance gap between 8- bit compressed and 16-bit uncompressed models" is itself constrained by the limitations of the current benchmarking landscape, making it difficult to produce abundant supporting evidence. Nonetheless. some reports have begun to highlight such gaps. For example, human evaluations on LLM Arena show a notable performance drop between (Llama-3.1-400)
1-5.1---0----nrruu [20] and its W8A8 counterpart -FP)) particularly under coding (1293 vs. 1277) and oong- (1282 vs. 1275) tasks. Similarly, quantizing -eepSeee-RR--istlllee DllstillLLlama-000 70B 21 from 16 bits 8 bits results in 23.7% drop on GPQA (from 9.51% to 7.25%) Furthermore, reasoning, a core capability of modern LLMs, appears especially sensitive to compression loss. Recent benchmark [39] reveals that quantizing DeepSeek-R -iiii--R---itti---eieeee 1.5B with 8-bit SmoothQuant [5I] (for weight, attention, and KV cache) leads to an average 9.09% drop reasoning tasks 4.8. to 44.29%) across datasets like AIME, MATH-500, GPQA-Diamond, and BiveCodeBench. We leave more evidence exploring
the performance gap between 8-bit quantized and uncompressed model in Appendix H
Although the broader question: "Which specific task, on which model, using which quantization technique, under what conditions, will lead to a noticeable drop compared to FP16/BF16?" is likely to remain open-ended simply due to the sheer amount of potential combinations. It is fair to say that lossy quantization introduces complexities that some end-users would prefer to avoid, since it creates uncontrolled variables that must be empirically stress-tested for each deployment scenario.
To eliminate this burden, DFloat1 I offers compelling alternative: delivering 100% identical performance to the original model, while consuming only ~70 % of the memory footprint with throughput benefits, which is unique and practical offering for resource-constrained deployment settings.
B Extended Related Works
Data Formats for Model Weights LLM weights are typically stored in compact floating point formats such as FP16 or BF (officially stylized as bffoatlds). FP16 allocates I sign bit, 5 exponent bits, and 10 mantissa bits, whereas 6 uses sign bit, 8 exponent bits, and 7 mantissa bits. Compared to FP16, B6 offers wider dynamic range at the cost of precision, which improves numerical stability and mitigates overflow issues during training [7]00]
Compressed data formats typically aim for lower bit-w..ths. For example, 1Pp8-which comes in both E4M3 (4 exponent bits, mantissa bits, plus sign bit) and E5M2 configurations has seen reasonable adoption in L.M training and development. Integer formats like INT8 have also been well explored, as in LLM. int8() ()())) 8 and its following works. Formats with stronger emphasis on efficiency, such
http:///sso//aaaeaa////////////a htte:a///uegiiiggocoo///oollllss htt.::aaaa....aa.aaaaaaaa0a-aaaa w8a8
15

---

as FP4, INT4, NF4 19. and AF4 [58, use only 4 bits. In this work, we primarily focus on formats with 28 bits, as benchmark literature [55] T9 39 often suggests that 8-bit quantization results in negligible performance though we show in Section that this claim is likely skewed due to evaluation selectiveness and benchmark limitations.
Lossless Model Compression
While lossy model compression techniques such as pruning and quantization [4 37 15] have received widespread attention, lossless model compression remains a relatively underexplored area. Upon careful investigation, we identified roughly four prior works that have made meaningful efforts in this space. Deep Compression [22) is a foundational work, applying Huffman coding [28] to quantized CNN models and achieving an additional ~222% compression gain
for model checkpoints. ZipNN (25] extended this idea to language models, comparing its results to
classic lossless compression tools such as zlib [10] and zste and demonstrated superior compression gains. However, this line of work-including their industry counterparts, such as ezm is limited in that its efficiency gains only apply to storage (reducing the size of model checkpoints) but offer no benefits during inference. While such storage savings are meaningful in large-scale scale training sastings-wertd frequent snapshotting and checkpoint rollbacks are needed [47]- have limited impact for everyday LLM end- users. Model downloading is typically one-time cost, so even if model checkpoint is compressed by 50%, it only cuts the download time at most by half, presumably over the model's Is entire lifecycle of deployment. Furthermore, checkpoints where terabytes of capacity are easily available, making up a much looser constraint compared to GPU HBM (High Bandwidth Memory); one of the main resource constraints during inference.
usually stored disk,
We argue that a lossless compression technique would be substantially more impactful it could deliver efficiency gains during inference liference-particulaann particularly on GPU-based systems, which is the default setup for LLM serving. In this context, NeuZip [23] is the only prior work we identify that supports GPU inference. NeuZip applies entropy encoding with layer-wise decompression to maintain reduced memory footprint throughout serving. However, it is built on NVIDIA's nvCOMP: "a high- speed data compression and decompression library optimized for NVIDIA GPUs" Unfortunately, nyCOMP is no longer open-source (only binary executables are available), which hinders future research. Moreover, we empirically find that nvCOMP's inference throughput and latency significantly worse than our proposed DFloatl kernel, resulting in a pipeline that trades memory efficiency for substantial inference overhead (see Figure 7
Another work referencing NeuZip is Huff-LLM [59], which also aims to reduce memory costs while maintaining efficient inference. However, its contributions and do not apply to GPUs. To the best of our knowledge, the DFloat data format we presented (and its respective kernel support in DFloat 1) shall serve the only PPUU-iffrenne-frcee friendll data format with lossless compression benefits.
specific FPGA-like architectures
Efficient LLM Inference LLMs are computationally intensive and resource-demanding, making the efficiency of LLM inference a key research focus (52). FlashAttention 77) accelerates exact attention computation on GPUs through kernel fusion, while NoMAD Attention [64] speeds up
attention on CPUs using in-register lookups. Model compression is another effective strategy to reduce resource requirements for serving LLMs and diffusion models. Quantization methods such as 37 SmoothQuant [5T] LeanQuant [1]. CQ 63 KVQuant (26). and KIVI 40) lower memory usage and enhance efficiency by compressing model weights, activations, or KV cache. Compression
GPTQ 13), AWQ
also applied in fine-tuning: methods like LoRA (27) QLoRA (9) and
TuetchTune [62] compress model weight deltas, whereas GaLore (5 and SARA (60] compress
optimizer states during training. One additional line of work relevant efficient LLM inference would be lossless efficient decoding, where paradigms such as speculative decoding [49
34 50
and n- r-gram candidate decoding 116 3] offer lossless generation quality with improved latency. DFloat 1 mainly differs from these works in that it provides substantial savings in memory footprint while maintaining lossless generation quality, whereas most-if not lls-eessless efficient decoding methods require memory consumption equal to or greater than that of the original model.
http:://gicibbbbcc//cccoooo////t
https://encode: su/ threads/
4067 Todd--oooddccpppeeeeo-o---------
https://developer.ut .vidia. cociiiia.co///cooom
16

---

Liama 8B
Freginey
65%
105
Roo kolas
Riw Kila
Rew Vile
Rin
2.5
2003
Sixzul Smeal 248
5.0%
190
95
Lam 703
100%
Kilun
Vike
Ron Voltn
Sign (1
Expanant 18 bits)
Mantissa (7 bits)
Figure 8: Relative frequency distribution of sign, exponent, and mantissa values in the 1F weights of all linear projection layers across various LLMs.
C Frequency Distribution of BFloat16 Values
Figure 8 presents the frequency distribution for distinct values of sign, exponent, and mantissa bits in the BFloat16 weights of LLMs. Figure shows the sorted frequency of exponent values of LLM weights.
Liana 3.1
Gamma 28
Cwvin 2.5 148
sasal Small
Liama 121
10"
10P
Exporeat Mark
Expeeed Paakk
Paaks
Boonard Rark
Figure Distribution of 6 exponent values across various models. The frequency of exponent values (shown in log scale) decays rapidly with exponent rank.
D Pseudo-code of the GPU kernel for DFloat11 Decompression
Algorithm D presents the pseudo-code of the two-phase GPU kernel for decompressing DFloat I to BF.
Table 4: System specifications of servers used for experiments.
GPU
GPU Memory CPU
CPU Memory
Server 1
NVIDIA RTX A5000
24564MiB
AMD EPYC 7513 32-Core
504GB
Server 2
NVIDIA A100
40960MiB
AMD EPYC 7742 64-Core
1.48TB
Server 3
NVIDIA Quadro RTX 8000
49152MiB
AMD EPYC 7742 64-Core
1.48TB
17

---

Algorithm GPU kernel for decompressing DFloat1 I to BF
I: procedure DF1 1ToBF16
require:
EncodedExponeet, PackedSignMaantisa:: byte arrays
LUT ...... LUTx, CodeLengths: 8-bit unsigned integer arrays of size 256
Gaps: 5-hit unsigned integer arry (one entry per thread in each block)
BlockOutputPos: 32-bit unsigned integer array (one entry per block)
Outputs: BF array, for storing results
n,, n, A: the number of thread blocks, numher of threads, number of bytes processed by each thread,
number compact LUTs,, respectively
2: Divide EncodedExponent into chunks:
EncodedExponentt EncodedExponent, of size n' bytes cach
1........ l..... B (in parallel across blocks) do Load EncodedExxoonett, into SRAM
Divide EncodedExponent, into chunks:
EncodedExponentt.u
Encoded of size n bytes each
6:
Load LUT LUT CodeLengths into SRAM
7:
Initialize integer arrays NumElements) . ThreadOOutputPos[ -T] with all Os Initialize 6F write buffer WriteBuffer in SRAM for all ,,.., T (in parallel across threads) do
8:
Phase Each thread determines
initial output position
10:
BitOffist <- Gaps[5T ++]
11:
while BitOffset < 8n do
12:
Read the next bytes of EncodedExponentt.r starting from the BitOffiset-th bit, into
Byte
13:
14:
Exponent + LUT, (Byte]
15:
while Exponent 2 240 do
Exponent 2 240 means that it pointer the next LUT
i ++1++
17:
Exponent + LUT (277 [Byte,]
18:
end while
19:
BitOffiet + BitOffist + [odeLennghh|EExpoeenj
20:
NumElements({] +- NumElements (
21:
end while
22:
Thread Synchronization Barrier
Compute prefix-sum using Blelloch's Algorithm:
23:
ThreadOutputPos/// <- BlockOutputPos(i) ΣE= NumElements
>> Phase 2: Writing decoded 1 to the appropriate positions
24:
Offset Gaps(bT +
25:
while BitOffset < 8n do
26:
Read the next bytes of EncodedExponent,. starting from the BitOffset-tt bit, into
Byte
27:
28:
Exponent + LUT, [Byte
29:
while Exponent 2 240 do
> Exponent: 2 240 means that it is pointer to the next LUT
30:
1 iti+1
31:
Exponent LUT LUT [Byte,]
32:
end while
Byte PackedSignMantissal Satpgn/Mnnntss/TTeeaaadourrrrrs [ThreadOtppuuPoosss]]]
33:
34:
Sign
< Byte bitwise_and 0b10000000
35:
Mantissa <- Byle bitwise
bitwiseeandn and 0b01111111
36:
WriteBuffe(TTheadOOrrrpoooss bitwise_or
BloocOOtppuPos(b] +
(Sign wwwtwsss_eett_ehheeeeeeeeeeeeeee
(Exponent hettwsse_eef_shhhe 7) bitwise_or Mantissa
37:
BitOffset +- BitOffiet + BoodeeLngths(Exxpoore
38:
ThreadOutputPos({) ThreadOOutputPoss(
39:
end while for b Perform coalesced writes to HBM:
40:
41:
Outputs BlockOutputPos + WriteBuffer(O. (BlockOutputPos() BlockOutputPos 1)]
42:
end for end procedure
43:
18

---

E
Hardware for Experiments
Table 4 presents the hardware configuration of servers used for experiments.
F
DFloat11 Compression Time
Table 5: Compression time per transformer block for different models.
Model
Compression Time per Transformer Block (s)
Llama 3.1 8B Instruct
191
Llama 3.3 70B Instruct
547
Llama 3.1 405B Instruct
2133
Table repoors the time required to compress a single transformer block for models of different sizes. Compression is a one-time preprocessing step for each model and is performed using single CPU thread. Since transformer blocks are independent in terms weight storage, their compression can be parallelized across multiple CPU threads, making the overall process highly scalable and efficient.
BF16 Model
Model
Liama 3.1 88 (16GB) GPU: A100 (40GB)
Owen 148 (28GB)
Liama 3.3 708 4140B) GPU: ATP0:A100:0 A100X4 (4GGB44)
GPU: A100 (40GB)
Thoughout [Toxatt (ocaadd)
Thoughout [Toknn(Baaaddd
Thouughuut Trooons/Seoaan
0000
600
200
150
100
200
134
Owan (28GB) (40GB)
(40GB)
GPU: A100 (40GBx4)
Lavercy (Seconds/Tol T
8:000
Lavercy (Seconds/Token)
0.5
5
3.08
Batch
Figure 10: Comparison of average latency and throughput for token decoding between the original (BF16) models and their losslessly compressed (DF11) counterparts. The BF16 and DF1 I models are run on the same GPU configurations, with Flash Attention [ turned on for both methods.
G GPU Inference Efficiency Comparison: BF16 vs. DF11
We present the GPU inference efficiency of BF16 and DF11 models in Figure 10 for various models and batch sizes on A100 GPUs.
H Impact of Lossy Quantization
An accuracy comparison of the original and INT8- Llama model is presented in table 6
19

---

Table 6: INT8 quantization error on different tasks. "Math" denotes MATH Hard with shots.
"GPQA CoT is with 2 shots. "A" denotes the error gap via INT8 quantization.
Model
Data Type
Math
GPQA CoT
BF16
23.92
15.18
Llama-3.
INT8
19.92
14.06
A
4.0
1.12
Huffman Tree
Internal Node
Lest hoods
Lookup Table (LUT)
Indios
10
11
12
13
14
15
Fax Sit
A
8
C
D
F
Gieary
2nd DH
Code Lengths
3rd BR
3rd DIL
Symbel
4th DI
Tystial
E
Figure 11:
Decoding Huffman codes can be performed either by traversing the Huffman tree or by using two lookup tables: one that maps each L-bit binary code to its corresponding symbol, and another that stores the code length for each symbol.
I Efficient Decoding of Huffman Codes Using Compact Lookup Tables
L1 The Dual Lookup Table Approach
Huffman decoding can be performed by traversing the Huffman tree: starting from the root, each bit of the encoded bitstream determines the branch to follow, and the symbol is fully decoded upon reaching a leaf node. While this bit-by-bit traversal is conceptually simple, it inefficient in practice. Each branching decision depends the previous one, leading to frequent memory accesses and conditional jumps. This pattern is especially problematic on GPUs, where it causes branch divergence and limits instruction-level parallelism. A widely adopted alternative is lookup-table-bused decoding (53), which flattens the Huffman tree into two compact lookup tables. This enables decoding of each symbol using just two array lookups and a bit shift, significantly improving throughput.
We employ two lookup tables, LUT and CodeLengths, to achieve efficient, branch-free Huffman decoding. Let L denote the length of the longest codeword in the Huffman codebook. We construct the primary lookup table LUT as an array of size 2 where each entry maps an L-bit binary sequence to the first symbol it encodes.
Figure TT shows an example with L, = 4 and a set of symbols A, B, C, D, E, For clarity, we use letters to represent symbols, though in practice these correspond to exponent values in BF weights. The lookup table LUT contains 2 = 16 entries, indexed by all possible 4-bit binary sequences. Each entry in LUT stores the symbol whose Huffman code matches the prefix of that index. If symbol's Huffman code is shorter than L bits, it will fill multiple consecutive entries. For example, if symbol is encoded the single bit 0, then all binary sequences from 0000 to 0111 begin with 0, so entries 0 through in LUT are assigned to A. In contrast, symbols with Huffman codes of length L occupy exactly one entry each. For instance, E 1110 and F 1111 map to entries 14 and 15, respectively. This construction yields a dense prefix table that allows decoding a symbol with a single array lookup using L-bit segment from the encoded bitstream.
To advance the encoded bitstream for decoding the next symbol, we also store the code lengths of all symbols. The second lookup table, CodeLengths, maps each symbol to its Huffman code length. In
20

---

Compact Lookup Tables (LUTs)
Huffman Tree
LUT
Indeax:
LUT
0
A
LUT
LUT
Binary Code
01
10
11
Desoked Sprciall
BH
LUT
B
B
C
C
3rd Bit
3rd Bit
Binary Gode
00
01
10
B
LUT 2
D
E
F
Binary Gode
01
10
Decnder Symte
Depode Sprterd
Figure 12: A Huffman tree can be decomposed into hierarchy of subtrees, each represented by compact lookup table (LUTT). Each LUT may reference another lower-level LUT in the hierarchy. This hierarchical decoding approach functionally equivalent to using single monolithic LUT,, but significantly more memory efficient.
the example, the lengths are: A:1, B:3, C:3, D:3, E:4, F:4. Together, these two tables allow fast, deterministic decoding by repeating the following steps:
1. Use the next L bits from the encoded bitstream to index LUT and retrieve the decoded symbol.
2. Look up the code length of the decoded symbol from CodeLengths determine how many bits to consume.
3. Advance the encoded bitstream and repeat.
This approach eliminates conditional branches and pointer chasing during decoding, making it highly suitable for parallel computation on GPUs.
I.2 Decomposing LUT into Hierarchical, Compact Lookup Tables
The primary lookup table LUT contains 2 entries, where L is the maximum code length in the Huffman codebook. While this enables constant-time decoding, the table size grows exponentially with In practice, ranges from 24 to 32 for Huffman trees built with BF exponents. This results in table sizes of 2 to 2 entries, which far exceeds the capacity of GPU SRAM. To address this, we decompose LUT into multiple smaller lookup tables that fit within on-chip memory, while still enabling fast decoding.
Hierarchical Table Structure Instead of storing single flat table of size 24, we decompose LUT into a hierarchy of compact lookup tables. Each table corresponds to subtree of the Huffman tree and is responsible for decoding bits. Each table processes the next b bits and either (i) directly returns decoded symbol, (ii) delegates to table next in the hierarchy for decoding the next b bits. This hicrarchical organization mirrors the structure of the original Huffman tree and significantly reduces total memory usage.
illustrates an example where the Huffman tree partitioned into three subtrees, each mapped to a separate lookup table responsible for 2 bits. The decoding process using these three LUTss proceeds as follows:
Figure 12
LUT,: Uses the first and second bits of the encoded bitstream to determine how to proceed, leading to possible cases:
- 00, 01 - decode the next symbol as A.
10 - delegate to LUT,.
21

---

11 + delegate to LUT2.
LUTI: Uses the third and fourth bits of the encoded bitstream to continue decoding:
00, 01 - decode next symbol as
- 10, 11 + decode the next symbol as C
LUT2: Uses the third and fourth bits of the encoded bitstream continue decoding:
-0, 01 - decode the next symbol as
10 - decode the next symbol as
- 11 + decode the next symbol as F
For decoding Huffman-cond coded 6 exponents, we decompose the LUT into multiple compact lookup tables, each responsible for decoding bits (i.c. b 8). This allows read the next byte from the encoded bitstream and perform an array lookup from a 256-entry array in each step. In practice, the decomposition of LUT leads to 4 to 8 compact LUTs, each with 256 entries, which comfortably fits within fast SRAM.
J Text-to-image Results of BF16 and DF11 Diffusion Models
Figure 13: Images generated by Stable Diffusion 3.5 Large in the original BFloat16 precision (top 5) pixel-wise identical to those produced by the DFloat1 l-compressed model (bottom 5), using the same prompt and random seed.
Figure 13 presents the comparison of images generated using Stable Diffusion 3.5 Large BFloat 16 and D weight format. The images are pixel-wise identical, when using the same prompt and random seed.
K
Limitations
This work focuses exclusively on losslessly compressing BFloat16 weights. We do not consider other formats such as FP32, FP16, or FP8, which may require different compression strategies. While DF11 improves memory efficiency, it introduces small but non-zero latency overhead due to decompression. This overhead amortized at larger batch sizes but may impact latency- applications with small batches. Our evaluation is limited to GPUs. We do not assess performance other hardware such as CPUs, TPUs, or custom accelerators, which may require -seeciic-ppecfficc optimizations.
22

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The main claims accurately reflect the paper's contributions and scope, as supported by evidences in the Method and Experiments section.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We have included Limitations section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
23

---

Justification:
This paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We have fully disclosed all information needed for result reproduction.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
24

---

Answer: [Yes]
Justification:
Our code and models are publicly available.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https Cooeeeoodeesssseeeeee 7/nips.com htte://ulllllliiiee/e//eeeuuaass ssiniioiicyp for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification: This paper does not include training experiments. All evaluation details necessary reproduce and understand the results provided in the full paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification:
We include error bars for experiments where appropriate.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
25

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We have provided detailed information on the computational resources required reproduce our results.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: After reviewing the NeurIPS Code of Ethics, we have verified that our research is in compliance.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations their jurisdiction..
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This work focuses on improving the inference memory efficiency of existing pre-trained models and does not involve the development of new model, training data, or training method, which limits its direct societal impact.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
26

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We do not introduce new data or models.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
The assets used are properly credited.
Guidelines:
The answer NA means that the paper does use existing assets. The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswithcone com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
27

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer:
[NA]
Justification:
No new assets are introduced by this paper.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
28

---

16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification: LLMs are not involved in any originl or important parts of this research.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to L.LM policy for what should or should not be described.
29


Reviews:
[{"review_id": "IneYwLg2Pk", "reviewer": "Reviewer_uiTc", "summary": "The paper introduces Dynamic-Length Float (DFloat11), a lossless compression method for storing BF16 model weights by applying Huffman coding only to the exponents. This reduces the original 8-bit exponent representation to an average of 2.6 bits. Additionally, the paper proposes an efficient GPU-friendly implementation of Huffman en/decoding, addressing parallelization challenges to maintain acceptable inference speed.", "questions": "- Since gap arrays, index storage, and Huffman encoding are not novel techniques, what do you consider the key contribution or novelty of your paper?\n\n- Could the proposed exponent-only compression strategy be extended to other formats such as FP16, FP8, or even 8-bit quantized models? If so, what kind of savings or trade-offs would you expect in those settings?", "limitations": "N/A", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 4, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n- The core idea of compressing exponents—given their lower entropy—is well-motivated and supported by entropy analysis across multiple models, making the approach convincing.\n\n- The authors thoughtfully address implementation challenges, particularly for GPUs, and their proposed two-phase kernel to handle Huffman coding appears sound.\n\n- The paper provides solid experimental results. End-to-end evaluations on large LLMs demonstrate that DFloat11 delivers real-world improvements.\n\n**Weaknesses**\n- Some of the proposed implementation optimizations for Huffman encoding—such as using a gap array and index storage—have already been explored in prior work, which may make these contributions appear incremental.\n\n- The method is BF16-specific, while many production systems are shifting toward INT8, FP8, or mixed-precision formats. Although the paper mentions this limitation in the introduction, it lacks a more detailed discussion comparing BF16 accuracy and throughput with INT8, which would help clarify DFloat11’s relevance in current deployment contexts.", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for providing thoughtful feedback, and for recognizing the motivation of our work and the effectiveness of our proposal. We address your concerns and questions below.\n\n**[W1, Q1] Novelty and Contributions**\n\nWhile Huffman coding optimizations for GPUs have been studied by previous works, lossless model compression that supports GPU inference is relatively unexplored. Most previous work has focused on lossy compression (such as quantization) or lossless methods intended only for reducing storage on disk, rather than for use during inference. Supporting GPU inference of losslessly compressed models requires non-trivial algorithmic designs and engineering efforts. Our key contributions include:\n1.  Practical solution for deployment: We bridge the gap between the low-entropy observation and real-world deployment by designing a practical method for online inference using losslessly compressed weights. This includes proposing transformer-block-level decompression to achieve better throughput on GPUs.\n2.  Proven benefits: Our results demonstrate that significant compression gains can be achieved with minimal loss in GPU inference speed, while preserving losslessness.\n\nWe believe our work opens up new directions for lossless model compression in online inference and can inspire further progress in efficient model deployment.\n\n**[W2] Comparison with INT8 Accuracy and Throughput**\n\nWe have added additional accuracy comparisons between BF16, DF11, and INT8 models. The INT8 models use GPTQ-W8A8 quantization with the llm-compressor library. Our results show that BF16 and DF11 achieve identical accuracy across all tasks. In contrast, the INT8 models show a non-zero drop in accuracy compared to BF16/DF11, which can be as high as 6.37% on GSM8K with quantized Gemma-3-27b-it. We will include further throughput comparisons between BF16, DF11, and INT8 models in the final version of the paper. According to the SmoothQuant paper, INT8 quantization can achieve up to 1.35x to 1.42x speedup over 16-bit inference.\n|Model|MMLU|ARC Challenge|GSM8K|Winogrande|\n|-|-|-|-|-|\n|Gemma-3-27b-it (BF16/DF11)|77.5|72.5|92.1|79.4|\n|Gemma-3-27b-it (INT8)|76.4|70.8|85.8|79.7|\n|Llama-3.1-8B-Instruct (BF16/DF11)|68.0|82.0|82.8|78.5|\n|Llama-3.1-8B-Instruct (INT8)|67.8|81.7|84.8|78.5|\n|Llama-3.3-70B-Instruct (BF16/DF11)|86.6|49.2|94.2|84.8|\n|Llama-3.3-70B-Instruct (INT8)|85.9|48.0|94.0|83.7|\n\n**[Q2] Extension to Other Formats (FP16, FP8, and 8-bit Quantized Models)**\n\nOur method is also applicable to FP16, FP8, and other 8-bit quantized formats. The table below shows the entropy of Qwen3-8B weights in FP16, FP8, and INT8 formats, along with their corresponding compression factors. Dynamic-length float encoding can be extended to support these formats to enable their lossless compression.\n\n|Format|Bit Width|Entropy|Compression Factor (entropy/bitwidth)|\n|-|-|-|-|\n|FP16|16|13.69|85.6%|\n|FP8|8|6.78|84.8%|\n|INT8|8|7.50|93.8%|"}, {"review_id": "ljUPJemhAI", "reviewer": "Reviewer_K1Gb", "summary": "This paper introduces DF11, a novel data format to losslessly reduce the size of LLMs by applying entropy coding. DFloat11 assigns dynamic length encoding to weights. The authors also write custom GPU kernels for fast online decompression. This method shows improvement over a large range of models, improving efficiency, generation length and the capability of GPU hosting LLMs.", "questions": "1. Is it possible to apply the dynamic length weight to FP8/FP4 models? What is the room for improvement?\n2. How long does it take to compress LLMs in DF11? For example, 1B/8B/32B/405B.\n3. How is the latency when batch size is 1/2/4/8/16/32?", "limitations": "yes.", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strenghs\n1. The idea of losslessly compressing LLMs is interesting and novel.\n2. The experiments are convincing and detailed.\n3. The writings are easy to follow.\n4. The algorithms to support the lossless compress make sense and are hardware-efficient.\n\nWeeknesses\n1. The paper does not show the performance gap between DF11 and AWQ/GPTQ/FP8, even if I acknoledge that these methods are lossy.\n2. The paper does not provide quantitative information on the time consumption of compression.\n3. DFloat11 is less helpful when FP8 becomes the default data format. (e.g., DeepSeek-V3/R1)\n4. Token Batch Size 16384 is not that convincing, since I assume the use case of DF11 is when VRAM is limited. That means this token batch size should refer to prefilling stage. However, except for super-long context, (where KV cache memory will become the major bottleneck), the major overhead is in decoding. Assuming 40GB A100 and Llama3.1-8B, the common serving batch size is hardly beyond 16-32. In this regime, the parameter loading will become the main cost. DF11, to my understanding, will at least cost 70% additional overhead since it will load the 11-bit and 16-bit weights.", "was_revised": true, "final_justification": "While the proposed algorithm is interesting, the demonstrated performance gain is limited (only 30% model compression). Although it is a lossless method, it does not seem to bring significant performance gain compared with FP8/FP4, especially in the context that lower bits computation by hardware is growing every year.", "rebuttal": "We thank the reviewer for providing valuable feedback and for highlighting the novelty of our work and the effectiveness of our method. We address your concerns and questions below.\n\n**[W1] Performance comparison between DF11 and AWQ/GPTQ/FP8**\n\nWe present additional accuracy comparisons between BF16, DF11, FP8, and GPTQ (INT4). As shown in the table below, FP8 and GPTQ introduce non-zero accuracy degradation on various benchmarks, while DF11 maintains identical accuracy to BF16. Moreover, even in cases where the accuracy remains roughly unchanged, the quantized model's behavior can diverge significantly from that of the original model, as discussed in Line 44 of the paper.\n\n| Model | MMLU  | ARC-C | Winogrande | BBH  | MuSR |\n|-|-|-|-|-|-|\n| Llama-3.1-8B-Instruct (BF16)       | 68.0 | 82.0  | 78.5       | 30.1 | 7.6  |\n| Llama-3.1-8B-Instruct (DF11)       | 68.0 | 82.0  | 78.5       | 30.1 | 7.6  |\n| Llama-3.1-8B-Instruct (FP8)             | 68.0 | 81.2  | 77.7       | 29.7 | 7.5  |\n| Llama-3.1-8B-Instruct (INT4,GPTQ)      | 66.9  | 80.2  | 78.0       | 28.9 | 6.3  |\n\nWe also provide throughput results (prefill 2K tokens, decode 1K tokens) for DF11, BF16, and GPTQ (INT4) with different batch sizes below. These results show that while DF11 incurs additional decompression overhead, it remains practical across batch sizes. Importantly, DF11 guarantees accuracy and behavior that are exactly identical to the original model, whereas quantized models (such as FP8, GPTQ, AWQ, etc.) may experience accuracy loss or behavioral changes.\n\n| Model  | BS 1 | BS 4 | BS 16 | BS 64 |\n|-|-|-|-|-|\n| Llama-3.1-8B-Instruct (BF16) | 27.51 | 108.80  | 494.20  | 1883.91 |\n| Llama-3.1-8B-Instruct (DF11) | 13.17 |  51.91  | 203.64  |  774.52 |\n| Llama-3.1-8B-Instruct (INT4,GPTQ) | 32.01 | 126.72  | 502.89  | 1929.24 |\n\n**[W2, Q2] Time consumption of compression**\n\nWe report detailed compression times in Appendix F. Compressing an 8B model takes approximately 5 minutes, while a 405B model requires about 4 hours. Compression is performed only once per model and is fast and efficient. Since each transformer block can be compressed independently, we parallelize the process across multiple CPU cores to further accelerate compression.\n\n**[W3,Q1] Applying DFloat to FP8/FP4 models**\n\nDynamic-length float can be applied to FP8 and FP4 weights to enable lossless compression for these formats. Similar to BF16, we apply Huffman coding to compress the exponent bits while leaving the sign and mantissa uncompressed. Since FP4 and FP8 have fewer exponent bits than BF16, the resulting lookup tables are smaller, which may allow for faster decompression. The compression gains vary across these formats, as the effectiveness depends on the entropy of the weights. For example, we measured the entropy of FP8 weights in Qwen3-8B to be 6.78 bits, corresponding to a 15.3% compression ratio. In contrast, FP4 weights in Llama-3.3-70B have an entropy of 3.97 bits (a 0.8% compression ratio), likely too small to be impactful. We thank the reviewer for this suggestion and plan to explore it further in future work.\n\n**[W4] Relevance of Large Token Batch Sizes and Decompression Overhead**\n\nWhile we agree that typical serving batch sizes for decoder-only LLMs are usually limited to 16-32, larger token batch sizes are relevant in other important scenarios. For example, when using LLMs as embedding models (where no KV cache is involved) or in diffusion models for image and video generation, the models process a large number of tokens at once. In these settings, the decompression overhead is effectively amortized and becomes negligible. As shown in Table 3, DF11 inference latency for diffusion models increases by only 4-5% compared to BF16.\n\nFor smaller batch sizes (e.g., 16–32), the additional overhead is not a fixed 70%. Instead, it is the constant overhead required to decompress the model from DF11 to BF16. Figure 10 provides a detailed comparison of inference efficiency between BF16 and DF11 across various batch sizes and models.\n\n**[Q3] Latency for different batch sizes**\n\nPlease refer to Figures 4 and 10 for latency measurements across batch sizes of 1, 2, 4, 8, 16, and 32. As the batch size doubles, throughput approximately doubles while latency increases slightly."}, {"review_id": "NQqk3MpF4P", "reviewer": "Reviewer_JfK6", "summary": "The paper proposes Dynamic-Length Float (DFloat11), a lossless compression framework that compresses weights of bfloat16-based LLMs and diffusion models to ~70% of their original size while preserving them bit-for-bit. The authors observe that the 8-bit exponent field in bfloat16 weights carries about less than 3 bits of information. They exploit this redundancy by Huffman-coding the exponent bits and leaving the sign and mantissa untouched, yielding an effective 11-bit representation. The weights are meant to be quickly decompressed with a GPU-based algorithm which stores decomposed LUTs in SRAM of every thread block, rendering the decompression operation fast and memory-bound.", "questions": "- Can DFloat11 be enabled on-demand only when the memory is exceeded (e.g., when the context gets too long)?\n - What would it take to fuse DF11 decompression into GEMM/attention kernels? How would this hypothetically lower the cost of decompression?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n - The paper is clearly written and carefully checked even for minor errors.\n - It offers a clever solution to a rarely touched problem of lossless model compression.\n - The appendix is well-thought out and complements the paper with insightful details.\n\n**Weaknesses**\n - The paper is missing a clear presentation of the overhead introduced by DFloat11, e.g., showing how large is the latency overhead in relation to BF16 calculation, when weight matrix dimensions and batch size change. This is somewhat visible _in vivo_ on Figure 6, which hints that for small token batch size the latency roughly doubles because DF11 is memory bound, and for large batch sizes it hides behind FLOPs. It would be useful to have a performance benchmark for, e.g., a single linear layer (of different sizes) and not an entire model.\n- Quantization of bfloat16 models to > 8 bits is a niche problem, since proficient methods of quantizing to 8-bit formats or lower exist and the format is widely supported in hardware ; however, the authors well justify the lossless approach, which is a small but nonetheless valuable contribution.\n - Figure 4: The font on the plots is hard to read after printing. Perhaps a shared ylabel would save some space that could be spent there.\n - In general, the figures are hard to read, some fonts are too small.\n - Minor: it seems that the common capitalization (also used by Google which introduced this format) is “bfloat16”, while the paper uses “BFloat16”.\n - Source code absent (but promised to be released upon acceptance). In a paper like this, where the kernel is the central contribution, releasing the code is in my opinion a necessity.", "was_revised": true, "final_justification": "The problem under consideration is rather niche, but the contribution is focused, well engineered and well written. A strong factor in my recommended score is the author's promise to release the source code, so a broader audience could learn how to implement such a method in practice.", "rebuttal": "We thank the reviewer for giving insightful feedback and for highlighting the clarity and contribution of our work. We address the comments and questions below.\n\n**[W1] Performance benchmark for a single layer**\n\nBelow, we provide additional performance benchmarks for a single transformer layer from Llama 3.1 8B Instruct, evaluated at various token batch sizes. Additional results for single layer decompression can be found in Figure 7, which presents the decompression throughput and latency for various matrices ranging from 4096x2048 to 4096x65536 in size.\n\n|Token Batch Size|BF16 (ms)|DF11 (ms)|Decompression Overhead (%)|\n|-|-|-|-|\n|1    |1.190    |2.414    |102.8       |\n|1024 |2.823    |4.361    |54.5        |\n|2048 |5.172    |6.736    |30.2        |\n|4096 |10.148   |11.687   |15.2        |\n|16384|45.436   |47.030   |3.5         |\n|32768|107.156  |108.478  |1.2         |\n\n**[W2] Quantization of bfloat16 models to more than 8 bits is a niche problem**\n\nOur work addresses a different goal from quantization methods. While quantization introduces a trade-off between accuracy and compression, our approach guarantees bit-identical outputs to the original bfloat16 model. This can be valuable in scenarios where exact numerical accuracy is critical or where the model's performance is highly sensitive to quantization.\n\n**[W3,W4,W5] Figure font size and the capitalization of bfloat16**\n\nWe appreciate the reviewer's careful review of our paper. We will revise the figures in the camera-ready version to improve readability. We will also add a note to acknowledge the alternate capitalization of bfloat16.\n\n**[W6] Source code**\n\nWe will release the source code upon paper acceptance.\n\n**[Q1] Can DFloat11 be enabled on-demand?**\n\nYes, enabling DFloat11 on demand is feasible. Since our method compresses only the exponent bits while leaving the sign and mantissa unchanged, the compressed exponents can be offloaded to the CPU and transferred to the GPU to save memory when needed.\n\n**[Q2] Fusing DF11 decompression with GEMM**\n\nFusing DF11 decompression into GEMM or attention kernels is a promising direction that could further reduce decompression overhead. This approach would involve grouping compressed exponent values into tiles or blocks, which each thread block can load and decompress in SRAM. The decompressed weights could then be used directly on chip for matrix multiplications, avoiding the bottleneck of an additional write and read of decompressed weights to and from HBM. We thank the reviewer for this insightful suggestion and plan to explore it in future work."}, {"review_id": "s8RPIFOEEc", "reviewer": "Reviewer_eUCL", "summary": "This paper introduces Dynamic-Length Float (DFloat11), a lossless compression framework for BFloat16 model weights, achieving ~30% size reduction while preserving bit-for-bit accuracy. The method leverages entropy coding for exponents and employs optimized GPU kernels for efficient decompression during inference. Experiments on models like Llama 3 and Stable Diffusion demonstrate significant memory savings and throughput improvements.", "questions": "Please refer to the weaknesses.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 4, "strengths_and_weaknesses": "## Strengths\n\n1. Great insight. The authors identify the redundancy of BF16 and then propose to leverage entropy coding for exponents. The proposed DFloat11 maintains almost 100% model accuracy, which is impressive.\n\n2. The authors implemented fast GPU kernels, which enables practical usages for on different hardwares.\n\n3. Overall, the performance of this method is great. The paper is well-written and in a good shape. \n\n## Weaknesses\n\n1. My main concern is the evaluation on LLM is not very comprehensive. Specially, recent LLM releases tend to report performance across different tasks, including code, math, multilingual, instruction following, etc. It is not very clear how the proposed DFloat11 affect LLM's performance in these tasks. In my opinion, WikiText and C4 cannot fully represent the capability of today's LLMs.\n\n2. Latency Overhead: Small but non-zero decompression costs may affect latency-sensitive applications, though amortized at larger batches.", "was_revised": true, "final_justification": "Nice work. The overall results are very positive. The authors have also made a good rebuttal.", "rebuttal": "We sincerely thank the reviewer for the thoughtful feedback and for recognizing the insightfulness of our findings and the effectiveness of our approach. Below, we address the concerns raised in your review.\n\n**[W1] Evaluation on LLM is not very compreshensive.**\n\nWe present additional evaluations of BF16 and DF11 models on various benchmarks in the table below. Since DF11 models produce bit-for-bit identical outputs to their BF16 counterparts, their performance on any benchmark is exactly the same. Figure 13 further illustrates the identical outputs generated by DF11 and BF16 models.\n\n| Model | MMLU  | ARC-C | Winogrande | BBH  | MuSR |\n|-|-|-|-|-|-|\n| Llama-3.1-8B-Instruct (BF16)       | 68.0 | 82.0  | 78.5       | 30.1 | 7.6  |\n| Llama-3.1-8B-Instruct (DF11)       | 68.0 | 82.0  | 78.5       | 30.1 | 7.6  |\n\n**[W2] Latency overhead**\n\nWe agree that the non-zero decompression overhead may impact latency-sensitive applications at small batch sizes. However, as shown in Figure 6, this overhead becomes negligible in large batch size settings due to amortization. Additionally, we plan to explore further hardware-aware optimizations (such as fusing the decompression and GEMM kernels) in future work to reduce latency overhead."}]

Author rebuttal:

Meta-review: This paper introduces DF11, a novel data format to losslessly reduce the size of LLMs by applying entropy coding. DFloat11 assigns dynamic length encoding to weights. All reviewers agree that this work makes solid contributions, and the AC recommends acceptance.
Decision: Accept (poster)