To Compress or Not? Pushing the Frontier of Lossless GenAI Model Weights Compression with Exponent Concentration
Authors: Zeyu Yang
Tianyi Zhang
Jianwen Xie
Chuan Li
Zhaozhuo Xu
Anshumali Shrivastava
Venue: ICLR 2026 Poster (2026)
Keywords: Compression
LLM
Primary area: foundation or frontier models, including LLMs
TL;DR: By understanding that the exponents of generative AI model weights have a low-entropy structure, we developed ECF8, a lossless 8-bit floating-point format that significantly compresses these models without losing accuracy.

Abstract: The scaling of Generative AI (GenAI) models into the hundreds of billions of parameters makes low-precision computation indispensable for efficient deployment. We argue that the fundamental solution lies in developing low-precision \emph{floating-point} formats, which inherently provide numerical stability, memory savings, and hardware efficiency without dequantization overhead. In this paper, we present a theoretical and empirical study of an \emph{exponent concentration} phenomenon in GenAI weights: exponents consistently exhibit low entropy across architectures and modalities. We show that this arises naturally from $\alpha$-stable distributions induced by stochastic gradient descent, and we prove tight bounds on the entropy of exponents. Our analysis establishes a theoretical compression limit near FP4.67, which motivates the design of a practical FP8 format. Building on these insights, we propose Exponent-Concentrated FP8 (ECF8), a lossless compression framework with entropy-aware encoding and GPU-optimized decoding. Experiments on LLMs and DiTs up to 671B parameters demonstrate up to 26.9\% memory savings and 177.1\% throughput acceleration, with perfectly lossless computations, i.e., no deviation in model outputs. Our results establish exponent concentration as a statistical law of trained models and open a principled path for lossless low-precision floating-point design in the FP8 era. Code is available at https://github.com/zeyuyang8/ecf8.

Published as a conference paper at ICLR 2026
To COMPRESS OR NOT? PUSHING THE FRONTIER OF LOSSLESS GENAI MODEL WEIGHTS COMPRESSION WITH EXPONENT CONCENTRATION
Zeyu Yang', Tianyi Zhang', Jianwen Xie², Chuan Li,, Zhaozhuo Xu², Anshumali Shrivastava'
Rice University, "Lambda, Inc., Steevens Institute of Technology {zy45, tz21, (s143]er e @rice......e.. edu, {jianwen. xie, <) c]elambdal. com, 2x179@stevens.ed.
ABSTRACT
The scaling of Generative AI (GenAl) models into the hundreds of billions of parameters makes low-precision computation indispensable for efficient deploy- ment. We argue that the fundamental solution lies developing low-precision floating-point formats, which inherently provide numerical stability, memory sav- ings, and hardware efficiency without dequantization overhead. In this paper, we present a theoretical and empirical study of an exponent concentration phe- nomenon in GenAl weights: exponents consistently exhibit low entropy across architectures and modalities. We show that this arises naturally from o-stable distributions induced by stochastic gradient descent, and we prove tight bounds on the entropy of exponents. Our analysis establishes thcoretical compression limit near FP4.67, which motivates the design of a practical FP8 format. Building on these insights, we propose Exponent-Connentratee compression framework with entropy-aware encoding and GPU-optimized decod- ing. Experiments on LLMs and DiTs up to 671B parameters demonstrate up to 26.9% memory savings and 177.1% % throughput acceleration, with perfectly loss- less computations, i.e., no deviation in model outputs. Our results establish expo- nent concentration statistical law of trained models and open a principled path for lossless low precisiin floating-poit design in the FP8 era. Code is available at hhttts://ggthhbs com htt.://c//meyyaggg/eeeeeeo
FP8 (ECF8), a lossless
INTRODUCTION
The rapid growth of generative AI (GenAl) models, from large language models (LLMs) (Dubey et al., [0244] Team et al. 2023 Hurst et al. 2024: Liu et al. 2024a Yang et al. 2025) to diffusion transformers (DiTs) (Wan al. 2025 Batifol et al.] 2025 Wu et al. 2025). has led parameter counts scaling into the hundreds of billions. Such purameter-intensie GenAl models make low- precision computation indispensable. General matrix multiplication (GeMM) (Dongaraa al. 1990] Goto & Geijn 2008 Springer & Bientinesi 2018) with reduced precision is the most straighttorward way to achieve memory savings and possible acceleration, as eliminates redundancy in weight representation while directly reducing hardware workloads.
Existing work has primarily focused on integer- based quantization (Zhang et al.
2024 Zhang &
Shrivastava 2024 Yao et al. 20222] Dettmers et al.) 2022 Frantar et al. 2022 Xiao et al. 2023 Lin
al. 2024a Liu et al. 2024b). While effective reducing model size, these approaches suffer from two fundamental drawbacks: (a) they are lossy, often introducing accuracy
generative quality
degradation (Zhang et al. 2025); ; and (b) they incur efficiency penalties in large-batch inference due
to the required dequantization procedure and mixed-precisee precision execution (Lin et al. 2024b] Jin al.,
2024]. Since integer tensors must be converted back into floating doating-point values before computing, throughput suffers, limiting their utility in high- performace deployment.
Motivated by the limitations of lossy quantization, D (Zhang et al. 2025) revealed that the exponents of BF16 weights in LLMss have far lower entropy than the bitwidth allocated store them,

---

Published as a conference paper at ICLR 2026
creating headroom for lossless compression via entropy coding. Heilper & Singer (025 reported similar findings about low-entropy exponents in neural networks. Yet these findings puzzled the community: Is there fundamental principle that explains its success? Can the idea extend beyond BF16 to other formats? What is the fundamental lower bound on exponent entropy, and how does inform the design of future numerical formats for neural networks? And most critically, memory compression be transformed into actual inference acceleration by getting fast decoding kernels? All of these unanswered questions can be answered positively by our paper.
In this paper, we present theoretical and empirical analysis revealing an exponent concentration phenomenon in GenAI weights: exponents exhibit low entropy (Shamnon 1948) and cluster within narrow ranges across architectures and modalities. We trace this to the heavy-tailed dynamics of stochastic gradient descent (Amari 1993), which lead neural network weights to follow a-stable distributions (Nikias & Shao 1995). This provides a rigorous explanation of why exponents con- centrate and allows us to bound their entropy. Our analysis proves that the ultimate compression limit corresponds to floating-poit point format of roughly FP4.67. While such fractional format is impractical in modern GPU hardware, motivates our design of a powerful, lossless FP8 variant.
Building these insights, we introduce Exponent-Conneenraate FP8 (ECF8), novel lossless compression framework that encodes exponents with entropy-aware coding and efficient GPU de- coding. We show that ECF8 can transform memory compression into inference acceleration, delivering up to 26.9% memory savings and up to 177.1% throughput acceleration across state-of- the-art GenAl models, without any observed deviation in generation. ECF8 establishes principled path toward the theoretical limit of lossless weight compression in neural networks. To the best of our knowledge, ECF8 the first lossless compression method that delivers end-to-end inference acceleration and is validated across GenAl models up to 671B parameters.
In summary, our contributions are:
We provide a theoretical analysis of exponent concentration in GenAl weights, proving that a- stable distributions lead to bounded low-entropy exponents with a compression limit near FP4.67.
We empirically validate exponent concentration across large LLMs and DiTs, showing that layer- wise entropy consistently lies around 2-3 bits.
We design and implement ECF8, a lossless FP8 compression framework with encoding and GPU- optimized decoding, leading to inference acceleration.
We demonstrate practical benefits on models up to 671 parameters, achieving significant memory reduction and throughput gains while preserving bit-exact fidelity.
2 EXPONENT CONCENTRATION LEADS TO LOSSLESS WEIGHT COMPRESSION IN GENERATIVE AI
Preliminary. Floating-poin numbers in JEEE-754 format consist of three components: sign bit 8, an exponent E,, and a mantissa M. In the low-precision computing paradigm of deep learning, the exponents determine the dynamic range of representable values. If the exponent values are highly concentrated rather than spread uniformly, their entropy is low, which directly implies that fewer bits are needed for lossless representation. This motivates analyzing the statistical laws that govern exponent distributions in trained model weights.
2.1 OBSERVATION:
EXPONENTS IN GENERATIVE AI MODEL. WFIGHTS
We empirically examine weight matrices from parameter-intensie GenAI models, including trans- formers used for language and vision tasks. Across diverse architectures (LLMs, diffusion models, and multimodal transformers), we consistently observe that the exponents of weight values occupy very narrow range and have much lower entropy than the allocated bitwidth would suggest. As shown in Figure I histograms of exponent values cluster tightly around few modes, and the mea- sured Shannon entropy typically close to 2-3 bits, in contrast to the 8 more bits allocated in standard floating-pott point formats. This persistent low-entropy phenomenon strongly suggests an un- derlying distributional principle.
2

---

Published as a conference paper at ICLR 2026
DeepSeek-RL endropy.
Qwen3 1ata--Az---nssruut-5500000
Swen3-Coder-308-ABB..
3.50
Rotal Blocks: 14967
Model Layers
Tosal blocks: 12326
entropy:
Model Layes
3.5 Tstal blocks: 6182
Mean antropy
Modell Layers
Min 2 5500
0,5442
2,2775
3.25
Max entropy 3.6131
Max entrapy: 3,8839
Hax entropy 3.5218
50d entrapy: 0.1358
Std entropy 0.2136
Std entrady: 0.0575
48 3.00
2.75
2.50
2500
5000 7500 1000 12500 15000
2000
4000 5000 8000
1000 12000
1000 2000 2000
4000 5000 6000
Owen3-8B-FP8
FLUX.1-der
3.5
Klean entropy. 2.40 2.4064
Totial Blocks:
Min entropy: 2.3004
Model Layers
3.4 Mean entropy: 2,7500
Kin entropy:
blocks:
Model
Mean entropy: 2 2.0850
Total blocks: 81
entropy:
DIT Single
DIT
56d untrropy: 0.2312
Sta entropy: 0.2515
2.2
Stdl entropy: 0.2047
2.5497
Text Encoder
1.0
1.8
20
80
80
2.8
10
20
50
40
50
40
50
Reta Wam2-1-T2v.14B
Wa2222-TZV--1448
Owen-Image
2.5
Mranertrppp:: 1.6529
Blocks Text Encoder
2.5
blecks Mean 1,5199
Blocks
bleces Mean 1,9503
DIT
Min artropy 0,9758
Min anitopy: 0. 0.0756 entrapy: 0.5445
DIT2 Blects
Min entropy: 0 8.845
Text Encoder
2.0
entrapy: 2.5521
Text Encoder
Max entragy:
0,4748
1.5
2.0
Figure 1: Entropy analysis across transformer blocks for different model architectures. The x-axis represents the block index within each model, and the y-axis shows the entropy values. Different colors indicate different block types within each architecture.
2.2 ANALYSIS: EXPONENT CONCENTRATION OF a STABLE VARIABLES
Set-up and notations. Let X E R be continuous random variable representing neural network weight. In the IEEE-754 floating-pote representation, a nonzero real number x encoded as:
x = (-1)" 2 M, with E E 2, M E [1,2),
where is the sign bit, E. is the exponent, and M the mantissa. We define:
E
[loga X
as the floating poins exponent of X. Our aim is to analyze the entropy and compression limit of E when X follows symmetric a-stable distribution:
X ~ (B 0,),),, with a E (0, 2].
2.2.1 WHY NEURAL NETWORK WEIGHTS FOLLOW a STABBLE DISTRIBUTIONS
Trained neural network weights are formed by accumulating stochastic updates over time. Under stochastic gradient descent (SGD), the update rule 0.+1 = 0, -1 V (00; & introduces heavy-tailed noise from random mini-batch sampling. Empirical work shows that the gradient noise often exhibits power-law tails Pl > æ)) - with < 2. By the Generalized Central Limit Theorem, sums of such heavy-tailed variables converge to a-stable distributions. Thus, neural network weights after many updates approximately follow symmetric a-stable laws, providing the thcoretical foundation for our analysis.
2.2.2 EXPONENT ENTROPY CONCENTRATION
Theorem 2.1 (Exponent Entropy Concentration). Let E = [log2 [X]] where X ~ S. = 0)), Then E follows a discrete two-sided geometric distribution with parameter q ~-a;
- 4 1k) ke
=(E=k)
1 +
The Shannon entropy of E is bounded by:
a
a 1 2-a
< H(E) <
1+2-a
In particular, (F.) is finite for all 0.
3

---

Published as a conference paper at ICLR 2026
Proof. For large jxl, the tails of an a-stable distribution behave like P([X) > 2)
Cax Thus,
the probability exponent = k is:
P(E k) IP(2 2 X < 2 2 ka (1-2 -)
which corresponds to two-sided geometric distribution. The entropy of this distribution is:
2q
|loge ql
H(E) - - P(E = k) log, PEE = =k) h2
1+4
1-9
REZ
where (p) - bounding terms gives the inequality.
log, (1 p) 1 p) S is the binary entropy. Plugging @ - 2- and
This theorem shows that the floating-pois exponents of a-stable weights do not spread evenly across the integer line but instead concentrate around zero, decaying geometrically with rate 2 The entropy bound demonstrates that this concentration is strong enough to guar- antee finite entropy regardless of a, and tighter concentration (smaller ()) leads to smaller entropy. Practically, this means that exponents carry very limited uncertainty, which enables efficient com- pression.
Interpretation.
2.3 IMPACT: EXPONENT CONCENTRATION GUIDES LOSSLESS COMPRESSION
Corollary 2.2 (Compression Limit), The minimal expected number of bits required to losslessly encode the exponent E is:
Lmin (E).
Therefore, exponent values of neural network weights drawn from an a-stable distribution can be encoded in:
o
a 2-a
bits on average.
Numerical instance (a 2). When (the Gaussian-like case), we have 2 1/4. The entropy bounds give:
1.6 S (E) S 2.67.
Thus, the exponent itself has compression limit of about 2.67 bits in the extreme case. However, Soating-poin representation also requires one sign bit and several bits for the mantissa to preserve numerical precision. Even with minimal mantissa allocation (e.g., bit) plus the sign, the absolute floor is around:
2.67 + +
4.67 bits.
In practice, infeasible to implement a "FP4.67" or even FP5 format efficiently due aligmment and hardware constraints. Therefore, our proposed FP8 format (ECF8) represents a practical engi- neering choice: it is close to the entropy-driven theoretical limit, while retaining sufficient mantissa precision and hardware compatibility for efficient inference.
3 ECF8: LOSSLESS LLM WEIGHT COMPRESSION WITH EXPONENT CONCENTRATION
We present ECF8, lossless compression algorithm that exploits the statistical properties of FP8 weights in pre-trained generative AI models. Our method consists of three core components: an encoding scheme based on Huffman coding, a parallel GPU decoding kernel for variable-length de- coding, and a dynamic tensor management system that reduces memory footprint during inference.
3.1 ENCODING
Our CUDA-based Huffman encoding pipeline transforms neural network weights into compressed format optimized for parallel GPU decoding through three sequential stages. First, we generate op- timal Huffman codes by analyzing weight exponent frequencies and constructing the corresponding binary tree. Second, we build hierarchical lookup tables that enable efficient variable-lengle code decoding using b-t subtables aligned with GPU memory architecture. Third, encode weight exponents into a compressed bitstream while computing synchronization metadata that enables au- tonomous parallel decoding across GPU thread blocks.
4

---

Published as a conference paper at ICLR 2026
Huffman code generation. Neural network weights in FP8 format allocate 4 bits for exponents, yet empirical analysis reveals that exponent entropy is substantially lower due to non non-uniform fre- quency distributions. We exploit this entropy gap through Huffman coding, which assigns variable- length prefix-rree codes with shorter sequences for frequent exponents. Our encoding procedure extracts exponents from model weights, computes their empirical frequency distribution p(æ:) for exponent values z E (0,1, ...... 15}. and constructs the optimal Huffiman tree minimizing expected code length E[{] -. p(=)((())))) ((x))(x).. To ensure GPU compatibility, we constrain maximum code length to 16 bits, requiring frequency adjustment for rare symbols while preserving near-optimality. This constraint is rarely violated transformer layers empirically.
Hierarchical lookup table construction. We construct multi-level lookup system that processes variable-length Huffman codes through sequential 8-bit operations. The system comprises two com- ponents: a cascaded decode table mapping codes to symbols, and length table storing code lengths indexed by symbol value.
The cascaded structure organizes codes by their byte-aligned prefixes. Let P = {PI. P2..... .P. }k
denote the set of all byte-aligned prefixes extracted from Huffmmnn codes, ordered by length. For each prefix p: of length 8j bits, we construct a lookup subtable LUT; with 256 entries. Fach entry I.UT, [b] for byte value E .O,...; 255) contains either
a decoded exponent & E complete Huffman code,
15} if the code pill4, where denotes concatenation, corresponds
or a pointer value 156-indee((p)) directing lookup to subtable LUT; for longer prefix = =p:ll4. ||..
This design bounds memory usage O(PPI 256) entries while maintaining m([ lookup time for codes of length max- The length table L[x] stores the length of each symbol æ, enabling proper bitstream advancement during decoding. Figure iillustrates the construction process of the cascaded decode table using a simplified example using the characters "a", "b", "c", "d", and "e" as symbols rather than exponents {O..... 15).
Lookup Tables
Codebook
Huffman Tree
Index
00
00
10
0
Start
LUT,
Value
0
10
O/e
110
O/a
Index
00
00
10
1111
LUT
OAD
Value
b 0
1110
O/c
1/0
Figure 2: A simplified illustration lookup table construction. Huffman tree is built from the string "aaabbeddeeees". Lookup tables configured to be 2-bit, each subtable has 4 entries. Codes for symbols "e" and "a" are at most 2 bits long and appear directly in the first table. In contrast, codes for "b", "c", and "d" exceed 2 bits and begin with "11", entry "11" in the first table points to secondary table. The pointer value is I, indicating subtable 1. A second lookup in this subtable resolves the final symbol.
Encoding and synchronization metadata generation. We encode exponents into compressed bitstream while generating coordination metadata for parallel GPU decoding. Given fixed parame- ters B (bytes per thread) and T (threads per block), we sequentially process symbols 21, T2, ..., Iy, concatenating their Huffman codes into continuous bitstream and extracting complete bytes for storage. Since variable-length codes may span thread boundaries, we compute gap values to ensure proper synchronization. For thread í processing bytes [LB, tB 2), the gap 9r represents the bit offset from the previous thread:
((x,)) mod 8B x, (symbols(s)
9t
where symbols(i) denotes the set symbols processed by thread i. Additionally, we compute output positions ob for each thread block b, tracking the cumulative count of symbols processed
5

---

Published as a conference paper at ICLR 2026
by preceding blocks. This metadata enables autonomous thread block operation during parallel decompression without requiring inter-block synchronization.
3.2 DECODING
The decoding process reconstructs original exponent values from the Huffman-encoded bitstream using highly optimized CUDA kernel that exploits GPU parallelism through hierarchical mem- ory management and coordinated thread execution. The decoding algorithm operates in five dis- tinct phases: i. memory initialization, where each thread allocates a register buffer of fixed size (bytes per thread plus additional lookahead bytes) and the thread block establishes shared memory for coordination and output staging; ii data loading, where threads load their assigned segments of the encoded bitstream from global memory into thread-local registers; iii. parallel counting, where each thread performs initial decoding based on gap values to determine the number of de- codable symbols, followed by a parallel reduction algorithm across the thread block to compute cumulative symbol counts and establish output positions for each thread (ensuring threads write to non-overlapping memory regions); iv. coordinated decoding. where threads decode their full sym- bol sequences and write results to shared memory using the previously computed output positions; and global memory write-back, where decoded symbols are transferred from shared memory to global memory via coalesced writes. This approach minimizes global memory access overhead while maximizing parallelism through careful coordination of thread-local computation and block- level synchronization primitives. A detailed description of the decoding algorithm is provided in Algorithm D in Appendix M
3.3 TENSOR MANAGEMENT
Our tensor management system enables on-demand weight decompression during model inference through strategic memory allocation and layer-wise processing. We implement just-in-time de- compression mechanism using PyTorch forward hooks that intercept layer execution and perform weight reconstruction immediately before computation. The system maintains single pre-allocated GPU memory buffer of size equal to the largest layer's weight tensor, eliminating dynamic memory allocation overhead during inference. For each layer & in the sequential model architecture, the forward hook invokes our CUDA decompression kernel to reconstruct weights W; from their com- pressed representation and writes the result to the shared memory buffer. Upon completion of layer G,'s forward pass, the buffer becomes available for layer f++1, enabling memory-efficient inference with constant GPU memory overhead regardless of model depth. Note that ECF8 is designed to optimize inference performance, Appendix shows how ECF8 can be integrated with popular in- ference frameworks such as vL..M. Appendix H exxlainn why ECF8 is compatible with LoRA (Hu et al. 2022).
EXPERIMENTS
In this section, we test nine models spanning autoregressive language models, diffusion transform- ers, and mixture-of-experts variants from 8B to 671B parameters. We evaluate ECF8 across two critical dimensions: compression effectiveness and end-to-end inference performance. Specifically, aim to address the following two fundamental research questions related to the deployment of production GenAl models:
RQ1: What memory reduction can production-scale transformers achieve while maintaining bit- exact weight reconstruction?
RQ2: How does weight compression affect memory footprint and inference latency and can we achieve faster inference under the same memory budget?
4.1 LOSSLESS MEMORY SAVING FOR FP8 WEIGHTS
This section addresses RQ1 regarding the rate of memory reduction for production-scale transform- ers. ECF8 achieves memory reductions from 9.8% to 26.9% across evaluated models. LLMs demonstrate consistent reductions between 9.8% and 14.8%, while DìTs achieve higher compres-

---

Published as a conference paper at ICLR 2026
sion ratios, with the Wan2.2-T2V-A14B model reaching 26.9% reduction. Figure shows that the compression lossless.
Table 1: Memory savings and end-to-end throughput improvements under fixed memory constraints. For throughput evaluation, DeepSeck-RI-0S22 is tested on 8xH200 systems, Qwen3-235B-AD2 Instruct-2507-FPS on 4xH200 systems, and the remaining models on single GH200 (96GB) sys- tems. The "Supported Machine" column indicates the minimum hardware configuration capable of running the ECF8-compressed model. Memory savings and throughput improvements under fixed memory constraints demonstrate ECF8's practical deployment benefits. Appendix provides entropy analysis for the evaluated models, while Appendix summarizes the ECF8 encoding time.
Model
Memory Change (GB)
Memory (%)
Supported Machine
Throughput (%)
DeepScek-R1-op 1--052
623.19
530.26
14.8
8x (80 GB)
150.3
Qwen3-235B-A222 200-2550--A22-----mt----00000000
217.77 + 185.98
14.1
4xH100 (80 GB)
35.9
Llama-33-708-Imstruc.
Btt.BB--msstut-PP-Pyymnaaa -yy-dymmic
63.76
13.4
Moen3-Coder-3B-AA33 B--ttntttett-FFPP
27.85
23.69
14.3
1XTXX5090 (32 GB)
23.7
Qwen3-8B-FPs
6.47 7 3.61
9.8
1XRTX4070 (12 GB)
12.6
FLUX.1 dev
10.52 8.29
14.1
1XRTX4070 (12 GB)
177.1
17.40 + 12.65
25.4
1 4TX (16 GB)
55.1
Wan2.2/T2V-A14B
26.9
1XRTX4090
108.3
Qwen-Image
26.20 20.56
21.0
1XRXX4090 (24 GB)
126.6
ECF8 delivers memory reductions from 9.8% to 26.9% across all evaluated models, with effective- ness correlating strongly with architecture type. Language models achieve moderate but consistent reductions between 9.8% and 14.8%. while diffusion transformers show substantially higher com- pression potential, with the Wan2.2-T2V-A14B model reaching 26.9% reduction.
This pattern validates our theoretical foundation: FP8 exponent distributions in trained networks contain exploitable redundancy through lossless compression. Compression effectiveness remains remarkably stable across model scales, from 8B parameter Qwen3-8B-Fil to 671B parameter DeepSeek-R1-on 1-0528, indicating that ECF8's performance depends on fundamental weight distribu- tion properties rather than model size.
The practical impact extends far beyond storage efficiency. Memory reductions enable deploy- ment on lower-capacity hardware: the 14.8% reduction for PonScck-R110522 allows 8xH100 deployment instead of requiring 8xH200 or 2-node 8xH100 systems, while the 25.4% reduction for Wan2. 1-111-T2-14444 14B fits within single RTX4080 constraints where uncompressed models exceed memory limits.
Under fixed memory constraints, ECF8 enables throughput improvements ranging from 11.3% to 177.1% by supporting larger batch sizes within the same memory budget. These gains compound ECF8's benefits beyond simple storage efficiency deliver measurable inference performance im- provements. Details of inference speed evaluation are presented in Section 4.2
ECF8 FAST LOSSLESS
ECF8 LOSSLESS
ECFa IS AND
FAST LOSSLESS
BAST LOSSLESS
ECF8 PAST LOSELESS
ECF8 AND LOSSESS
ECF8 FAST AND LOSSLESS
Figure 3: Images generated by CCFF8-compressed Qwen-Image model, demonstrating reconstruction quality compared the original FP8 model. The numerical difference between im- ages generated by ECF8 model and FP8 model using the same random seed and inference parameters perfectly zero.

---

Published as a conference paper at ICLR 2026
Figure 3] provides direct evidence of ECF8's lossless compression property through visual qual- ity assessment. The CCF8-compressed Owen-Image model generated these images using identical random seeds and inference parameters the uncompressed FP8 baseline. Each output is pixel-by- pixel identical to the original model's results, confirming zero quality degradation. Appendix D also demonstrates the lossless nature of ECF8 through numerical evaluations of compressed LLMs standard downstream language tasks.
This lossless property proves critical for production deployment, where model behavior must remain completely unchanged while achieving significant memory savings. The diverse generated content demonstrates that ECF8 preserves full generative capabilities across different image domains and styles without introducing any artifacts or quality loss.
4.2 IMPROVEMENT ON INFERENCE SPEED UNDER FIXED MEMORY FOOTPRINT
This section addresses RQ2 regarding the inference acceleration under fixed memory footprint. We evaluate inference speed improvements by measuring maximum achievable throughput within fixed memory budgets, simulating real-world scenarios where hardware capacity limits batch sizes. In conclusion, the memory savings achieved by ECF8 translate directly into substantial inference performance improvements under realistic deployment constraints. The rationale behind this ob- servation is that smaller memory footprints allow for larger batch sizes, which in turn enable higher throughput, well as shorter per-request latency. For instance, under a 640 GB memory limit, ECF8 DeepSeek- -1-55228 sustains a batch size of 16, delivering 150.3% higher throughput and 60.1 lower per-request latency than FPS, which only accommodate batch size of 2. In addition, Appendix E provides a detailed analysis of the kernel-level performance of ECF8 decompression and Appendix F compares the throughput of FP8 and ECF8 LLMs at the same batch size.
Table 2: Comparison of FP8 and ECF8 across LLMs of varying scales. Under fixed memory con- straints, we report per-request latency (s), throughput (tokens/s), and maximum batch size (each batch generates 1024 tokens). DeepSeek- BeasSeek-R--05 R1-0528 and Qwen3-235B-NN 2-2-232--------- -an-s---2---nnttettt--22-- 2507 are eval- uated on H200 GPUs (141 GB), while other models are evaluated
single GH200 GPU (96 GB).
Model/ / Constraint
Max Batch Size
Per Request Latency (6)
Throughput (tokens/s)
Model
Constraint
FP8
ECF8
FP8
ECF8
(%)
FP8
ECF8
(%)
DeepSeek-RII PeepSeck-R1-026
640 GB
16
660.65
263.95
60.1
1.55
3.88
150.3
Qwen3-235B-Bo
240 GB
32
64
107.56
79.14
26.4
9.52
12.94
35.9
Llama-3.3-70B Instruct- FP8 dyn-ddnamic
48
24.80
22.28
10.2
41.28
45.96
11.3
Owen3-Coder330-ADD -aa.:-00---S----s-----P---
32 GB
16
32
107.33
86,70
19.2
9.54
11.80
23.7
Qwen3-8B-FFP
12 GB
16
24
4.90
4.35
11.2
208. 80
235.22
12.6
Language model inference acceleration. Table results demonstrate consistent throughput im- provements across language models ranging from 11.3% to 150.3%. The SheppSek-RR105522 model shows the most dramatic improvement, achieving 150.3% higher throughput (3.88 vs 1.55 tokens/s) by enabling 8x larger batch sizes (16 vs 2) within the 640 GB memory constraint. This improve- ment stems from ECF8's ability to reduce the model's memory footprint from 623GB to 530GB, creating sufficient headroom for larger batches.
Smaller models exhibit more modest but still significant gains. The Qwen3-8B-FFF model achieves 12.6% throughput improvement by increasing batch size from 16 to 24 within 1 12GB constraint. Even with constrained memory budgets typical of consumer hardware, ECF8 consistently enables meaningful performance improvements through more efficient memory utilization.
Diffusion model inference acceleration. Diffusson models are primarily compute-boutd bound rather than memory-bound with latency dominated by extensive denoising computations across multiple timesteps. VRAM management, which dynamically offloads and reloads model components be- tween GPU and CPU memory, represents a common deployment strategy for diffusion models to handle memory constraints during inference. As shown in Table 3 ECF8 consistently outperforms FP8 baselines across four representative diffusion transformer architectures, achieving memory re- ductions from 7.9% to 17.8% and latency improvements from 3.3% to 55.9% under controlled exper- imental conditions using the DiffSynth library. FLUX.I-O0 exhibits the most substantial gains with 45.9% end-to-end latency reduction (13.15s 24.29s) and 12.1 12 memory savings, while Qwen-
8

---

Published as a conference paper at ICLR 2026
Table 3: Comparison of FP8 and ECF8 across popular DiTs. Reported metrics include end-to-end (E2E) latency (s), step latency (ms), and GPU peak memory (MB). All experiments are conducted with the DiffSyynt library on a single GH200 GPU (96 GB) using an identical batch size, random
seed, and prompt. DiftSyyth enables VRAM management by default, dynamically offloading and reloading model components between GPU and CPU to reduce peak memory usage.
Model
DType
E2E Latency (s)
Step Latency (ms)
Memory (MB)
Memory J (%)
Latency (%)
ECF8
13.15 + 0.08
438.4 = 2.8
14274
12.1
45.9
FLUX.I-dev
FP8
24.29 + 0.10
809.5 3.4
16243
0.0
0.0
Wan2. 1-12V-44B
ECP8
460.67 0.92
9213.4 + 18.5
18036
7.6
3.3
FP8
476.21 + 3.34
9524.3 = 66.8
19529
0.0
0.0
ECF8
461.41 # 1,06
9228.2 # 21.3
27560
17.8
4.0
Wun2.2-T2V-A14B
FP8
480.45 0.68
9608,9 = 13.5
33517
0.0
0.0
ECF8
49,05 + 0.07
1226.3 + 1.7
25766
7.9
55.9
Qwen-Image
FP8
111.14 1.39
2778.4 = 34.8
27963
0.0
0.0
Image demonstrates exceptional step-level efficiency with 55.9% per-step latency improvement (1226.3ms 2778.4ms). The video generation models Wan2.1- 1-2211-TVV-14 14B and Wan2.2- --22--12 show more modest latency improvements of 3.3-4.0% though 10n2.2-T2--148 achieves 17.8% memory savings. These results validate that ECF8's 's compact weight representation reduces commu- nication overhead during the frequent weight loading operations characteristic of VRAM- diffusion inference, translating storage efficiency into measurable performance gains across diverse model architectures and scales. Beyond single-batch performance gains, the reduced peak memory consumption enables larger batch sizes within fixed memory constraints, translating remarkable throughput improvements of 55.1 1 % to 177. 1 % as demonstrated in Table
RELATED WORK
Weight quantization of GenAl models. Quantization (Hubara et al. 2018) has become the dom- inant compression paradigm, which reduces memory by converting 16-bit weights to lower preci- sion formats. LLM.int8() (Dettmers et al.) (022) achieved practical 8-bit inference through mixed- precision matrix multiplication, and delivered 2x memory reduction by processing most operations in INT8 while it handled outliers in FP16. SmoothQuant (Xiao et al. 2023) enabled W8A8 quan- tization by migration of activation outliers into weights, while GPTQ (Frantar al.] 2022), AWQ Lin et al. 2024a), and SpinQuant (Liu et al. 2024b) pushed boundaries to 4-bit precision through second-order optimization, activation-awarr selection, and learned rotations. However, quantiza- tion is inherently lossy and can cause unpredictable performance degradation, which translates to significant revenue losses at scale. DFloat 1 (Zhang et al., 2025) addressed this by exploiting the low entropy of BF16 weights, and proposed lossless compression via entropy-coded dynamic-length floats with efficient GPU decompression, which achieved 30% memory reduction with bit-exact re- construction. Nevertheless, these methods are specifically designed for BF16 weights. With the emergence of native FPS models (Micikevicius et al., 2022) as the future trend in GenAl, there re- mains unaddressed need for efficient lossless compression techniques tailored to FP8 weights, which motivates our work.
Weight distribution analysis of neural networks. A growing body of work indicates that the weight matrices of deep neural networks are not well captured by classical Gaussian assumptions but instead exhibit heavy-tailed behavior that can often be modeled by a-stable distributions. On the optimization side, GGubbzbalaaan et al. (202T) showed that stochastic gradient descent gives rise to heavy-tailed dynamics, where the properly rescaled itcrates converge in distribution to a- stable random variables, with heavier tails (smaller a) linked improved generalization. From the perspective of infinite-width limits, Jung et al. (2021) proved that networks with heavy-tailed initializations converge to a-stable processes, while Lee et al. (2023) extended these results to archi- tectures with dependent weights, showing that heavy tails also induce sparsity and compressibility. On the spectral side. |Mhoneyy & Martin (2019) and Martin & Mahoney (202I) analyzed trained weight matrices and observed power-law spectral densities, p()) x X with most exponents in the range a E (2, 4) across diverse architectures. Taken together, these findings suggest that heavy- tailed statistics are robust and recurring feature of well-trained networks. As show in Section

---

Published as a conference paper at ICLR 2026
a-stable distributions lead to exponent concentration and, consequently, low entropy, offering op- portunities for lossless compression via entropy coding.
CONCLUSION
We revisited the problem of efficient model deployment for generative Al systems through the lens of low-precision computation. While integer quantization has been the prevailing approach, its lossy nature and reliance on dequantization limit its scalability in high-throughput environments. We demonstrated that neural network weights obey an exponent concentration principle, rooted in o-stable dynamics of stochastic gradient descent, which ensures that exponents carry bounded low entropy. This theoretical insight yields fundamental compression limit around FP4.67. Motivated by this, we developed ECF8, a practical, lossless FP8 format with entropy- aware encoding and GPU-efficient decoding. Our experiments across LLMs and diffusion transformers show that ECF8 reduces memory consumption by up to 26.9% and improves throughput by much 177.1 %, all while preserving bit-exact fidelity. Beyond immediate systems benefits, this work highlights expo- nent concentration universal property of trained models and lays the foundation for principled design of next-generation low-precision floating- point formats for GenAl.
ACKNOWLEDGMENTS
We gratefully acknowledge the support of Lambda, Inc. for providing compute resources for this project. The work of Zhaozhuo Xu was supported by National Science Foundation awards 2451398 and 2450524. The work of Zeyu Yang and Anshumali Shrivastava was supported by Rice Ken Kennedy Institute (K2I) Generative Al Cluster Funding. The work of Anshumali Shrivastava was additionally supported by National Science Foundation award 2336612.
ETHICS STATEMENT
This work focuses improving the efficiency of generative AI systems. We do not anticipate any direct negative ethical implications. On the contrary, our approach has the potential to reduce the en- ergy consumption and carbon footprint of AI deployment by lowering GPU requirements. Further- more, by improving accessibility and reducing computational costs, our work can help democratize the of generative AI technologies for broader range of users.
REPRODUCIBILITY STATEMENT
We have taken several steps facilitate reproducibility. An open-source GitHub repository con- taining the full source code and experiment scripts is available at https: //tp::///ttttt com/ k e All theoretical results are accompanied by detailed proofs in Section [ To- gether, these resources ensure that our results can be independently verified and extended.
REFERENCES
Shun-ichi Amari. Backpropagation and stochastic gradient descent method. Neurocomputing, (4-5):185-196, 1993.
Stephen Batifol, Andreas Blattmann, Frederic Boesel, Saksham Consul, Cyril Diagne, Tim Dock- hom, Jack English, Zion English, Patrick Esser, Sumith Kulal, et al. FLUX.1 Kontext: Flow matching for in-context image generation and editing in latent space. arXiv e-prints, pp. arXiv- 2506, 2025.
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. PIQA: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI Conference on Artificial Intelli- gence, volume 34, pp. 7432-7439, 2020.
10

---

Published as a conference paper at ICLR 2026
Christopher Clark, Kenton Lee, Ming- Toutanova. BoolQ: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint arXiv:1905.10044, 2019.
Ming-Wei Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:21 14168, 2021.
Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zeutlemoyer. LLM.int00: 8-bit matrix multiplication for transformers scale. Advances in Neural Information Processing Systems 2022.
(NeurIPS),
Jack Dongarra, Jeremy Du Croz, Sven Hammarling, and Iain S Duff. A set of level 3 basic linear algebra subprograms. ACM Transactions on Mathematical Software (TOMS), 16(1):1-17, 1990.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The Llama 3 herd of models. arXiv e-prints, pp. arXiv-2407, 2024.
Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. GPTQ: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210.173233 2022.
Kazushige Goto and Robert A van de Geijn. Anatomy of high-performance matrix multiplication. ACM Transactions on Mathematical Software (TOMS), 34(3):1-25, 2008.
Mert Gurbuzhalaban, Umut Simsekli, and Lingjiong Zhu. The heavy-tail phenomenon in SGD. In International Conference on Machine Learning (ICML), pp. 3964-3975. PMLR, 2021.
Anat Heilper and Doron Singer. Lossless compression of neural network components: Weights, checkpoints, and K/V caches in low-precision formats. arXiv preprint arXiv:2508.19263, 2025.
Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li. Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations (ICLR), 2022. URL htttps://ppenrevatt
Corum??1d=neeveeerFf5
Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El-Yaniv, and Yoshua Bengio. Quantized neural networks: Training neural networks with low precision weights and activations. Journal of Machine Learning Research (JMLR), 18(187):1-30, 2018.
Aaron Hurst, Adam Lerer, Adam Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os- trow, Akila Welihinda, Alan Hayes, Alec Radford, et al. GPT-4o system card. arXiv preprint arXiv:2410.2127,, 2024.
Renren Jin, Jiangeun Du, Wuwei Huang, Wei Liu, Jian Luan, Bin Wang, and Deyi Xiong. A comprehensive evaluation of quantization strategies for large language models. In Findings of the Association for Computational Linguistics (ACL), pp. 12186-12215, 2024.
Paul Jung, Hoil Lee, Jiho Lee, and Hongseok Yang. a-stable convergence of heavy-tailed infinitely-
wide neural networks. arXiv preprint arXiv:2116. 1064, 2021.
Hoil Lee, Fadhel Ayed, Paul Jung. Juho Lee, Hongseok Yang, and Francois Caron. Deep neu- ral networks with dependent weights: Gaussian process mixture limit, heavy tails, sparsity and compressibility. Journal of Machine Leurning Research (JMLR), 24(289):1-78. 2023.
Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang. Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. AWQ: Activation-aware weight quantization for on-device LLM compression and acceleration. Proceedings of Machine Learning and Systems, 6:87-100, 2024a.
Yujun Lin, Haotian Tang, Shang Yang, Zhekai Zhang, Guangxuan Xiao, Chuang Gan, and Song Han. QServe: W4A8KV4 quantization and system co-design for efficient LLM serving. arXiv preprint arXiv:2405.044 04532,, 2024b.
11

---

Published as a conference paper at ICLR 2026
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chenggi Deng, Chenyu Zhang, Chong Ruan, et al. DeepSeek- V3 technical report. arXiv preprint arXfv:2412.19437. 2024a.
Zechun Liu, Changsheng Zhao, Igor Fedorov, Bilge Soran, Dhruv Choudhary, Raghuraman Krish- namoorthi, Vikas Chandra, Yuandong Tian, and Tijmen Blankevoort. SpinQuant: LLM quantiza- tion with learned rotations. arXiv preprint arXiv:2405. 16400, 2024b.
Michael Mahoney and Charles Martin. Traditional and heavy tailed self regularization in neural network models. In International Conference Machine Learning (ICML), pp. 4284-4293.
PMLR, 2019.
Charles Martin and Michael W Mahoney. Implicit self-regularization in deep neural networks: Evidence from random matrix theory and implications for learning. Journal of Machine Learning Research (JMLR), 22(165)) 2021.
Paulius Micikevicius, Dusan Stosic, Neil Burgess, Marius Cornaa, Pradeep Dubey, Richard Grisen- thwaite, Sangwon Ha, Alexander Heinecke, Patrick Judd, John Kamalu, et al. FP8 formats for deep learning. arXiv preprint urXiv:2209.05333 2022.
Chrysostomos L Nikias and Min Shao. Signal processing with alpha-stable distributions and appli- cations. Ieley-Interscience, 1995.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. WinoGrande: An ad- versarial winograd schema challenge at seale. Communications of the ACM, 64(9)999-06, 2021.
Claude E Shannon. A mathematical theory of communication. The Bell System Technical Journal, 27(3):379-422 1948.
Paul Springer and Paolo Bientinesi. Design of a high-performance gemm-like tensor-tensor multi- plication. ACM Transactions Mathematical Software (TOMS), 44(3):1-29, 2018.
Gemini Team, Rohan Anil, Sebastian Borgeaud, -aantBap Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: family of highly capable multimodal models. arXiv preprint arXiv:23 /2. 1805, 2023.
Team Wan, Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Chen-Wei Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianxiao Yang, et al. Wan: Open and advanced large-scale video generative models. arXiv preprint arXiv:2503.20314 2025.
Chenfei Wu, Jiahao Li, Jingren Zhou, Junyang Lin, Kaiyuan Gao, Kun Yan, Sheng-ming Yin, Shuai Bai, Xiao Xu, Yilei Chen, et al. Qwen-Image technical report. arXiv preprint arXiv:2508.02324 2025.
Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. SmoothQuant: Accurate and efficient post-training quantization for large language models. In International Conference on Machine Learning (ICML), pp. 38087-38099. PMLR, 2023.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025.
Zhewei Yao, Reza Yazdani Aminabadi, Minjia Zhang, Xiaoxia Wu, Conglong Li, and Yuxiong He. ZeroQuant: Efficient and affordable post-training quantization for large-scale transformers. Advances in Neural Information Processing Systems (NeurIPS), 35:27168-211 68-87183, 2022.
Tianyi Zhang and Anshumali Shrivastava. LeanQuant: Accurate and scalable large language model quantization with loss-error-aware grid. arXiv preprint arXiv:2407.10033 2024.
Tianyi Zhang, Jonah Yi. Zhaozhuo Xu, and Anshumali Shrivastava. KV cache is bit per chan- nel: Efficient large language model inference with coupled quantization. Advances in Neural Information Processing Systems (NeuriPS), 37:3304-3333 2024.
Tianyi Zhang, Yang Sui, Shaochen Zhong, Vipin Chaudhary, Xia Hu, and Anshumali Shrivastava. 70% size, 100% accuracy: Lossless LLM compression for efficient GPU inference via dynamic- length float. arXiv preprint arXiv: /65/, 2025.
12

---

Published as a conference paper at ICLR 2026
APPENDIX
A THE USE OF LARGE LANGUAGE MODELS (LLMS)
We used LLMs general-purpont assistive tool during the preparation of this work. Specifically, LLMs were employed for:
refining grammar, improving clarity of sentences, and suggesting alternative phrasings without altering the technical content.
Writing assistance:
generating 1 code snippets for tables, figures, and algorithm blocks, which were then verified and adapted by the authors.
Formatting:
Brainstorming:
offering initial organizational structures for sections and summarizing related work, which the authors subsequently reviewed, fact-checked, and rewrote.
No parts of the research methodology, experiments, data analysis, conclusions LLMs. The authors take full responsibility for the final content of the paper. LLMs were not used for creating fabricating research results. As per ICLR policy, LLMs are not listed authors.
generated by
B GENERATIVE AI MODELS USED FOR EVALUATION
We evaluate diverse set of state-of-the the-ttt--rt art generative AI models spanning language, code, image, and video modalities. All models are released FP8 or have community supported FP8 versions.
B.1 LLMS
DeepSeck-RI. R1-0528 671B reasoning-focused mixture-of-experr ef-eeersss (MoE) model, one of the first native FP8 L.Ms,, achieving strong performance on complex reasoning tasks.
235B MoE FP8 model tuned for instruction following. reasoning, mathematics, coding, and tool use.
a 70B dense instruction-tunee model using symmetric FP8 quantization for weights and activations, improving throughput and reducing memory cost.
a 30.5B MoE FP8 model specialized for code generation and agentic tool use, supporting long contexts up to 256K tokens (extendable to 1M with Yarn).
Ovend--Cader:3B-AABB-nnttuee
Qwen3-8B-FPS7 an 8B dense FP8 model released by Qwen, serving as lightweight but com- petitive baseline.
B.2 DITs
FLUX.I-dev
16B DiT for text-to-image generation, originally in BF16 with community FP8 implementations.
1-n2.1-72 a 14B DiT for text-to-video generation, BF16 release with FP8 support by
community.
Wan2.2-T2v-A141 30B MoE DiT for text-to-video generation, representing major architec- tural upgrade. Released BF16 with FP8 support by community.
Qwen-Image"l 20B biT-d model for image generation and editing, notable for high-fidelity text rendering and precise editing. Released in BF16 with FP8 implementations by community.
https:
13

---

Published as a conference paper at ICLR 2026
C ENTROPY ANALYSIS OF GENAI MODELS
Table 4: Exponent entropy of GenAl models before and after the Hadamard transform. The eval- uated models span diverse architectures (L.LMs and DiTs) and modalities (language, code, image, and video), with weights drawn from both native FP8 and quantized FP8 formats. Across all models, exponent entropy remains consistently low (1.52-2.73 bits), demonstrating that exponent concentra- tion is a universal property of trained networks. Moreover, applying the Hadamard transform does materially alter the exponent entropy.
Exponent Entropy
Model
Model Type
FP8 Source
(Original + Hadamard)
DeepSeck-R1-0522
LLM (MoE)
Native FP8
2.6037 2.5529
Owen3-235B-A22H6
LLM (MoE)
Native FP8
2.6264 2.6259
Qwen3-Coder-30BB 1-0---3--Insttut--F-
LLM (MoE)
Native FP8
2.5986 2.5979
LLM
Quantized FP8
2.6564 + 2.6249
Owen3-8B-FPS
LLM
Native FP8
2.7295 + 2.6368
FLUX.1-dev
Image DIT
Quantized FP8
2.0850 + 2.0861
Wan2.1-T2V-14B
Video DiT
Quantized FP8
1.6529 1.6494
Wan2.2-12V-A14B
Video DiT (MoE)
Quantized FP8
1.5199 1.5178
Qwen-Image
Image
Quantized
1.9503 1.9504
Our study evaluates exponent concentration across a broad set of FP8 generative models, including dense and MoE LLMs as well image and video image DiTs. Figure shows the block wise entropy distributions, and Table 4 summarizes the entropy values for each model together with its type and FP8 source. Across all architectures, exponent entropy remains low. Large LLMs concentrate around 2.5-2.7 bits, while diffusion transformers reach values of near 1.5-2.1 bits. Post training quantized FP8 models exhibit the same pattern native FP8 models, and the Hadamard transform leaves the exponent entropy essentially unchanged.
Exponent (4 Bits)
Sign and Mantissa (4 Bits)
Entropy (Bits)
Entropy [Bits]
2
2
Exact 4 Bits
Exact 4 Bits
Raw FP8
Raw FP8
Post-Hadamard
Post-Hadamard
2048
4096
8192
1024
2048
Matrix Size (N)
Matrix Size (N)
Figure 4: Exponent and sign-mantissa entropy analysis of FP8 weights sampled from A(0 1).
We also visualize the exponent and sign-mantissa entropy of FP8 weights sampled from AN 1) in Figure 4 Sign and mantissa have almost exactly 4 bits of entropy regardless of whether they are Hadamard transformed or not. By comparison, the exponent entropy of FPS weights sampled from (0, 1) is around 2.6 bits, which is consistent with our observations on real models and our theoretical analysis.
14

---

Published as a conference paper at ICLR 2026
D NUMERICAL VERIFICATION OF LOSSLESS COMPRESSION
We explicitly verify that ECF8 is strictly lossless through numerical evaluation. For image gen- eration Figure 3 the pixel-wise difference between outputs from the FP8 model and its ECF8- compressed counterpart is exactly zero. Furthermore, language benchmarks, finding that ECF8 and FP8 produce identical outputs. Table confirms that both formats achieve the scores across BoolQ (Clark et al. 2019). GSM8K (Cobbe al. 2021 PIQA Bisk et al. 2020). and Winogrande (Sakaguchi et al. 2021)
evaluated Qwen3-8B
standard
Table 5: Numerical verification of lossless compression for Qwen3-8B-FPS ECF8 and FP8 yield identical scores across all tasks, confirming bit bit-perfect reconstruction.
Benchmark Metrie
FP8
ECF8
GSM8K
Exact Match
0.876 0.009
0,876 = 0.009
BoolQ
Accuracy
0.867 = 0.006
0.867 = 0.006
PIQA
Accuracy
0.764 0.010
0,764 + 0.010
Winogrande
Accuracy
0.691 0.013
0.691 0.013
KERNEL-LEVEL BENCHMARK OF ECF8 DECOMPRESSION
In this section, we measure the absolute throughput and latency of the ECF8 decompression kernel across different amounts of FPS weights. In H100, ECF8 decompression achieves 201-235 GB/s effective throughput and producing only 39-2177 μus of latency depending on the tensor size. The FPS tensors used Table are generated by sampling from a normal distribution A/ 1).
Table 6: Speed of the ECF8 decompression kernel on H100 GPU across different tensor sizes.
Size (MB)
Elements (M)
Decompression Latency (μs)
Decompression Throughput (GB/s)
8
8.4
39.6
201.8
32
33.6
143.3
223.3
64
67.1
279.1
229.3
128
134.2
550.4
232.6
256
268.4
1092.9
234.2
512
536.9
2177.1
235.2
We also measure per-layer latency for Qwen3-8B-a comparing native FP8 and ECF8 formats. As shown in Table[7] ECF8 decoding overhead decreases batch size increases due higher arithmetic intensity that amortizes the decompression cost. At small batch sizes, overhead ranges from 52% (at batch size 1 1-64), but drops rapidly to 3% batch size 16384. Decompression becomes negligible once GEMMs dominate the forward pass.
Table Per-layer latency Qwen3 8B-FP8 comparing native FP8 and ECF8 formats. ECF8 de- coding overhead decreases as batch size increases due to higher arithmetic intensity, which amortizes the decompression cost.
Batch Size
FP8 Latency (ms)
ECF8 Latency (ms)
Overhead (%)
0.76
1.57
51.86
16
0.74
1.55
52.51
32
0.78
1,59
51.18
64
0.74
1,55
52,54
128
1.54
2.36
34.61
256
1.48
2.29
35.59
512
1,70
2.52
32.37
1024
2.03
2,84
28.70
2048
3,48
4.30
18.97
4096
6.56
7.37
11.06
8192
12.83
13.65
5.97
16384
25.17
25.99
3.14
15

---

Published as a conference paper at ICLR 2026
To provide comprehensive kernel benchmark, compare ECF8 and FP8 inference three representative layer sizes, including a dense linear layer with 583.5M parameters from DeepSeek- RI-0528, MoE layer with 44.0M parameters from DeepSeee-R110522 and dense linear layer with 193.0M parameters from Qwen3-8B-FP8.
DeepSeek-R1-00 R1-0528: Layer
DeepSeek-R1-0522 MoE
Qwen3-8B-FP8: Layer
6
FP8
Latency (ms)
60
ECF8
ECF8
20
ECF8
4
40
0
128
512
2048
8192
32
128
512
2048
8192
32
128
512
2048
8192
Batch Size
Batch Size
Batch Size
Figure 5: Layer-wiss inference latency of ECF8 and FP8 versus batch size on an H100 GPU.
Figure 5 compares the layer-wise inference latency of ECF8 and FP8. For all layer sizes, the latency of ECF8 reaches increasingly closer to FP8 as batch size grows and eventually almost overlaps with FP8 when batch size is greater than 8192.
DeepSeek- R1-0528: Layer
DeepSeek-R1- R1-0528: MoE
Qwen3-8B-FF Layer
Performance (TFLOPS)
10a
10a
10a
HBM BW TB55
HBM BW P.35 TB/a)
HBM BW (3.35 TBIs)
102
10
102
10
10
109
ECF8 FP8
10
ECF8
100
ECF8
FP8
FP8
10
10a
100
10a
10
10a
Arithmetic Intensity
Arithmetic Intensity
Arithmetic Intensity
Figure 6: Layer-wise roofline analysis of ECF8 and FP8 inference on H100 GPU.
Figure 6 illustrates the roofline analysis for ECF8 versus native FP8. In low arithmetic intensity regions (memory-bound), ECF8 incurs slight throughput penalty due to the additional decom- pression steps. However, as arithmetic intensity increases, driven by larger batch sizes sequence lengths, execution becomes compute-bound. these regimes, the decompression overhead is al- most amortized by matrix multiplication, allowing ECF8 to match the peak TFLOPS of native FP8 inference.
F THROUGHPUT COMPARISON BETWEEN ECF8 AND FP8 IN LLMs AT THE SAME BATCH SIZE
Figure presents an end-to-end throughput comparison between ECF8 and standard FP8 across multiple L.MMs at matched batch sizes. ECF8 introduces modest ~00 throughput penalty due to the lightweight on-the-fly decompression performed before each transformer block. However, this overhead is more than compensated by ECF8's substantial memory savings: under fixed memory budget, it can typically sustain nearly twice the batch size of FP8, as shown in Table 2 Once the workload enters the compute-bee bound regime, the decompression cost is fully amortized by increased arithmetic intensity, enabling ECF8 to match or even exceed FP8 throughput, consistent with the results in Table
16

---

Published as a conference paper at ICLR 2026
Qwen3-8B
Qwen3-Coder-30B-ABB
Llama-3.3-70B
45
400
FP8
391.
14
FP8
127
40
FP8
41.3
Tokens Second
350
ECF8
12
ECF8
ECF8
300
250
200
208 8
8
20
20.4
105.3
5.0
52
15
14.9
100
86.1
3.3
3,42
3.
10
10.3
7.5
13.50.1
25.91.4
2
5
4.6
2.5
5.1
1.8
2
16
16
2
16
Batch Size
Batch Size
Batch Size
Swen3-235B-A222
DeepSeek-Re
10
FP8
9.5
FP8
4.5
Tokens per Second
ECF8
4-
ECF8
3.4
3
0.2
2.4
4.
2
3.7
3.1
1.6
1.4
2.5
2.1
1
0....
0
4
8
32
16
Batch Size
Batch Size
Figure 7: Throughput comparison between ECF8 and FP8 in LLMs at matched batch sizes.
G STEPS TO INTEGRATE ECF8 INTO VLLM OR OTHER INFERENCE FRAMEWORKS
Integrating ECF8 into production inference frameworks such as vLLM requires only small, localized modifications because ECF8 is plug-and-play design.
Custom weight loader: The framework loads compressed ECF8 weights directly into GPU mem- ory rather than loading standard FP8 tensors. The loader reads the entropy coded exponent stream and the packed mantissa and sign bits.
On-demand decompression. A small forward hook is inserted the beginning of each trans- former block. This hook invokes our GPU optimized decoder and materializes standard FP8 tensor in preallocated workspace buffer.
Once decompressed, the tensor is an ordinary FP8 weight array and is indistinguishable from a conventionally loaded tensor. All existing computational kernels, scheduling logic, attention imple- mentations, and pipeline/data parallel execution paths remain unchanged.
H ECF8 Is COMPATIBLE WITH LORA FINE-TUNING
ECF8 intentionally designed for static weights. Training time weights change every step, so repeatedly compressing them is impractical. Each optimizer update would require regenerating the Huffman codebook and rewriting the bitstream, which introduces overhead incompatible with training throughput. For this reason, ECF8 is positioned weights remain fixed and can stored once a compact entropy coded form.
inference oriented method where
However, ECF8 is fully compatible with fine-tuning based LoRA (Hu et al. 2022). The mem- ory footprint of a model dominated by the base weights, while LoRA adapters are comparatively small. In LoRA setting the large base weights remain the compressed ECF8 format, while
17

---

Published as a conference paper at ICLR 2026
the small LoRA adapters stay uncompressed and trainable. This preserves the standard LoRA work- flow and provides meaningful memory savings without modifying training logic.
I RUNTIME MEASUREMENTS FOR ECF8 ENCODING
We benchmark the speed the ECF8 encoder and observe a sustained throughput of approximately 3.2 MB/s across various tensor sizes. Because encoding is one-time offline preprocessing step performed prior to model release, this overhead does not impact inference latency. Table X shows the encoding throughput for various tensor sizes.
Table 8: ECF8 encoding throughput for various tensor sizes.
Size (MB)
Elements (M)
Encoding Throughput (MB/s)
8
8.4
3.20
32
33.6
3.17
67.1
3.19
128
134.2
3.18
256
268.4
3.20
512
536.9
3.08
We also report end-to-end encoding time for full models using parallel CPU workers. Even with unoptimized CPU encoder, the largest 671B model is compressed in under hours, while smaller models take only minutes. Given that encoding a one-time cost amortized over millions of inference runs, this overhead is negligible for deployment.
Table 9: End-to-end ECF8 encoding time for full models using 32 CPU workers.
Model
Original Size (GB)
Encoding Time (Hours, 32 Workers)
DeepSeek-RI/ R1---21--55
623.19
1.73
217.77
0.60
27.85
0.08
63.76
Qwen3-8B-FPS
6.47
0.02
FLUX.I-dev
10.52
0.03
Wan2.1-T22 1-T2V114B
17.40
0.05
Wan2.2-T2V-A14B
30.49
0.08
Imane
26.20
0.07
18

---

Published as a conference paper at ICLR 2026
J SOFTWARE AND HARDWARE USED IN EXPERIMENTS
Our experiments are conducted using PyTTorch 2.7.1, CUDA 12.8, Transformers 4.56.0, and Dif- fusers 0.34.0. The hardware configuration varies based on model size requirements: we employ 8xH200 GPUs (141 GB memory each) for DeepSeek-R1-On R1-0228 due to its substantial memory de- mands, 4xH200 GPUs (141 GB memory each) for Qwen3-2355 A22B-IIstruut-225 2507-FF88. and single GH200 GPU (96 GB memory) for all remaining models in our evaluation suite.
K IMAGE GENERATION SETTINGS
10 for image generation. All DiT models are run with the default settings for image/video resolution, inference steps, and guidance scale. A fixed random seed (2025) is applied to guarantee comparability between the FP8 and ECF8 models. The prompts employed are listed in Table T0. The images generated by the ECF8 Qwen-Image model presented in Figure
We use the open-source Diff'Sytth implementation
Table 10: Prompts used for image generation.
Prompt
A futuristic neon-lit cityscape with a cheerful cyherpunk character in a glowing high-tech exosuit, smiling with holographic tattoos and reflective visor. The atmosphere is bright and vibrant with neon blues and purples, blending anime and sci-fi concept art aesthetics. Holding sign saying ECF8 IS FAST AND LOSSLESS.
A playful surrealist artwork where colorful balloons float through sunny meadow, and a joyful faceless figure relaxes in midair. The palette is light and cheerful with splashes of gold and pastel tones, evoking a sense of carefree happiness. Holding a sign saying ECF8 IS FAST AND LOSSLESS.
An anime female character, lofi style, soft colors, gentle natural lincwork, key art, emotion is happy. Hand drawn with an wward-winning anime aesthetic and a well-defined nosc. Holding sign saying ECF8 IS FAST AND LOSSLESS.
A festive parade scene with a vibrant character at the center of confetti- filledd-iileddddd street, smiling brightly, with balloons and streamers in the background. Holding sign saying ECF8 IS FAST AND LOSSLESS. A radiant anime-style character standing in glowing crystal meadow, surrounded hy rainbows and mag- ical sparkles, smiling with pure happiness. Holding sign saying ECF8 IS FAST AND LOSSLESS.
A cheerful fantasy scene where character rides friendly dragon while flying a kite, filled with vibrant colors and joyful energy. Holding sign saying ECF8 IS FAST AND LOSSLESS.
A pastel illustration of character sitting by glowing campfire, with warm lanterns floating above, smiling peacefully. Holding sign saying ECF8 FAST AND LOSSLESS.
A joyful carnival sunset with character in front of Ferris wheel, holding cotton candy, illuminated by golden evening light. Holding a sign saying ECF8 IS FAST AND LOSSLESS.
futuristic festival scene where neon-clad character dances joyfully under holographic fireworks. Hold- ing a sign saying ECF8 IS FAST AND LOSSLESS.
serene beach scene with character building sandcastle under rainbow, while dolphins leap the distance. Holding a sign saying ECFS IS FAST AND LOSSLESS.
19

---

Published as a conference paper at ICLR 2026
L NOTATIONS
Table 11: Symbols in Algorithm
Symbol
Description
Type Shape
LUT encoded
Hierarchical lookup tables
MMls X 256 integers
Encoded byte stream
Vbyyss bytes
packed
Packed sign/mantissa nibbles Block output positions
[Teeem |2] bytes
outpos
(Tbiocks 64-bit integers
gaps
Packed 4-bit gap values (two per byte), across all threads
[[ (T') bytes
64-bit sliding bit window
64-bit integer
S
16-bit tail buffer
16-bi, integer
f
Free bits in headroom Symbols counted per thread Thread output start index Thread output end index Block output base index
8-bit integer
c
32-bit integer
Ount
32-bit integer
32-bit integer
Ohase
32-bit integer
localbf
Thread register buffer Shared write buffer Shared prefix-sun arry Number of lookup tables Length of encoded stream Number of output elements Bytes per thread window Number of thrcads per block Decoded FP8 output bytes Thread-block index Thread index within block Global thread index (b T t)
Bytes of length +
writebf
outpos
oulpos(U) bytes
occumn
T 1 32-bit integers
Thurs
32-bit integer
Toytes
32-bit integer
Telem
32-bit integer
B
Constant integer
T
Constant integer
outputs
Neeem bytes
b
32-bit integer
t
32-bit integer
ts
64-bit integer
g
Per-thread gap (extracted from gaps) Decoded symbol (exponent code)
4-bit integer
8-bit integer
The packed sign/mantissa nibble Bit Bit-length of the codeword for
4- bit integer
8- b-bit integer
20

---

Published as a conference paper at ICLR 2026
M PSEUDOCODE FOR ECF8 DECOMPRESSION
Table 11 summarizes the notations used in this section.
Algorithm Block-level decompression from ECF8 to FP8
Require: LUT,, encoded, packed, outpos, gaps, Thuts Thytes Telem
Ensure: outputs
Parallel for each thread in block b of size
2: Global thread id to < ---T---- +t.
3: Load B +
bytes from encoded offset lg . B into local buffer localhf.
4: Form 64-bit head 7. from localb5(0.. [0..7] so the oldest byte is the most significant; form a 16-bit tail S from localb f|s..9].
5: Extract 4-bit gap g from packed gaps using tg: g (gaps|1ty/21) > (4 (tg mod 2)
4)) & 0x0ff. Set L - L < g. + g. c. O.
Phase 1: symbol counting
6: while f < 16 do
7:
< LUT >> 56]
8:
if 2 240 then I + LUT (5U(2556(256 - x) + ((L 48) & 255)
9: end
10: be < LUT (256
1) a < be + bg
11: end while
12: Stitch tail: + LV < < ( f f 1-16
13: while + < B do
Decode one symbol above; update L, f.
15: end while
Block-level prefix sum
16: Each thread writes its count to shared array accum|t] thread 0 writes occum[() oulpos[b] e.
17: Perform in-place up-sweep and down-sweep accum((0..T 1] to obtain exclusive starts.
18: Thread sets accumn() (0) & outpos[b) and accum[T]] +- outpos
19: Set Obase f- accumn Ostart + accum [t). Oend min(Ostat + c, Relem). Phase 2: decode and assemble FP8
20: Reinitialize S.
21: while f < 16 and Osarrt < Oend do
22:
Decode next symbol x.
23:
f- paacedd packee[[[samm/2]] < ((ostan mod 2) - -4). 24: byte + (x: < 3) V (q&0x80) V ((g > 4)80070)).
25:
writebf(osua
Orase] byte.
26:
Ostrr + Ostart + 1; update L, using be = LUT (256 (Muss - 1) + x].
end while
28:
Stitch remaining: L + LV(S < 16);; f 16.
while Osuu < Oend do
30:
Decode and pack as above to fill writebf.
31:
end while
Output write-back
32: Synchronize all threads.
33:
Each thread copies its slice of writebf to outputs.
Our ECF8 decompression kernel maps each stage to the fastest GPU memory tiers. The Huffman bitstream state is maintained entirely in registers, minimizing global memory traffic. Shared memory handles block-wide prefix sums and serves as a temporary buffer for coalesced global writes. Hier- archical Huffman tables and packed mantissa data are accessed via the read-only path and cached in L2 for low- low-latency lookups. As result, the decompression loop executes almost entirely out of registers and L2, with shared memory coordinating writes, enabling high sustained throughput.
21


Reviews:
[{"review_id": "aTqAxkVvGM", "reviewer": "Reviewer_m6i1", "summary": "The paper introduces Exponent-Concentrated FP8 (ECF8), a lossless compression framework for generative AI model weights. The authors discover that model weight exponents have low entropy (around 2–3 bits) across architectures, a phenomenon they term exponent concentration. They explain it theoretically via α-stable distributions arising from stochastic gradient descent and derive a compression limit near FP4.67, motivating FP8 as a practical format.\nECF8 implements entropy-aware Huffman coding and a GPU-optimized decoding kernel, achieving up to 26.9% memory savings and 177.1% throughput gains with zero accuracy loss across LLMs and diffusion models.", "questions": "- You claim exponent concentration is a statistical law of trained models. Could you quantify how universal this phenomenon truly is?\nFor instance, do all model families (transformers, CNNs, diffusion models, MoEs) exhibit similar entropy patterns, or are there exceptions (e.g., sparse or quantized weights)?\n- The α-stable assumption underpins your entropy bound derivation. Have you empirically verified that the tails of real model weight distributions match α-stable fits (versus log-normal or Laplacian alternatives)?\n- How easily can ECF8 be integrated into existing inference frameworks (like TensorRT, vLLM, or DeepSpeed-Inference)?\n- While ECF8 accelerates inference under memory constraints, have you measured absolute decoding overhead compared to a non-compressed FP8 baseline when memory is not a bottleneck?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The authors provide a rigorous explanation for exponent concentration through α-stable distributions and derive entropy bounds analytically. This bridges an important conceptual gap between statistical properties of trained models and practical compression techniques.\n- The GPU-friendly Huffman decoding kernel and tensor management system show careful systems design. The just-in-time decompression via PyTorch hooks is clever.\n- The study spans diverse models (8B–671B parameters), modalities (text, image, video), and hardware setups, lending credibility to the generality of the proposed approach.", "weaknesses": "- While the paper demonstrates strong GPU-side results, there’s limited discussion of how easily ECF8 could be integrated into existing frameworks or inference stacks beyond PyTorch (e.g., TensorRT, vLLM, serving pipelines).\n- The focus is exclusively on inference and storage. It remains unclear if exponent concentration also appears during training or whether ECF8 could be adapted for that phase.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "R5FEH4PcK8", "reviewer": "Reviewer_6N2r", "summary": "The paper studies the empirical observation that FP8 model weights exhibit exponent concentration as the exponent in FP8 is used with much lower entropy (2-3 bits range) than the full 4-bit range. The authors provide an analytical justification for this approach via a heavy-tailed/α-stable argument, and then exploit it by proposing a lossless FP8-compatible encoding (ECF8) that entropy-codes only the exponent, while preserving the sign and mantissa unchanged. They implement a GPU-friendly Huffman decoder with hierarchical LUTs and show that, on high-end GPUs and large FP8 models (including large LLMs and diffusion models), they can reduce memory footprint and translate that into real throughput benefits.", "questions": "1. Can you provide kernel-level benchmarks across a range of matrix/tensor sizes? In particular, how do compression ratio and effective runtime change for small vs large tensors? A plot of size vs throughput would clarify the operating regime.\n\n\n2. Can you show a roofline or bandwidth/compute tradeoff (similar to MARLIN[Frantar et al.]) plots, comparing vanilla FP8 loads vs FP8+your decoder? That would make the performance story crisper.\n\n\n3. Do the reported end-to-end measurements include the prefill stage for LLMs, or are they only for decode?\n\n\n4. How exactly do you use the GPU memory hierarchy (shared/L2/registers) to hide the decode overheads? A description would be helpful.\n\n\n5. Can you report the absolute overhead of decoding (e.g. μs per MB or % of layer time) so we can tell how often this is actually free?\n\n6. Is the current implementation restricted to weights, or have you tried the same exponent-entropy analysis on activations and gradients? If not, do you expect the same level of concentration? What if outlier mitigation methods(like Hadamard) are applied to them?\n\n7. Do the quantized models (from bf16/fp16 to fp8 using PTQ methods) exhibit the same distribution in the exponent?\n\n8. Could you provide runtime measurements for the encoder? compared to the decoder, of course.", "limitations": "", "rating": 2, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper has a clear motivation and analysis by showing the phenomenon (low-entropy exponents in FP8 weights) and then giving an analytical story for it.\n\n\n2. Real speedups with implemented CUDA kernels. The authors implement a GPU-friendly, hierarchical Huffman decoder and demonstrate that it can induce throughput gains.\n\n\n3. Evaluations on high-end, current GPUs using recent NVIDIA GPUs make the results relevant for current inference/serving stacks.\n\n\n4. Large-scale, end-to-end results by showing the method on large FP8 LLMs / diffusion-style models.", "weaknesses": "1. Incrementality vs DF11 / prior compressed-float ideas. A substantial part of the contribution can be read as taking the DFloat/DF11 observation that float fields are compressible, but tailoring it to FP8. That’s useful, but it’s not a completely new compression principle. It’s a focused, FP8-era instantiation.\n\n\n2. Missing detailed kernel/micro benchmarks. The paper reports headline end-to-end numbers, but it doesn’t fully break down kernel performance across tensor shapes/sizes, so it’s hard to see where the method starts to pay off and where decode overhead starts to dominate.\n\n\n3. The scope is unclear beyond weights. The paper primarily discusses compressing weights. There is no parallel, equally detailed analysis for activations (or even gradients), even though those often have different distributions and would be interesting for offloading or distributed setups.\n\n\n4. Interaction with distribution-smoothing methods is not discussed. Recent methods that apply PTQ for making FP8 models can change exponent usage. It’s not clear whether the proposed entropy gap persists under such transformations, or whether the gain shrinks.\n\n5. Runtime measurements for the encoder are missing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7Ni5Vdf2RR", "reviewer": "Reviewer_XgxS", "summary": "The paper introduces a new format, Exponent-Concentration FP8, which utilises entropy-aware exponent bit encodings. Introduction of this format is motivated by an empirical observation: exponent entropy for models stored in FP16 being significantly smaller than the number of allocated bits. Memory savings and inference speedups are demonstrated experimentally for generative models across vision and language domains.", "questions": "1. Regarding the inference acceleration: at identical batch size and hardware, what are the latency and throughput improvements over native FP8?\n2. How does your method compare with [2] ?\n\nReferences:\n\n[1] 70% Size, 100% Accuracy: Lossless LLM Compression for Efficient GPU Inference via Dynamic-Length Float, Zhang et al.\n\n[2] Lossless Compression of Neural Network Components: Weights, Checkpoints, and K/V Caches in Low-Precision Formats, Heilper & Singer.\n\n[3] ZipNN: Lossless Compression for AI Models, Hershcovitch et al.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The paper leverages exponent values concentration (as an instance of weights distribution trait) to propose a new numerical format.\n2. Validation range is solid: it spans language, vision, and multimodal models up to ~670B parameters. The authors report throughput gains via larger batch sizes under fixed memory constraints.", "weaknesses": "1. Theoretical claim that model weights follow $\\alpha$-stable distributions is presented as an intuition with no guarantees or conditions provided: \"often exhibits\", “after many updates”, “approximately follow” (lines 136 - 140). No reference to the empirical work showing the key observation of gradient noise being power-law distributed (line 137).\n2. The derivation of the compression limit is not mathematically strict. The 2.67 entropy bound for exponent is derived from a single case of Normal distribution. Adding 2 more bits on top of that number does not give a strict bound for the format compression. Again, it’s more of an observation rather than a theoretical limit.\n3. The “statistical law of trained models” (in abstract) is an overstatement. It is an empirical observation on a relatively small set of models.\n4. Limited novelty: the key idea of exponent concentration and Huffman encodings is similar to that of DFloat11 [1] or ZipNN [3], only applied to FP8 format instead of FP16. Applying Huffman encodings to exponent streams is an implementation detail, not novel improvement. \n5. Missing baselines: no experimental comparison with other FP8-specific compression formats like [2].\n6. Mismatch in Table 1, it mentions H200 in the caption while the table lists numbers for H100. Authors should either provide H200 numbers or modify the caption.\n7. No similar analysis for mantissa values distribution or justification why mantissa compression is not worth pursuing.\n8. “Visual quality assessment” from Figure 3. is not sufficient to demonstrate lossless compression. The authors should provide numerical loss results to back that statement.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vqfI1u6lBf", "reviewer": "Reviewer_n2wB", "summary": "This paper builds on prior observations that the exponents of weights in modern transformer models have low entropy and concentrate near zero. Motivated by this, the authors propose ECF8, a lossless FP8 weight compression format. Beyond the format itself, they implement a full pipeline that exploits ECF8 to reduce memory usage and improve latency/throughput, while preserving bit-exact outputs.", "questions": "Questions for the authors:\n\n1. Entropy bounds vs. closed form.\nTheorem 2.1 claims the entropy is \"finite for all \\alpha > 0\" based on lower and upper bounds for the two-sided geometric distribution. However, the Shannon entropy of the two-sided geometric has a closed-form expression. Why derive bounds instead of using the closed form directly?\n\n2. Compute-bound regimes and FP8 tensor cores.\nThe reported throughput gains appear to come from memory/VRAM-bounded scenarios. In a compute-bound regime, FP8 weights-and-activations baselines can exploit FP8 tensor cores (with ~2× the FLOPs of BF16 cores). How would a weight-only compression scheme like ECF8 sustain the large speedups shown (e.g., Table 1) once the model is compute-bound? It would help to: (i) explicitly characterize when results are memory- vs. compute-bound, (ii) quantify decode overhead (GB/s, kernel occupancy) and any GEMM slowdowns, and (iii) report the break-even batch/sequence lengths where ECF8 no longer provides throughput gains relative to FP8 weights-and-activations.\n\n\n3. Verification beyond visuals.\nFigure 3’s visual comparisons are helpful but not sufficient to demonstrate bit-exactness. Please include bit-exact checks for LLMs (e.g., matching logits/activations over a corpus with deterministic seeds) and add accuracy evaluations using lm-evaluation-harness, LightEval, or similar to strengthen the lossless and accuracy claims.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper investigates a fairly new direction in this area, the results are promising, and the evaluation is fairly thorough.", "weaknesses": "Please see questions below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)