FPE2M2: Approaching Lossless and Efficient Quantization with Native Floating Point
Authors: Ke Yi, jianwei zhang, Zhiying Xu, Xinlong Yang, Yang Zhou, Minmin Sun, Zengke Liu, Jia Li, Tong Zhang, Junyang Lin, Jingren Zhou
Accepted at: acl25

FPE2M2: Approaching Lossless and Efficient Quantization with Native Floating Point

Ke Yi1,2  Jianwei Zhang2  Zhiying Xu2  Xinlong Yang2  Yang Zhou2
Minmin Sun2  Zengke Liu3  Tong Zhang1  Junyang Lin2  Jingren Zhou2

Abstract
Auto-regressive decoding is a memory-bound job, meaning decoding inference performance is limited by the bandwidth rather than the computational capabilities of the GPU. Weight-only quantization is a promising method to address the memory-bound limitations. Previous studies have followed one of two approaches. Some have exclusively studied integer quantization while ignoring the Gaussian distribution nature of LLMs’ weights. Others have proposed non-uniform quantization, but incurred additional memory overhead due to lookup tables. In this work, we extend the floating-point standard to the ExMy quantization schema, which allocates x bits for the exponent and y bits for the mantissa to represent a number. In terms of runtime efficiency, we demonstrate that the conversion from ExMy to FP16 can be realized through register-level operations, which can get almost the same performance as INT5. In terms of quantization loss, we analyze that of different ExMy settings, where the E2M2 schema achieves an optimal balance, offering the highest efficiency with lossless accuracy. We further propose the FPE2M2 framework that supports lossless weight-only quantization inference and validate the FPE2M2 framework on Qwen and LLaMA Models across various modalities, such as text, image, and audio tasks, which achieves a faster inference speed while maintaining nearly lossless accuracy. 1

1 Introduction
LLMs have demonstrated outstanding performance and potential for various tasks. However, the high cost of inference has limited their application in practical commercial scenarios. Specifically, the pipeline of LLM inference can be divided into two stages: pre-filling and decoding. Owing to the auto-regressive nature of LLMs, the decoding stage is afflicted by the I/O bottleneck, as it generates only one token at a time. These challenges motivate the research on model compression techniques aiming to alleviate the bottleneck.

Quantization is a prevalent model compression technique that represents weights with lower precision. When the compression ratio is high, for instance, 4-bit quantization, the loss in accuracy becomes non-negligible. Previous studies (Frantar et al., 2022; Lin et al., 2023) employ calibration sets to adjust the weights of LLMs for reduced quantization loss. Nevertheless, the quantized LLM may overfit the calibration set and exhibit poor performance on unseen data, as noted by (Williams and Aletras, 2024). From another perspective, those works primarily focus on integer quantization, which overlooks the characteristic that model weights adhere to a Gaussian distribution. (Dettmers et al., 2024) observes the characteristic and introduces a non-uniform quantization schema to decrease quantization loss further. However, the conversion from non-uniform quantized values to full precision values relies on lookup tables. Notably, accessing the lookup table can incur significant overhead during the memory-bound decoding stage.

This work extends the IEEE 754 floating-point standard to a low-bit schema. Specifically, a floating point with an x-bit exponent and a y-bit mantissa is denoted as ExMy. ExMy differs from the previous standard by replacing Infinity and Nan with regular numbers. Based on our observation, the conversion between ExMy and FP16 can be accomplished by register-level operations, obviating the need for lookup tables. Furthermore, we comprehensively analyze the quantization loss for various non-uniform quantization schemes, and FPE2M2 can reach the sweet spot of achieving lossless quantization of the minimum bit width. We comprehensively validate the Qwen, LLaMA, and DeepSeek Distillation Models framework on text,
image, and audio tasks. We achieve a faster inference speed with negligible loss. Our contributions are as follows:

• We review the loss of different bit width allocation schemas and validate that FPE2M2 is the lossless quantization with minimum bit width.

• We propose a novel approach for rapid conversion from FPE2M2 to FP16 with only register-level operations, which is highly efficient in the memory-bound decoding stage.

• We further comprehensively validate the framework on mainstream LLMs for text, image, and audio tasks, achieving a faster inference with negligible accuracy loss.

2 Related Work

2.1 LLM Quantization

Quantization represents a practical methodology for reducing model size and accelerating inference. From a serving perspective, quantization can be categorized into weight-only and weight-activation quantization. Weight-activation quantization can accelerate computation by leveraging low-bit GEMM kernels suitable for compute-bound scenarios, namely the pre-filling stage. Qserve (Lin et al., 2024) further implement the A8W4KV4 and better accelerate. LLM.int8() (Dettmers et al., 2022) employs mixed INT8/FP16 decomposition to handle activation outliers. Subsequent work (Yuan et al., 2023) rearranges the channels to reduce the variance within one quantization group, further enhancing accuracy. Atom (Zhao et al., 2024) integrates the reorder technique and mixed INT4/INT8 precision to maintain accuracy and accelerate compared to the FP16 baseline. (Ashkboos et al., 2024; Liu et al., 2024; Yi et al., 2024) pairwise rotates the activation and weight to suppress outliers and maintain output equalization, enabling INT4 inference with well-smoothed activations. It should be noted that the above methods only serve the compute-bound pre-filling stage. Weight-only quantization is more suitable for the I/O bound decoding stage, as it employs low-bit representations for weight matrices, thereby saving memory movement. One effective way to reduce quantization error is shrinking the quantization range, i.e., sub-channel quantization. In the general case, quantization is performed on the channel level, which has a much less impact on the accuracy than the whole weight matrix. Applying sub-channel quantization could further reduce the quantization error, but it could result in a remarkable overhead. Orthogonal to Sub-channel quantization, GPTQ (Frantar et al., 2022) used Hessian-based error compensation to reduce quantization errors. AWQ (Lin et al., 2023) compressing weight quantization error according to the activation outliers. Those methods can achieve comparable accuracy compared with sub-channel quantization under the per-channel setting. However, they depend on the calibration dataset, which leads to a potential over-fitting problem (Williams and Aletras, 2024).

2.2 Non-uniform Quantization

Previous works focus on optimizing the quantization error for integer quantization. However, uniform quantization naturally causes more errors due to the Gaussian distribution of LLMs’ weights. NF (Dettmers et al., 2024) propose a non-uniform quantization structure that fits the Gaussian distribution assumption well and achieves better accuracy than Integer quantization. AFQP (Zhang et al., 2023) observes LLM’s weights as asymmetric, and applying asymmetric quantization further improves the accuracy. LLM-FP4 (Liu et al., 2023) represents FP4 in both weight-only quantization and activation-weight quantization and validates the effectiveness of FP4. The above methods are all based on 4-bit quantization, which makes it hard to achieve a negligible accuracy drop. Moreover, some of those methods ignore the importance of conversion speed (Liu et al., 2023) or find it hard to optimize the conversion speed (Zhang et al., 2023).

3 Preliminaries and Motivation

3.1 Definition of ExMy

Extending the IEEE 754 standard, the ExMy is represented as a sign bit, exponent bit, which can be represented as:

\[
\mathbf{X}_{\text{normal}} = (-1)^s 2^e (1 + \frac{d_1}{2^1} + \frac{d_2}{2^2} + \cdots + \frac{d_m}{2^m})
\]
\[
\mathbf{X}_{\text{subnormal}} = (-1)^s (\frac{d_1}{2^0} + \frac{d_2}{2^1} + \cdots + \frac{d_m}{2^{m-1}})
\]

where \( s \in \{0, 1\} \) denotes the sign bit, and \( d_i \in \{0, 1\} \) is the mantissa bits. \( e \) denotes the exponent parts; the subnormal number has \( e = 0 \). To be noticed that the bias of ExMy’s exponent part is 0, and the special values, i.e., ‘Nan’ and ‘Inf’, are
replaced by regular numbers. An example of E2M1 is shown in Figure 1.

3.2 Rethinking the Sweet Point of Quantization

Based on the binary system, representation with \(2^n\) bits is more conducive for hardware implementation, e.g., 2-bit, 4-bit, 8-bit, 16-bit. FP16/8F16 is widely used for training and inference without accuracy loss. Recent works (Detmers et al., 2022; Xiao et al., 2023) show that INT8/FP8 could also achieve lossless inference with higher efficiency than FP16/BF16. To further improve the efficiency, some works quantize the weights into 4-bit or 2-bit. The latter option suffers from severe accuracy loss, while the former balances efficiency and accuracy. As a result, previous works (Frantar et al., 2022; Lin et al., 2023) consider 4-bit as a sweet spot and attempt to restore the accuracy loss based on the 4-bit quantization.

As mentioned in Section 2.1, Sub-Channel Quantization and GPTQ are two popular methods for restoring the accuracy loss of INT4 quantization. Besides that, we also involve FP4 quantization, which adheres to the Gaussian distribution of LLMs’ weights and brings less accuracy. However, as depicted in Figure 2, quantization utilizing the aforementioned methods still exhibits a significant accuracy loss (10%) compared to FP16. The accuracy loss becomes negligible when the bit width exceeds 5. In terms of providing negligible quantization error and attaining higher inference speed, 5-bit quantization represents the optimal choice and deserves more attention.

3.3 E2Mx Consistently Dominates under Different Bit Width

The issue is regarding allocating the bits for the exponent and mantissa portions for optimal performance. Under the assumption that the weight of LLMs obeys a Gaussian distribution, we examine the relationship between quantization error and Sigma of the distribution for different ExMy as shown in Figure 3. The result reflects two important observations:

• E2Mx consistently dominates other ExMy under the Sigma of LLMs’ weight distribution.

• The trend of quantization error remains consistent across different bit widths.

To simplify the analysis among different quantization schemas, we scale both the quantization grids and weights to the range of [-1,1]. The sigma is collected on the scaled weights.

For the first observation, we provide a statistical analysis of the Sigma of different LLMs’ weight distribution C, which is in accordance with the sweet point of E2Mx. For the second observation, we provide a simple proof in Appendix B that the quantization error of ExMy is four times that of ExM(y+1), approximating a linear relationship.

Based on the above analysis, the significance of E2M2 is highlighted, thereby motivating us to devise an efficient implementation of E2M2 quantization in the next section.

4 Method

4.1 Preliminaries

Floating Point Quantization  For integer quantization, the high-precision floating point number is quantized through scaling and rounding, which can be formulated as:

\[
s = \frac{\max(|X|)}{Q_{max}}, Q = \lfloor \frac{X}{s} \rfloor,
\]

where \(Q_{max}\) represents the maximum value of the quantized number. However, the rounding operation could not be strictly applied to floating point
Figure 3: Preliminary analysis on the impact of different bit width allocation schemes. The quantization error is metricized by the L2 distance between the original and quantized weights. Quantization schemes with the same bit width are compared within the same figure range of the Sigma of weight results from mainstream LLMs.

quantization since the quantization slot is not uniform, as illustrated in Figure 1.

In order to efficiently determine the nearest quantized number for \( \mathbf{X} \): Firstly, obtain the length of the interval \( L = \log_2(\max(\mathbf{X}, 2)) \). Secondly, obtain the length of slot \( v = L/2^M \). Thirdly, rescale the interval and \( \mathbf{X} \) to ensure that the slot \( v = 1 \) and \( \mathbf{X}' = \mathbf{X}/v \). Fourthly, now the slot is uniform, round \( \mathbf{X}' \) to the nearest integer and then scale back to the original scale. The above rounding process could be formulated as follows:

\[
v = \log_2(\max(\mathbf{X}, 2))/2^M \\
\mathbf{X}' = \mathbf{X}/v, Q' = \lfloor \mathbf{X}' \rfloor, Q = Q' * v
\]

where \( M \) is the number of bits of the exponent part.

Dequantization Unlike the quantization process, the conversion from quantized ExMy to FP16 is performed in real-time, necessitating low time complexity. In integer dequantization, prior works have employed register-level operations to replace the low-throughput built-in conversion function, as shown in Figure 4 (a). This conversion depends on the following insight: For any unsigned integer \( x \) less than 8 bits, the bitwise OR operation \( 0x6400|x \) yields a result that is numerically equivalent to \( x + 1024 \).

Extending to the signed integer, \( Q_{max} \) is first added to the signed integers immediately after quantization to convert them into unsigned integers. Subsequently, the operation of subtracting \( Q_{max} \) is integrated into the final step, as depicted in Figure 4 (a).

For non-uniform quantization, the method above is not applicable. Previous research (Dettmers et al., 2024) has employed a lookup table for conversion, as shown in Figure 4 (b). However, accessing the lookup table is I/O intensive and inefficient for the memory-bound decoding stage.

4.2 Fast Conversion from ExMy to FP16

With the constraint of equation 3.1, we propose a fast conversion method from ExMy to FP16 based on numerical operations, as shown in Figure 4 (c). The conversion depends on the key insight:

\[
\mathbf{X} = (\mathbf{Q} << 8) * 2^{15}
\]

where \( \mathbf{X} \) denotes the FP16 value before quantization, and \( \mathbf{Q} \) denotes the quantized E2M2. The bias of the exponent part of FP16 is 15, whereas the bias of E2M2 is 0. Hence, we scale the shifted \( \mathbf{Q} \) with \( 2^{15} \) in the equation above. It is worth noting that 1) The above conversion supports both normal and subnormal numbers; 2) The conversion assumes that \( \mathbf{X} \) is positive. 3) The shifted \( \mathbf{Q} \) is approximately \( 2^{-15} \). Directly calculating on that without scaling can easily result in underflow; 4) It is not equivalent to directly setting some exponent bits to 1 to avoid underflow, since it would violate the definition of a subnormal number.
Figure 4: Illustration of the conversion from quantized value back to FP16 value. (a) A fast conversion from INT5 to FP16 involves register-level operations, such as ADD and OR operations. (b) A naive conversion from non-uniform quantized value to FP16, including accessing the lookup table, which is I/O intensive. (c) The proposed conversion from E2M2 to FP16. Given that its definition is extended from the IEEE 754 standard, the conversion can be implemented by only register-level operations.

To efficiently handle both underflow and negative numbers, the sign bit is fused into a multiplication operation, as demonstrated in the final step of Figure 4 (c).

\[
S = (-1)^s 2^{15}
\]
\[
X = ((Q << 8)|(0xf0f0)) * S
\]

where s is the sign bit.

4.3 System Level Optimization

The above description is based on single-element conversion. For real-time inference, it is necessary to optimize the bandwidth utilization by integrating multiple elements into INT32 and processing them in parallel in real time. To achieve storage of 5-bit E2M2 in INT32 without any blank space, 32 instances of 5-bit E2M2 are compressed into 5 INT32. Subsequently, the method of reconstructing FP16 from E2M2 will be demonstrated through careful design.

Sign Splitting   To simplify the index calculations, the 5-bit E2M2 is divided into a 4+1 scheme; in this scheme, the first 4 bits are exponent bits and mantissa bits, i.e., E2M2, and the last bit is the sign bit. The 4-bit portion of E2M2 is stored in four INT32 continuously, and the remaining sign bits are stored in a single INT32.

Layout Organization   In the mainstream system, FP16 (Half) is stored within INT32 as a pair, as shown in Figure 6 (a). This design results in efficient memory access and high memory bandwidth utilization. Under this design, it is essential to organize the data layout as shown in Figure 6 (b), which enables the parallel extraction of the pair weights. Through the masking (AND) operation, we could extract the E2M2 from the compressed INT32. Similarly, the masking operation is applied for the sign parts, and certain bits are set to 1 for the latter multiplication operation. The extracted E2M2 and sign are multiplied to obtain the final dequantized FP16. The entire layout is illustrated in Figure 6, and the pseudo-code is shown in Algorithm 1.

Algorithm 1: Dequantization from E2M2 to FP16
Input : Quantized tensor Q[5]
Output : Dequantized tensor DQ[16]
1 Initialize array e2m2[16];
2 for \( i \leftarrow 0 \) to \( 3 \) do
3   \( e2m2[i \cdot 4] \leftarrow (Q[i] << 8)\&0x0f000f00; \)
4   \( e2m2[i \cdot 4 + 1] \leftarrow (Q[i] << 4)\&0x0f000f00; \)
5   \( e2m2[i \cdot 4 + 2] \leftarrow (Q[i])\&0x0f000f00; \)
6   \( e2m2[i \cdot 4 + 3] \leftarrow (Q[i] >> 4)\&0x0f000f00; \)
7 end
8 for \( i \leftarrow 0 \) to \( 15 \) do
9   \( s \leftarrow Q[4]\&0xf800f800; \)
10  \( s \leftarrow s\&0xf3c003c00; \)
11  \( DQ[i] \leftarrow hmul2(s, e2m2[i]); \)
12  \( Q[4] \leftarrow Q[4] >> 1; \)
13 end
Figure 5: Illustration of the low-level implementation. (a) The basic structure to store FP16/half value in the low-level computation. (b) To extract two quantized FPE2M2 values in parallel, we first split the sign part and E2M2 part and store them in separate INT32. For the pair of FP16 values in one INT32, their quantized E2M2/sign part should possess a gap of 16 bits.

Figure 6: The overview of 32 quantized FPE2M2 value stored in five INT32.

5 Experiments

5.1 Models and Datasets
We conduct experiments on mainstream LLMs, including LLaMA families (Touvron et al., 2023) (LLaMA3-8B), Qwen families (Yang et al., 2024) (Qwen2.5-3B, Qwen2.5-7B, Qwen2.5-14B, Qwen2.5-72B) and Model distilled by DeepSeek-R1 (DeepSeek-R1-Qwen-14B, DeepSeek-R1-LLaMA3-8B). Quantization employs a per-channel symmetric scheme with a round-to-nearest (RTN) strategy. We evaluate the performance of the models on zero-shot Common Sense QA benchmarks(ARC-e, ARC-c (Clark et al., 2018), BoolQ (Clark et al., 2019), and OBQA (Mihaylov et al., 2018)), MMLU (Hendrycks et al., 2021), GSM8K (Cobbe et al., 2021), Chinese benchmarks (Ceval(Huang et al., 2023)), Visual benchmarks (MME (Fu et al., 2023), POPE (Li et al., 2023), ChartQA (Masry et al., 2022), AI2D (Kembhavi et al., 2016)) and Audio benchmarks (MELD (Poria et al., 2019)).

5.2 On Text tasks
We conduct experiments on text tasks, including ArcEasy, ArcChallenge, BoolQ, OBQA, MMLU, GSM8K-Flex, GSM8K-Strict, and Ceval. We compare the performance of the models on text tasks with different quantization schemes. The results are shown in Table 1. Non-uniform quantization (NF5 and FPE2M2) consistently outperforms integer quantization (INT4, INT5) across all models and benchmarks. Notably, non-uniform quantization with 5-bit has achieved nearly lossless performance, which fluctuates around 0.3% compared with complete precision. It is worth noting that the conversion from NF5 to FP16 relies on the lookup table, which causes remarkable overhead, as shown in Section 5.5.

5.3 On Multi-modal tasks
We also compare the proposed approaches and the state-of-the-art methods on the multi-modal tasks, as presented in Table 2. We evaluate Visual tasks with Qwen2-VL-7B and audio functions with Qwen2-Audio-7B. FPE2M2 quantization consistently surpasses other integer quantization schemes and performs fluctuation around baseline performance. While INT4 has a 6% accuracy drop and INT5 has a 0.6% accuracy drop compared with complete precision. In some instances, e.g., Audio Taks, FPE2M2 even surpasses full precision performance by 2%.

5.4 Ablation Studies
Different Allocation for Exponent and Mantissa
We conduct an ablation study on the allocation of exponent and mantissa for FPE2M2, as presented in 3. The results are consistent with the observation in 3.3, where the E2M2 achieves the best performance, and E4M0 has the worst.

Orthogonal Approach for Quantization Loss Optimization
We conduct an ablation study that employs an orthogonal approach to alleviate the accuracy drop of quantization, as presented in 4. Sub-channel settings have a consistent performance gain on integer quantization, e.g., INT4 and INT5, while the gain is uncertain for non-uniform quantization, e.g., FPE2M1 and FPE2M2. The reason is that the Sub-channel settings have fewer elements within a quantization group, i.e., 128, violating the assumption that the quantization group obeys s Gaussian distribution. For GPTQ settings, we perform the algorithm with MMLU as a calibration
Table 1: Downstream accuracy (%) on text tasks including ArcEasy, ArcChallenge, BoolQ, OBQA, MMLU, GSM8K-Flex, GSM8K-Strict, Ceval. Each block is based on the same foundation model specified in the first row. The best accuracy is highlighted in red, and the second best is highlighted in blue.

<table>
  <tr>
    <th>Quant Method</th>
    <th>ArcEasy</th>
    <th>ArcChallenge</th>
    <th>BoolQ</th>
    <th>OBQA</th>
    <th>MMLU</th>
    <th>GSM8K-Flex</th>
    <th>GSM8K-Strict</th>
    <th>Ceval</th>
    <th>Avg.</th>
  </tr>
  <tr>
    <td>Qwen2.5-3B-Instruct</td>
    <td>48.1</td>
    <td>72.9</td>
    <td>80.1</td>
    <td>42.0</td>
    <td>65.2</td>
    <td>63.4</td>
    <td>10.7</td>
    <td>74.2</td>
    <td>57.1</td>
  </tr>
  <tr>
    <td>INT4</td>
    <td>26.7</td>
    <td>24.4</td>
    <td>37.8</td>
    <td>31.8</td>
    <td>25.5</td>
    <td>0</td>
    <td>0</td>
    <td>25.5</td>
    <td>21.4</td>
  </tr>
  <tr>
    <td>INT5</td>
    <td>49.3</td>
    <td>71.1</td>
    <td>79.4</td>
    <td>41</td>
    <td>61.7</td>
    <td>63.0</td>
    <td>11.9</td>
    <td>69.6</td>
    <td>55.9</td>
  </tr>
  <tr>
    <td>NF5</td>
    <td>47.4</td>
    <td>71.3</td>
    <td>80.2</td>
    <td>41.8</td>
    <td>64.9</td>
    <td>60.7</td>
    <td>19.8</td>
    <td>71.7</td>
    <td><b>57.2</b></td>
  </tr>
  <tr>
    <td>FPE2M2</td>
    <td>47.1</td>
    <td>69.7</td>
    <td>78.2</td>
    <td>41.6</td>
    <td>64.9</td>
    <td>64.8</td>
    <td>19.1</td>
    <td>74.1</td>
    <td><b>57.4</b></td>
  </tr>
  <tr>
    <td>Qwen2.5-7B-Instruct</td>
    <td>55.0</td>
    <td>81.3</td>
    <td>86.3</td>
    <td>48.8</td>
    <td>72.2</td>
    <td>82.9</td>
    <td>76.3</td>
    <td>79.6</td>
    <td>72.7</td>
  </tr>
  <tr>
    <td>INT4</td>
    <td>47.7</td>
    <td>66.5</td>
    <td>80.4</td>
    <td>43.2</td>
    <td>64.7</td>
    <td>11.3</td>
    <td>6.9</td>
    <td>67.4</td>
    <td>48.5</td>
  </tr>
  <tr>
    <td>INT5</td>
    <td>54.7</td>
    <td>79.0</td>
    <td>85.5</td>
    <td>47.6</td>
    <td>71.3</td>
    <td>75.9</td>
    <td>68.9</td>
    <td>78.3</td>
    <td>70.1</td>
  </tr>
  <tr>
    <td>NF5</td>
    <td>55.1</td>
    <td>80.2</td>
    <td>86.1</td>
    <td>47.8</td>
    <td>71.7</td>
    <td>82.4</td>
    <td>77.6</td>
    <td>78.9</td>
    <td><b>72.5</b></td>
  </tr>
  <tr>
    <td>FPE2M2</td>
    <td>55.2</td>
    <td>80.3</td>
    <td>86.4</td>
    <td>48.2</td>
    <td>71.6</td>
    <td>83.2</td>
    <td>80.1</td>
    <td>78.1</td>
    <td><b>72.9</b></td>
  </tr>
  <tr>
    <td>Qwen2.5-14B-Instruct</td>
    <td>62.4</td>
    <td>81.5</td>
    <td>89.6</td>
    <td>47.4</td>
    <td>78.8</td>
    <td>50.9</td>
    <td>79.1</td>
    <td>83.8</td>
    <td>71.6</td>
  </tr>
  <tr>
    <td>INT4</td>
    <td>54.3</td>
    <td>76.0</td>
    <td>84.8</td>
    <td>44.8</td>
    <td>74.4</td>
    <td>44.5</td>
    <td>65.9</td>
    <td>77.5</td>
    <td>65.3</td>
  </tr>
  <tr>
    <td>INT5</td>
    <td>59.7</td>
    <td>79.5</td>
    <td>87.9</td>
    <td>46.0</td>
    <td>78.2</td>
    <td>39.7</td>
    <td>77.4</td>
    <td>82.7</td>
    <td>68.9</td>
  </tr>
  <tr>
    <td>NF5</td>
    <td>60.9</td>
    <td>80.2</td>
    <td>87.8</td>
    <td>46.4</td>
    <td>78.6</td>
    <td>50.5</td>
    <td>81.5</td>
    <td>82.8</td>
    <td><b>71.1</b></td>
  </tr>
  <tr>
    <td>FPE2M2</td>
    <td>62.5</td>
    <td>82.9</td>
    <td>87.9</td>
    <td>47.6</td>
    <td>78.4</td>
    <td>58</td>
    <td>76.8</td>
    <td>83.2</td>
    <td><b>72.2</b></td>
  </tr>
  <tr>
    <td>Qwen2.5-72B-Instruct</td>
    <td>63.4</td>
    <td>83.2</td>
    <td>90.4</td>
    <td>48.8</td>
    <td>83.7</td>
    <td>89.4</td>
    <td>90.7</td>
    <td>89.3</td>
    <td>79.9</td>
  </tr>
  <tr>
    <td>INT4</td>
    <td>57.5</td>
    <td>77.6</td>
    <td>88.2</td>
    <td>45.2</td>
    <td>80.8</td>
    <td>55.4</td>
    <td>63.6</td>
    <td>84.5</td>
    <td>69.1</td>
  </tr>
  <tr>
    <td>INT5</td>
    <td>63.4</td>
    <td>82.8</td>
    <td>90.6</td>
    <td>47.8</td>
    <td>83.0</td>
    <td>84.6</td>
    <td>90.6</td>
    <td>88.4</td>
    <td>78.9</td>
  </tr>
  <tr>
    <td>NF5</td>
    <td>63.8</td>
    <td>82.7</td>
    <td>90.5</td>
    <td>50.6</td>
    <td>83.6</td>
    <td>88.4</td>
    <td>90.9</td>
    <td>88.9</td>
    <td><b>79.9</b></td>
  </tr>
  <tr>
    <td>FPE2M2</td>
    <td>64.5</td>
    <td>83.2</td>
    <td>90.8</td>
    <td>49.4</td>
    <td>83.4</td>
    <td>87.7</td>
    <td>90.3</td>
    <td>89.6</td>
    <td><b>79.8</b></td>
  </tr>
  <tr>
    <td>LLaMA3-8B</td>
    <td>53.2</td>
    <td>77.4</td>
    <td>80.9</td>
    <td>44.8</td>
    <td>62.05</td>
    <td>50.8</td>
    <td>50.57</td>
    <td>47.4</td>
    <td>58.4</td>
  </tr>
  <tr>
    <td>INT4</td>
    <td>42.2</td>
    <td>67.4</td>
    <td>75.1</td>
    <td>40.8</td>
    <td>47.1</td>
    <td>8.8</td>
    <td>5.1</td>
    <td>33.1</td>
    <td>39.9</td>
  </tr>
  <tr>
    <td>INT5</td>
    <td>50.2</td>
    <td>78.5</td>
    <td>81.4</td>
    <td>46.0</td>
    <td>59.6</td>
    <td>39.6</td>
    <td>39.3</td>
    <td>43.8</td>
    <td>54.8</td>
  </tr>
  <tr>
    <td>NF5</td>
    <td>53.0</td>
    <td>78.5</td>
    <td>81.5</td>
    <td>45.2</td>
    <td>61.4</td>
    <td>46.4</td>
    <td>45.8</td>
    <td>45.5</td>
    <td><b>57.2</b></td>
  </tr>
  <tr>
    <td>FPE2M2</td>
    <td>53.2</td>
    <td>78.4</td>
    <td>80.2</td>
    <td>45.4</td>
    <td>61.2</td>
    <td>46.9</td>
    <td>46.0</td>
    <td>45.8</td>
    <td><b>57.2</b></td>
  </tr>
  <tr>
    <td>DPSK-DS-Qwen-14B</td>
    <td>53.5</td>
    <td>74.9</td>
    <td>87.7</td>
    <td>43.4</td>
    <td>73.2</td>
    <td>87.1</td>
    <td>86.8</td>
    <td>76.4</td>
    <td>72.9</td>
  </tr>
  <tr>
    <td>INT4</td>
    <td>47.7</td>
    <td>70.4</td>
    <td>85.0</td>
    <td>40.2</td>
    <td>68.5</td>
    <td>69.7</td>
    <td>75.8</td>
    <td>69.1</td>
    <td>65.8</td>
  </tr>
  <tr>
    <td>INT5</td>
    <td>53.6</td>
    <td>73.7</td>
    <td>87.3</td>
    <td>43.4</td>
    <td>71.8</td>
    <td>66.4</td>
    <td>86.5</td>
    <td>73.6</td>
    <td>69.5</td>
  </tr>
  <tr>
    <td>NF5</td>
    <td>53.9</td>
    <td>75.5</td>
    <td>87.5</td>
    <td>42.8</td>
    <td>72.1</td>
    <td>87.1</td>
    <td>87.4</td>
    <td>75.7</td>
    <td><b>72.7</b></td>
  </tr>
  <tr>
    <td>FPE2M2</td>
    <td>54.3</td>
    <td>75.3</td>
    <td>87.6</td>
    <td>43.6</td>
    <td>72.8</td>
    <td>86.5</td>
    <td>85.8</td>
    <td>76.1</td>
    <td><b>72.7</b></td>
  </tr>
  <tr>
    <td>DPSK-DS-Llama-8B</td>
    <td>42.4</td>
    <td>66.1</td>
    <td>83.0</td>
    <td>41.6</td>
    <td>54.1</td>
    <td>63.9</td>
    <td>62.1</td>
    <td>44.2</td>
    <td>57.2</td>
  </tr>
  <tr>
    <td>INT4</td>
    <td>40.6</td>
    <td>59.2</td>
    <td>77.4</td>
    <td>35</td>
    <td>43.7</td>
    <td>53.4</td>
    <td>48.5</td>
    <td>34.3</td>
    <td>49.0</td>
  </tr>
  <tr>
    <td>INT5</td>
    <td>42.2</td>
    <td>64.6</td>
    <td>82.3</td>
    <td>41.2</td>
    <td>52.6</td>
    <td>63.7</td>
    <td>60.9</td>
    <td>41.0</td>
    <td>56.0</td>
  </tr>
  <tr>
    <td>NF5</td>
    <td>42.6</td>
    <td>64.4</td>
    <td>83.3</td>
    <td>40.6</td>
    <td>53.7</td>
    <td>64.8</td>
    <td>62.4</td>
    <td>43.6</td>
    <td><b>56.9</b></td>
  </tr>
  <tr>
    <td>FPE2M2</td>
    <td>42.7</td>
    <td>64.7</td>
    <td>83.4</td>
    <td>42.8</td>
    <td>53.3</td>
    <td>64.9</td>
    <td>62.0</td>
    <td>42.7</td>
    <td><b>57.1</b></td>
  </tr>
</table>

Table 2: Downstream accuracy (%) on multi-modal tasks including POPE, MMMU, ChartQA, RealWorldQA, MME, AI2D, OCRBench, Meld_dev, Meld_test. We evaluate visual tasks with Qwen2-VL-7B, and audio tasks with Qwen2-Audio-7B.

<table>
  <tr>
    <th>Method</th>
    <th>POPE</th>
    <th>MMMU</th>
    <th>ChartQA</th>
    <th>AI2D</th>
    <th>OCRBench</th>
    <th>Avg.</th>
    <th>MME</th>
    <th>Meld_dev</th>
    <th>Meld_test</th>
  </tr>
  <tr>
    <td>FP16</td>
    <td>88.4</td>
    <td>50.8</td>
    <td>81.7</td>
    <td>80.3</td>
    <td>80.9</td>
    <td>76.4</td>
    <td>2325.3</td>
    <td>54.5</td>
    <td>54.7</td>
  </tr>
  <tr>
    <td>INT4</td>
    <td>86.9</td>
    <td>45.3</td>
    <td>77.2</td>
    <td>77.7</td>
    <td>69.9</td>
    <td>71.4</td>
    <td>2248.5</td>
    <td>47.8</td>
    <td>50</td>
  </tr>
  <tr>
    <td>INT5</td>
    <td>88.4</td>
    <td>49.8</td>
    <td>80.8</td>
    <td>79.8</td>
    <td>80.7</td>
    <td>75.9</td>
    <td>2283.6</td>
    <td>54.3</td>
    <td>54.3</td>
  </tr>
  <tr>
    <td>NF5</td>
    <td>88.4</td>
    <td>50.4</td>
    <td>81.6</td>
    <td>79.7</td>
    <td>81</td>
    <td>76.2</td>
    <td>2313.9</td>
    <td>54.4</td>
    <td>54.3</td>
  </tr>
  <tr>
    <td>FPE2M2</td>
    <td>88.9</td>
    <td>50.6</td>
    <td>81.8</td>
    <td>80.3</td>
    <td>81.1</td>
    <td>76.5</td>
    <td>2321.0</td>
    <td>55.6</td>
    <td>55.9</td>
  </tr>
</table>

set. It performs well in cases with large accuracy drops, i.e., INT4. However, the phenomenon of over-fitting to the calibration set is seen in cases that have a small accuracy drop, leading to an overall performance drop.

5.5 Efficiency Analysis

We evaluate the GEMM kernel with low-bit quantization on RTX 4070 Ti and H100, as shown in Figure 7. We implement INT4, INT5, and NF5 as baselines and take speedup compared with FP16 as
Table 3: Ablation study on the different ExMy quantization schemes with five bits.

<table>
  <tr>
    <th>Quant Method</th>
    <th>ArcEasy</th>
    <th>ArcChallenge</th>
    <th>BoolQ</th>
    <th>OBQA</th>
    <th>MMLU</th>
    <th>GSM8K-Flex</th>
    <th>GSM8K-Strict</th>
    <th>Ceval</th>
    <th>Avg.</th>
  </tr>
  <tr>
    <td>Qwen2.5-7B-Instruct</td>
    <td>54.9</td>
    <td>81.2</td>
    <td>86.3</td>
    <td>48.8</td>
    <td>72.1</td>
    <td>82.8</td>
    <td>76.2</td>
    <td>79.6</td>
    <td>72.7</td>
  </tr>
  <tr>
    <td>FPE1M3</td>
    <td>54.7</td>
    <td>79.0</td>
    <td>85.5</td>
    <td>47.6</td>
    <td>71.3</td>
    <td>75.9</td>
    <td>68.9</td>
    <td>78.3</td>
    <td>70.1</td>
  </tr>
  <tr>
    <td>FPE2M2</td>
    <td>55.2</td>
    <td>80.3</td>
    <td>86.4</td>
    <td>48.2</td>
    <td>71.6</td>
    <td>83.2</td>
    <td>80.1</td>
    <td>78.1</td>
    <td>72.9</td>
  </tr>
  <tr>
    <td>FPE3M1</td>
    <td>53.3</td>
    <td>79.3</td>
    <td>84.6</td>
    <td>47</td>
    <td>70.9</td>
    <td>79.8</td>
    <td>69.8</td>
    <td>77.8</td>
    <td>70.3</td>
  </tr>
  <tr>
    <td>FPE4M0</td>
    <td>48.5</td>
    <td>70.9</td>
    <td>77.6</td>
    <td>43.4</td>
    <td>59.8</td>
    <td>56.4</td>
    <td>56.0</td>
    <td>60.9</td>
    <td>59.2</td>
  </tr>
</table>

Table 4: Downstream accuracy (%) on text tasks including ArcEasy, ArcChallenge, BoolQ, OBQA, MMLU, GSM8K-Flex, GSM8K-Strict, Ceval. Each block is based on the same quantization schema specified in the first row.

<table>
  <tr>
    <th>Quant Method</th>
    <th>ArcEasy</th>
    <th>ArcChallenge</th>
    <th>BoolQ</th>
    <th>OBQA</th>
    <th>MMLU</th>
    <th>GSM8K-Flex</th>
    <th>GSM8K-Strict</th>
    <th>Ceval</th>
    <th>Avg.</th>
  </tr>
  <tr>
    <td>INT4</td>
    <td>47.7</td>
    <td>66.5</td>
    <td>80.4</td>
    <td>43.2</td>
    <td>64.7</td>
    <td>11.3</td>
    <td>6.9</td>
    <td>67.4</td>
    <td>48.5</td>
  </tr>
  <tr>
    <td>+Sub-Channel</td>
    <td>53.1 ↑</td>
    <td>77.7 ↑</td>
    <td>84.6 ↑</td>
    <td>47.8 ↑</td>
    <td>70.0 ↑</td>
    <td>76.8 ↑</td>
    <td>69.1 ↑</td>
    <td>75.9 ↑</td>
    <td>69.4</td>
  </tr>
  <tr>
    <td>+GPTQ</td>
    <td>51.7 ↑</td>
    <td>74.9 ↑</td>
    <td>83.6 ↑</td>
    <td>43.3 ↑</td>
    <td>70.0 ↑</td>
    <td>73.1 ↑</td>
    <td>67.8 ↑</td>
    <td>73.3 ↑</td>
    <td>67.2</td>
  </tr>
  <tr>
    <td>FPE2M1</td>
    <td>55.7</td>
    <td>80.5</td>
    <td>84.7</td>
    <td>47.0</td>
    <td>69.6</td>
    <td>74.2</td>
    <td>68.9</td>
    <td>75.4</td>
    <td>69.5</td>
  </tr>
  <tr>
    <td>+Sub-Channel</td>
    <td>53.2 ↓</td>
    <td>79.3 ↓</td>
    <td>86.2 ↓</td>
    <td>47.8 ↓</td>
    <td>71.1 ↓</td>
    <td>79.8 ↓</td>
    <td>73.2 ↓</td>
    <td>78.8 ↓</td>
    <td>71.2</td>
  </tr>
  <tr>
    <td>+GPTQ</td>
    <td>52.7 ↓</td>
    <td>78.8 ↓</td>
    <td>86.1 ↓</td>
    <td>46.0 ↓</td>
    <td>70.0 ↓</td>
    <td>79.1 ↓</td>
    <td>75.6 ↓</td>
    <td>76.3 ↓</td>
    <td>70.6</td>
  </tr>
  <tr>
    <td>INT5</td>
    <td>54.7</td>
    <td>79.0</td>
    <td>85.5</td>
    <td>47.6</td>
    <td>71.3</td>
    <td>75.9</td>
    <td>68.9</td>
    <td>78.3</td>
    <td>70.1</td>
  </tr>
  <tr>
    <td>+Sub-Channel</td>
    <td>55.0 ↑</td>
    <td>80.5 ↑</td>
    <td>86.2 ↑</td>
    <td>48.2 ↑</td>
    <td>71.6 ↑</td>
    <td>79.2 ↑</td>
    <td>71.6 ↑</td>
    <td>79.4 ↑</td>
    <td>71.4</td>
  </tr>
  <tr>
    <td>+GPTQ</td>
    <td>54.0 ↓</td>
    <td>80.4 ↓</td>
    <td>85.4 ↓</td>
    <td>47.5 ↓</td>
    <td>71.6 ↓</td>
    <td>82.1 ↑</td>
    <td>72.7 ↑</td>
    <td>78.1 ↑</td>
    <td>71.5</td>
  </tr>
  <tr>
    <td>FPE2M2</td>
    <td>55.2</td>
    <td>80.3</td>
    <td>86.4</td>
    <td>48.2</td>
    <td>71.6</td>
    <td>83.2</td>
    <td>80.1</td>
    <td>78.1</td>
    <td>72.9</td>
  </tr>
  <tr>
    <td>+Sub-Channel</td>
    <td>55.0 ↓</td>
    <td>80.0 ↓</td>
    <td>86.5 ↓</td>
    <td>48.0 ↓</td>
    <td>71.9 ↓</td>
    <td>81.6 ↓</td>
    <td>79.0 ↓</td>
    <td>79.6 ↓</td>
    <td>72.7</td>
  </tr>
  <tr>
    <td>+GPTQ</td>
    <td>54.4 ↓</td>
    <td>81.4 ↑</td>
    <td>86.2 ↓</td>
    <td>49.2 ↑</td>
    <td>71.8 ↑</td>
    <td>81.5 ↑</td>
    <td>74.4 ↑</td>
    <td>78.5 ↑</td>
    <td>72.2</td>
  </tr>
</table>

![Bar charts comparing throughput between quantized inference and full precision inference on RTX-4070ti and H100.](page_1012_1342_410_246.png)

Figure 7: Same-batch throughput comparison between quantized inference and full precision inference on RTX-4070ti and H100.

the metric. Compared with INT5, FPE2M2 brings limited overhead, including a few more register-level multiplications and masking operations. NF5 brings noticeable overhead, including access to a lookup table, which is inefficient under memory-bound scenarios.

6 Conclusion

This work presents FPE2M2, extending the IEEE 754 floating-point standard to low-bit quantization and achieving lossless quantization with 5-bit FPE2M2. FPE2M2 is built on the assumption that weights of LLMs obey a Gaussian Distribution, which brings less quantization error compared with Integer quantization. Compared with other non-uniform quantizations, e.g., NF5, FPE2M2 can be easily implemented with negligible overhead and is generalized across various large language models (LLMs). Through a comprehensive analysis of different quantization schemes, we take 5-bit as a sweet spot for LLM quantization, where FPE2M2 can achieve near-lossless performance with negligible overhead.
References

Saleh Ashkboos, Amirkeivan Mohtashami, Maximilian L. Croci, Bo Li, Martin Jaggi, Dan Alistarh, Torsten Hoeﬂer, and James Hensman. 2024. Quarot: Outlier-free 4-bit inference in rotated llms. Preprint, arXiv:2404.00456.

Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. BoolQ: Exploring the surprising difﬁculty of natural yes/no questions. arXiv preprint arXiv:1905.10044.

Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457.

Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word problems. Preprint, arXiv:2110.14168.

Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. 2022. Llm.int8(): 8-bit matrix multiplication for transformers at scale. Preprint, arXiv:2208.07339.

Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2024. Qlora: Efﬁcient ﬁnetuning of quantized llms. Advances in Neural Information Processing Systems, 36.

Elias Frantar, Saleh Ashkboos, Torsten Hoeﬂer, and Dan Alistarh. 2022. Gptq: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210.17323.

Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, et al. 2023. MME: A comprehensive evaluation benchmark for multimodal large language models. arXiv:2306.13394.

Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language understanding. In International Conference on Learning Representations.

Yuzhen Huang, Yuzhuo Bai, Zhiliao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su, Junteng Liu, Chuncheng Lv, Yikai Zhang, Jiayi Lei, Yao Fu, Maosong Sun, and Junxian He. 2023. C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models. Preprint, arXiv:2305.08322.

Aniruddha Kembhavi, Mike Salvato, Eric Kolwe, Minjoon Seo, Hannaneh Hajishirzi, and Ali Farhadi. 2016. A diagram is worth a dozen images. Preprint, arXiv:1603.07396.

Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. 2023. Evaluating object hallucination in large vision-language models. arXiv:2305.10355.

Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Xingyu Dang, and Song Han. 2023. Awq: Activation-aware weight quantization for llm compression and acceleration. arXiv preprint arXiv:2306.00978.

Yujun Lin, Haotian Tang, Shang Yang, Zhekai Zhang, Guangxuan Xiao, Chuang Gan, and Song Han. 2024. Qserve: W4a8kv4 quantization and system co-design for efﬁcient llm serving. Preprint, arXiv:2405.04532.

Shih-yang Liu, Zechun Liu, Xijie Huang, Pingcheng Dong, and Kwang-Ting Cheng. 2023. Llm-fp4: 4-bit floating-point quantized transformers. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, page 592–605. Association for Computational Linguistics.

Zechun Liu, Changsheng Zhao, Igor Fedorov, Bilge Soran, Dhruv Choudhary, Raghuraman Krishnamoorthi, Vikas Chandra, Yuandong Tian, and Tijmen Blankevoort. 2024. Spinquant: Llm quantization with learned rotations. Preprint, arXiv:2405.16406.

Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. 2022. Chartqa: A benchmark for question answering about charts with visual and logical reasoning. Preprint, arXiv:2203.10244.

Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018. Can a suit of armor conduct electricity? a new dataset for open book question answering. In EMNLP.

Soujanya Poria, Devamanyu Hazarika, Navonil Majumder, Gautam Naik, Erik Cambria, and Rada Mihalcea. 2019. Meld: A multimodal multi-party dataset for emotion recognition in conversations. Preprint, arXiv:1810.02508.

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lampl. 2023. Llama: Open and efﬁcient foundation language models. Preprint, arXiv:2302.13971.

Miles Williams and Nikolaos Aletras. 2024. On the impact of calibration data in post-training quantization and pruning. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 10100–10118, Bangkok, Thailand. Association for Computational Linguistics.

Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. 2023. Smoothquant: Accurate and efﬁcient post-training quantization for large language models. In International Conference on Machine Learning, pages 38087–38099. PMLR.
An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jianxin Yang, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang, Mei Li, Mingfeng Xue, Na Ni, Pei Zhang, Peng Wang, Ru Peng, Rui Men, Ruize Gao, Runji Lin, Shijie Wang, Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wenbin Ge, Xiaodong Deng, Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei, Xuancheng Ren, Xuejing Liu, Yang Fan, Yang Yao, Yichang Zhang, Yu Wan, Yunfei Chu, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, Zhifang Guo, and Zhihao Fan. 2024. Qwen2 technical report. Preprint, arXiv:2407.10671.

Ke Yi, Zengke Liu, Jianwei Zhang, Chengyuan Li, Tong Zhang, Junyang Lin, and Jingren Zhou. 2024. Rotated runtime smooth: Training-free activation smoother for accurate int4 inference. Preprint, arXiv:2409.20361.

Zhihang Yuan, Lin Niu, Jiawei Liu, Wenyu Liu, Xinggang Wang, Yuzhang Shang, Guangyu Sun, Qiang Wu, Jiaxiang Wu, and Bingzhe Wu. 2023. Rptq: Reorder-based post-training quantization for large language models. Preprint, arXiv:2304.01089.

Yijia Zhang, Sicheng Zhang, Shijie Cao, Dayou Du, Jianyu Wei, Ting Cao, and Ningyi Xu. 2023. Afpq: Asymmetric floating point quantization for llms. Preprint, arXiv:2311.01792.

Yilong Zhao, Chien-Yu Lin, Kan Zhu, Zihao Ye, Lequn Chen, Size Zheng, Luis Ceze, Arvind Krishnamurthy, Tianqi Chen, and Baris Kasikci. 2024. Atom: Low-bit quantization for efficient and accurate IIm serving. Proceedings of Machine Learning and Systems, 6:196–209.
A  Limitations

Non-uniform quantization depends on the assumption that the weights of LLMs obey Gaussian Distribution, which is not always the case. For example, the weights of LLaMA3-70B have remarkable outliers, leading to a server accuracy drop for integer and non-uniform quantization based on per-channel settings. While GPTQ and Sub-Channel can address this issue, they still suffer from overfitting or inefficiency. As a result, the sweet point of quantization is still open to explore for the models with remarkable outliers.

B  A simple proof of linear relationship between quantization error of ExMy and ExM(y+1)

First, recall that in ExMy representation with exponent bits \( x \) and mantissa bits \( y \). We proceed as follows::

\[
\mathbf{X}_{\text{normal}} = (-1)^s 2^e (1 + \frac{d_1}{2^1} + \frac{d_2}{2^2} + \cdots + \frac{d_m}{2^m})
\]
\[
\mathbf{X}_{\text{subnormal}} = (-1)^s (\frac{d_1}{2^0} + \frac{d_2}{2^1} + \cdots + \frac{d_m}{2^{m-1}})
\]

where \( s \in \{0, 1\} \) is the sign bit, and \( d_i \in \{0, 1\} \) is the mantissa bits. \( e \) denotes the exponent parts; the subnormal number has \( e = 0 \). Here we define the interval \( L_i, i \in [2, 2^e] \), which starts from \( 2^{i-1} \) and ends at \( 2^i \). \( L_1 \) is the first interval, starting from 0 and ending at 2. To be noticed that, the number of \( x \) defines the number of intervals, and the number of \( y \) defines the number of slots in each interval. Hence, compared with ExMy, ExM(y+1) just have two times more slots in each interval, and the length of each slot is 1/2 times smaller, as shown in 8. The quantization error can be calculated by:

\[
E[Q] = \int_{-\infty}^{+\infty} p(x) * e(x)dx \tag{2}
\]

where Q is the quantization error, \( p(x) \) is the probability density function of the weight, and \( e(x) \) is the L2 error.

Here we briefly approximate the \( p \) to be constant function \( p(x) = 1 \). For the first interval of E2M1,

the quantization error is:

\[
E[Q] = \int_0^2 p(x) * e(x)dx \\
= \int_0^2 1 * e(x)dx \\
= 2 * \int_0^1 x(1-x)dx \\
= \frac{1}{3}
\]

For the first interval of E2M2, the quantization error is:

\[
E[Q] = \int_0^2 p(x) * e(x)dx \\
= \int_0^2 1 * e(x)dx \\
= 4 * \int_0^{1/2} x(1/2 - x)dx \\
= \frac{1}{12}
\]

The real quantization error with \( p(x) \) being the normal distrition has been shown in the main text 3.

C  Statistical analysis of Sigma of different LLMs’ weight distribution

The quantization process entails two steps: Firstly, scale the weight to the maximum value of the quantization range. Secondly, round the weight to the nearest integer. To simply the analysis among different quantization schema, we scale the quantization grids to the range of \([-1, 1]\) and scale the weight to the range of \([-1, 1]\). Subsequently we collect the sigma of different LLMs’ weight distribution after scaling, and show the results in 9. For cases where the model has a size larger than 14B, the sigma is relatively small in the very early layer. Apart from that, the sigma is typically in the range of 0.12 to 0.28, which accounts for 97% of the weight distribution. Moreover, 0.25 is the prevalent value for the sigma in most layers, which is the optimal point of E2Mx.
E2M1: \(2^1\) slots per \(2^2\) interval
E2M2: \(2^2\) slots per \(2^2\) interval

L2 Error
The first interval of E2M1
The first interval of E2M2

Figure 8: E2M1 and E2M2

(a) DeepSeek-R1-Distill-Qwen-14B
(b) DeepSeek-R1-Distill-Qwen-32B
(c) Llama-3-8B
(d) Qwen-0.5B
(e) Qwen-1.5B
(f) Qwen-3B
(g) Qwen-7B
(h) Qwen-14B
(i) Qwen-32B

Figure 9: Sigma of different LLMs’ weight distribution

Reviewer paper summaries: This paper demonstrates that FPE2M2 is the sweet-spot choice for balancing quantization accuracy and system efficiency compared to other formats. In terms of accuracy, FPE2M2 achieves quantization performance close to FP16. Regarding system efficiency, by leveraging the proposed sign splitting and layout organization methods, the dequantization process can be completed more quickly. Therefore, the authors recommend using FPE2M2 as the quantization choice for LLMs.

The paper proposes FPE2M2, a novel quantization framework that extends the IEEE 754 floating-point standard into a low-bit format (denoted as ExMy) tailored for quantization in LLMs. A key contribution is a fast, register-level conversion algorithm from FPE2M2 to FP16 that eliminates the need for memory-intensive lookup tables, thus enhancing inference speed during the memory-bound decoding stage. Extensive experiments across various modalities and models are performed.
Summary of strengths: Low-bit quantization is an important topic for LLMs.

The findings are interesting that the quantization error of E2My is lower than other bit width allocations with the same bit widths.

The system implementation is efficient for faster de-quantization process.

1. Develop a fast, register-level conversion algorithm from FPE2M2 to FP16 for faster computation. 
2. Easy quantization compared to quantizing to fixed point representations.
Summary of weaknesses: My major concern is that this paper lacks a theoretical explanation for why E2My has lower quantization errors than other bit-width allocations (Figure 3). Is it because the numerical distribution of FPE2My is closer to the numerical distribution of the selected weight tensor? Could it be that the quantized weight distribution happens to match the numerical distribution of E2My, and if different weights were used, this conclusion would no longer hold? There should be some relevant analyses to make this finding more convincing.

The method mentioned in the paper for Fast Conversion from ExMy to FP16 appears to be similar to the common approaches used for converting other low-precision floating-point numbers (such as FP4/FP8) to high-precision floating-point numbers. The method itself does not seem to be innovative.

Among the quantization choices, FP4 (E2M1) is more commonly used than NF4 because its dequantization step is simpler. Therefore, FP5 should also be compared with FP4 in terms of accuracy and efficiency, making it easier for users to choose the FP format that best suits their specific scenarios.

In Figure 7, an end-to-end model throughput test is needed to fully compare the efficiency of different formats.

1. The idea of using low-bit floating point for quantization is not novel. 
2. The proposed format uses 5-bit. It is not hardware-friendly, though the authors have proposed a way to separate the value and sign for this issue.
Comments, suggestions and typos: No Figure6(a)(b) (Line 330,333) can be found in this paper. Perhaps they should be Figure5?

1. The paper directly compares the E2M2 format with the naive RTN method for integer quantization. It is not fair since the nature of quantizing into floating points and integer points are very different. It will be more convincing to compare with existing post-training quantization algorithms, like GPTQ, Quip, MagR, etc.
2. There exists other work using FP4 to quantize LLMs. One example is FP4-Quantization: Lossless 4bit Quantization for Large Language Models. It is better to also compare with them. 
3. The NVIDIA Blackwell architecture supports FP4 precision. It might be too late to work on this at this moment.
Overall assessment scores: 3

1.5
Metareview score: 2.5
Metareview: This paper introduces FPE2M2, a novel 5-bit floating-point quantization scheme designed to balance efficiency and accuracy for LLM inference. By extending the IEEE 754 standard into an ExMy format (2 exponent bits, 2 mantissa bits), the authors achieve near-lossless weight quantization while enabling register-level conversion to FP16—avoiding the memory overhead of lookup tables (e.g., NF4). Experiments on models like LLaMA and tasks across text/image/audio modalities show competitive accuracy with faster inference speeds compared to integer quantization and NFS baselines.
Summary of reasons to publish: It challenges the "4-bit or bust" trend in quantization, showing 5-bit floating-point can be both accurate and efficient—a nuanced but valuable insight.

The FP16 conversion trick is clever and well-validated, though its real-world impact needs clearer benchmarks (e.g., latency vs. INT4).