SuFP: Piecewise Bit Allocation Floating-Point for Robust Neural Network Quantization
Authors: Geonwoo Ko
Sungyeob Yoo
Seri Ham
Seeyeon Kim
Minkyu Kim
Joo-Young Kim
Venue: Accepted by TMLR ()

Abstract: The rapid growth in model size and computational demand of Deep Neural Networks (DNNs) has led to significant challenges in memory and compute efficiency, necessitating the adoption of lower bit-width data types to enhance hardware performance. Floating-point 8 (FP8) has emerged as a promising solution, supported by the latest AI processors, due to its potential for reducing memory usage and computational load. However, each application often requires its own optimal FP8 configuration to achieve high performance, resulting in inconsistent performance and increased hardware complexity. To address these limitations, we introduce Super Floating-Point (SuFP), an innovative data type that integrates various floating-point configurations into a single representation through a piecewise bit allocation. This approach enables SuFP to effectively capture both dense regions near zero and sparse regions with outliers, thereby minimizing quantization errors and ensuring full-precision floating-point performance across different models. Furthermore, SuFP’s processing element design is optimized to reduce the hardware overhead. Our experimental results demonstrate the robustness and accuracy of SuFP over various neural networks in the vision and natural language processing domains. Remarkably, SuFP shows its superiority in large models such as large language model (Llama 2) and text-to-image generative model (Stable Diffusion v2). We also verify training feasibility on ResNet models and highlight the structural design of SuFP for general applicability.

Published in Transactions on Machine Learning Research (07/2025)
SuFP: Piecewise Bit Allocation Floating-Point for Robust Neural Network Quantization
Geonwoo Ko
geonanoko@@aiis..cckk
Korea Advanced Institute of Science and Technology (KA1ST)
Sungycob Yoo"
wwngyeob.yoo&kkaistt..
Korea Advanced Institute of Science ond Technology (KA1ST)
Seri Ham
seri1215@kaisk or.kr
Korea Advanced Institute of Science ond Technology (KAIST)
Secycon Kim Korea Advanced Institute of Science and Technology (KAIST)
secyakim@kaast..cc ac.
Minkyu Kim
münkkgu.iin&&kraflonn.cm
KRAFTON
Joo- Young Kim Korea Advanced Institute of Science and Technology (KAIST)
jooyoung 120 200@@kiish...m. ac. ac.kr
Reviewed on OpenReview: https: // openreview. net/ por/forum? id- 7Maadiinf
Abstract
The rapid growth in model size and computational demand of Deep Neural Networks (DNNs) has led to significant challenges in memory and compute efficiency, necessitating the adoption of lower bit-width data types to enhance hardware performance. Floating-point 8 (FP8) has emerged as a promising solution, supported by the latest. AT processors, due to its potential for reducing memory usage and computational load. However, each application often requires its own optimal FP8 configuration to achieve high performance, resulting in inconsistent performance and increased hardware complexity. To address these limitations, we introduce Super Floating-Pon PPint (SuFP), an innovative data type that integrates various floating-poin poilingg-poitt configurations into single representation through a piecewise bit allocation. This approach enables SuFP to effectively capture both dense regions regions with outliers, thereby minimizing quantization errors and ensuring full-precision floating-point performance across different models. Furthermore, SuFP's processing clement design is optimized to reduce the hardware overhead. Our experimental results demonstrate the robustness and accuracy of SuFP over various neural networks in the vision and natural language processing domains. Remarkably, SuFP shows superiority large models such as large language model (Llama 2) and text-to-image generative model (Stable Diffusion v2). We also verify training feasibility ResNet models and highlight the structural design of SuFP for general applicability.
and sparse
Introduction
Deep Neural Networks (DNNs) have demonstrated exceptional performance across a wide range of applica- tions, including Computer Vision (Deng et al., 2009a) and Natural Language Processing (NLP) (Chowdhary & Chowdhary, 2020). Moreover, DNNs are now excelling in stale-of-the-arl generative models, such
"These authors contributed equally to this work

---

Published in Transactions on Machine Learning Research (07/2025)
0.08
0.08
0.04
0.08
0.02
002
0.02
0.00
100
80
60
60
500
1000
2000 Dim
1500
2500
Din
500
1000
Dim 0
2000
2500
3000
Dim
500
1000
Dim 0
1500
2000
3000
40
Dim
FP8 (E5M2)
FP8 (E4M3)
SuFP
Figure 1: Comparison of the Mean Square Error (MSE) for FP8 formats and SuFP in ViT-B/16 model activation layer. F5M2 and E4M3 are used as representative FP8 formats supported by NVIDIA GPUs. The SuFP format exhibits significantly lower MSE values compared to these IEEE-like FPS formats.
text-to-image models (Rombach et al., 2022) and Large Language Models (LLMs) (Touvron et al., 2023). This extended applicability further elevates the standing and importance of DNNs within the broader AT landscape.
Recent progress in DNNs has largely come from scaling them up. This trend has led to exponential growth in model size and computational complexity, which in turn demands larger memory footprints and greater compute capacity (Gholami et al., 2021). This growth creates significant bottlenecks in every type of com- putational hardware, from servers to edge devices. Consequently, adopting lower bit-width data types is essential to enhance hardware performance. To address this demand, recent studies have focused on main- taining original model performance while improving its memory footprint and compute efficiency through low-bit operations.
leading low-precision format for Al workloads and is already supported by recent NVIDIA GPUs such as H100 and Blackwell (NVIDIA, 2023; 2024). FP8 represents data through exponent and mantissa, similar to Ploating-point 32 (FP32) and Floating point 16 (FP16). However, due to its insufficient bit-width, FP8 struggles to accurately represent data distributions, as illustrated in Figure 1. The complexity and diversity tensor distributions make it difficult to apply a single FP8 configuration across various applications. Each application requires an optimal configuration of exponent and mantissa bits (e.g., E5M2, E4M3, E3M4, and E2M5) tailored its specific data distribution characteristics. Consequently, using single FP8 configuration for multiple applications can lead to performance degradation. To address this, mixed FP8 configurations have been explored as one approach to optimize performance (Shen et al., 2023), but at the cost of extra algorithmic complexity and hardware overhead. These challenges highlight the need for innovative solutions to effectively leverage FP8 across varions applications without compromising performance or increasing hardware complexity.
Floating-poih (FP8) has emerged
In this paper, we introduce Super Floating-Point (SuFP), a novel data type designed to overcome the limitations of FP8 by ensuring model performance, enhancing robustness, and optimizing hardware efficiency, SuFP employs piecewise bit allocation strategy, integrating various floating-poins configurations optimized for different regions of the data distribution into single data type. This enables the elfective representation of the data distribution across wide range of models. Specifically, SuFP allocates different numbers of bits to capture both the donse ncar-zero region and the sparse region containing outliers, maximizing representation efficiency within a constrained bit-width. Figure highlights the effectiveness this approach by showing that our data type achieves lower Mean Square Frror (MSF) across the entire tensor compared to various FP8 configurations. In this way, SuFP maintains low quantization errors across diverse data distributions in various models, ensuring broad applicability. Moreover, SuFP is highly optimized for hardware; its scaling bias, using power-of-two www.reewwwwwwweweeeeeeeeeeeeeeee representation, only requires integer addition for scaling, whereas conventional FP8

---

Published in Transactions on Machine Learning Research (07/2025)
necessitates FP32 multiplication. This enables coverage of a diverse dynamic range with negligible hardware overhead. Additionally, the tailored hardware for SuFP consists solely of an integer arithmetic unit and a shifter, allowing for a highly compact hardware configuration.
In summary, the contributions of SuFP are:
Ensuring Model Performance: SuFP employs a piecewise bit allocation to integrate various floating-point configurations optimized for different regions of the data distribution into a single data type. This method ensures effective representation and minimizes quantization errors across diverse models, as demonstrated by the lower MSE compared to various FP8 configurations.
Enhancing Robustness: By allocating different precisions to capture both the dense near-zero region and the sparse region containing outliers, SuFP adapts to wide variety of data distributions. This flexibility ensures broad applicability and maintains low quantization errors across different models.
Optimizing Hardware Efficiency: SuFP is highly optimized for hardware, using a power-of-two scaling bias that requires only integer addition for scaling. This approach, combined with a tailored hardware design consisting solely of an integer arithmetic unit and a shifter, enables coverage of a diverse dynamic range with negligible hardware overhead, resulting in a compact and highly efficient hardware configuration.
2
Related Works
Integer data type. Low-bit quantization in neural networks commonly uses inleger data types due to their operational efficiency and straightforward implementation. With these data types, the full range of data values is uniformly mapped to the available integer levels. However, this method makes it difficult accurately represent both dense data regions and sparse outliers, making hard to maintain model accuracy.
To address these issues, various studies have proposed several methods. Xiao et al. (2023) introduces extra scaling factor to handle outliers better. Similarly, Dettmers et al. (2022) employs higher precision for specific parts of the data to mitigate this issue. Due to the significance of handling quantization error, Wu et al. (2020) implements additional recovery methods, while Yao et al. (2022) incorporates knowledge distillation to improve accuracy. Additionally, Frantar et al. (2022) chooses to quantize only the weights simplify the process and reduce errors. These various approaches highlight the ongoing challenges and complexities associated with low-bit integer quantization.
Floating-poine data type. Floating-pon point data types present a promising alternative for low-bit quan- tization, especially many neural network models exhibit Gaussian-like distributions in their weights and activations. Various studies have explored different aspects and methodologies for implementing low-bit ploating-point quantization. Zhang et al. (2023) demonstrates that doating-point dała types often oulper- form integer data types in activations. However, due to the complexity and diversity of tensor distributions, there is no universally superior format for all tensors or layers. This complexity and diversity indicate limitations the general applicability of low-bit floating-point formats. response to these limitations, alternative approaches such as the posit number system and EFloat have been explored. Gustafson & Yonemoto (2017) and Langroudi et al. (2020) propose the posit number system which offers higher accu- racy and dynamic range compared to traditional floating-poie point formats. However the posit number system requires more complex hardware implementations. Our analysis additionally reveals that the posit number system is ill-suited to represent the tensor distribution of various DNN models. We discuss the details of those obsorvations in the Appendix D. Similarly, Bordawckar et al. (2021) introduces EFloat, an entropy- coded format designed to reduce memory usage while maintaining precision; however requires additional complex decoding hardware. To further address the diverse needs of low-bit quantization, Sun et al. (2019) suggests that mixed configurations of floating-poiut formats can provide robust accuracy improvements. Such insights, as highlighted in Kuzmin et al. (2022), may lead to hardware overheads since they do not take hardware efficiency into account. Furthermore, Micikevicius et al. (2022) and Wu et al. (2023) suggest
3

---

Published in Transactions on Machine Learning Research (07/2025)
Variable Encoding Field
Algorithm 1: SuFP Decoding Algorithm
Variable Data Field
Input Binary 8-bit Data [9T, for... boj; scaling
Output: SuFP Quantized Data
b
b
b
b+
b
b
b
Seafy by
bias + Sealing Bias if be == then
S
Caufp (100s) ...
mawfp bybn ...
1
s
mantissa
else
7
If byde # 11, then
coufp + (102)
2
S
exponent
mantissa
9
Mzufp + 163b2b1b0
3
s
exponent mantissa
Goufp 112 byba manly t- 162 ...
Figure 2: Visual representation of multi-region piece- wise bit allocation of SuFP.
18 return (-1)"aufe moufy" 2 2e thine
that achieving high performance with low-bit Noating-point data types often requires additional fine-tuning the use of quantization error compensation schemes, which can introduce further complexity.
A common observation from these studies is that floating seint data types offer a more versatile approach to addressing diverse data distributions compared to integer data types. This versatility arises from their ability to adjust the exponent and mantissa bits to handle varying data ranges and resolutions. However, if the bit allocation between the exponent and mantissa not optimized for each data distribution, it can negatively impact the overall accuracy and efficiency of the models.
Task-specific quantization. Task-specific quantization strategies aim to tailor quantization schemes to the specific characteristics of each model or task. Yuan et al. (2022) proposes twin-uniform quantization with Hessian-guided scaling to achieve near-lossless 8-bit performance on Vision Transformers. Wang et al. (2024) adopts distribution-awwre activation quantization and structural risk mininization-based timestep selection to preserve image generation quality in diffusion models at 6-bit. In addition to post-training methods, recent studies have explored the application of low-bit precision during model training. Peng et al. (2023) introduces an automatic mixed-precision framework that extensively applies FP8 to most computation and storage paths for large-scale language models. However, task-speeific quantization approaches often rely customized functions and calibration procedures, which limiting their applicability to diverse models and tasks.
Recognizing these limitations, our work introduces SuFP, which overcomes the limitations of previous studies by optimizing bit allocations and utilizing variable encoding fields to ensure full- precisioon model performance, enhance robustness, and optimize hardware efficiency.
3 Super Floating-Pon Point (SuFP)
The key idea of SuFP multi-region piecewise bit allocation, which integrates various floating point config- urations optimized for different. regions within data distribution into single data type. This scheme allows SuFP to accurately represent both dense regions near zero and sparse regions with outliers, ensuring wide dynamic range with efficient bit utilization.
Multi-Region Piecewise Bit Allocation. SuFP data type comprises the sign bit (MSB), an encoding field, and a data field. SuFP can present three different data representations of exponent-and-non mantissssss combinations based on the encoding field, as shown in Figure 2, and the data field is interpreted according each representation.
The overall decoding process is described in Algorithm 1. Once the representation is determined, exponent (Czufp) and mantissa (mzufp) are determined based the data field. Meanwhile, to achieve different

---

Published in Transactions on Machine Learning Research (07/2023)
precisions, each representation has a distinct exponent baseline. The exponent baselines for representations ! and are -4, -2, and 3, respectively.
There are other features that influence the precision of each representation. The representation ! is designed express numbers close to zero with high gramularity by using the entire data field for the mantissa. Notably, by setting by and by to 002, achieves the representation of subnormal numbers in the IEEE Hoating-poii standard. On the other hand, both the representation @ and 3 divide the data field into exponent and mantissa sections. Specifically, the represeniation ② aims to caplure broader range of numbers, including outliers, even with its fewer mantissa bits compared to representation @ In contrast, the representation 2 focuses on numbers within an intermediate range between representation ! and 3 It offers finer precision due to its wider bit-width mantissa, allowing for an optimal expression mumbers between the main body and outliers.
E2M5
E3M4
Data Type
E4M3
E5M2
59
153
Original
E4M3
SuFP
E5M2
SuFP
-400
-200
400
Data Distribution
(a)
(b)
Figure 3: (a) Illustrates the data distribution according to each FP8 configuration applying the scaling bias and (b) compares the quantized tensor distributions of the representative FP8 formats, E5M2 and E4M3, with SuFP. As shown in (b), SuFP data distribution most accurately represents the original tensor.
In summary, the encoding field is determined by comparing the exponent value of given number against predefined threshold values. These thresholds segment the data distribution into regions, each represented by one of the three encoding schemes. Thus, numbers closer to zero are allocated more mantissa bits, while larger outlier values receive additional exponent bits.
Building on these specifics, each representation offers different levels of precision within its respective range, effectively covering a wide spectrum of numbers. For example, nunbers the range to 4, which belong the main body, are captured with a granularity of 2-4 On the other hand, outliers in the range of 256 to 448 are captured with granularity of 20 Due to the implementation of the variable encoding field and variable data field, SuFP is capable of effectively representing both sparse and dense regions. This capability is illustrated in Figure 3, which demonstrates how SuFP follows data distributions compared to other FP8 configurations. For a detailed analysis of multi-region piecewise bit allocation, please refer to Appendix A and Appendix B.
Computation with SuFP. Before beginning the computation, we need bias-sclecting process detailed in Appendix C. A scaling bias, which is the value added to the exponent baseline to determine the actual exponent value, allows for more diverse range of exponent values than what can be originally represented with the limited exponent bit-width. Once the scaling bias for each tensor is predetermined, a tensor is quantized as SuFP without the need to search for the optimal scaling bias at runtime. This enables in-situ quantization for weight and activation tensors. Real-time quantization of the activation tensor possible due the highly consistent distribution of activation data during inference, regardless of input variations.
The inference process of SuFP consists of two main steps: decoding and computation. The decoding step is straightforward and involves first extracting the encoding field from the SuFP representation. Depending
5

---

Published in Transactions on Machine Learning Research (07/2023)
on this encoding, the corresponding bit fields for exponent and mantissa are isolated. The exponent value is computed by adding the extracted exponent bits to the baseline exponent and the predetermined scaling bias.
Following the decoding step, the SuFP Arithmetic Logic Unit (ALU) utilized to execute multiplications on mantissa and additions on exponents, as described Equation (1) and (2). The largest value from the exponent addition results is identified. To align all the results with the maximum exponent value, the results shown in Equation (3). The aligned mantissa is then
of the manlissa multiplications are right-shifedd, processed through an adder tree to compute the partial sum. Finally, after all computations on the two tensors are completed, the scaling bias is applicd, as described in Equation (4). Unlike conventional FP8, which requires an FP32 multiplication unit for scaling, our SuFP only necessitates a simple integer adder since the scaling bias a power of two. This difference emphasizes the hardware efficiency of our approach.
(w: 2biasw) ) (a; 2biasA)
1) 6and Mar.j 2 2 (-1)6am maj m 2ea.t tbass
(1)
(-1) 1) s0 MRn. my Me f 20 teait ghiasw biasa
(2)
1=0
(mw,ma.)) Ma.j)
>> Cmazi Emaz
(Cw,t + Ca,))
(3)
0
(w; 2blnsw) ). (ax 2blusa) xo 2hiasw biis +omer (-1) 11. (Mroo (Mon.jHaa.))
(4)
Hardware Implementation for SuFP. Figure 4 shows the architecture the proposed SuFP PE, designed process 16 parallel input data streams. While the FP8 data types use maating-oiintbasct PE, SuFP PE composed mainly of integer operation ALUs and shifters, requiring fewer hardware resources. The SuFP PE can be readily integrated into any architecture, with the systolic array being a prime example. For detailed explanation of the systolic array, please refer to Appendix G.
Mantissa Alignment
Adder Tree
FP32 Accumulator
16 SuFP ALUs
blas, blas, mxo
blas, blas,
0x
SuFP ALU
mg
mg
m'o
myo
8yo
e0
mx1
xx1
SuFP
m
m
m
Fixed Point
myt
8y1
ALU
e
BF16
Mx15
xx15
SuFP
m15
m15
15
My15
By15
ALU
e15
mmax
SuFP ALU
Alignment Process
Max Exponent Finder
Integer Muitiplier
ep
my
e
I
mx2e
(shift right)
0x
2's Complement
@max
m
m' max
Adder
mmax
055
Figure 4: Proposed processing clement architecture for SuFP.

---

Published in Transactions on Machine Learning Research (07/2023)
4 Experiments
This section evaluates the proposed SuFP. We comprehensively assess SuFP's performance by quantizing both weights and activations across various models, including vision, language, and generative tasks. By comparing Quantization Signal-to-Noise Ratio (QSNR) (Darvish Rouhani et al., 2023) and hardware effi- ciency, we highlight the superior performance of SuFP over conventional FP8 configurations. Quantitative results show that SuFP achieves high accuracy across different models with minimal performance degrada- tion, while qualitative results demonstrate high fidelity image generation tasks. Additionally, we validato the applicability of SuFP in training neural networks, demonstrating its stability and precision with minimal performance drop.
4.1 Baselines and Experimental Setup
We implement SuFP using PyTorch with HuggingFace transformer and TorchVision libraries. We adopt post-training quantization (PTQ) for evaluating the effectiveness of our approach across various pre-trained models. For computer vision tasks, we benchmark our method on the ResNet18, ResNet50 (He et al., 2016), Vision Transformer (ViT) (Dosovitskiy
al., 2020), and EfficientNet-v2 (Tan & Le, 2021) models with the ImageNet dataset (Deng et al., 2009a). For natural language tasks, we benchmark our method using the BERT-base model (Devlin et al., 2018) datasets such MRPC, CoLA (Warstadt et al., 2018), and SQuAD 2.0 (Rajpurkar et al., 2018). For text-to-image generative tasks, we benchmark the Stable Diffusion v2 (Rombach et al., 2021) on the COCO dataset (Lin et al., 2014). For LLMs, we benchnark our method using Llama 2 model Touvron et. al. (2023) on MMLU. We compare the performance of the proposed SuFP with the baseline data types, including FP32, FP16, Brain Floating-Poii 16 (BF16), and different configurations of FPS.
approach using
Training experiments (Section 4.4) are conducted by emulating representable values of SuFP. We focus on input tensors of computation-iintessive operations, such convolutions matrix multiplications, which contain weight, activation, and gradient tensors. We consider FP16 as a baseline and use the same hy- perparameters from the baseline for SuFP. We train image classifier using ResNet-18 (He et al., 2016) and ResNet-50 (He et al., 2016) backbone on CIFAR-10, CIFAR-100 (Krizhevsky et al., 2009), ImageNet- 100 (Deng et al., 2009b) and Tiny-ImageNe (Le & Yang, 2015) datascts. We train each classifiers three times with different seeds and report the mean of final accuracy.
Additionally, we use SystemVerilog implement the SuFP PE and various configurations of FP8. All designs are synthesized using the Synopsys Design Compiler, optimized for the 28nm CMOS technology. and set to operate at 1 GHz clock frequency. In addition, we evaluate the power estimation of each PE with internal power, switching power, and leakage power. The implementation details of the previously mentioned PEs are claborated in Appendix F.
4.2 Comparison of QSNR and Hardware Efficiency
In this section, we present an analysis of QSNR and hardware efficiency for SuFP and various FP8 con- figurations. QSNR is used measure the numerical fidelity of various quantization schemes, with higher QSNR values demonstrating that the quantized values better follow the original values and their distribu- tion (Darvish Rouhani et al., 2023). Hardware elficiency is expressed as the product of throughput and the number of operations, divided by the product of area and power consumption. This metric indicates how efficiently a system can perform a large number of operations within a given area and power budget. As shown in Figure 5, while conventional FP8 configurations face a trade-off between hardware efficiency and QSNR, SuFP overcomes this trade-off and performs highly in both aspects. The results show that SuFP consistently achieves high QSNR values and maintains advantages in hardware efficiency across various models and tasks, including vision, language, and generative applications. Specifically, SuFP shows reliable performance in both weight and activation quantization, indicating its robustness and versatility.

---

Published in Transactions on Machine Learning Research
Hardware Efficiency vs. QSNR (weight)
Hardware Efficiency vs. QSNR (activation)
38
Better
Better
30
25
14
0SNR
QSNR
32
20
15
30
E4M3
E5M2
SuFP
E5M2
SUFP
10
0.3
0.4
0.5
0.7
0.8
0.9
0.6
0.7
0.8
Normalized Hardware Efficiency
Normalized Hardware Efficiency
Model
ResNet-18
ResNet-50
EfficientNet
VIT-B
Bert
Llama2
Stable Diffusion
Figure 5: Comparison of hardware efficiency and QSNR for weights (left) and activations (right) across various models.
4.3 Inference Performance Analysis of Various Models
To highlight the robustness of SuFP across various domains, we evaluate the impact on the performance of our SuFP on various models. We extend our experiments to models in the vision, language, and generative tasks areas and compare the performance of SuFP with that of the standard formats like FP32, FP16, BF16, and different variations of FPS.
Table 1: Comparison of inference performance among various models with SuFP and other data types. For vision tasks, the models used in our experiments include ResNet (RN), EfficientNet (Eff.Net), and ViT- B/16, evaluated on the ImageNet benchmark for computer vision. For the NLP tasks, we use a BERT-base model, evaluated MRPC, CoLA, and SQuAD 2.0 benchmark. Bold denotes the best result among 8-bit representation.
Vision
NLP
Method
bits
RN18
RN50
Nef.
ViT-B/16
MRPC
CoLA
SQuAD 2.0
FP32
32
69.76
76.13
81.31
81.07
0.8307
0.5678
78.87
BF16
16
69.80
76.11
81.31
81.04
0.8307
0.5635
78.86
E5M2
8
67.04
72.96
78.59
80.42
0.8388
0.5793
78.55
E4M3
8
68.82
75.32
80.43
80.84
0.8319
0.5566
78.56
F3M4
8
69.55
75.83
81.15
80.99
0.8319
0.5604
78.70
E2M5
8
69.70
76.00
77.86
80.88
0.8157
0.5511
78.46
SuFP
8
69.81
76.17
81.26
81.07
0.8371
0.5866
78.95
4.3.1
Quantitative Results
Table compares the performance in the vision and NLP domains. For vision tasks, evaluate ResNet-18, ResNet-50, NeficientNet-v2,, and VIT-B/I 16 models using the top-1 accuracy metric. According to the exper- iments, SuFP rosults in a nogligible accuracy decrease of less than 0.06% across all vision models, indicating that SuFP is comparable to full fall-precision in vision tasks. Moreover, SuFP outperforms conventional FP8 representations terms of accuracy. For instance, SuFP achieves 81.26% accuracy for EfficientNet, only 0.06% below the original accuracy of 81.31%. In contrast, E3M4, the highest-performing conventional FP8 representation in this scenario, exhibits an accuracy of 81.15%, representing 0.20% decrease compared to full-precision.
8

---

Published in Transactions on Machine Learning Research (07/2025)
For NLP task, we evaluate the BERT-base model on the MRPC, CoLA, and SQuAD 2.0 datasets using accuracy, MCC, and F1-Score as the performance metrics, respectively. Based on the experimental results, SuFP demonstrates notably better results across varions benchmarks for the BERT-base model comparod to FP32. In addition, SuFP performs better than FP8s most scenarios. These results indicate that. SuFP delivers outstanding performance compared FP8s with the same bit-width.
Comparison of performance of Llama 2-7b with SuPP and other data types. Bold denotes the best result among 8-bit representation.
Table 2:
Method
# bits
STEM
Humanities
Social Sciences
Other
Average
FP16
16
0.369
0.433
0.518
0.525
0.459
E5M2
8
0.214
0.242
0.217
0.238
0.229
E4M3
8
0.363
0.421
0.509
0.517
0.450
E3M4
8
0.338
0.361
0.454
0.465
0.401
E2M5
8
0.274
0.249
0.269
0.276
0.266
SuFP 8
0.370
0.426
0.516
0.516
0.455
Table 9 shows the performance of SuFP on LLM. We evaluate the Llama 2 model on the MMLU bench- mark. Based on the experimental results, SuFP matches the baseline FP16 for this task, with performance decreasing only slightly by 0.87% from 0.459 to 0.455 on average. Notably, the fact that SuFP achieved high performance with just bit-width quantization through direct adoption without fine-tuning is highly meaningful.
Based on the overall results, the optimal configuration for each model within conventional FP8 representa- tions is inconsistent. For example, in the case of RN18, the F2M5 configuration achieves the best accuracy among conventional FP8 configurations, whereas in the NLP domain, the E5M2 configuration achieves the highest accuracy on the CoLA benchmark. In the LLM domain, the E4M3 configuration performs the best. This inconsistency makes difficult. to rely on single FP8 representation consistently achieve high performance across various models, highlighting the superior performance and versatility of SuFP.
4.3.2 Qualitative Results
We evaluate the performance of text-lo-image generation model to demonstrate the effectiveness of the SuFP. As shown in Figure 6a, SuFP produces images with quality similar to FP32, surpassing the conventional FP8 representations, such as E4M3, E3M4, and E2M5. This indicates that our SuFP has minimal information loss compared to these conventional methods. Additional samples of images generated by SuFP are included in Appendix H. Moreover, as shown Figure 6b, unlike the conventional FP8 approaches, SuFP achieves details the level of FP32, resulting more realistic images. In particular, SuFP achieves an FID scroe of 25.6262 in our experiment using the COCO dataset, showing improved performance over FP32 recording FID score of 27.0643. These results reinforce the reliability of SuFP and demonstrate its potential as efficient alternative to FP32 for high-quality image generation.
4.4 Training
We further validate the broad applicability of SuFP, especially in training neural network scenarios, which typically requires precise approximation compared to inference with pre-trained models (Wang et al., 2018). Figure visualizes the training process terms of training loss. SuFP consistently shows stabilized training without severe fluctuation in the training loss curves, Table 4, 5, 6 presents the final accuracy of each classifier. Classifiers trained with SuFP show negligible drop in performance, at most 0.85%, which demonstrates stability of SuFP in neural network training.

---

Published in Transactions on Machine Learning Research (07/)025)
FP32
SuFP
E4M3
E3M4
E2M5
(a)
FP32
SuFP
E4M3
E3M4
E2M5
FP32
SUFP
E4M3
E3M4
E2M5
Eyes
Eyes
"An old man in suit and tie is staring."
"A cat that is staring at the camera.
(b)
Figure 6: (a) Comparison sample images and (b) details of images generated by the Stable Diffusion model on the COCO dataset using FP32, SuFP, and FP8 representations (E4M3, E3M4, and E2M5). Note that E5M2 excluded from the comparison because it fails to generate images. SuFP follows the original FP32 images more closely than other FP8 representations, capturing not only the overall composition but also subtle textures and fine-grained details, thereby demonstrating higher fidelity in quantization.
Table 3: R8sNet18 Classification results on CIFAR- 10 and CIFAR-100 dataset.
Table ResNet50 Classification results on CIFAR- 10 and CIFAR-100 dataset.
CIFAR-10
CIFAR-100
CIFAR-10
CIFAR-100
Method
Method
Top-1
Top-5
Top-1
Top-5
Top-1
Top-5
Top-1
Top-5
FP16
94.87
99.84
76.95 76.40
93.61 93.27
FP16 SuFP
95.37
99.92
78.29
94.57
SuFP
94.14
99.81
94.83
99.84
78.65
94.76
10

---

Published in Transactions on Machine Learning Research (07/2025)
Table 5: ResNet18 and ResNet50 Classification re-
Table ResNet18 and ResNet50 Classification re- sults Inageee dataset.
sults on ImageNet-110 dataset.
ResNet18
ResNet50
ResNet18
ResNet50
Method
Method
Top-1
Top-5
Top-1
Top-5
Top-1
Top-5
Top-1
Top-5
FP16
81.98
95.50
84.54 83.69
96.48 96.29
FP16
51.38
75.98
54.51
78.51
SuFP
81.41
95.43
SuFP
51.26
76.13
54.79
78.81
FP16
SuFP
SuFP
SuFP
SuFP
81
32
9.3
0
20
40
60
100
10
20
10
20
40
50
Epoch
Epoch
Epech
Epoch
(a.) ResNc116-CIFAR10
(c) ResNet18-CIPAR100
(c) ResNet Immale/0e010aaaeee110100
(g) ResNet ResNeet1--innymaaaaee TinyImageNet
FP16
FP16
FP16
PP1
SuFP
1.5
Loss
0.0
20
00
100
40
60
100
10
40
Epoch
Epech
Epoch
Epach
(b) 1TFAARI
(d) ResNetsO-CUFARR10
(f) ResNet55--mageNNttt
(h) ResNee55- Tineem
Figure 7: Visualization of training dynamics through training loss.
5
Conclusions
This paper introduces the Super Floating-Point (SuFP), designed to address the challenges of large and complex DNNs. SuFP optimizes precision in each data region by employing multi-region piecewise bit allocation. By integrating various floating-point configurations into a single representation, SuFP effectively captures data in both dense near-zero regions and outlier regions, adapting to diverse data distributions. This approach overcomes the limitations of FP8, which often requires different configurations for each application, leading to inconsistent performance and increased hardware complexity. The tailored SuFP PE utilizes only integer units and shifters, ensuring a compact and optimized hardware implementation. Through this innovative method, SuFP demonstrates significant potential in enhancing the performance and efficiency of DNN computations, achieving accuracy comparable to FP32 and FP16. Consequently, SuFP is compelling solution for the robust performance of overall AI applications.
Limitation and Future work The quantization mothod proposed this paper utilizes a static quanti- zation technique that employs a power-of-twa scaling bias. Therefore, applying a method that dynamically caleulates precise floating-point scaling factors for tensors generated runtime could potentially enhance performance, Nevertheless, our proposed method achieves near high-precision floating-point performance with hardware-friendly scaling bias. Research into using more precise scaling factors for quantization remains area, for future investigation.
11

---

Published in Transactions on Machine Learning Research (07/2025)
Acknowledgements
This work was supported by ПTР grant funded by the Korea government (MSIT) (No. 2022-0-01036, Development of Ultra-Performance PIM Processor SoC with and TTTP-2025-RS 20220005422,2022220557777,,,,,,2 Graduate School of Artificial Intelligence Semiconductor).
and GByte-Memory
12

---

Published in Transactions on Machine Learning Research (07/2023)
References
Rajesh Bordawekar, Bulent Abali, and Ming-Hung Chen. Efloat: Entropy-coded floating point format for compressing vector embedding models. arXiv preprint S1K:::2022027055 2021.
KR1442 Chowdhary and KR Chowdhary. Natural language processing. gence, pp. 603 649, 2020.
Fundamentals of artificial intelli-
Bita Darvish Rouhani, Ritchie Zhao, Venmugil Elango, Rasoul Shafipour, Mathew Hall, Maral Mesmakhos- roshahi, Ankit More, Levi Melnick, Maximilian Golub, Girish Varatkar, et al. With shared microexponents, little shifting goes long way. In Proceedings of the 50th Annual International Symposium Compuler Architecture, pp. 1 1-13, 2023.
J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. ImageNet: A Large-Scale Hierarchical Image Database. In CVPR09, 2009a.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248 255. IEEE, 2009b.
Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Llm. int8 (): 8-bit matrix multiplication for transformers at scale. arXiv preprinl urXiv:2208. 07339, 2022.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding, arXiv preprint urXiv:1810.0440 2018.
Alexcy Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Un- terthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. 11929, 2020.
Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. Optq: Accurate quantization for gen- crative pro-trained transformers. In The Eleventh International Conference on Learning Representations, 2022.
Ofer Geva, Chris Berry, Robert Sonnelitter, David Wolpert, Adam Collura, Thomas Strach, Di Phan, Cedric Lichtenau, Alper Buyuktosunoglu, Hubert Harrer, et al. 1bm telum: 16-core 5+ ghz dem. In 2022 IEEE International Solid-State Circuits Conference (ISSCC), volume 65, pp. 46-48. IEEE, 2022.
Amir Gholami, Zhewei Yao, Sehoon Kim, Michael W Mahoney, and Kurt Keutzer. Ai and memory wall. RiseLab Medium Post, 1:6, 2021.
Wilfred Gomes, Altug Koker, Pat Stover, Doug Ingerly, Scott Siers, Srikrishnan Venkataraman, Chris Pelto, Tejas Shah, Amroesh Rao, Frank O'Mahony, et al. Ponte vecchio: A multi-tile 3d stacked processor for exascale computing. In 2022 IEEE International Solid-State Cirewits Conference (ISSCC), volume 65, pp. 42 44. IEEE, 2022.
John L Gustafson and Isaac T Yonemoto. Beating floating point at its game: Posit arithmetic. Super- computing frontiers and innovations, 4(2):71-86, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Norman P Jouppi, Doe Hyun Yoon, Matthew Ashcraft, Mark Gottscho, Thomas Jablin, George Kurian, James Laudon, Sheng Li, Peter Ma, Xiaoyu Ma, et al. Ten lessons from three generations shaped google's (puv4i: Industrial product. In 2021 ACM/IEEE 48th Annual International Symposium on Compuler Architecture (ISCA), pp. 1-14. IEEE, 2021.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
13

---

Published in Transactions on Machine Learning Research (07/2025)
Andrey Kuzmin, Mart Van Baalen, Yuwei Ren, Markus Nagel, Jorn Peters, and Tijmen Blankevoort, Fp8 quantization: The power of the exponent. Advances in Neural Information Processing Systems, 35:14651- 14662, 2022.
Hamed F Langroudi, Vedant Karia, John Gustafson, and Dhireesha Kudithipudi. Adaptive posit: Param- eter aware numerical format for deep learning inference on the edge. In Proceedings of the IEEE/CVP Conference on Computer Vision and Patlern Recognition Workshops, pp. 726 727, 2020.
Yann Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 7(7):3, 2015.
Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C LLwwrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pp. 740-755. Springer, 2014.
Naveen Mellempudi, Sudarshan Srinivasan, Dipankar Das, and Bharat Kaul. Mixed precision training with 8-bit floating point. arXiv preprint XrXiv:1905. 12334, 2019.
Paulius Micikevicius, Dusan Stosic, Neil Burgess, Marius Cornea, Pradoep Dubey, Richard Grisenthwaite, Sangwon Ha, Alexander Heinecke, Patrick Judd, John Kamalu, al. Fp8 formats for deep learning. arviv preprint ar Xrv::2209.005: 254433 2022.
NVIDIA. Nvidia h100 tensor core gpu. http:///wwww.aaaaaaaaccccacoaaad 2023. Ac-
cessed: 2024-05-19.
NVIDIA. Nvidia blackwell architecture technical brief.
httpps//ressurrces.vvidd.. com/
aar-e--baactelll-arrrittttt 2024. Accessed: 2024-05-19.
Houwen Peng, Kan Wu, Yixuan Wei, Guoshuai Zhao, Yuxiang Yang, Ze Liu, Yifan Xiong, Ziyue Yang, Bolin Ni, Jingcheng Hu, et al. Fp8-Im: Training fp8 large language models. arXiv preprint arXiv:2310.1883 183113 2023.
Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don't know: Unanswerable questions for squad. arXiv preprint arXio:1800 100.:2200..0322, 2018.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models, 2021.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and paltern recognition, pp. 10684 10695, 2022.
Haihao Shen, Naveen Mellempudi, Xin He, Qun Gao, Chang Wang, and Mengni Wang. Efficient post-training quantization with fp8 formats. arXie preprint ar 2rXiv:2309.14556 2023.
Xiao Sun, Jungwook Choi, Chia-Yu Chen, Naigang Wang, Swagath Venkataramani, Vijayalakshmi Viji Srinivasan, Xiaodong Cui, Wei Zhang, and Kailash Gopalakrishnan. Hybrid 8-bit floating point (hfp8) training and inference for deep neural networks. Advances in neural information processing systems, 32, 2019.
Mingxing Tan and Quoe Le. Efficientnet22: Smaller models and faster training. In International conference on machine learning, pp. 10096-10106, PMLR, 2021.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bash- lykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama Open foundation and fine-tuned chat models. arxee preprint 3rXiv:2307.09288 2023.
Swagath Venkataramani, Vijayalakshmi Srinivasan, Wei Wang, Sanchari Sen, Jintao Zhang, Ankur Agrawal, Monodeep Kar, Shubham Jain, Alberto Mannari, Hoang Tran, et al. Rapid: Ai accelerator for ultra-low precision training and inference. In 2021 ACM/IEEE 48th Annual International Symposium Computer Architecture (ISCA), pp. IEEE, 2021.
14

---

Published in Transactions on Machine Learning Research (07/2025)
Changyuan Wang, Ziwei Wang, Xiuwei Xu, Yansong Tang, Jie Zhou, and Jiwen Lu. Towards accurate post- training quantization for diffusion models. Proceedings of the IEEE/CV /VFF Conference on Computer Vision and Pattern Recognition, pp. 16026-16035, 2024.
Naigang Wang, Jungwook Choi, Daniel Brand, Chia- Yu Chen, and Kailash Gopalakrishnan. Training deep neural networks with 8-bit floating point numbers. Advances in neural information processing systems. 31, 2018.
Alex Warstadl, Amanpreet Singh, and Samuel R Bowman. Neural network acceptability judgments. arv preprint 1rXx::1055.24 71, 2018.
Hao Wu, Patrick Judd, Xiaojie Zhang, Mikhail Isaev, and Paulius Micikevicius. Integer quantization for deep learning inference: Principles and empirical evaluation. arXiv preprint arXiv:2004.09602, 2020.
Xiacxia Wu, Zhewei Yao, and Yuxiong He. Zeroquant-fp: A leap forward in lims post-training w4a8 quan- tization using floating-point formats. arXiv preprint arXiv:2307.0975: 2023.
Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models. In International Conference Machine Learning, pp. 38087-38099. PMLR, 2023.
Zhewei Yao, Reza Yazdani Aminabadi, Minjia Zhang, Xiaoxia Wu, Conglong Li, and Yuxiong He. Zero- quant: Efficient and affordable post-training quantization for large-scale transformers. Advances in Neural Information Processing Systems, 35:27168 27183, 2022.
Zhihang Yuan, Chenhao Xue, Yiqi Chen, Qiang Wu, and Guangyu Sun. Piq4vit: Post-training quantization for vision transformers with twin uniform quantization. In European conference on computer vision, pp. 191-207. Springer, 2022.
Yijia Zhang, Lingran Zhao, Shijic Cao, Wenqiang Wang. Ting Cao, Fan Yang, Mao Yang, Shanghang Zhang, and Ningyi Xu. Integer or floating point? new outlooks for low-bit quantization on large language models. Xiv preprint Xrx:223005. 12356, 2023.
15

---

Published in Transactions on Machine Learning Research (07/2023)
Appendix A
Effectiveness of Multi-Region Piecewise bit allocation
99%
Dense Region
1% Sparse Region
4.0
3.5
3.0
Frequency
2.5
2.0
0.1
0.2
0.4
0.6
1.0
max
max
max
max
max
1.5
1.0
0.5
0
0
5
10
15 Value
20
25
30
Figure 8: Real data distribution and piecewise bit allocation boundaries.
In this section, we discuss the effect multi-region piecewise bit allocation applied to SuFP. As shown Figure 8, the real data distribution can be divided into a dense region in the near-zero range and sparse region with rare large values. The dense region contains most of the data, and the sparse region consists of values that significantly impact model accuracy. Therefore, accurately representing both regions is essential to minimize model performance degradation due to quantization.
In this experiment, we analyze the performance of piecewise bit allocation in both dense and sparse regions. The effect depends on the location and number of boundaries that divide regions, which means when setting the boundaries, it is crucial to accurately ensure the granularity required by each region to minimize accuracy degradation in the model. Therefore, in this experiment, we set four boundarics 10%, 20%, 40%, and 60% for 2-region piecewise bit allocation and measure the MSE in both dense and sparse regions for each setting, segmented the data into four distinct regions. These
In the 4-region piecewise bit allocation experiment, segmentations are determined by the three boundaries that demonstrated the lowest MSE in the previous 2- region quantization experiment. The models used in this experiment are Vision Transformer, Stable Diffusion v2, and Llama 2, and the dense and sparse regions are divided into 99% and 1% of the total data region, respectively.
To perform in-situ quantization and evaluate the accuracy of quantization process, we set a specific batch size for each model and obtain the maximum value within batch to determine a scaling bias. By using the scaling bias, quantization is performed independently each tensor within every batch. Subsequently, calculate the MSE between the original and quantized tensors within the dense regions and the sparse regions.
As shown in Figure 9, achieving low MSE values in both the sparse region and the donse region challenging in the case of 2-region piecewise bit allocation. On the other hand, 4-region bit allocation exhibits relatively consistent MSE values but is higher in both of the regions. In the case of n-bit piecewise bit allocation with N-regions, the data bit-width allocated to each region is fixed to log2(N) - 1. This means that even if the number of regions is increased to represent the entire data distribution more finely, the granularity for each region is suboptimal. This issue becomes hurdle in accurately representing the entire region, including both sparse and dense regions. Consequently, this inflexible way of allocating bits leads to lower the overall accuracy the model.
In contrast, SuFP consistently shows the lowest MSE values across all models and data regions. This is because SuFP ensures optimized granularity for each data region. Specifically, explained in Section SuFP uses variable encoding field and variable data field to represent the data distribution, allowing for sufficient granularity for the high-grammlariy-rrquiieee dense region while maintaining extensive dynamic
16

---

Published in Transactions on Machine Learning Research (07/2025)
2-Region
4-Region
SuFP
x10
x10
0.5
3.5
MSE Value
MSE Value
0.3
2.5
MSE
1.5
0.2
1.0
0.0
2-Region 60%
2-Region 40%
2-Ragion 20%
2-Raglon 10%
4-Ragion
SuFP
2.Rogion 00%
22Reglon 40%
2-Reglon 20%
2-Region 10%
4-Region
SuFF
2-Region 40%
2-Rogion 20%
22Reglon 10%
4-Region
SuFP
2.Region 60%)
22Ragion 40%
2-Region 20%
2-Region 10%
4-Region
SuFP
0.0
2.Rogion 40%
2.Region 20%
2-Reglon 10%
]
4-Region
SuFP
2.Reglon 60%
2.Raglon 40%
2-Region 20%
2-Region 10%
4-Region
SuFP
2-Regoon
2.Region
Dense Region
Sparse Region
Denae Reglon
Sparse Reglon
Dense Reglon
Sparse Reglon
(a) Vision Transformer
(b) Stable Diffusion v2
(c) Llama
Figure 9: MSE value comparison of SuFP and other piecewise quantization techniques.
range for the sparse region demanding wide coverage. This approach effectively represents the real data distribution across each region, minimizing degradation in model performance.
Appendix B Analysis of SuFP Representations' Impact on SuFP PE Bit Utilization
In this session, we analyze the impact of three different representations of SuFP on bit utilization in SuFP PE. Each representation has varied mantissa and exponent bit-widths, enabling optimized bit allocation in both dense and sparse regions. Reprosentation ! has the largest mantissa bit-width, while representation 3 has the smallest. These differences also alfect the design of SuFP PE ALU. The bil-width of SuFP PE ALU is set based on representation ! which has the largest mantissa bit-width. Although representation
Representation (%)
Ratio of SuFP
100
85.03
85.02
97.93
91.27
84.94
80
69.94
70.38
18.32
24.72
20
11.74
12.01
2.98
4.94
0.004
12.32
2.03
8.44
0.29
11.83
3.23
Rep.
Rep.
Rep.
Rep.
Rep.
Rep.
Rep.
Rep.1
Rep.
Rep.
Rep.
Rep.
Rep.
Rep.
Rep.
Rep.
Rep.
Rep.
Rep.
Rep.
Rep.
Rep.
Rep.3
ResNet
ResNet 50
EfficientNet -v2
VIT-B
Stable
Liama-
18
/16
BERT-base
Diffusion v2
2-7b
Average
Figure 10: Comparison of SuPP representation proportion in weight and activation tensors across various models.
17

---

Published in Transactions on Machine Learning Research (07/2023)
is primarily utilized to effectively represent outliers, it results in relatively lower bit utilization within SuFP PE.
Figure 10 shows the ratio of the three representations when applying SuFP quantization across various models. As shown in Figure 10, the ratio of representation @ is noticeably low in all models. For instance, in EfficientNet-v2, representation & accounts for only about 0.004%. In conclusion, while representation is essential for model accuracy, its low representation ratio indicates a minor impact on SuFP PE hardware's bit utilization. Consequently, SuPP demonstrates its elfectiveness in terms of both accuracy and hardware's bit utilization.
Appendix C Implementation Details of Quantization Flow
Algorithm 2: scaling bias-optimal quantization flow
Input Model M, Bias searching range [b. by].
Full-precision weight W fv, Full-precision Activation Afp
Output: Optimal scaling weight bias bw opt
11, Optimal scaling activation bias ba_ opt (0... Optimize weight bias do
(0... ope
for layer + accuracymer +- Wynant (0... layer - + 5W layer - 1), ber_____ ope (0... layer - 1))
for bias = b, to by do
"Yquanslayer W' <- (W (Wguane(O.. .oo.ellrrr...... W layer | 1...n 1}}
{ Quant(W fp spllayer], bias)
accuracy &- Teat((M, W AAp)
if accuracy > accuracsmas then
accuracymaz f- accuracy
bw [u_opt[layer] < bias
12 // Optimize activation bias set
13. for layer +0 to n do accuraсymaz +
14
15
+ ( Ia.... 11, ooo__oppp....ooogp 1])
16
for bias +- by to by do Aquentlaayer] +- Quant(AA ( bias) A' + accuracy + Test(M, W Wqwant.44 A')
Asp layer 1... 1}}
if accuracy accuracymaz then accuracymaz: accuracy by br_opt [noyer] bias
29 return opt bn opp
The scaling bias of SuFP is set single value within each layer. In addition, once the bias is determined, it remains invariant throughout the entire inference process of the model. The bias determines the quanti- zation range and precision, significantly impacting the overall accuracy. Thus, employing a proper bias for optimization is extremely important.
In seeking the tensor-wise optimal bias, there are potential risks slipping into local optimization instead of achieving global optimization. Concurrently, individual optimal bias for weight and activation might not yield the best outcome when computed together.
Based on these insights, structure our optimization process as described in Algorithm 2. This process considers (i) interactions among adjacent layers, (ii) cumulative influences across the layers, and (iii) the syn- ergistic relationship between weight and activation. Additionally, the total time required for this procedure depends on the target accuracy level, which can dynamically change the bias searching range.
As additional implementation details for the training experiments described Section 4.4, we trained the classifier for 100 epochs using SGD optimizer with a learning rate of 0.1, momentum of 0.9, and weight decay of 5e-4. We applied loss scaling (Mellempudi et al., 2019), which allows small gradient values to be represented with smaller bit-width representation. To dynamically find the scaling bias for tensor quanti-
18

---

Published in Transactions on Machine Learning Research (07/2025)
zation, we set bias as bios = ((oor((og((mmmm(ssttt) margin. For weight and activation quantization, we set the margin term to for the CIFAR-10 dataset and 2 for the CIFAR-100 dataset. For gradient quan- tization, we set the term to for both datascts. We do not quantize the input of the first convolution and final fully connected layer to stabilize the training procedure (Mellempudi et al.; 2019). We used modified ResNet-18 architecture for CIFAR-10/ 1000 datasets; the first convolution layer replaced by kernel size 3 and stride 1, and the last pooling layer is replaced by the identity function.
Appendix D
QSNR results between SuFP and existing formats
QSNR Comparison (Weight)
QSNR Comparison (Activation)
40.0
40.0
Posit
35.0
35.0
Adaptive Posit
OSNR
OSNR
SuFP
30.0
30.0
25.0
25.0
20.0
20.0
ResNet-18
ResNet-50
EfficientNet
VIT-B
Stable Dilfusion
Berl
Llama2
ResNet-1 18
ResNel-50
EfficieniNet
ViT-B
Stable Diffusion
Bert
Llama2
Model
Model
Figure 11: QNSR analysis of SuFP and Posit data types across various models
Posit covers a wider range values, including those extremely close to zero. This allows Posit to handle a broader spectrum of numerical values effectively. On the other hand, SuFP is designed cover a relatively narrower range of data, focusing on higher precision near zero but not at the extreme near-zero values. SuFP does not represent values extremely close to zero as effectively Posit, but it allocates more bits to the data near zero, thereby achieving higher precision for these values.
It is important to note that these extremely near-zero values generally have minimal impact on the results after operations such as GEMM (General Matrix Multiply). In neural network computations, allocating more bits to significant values rather than extremely near-zero values often results in improved accuracy. This is because the more meaningful values contribute more significantly to the overall computation.
To further substantiate this point, used the QSNR, metric (Quantization Signal-to-Noise Ratio), which is widely adopted in recent data type and quantization studies. QSNR provides comprehensive measure of how well the quantized values preserve the original data distribution, which is crucial for maintaining model accuracy and performance. We compared the QSNR performance of SuFP, Posit, and Adaptive Posit across various models, including ViT, ResNet18, ResNet50, Efficient.Net, Stable Diffusion, BERT, and Llama2. As shown in Figure 11, and they consistently show that SuFP achieves higher QSNR values. This indicates that SuFP maintains higher numerical accuracy and robustness across diverse tasks.
In conclusion, the specific allocation of these bits within 8-bit data type, trivial as it may seem, has significant implications for performance when applied to neural network models. SuFP is meticulously opti- mized for the data. distributions found in neural networks, particularly targeting near-zero dense regions and sparse outliers. As previously mentioned, our QSNR measurements indicate that SuFP consistently outper- forms Posit in representing these distributions, especially in large models and diverse tasks, demonstrating superior accuracy.
Additionally, SuFP employs fixed encoding fields with corresponding fixed exponent/ maantetsmmanissaa bit-widths, which simplifies its hardware implementation compared to Posit. Posit formats require dynamic detection of leading zeros and ones, leading to increased hardware overhead. For instance, Posit decoders, particularly
19

---

Published in Transactions on Machine Learning Research (07/2025)
for es=2, can occupy roughly twice the area of a 4-bit multiplier, adding considerable hardware complexity, SuFP, on the other hand, maintains a simpler decoder design while achieving high performance across various modols and tasks.
Appendix E
Comparison with Block Floating-Point: Quantization Accuracy and Hardware Efficiency
We compare SuFP accuracy against standard formats like FP32 and BF16 as well as other PTQ techniques, including MSFP and BSFP. We use the configuration of MSFP and BSFP based on their reported superior accuracy in previous studies. Furthermore, we set the number of elements in a block to 16, as this value gives optimal performance for both MSFP and BSFP.
Table 7: Comparison of normalized accuracy among vision models with SuFP and other data types.
Data Type
Vision
Method
Weight / Activation
RosNet-18
ResNet-50
EllicientNet-v22 (s)
VIT-B/IN
FP32
FP32 FP32
1.0000 (69.76/69.7)
1.0000 (76.13/76.15
1.0000 (81.31/81331)
1.0000 (81.07/81.07)
BF16
BF16 / BF16
1.0006 (69.80/69.7)
0.9997
(76.11 (3)11)))6.3)
1.0000 (81.31 /8L31)
0.9996 (81.04/81.07)
MSFP!
MSFP / MSFP
0.9990 (69.69/80 69/69.76)
0.9991 (76.06/76. 00.066676..
0.9983 (81.00/88 (0.09///4.3))
0.9993 (81.01/81.07)
BSFPa
BSFP
MSFP
0.9987 (69.67 /69.76)
0.9993 (76.08/76.)) 1101//6613)
0,9980 (84.06/844 (84.23)
0.9981 (80. (80.92/81.07)
SuFP
SuFP / SuFP 1.0007 (69.81/69.76)
1.0005 (76.17/76.13) 0.9994 (81.26/81.55
0.9999 (81.06/81.07)
1 The precision of MSFP is characterized as MSFP16 (Ibit sign, 7bit muntissa, 8hit exponent).
2 BSFP is structured with b-bit and 2 bit mantissa, accompanied by 8-bit and 7- scale fartors corresponding to each mantissa.
Evaluation on Vision Models. Table compares the performance of various quantization techniques in the vision domain. The top-1 accuracy metric is used for performance evaluation. For consisteent comparison, source the accuracy results for MSFP and BSFP from the BSFP paper. To ensure consistency, we set up our environment on FP32 accuracy from the BSFP paper. However, FP32 accuracy for EfficientNet- v2 differs from the reported value. Thus, we normalize the accuracies of MSFP, BSFP, and SuFP relative to FP32 and focus on comparing their changes.
Table 8: Comparison of performance among language and text-to-image generative models with SuFP and other data types.
Data Type
BERT-basc
Stable Diffusion v2
Method
MRPC 1
CoLA 1 (MCC)
SQUAD 2.0 (F1-score)
COCO (FID-score)
Weight / Activation
(Accuracy)
FP32 MSFP
FP32 / FP32
0.8307
0.5678
78.8684
27.0643
MSFP / MSFP
0.8319
0.5636
78.8113
27.2551
BSFP
BSFP / MSFP
0.8336
0.5636
78.7647
SuFP
SuFP
SuFP
0.8371
0.5866
78.9547
25.6262
Evaluation Language and Text-to-Image Models. Table 8 shows the performance of SuFP on the Language and Text-to-image categories. For comparison, use FP32, MSFP, and BSFP as baseline. Tn the Language category, BERT-bbae serves as our representative model. We evaluate the BERT-base model on the MRPC, CoLA, and SQuAD 2.0 datasets using accuracy, MCC, and F1-Score as the performance metrics, respectively.
For the Text-to-image category, we experiment with the Stable Diffusion v2 model. We use the COCO dataset in this experiment, adopting the FID score as performance metric. For the experiment, we adopted our SuFP only to diffusion.
20

---

Published in Transactions on Machine Learning Research (07/2023)
Table 9: Comparison of performance among LLMs with SuFP and other data types.
Data Type
Llama 2-7b
Method
Weight / Activation
STEM
Humanities
Social Sciences
Other
Average
MSFP
MSFP / MSFP
0.372
0.431
0.523
0.524
0.460
SuFP
SuFP / SuFP
0.370
0.426
0.516
0.516
0.455
Evaluation LLMs. Table 9 shows the performance of SuFP and MSFP on the LLMs. We evaluate the Llama 2 model on MMLU benchmark.
Weight & AAtivation Size
1.0
1000
0.989
1.00
0.999
1000
0.990
1.000
1000
0.000
1,000
0.98
Normalized
380
388
0.96
0.94
1941
0.944
347
0.941
0.92
MSFP
BSFP
SUFP
MSFP
BSFP
SUFP
MSFP
BSFP
SuFP
MSFP
BSFP
SuFP
MSFP
BSFP
SuFP
MSFP
BSFP
SuFP
MSFP
BSFP
SuFP
MSFP
BSFP
SuFP
ResNet
ResNet
EfficientNet -V2
VIT-B
BERT
Stable
Llama-
Average
18
50
/16
-base
Diffusion v2
2-7b
Figure 12: Memory footprint of MSFP, BSFP and SuFP normalized to MSFP.
Memory Footprint Figure 12 shows the required memory footprint for the weight and activation tensors of models applied with the quantization techniques of SuFP, MSFP, and BSFP. They are normalized with respect to MSFP to rllustrate the effective reduction in footprint clearly. Tt worth noting that BSFP uses 128-bit bit memory due to the standard byte alignment despite its 127-bit configuration, causing 0.8% overhead. Based on the calculations, SuFP occupics 0.941x x of the memory of MSFP and 0.960 that of BSFP on average.
10
Throug Throughput hpptt Area
8.53
9.00
Normalized Operations Watt
16.47
17.04
16
Normaliized
5.71
12
13.08
4.00
4
6:40
1.00
1.25
2.06
1.00
FP32
BF16
FP8
BSFP MSFP SuFP
FP32
BF16
FP8
BSFP MSFP SuFP
(a)
(b)
Figure 13: (a) Normalized throughput per area and (b) normalized operations per watt comparison of SuFP with other baselines. The values normalized with respect to FP32.
Hardware Efficiency We compare the hardware efficiency of SuFP with that of other methods. Figure 13 (a) shows the throughput per area of PEs for various data formats, normalized with the result of FP32. The SuFP PE demonstrates the highest efficiency, 9.00x compared to FP32 PE. On the other hand, BSFP shows lower value due its use of bit-serial PE. The bit-serial operation in BSFP PE necessitates 16
21

---

Published in Transactions on Machine Learning Research (07/2025)
computations to process the multiplication of 7-bit BSFP mantissa with 7-bit MSFP mantissa using a 2-bít multiplier. In the case MSFP, while MSFP utilizes a 7-bit multiplier to multiply mantissa, SuFP employs a 6-bit multiplier, enabling SuFP PE to achieve enhanced throughput-per--nt efficiency. In more detail, SuPP PE is up to 7.20x more elficient. than that of stale-of-the-arl MSFP and BSFP PE.
We also evaluate the performance of SuFP PE in terms of energy efficiency. Figure 13 (b) shows the comparison results, presenting the number of operations per watt, which is normalized to FP32. SuFP oulperforms FP32 PE by being 1 more energy-efficiet. Even when compared to the stale-of-the-rt MSFP and BSFP, our PE is up to 8.27x more energy-effciient
Appendix F Implementation Details of Various PEs
In Section 4.2, we present an analysis of QSNR and hardware efficiency for SuFP and various FP8 config- urations. For these experiments, we implemented SuFP PE and various baseline PEs (FP32 and various FP8 configurations) using SystemVerilog and synthesized them using the Synopsys Design Compiler in 28mm CMOS technology. All implementations include pipeline flip-flops inserted to achieve a target frequency of GHz. By using this sctup, we measured the area and power values of the PEs. For a clearer compari- son, the results presented in Table 10 with the same throughput value for each PE. To provide further understanding of these results, the detailed configurations for each PE are described as follows:
FP32 PE supports full-precision FP32 operations, involving multiplication and accumulation op- erations in FP32 format for precise outcomes.
FP8 PE is configured with different variations including E2M5, E3M4, E4M3, and E5M2. Each configuration performs multiplication operations in the specified FP8 format, with BF16 format accumulator used to ensure consistent accuracy.
SuFP PE conducts 16 SuFP multiplication operations in parallel, shown in Figure 4. SuFP PE also utilizes BF16 accumulator, similar to the other PEs.
Table 10: Iso-throughput area and power of SuFP and other basclines.
16x FP32
16x FP8(E2M5)
16x FP8(E3M4)
16x FP8(E4M3)
16x FPS(E5M2)
SuFP
Area (um²)
29731.967
5844.3840
5628.6720
5205.3120
4806. 1440
3303.7200
Power (mW)
19.8528
1.9920
1.8080
1.5173
1.0966
1.1649
addition, to further clarify the hardware composition, we analyzed the area contributions of each con- stituent module in the proposed SuFP PE. Specifically, the SuFP ALUs account for 36.8% of the total area, the mantissa aligument for 28.8%, the adder tree for 23.3%, and the accumulator for 10.9%, respectively.
22

---

Published in Transactions on Machine Learning Research (07/2023)
Appendix G
Extension of SuFP PE to Systolic Array
Input SRAM #0
SuFP Processing Element
Input Decoder
SuFP ALUs
Mantissa Aligner
Input SRAM #1
Input Decoder
Adder Tree
FP32 Accumulator
PE
Ctrl
Output SRAM
Figure 14: Systolic array architecture containing proposed SuFP PEs.
A systolic array architecture consists of multiple PEs arranged a 2D array format, allowing parallel data processing. This architecture is not only adopted by Google's TPU (Jouppi et al., 2021) but is also widely utilized across various NPUs such as Venkataramani al. (2021); Geva et al. (2022); Gomes et al. (2022). Our proposed SuFP PE can also scamlessly integrate into the systolic array architecture, potentially lcading to significant performance enhancements. Figure 14 illustrates the SuFP PE (on the right) and the systolic array architecture composed of these SuFP PEs (on the left). The PEs adjacent to the SRAM directly receive the decoded data from the SRAM. Subsequently, these PEs use this data for computations and transmit the computed results and input data to neighboring PEs. Through this process, once-decoder data efficiently propagates among PEs, thus minimizing the recoding-related overhead in the systolic array.
23

---

Published in Transactions on Machine Learning Research (07/2023)
Appendix H Additional Quantization Results on Text-to-lmage Generation
In this section, we provide the results of text-to-image generation using SuFP quantization applied to full- precision diffusion models. As shown in Figure 15, there almost no difference between images generated with full-precision and those produced using SuFP quantization.
(a) Full-Preciess Precision
(b) SuFP
Figure 15: Sample images generated from Stable Diffusion model on COCO dataset with full precision and our SuFP.
24


Reviews:
[{"review_id": "8hd5tqe4Q2", "reviewer": "Reviewer_Gaj3", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n- Practical value:\nThe proposed method directly addresses industrial needs for low-latency encoding, making it highly relevant for real-world applications.\n- Comprehensive experiments:\nThe experimental validation is thorough, with extensive comparisons against strong baselines such as ELIC and VTM across multiple standard datasets.\n- Clear and well-organized presentation:\nThe paper maintains a good balance between theoretical motivation and experimental validation, with a clear and logical flow that makes it easy to follow.\n\n## Weaknesses\n- Unclear Generalization to Non-Natural Domains:\nThe experiments focus mainly on natural image datasets. It would be valuable to discuss or test how well the approach generalizes to other domains, where patch characteristics can differ significantly.\n- Ablation Studies Could Be More Comprehensive:\nWhile some ablations are provided, deeper analysis would be beneficial. For instance, the sensitivity of performance to the design of the importance indicator (e.g., using classification vs regression losses) and to parameters like skip thresholds could offer more insight into the robustness and flexibility of the approach.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes SuFP, a piecewise floating-point format designed to enable efficient, low-bit quantization across a broad range of neural network models. The core contribution lies in a patch-wise bit allocation mechanism that allows fine-grained complexity control in a single-pass encoding setup. Unlike prior methods that rely on global heuristics or multiple encoding stages, the proposed method dynamically adjusts bit allocation based on local content importance, significantly improving coding efficiency and task performance under constrained precision budgets.\nIn addition to the algorithmic novelty, the authors present a hardware-aware design that simplifies implementation using only integer units and bit-shifters, enhancing deployment feasibility. Extensive experiments are conducted on vision, language, and generative models (e.g., ResNet, ViT, BERT, LLaMA 2, Stable Diffusion), showing that SuFP achieves strong trade-offs between performance, complexity, and generalization. The paper demonstrates both quantitative and qualitative improvements over state-of-the-art quantization baselines and standard floating-point formats, confirming the versatility and robustness of the proposed format.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Provide a more thorough theoretical discussion or justification for the bit allocation mechanism.\n- Discuss potential limitations or adaptations needed for non-natural image domains.\n- Add comparisons against efficient traditional codecs (e.g., JPEG XL) to highlight practical advantages.", "broader_impact_concerns": "This work does not raise any ethical or societal concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "FMRGE8oEUS", "reviewer": "Reviewer_uZgK", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths:\n\n1. Comprehensive Evaluation: The paper provides extensive experiments in multi taks (vision, NLP, generative models) and various models (e.g., ResNet, ViT, BERT, Llama 2, Stable Diffusion), demonstrating SuFP’s versatility and robustness.\n2. Hardware Efficiency: The hardware-implementation design, relying on integer units and shifters, reduces complexity and overhead compared to existing FP8 formats, making it practical for real-world deployment.\n3. Strong Quantitative and Qualitative Results: The Quantitative results like QSNR, MSE, and accuracy, as well as the qualitative assessments (e.g., image generation quality), provides convincing evidence of SuFP’s superiority.\n\n\n## Weaknesses:\n1. Limited Discussion on Dynamic Scaling: The paper relies on a static power-of-two scaling bias, which may not fully adapt to runtime data variations. This limitation is acknowledged, but more exploration of dynamic scaling solutions could strengthen the work.\n2. Lack of latest related works: The discussed related works is relative limited, add more recent and related works is necessary. Such like: \n\n * PTQ4ViT: Post-training quantization for vision transformers with twin uniform quantization\n * Fp8-lm:Training fp8 large language models.\n * Using fp8 with transformer engine,\n * Post-training quantization on diffusion models\n\n3. Method Details: While the paper includes appendices with technical details, some aspects (e.g., exact bit allocation strategies, decoding algorithms) could be more clearly explained for readers unfamiliar with low-bit quantization.\n\n4. Lack of Comparison with Emerging Formats: The paper compares SuFP primarily with FP8, FP16, and FP32 but could benefit from benchmarking against other recent quantization formats (e.g., adaptive posit, EFloat) beyond what is briefly mentioned.\n\n5. Lack of Experimental Details: The experiments section lacks sufficient details. For instance, in Section 4.3, it is unclear whether the results are from Post-Training Quantization (PTQ). Similarly, in Section 4.4 (Training), it is ambiguous whether the results stem from direct Quantization-Aware Training (QAT). \n\n6. Insufficient Metrics for SD Models: The quantization results for the Stable Diffusion (SD) model lack specific quantization metrics. Such metrics are essential to clearly demonstrate the method’s generalization ability and should be included.\n\n7. Limited Evidence for Generalization Claim: If the authors claim that SuFP has strong generalization in both post-training and pre-training scenarios, Section 4.4 (Training) lacks pre-training results for models like ViT, LLMs, and SD. Including these would better support the claim of broad applicability.\n\n8. Lack of Memory and Inference Speed Results: The paper didn't report results on memory usage and inference speed. Given that SuFP seems to require proprietary hardware support, a comparison with FP8 in terms of memory footprint and latency (including training speed) on the same hardware would be critical. The lack of this data hinders a complete evaluation of SuFP’s practical benefits.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces Super Floating-Point (SuFP), a 8-bit floating-point data type that integrates various floating-point configurations (E4M3/E5M2/E0M7) into a single representation for DNNs quantization. The core design in SuFP is a multi-region piecewise bit allocation strategy. Besides, this paper also provide the hardware implementation. The experiments demonstrates SuFP’s PTQ accuracy across diverse DNN models, including vision (e.g., ResNet, ViT), natural language processing (e.g., BERT), and generative tasks (e.g., Stable Diffusion, Llama 2).  Besides, the experiments also validates SuFP’s applicability in training scenarios.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The requested changes includes: \n * more related works discussion \n * more clear method details\n * more clear experiments details\n * more results about memory and inference latency\n * more experiments comparison\n * more clear generation claim\n\nthe details could refer to **weakness**", "broader_impact_concerns": "This paper does not explicitly include a Broader Impact Statement.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Ba1j74vJpw", "reviewer": "Reviewer_MCFQ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. Interesting design: The proposed FP8 data type improves representational ability by dynamically adjusting the range and granularity using a bit-encoded selection mechanism.\n\n2. High-quality visualizations: The figures are well-presented, and the paper is easy to follow.\n\n3. Comprehensive experiments: The study includes evaluations on both vision and NLP tasks.\n\nWeaknesses:\n1. Reliance on customized hardware: The proposed SuFP data type requires a specialized hardware architecture for efficient execution. For efficiency experiments, I suggest adding more details on the implementation, particularly in terms of area and power comparisons. Given that the proposed data type introduces additional control logic, I would expect its area and energy consumption to be higher than static FP8. A more detailed explanation of these overheads would be appreciated by the algorithm community.\n\n2. Comparison with alternative FP8 enhancements: Another approach to improving FP8 involves combining per-block scaling factors with narrow floating-point and integer types, as demonstrated in MXFP. However, MXFP-like methods are not explicitly compared in this work. The authors mention related discussions in the limitations section, but it is unclear whether this includes a direct comparison with MXFP-like methods. If this work does not show clear improvements over MXFP, its significance may be diminished, as MXFP can be supported by existing hardware.\n\n3. Clarifications on experimental settings: According to Appendix C, fine-tuning is required after quantization for the ResNet experiments. However, the description of experiments on LLaMA suggests a setup without fine-tuning. I recommend clarifying the experimental settings at the beginning of the experiments section to ensure consistency and better readability.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents a new FP8 data type, Super Floating-Point (SuFP), which dynamically allocates bits between the mantissa and exponent by using some bits to encode the specific mantissa-exponent combination for a given value. Specifically, the first format uses 6 bits for the mantissa to represent numbers close to zero with high granularity. The second format uses 2 bits for the exponent and 4 bits for the mantissa, allowing for a wider numerical range. The third format uses 2 bits for the exponent and 2 bits for the mantissa to accommodate certain outliers. All three formats can be stored within the FP8 representation.\n\nTo support computations with dynamically assigned mantissa and exponent sizes, a dedicated processing element (PE) hardware architecture is proposed for efficient execution. Experiments across both vision and NLP tasks demonstrate superior accuracy compared to baseline fixed FP8 data types.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please refer to the Weaknesses section above.", "broader_impact_concerns": "No concerns on the ethical implications of this work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Nv4XvR0wis", "reviewer": "Reviewer_LrhJ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper offers a limited evaluation of the proposed SuFP and its supporting hardware, which makes it challenging to fully assess the effectiveness of the proposed method. \nAdditionally, some of the approaches presented are not entirely novel, as they have been explored in previous works. It raises concerns about the novelty of this paper.\n\nDetailed concerns include: \n\n1) The paper discusses a bias-selecting process to tune the scaling bias for the exponent according to data distribution. However, the concept of a configurable exponent bias is already proposed in [1].\n\n2) SuFP is described as the 'best solution' for the 8-bit FP format, and this paper claims its superior ability to represent values near zero more densely than E4M3 and offering a wider dynamic range as depicted in Figure 3. However, I have reservations about these assertions. From my analysis of Algorithm 1 presented in the paper, it appears that the dynamic range of SuFP is actually narrower than that of both E4M3 and E5M2. Furthermore, there is a discrepancy in Figure 3 which shows the maximum normal of E4M3 as being near 200, whereas the actual maximum normal for E4M3 is documented to be 448 [2]. This discrepancy raises concerns about the accuracy of the representations and claims in the paper.\n\n3) The paper asserts that SuFP can universally fit any data distribution, as depicted in Figure 4, and achieve superior accuracy across most benchmarks, as shown in Tables 1 and 2. However, it remains unclear whether these advantages are attributable to the variable encoding field or the bias-selecting process. The paper does not provide a compelling justification for why its approach would inherently offer a better fit for various data distributions. \n\n4) In my opinion, the efficiency of the proposed SuFP hardware is primarily attributed to merging 16 inputs for concurrent processing. This design strategy offers two notable advantages over conventional PE designs. Firstly, by merging 16 inputs, it significantly reduces the number of flip-flops required, which can lead to a more compact and potentially less power-intensive layout. However, this approach of merging inputs might impose limitations on performance in high-frequency processing scenarios, where the ability to handle rapid sequential inputs or operations is critical. Such drawbacks should be carefully considered when evaluating the practicality of implementing SuFP in high-frequency computing environments.\nSecondly, this architecture allows for the processing of floating-point operations using integer units, which can simplify the hardware design and improve the speed of certain computational tasks. However, the technique of integer-based operation and pre-alignment of floating-point values is already proposed in [3], [4]. \n\n[1] Kuzmin, Andrey, et al. \"Fp8 quantization: The power of the exponent.\" Advances in Neural Information Processing Systems 35 (2022): 14651-14662.\n\n[2] Micikevicius, Paulius, et al. \"Fp8 formats for deep learning.\" arXiv preprint arXiv:2209.05433 (2022).\n\n[3] Park, Wonhoon, et al. \"A 5.99 TFLOPS/W Heterogeneous CIM-NPU Architecture for an Energy Efficient Floating-Point DNN Acceleration.\" 2023 IEEE International Symposium on Circuits and Systems (ISCAS). IEEE, 2023.\n\n[4] Jang, Jaeyong, et al. \"Figna: Integer unit-based accelerator design for fp-int gemm preserving numerical accuracy.\" 2024 IEEE International Symposium on High-Performance Computer Architecture (HPCA). IEEE, 2024.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces a novel 8-bit floating-point format named Super Floating-Point (SuFP). SuFP adjusts the bitwidth of the exponent and mantissa through a variable encoding field. Additionally, the paper presents hardware designed to efficiently support SuFP-based dot products.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "No", "audience_explanation": "", "requested_changes": "1) **Ablation Study**: It is necessary to conduct an ablation study to separately evaluate the impacts of the bias-selecting process and the variable encoding field on QSNR and accuracy.\n\n2) **Impact of Variable Encoding Field**: The paper should discuss more thoroughly how the variable encoding field impacts the representation of data distributions compared to conventional FP8.\n\n3) **Operational Frequency and Hardware Details**: Clarifications are needed regarding the operational frequency of the tested hardware components. Details should also be provided on whether the area/power metrics for the FP units in Table 7 include the flip-flops. For a fair comparison, merging 16 FP8 units as a single unit for processing the dot product should be considered.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)