Block and Subword-Scaling Floating-Point (BSFP) : An Efficient Non-Uniform Quantization For Low Precision Inference
Authors: Yun-Chen Lo
Tse-Kuang Lee
Ren-Shuo Liu
Venue: ICLR 2023 poster (2023)

Abstract: In this paper, we propose Block and Subword-Scaling Floating-Point (BSFP), a non-uniform quantization scheme for the skewed and non-uniform distribution of weight vectors in neural networks. By quantizing each weight vector as the superposition of multiple subword vectors (in two's complement) with scaling factors (in Low-bit Floating-Point, LBFP), BSFP can effectively fit the distribution of weight vectors while maintaining high computation efficiency. Furthermore, we present a grid search-based MSE-optimal quantization flow and a scaled serial processing engine to complete the quantization pipeline and the infrastructure. 

The experimental results on the ImageNet classification task show that our proposed method outperforms state-of-the-art Microsoft Floating Point (MSFP) by up to 20.56% top-1 accuracy at the same weight precision and reduces up to 10.3% model size. Furthermore, BSFP outperforms MSFP by up to 2.0$\times$ computing throughput and up to 5.3$\times$ energy efficiency under the same silicon area budget.

Published as a conference paper at ICLR 2023
BLOCK AND SUBWORD-SCALING FLOATING-POINT (BSFP) AN EFFICIENT NON- CON-UNIFORM QUANTIZA- TION FOR Low PRECISION INFERENCE
Yun-Chen Lo, Tse-Kuang Lee, Ren-Shuo Liu Department of Electrical Engineering, National Tsing Hua University, Hsinchu, Taiwan
ABSTRACT
In this paper, we propose Block and Subword-Scaling Floating-Poin (BSFP), a datatype with non-uniform quantization scheme for the skewed and non-uniform distribution of weight vectors in neural networks. By quantizing each weight vector the superposition of multiple subword vectors (in two's complement) with scaling factors (in Low-bit Floating- Point, LBFP), BSFP can effectively fit the distribution of weight vectors while maintaining high computation efficiency. Furthermore, we present grid search-based MSE-optimal quantization flow and a scaled serial processing engine to complete the quantization pipeline and the infrastructure.
The experimental results on the ImageNet classification task show that proposed method outperforms sate--o--th--a Microsoft Floating Point (MSFP) by up to 18.57% top-1 accuracy the same weight precision and reduces up to 10.3% model size. Furthermore, BSFP outperforms MSFP by up to 2.0x computing throughput and up to 5.3x energy efficiency under the same silicon area budget.
INTRODUCTION
Deep Neural Networks (DNNs) have continuously enabled more and more eye-catching artificial intelligence (AI) applications Johnson et al.) (2016); [in al.
Deng et al. (2009). However, their large model size and high computational complexity hinder the wide deployment of DNNs to latency-sensitive cloud services and emergy-constrained edge devices. To address the performance and energy challenges, in addition to compacting neural network structures Sandler et al. (2018); Ma et al. (2018), reducing the bitwidths of weights or activations also have been extensively explored Jacob Jacob
al. 2018); Darvish Rouhani al. (2020); Tambe al. (2020): Li al. (2020).
Particularly, non-conventional datatypes and custom hardware are emerging to optimize the per- formance, energy efficiency, efficiency, and memory requirements of DNN inference. Prior industry and academia researches have explored low-bit floating-point datatypes Kalamkar et al. (2019); Jouppi al. (2020); NVIDIA (2022); Tambe al.) (0200), block-based floating-point datatypes Darvish Rouhani et al. (2020); Köster et al. (2017), low-bit fixed-point datatypes N NVIDIA (2020); Jacob al. (2018), and power-of-lwe fixed-point datatypes Miyashita et al.) (2016); Zhou ct al. (201)); Li al. (2020) as the potential candidates in efficient DNN inference. Among many datatypes, Microsoft Floating Point (MSFP), a kind of block-based floating-pon type as shown in Figure bb), claims to achieve the mftte-oftthe-rt tradeoff among dynamic range, DNN accuracy, and hardware complexity Darvish Rouhani et al. (2020).
This work focuses on post-training quantization, which is preferable in practice. First, for end users, involves no data (including private data) and enables low-friction deployment pipeline Nagel al. (2019). Second, according to discussions with an IC design house that tapes out Al chips in advanced technology nodes, the industry (at least their application-side customers) does appreciate post-training quantization because, in most cases, AI application companies reluctant release AI models and training data to AI accelerator companies. Although we focus on post-training quantization, still include the fine-tur fine-tuning results in Appendix A
This paper proposes Block and Subword-Scaling Floating-Point (BSFP), a new class of datatypes with bit-efficient, non-uniform quantization method and custom hardware to improve the energy

---

Published as a conference paper at ICLR 2023
Sign (s)
Exponent (e)
P Mantisse or integer (m)
(a) Block and Subword-Scaling FP (BSFP)
(b) Microsoft FP (MSFP)
(c) IEEE FP16, Google BF16, Nvidia TF19
A vector of words
A vector of
A vector of words
Each ward Single sign-mag. integer
word One FP
Dual subwords > Dual low bit (2 bits this example]
16-19
Dual low-bit bit FP scalings
One powerof-twe staling
bits
7 bits
bits
Figure I: Number system comparison between (a) the proposed Block and Subword-Scaling Floating- Point (BSFP), (b) Microsoft FP (MSFP Darvishh Rouhani et al. (2000), and (c) floating-point numbers (EEEE 754 FP16, Google BF16Jouppe et al. (2020), and Nvidia TensorFloat (TF19) NVIDIA (2022))
efficiency and performance over state of-te-of-the-ar MSFP. As shown in Figure Ta), the key idea of BSFP is to approximate each full-precision weight vector using the sum of two subword vectors with two scalings, respectively. More specifically, each subword is a low-hit (e.g., 2-bit), signed (two's complement) integer, and each scaling is a low-hit floating-poing (LBFP) number (e.g. a 7-bit one). We will show that BSFP is superior to MSFP in capturing the nonuniformity and skewness of per-vector weight distributions, which are common cases for vector of small number (e.g., 16) of weights. In addition, although BSFP adopts two scalings and two subword vectors, it can still be efficiently computed for the following three reasons. First, the computation cost of scaling is amortized over 16 weights. Second, each scaling is an LBFP and involves only low-bit operations. e.g., multiplications with 3-bit mantissa. Third, the subword vector structure happens to fit bit-serial computation architectures Qian Zhang et al. (2022); Judd et al. (2016).
One property that BSFP exhibits is to approximate the desired weight vector using both coarse and fine vectors. One subword vector with a large scaling captures large weights, and the other subword vector with small scaling mitigates the remaining deviations. Therefore, BSFP can adapt to large outliers and small resolutions simultaneously.
Figure a compares the quantization results of a real 16-element weight vector from ShuffleNet-v2 in either 8-level BSFP or 15-level MSFP. This example clearly demonstrates the potential that even BSFP with relatively fewer quantization levels achieve smaller quantization errors (e.g.. in terms of MSE) than MSFP with more quantization levels. We summarize the rationales for BSFP's superiority below:
No waste of quantization level: BSFP utilizes two's complement for each subword and does not waste precious quantization levels. In comparison, MSFP resembles and wastes one quantization level (i.e., duplicated +0 and -0). Even worse, the impact of wasting quantization levels increases as the bitwidth goes down. For instance, 3-b two's s complement number can represent eight quantization levels, 12.5% more than the seven levels of the 3-b maggittt number.
Adaptation skewed distribution: BSFP exploits the asymmetrical nature of two's complement numbers (e.g., -2, -1, 0, for 2-b two's complement numbers) and the sign of the associated scaling to adapt to the asymmetrical weight distribution of in weight vectors. In comparison, MSFP is permanently restricted to symmetrical quantization levels and leads to a waste of quantization levels fitting asymmetrical distributions.
Adapaation to non-uniform distribution: BSFP can offer non-uniform quantization levels by combining two subword-scaling vectors. In comparison, MSFP always uniformly quantizes weight vectors, which may instead exhibit non-uniform weight distributions.
Better freedom of quantization step size: The quantization step size of BSFP is defined by the two scalings, which are (low-bitee bitwidht)) floating- point values. In contrast, the quantization step size of MSFP cannot be any value other than power-of-two, e.g., 0.5, 0.25, 0.125.
2

---

Published as a conference paper at ICLR 2023
fy42b comppleentt fod 2% complement)
Properties
This Work
MSFP
No Wooded
Wested
(e.g., levels)
Quantization
leesasse of Sign Magnitall
A block of 16 weights (reat weight volues)
Wright Biorbuutin
Muaays Symmattci))
MSFP (e.g. 15 levels)
Weight Diprituttin
Adept Nan
Yes
[Waays (iitto)
Treadom Cuantication
Step Size
High
(Mast Power of Twc)
Low
14b sign-magnitude]
Figure 2: (a) Quantizing 16 real weights ShuffleNet-v2 using 3-b Block and Sebborrd-Scalinng Ploating-Point (BSFP) can achieve both lower quantization error (MSE) and lower storage than 4-b Microsoft Floating Point (MSFP). (b) Properties comparison of BSFP with MSFP.
BSFP can be deployed to servers and edge devices, achieving high performance and energy efficiency, Please note that the main advantages of BSFP are not from specific MAC architecture. Therefore, select one MAC architecture and focus on comparing BSFP to MSFP datatype on that architecture. This is also the case for the MSFP paper, which compares MSFP against BF16 and INT). Our experiments will demonstrate the robustness and generality of BSFP using several mainstream DNNS on the ImageNet classífication task. summary, this work makes the following contributions:
We propose Block and Subword-Scaling Floating-Point (BSFP), hardware-algorithm co-designed numerical datatype for DNNs that can achieve higher accuracy with better computing throughput and energy efficiency than MSFP.
We design scaled serial processing engine to support the proposed BSFP. This custom hardware enjoys small area footprint and can support various configurations of BSFP.
We identify mean squared error (MSE) as an effective criterion in determining the LBFP scaling factors of BSFP and a grid search-based MSE-optimal quantization flow to enable low-friction deployment pipeline. We present both post- post-training and fine-tuning procedures for preparing DNNs in the proposed BSFP format.
We perform extensive evaluations various DNNs of the ImageNet classification task and demonstrate that BSFP successfully outperforms MSFP in model size, quantization error, accuracy, throughput, and energy efficiency.
RELATED WORKS
There is rising attention designing custom datatypes for efficient inferencing. The wide range of formats can be categorized into four classes, i.c., low-bit floating-point low-bit fixed-point, power-of-two, and block-based floating-point.
The first category, low-bit floating-point numbers (FP16, Bfloat16 (BF16), and TensorFloat), sim- plifies the IEEE-754 floating-point formats. Representative commercial hardware include Google's TPUs (BF16) Jouppi et al. 2020) and NVIDIA's A100 GPUs (Tensorfloat, TF19) NVIDIA (2022).
The second class of datatype low-bit fixed-point datatype (e.g., INT4), whose operations are equivalent to integer operations and can achieve low hardware cost and high performance. Such fix- point numbers include 8-bo al. (2018). 4-b NVIDIA (2020): Dai al. (202T). 3-b Mellemuuli et al. (2017). and ultimately binary Hubara et al. (2016). Although fixed-point datatype achieves small hardware area and receives great popularity, requires careful model re-calibration and suffers from large accuracy drop when representing values with high dynamic range.
The third class is power-of-two numbers, including Power-of-Twee Zhou al. (2017): Miyashita et al. (2016) and Additive Power-of-Twoll et al, (2020), which utilizes approximate floating-poin weights. The power-of-two format is appealing for replacing multipliers with low-cost shifters. However, the available values are limited by the power-of-two form.
power-of-two terms to
Additionally, some prior research also explores non-uniform quantization. For example, LQ- Nets Zhang al. (2018) learns quantization levels to minimize the quantization error. Distilla- tion Polino et al. (2018) optimizes the quantization levels by learning to minimize a task loss with their teacher networks. However, these methods use limited number of floating-point numbers
3

---

Published as a conference paper at ICLR 2023
to quantize the full full-precision data points, leading to significantly larger overhead than fixed-ponn -oint
computation.
The last category is block-based fooatig--piint which forces block of (c.g., 16) Moating-point numbers to share one exponent. The block-based Roating-poinn offers low area overheads and large dynamic range. This format has enabled state-of-the-an accuracy-
Pareto frontier.
Representative examples include Intel's Lake Crest (Flexpoint Köster et al. (201)) and Microsoft's
Brainwave Fowers al. (2018) Darvsh Rouhani al. (020)).
The closest work related to this paper is MSFP in NeurIPS 2020 Darvish Rouhani et al. 2020), which is the current state-of-the-art format in terms of prea-to-aceuracy trade-offs. Our BSFP design is fundamentally different and novel compared to MSFP for the following reasons: 1) BSFP approximates full-pre pill-ireeiioon weight vector using the superposition of multiple subword-scaling vectors, which MSFP and other prior works do not explore before. Please refer to the properties comparison in Figure 2 and the corresponding discussion of BSFP's superiority. 2) We additionally design bit-serial processing engine that can support different configurations of BSFP. In comparison, the MSFP work considers bit-parallel processing engine and does not support format changes.
BLOCK AND SUBWORD-SCALING FLOATING-POIN (BSFP)
We propose BSFP that quantizes an I-element full-precision weight vector (Wip) using Neud I- element subword-scaling vectors (W (Wsuh, x scale; snn.. x (-1) m;)) where the subwords adopt the two's complement format and the scaling factors adopt the LBFP format. The default / 16 unless explicitly mentioned. These scaling factors ( (-1)#t 2 m, determine the quantization points that BSFP vector can represent. Since using two subwords achieves good accuracy-storage trade-offs, we set Nuub =
Naub
W Weub, x (-1)8:2emm, 2 (Weut, X (-1) m1) + (Weubs (-1) 20
Wsp
We apply BSFP to weights, In comparison, activations are left as MSFP as the MSFP paper does does Darvish Rouhani al. (2020). The rationales behind this decision are as follows. Weights differ from activations in that weights are available offline. Thus, weights can and also should enjoy a longer quantization time budget. Clearly, if both weights and activations adopt MSFP, as the MSFP paper does, it directly suggests that some optimization opportunities for weights are left on the table.
Computing with BSFP format. Dot products the fundamental operations in DNNs. Here show the dot product of an l-element Whefp weight vector and a Amefp activation vector.
Naub W sub, x (-1)s:29-ma) (Amsfo 28shared)
Whsfp'
Amsfp
Nanb
((Wsub)
Amssp) x ((-1)* m, 2 20
For each subword vector of BSFP, it first multiplies with the MSFP activation vector using compact multipliers and an adder tree to obtain the partial sum. Secondly, the LBFP scaling factor of BSFP is combined with the shared exponent of MSFP to get the correct scaling factor. More specifically, the exponent fields of BSFP and MSFP are summed together. Finally, we scale the partial sum with the combined scaling factor. The hardware overheads of BSFP over MSFP is to multiply the partial sum by a low-bit (3-b or 4-b) mantissa of the scaling factor.
Scaled Serial Processing Engine (S²PE). Figure 3 shows the proposed scaled serial processing engine PE) and its example in computing a 4b BSFP weight vector. Within an S'PE, the 2b multipliers chosen to enable flexibility in computing different bitwidth configurations with bit- serial computation Qian Zhang et al. (2022). This design choice achieves a compact silicon footprint and high computation flexibility.
4

---

Published as a conference paper at ICLR 2023
S4p
Exponent Maniisa
Scaled Serial Processing Engine (S"PE)
Wou
Saling
Shiting
Controller
Irput SRAM
Input SRAM
SRAM
Input
Weight SRAM
SriE
SAPE
SPPE
Weight SRAM
SAPE
SYPE
SYPE
Cycle
Weight SRAM
SPPE
Cysie
Figure 3: Proposed scaled serial processing engine (S²PE) and systolic architecture.
In addition, the proposed PE only requires an integer multiplier to scale the partial sum by the (at most) 4-b mantissa of the LBFP scaling factor. The exponent terms of BSFP and MSFP are added together to convert the scaled partial sum back to BF16 for accumulation. For BSFP with a bitwidth larger than 2, the proposed PE computes the result using multiple cycles. We finally present systolic architecture, which integrates multiple scaled serial processing engines.
Criterion-optimal quantization flow. The flexibility of BSFP requires a flow to determine a suitable LBFP scaling factors setup. We observe that the number of combinations for L.FPP scaling factors is limited; hence exhaustively evaluating all of them GPU is feasible. Specifically, every weight vectors are parallelly quantized and evaluated using the same LBFP scaling setup. For example, an optimal setup for ShuffleNet-v2 can be found within 30 minutes using one NVIDIA V100 (ResNet-50: less than hour; VIT: less than 1.5 hours).
Figure 4 shows the quantization flow, which consists of four steps: 1) Generating subword con- figurations, which generate several potential subword configurations that satisfy the target weight bitwidth. 2) Generating I.FP scaling combinations, which creates search space by exploring all potential combinations (grids) of LBFP scaling factors. 3) Iterative rounding, which quantizes the full-precision weight vectors using generated scaling factor combinations. 4) Criterion evaluation, which evaluates the difference between the full-precision weights and quantized counterparts using chosen criterion. The criterion candidates are L1, Mean Square Error (MSE), and Cosine Similarity, where MSE is selected as the final criterion. The details of the flow are presented in Appendix
Sabnward (2,2)
ScolPs: Sool 1x2
Target bitwdith e.g. 4
(3,1)
Generate LBFP
Iterative Rounding
Criterion Evaluation
Criterion- Optimal Output
Configs
Scaling Factors
1xx 2
(4,0)
Figure 4: Criterion-optime quantization flow for BSFP format.
4 BSFP CONFIGURATIONS
This section discusses the effects of different configuration setups for BSFP.
Block Size (or vector length, 1). The vector length affects the model size, the PE area efficiency. and the model accuracy. A large vector length generally amortizes the hardware overheads and reduces the storage overheads of LBFP's scaling factors. In practice, we found vector lengths of 16 to 64 to be effective for BSFP in preserving accuracy while incurring moderate hardware cost.
Quantization Criterion. The quantization criterion plays an important role our quantization flow to decide the final accuracy. Common criterion candidates are Manhattan distance (L1 distance), Mean square error (MSE, Euclidean distance, L2 distance) Zhao et al. (2019), and Cosine Similarity Zhu et al. (2019 Zhang et al. (201)). We identify MSE better criterion in quantization flow. Furthermore, users can easily define their quantization criterion in the framework.
Number of Subwords. We can tune the number of subwords (N factor explore the trade-offs for BSFP datatype. To fully obtain the benefit of BSFP, we suggest setting Nowb larger than 1, c.g.. to 4. In practice, the number of subwords set to 2 to balance storage overheads and accuracy.
5

---

Published as a conference paper at ICLR 2023
Low-bit dloating-point (LBFP) Scaling Factor Format. The configuration for LBFP scaling factor format is also crucial in determining our quantization flow's 's accuracy and search space. Appendix H shows that scaling format with 1s4m3e and 1s3m3e achieves good trade-offs.
Besides, we propose to equip separate exponent biases for different LBFP scaling factors. The key idea is to set suitable biases minimize the LBFP scaling factor's precision while achieving satisfying accuracy. We empirically set exponent biases the first and second subword scaling factors be and -8, respectively. noteworthy that both biases are the same across the whole model, which incurs negligible storage overheads. Setting per-layer biases is left as future work.
EVALUATIONS
Baselines and Network Architectures. We compare BSFP with MSFP Darvish Rouhani et al. (020)) DSQ Nagel et al. (2019). TFlite Krishnamoorthi (2018). and APoT Li et al. (2020) in accuracy-tor to-precision trade-off. Furthermore, we also compare area and power efficieccy with other strong baselines, such as BFl]]oouppi et (2020), fixed-point Nagel et al. (2019) and Power-of-Two
PEs Zhou et al. 2017 Li et al. (2020).
We select six mainstream DNNS for evaluating BSFP and MSFP. ShuffleNet-v2 Ma al. (2018 and MobileNet-v2 Sandler et al. (2018) are chosen represent compact DNNs, ResNet- 8 and ResNet-50 He et al. (2015) are selected to represent classical DNNs. EfficientNet-v2 Tan & Le (2019)
and VIT Dosovitski et al. (2020) are chosen to represent modern DNNs. All of the results based on ImageNet classification dataset Deng et al. (2009).
Hardware Evaluations. We implement the BSFP PE and baseline PEs in Verilog and validate the behavior against the software functional simulator. All designs are synthesized at 500 MHz under TSMC 40mm using Synopsys Design Compiler (Topographical mode). We set the parallelism to be 16 for all number systems.
5.1
MODEL
AND VECTOR LENGTII-O--MODEL SIZE TRADE-OFFS
Figure 5 shows the storage-to- to-acuaaccyc Pareto frontier of BSFP and MSFP on ShuffleNet-v2 (post-training quantization). BSFP consistently improves the accuracy using a smaller model size. For example, BSFP simultaneously obtains 0.4% top-1 accuracy gain and saves 5% model size over MSFP. The superiority of BSFP is consistent across four evaluated models, where other results are omitted for brevity.
Figure 5 b) compares the accuracy-to-storage trade-off's of different vector lengths for BSFP and MSFP on ShuffleNet-v2 (post-training quantization). On BSFP, the vector length ranging from 16 to 64 leads to only marginal accuracy drop, while vector length of 128 significantly decreases the accuracy. Thus, we suggest that the vector length for BSFP be 16 to 64. In comparison, MSFP's accuracy drops significantly from length of 16 to 32, and the accuracy does vary significantly while further enlarging the vector length. It also noteworthy that BSFP consistently outperforms the Pareto frontier regardless of vector length compared to MSFP.
Tog-1 Accuracy 880
better
1og-1 Accuracy 196)
better
F=33
Top-4 Accuracy
86
1448
55
84
40
62
32
-2- Lt
-0. Maz
MSRP
Simdadty
45
Model Sice (M8)
Model Sten (M1)
Subword contgurations
(a)
(b)
(c)
Figure 5: (a) Model size-to- accuracy Pareto frontiers of BSFP and MSFP with various bitwidths (fixing vector length). (b) Model size-to-accuracy Pareto frontiers BSFP and MSFP with various vector length, (fixing bitwidth). (c) Quantization using different criteria. All experiments are conducted on ShuffleNet-v2 using post-training quantization.

---

Published as a conference paper at ICLR 2023
X10+
[a] ShuffleNet-v2
12
X10
(b) MobileNet-v2
HL DlDieegence
-
MBFF
KL Divergence
10
-0-85FP
better
better
(oodel Site (MIf
Model Size (MB)
Figure 6: KL Divergence-te poorgence-to-modee to--odel size comparison of MSFP and BSFP on (a) ShuffleNet-v2 and (b) MobileNet-v2. The KLD BSFP is consistently lower than MSFP while using a smaller model size.
5.2 QUANTIZING USING DIFFERENT CRITERION
Figure ([c) shows the accuracy of quantizing ShuffleNet-v2 using different criteria, i.c., L1 distance, MSE, and Cosine Similarity. The MSE is the best criterion our grid search-based quantization flow. It is noteworthy that different criteria have little accuracy difference in higher bitwidth and vary considerably in lower bitwidths. One reason for MSE's superiority is that it punishes significant element-wiss distortion, which the other two criteria cannot. Specifically, selecting L1 norm optimizes overall distortion, and selecting Cosine Similarity optimizes the angular difference.
5.3 KULLBACK-LEIBLA DIVERGENCE ANALYSIS
Kullback-Leibler Divergence (KL Divergence) chosen for evaluating quantization quality in the MSFP paper. Intuitively, lower KL Divergence demonstrates the quantization can fit the original data much better, which results in higher accuracy. We compare the KL Divergence of MSFP and BSFP to demonstrate the superiority of BSFP.
Figure 6(a) and Figure (b) respectively sample a layer from ShuffleNet-v2 and MobileNet-v2 to compare the KL. civeegencc-to-mooee size Pareto frontier of BSFP and MSFP. The key takeaway is that BSFP consistently obtains better KL Divergence for both models while allocating smaller model sizes. Furthermore, the discrepancy between MSFP and BSFP enlarges as the model size reduces.
5.4 PER-VECTOR ABSOLUTE PEARSON'S SKEWNESS ANALYSIS
Figure analyzes the per-vector absolute Pearson's 's skewness coefficient (SK) for each layer of ShuffleNet-v2 and R8sNet-18. We report the proportion of vectors with different degrees of skewness, iee. moderate (0.5 < 5K < 1.0), high (1.0 < SK < 1.5), and very high (1.5 < SK). Two takeaways are: 1) The weight vectors modern networks, i.c., ShuffleNet-v2 and ResNet-18, skewed. Specifically, up to 50% ~ 75% of the weight vectors are skewed. The proposed BSFP can adapt to skewed distributions and outperform prior works with uniform quantization. 2) Even weight vectors that are less skewed, BSFP can still outperform prior works because it does not waste quantization levels and provides better step size flexibility.
(al ShuffleNet-22
(b) ResNet-18
Pooportoan (6)
Proportion
8 8
30
O BTO UE B1
Layer
Layer
Figure 7: Per-vector absolute Pearson's skewness coefficient (SK) of (a) ShuffleNet-v2 and (b) R8.Net-18. In general, very highly skewed means SK 1.5, highly skewed means SK > 1.0, and moderately skewed means 1.0 > SK > 0.5. Appendix GrrendixGssheee G shows that slicing weight into vectors is one potential rationale for the large skewness.

---

Published as a conference paper at ICLR 2023
ShutffeNeeww2
MobileNeee-2
ResNet 18
ResNet-S0
bottor
bach
MV7
Weight Brocidas
We gig freciioo
Figure& 8: censsionntt-Acuraae comparison of BSFP with MSFP, DFQ, TFlite, APoT. All number systems utilize post-training quantization. The accuracy of DFQ Nagel et al. (2019) and TFlite TFlst namoorthi (2018) are derived from the papers. We implement APoT to obtain the accuracy.
Table Post-training comparison of accuracy and hardware performance on ImageNet with MSFP and BSFP.
Method
Precision
Tog-1 (%)
Model Size
FLOPs FieOPs
Method
Precision A)
Top-1 (56
Model Size
FLOPs FizOPs
blobiloNet-11 StuutaNee-22
FP
82
60.15
911
148.831
8/8
20.05
2.4 ML3
148.8M
15
7.8
69.10
2,8 MB
180.2M
MSFF
7
68.27
211 XB8
113.00
DSFP
6
2.0
68.62
2.0 MB
68.7M
0.0
19
5/5
47.89
18.
67.30
1.4 MB
37 2M
FP
32
12
71.84
14.0 3013
$14.33
8
$1.08
57 MB
516.1M
7
F1.47
5.5 MB
2M
MSFP
727
8.6
63.05
24.
178.7M
BSFP
06.01
2.6 MD
776.7M
98.2M
575
$7.49
24
752751
50.85
2.2 MB
78.5M
FP
92/30
00.76
26,8 MD
1.90G
RaaNoe-18
1,39G
MSTT
6/6
7/77
67.75
00.27
00.54
11.0 30
9.5 XD
38d
710.90
1,200
1,300
(OURS)
RSFP
6
6
69.40
69.58
10.1 MB
8.7 MB
568.8M
465.0M
1.22G
4.7
$7.90
56.
455.001
01.04
268.8M
FP
02/32
76.13
100,2 MD
4.34G
RaaNet-50
8
76.06
27.2 X118
4.14G
7
2--2
18
76.08
25.4 MB
S.6G
8.17G
MSFP
6.5
78.45
75.54
23,8 MR
17.6 MB
2.3G
1.6
(OUR5))
DSFP
3-22
75.95
15.87
19.0 MB
15.8 MB
1.8G
1OG
4.24
61.95
14.0 MB
1:06
21
65.05
12.6 MB
582 2M4
FP
32/37
88.0
8.8G
878
$4.19
25.4 MU
9.80
15
84.06
21.8 MB3
7.7G
7/7
$3.94
2006 MD
6.00
6
88.96
18.1 MB
6.00
MISPP
6/66
3013
5.00
6/7
77.45
15.1 MR
3.4G
MR
2.2G
4.4
3:27
12.4 Mid
2.303
2 1
27.20
10.8 MB
1.7G
Naiion Theaalome
FP
82/322
81.07
365.1 5018
56.0
(V/7-8/16)
8/8
$1.01
92.4
56.0G
5 2
86.8 MR
49.0G
7/7
90.94
81.6 MB
42.9G
6
8087
75.4
$1.8G
MSFP
0.1.6
90.84
00.7 MB
31.56
(OURS)
DSFP
5
11
80.11
04.6 MB
17.50
57.5
21.9G
22-2
4.74
78.89
28.9 MD
1400
312-1
79.77
428 MB
19,50
5.5 WEIGHT PRECISSON-TO-ACCURAN PARETO FRONTIER
cooure&compares the weight precision-to-accueee tocacuracy Pareto frontier of MSFP and BSFP on four DNNs. BSFP consistently outperforms MSFP on accuracy, given the same weight precision. Furthermore, the accuracy gap between MSFP and BSFP enlarges while reducing the weight precision. Let us take ShuffleNet-v2 as an example. When allocating 7b weight precision for both BSFP and MSFP, BSFP achieves 69. 69.10% top-1 accuracy. outperforming MSFP's 68.27% accuracy by 0.83%. However, when further narrows down the weight bit to five, the accuracy benefit of BSFP over MSFP enlarges. Further, BSFP consistently outperforms MSFP and other numerical systems under the same precision.
Table 1 summarizes the accuracy, model size, and the number of operations on six DNNs. For full precision models, we report their computation complexity in FLOPs. We define a FixOP as one operation between an 8-bit fixed-point weight and an 8-bit fixed-point activation, which takes 64 binary operations for quantized models. For example, 4 by 6-bit multiplication is equivalent to 4x6 /8x8 =3 /s FixOP (8b). To sum up. BSFP consistently achieves higher accuracy with smaller model size and computation complexity than MSFP. Smaller FixOPs improve the computation steps on serial PE or reduce hardware overheads on parallel PF.
8

---

Published as a conference paper at ICLR 2023
Table 2: BSFP versus other mainstream number formats for DNN inference. Memory and MAC density of various formats are normalized to BFI6. The results listed are based on topographical synthesis results using TSMC 40mm process at 500 MHz.
16x BP16
16x INT4
16x INTS
18x Power
Dit Serial
Scaled Serial PE (BSFF)
2b.moul
Area (per PE)
1,x
22,6x
8.7x
7ax
13:0x
26.6x
88.1x
Power (por PF)
13.0%
ShuffleNetv2
MoblleNet-v2
ShuffleNettv2
MobileNet-22
better
Top- Meniney
201 125-
- Idlt pal-mali
--- SBFP (1b-uut))
Obrealt) NPOT
9
APUT
IMT
22
Normsilzed lsoxlree Throughput
Normalized Iso Arca Throughput
Normalaad Throughput per Wott
Normalloed Throughost
Figure 9: Iso-area Throughput to Accuracy (left) and Throughput per Watt to Accuracy (right) comparison BSFP with other strong baselines. We normalize the throughput and throughput/W to MSFP-16.
5.6 ACCURACY TO PERFORMANCE AND ACCURACY TO ENERGY EFFICIENCY COMPARISONS
Table 2 shows the area and power of PEs with different number formats. We normalize all of the designs with 6F16 PE. In general, scaled serial PE achieves the smallest area and power because of the compact MAC circuit. The 2-b scaled serial PE can outperform 2-b serial MSFP PE Qian Zhang al. (2022) because of the following reasons:
Larger adder tree induced by additional XOR gate: Besides area overheads XOR gates, the bitwidth of the multiplication results must be enlarged because of the potential negation. Consequently, the 16x4-b adder tree has to increase each input port by 1-b, causing 25 % higher complexity than the original adder tree. In brief, the XOR gates, the negation logic, and the enlarged adder tree jointly cause the area and energy in-efficiency.
The alignment overheads for MSFP are more significant than for BSFP: Although the bitwidth of scaling factors are similar for BSFP and MSFP, MSFP allocates all 8-b to be the exponent. In contrast, BSFP allocates only 3-b to be exponent and lets the rest be sign and mantissa. As a result, the alignment procedure before accumulation is largely simplified for BSFP, which improves energy efficiency.
Figure (left) shows the iso-area throughput to accuracy comparison. BSFP consistently outperforms MSFP across four models. Specifically, BSFP outperforms MSFP by 1.3x to 2.0x throughput while achieving higher accuracy. Again, the throughput improvement results from a smaller PE area and lower weight precision. Figure 9 (right) shows the throughput per to accuracy comparison. BSFP also achieves significantly better throughput per Watt than MSFP. Specifically, BSFP outperforms MSFP by 1.3x to 5.3x while achieving higher accuracy.
6 CONCLUSIONS
This paper introduces Block and Subword-Scaling Floating-Poin (BSFP) for quantizing weight vectors in neural networks, which typically exhibit skewed and non-uniform distribution. The quantized vector BSFP is the sum set of subword -wwoodd-callng vectors, bringing up 2x speedup and 5.3x energy efficiency improvements compared with MSFP.
In addition, our grid search method guarantees finding the optimal BSFP configurations for capturing skewed and non-uniform weights and optimizing the MSE criterion. With proposed scaled serial PE. BSFP obtains better area efficiency and energy efficiency compared to prior number systems. In sum, BSFP successfully reaches state-of-the-art model size, throughput, and energy efficiency.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENTS
We thank reviewers for their insightful comments. We thank NCHC (National Center for High- performance Computing, Taiwan) and TSRI (Taiwan Semiconductor Research Institute) for providing computational and storage resources. This work is supported in part by NSTC (National Science and Technology Council, Taiwan) grants 111 111-2823-8-007-00 and 111 1011-2218-E-007-20 E-007-009, Synopsys Scholarship, and NSTC Scholarship.
REFERENCES
Arm, 2020. URL http::///eeaaloeerrr..c...ooooct
Yu-Hsin Chen, Tushar Krishna, Joel S. Emer, and Vivienne Sze. Eyeriss: An Energy- -ffeice-ffic Reconfigurable Accelerator for Deep Convolutional Neural Networks. IEEE Journal of Solid-State Circuits (JSSC), 52(1):127-138, 2017. doi: 10. 109///SC.2201.22115533
Lucian Codrescu. Qualcomm Hexagon DSP: An architecture optimized for mobile multimedia and communications. In 2013 IEEE Hot Chips 25 Symposium (HCS), pp. 1-23, 2013. doi: 10.1
Steve Dai, Rangha Venkatesan, Mark Ren, Brian Zimmer, William Dally, and Brucek Khailany. VS-Quant: Per-vector Scaled Quantization for Accurate Low-Precision Neural Network Inference. In Proceedings of Machine Learning and Systems (MLsyss, volume pp. 873-884, 2021. URL http:://pprceeedngg wttpa//cc..//pppe//22222222. ED3344cd520006/33555553333000000 Pape:.
Bita Darvish Rouhani, Daniel Lo, Ritchie Zhao, Ming Liu, Jeremy Fowers, Kalin Ovtcharov, Anna Vinogradsky, Sarah Massengill, Lita Yang, Ray Bittner, Alessandro Forin, Haishan Zhu, Taesik Na, Prerak Patel, Shuai Che, Lok Chand Koppaka, Xia Song. Subhojit Som, Kaustav Das, Saurabh T, Steve Reinhardt, Sitaram Lanka, Bric Chung, and Doug Burger. Pushing the Limits of Narrow Precision Inferencing at Cloud Scale with Microsoft Floating Point. In Advances in Neural Information Processing Systems (NeurIPS), volume 33, pp. 1027 1-00881, 2020.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition (CVPR), pp. 448-555. IEEE, 2009.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale.
CoRR, abs/2010.1 1929, 2020. http:: wrwii.iivvv///..a/aaaaaaaaaaavva org/ abs///b///////// 2010 11929
Jeremy Fowers, Kalin Ovtcharov, Michael Papamichael, Todd Massengill, Ming Liu, Daniel Lo, Shlomi Alkalay, Michael Haselman, Logan Adams, Mahdi Ghandi, Stephen Heil, Prerak Patel, Adam Sapek, Gabriel Weisz, Lisa Woods, Sitaram Lanka, Steven Reinhardt, Adrian Caulfield, Eric Chung, and Doug Burger. A configurable cloud-scale DNN processor for real-time AI. In 2018 ACM/IEEE 45th Annual International Symposium on Computer Architecture (ISCA), pp. 1-14, 06 2018. doi: 101.11//A...1...00000
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. CORR, abs/1512.00336 2015.
Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El-ian,, and Yoshua Bengio. Binarized neural networks. In Advances in Neural Information Processing Systems (NeurIPS), volume 29,
Curran Associates, Inc., 2016.
Imperas, 2019.
URL
10

---

Published as a conference paper at ICLR 2023
Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. Quantization and Training of Neural Networks for Efficient Enegge-AAithmettine Only Inference. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition (CVPR), June 2018.
Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual Losses for Real-Time Style Transfer and Super-Resolution CoRR, abs/1603.08 155, 2016. URL
08155
Norman P. Jouppi, Doe Hyun Yoon, George Kurian, Sheng Li, Nishant Patil, James Laudon, Cliff Young, and David Patterson. A Domain-Specific Supercomputer for Training Deep Neural Networks. Commun. ACM, 63 jun 2020. ISSN 0001-0782. doi: 10.1 10.1145/330037. URL
https: //toi//ooii...... oroloorg/10.. 1145/ 336033
Patrick Judd, Jorge Albericio, and Andreas Moshovos. Stripes: Bit-Serial Deep Neural Network Computing. IEEE Computer Architecture Letters, 16:1-1, 08 2016. doi: 10.1 1..1109/CC.20166 2597140.
Dhiraj D. Kalamkar, Dheevatsa Mudigere, Naveen Mellempudi, Dipankar Das, Kunal Banerjee, Sasikanth Avancha, Dharma Teja Vooturi, Nataraj Jammalamadaka, Jianyu Huang, Hector Yuen, Jiyan Yang, Jongsoo Park, Alexander Heinecke, Evangelos Georganas, Sudarshan Srinivasan, Abhisek Kundu, Misha Smelyanskiy, Bharat Kaul, and Pradeep Dubey. A Study of BFLOAT16 for Deep Learning Training. CoRR, abs/1905. 12332,, 2019. URL http::://axx..oog/////// abs//
1905. 12322
Urs Köster, Tristan J. Webb, Xin Wang, Marcel Nassar, Arjun K. Bansal, William H. Constable, Oğuz H. Elibol, Scott Gray, Stewart Hall, Luke Hornof, Amir Khosrowshahi, Carey Kloss, Ruby J. Pai, and Naveen Rao. Flexpoint: An Adaptive Numerical Format for Efficient Training of Deep Neural Networks. In Proceedings of the 31st International Conference on Neural Information Processing Systems (NeurIPS), pp. 1740-1750, Red Hook, NY, USA, 2017. Curran Associates Inc. ISBN 978151 178696999
Raghuraman Krishnamoorthi. Quantizing deep convolutional networks for efficient inference: A 2018. URL
whitepaper. CoRR,
08342
Yuhang Li, Xin Dong, and Wei Wang. Additive Pfwers-of-Tre Quantization: An Efficient Non- uniform Discretization for Neural Networks. In International Conference on Learning Reprexenta- tions (ICLR), 2020.
Tsung- Yi Lin, Michael Maire, Serge Belongie, Lubomir Bourdev, Ross Girshick, James Hays, Pietro Perona, Deva Ramanan, C. Lawrence Zitnick, and Piotr Dollár. Microsoft coco: Common objects in context, 2014.
/ http///axvvv///////aa/eeee//.aea org/ aaas/// / 405. 0312 cite 1rxiv:1405.03 1005.0312Commenn 1) updated annotation pipeline description and figures; 2) added new section describing datasets splits; 3) updated author list.
Ningning Ma, Xiangyu Zhang, Hai-Tao Zheng, and Jian Sun. ShuffleNet V2: for Efficient CNN Architecture Design. In Proceedings of the European Conference Computer Vision (ECCV), September 2018.
Practical Guidelines
Naveen Mellempudi, Abhisek Kundu, Dipankar Das, Dheevatsa Mudigere, and Bharat Kaul. Mixed low-precision deep learning inference using dynamic fixed point, 2017.
Moritz Menze and Andreas Geiger. Object Scene Flow for Autonomous Vehicles. In Conference on Computer Vision and Pattern Recognition (CVPR), 2015.
Daisuke Miyashita, Edward H. Lee, and Boris Murmann. Convolutional Neural Networks using Logarithmic Data Representation. CoRR, abs/1603.01025. 2016. URL http://arxiv..rrg
abs/ / 1600..636000 01025
Markus Nagel, Mart Baalen, Tijmen Blankevoort, and Max Welling. Data-Free Quantization through Weight Equalization and Bias Correction. CoRR, abs/ 0tss/1906.44211 2019. URL http:
aarxivaaa wwt./bo///aaaooo 1906.9066400 04721
11

---

Published as a conference paper at ICLR 2023
NVIDIA. Int4 precision for Al inference (NVIDIA Technical Blog). Aug 2020. URL https
//developer. nvi awvidia.. www.coo////oo/oo. www.aaa..cc///po//aaaa-aaaaa--aa
NVIDIA.
TensorFloat-32 in the A100 GPU Accelerates AI Training,
HPC up
to 20x, 2022.
URL o 2ttp://.......coooo./////2//0/30
NVIDIA, 2023. URL http:::/gggttubb..... comnnlla
Antonio Polino, Razvan Pascanu, and Dan Alistarh. Model compression via distillation and quantiza- tion. CoRR, abs/1802.05668, 2018. URL BRLthtt:://aaee www.vvixivvv. http:///...r//b///1/0/00 1582000566620080 05668
Sai Qian Zhang, Bradley McDanel, and H. T. Kung. FAST: DNN Training Under Variable Precision Block Floating Point with Stochastic Rounding. In 2022 IEEE International Symposium on Perfo-Perforaecce Computer Architecture (HPCA). pp. 846 846-860, 2022. doi: 10.1109/HPCA5366.. 2022.00067.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang- -highh Chen. Mo- bileNetV2: Inverted Residuals and Linear Bottlenecks. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), June 2018.
Sayeh Sharify, Alberto Delmas Lascorz, Mostafa Mahmoud, Milos Nikolic, Kevin Siu, Dylan Malone Stuart, Zissis Poulos, and Andreas Moshovos. Laconic deep learning inference acceleration. In 2019 ACM/IEEE 46th Annual International Symposium on Computer Architecture (ISCA), pp. 304-317, 2019.
Synopsys, 2020.
URL https:/ http://wwww..y.ee wwwp://wwwsssso..oo.oooo wow./opsy..com//sssgi......... ip/ processor sts-ssoluse solut iosiiins//sain aatp//rc---0000 family.htm.
Thierry Tambe, En Yu Yang, Zishen Wan, Yuntian Deng, Vijay Janapa Reddi, Alexander Rush, David Brooks, and GuoYeon Wei. Aagorah co-design of adaptive floating-point encodings for resilient deep learning inference. In 2020 57th ACM/IEEE Design Automation Conference (DAC), pp. 1-6, 2020. doi: 10. SWW.IDDACI8002.222.9925525
Mingxing Tan and Quoe Le. EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks. In Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. httpp:://prrcees
PMLR, 09-15 Jun 2019. URL
ngs ml presss...ss....... / 297/v9/tttt ttan 9a. html
Zachary Teed and Jia Deng. RAFT: Recurrent All-Pairs Field Transforms for Optical Flow (Extended Abstract). Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, (IJCAI), pp. 1939-48433 International Joint Conferences on Artificial Intelligence Organization, 8
2021. URL httthhtts: http://di.ooog//.////// http//10..4993/// 0400002244933/3//.2202222 /662 Papers.
Sister Conferences Best
Jianxun Yang. Zhao Zhang, Zhuangzhi Liu, Jing Zhou, Leibo Liu, Shaojun Wei, and Shouyi Yin. FuseKNA: Fused Kernel Convolution based Accelerator for Deep Neural Networks. In 202/ IEEE International Symposium High- Performance Computer Architecture (HPCA), pp. 894-907, 2021. doi: 10.1109/HCAA11447... 21211644..220007999
Dongqing Zhang, Jiaolong Yang, Dongqiangzi Ye, and Gang Hua. LQ-Nets: Learned quantization for highly accurate and compact deep neural networks. ArXiv, abs/180 105/1807.110 10029, 2018.
Tianyu Zhang, Lei Zhu, Qian Zhao, and Kilho Shin. Neural Networks Weights Quantization: Target None-retraining Ternary (TNT). 2019 Fifth Workshop on Energy Efficient Machine Learning and Cognitive Computing NeurIPS Edition (EMC2-NeurIPS), pp. 62 65, 2019.
Ritchie Zhao, Yuwei Hu, Jordan Dotzel, Christopher De Sa, and Zhiru Zhang. Improving neural network quantization without retraining using outlier channel splitting. ArXiv, 2019.
Aojun Zhou, Anbang Yao, Yiwen Guo, Lin Xu, and Yurong Chen. Incremental Network Quantization: Towards Lossless CNNs with Low-Precision Weights, 2017. URL https: : // wtaaxiiv.oro. /rg/ abs/ 1102 03044
12

---

Published as a conference paper at ICLR 2023
Feng Zhu, Ruihao Gong, Fengwei Yu, Xianglong Liu, Yanfei Wang. Zhelong Li, Xiuqi Yang, and Jun- jie Yan. Towards unified INT8 training for convolutional neural network. CoRR, abs/1912.126600 12607, 2019. URL/ttp:: / http://axvvvv///e org.abs..rooaa/aaoaaaoo.a/. httpab////19.....1.
13

---

Published as a conference paper at ICLR 2023
APPENDICES
A QUANTIZATION-AWARE FINE-TUNING (QAT)
Fine-tuning Setups. BSFP can also be applied to quantization procedure chooses SGD as the optimizer and sets the learning rate to be 10-6 In addition, we set the momentum to 0.9 and weight decay to 10-5. The models are fine-tuned on NVIDIA's V100 GPUs using a batch size of 100. We extend Pytorch to conduct the algorithmic experiments.
fine-tuning. The fine-tuning
3hows shows the quantize-aware fine-tuning comparison between MSFP and BSFP for wide range of models. The key takeaway is:
Consistency of the benefits: BSFP consistently achieves higher accuracy using smaller model size and fewer operations. This demonstrates that the benefits of BSFP is consistent regardless of post-training quantization or quantize-awe training. Further, we also point out that BSFP outperforms MSFP in accuracy given ultra low bitwidth.
Table 3: Quantization aware fne-tuning comparison accuracy performance as well as hardware performance of ShuffleNet-v2 Ma al. 2018 MobileNet-v2 Sandler al. (2018), ResNets He et al. (2013) on ImageNet with MSFP and BSFP.
Precision (W A3
FLOPs/ FixOPs (8b)
Procision
FLOPs/ FixOPs (8b)
Method
Top-1 (%)
Model Size
Method
Top-1 (%)
Model Size
(W A)
FP
32 7 82
69.18
9.1 MB
148.8M
ShuflleNet.e?
8 3 8
68.89
2.4 MB
148.8M
7 7 7
68.30
2.1 MB
113.9M
T
64-2
69.10
2.3 MB
130. 2M
MSFP
6 6/6 6
67.56
1.9 MB
83.7M
BSFP (OURS)
6
3+-3
/
68.63
2.0 MB
83.7M
5 5
64.17
1.6 MB
58. 1M
5
4++1
65.69
1.7 MB
46.5M
4.74 4
44.99
1.3 MB
37.2M
4
3++1
62.73
1.4 MB
37.2M
FP
32
71.81
14.0 MB
314.1M
MobsloNet-22
8
71.71
3.7 MB
314.1M
7 7
71.19
3.3 MB
240.5M
7
5-+-2
71.80
3.5 MB
274.8M
MSFP
6 6
70.27
2.9 MB
176.7 76.7M
BSFP
6
3++33
71.17
3.0 MB
176. 7M
5 5
65.25
2.4 MB
122.70 7M
(OURS)
5
8+
67.38
2.6 MB
98. 2M
1
45.73
2.0 MB
78.3M
1
2++2
66.29
2.2 MB
78.5M
FP
92/332
46.8 MB
1.82G
8
69.84
12.4 MB
1.82G
RewNet-18
7
69.73
11.0 MB
1,39G
5-+21
70.01
11.6 MB
1,59G
6
69.61
9.5 MB
1.02G
3-+33
10.1 MB
1.02G
MSFP
5
68.94
8.0 MB
710.9M
BSFP (OURS)
8.7 MB
568.8M
4
64.76
6.6 MB
455.0M
2+
69.02
7.2 MB
455.0 035.0M
3/3
46.59
5.1 MB
255.9M
64.85
5.8 MB
265.9M
FP
32/32
76.13
102.2 2 MB
4.1GG
8/8
76.26
27.2
1,G
ReaNet-58 50
7
77.
24.0 MB
3.G
5-+-2]
76.30
25.4
3.60
6
76.06
20.8 MB
2.3G
6
1-42
76.17
22.2
2.3G
MSFP
5./5
75.12
17.6 MIS
BSFP (OURS)
76.00
19.0
1,60
4./4
71.57
2
75.04
15.8
1.00
3/3
53.47
MB
582.2M
2+
/
70.95
12.6
582.2V
B COMPARISON BETWEEN STANDARD QAT AND THE PROPOSED Low-CoST QAT
To amortize the overheads incurred by grid search during quantization-awe aware training (QAT), we propose quantize only the first batch for every 100 batches to amortize the quantization overheads. We refer this proposal as low-cost QAT.
To answer the question whether the proposed low-cost QAT will affect the training process, Figure 10 compares the training curve of the standard QAT and low-cost QAT on fine-tuning ShuffleNet-v2 and Nob The takeaway is that the training curve of low-cost QAT is similar the curve
14

---

Published as a conference paper at ICLR 2023
(a) ShuffleNet-12
(a) MobileNet-v2
$8.5
Quantia: once butches
70.5
Quantic: one in bunches
Quartia batchei
Quantia 100 batche
Accurso
57
68.5
56
68.5
19.5
Training Steps
Training Steps
Figure 10: Training curve comparison between standard quantivation-aware training and low-cost quantization-wware training for (a) ShuffleNet-v2 and (b) MobileNet-v2..
standard QAT, which quantizes before every batch. This demonstrates that the low-cost QAT can preserve the training curve while requiring significantly lower search overheads.
C EVALUATIONS ON OTHER TASKS
C.I OPTICAL FLOW ESTIMATION USING RAFT KITTI DATASET
Since research direction focuses computer vision, we choose extend evaluate for BSFP DNN-based optical flow estimation Teed & Deng (202I) (KITTI
Geiger (2015) ).
Table shows that BSFP consistently outperforms MSFP using weight precision, which demonstrates the benefit of BSFP consistent regardless of the tasks.
Table 4: MSFP and BSFP on RAFT Teed & Deng (021) for KITTI dataset Menze Geiger (015))
BSFP consistently improves the error given same weight precision.
Optical Flow (RAFT)
MSFP
BSFP
Precision (W)
Average end-point
0.658
0.632
0.895
0.682
2.068
0.905
D DETAILS OF THE CRITERION-OPIMAL QUANTIZATION FLOW
In order find the optimal subword mantissas and scalings, adopt iterative rounding, which contains the following steps:
1. Enumerate the two scaling factors.
2. List the quantization levels the two scaling factors can generate. The number of quantization levels is moderate. For example, there are only 64 levels if the two subwords are 2+4 bits.
3. Calculate the MSE between the 16 original weights and their nearest quantization levels. Finding the nearest quantization level in a list is viable as other quantization schemes, such as L et al. 2020), also employ
4. Keep the scaling factors that achieve the lowest MSE and go to step 1.
The pseudo code of the criterion-optima quantization flow is presented in Algorithm
E SUPPORTING BSFP ON BIT-PARALLEL ARCHITECTURE
Since some of the hardware, e.g., CPU and GPU, prefers bit-parallel PE architecture, Figure []] presents the bit-parallel BSFP PE design. The difference between serial PE architecture and parallel
15

---

Published as a conference paper at ICLR 2023
Algorithm I Criterion-optimal quantization flow
Input: Full-precision weight Wf
Output: Criterion-optimal scalings SCopt, Criterion-optimal subword mantissas maRopt
Require: Subword configurations configgut Scaling configurations f'isse
Criterion + max
SC +- EEnum(con foruumcconffgge f
Enumerate all scaling setups to form search space SC > For every 8Ceur of the search space SC Find all quantization levels using SCzur b Quantize W to its nearest levels list b Compute the MSE criterion > Update the scale and subword mantissas
for 8Ceur E SC do
listquant BuildQlist (SCur,, configsubt
W Wquant &- Quant W [p: listquant)
Criterioneer + M MSS(WW fps W quant)
if Crilerioneur < Criterion then.
SCopt SCcur
maRopt < maneur
Criterion &- Criterioneur
end if end for
PE architecture is that the scaling units need to be unrolled, which incurs some overheads. BSFP's benefits of smaller model size and lower arithmetic complexity remain unchanged.
Further, we can achieve uneven subword configurations by fixing the upper multipliers and adjusting only the weight precision of lower multipliers (red weights).
Sign
Exponent
Mantissa
Scaled Parallel Processing Engine (SPPE)
Amsdp
W
bslp
vullwordd
subword
16
Fired-Porid Poidd-Point
byfoot36
16x
Scalt, expu
exp
Cycle 1
Scelings
Multipliers Adder
Scoling
Shifting
Accumulator
Figure 11: Proposed scaled parallel processing engine.
BSFP can be efficiently deployed in newly- wwwly-manufactured devices, including customized Al accelerator IPs, which represent significant amount of AI products including smartphones and smart appliances. For example, Nvidia NVDLA NVIDIA (2023), Qualcomm Hexagon Tensor Accelerator Codrescul (0013). and MIT Eyeriss Chen et al. 2017h. Such IPs are embedded in emerging Al chips for smartphones, appliances, robots, etc. Since customized, embedded Al accelerator IPs are not obligated support standard data types, the proposed BSFP type. which improves weight quantization quality (including post-training quantization, which is even difficult and desirable), will stand out candidate chip designers to consider and adopt in their new design.
As for CPUs, although they are not our only focus, letting newly-manufactured CPUs support BSFP not difficult, and the computation will efficient. Modern CPUs, including RISC-V, ARM, and Synopsys ARC, allow custom ISA extensions Imperas (2019): ; Arm (2000) Synopsys 2020).
Designers can provide an add-on digital circuit that natively and efficiently supports BSFP using low- bit, fixed-poine adders and multipliers. Vendors already offered the required design/simulation flow, CPU generator, and toolchain generator to facilitate designers to employ the extended instructions, e.g., vector product instructions between BSFP weights and MSFP activations.
16

---

Published as a conference paper at ICLR 2023
F DETAILS OF THE HARDWARE COMPARISON
Processing Engine Details: Table 2 compares 16-wide PEs in different number formats, i.c. BF16, INT4, INT8, Power-of-Two, MSFP, and BSFP, following similar setup in FAST Qian Zhang et al. 2022). The architectural details for these PEs are summarized below:
The BF16 (brain float) PE adopts bit- stt-paralle architecture, which computes 16-wide BF16 multiplications and reduces them single partial sum. The accumulator is in BF16 format.
The INT4 and INT8 PEs also adopt bit-parallel architectures, which perform 16-wide MAC and generate 12b and 20b partial sum, respectively. The accumulator then accumulates using INT32 format.
The Power-of-Two PE shifts 16 inputs based on the weight values and reduces 16 products to a single partial sum. The accumulator is in INT32 format.
The serial MSFP PiiQian Zhang et al. 2022) computes 16 2b-to-2b multiplications per cycle and shifts them to accumulate for the correct partial sum. We adopt the BF16 accumulator.
The proposed S IP supports BSFP, which computes 16-wide 2b or lb multiplications per cycle and scale-and-shin them accumulate for correct partial sum. The accumulator is in BF16 format.
Additional Notes:
The adder tree output precisions every PEs are optimized to their minimum.
ALL of the PEs obtain similar slack profiles.
Performance Analysis Setup: We adopt iso-area performance setup fairly compare different number systems, which is widely adopted by top architecture papers Sharify et al. (2019); Yang al. (2021). In other words, the size of BF16 PE is 26x larger than that of the BSFP PE, we allowed to allocate 26x more PEs for BSFP. Besides PE parallelism, we also consider the multi-cycle computation of serial architecture to reasonably estimate the performance.
We implement all of the above PEs using Verilog and synthesis them using Synopsys Design Compiler (topographical mode) on the TSMC 40 nm node. Specifically, the power of each module is analyzed with average switching activity. We then utilize the power profiles to estimate throughput per Watt.
Power Analysis Setup:
G PER-LAYER SKEWNESS ANALYSIS
Figure 12 analyzes the absolute skewness for weights each layer without slicing weights into vectors. We observe that most layers' skewness is low (i.e., 50.5), which matches the summary of prior works that weight values follow bell-shape and non-skewed distribution.
Through side-by-side comparison between Figure 12 and Figure 13 (same as Figure 7 We copy here for easier comparison), we clearly observe that vectorization is one source of more considerable skewness Figure
(a) ShuffleNet-v2
ResNet-18
Absolutte Shewness
Absolute
Low
Low skewness
Figure 12: Per-layer absolute skewness of (a) ShuffleNetv2 and (b) ResNet- 18. Most of the layers obtain low skewness, which matches the analysis reported by prior works.
17

---

Published as a conference paper at ICLR 2023
(a) ShuffleNet.v2
ResNet-18
Proportion 50
Proportion (16)
80
L0 LS 235
005 BLD-CK
Layer
Figure 13: Per-vector absolute Pearson's skewness coefficient (SK) of (a) ShuffleNet-v2 and (b) ResNet-18. In general, very highly skewed means SK 1.5, highly skewed means SK > 1.0, and moderately skewed means 1.0 SK 0.5.
H SEARCHING FOR BSFP CONFIGURATIONS
To obtain the scaling factor configurations (1-4-3, 1-3-3) and the exponent biases (-8, -3), the following steps to empirically set them. This method acceptable as neural networks naturally posses many hyper parameters.
1. Begin with setting scaling factors BF16 (1-7-8), sufficiently precise format.
2. Given the scaling factors, find the narrowest bitwidths of subwords that causes accuracy drop less than a first budget. For example, 5+2 bits are selected.
3. Given the subword settings, reduce the bitwidths of scaling factors and sweep the the biases to point that the overall accuracy drop is less than second budget. This step lead us to (1-4-3, 1-3-3) and biases (-3,-8).
I ADDITIONAL COMPARISON OF BSFP AND MSFP (BFP)
Compared to Vanilla BFP (MSFP), BSFP lowers the flexibility converting from full- full-precision weights, but BSFP gains richer flexibility to approximate and convert back to full-precision weights. In view of the different characteristics of weights and activations, should be reasonable option to adopt BSFP for weights and MSFP for activations.
BSFP focuses on tine-tuning-fee free, post-training quantization (as well as inference), which are our heard realistic demands from the industry. Though BSFP-based training is beyond the scope, here provide a proposal.
First, each BSFP vector is weighted sum of two Vanilla BFP vectors. Therefore, by letting the scaling factors be trainable parameters, is possible to leverage a Vanilla BFP framework to train BSFP networks the first place without resorting to decomposition- quantization.
Second, one can employ Vanilla BFP training for the majority of training epochs in the beginning and switch BSFP training only the last few epochs.
18


Reviews:
[{"review_id": "nkypRWcONf3", "reviewer": "Reviewer_guHP", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed a new number representation format for efficient post-training quantization of deep neural networks. The proposed format consists of a vector of words each of which is linearly decomposed by two subwords with reduced precision floating point scales. The authors claim that the proposed format is more suitable for representing asymmetric weight distribution. The authors evaluate the proposed method with several ImageNet classification models, and they also presented the hardware implementation for it.", "summary_of_the_review": "This paper proposed an interesting reduced-precision data representation, but the proposed method incurs hardware overhead. The authors tried to justify their choices with their hardware implementation, but the corresponding discussions are not thoroughly evaluating all the hardware options. Unfortunately, there is little theoretical analysis that justifies the algorithmic advantages of the proposed method, which the readers of ICLR would most appreciate. In this regard, it seems to the reviewer that this paper might fit better for the hardware architecture conferences such as DAC and ISCA.", "strength_and_weaknesses": "(Strengths)\n- A new data format (potentially) more representative of asymmetric distributions.\n\n- Evaluation of quantization accuracy on various ImageNet classification tasks.\n\n(Weaknesses) \n- The proposed methods incur more arithmetic operations (e.g., multiplication per subword and addition of psums from the sub-words). The authors claim that these additional computations can be efficiently handled by custom hardware. However, it is very difficult to properly evaluate their claims on hardware benefits without enough evidence and fair comparisons. \n\n- The authors claim several algorithmic advantages of the proposed format, such as adaptation to skewed/non-uniform distributions. However, these claims are not theoretically or analytically supported in the paper; E.g., there is no guarantee that asymmetric number representations of two's complement format are helpful for capturing asymmetric weight distributions. Note that the two's complement is significantly skewed to the negative regions. This fixed skew might be harmful if the weights are positively skewed or not skewed much. There is no analysis in the paper that provides theoretical reasonings about it. \n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "WMNqhybd4VD", "reviewer": "Reviewer_A7Vh", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors Block and Subword-Scaling Floating-Point (BSFP), a variant of block-floating point (BFP). BFP represents a vector of real numbers using a single shared exponent and a per-element signed mantissa. BSFP is a variant with the following key differences:\n 1. BSFP represents a vector of real numbers as the sum of *multiple* such BFP vectors. The paper only experiments with 2 vectors.\n 2. BSFP uses shared low-bit float scaling factors (which can be an arbitrary number) as opposed to a shared exponent (which must be a power of 2)\n\nBSFP is designed for post-training inference. BSFP is only used for the weights, while activations are left in BFP. This is because the weights are known ahead of time, so the authors can use a brute-force exhaustive algorithm to finding the best scaling factors. The dot products between weight and activation will be between BSFP and BFP, which is actually a positive as such a dot product is efficient and does not require floating-point multiplies:\n 1. Individual mantissas can be multiplied in fixed-point\n 2. The float BSFP scale factor(s) and the power-of-two BFP scale factor can be multiplied by an exponent summation.\n 3. The scale factor product can be multiplied onto the dot product sum with a fixed-point mantissa multiply.\n\nThe authors propose to use a custom bit-serial architecture to support BSFP. There's a architectural diagram given but not much detail in the text. The architecture is based on a bit-serial processing engine from another paper. Using bit-serial means the subwords in BSFP can be different bitwidths.\n\nThe authors compare against MSFP, Microsoft's BFP implementation which seems like a fairly strong baseline. This is a great improvement from many quantization papers which compare only against FP32. Experiments are done over a suite of CNNs and VIT on ImageNet. Convincing results show that BSFP achieves better accuracy-per-model-size compared to MSFP. However, the throughput comparison is done with both MSFP and BSFP on the custom bit-serial architecture - this feels weak to me since bit-serial is not widely used in practice and the MSFP paper uses a conventional accelerator. Still the experiment section as a whole is quite convincing.", "summary_of_the_review": "The paper proposes BSFP, a novel data format similar to block-floating point (BFP). BSFP uses multiple BFP vectors to represent a single vector of real numbers. BSFP uses floating point scale factors but can efficient multiple with a BFP vector with only fixed-point operations. BSFP has a complex quantization procedure and is only suitable for pre-quantized weights during inference. BSFP was designed for an unconventional bit-serial architecture and may not be efficient on commercial devices. Nevertheless, BSFP is quite novel and achieves improved accuracy per model size when benchmarked against a strong BFP baseline. This makes it an interesting paper for the quantization community.\n\nThe biggest change I advise is to focus the paper on the quantization algorithm and data format and away from the hardware architecture.", "strength_and_weaknesses": "Strength:\n - BSFP is a novel data type which combines multi-level quantization with BFP.\n - BSFP is clearly algorithmically better than BFP since it uses float scale factors. Despite this BSFP does not require float multiplies during computations when multiplied with BFP.\n - Paper is well written. The results surrounding accuracy and model size is convincing. The comparison is made to a strong quantization baseline in MSFP.\n\nWeaknesses:\n - The process of converting floating-point weights to BSFP is not fully clear. The exhaustive algorithm to find the scale factors makes sense, but how to compute both sub-word mantissas? The paper mentions \"iterative rounding\" but this needs to be made clearer. If the two mantissas are of uneven widths (as is the case in many of the experiments) which mantissa is computed first?\n - Most of the BSFP configurations tested in Table 1 use uneven subword mantissa widths. These configurations may not be efficient outside of bit-serial architectures. This is unlike BFP which is relatively easy to implement even on CPU and GPU.\n - BSFP trades away flexibility for efficiency. Vanilla BFP can be used easily for training and fine-tuning, but BSFP due to the complex quantization procedure can only be used for inference.\n - The throughput comparison between BSFP and MSFP was made on a custom bit-serial processor, which seems like it was designed specifically for BSFP. For example, MSFP uses a wider shared exponent which incurs a greater shift overhead in their design. But I think this is specific to bit-serial and would not be an issue for a conventional adder tree. This makes the throughput comparison somewhat unconvincing. I honestly think the authors should scale back the amount of the paper devoted to the hardware throughput comparison. ICLR is not the venue for this. Focus the paper on the model size and accuracy results instead.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Mlbz2p9sy-", "reviewer": "Reviewer_KfxR", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose a form of low-precision additive quantization in order to represent weights/parameters in NN inference as a sum of a small number (here, 2) of floating point values scaled by very low precision signed integers that represent each individual scalar value in the weights. This scheme goes by the name of BSFP and provides what appears to be an effective means of quantization of weights. \n\nIn order to reduce computation overheads in hardware with this additive quantization representation, it is also necessary to represent activations in such a manner so as to reduce the number of full floating-point additions and multiplications. They chose the pre-existing MSFP representation (a chunk of signed integers plus a shared scaling exponent) in order to represent activations. Together, the two representations work with each other to drastically reduce the number of floating-point type operations (significand adders / shifters for multiplication, etc) which provides a direct advantage to hardware PPA. This basic building block to represent an inner product between BSFP and MSFP (called the S2PE) is combined in a systolic array for generalized inner product evaluation.\n\nThe work requires some number of parameters to be fixed for HW purposes (e.g., quantization scheme and loss function, precision of the two BSFP components, chunk size, etc), but the scheme appears to provide even more significant compression of the weights over MSFP and improved accuracy on a variety of networks (ShuffleNet-v2, MobileNet-v2, VIT, etc).\n", "summary_of_the_review": "Overall, I am borderline accept to full accept, this does add something new to the literature (first time I've seen additive quantization used in this space for one) and as an existence proof with concrete results it is super interesting, but the BSFP technique itself (including the discussion about using signed integers instead of wasted +/-0 representation space with sign/significand for MSFP) do have antecedents. A little more evidence on broader applicability I think would be useful in order for HW people to consider it in their inference designs, but I think that will likely have to wait for others (or the authors themselves) to expand the work in subsequent papers.\n", "strength_and_weaknesses": "The main strength is in showing the empirical applicability of the BSFP technique to a variety of computer vision NNs, and while I am not super up on the universe of inference quantization techniques that have been proposed in recent years, I feel like I am fairly up on the computer arithmetic in silicon design space and this quantization method does provide significant advantages in accelerating computation.\n\nThe in-silicon hardware advantages of this design seem quite obvious to me and I don't doubt the PPA advantages at all, but a little more detail on the comparison baselines would be useful. In \"Hardware Evaluations\", it seems that synthesis was run on a 16-wide S2PE (guessing this is what is mean by \"parallelism to be 16 for all number systems), but what exactly was the synthesized design for a 16-wide bf16, int4, int8 design? Does the 16-wide int8 inner product tree produce a 32-bit output or a 8 * 2 + log2(16) = 20 bit output? Where did the bf16 blocks come from (DesignWare or elsewhere)? Any information on the slack in the various designs were produced? How was power estimated, using average switching activity or by using realistic test vector SAIFs from real networks?\n\nTable 1 is a little confusing to look at. The bolded numbers are for all top-1 results of BSFP, though some of the bolded numbers are under the neighboring MSFP numbers. However I would agree that the two techniques are not directly comparable (as BSFP does offer big HW efficiencies). The Pareto frontier charts are simply easier to understand. Regarding \"FLOPs / FixOPs (8b)\", how is this computed for BSFP as many of the adders / multipliers in the S2PE are sub-8b? ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "4dmDeP3wnB", "reviewer": "Reviewer_h3B6", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces a new floating point format for non-uniform quantization of weights (BSFP). The main idea is to split the input vectors (words) into subwords (usually 2) to which separate scaling factors are assigned. All possible combinations of scaling factors are evaluated and one is chosen according to a given criterion (e.g., MSE). The focus of the paper is on post training quantization (PTQ) but quantization aware training (QAT) results are also included. Results are reported on various network for image classification. A hardware implementation for BSFP is proposed and simulated. ", "summary_of_the_review": "The paper presents a novel format for weight quantization and related HW implementation. The proposed format generally outperforms direct competitors in terms of accuracy at a given model size and/or FLOPs, as tested over a range of models for image classification. Some empirical choices may benefit for clearer motivation and/or explanation. ", "strength_and_weaknesses": "Strengths:\n- the idea is conceptually simple and well-described: implementing non-uniform weight coverage by combining two (or potentially more) scales for weight quantization\n- comparison against MSFP competing format is favorable, showing improved accuracy at given model size across a range of networks for image classification\n- practical implementation is supported by a proposed HW design and process flow for the processing engine / architecture\n\nWeaknesses and other comments:\n- there are several empirical choices across the paper: scaling factor format (1-4-3, 1-3-3), biases (-3,-8), MSE. For the most part, it is not clear what considerations drove these selection, what other options were investigated, and what was their impact. As far as MSE is concerned, alternative criterions are shown in Fig. 5(c) but the authors don't even attempt to explain why MSE may perform better than L1 or cosine similarity\n- this technique explores all possible scaling factors combinatorially in search of the optimal one. This seems feasible for a small network but less practical for larger ones. Can the authors mention the search time required for the other networks they investigated? In addition, the number of possibilities is upper bounded to $2^7 * 2^7$. Is this value compounded by the number of subword configurations to explore (therefore larger for larger bit widths)?\n- no details on HW simulation makes any sort of verification of throughput and energy efficiency impossible. These may warrant their own section in the appendix\n- accuracy improvement at given model size or FLOPs can be modest in some cases (Table 1), although it is general more apparent when quantization is more aggressive\n- section 5.4 presents skewness analysis to support the claim that distributions of grouped weights can be skewed. I assume Fig.7 was done for vector length of 16. Is the skewness arising from the grouping of weights into blocks or from the actual skewness of the weight distribution of each layer? What would be the proportion of skewed vectors taken from a normally distribution? May want to present this as \"baseline\" skewness\n\nOther comments:\n- clarification on Table 1: when reporting weight bit precision for BSFP (for example: 7 [5 + 2]) does this mean the input vectors were split into 2 subwords in 2s complements format and respective size 5 and 2? Are the scaling factors fixed to 1-4-3  and 1-3-3 format?\n- regarding QAT in the appendix, how does the process work? Are the optimal scaling factors recomputed at every iterations (or every 10, 100 iterations in the case of low cost QAT)? This seems to be adding an enormous overhead to the training\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster