FLIQS: One-Shot Mixed-Precision Floating-Point and Integer Quantization Search

Authors: jordan  dotzel
gang  wu
andrew  li
muhammad  umar
yun  ni
mohamed s abdelfattah
zhiru  zhang
liqun  cheng
martin  dixon
norman p jouppi
quoc v le
sheng  li
Venue: automl_cc_AutoML_2024_Conference
Published: 2024-02-27 01:23:56

Abstract: Quantization has become a mainstream compression technique for reducing model size, computational requirements, and energy consumption for modern deep neural networks (DNNs). 
With improved numerical support in recent hardware, including multiple variants of integer and floating point, mixed-precision quantization has become necessary to achieve high-quality results with low model cost. 
Prior mixed-precision methods have performed either a post-training quantization search, which compromises on accuracy, or a differentiable quantization search, which leads to high memory usage from branching.
Therefore, we propose the first one-shot mixed-precision quantization search that eliminates the need for retraining in both integer and low-precision floating point models.
We evaluate our search (FLIQS) on multiple convolutional and vision transformer networks to discover Pareto-optimal models.
Our approach improves upon uniform precision, manual mixed-precision, and recent integer quantization search methods. 
With integer models, we increase the accuracy of ResNet-18 on ImageNet by 1.31\% points and ResNet-50 by 0.90\% points with equivalent model cost over previous methods.
Additionally, for the first time, we explore a novel mixed-precision floating-point search and improve MobileNetV2 by up to 0.98\% points compared to prior state-of-the-art FP8 models.
Finally, we extend FLIQS to simultaneously search a joint quantization and neural architecture space and improve the ImageNet accuracy by 2.69\% points with similar model cost on a MobileNetV2 search space.

Main paper: 
 introduction: 
 In recent years, deep neural networks (DNNs) have achieved state-of-the-art results on a wide range of tasks including image classification, speech recognition, image and speech generation, and recommendation systems. Each model iteration typically enhances quality but also tends to increase computation, memory usage, and power consumption. These increases limit DNN adoption in resource-constrained edge devices, worsen their latency across platforms, and expand their carbon footprint, especially within cloud systems. DNN quantization to low-precision formats has become the standard method for reducing model storage size, memory bandwidth, and complexity of MAC operations [1, 2] . These formats include both integer and low-precision floating-point, which has recently gained attention as a flexible alternative to integer formats.At the same time, DNN accelerators have become more diverse and now support a wide range of numerical formats. For example, the Google TPUv3 supports FP32, BF16, FP16, and INT8 [3] , while the latest NVIDIA Hopper architecture supports FP32, BF16, FP8, and INT8 [4] . Furthermore, reprogrammable systems such as FPGA devices allow arbitrary precision arithmetic such as INT5, FP11, FP9, or FP8 for more granular accuracy-performance trade-offs [5] . While these devices enable mixed-precision quantization, where layers take on different formats within the same model, it is challenging to optimally assign per-layer formats since layers exhibit different quantization characteristics. In simple cases, this assignment can be performed manually, yet with the explosion Figure 1 : FLIQS -The explosion of model architectures, numerical support, and deployment platforms requires automated methods for searching model configurations to utilize platform-specific numerical formats. We establish FLIQS as the first one-shot quantization and neural architecture search framework for searching for integer and floating point formats.of DNN architectures and accelerator designs, automated methods are more reliable, scalable, and reproducible for achieving high accuracy and performance.In this paper, we introduce FLoating-Point and Integer Quantization Search (FLIQS) to automate mixed-precision floating-point and integer quantization and automatically assign per-layer formats. In addition, FLIQS can jointly optimize for quantization formats and neural architecture to intelligently allocate compute across the kernel, channel, and bitwidth dimensions. FLIQS is a one-shot search based on reinforcement learning (RL) and unlike expensive multi-trial searches, it avoids training separate models for each configuration, leading to overall reduced search overhead. Furthermore, as the search takes place during training, FLIQS can achieve higher accuracies than post-training quantization (PTQ) searches. Coupled with additional entropy regularization, the final model can be deployed without the need for further retraining or fine-tuning. As shown in Figure 1 (a), FLIQS accelerates the process of adapting legacy models to new hardware, co-designing models and accelerators, and finding Pareto-optimal models on current hardware systems. We summarize our contributions as follows:1. Introduce the first one-shot quantization search without retraining through the addition of a new cosine entropy regularization schedule;2. Demonstrate state-of-the-art results for integer and low-precision floating-point quantization search across a range of convolutional and transformer networks;3. Perform the largest comparison of integer and floating-point mixed-precision networks;4. Conduct the first study of quantization and neural architecture search on low-precision floatingpoint networks and establish recommendations for allocating compute across bitwidth and neural architectural dimensions. 
related work: 
 Low-Precision Floating Point: Low-precision floating point is being discussed as the next generation format for DNN training and inference. [6] . Companies, including AMD, Intel, NVIDIA, and Qualcomm, have recently agreed to adopt 8-bit floating-point (FP8) in future deep learning systems. Within these formats, recent studies generally focus on two variants: E5M2 and E4M3, where E represents the number of exponent bits and M is the number of mantissa bits. For example, HFP8 suggests using E4M3 for the forward pass and E5M2 for backpropagation [7] . Building upon these uniform precision works [7, 8, 9, 10, 11] , FLIQS proposes an automated approach for finding Quantization Search: Prior work has explored mixed-precision integer quantization searches, as shown in Figure 1 (b). For instance, HAQ [12] and ReLeQ [13] both perform PTQ quantization searches that utilize RL to allocate bitwidths based on the model accuracy and cost estimates. In addition, the HAWQ series of works further develops these PTQ searches, using the Hessian spectrum to determine layer sensitivities and constrained ILP formulations to find optimal bitwidth configurations [14, 15, 16] . However, being PTQ-based, these methods cannot take advantage of the higher accuracy and more accurate feedback provided by quantization-aware training (QAT) during the search.Other efforts perform quantization search during training, often using neural architecture search (NAS) with super-networks or differentiable NAS [17, 18, 19, 13, 20] . For instance, MPQ uses an adaptive one-shot method that trains models using multiple bitwidths and automatically freezes the bitwidths of specific layers during training to improve the model convergence across bitwidths [21] . In addition, EDMIPS creates branches for each bitwidth, forms a linear combination of them, and then alternates training the layer weights and the branch weights [22] . These differentiable searches often have simpler formulations since the layer and branch weights are unified and trained together with gradient descent. However, because they replicate the weights and activations, they incur higher memory and computational costs compared to RL-based methods. In addition, both PTQ and QAT prior works require additional retraining steps on the model after the search, while FLIQS directly serves the final model without fine-tuning.Quantization Neural Architecture Search (QNAS): In addition, prior work has explored joint search spaces with quantization formats and neural architecture [23, 24, 25, 26, 27] . For example, APQ uses knowledge distillation from a full-precision accuracy predictor to optimize neural architecture, quantization formats, and pruning policies [25] . FLIQS expands on this line of work by jointly searching quantization formats and neural architecture and highlights trends for allocating compute across this joint search space for high accuracy and performance. 
fliqs framework: 
 As a one-shot method, FLIQS employs a controller to sample per-layer formats and model architectures during training. This method allows the search and model to adapt to each other yet it comes with certain challenges. First, the search may interfere with the original model training process, since modifying the architecture shifts the weight and activation distributions during training. In addition, one-shot search needs to evaluate the quality signal of different architectures on different batches of training data to avoid lengthening the training process. This introduces noise into the reward signal since different batches may have significantly different quality. Also, the controller and policy model must be efficient enough to be embedded within the training graph to not significantly increase the training time. This section addresses these challenges, while focusing on the search space involving per-layer formats and channel widths. As shown in Figure 2 , the model first trains without search, and the architecture is sampled uniformly at random to avoid overfitting to a single option. It uses standard fake quantization and employs a two-phase approach that delays activation quantization to improve stability (Appendix A.2). Next, at each training step, the controller proposes a new architecture and applies it to the model. The model then performs a standard forward and backward pass on the training data to produce the model gradients and a forward pass on the validation data to produce a quality signal for the controller. This quality signal is combined with the model cost in Figure 2 (b) to produce a reward and reward advantage, which the controller then uses to update its policy. After the search and training finish, the model is directly used for inference without additional fine-tuning or retraining.E2 M5 E3 M3 E2 M4 E2 M3 E2 M3 E2 M5 E2 M5 90 GBOPs 6B 5B 4B 4B 5B 4B 4B 7B 320 GBOPs ResNet18 Attention MLP 7B 6B 8B 8B 6B 7B 7B BF 810 GBOPs K Q V … … E2 M3 E2 M1 E2 M3 E2 M2 E2 M2 E2 M4 E2 M3 … 60 GBOPs DeiT-B16 … E3 M2 E3 M3 E2 M5 E3 M1 E2 M1 E2 M5Cost and Reward Function: FLIQS uses the quadratic cost model, bit operations (BOPs), as described in Equation 1where 𝑏 (𝛼) is the total bitwidth of the current layer architecture 𝛼 and 𝑀𝐴𝐶 𝑙 (𝛼) represents the number of multiply-accumulates (MACs) in layer 𝑙. Quadratic cost models, which predict power and area, are particularly useful in model-accelerator co-design where multipliers dominate resources and scale quadratically in power and area [28] .EQUATIONThis model cost is combined with the quality signal, 𝑄 (𝛼), in the absolute reward function shown in Equation 1 [29] . This quality signal is model and application dependent but in the simple case is the validation accuracy. The absolute reward function includes a cost target 𝐶 𝑇 that provides the user control over the accuracy-performance trade off. More restrictive targets tend to result in less compute-intensive models (as shown in Figure 3 ), which often have lower accuracy. This resultant cost term is combined with the model quality using the cost scalar 𝛾, which balances the importance of performance and quality.RL Controller: The RL controller is in charge of choosing the model architecture at each step. It learns a policy 𝜋 𝑙 (𝛼) for each layer 𝑙 that represents a probability distribution over each architecture 𝛼. At each training step, the controller samples and applies a new layer architecture 𝛼 𝑙 ∼ 𝜋 𝑙 (𝛼). The channel widths are efficiently searched by applying channel masks, which dynamically zero out channels and reuse the underlying weights during training. This policy 𝜋 𝑙 (𝛼) is parameterized by 𝜃 𝑙,𝛼 , where 𝜃 𝑙,𝛼 represents the logit for the 𝛼 𝑡ℎ decision in the 𝑙 𝑡ℎ layer. These logits are then passed through a softmax layer to produce the policy probability distribution. After sampling and assigning the model architecture, 𝜶 , the reward 𝑟 (𝜶 ) is calculated according to Equation 1. However, since the reward depends on the quality signal, which increases throughout training, the difference between the running average of previous rewards, r (𝜶 ), and the current reward is used instead: 𝑟 Δ (𝜶 ) = r (𝜶 ) -𝑟 (𝜶 ). Then, the REINFORCE algorithm [30] is used to update the policy 𝜋 𝑙 (𝛼) by performing gradient descent on the policy loss, L 𝜃 :EQUATION)where 𝜂 is the RL learning rate. This procedure is chosen due to its low complexity, and it helps address the performance concerns with one-shot searches (analysis shown in Appendix A.6). Other reinforcement learning methods, such as PPO, and more sophisticated policy models, such as multi-layer perceptron models, offered no quality improvements while being more costly.Format Search Space: For pure quantization search, this work evaluates FLIQS on two search spaces: FLIQS-S and FLIQS-L. FLIQS-S includes the standard power-of-two formats, while FLIQS-L includes a larger set of formats between four and eight bits. For floating point, FLIQS-L includes 16 formats, which to our knowledge is the largest quantization search performed. Full details of the quantization search spaces can be found in Appendix A.5.Switchable Clipping: FLIQS also introduces a switchable clipping threshold that changes based on the current format. This is necessary since smaller bitwidths require more aggressive clipping, and vice versa, as shown in Figure 4 (a). These clipping thresholds can either be pre-computed with synthetic data, or computed during the first phase of the search with real data. In general, pre-computing the thresholds leads to high-quality results with less complexity, and it is used for the experimental sections below. 
fliqs analysis: 
 Switching Error: The primary challenge for FLIQS is minimizing the effect of the search on the model training. Within a pure quantization search, this effect can be formalized by introducing the switching error. Consider the standard symmetric integer quantizer, 𝑄 (𝑥; 𝑠) with the scale factor 𝑠 = (2 𝑘 -1 -1)/𝜎 𝑇 , where 𝜎 𝑇 is the clipping threshold. This gives the absolute quantization error Δ(𝑥; 𝑠), defined as:𝑄 (𝑥; 𝑠) = ⌊𝑥 • 𝑠⌉ /𝑠, Δ(𝑥; 𝑠) = |𝑄 (𝑥; 𝑠) -𝑥 | (4)For a fixed 𝜎 𝑇 , 𝑄 (𝑥; 𝑠) and Δ(𝑥; 𝑠) can instead be parameterized solely by the bitwidth 𝑘. When this varies during the search, it produces a switching error:EQUATIONAs illustrated in Figure 4 (a), this switching error for standard search spaces, such as integer FLIQS-S, can be relatively large (setup details listed in Appendix A.8).Convergence: This switching error can be viewed as an additional source of noise for the model optimizer, typically SGD or Adam [31] . Intuitively, the expected switching error should be proportional to the total policy entropy 𝐻 𝑀 of the model 𝑀:EQUATIONThat is, as the policy decreases entropy over time by settling on specific formats, the expected switching error decreases and converges to zero as the entropy tends toward negative infinity. This can be seen explicitly by modeling 𝜋 𝑙 (𝑘) ∼ 𝑁 (𝑘; 𝜇, 𝜎) as a Gaussian distribution, which has an entropy 𝐻 = 1 2 log(2𝜋𝑒𝜎 2 ). Under these assumptions, lim 𝐻 →-∞ ⇒ lim 𝜎→0 ⇒ lim 𝑘 1 →𝑘 2 and thus: lim𝐻 →-∞ 𝐸 [Δ 𝑆 (𝑥; 𝑘 1 , 𝑘 2 )] = 𝐸 [ lim 𝑘 1 →𝑘 2 Δ 𝑆 (𝑥; 𝑘 1 , 𝑘 2 )] = 𝐸 [Δ 𝑆 (𝑥; 𝑘 2 , 𝑘 2 )] = 0 (7)since Δ 𝑆 (𝑥; 𝑘, 𝑘) = 0. Therefore, as the model entropy decreases, the search no longer interferes with the model training, and this interference can be formulated in terms of additional optimization noise. The noise ball around the optimum is proportional to the entropy, and therefore convergence requires carefully controlling the entropy.Entropy Regularization: FLIQS introduces entropy regularization to reduce the entropy toward the end of the search and enable searches without a final retraining. This addresses the key challenge of one-shot quantization search by diminishing the effects of the search on the model training. The entropy regularization adds a new loss term to the policy loss L 𝜃 , balanced by a factor 𝛽 𝐻 . In addition, FLIQS introduces a cosine entropy regularization schedule in Equation 9, where 𝑠 ∈ [0, 1] represents the current training progress and 𝛽 𝑒𝑛𝑑 𝐻 = 0.5. Figure 4 (b) demonstrates the characteristics of this schedule and the tradeoffs in choosing 𝛽 𝐻 . It can achieve high quality results through high exploration at the beginning of the search (high 𝐻 𝑀 ) and final stability for the quantization-aware training at the end. Appendix A.11 demonstrates that retraining after the search adds no benefit with entropy regularization.EQUATION)EQUATION 
quantization search: 
 We begin by evaluating FLIQS on pure quantization search spaces, since this allows comparisons to the most previous work. All models were trained from scratch with cloud-based TPUv3 cluster, and all training and search hyper-parameters are listed in Appendix A. 3 .Pareto Curves: Figure 5 shows the Pareto curves for uniform precision and FLIQS models. It demonstrates that FLIQS outperforms uniform precision methods across ImageNet models, often with large margins. The FLIQS-L searched models lead to the highest accuracy overall, yet this search space requires support for arbitrary precision in hardware, e.g. within FPGA platforms. In addition, when comparing models together, the FLIQS-L MobileNetV2 outperforms all others models across floating-point and integer formats, with FLIQS-L EfficientNet following closely behind. Finally, the integer and floating-point models are plotted together and show that in nearly every case, floating-point outperforms integer.To achieve these results, FLIQS makes different decisions for each model guided by the reward signal. For the ResNet models, it assigns most layers to low-precision, except for the first and last. It further increases the precision of the pointwise convolutions in the downsampling skip branches (the top 8B convolutions in Figure 3 ). In contrast, for EfficientNet and MobileNetV2 the pointwise convolutions are typically in lower precision while the depthwise convolutions are in higher precision. Lastly, the vision transformer model, DeiT, shows similar behavior to the other ResNet Performance -The ResNet18 area estimates demonstrate a small impact from the additional layers in higher precision with FLIQS-L and additionally show the correlation between GBOPs and area. The precision column for each of the three layers in the ResNet-18 downsampling block: 3×3, 3×3, 1×1. The ResNet50 results demonstrate that the integer FLIQS-S mixed-precision model does not add significant overhead over HAWQ-V3. FPGA results were gathered on the Xilinx UltraScale+ FPGA platform, where look-up tables (LUTs) are the primary resource. For example, it outperforms the recent mixed-precision QS method HAWQ-V3 [16] across multiple model cost targets. For ResNet-50, FLIQS improves the Top-1 accuracy by 0.61% while using only 51% of its GBOPs. In addition, FLIQS-L outperforms many recent works on FP8 model inference. For example, against MPFP [8] on ResNet18, FLIQS finds a variant with 1.93% higher accuracy with a third of the model cost by allocating more bits to the downsampling convolutions and first convolutions in the network.These results demonstrate that the searched models consistently outperform their uniform precision baselines. Moreover, this section to our knowledge shows the first large-scale comparison of floating-point and integer mixed-precision models and shows that floating-point models outperform their integer counterparts for the same total bitwidth. Joint integer and floating-point searches were attempted; however, since floating-point dominates integer formats at the same total bitwidths, the outputs of these searches were the same as the pure floating-point searches.Performance: To evaluate the performance of the searched models, we use an infrastructure developed by the authors of HAWQV3 [16] that extends the TVM [37] compiler to support INT4 inference. Table 3 shows that on Turing GPUs, the FLIQS-S model improves accuracy significantly with only 1% lower inference speed compared to the INT4 model. In addition, Table 2 shows that LUTs scale quadratically with the precision bitwidth, and since LUTs act as a proxy for area, this verifies the usefulness of the BOPs cost model. This table also confirms the overhead from these searched models is relatively small compared to the accuracy improvements shown in Table 1 . 
quantization neural architecture search: 
 FLIQS can efficiently traverse large quantization search spaces and achieve Pareto-optimal combinations of accuracy and model cost within fixed model architectures. Yet, further improvements can come from combining the quantization search of FLIQS with neural architecture search, which is referred to as FLIQNAS in this section.Figure 6 evaluates this method on a MobileNetV2 search space, which incorporates tunable filter widths on inverted bottleneck projection layers and adjustable kernel sizes on central depthwise layers. Altogether, there are 230 tunable values leading to a search space of over 10 100 configurations for FLIQNAS-S. This search space is significantly larger than that of the original MobileNetV2 FLIQS-S with 53 options and approximately 10 25 configurations. This figure compares FLIQNAS to FLIQS and quantized NAS, which fixes the quantization format for all layers and only searches for the architecture. It shows that FLIQS-S and FLIQS-L searches perform well for low model costs, yet as the model scales to higher costs, the compute is better allocated by increasing the size of the architectural components. In this region, both quantized NAS and FLIQNAS yield the best performance. For all model costs, FLIQNAS-L is able to reach the Pareto-optimal tradeoff of accuracy and model cost. Lastly, when compared at identical cost targets, floating-point FLIQNAS surpasses the performance of the integer search space.In Figure 6 , we include a FLIQNAS comparison against APQ [25] , which performs a joint architecture, pruning, and quantization search by using a large once-for-all network. Its search space is similar and includes multiple kernel sizes, channel widths, and integer bitwidths built on top of the original MobileNetV2 architecture. This table shows that for similar GBOPs, FLIQNAS leads to higher accuracy over APQ across its three published design points. Further layer-wise analysis of these results is located in Appendix A.7. 
conclusion: 
 As AI hardware supports an increasing number of numerical formats, DNN quantization search to integer and low-precision floating-point grows increasingly important for reducing memory and compute. This paper proposes FLIQS, the first one-shot RL-based integer and low-precision floating-point quantization search without retraining. Compared to prior work, FLIQS can achieve higher accuracy without involving additional fine-tuning or retraining steps by introducing a cosine entropy regularization schedule. Moreover, as an RL-based method, it reduces the amount of memory needed for weights, activations, and gradients during the search compared to recent differentiable NAS searches.These enhancements accelerate research progress and enable quantization searches on larger search spaces and more substantial models, such as DeiT-B16, which has 10 times the model cost as BF16 MobileNetV2. In addition, FLIQS conducts the first floating-point quantization search and produces mixed-precision models that outperform the latest works on FP8 formats. When further combined with architecture search, it identifies even stronger MobileNetV2 models than NAS and quantization search alone. It further suggests that for a fixed compute budget, larger models benefit from increasing architectural dimensions over bitwidth. Overall, FLIQS represents an efficient framework for searching multi-precision models on current hardware and gives further insight into model and hardware co-design for future accelerator generations. 
 
 Appendix: The following sections contain additional experimental details, small experiments, ablation studies, and example output bitwidths. The listed hyper-parameters attempt to make the results more reproducible and interpretable. In addition, the small-scale experiments motivate certain hyperparameter selections discussed in the main paper. And finally, the example configurations give more insight into how FLIQS allocates bitwidth across different models and cost targets.FLIQS bitwidth configurations vary based on the model and search space. Figure 7 shows a set of configurations for FLIQS-L and FLIQS-S searches on a ResNet18 across four different model cost targets. Lower bitwidths are represented with colors closer to red and higher bitwidths are closer to green. This figure shows that FLIQS typically gives higher bitwidth to the first and last layers of the model. It also consistently gives higher bitwidths to the 1x1 convolution on the upper branch, and although not obvious in this figure, it usually allocates more bitwidth to the earlier stages of the model compared to later stages.Figure 8 shows example bitwidth configurations for all models evaluated. It reveals that ResNet50 has similar trends to ResNet18: more bitwidth for the first and last layers, 1x1 convolutions on the upper branch, and generally more in the early stages. Unlike the ResNet models, MobileNetV2 has a main block that comprises a sequence of a pointwise convolution, depthwise convolution, and then pointwise convolution. FLIQS allocates more bitwidth to the central 3x3 depthwise convolution in this block (groups of three in the figure). InceptionV3 has a more complicated branched architecture of 1x1, 3x3, and 5x5 convolutions. This block is shown in the figure as the repeated structure of one, three, two, and then one convolution, from top to bottom. FLIQS likewise gives more bitwidth to the earlier stages of InceptionV3 and its first and last layers. Additionally, it increases the precision of the 1x1 convolutions on the top and bottom of the repeated block. … K Q V K Q V DeiT-B16 EfficientNet 6b 10 GBOPs 6b 5b 6b 4b 5b 6b 4b 4b 5b 4b … E2 M5 E3 M3 E2 M4 E2 M3 E2 M3 E2 M5 E2 M5 90 GBOPs 6B 5B 4B 4B 5B 4B 4B 7B 320 GBOPs ResNet18 Attention MLP 7B 6B 8B 8B 6B 7B 7B BF 810 GBOPs K Q V … … E2 M3 E2 M1 E2 M3 E2 M2 E2 M2 E2 M4 E2 M3 … 60 GBOPs DeiT-B16 … E3 M2 E3 M3 E2 M5 E3 M1 E2 M1 E2 M5 Figure 8: Integer FLIQS-L Examples ResNet18 E2 M2 E2 M1 E2 M2 E2 M1 E2 M1 E2 M3 30 GBOPs E2 M1 E2 M1 E2 M5 E2 M1 E2 M1 E3 M4 … ResNet50 E2 M2 E2 M1 E2 M2 E2 M3 67 GBOPs E3 M4 … E2 M2 E2 M1 E2 M1 E2 M2 E2 M1 E2 M2 E2 M3 E2 M1 E2 M3 E2 M2 E2 M2 7 GBOPs E2 M1 E2 M3 E2 M2 E2 M1 E2 M2 E2 M1 E2 M1 E2 M3 … EfficientNet E2 M3 E2 M4 E2 M2 7 GBOPs E2 M1 E2 M4 E2 M3 E2 M1 E2 M3 E2 M2 E2 M2 E2 M1 … MobileNetV2 E2 M2 E2 M1 E2 M1 E2 M1 E2 M2 E2 M1 E2 M1 320 GBOPs Attention MLP DeiT-B16 E2 M3 … E2 M4 E2 M2 E2 M1 56 GBOPs E2 M3 E2 M1 E3 M1 E2 M2 E2 M4 E3 M2 E2 M1 InceptionV3 E3 M1 E2 M1 E2 M2 E4 M1 E2 M3 E2 M1 E2 M1 E2 M1 E2 M2 E2 M4 …GBOPsThese shared weights are quantized dynamically with a method adapted from the open-source library Accurate Quantized Training (AQT) [38] , which supports both integer and emulated lowprecision floating point quantization. This process can be summarized as:EQUATION𝜎 (𝑥 𝑓 ; 𝜎 𝑡 ) = max(-𝜎 𝑡 , min(𝑥 𝑓 , 𝜎 𝑡 )) (11) where 𝑥 𝑞 is the quantized value, 𝑥 𝑓 is the original full-precision number, 𝑠 is the scale factor, and 𝜎 𝑡 denotes the clipping threshold. In addition, 𝜎 (•) represents a clipping function, and ⌊•⌉ represents a generic rounding function that pushes the value to the nearest integer or low-precision floating-point value.The scale factor 𝑠 normalizes the input to the chosen maximum representable number and then rescales to the maximum quantized value. The clipping threshold and scale factor are determined by the run-time statistics of the weights and activations. Additionally, FLIQS uses a two-phase quantization approach where the weights and activations begin quantization at different training steps, as shown in Figure 2 .The two-phase quantization approach has been found empirically to improve the final accuracy of the model. In the first phase, only the weights are quantized and in the second phase, the weights and activations are quantized. The start step of the second phase has a large effect on the final accuracy. Table 10a shows the effect of sweeping the starting step for activation quantization on a ResNet50 trained to 30,200 steps. On one extreme, with the second phase starting as soon as possible, this method degenerates into a single-phase quantization method where weight and activation quantization begin immediately. On the other extreme, where the second phase begins as late as possible, it becomes a hybrid QAT-PTQ method where the weights are quantized during training and the activations are quantized after training.Table 10a shows that accuracy peaks around 15-20% of the total training time. For this reason, FLIQS uses 7500 steps as the start step for activation quantization for ResNets and InceptionV3, which train to 30,200 steps, and 20,000 as the start step for MobileNetV2, EfficientNet, and DeiT, which train to 112,000 steps or longer.The quantization method additionally depends on the activation clipping threshold, which is calculated as a multiple of the profiled activation standard deviations per layer. With too small a clipping threshold, there is lower rounding error on the more numerous central values distribution, yet there is significant clipping error on the larger values. With too large a threshold, there is more rounding error on the central values and less clipping error on the larger values.This trade-off is demonstrated empirically in Table 10b , where standard deviation multiples are swept from 1 to 8 and applied globally to a ResNet18. This table shows that the best accuracy are achieved around 3-4 times the standard deviation in ResNet18. For simplicity, we apply this multiple to all models for our experiments. Table 10c shows that the final accuracy is not sensitive to the number of profiling batches. This is likely because we use a large batch size of 2048, and since it is shuffled, it likely already provides a strong estimate of the statistics.The training hyper-parameters are chosen to be close to original paper hyper-parameters or recent related work. Table 4 shows the hyper-parameters used to produce Table 1 and Figure 5 . For our search, the RL controller warmup period lasts the first 25% of the training It uses an Adam optimizer with learning rate of 4.6E-3 and momentum of .95. The loss function is a standard softmax cross entropy loss with a label smoothing coefficient set to 0.1. A cosine entropy regularization schedule is applied to all runs beginning with no regularization and ending with 𝛽 𝐻 = .5. For QNAS, during the RL controller warmup period, the branches corresponding to various kernel sizes are sampled jointly with a probability schedule. This schedule begins at 1 at the beginning of training and decreases linearly to 0 at the end of the warmup period. After the warmup period, only a single branch is sampled at a time.In general, the search spaces used with FLIQS should reflect the capabilities of the target hardware. Small search spaces are useful for adapting a model to existing hardware such as the TPUv3 or NVIDIA A100. Large search spaces are useful for reconfigurable hardware such as the AMD Xilinx UltraScale+ FPGA and for co-designing models with future accelerators. The largest search space evaluated in this work includes 16 floating-point formats.For the integer FLIQS-S search space, we include INT4, INT8, and BF16. These are the standard formats supported in modern GPU micro-architectures, such as NVIDIA Ampere. Many platforms additionally support FP16, yet this format typically performs worse than BF16 in most common use For floating-point FLIQS-S, we include three formats to be consistent with the integer search variant. BF16 is the most common half-precision format, E4M3 is the FP8 variant most useful for inference (E4M2 primarily used for gradients), and E2M1 is a custom FP4 format. For FLIQS-L, we include all the formats with total bitwidths between four and eight.All custom formats support subnormals and do not support infinity. The bias terms are selected so the exponent range is symmetric about zero. However, this bias term is not relevant to FLIQS, since continuing from prior work [38, 6] , it uses a profiled scale factor during training and search. This means that the bias term combines with the profiled scale factor and has no additional effect. Therefore, the dynamic range is controlled more by the additional scale factor than the format itself and can adequately scale to various data distributions; the format instead primarily determines the distribution of quantization points (non-linear for floating-point and linear for integer ).A. 6 In general, QNAS searches tend to allocate more of their compute to architectural components, especially at high cost targets. This behavior is shown in Figure 6 , where expanding quantization searches to include flexible channels and kernel size dimensions increases the accuracy of the model at similar costs. Within these architectural components, typically the channel dimension is increased before the kernel size to reach cost targets. This could be due to the fundamental difference between kernel size and channel width; kernel size reflects the ability to aggregate information faster across spatial dimensions and channel width controls the depth of a neural network representation.The channel dimension allocations also show an interesting trend in that lower bitwidths typically receive a larger number of channels. This is intuitive since increasing the channel width can potentially recuperate losses in representational ability from the lower bitwidth. There is a weaker trend in this direction with kernel size, where the kernel size can tend to be larger with lower bitwidths, although it is not as strong.For shifting error and clipping analysis, we simulate the data distributions commonly found within neural networks. For this, we use Gaussian and Laplacian distributions and inject additional outlier values. These outliers are set at 3× the maximum value in the original tensor and are injected at various rates from 1:10 to 1:10000. These outliers are especially common in activation tensors.For the shifting error, we then sample 1000 tensors independently at random, and quantize them with two different symmetric linear quantizers that vary only in their bitwidths. We then calculate the RMS error between the two output tensors and average over all 1000 tensors. Finally, we fit the best exponential function with the form: 𝐴𝑒 (-𝐵𝑥 ) + 𝐶.Similarly, for the clipping analysis, we sample 100 tensors and calculate the quantization error between the original FP32 and quantized tensors for each percentile value. For the percentiles, we use a linear grid of 100 values from [1, 101] . We then plot the average MSE error over the 100 tensors and separately plot the optimal percentile. We experimented with different metrics, such as the Kullback-Liebler (KL) divergence, yet these did not lead to qualitatively different results. Table 7 shows the general effects of different possible FP8 formats on ImageNet accuracy. The models are generally resilient to FP8 quantization with MobileNetV2 having the largest accuracy degradation with the E5M2 format. This is analogous to integer quantization, where typically INT8 is sufficient for most models to maintain neutral accuracy and where MobileNetV2 is more sensitive to low-bit quantization. In this table, the accuracy trends upward with more mantissa bits, and therefore not only do they determine the majority of the area in floating-point units, they increase the accuracy of the models. This leads to the classical accuracy-performance trade-off that floating-point quantization search attempts to navigate for optimal configurations. Yet for hardened accelerators, the peak throughput for different FP8 formats is the same, and therefore higher mantissa bitwidth is preferable.Table 2 shows the hardware area estimates and accuracy of a set of ResNet-18 models on an AMD Xilinx UltraScale+ FPGA, implemented using Vivado HLS [39] . Since the whole model does not fit on the board, we estimate the cost with the first residual block in the model, which consists of two convolutional layers on one branch, and a pointwise convolution on the other, followed by their sum. Since all MAC operations are mapped to look-up tables (LUTs), the LUT count acts as a proxy for the area and power overhead of the model. The precision settings for FLIQS-L are taken from actual runs and represent the general bitwidth allocation to the ResNet blocks, although there may be some deviation within individual blocks.This table shows that LUTs scale quadratically with the precision bitwidth. Since the LUTs act as a proxy for area, this verifies the core assumption of the BOPs model (Section 1) that BOPs are proportional to the model area and power on chip. This table also confirms the overhead from these searched models is indeed relatively small compared to the accuracy improvements shown in Table 1 .With sufficient entropy regularization, retraining the model after FLIQS is unnecessary. Table 8 shows a sweep for ResNet18 with and without retraining. With retraining, the search occurs as described in Section 3, except that the best configuration is taken and retrained from scratch for the original training length. The table shows natural variance between the retraining and no-retraining methods, but there is no noticeable advantage to retraining across model widths. A.12 All ModelsFigure 11 plot all models with corresponding colors for methods and corresponding symbols for models. It shows that FLIQS MobileNetV2 and EfficientNet models consistently outperform other models in terms of accuracy and model cost, and BF16 models consistently perform the worst. This is expected since, as their name suggests, these models are designed specifically to be efficient and both use inverted bottleneck structures to reduce overall compute. The worst performing model overall is ResNet18, which is followed in the higher model costs by ResNet50.Next, we briefly explore FLIQNAS on recommendation models using the Criteo dataset [40] , which is the most popular public advertisement click-through-rate (CTR) prediction benchmark. We evaluate a multi-layer perceptron (MLP) model with four hidden layers and layer factorization technique [41] similar to the method used in DCN-V2 (Deep & Cross Network) [42] . We use the AUC metric for evaluation, and list additional details about the dataset, model architecture and search space. Figure 13 compares FLIQNAS and FLIQS with uniformly quantized models on both integer and float quantization. We focus only on FLIQS-L due to the small search space and do not include the uniformly quantized INT4 and E2M1 models since they show significant quality loss. Figure 13 shows that FLIQNAS-L performs better than FLIQS-L especially at larger MBOPs. Both of them show better quality and performance trade-offs than uniform quantization. Criteo: The Criteo dataset [40] contains user logs over a period of 7 days with a total of 45M examples. Each example has 13 continuous features and 26 categorical features with a binary label indicating if an advertisement was clicked or not.E2 M2 E2 M1 E2 M1 E2 M1 E2 M2 E2 M1 E2 M1 E2 M3 … E2 M4 E2 M2 E2 M1 56 GBOPs E2 M3 E2 M1 E3 M1 E2 M2 E3 M2 E2 M1 E3 M1 E2 M1 E2 M2 E4 M1 E2 M1 E2 M1 E2 M1 E2 M2 E2 M4 … Integer Models GBOPs ImageNet Float Models GBOPs ImageNet Integer vs. Float GBOPs ImageNet E2 M2 E2 M1 E2 M1 E2 M1 E2 M2 E2 M1 E2 M1 320 GBOPs E2 M3 … E2 M4 E2 M2 E2 M1 56 GBOPs E2 M3 E2 M1 E3 M1 E2 M2 E3 M2 E2 M1 E3 M1 E2 M1 E2 M2 E4 M1 E2 M1 E2 M1 E2 M1 E2 M2 E2 M4 … Integer Models GBOPsE2 M2 E2 M1 E2 M1 E2 M1 E2 M2 E2 M1 E2 M1 320 GBOPs Attention MLP DeiT-B16 E2 M3 … E2 M4 E2 M2 E2 M1 56 GBOPs E2 M3 E2 M1 E3 M1 E2 M2 E2 M4 E3 M2 E2 M1 E3 M1 E2 M1 E2 M2 E4 M1 E2 M3Architecture: The recommendation model architecture starts with an embedding layer to project the sparse categorical features into dense vectors. The embedded vectors are then concatenated with the continuous features and fed into the MLP with four hidden layers and low-rank on each layer to reduce the computational cost.Search Space: For FLIQS-L, the search space uses the same configurations for integer or floatingpoint search on each layer. For FLIQNAS-L, besides the quantization search space, we also include 128 and 512 × [0.25, 0.5, 0.75, 1.0, 1.25, 1.5, 1.75, 2.0] for rank values and layer widths respectively on each layer.  


Decision: The paper describes a new neural architecture search (NAS) method for mixed floating point and integer precision to quantize network. 
All reviewers are positive about the paper. Reviewer rnwf and 4KXy also highlight the strong experimental results. Overall, I think this paper could have a positive impact on NAS to stir the development of automated methods for quantization, which arguable becomes increasingly more important with ever larger networks.

Reviews: review: paper_summary: paper_summary: The authors consider quantization, a common method to reduce costs and runtime for model inference. The paper a/ introduces the first one-shot quantization search that doesn't require retraining by adding a cosine entropy regularization schedule, b/ compares both integer and low-precision floating-point quantization search across network types, b/ performs the largest comparison of integer and floating-point mixed-precision networks, and d/ studies quantization and NAS on low-precision floating-point networks and establishes compute resource recommendations.
main_review: main_review: + Introduces an automated quantization search for both integer and floating point models
+ Introduces a reinforcement-learning based controller
+ Establishes impactful SOTA performance on model architectures with quantization
+ Does not require re-training, resulting in lower costs / improved productivity
- Does not comment on outcomes/findings from ablation study in body of paper
review_summary: review_summary: The paper presents an impactful contribution to the field of neural network contribution, and is the first to do so for both integer and floating point; specifically, it outlines the first one-shot quantization search that doesn't require retraining via an RL-based controller, which would greatly lower costs / challenges with previous quantization methods. The paper provides a strong experiment setting and detailed comparisons to confidently establish state-of-the-art performance in quantized ResNet/MobileNet, and draws meaningful conclusions on trends encountered during experimentation.
paper_summary: paper_summary: The paper presents FLIQS which is the first one-shot mixed-precision quantization search to eliminate the need for retraining in both integer and low-recision floating point models. The method is evaluated fairly exhaustively across different model types (transformers and cnns) and 2 different search space types and achieves quantized models with lower latencies without additional retraining costs.
main_review: main_review: Positives:
1. Strong motivation of the problem
2. Presentation is clear in most parts
3. Evaluation across architecture types is fairly exhaustive 
5. Strong empirical results

Negatives
1. Some parts of the search pipeline are unclear (check Technical Quality And Correctness)
2. 2-stage one-shot methods and RL-based search algorithms are quite well studied in literature and could be applied for quantization search too. Though the paper is the first to propose a one-shot model (where in archs don't require retraining), I think the approach should compare to baselines like random-search, evolutionary-search on the shared model too.
review_summary: review_summary: This paper makes an important contribution in studying neural architecture search methods in a very large space of quantizations. The method is studied across architecture types, different quantization search spaces, and metrics (GBOPs,latency speedups). Moreover, since the architectures don't require further retraining this approach is also very much aligned with "Green AutoML" and need to reduced carbon consumptions in deep learning models (due to the efficiency of the method and the lower energy consumption of architectures discovered)
paper_summary: paper_summary: This paper introduces FLIQS, a technique for automated mixed-precision quantization, which involves both floating-point and integer formats. FLIQS employs a layer-wise approach and utilizes a one-shot reinforcement learning search method to optimize both the model architecture and the quantization formats simultaneously. Compared to traditional post-training quantization methods, FLIQS demonstrates higher accuracy performance.
main_review: main_review: # Strengths:
- Paper is well-written and addresses important research questions. 
- The proposed method does not require re-training.

# Weaknesses:
- Novelty of the paper is under question. I would like to hear from authors in which way their proposed method differs from SoTA papers (some are listed in the Clarity section). 
- Compared to related studies, results are limited to CNN and ViT models. I would like to see if (and how) the proposed method could be extended for language models.
    - Sridhar, Sharath Nittur, et al. "SimQ-NAS: Simultaneous Quantization Policy and Neural Architecture Search." arXiv preprint arXiv:2312.13301 (2023).
- Table 1: Compare the results of FLIQS with:
    - Dong, Peijie, et al. "Emq: Evolving training-free proxies for automated mixed precision quantization." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.
    - Huang, Wei, et al. "OHQ: On-chip Hardware-aware Quantization." arXiv preprint arXiv:2309.01945 (2023).
    - Bablani, Deepika, et al. "Efficient and effective methods for mixed precision neural network quantization for faster, energy-efficient inference." arXiv preprint arXiv:2301.13330 (2023).
- Reproducibility of results is under question.
- Comparison baselines are relatively old.
review_summary: review_summary: The proposal of a one-shot NAS method for designing a mixed-precision CNN/ViT is interesting, and I found the paper engaging to read. However, there are several areas where the paper could be strengthened. 
- Firstly, extending the results to include language models like BERT would enhance its applicability, although I understand this may not be feasible for this submission. 
- Secondly, evaluating latency results on other platforms such as CPUs or FPGAs would provide a more comprehensive understanding of the method's performance across different hardware. 
- Lastly, comparing the proposed method with state-of-the-art techniques published in 2023 and 2024 would help its effectiveness in the field.
paper_summary: paper_summary: This paper proposed to search for mixed-precision and quantization networks based on a one-shot and RL-based search algorithm.
main_review: main_review: This paper is well-motivated and well-organized.  Although its algorithm novelty is not very high, I'd like to see the authors apply NAS method to new applications. Experiments are solid. I believe this paper should be accepted.
review_summary: review_summary: see Overall Review