Nearly Lossless Adaptive Bit Switching

Venue: NeurIPS_cc_2024_Conference
Published: 2024-04-27 11:50:19

Abstract: Model quantization is widely applied for compressing and accelerating deep neural networks (DNNs). However, conventional Quantization-aware training (QAT) focuses on training DNNs with uniform bit-width. The bit-width settings vary across different hardware and transmission demands, which induces considerable training and storage costs. Hence, the scheme of one-shot joint training multiple precisions is proposed to address this issue. Previous works either store a larger FP32 model to switch between different precision models for higher accuracy or store a smaller INT8 model but compromise accuracy due to using shared quantization parameters. In this paper, we introduce the ${\bf  Double Rounding}$ quantization method, which fully utilizes the quantized representation range to accomplish nearly lossless bit-switching while reducing storage by using the highest integer precision instead of full precision. Furthermore, we observe a competitive interference among different precisions during one-shot joint training, primarily due to inconsistent gradients of quantization scales during backward propagation. To tackle this problem, we propose an Adaptive Learning Rate Scaling (${\bf ALRS}$) technique that dynamically adapts learning rates for various precisions to optimize the training process. Additionally, we extend our \emph{Double Rounding} to one-shot mixed precision training and develop a Hessian-aware Stochastic Bit-switching (${\bf HASB}$) strategy. Experimental results on the ImageNet-1K classification demonstrate that our methods have enough advantages to state-of-the-art one-shot joint QAT in both multi-precision and mixed-precision. Our codes are available at https://anonymous.4open.science/r/Double-Rounding-EF78/README.md.

Main paper: 
 introduction: 
 Recently, with the popularity of mobile and edge devices, more and more researchers have attracted attention to model compression due to the limitation of computing resources and storage. Model quantization [1; 2] has gained significant prominence in the industry. Quantization maps floating-point values to integer values, significantly reducing storage requirements and computational resources without altering the network architecture.Generally, for a given pre-trained model, the quantization bit-width configuration is predefined for a specific application scenario. The quantized model then undergoes retraining, i.e., QAT, to mitigate the accuracy decline. However, when the model is deployed across diverse scenarios with different precisions, it often requires repetitive retraining processes for the same model. A lot of computing resources and training costs are wasted. To address this challenge, involving the simultaneous training of multi-precision [3; 4] or one-shot mixed-precision [3; 5] have been proposed. Among these approaches, some involve sharing weight parameters between low-precision and high-precision models, enabling dynamic bit-width switching during inference. However, bit-switching from high precision (or bit-width) to low precision may introduce significant accuracy degradation due to the Rounding operation in the quantization process. Additionally, there is severe competition in the convergence process between higher and lower precisions in multi-precision Submitted to 38th Conference on Neural Information Processing Systems (NeurIPS 2024). Do not distribute. scheme. In mixed-precision scheme, previous methods often incur vast searching and retraining costs due to decoupling the training and search stages. Due to the above challenges, bit-switching remains a very challenging problem. Our motivation is designing a bit-switching quantization method that doesn't require storing a full-precision model and achieves nearly lossless switching from high-bits to low-bits. Specifically, for different precisions, we propose unified representation, normalized learning steps, and tuned probability distribution so that an efficient and stable learning process is achieved across multiple and mixed precisions, as depicted in Figure 1 .To solve the bit-switching problem, prior methods either store the floating-point parameters [6; 7; 4; 8] to avoid accuracy degradation or abandon some integer values by replacing rounding with floor [3; 9] but leading to accuracy decline or training collapse at lower bit-widths. We propose Double Rounding, which applies the rounding operation twice instead of once, as shown in Figure1 (a). This approach ensures nearly lossless bit-switching and allows storing the highest bit-width model instead of the full-precision model. Specifically, the lower precision weight is included in the higher precision weight, reducing storage constraints.Moreover, we empirically find severe competition between higher and lower precisions, particularly in 2-bit precision, as also noted in [10; 4] . There are two reasons for this phenomenon: The optimal quantization interval itself is different for higher and lower precisions. Furthermore, shared weights are used for different precisions during joint training, but the quantization interval gradients for different precisions exhibit distinct magnitudes during training. Therefore, we introduce an Adaptive Learning Rate Scaling (ALRS) method, designed to dynamically adjust the learning rates across different precisions, which ensures consistent update steps of quantization scales corresponding to different precisions, as shown in the Figure 1 (b) .Finally, we develop an efficient one-shot mixed-precision quantization approach based on Double Rounding. Prior mixed-precision approaches first train a SuperNet with predefined bit-width lists, then search for optimal candidate SubNets under restrictive conditions, and finally retrain or fine-tune them, which incurs significant time and training costs. However, we use the Hessian Matrix Trace [11] as a sensitivity metric for different layers to optimize the SuperNet and propose a Hessian-Aware Stochastic Bit-switching (HASB) strategy, inspired by the Roulette algorithm [12] . This strategy enables tuned probability distribution of switching bit-width across layers, assigning higher bits to more sensitive layers and lower bits to less sensitive ones, as shown in Figure 1 (c). And, we add the sensitivity to the search stage as a constraint factor. So, our approach can omit the last stage.In conclusion, our main contributions can be described as:• Double Rounding quantization method for multi-precision is proposed, which stores a single integer weight to enable adaptive precision switching with nearly lossless accuracy.• Adaptive Learning Rate Scaling (ALRS) method for the multi-precision scheme is introduced, which effectively narrows the training convergence gap between high-precision and low-precision, enhancing the accuracy of low-precision models without compromising high-precision model accuracy.• Hessian-Aware Stochastic Bit-switching (HASB) strategy for one-shot mixed-precision SuperNet is applied, where the access probability of bit-width for each layer is determined based on the layer's sensitivity.• Experimental results on the ImageNet1K dataset demonstrate that our proposed methods are comparable to state-of-the-art methods across different mainstream CNN architectures. 
related works: 
 Multi-Precision. Multi-Precision entails a single shared model with multiple precisions by one-shot joint Quantization-Aware Training (QAT). This approach can dynamically adapt uniform bit-switching for the entire model according to computing resources and storage constraints. AdaBits [13] is the first work to consider adaptive bit-switching but encounters convergence issues with 2-bit quantization on ResNet50 [14] . Bit-Mixer [9] addresses this problem by using the LSQ [2] quantization method but discards the lowest state quantized value, resulting in an accuracy decline. Multi-Precision joint QAT can also be viewed as a multi-objective optimization problem. Any-precision [6] andMultiQuant [4] combine knowledge distillation techniques to improve model accuracy. Among these methods, MultiQuant's proposed "Online Adaptive Label" training strategy is essentially a form of self-distillation [15] . Similar to our method, AdaBits and Bit-Mixer can save an 8-bit model, while other methods rely on 32-bit models for bit switching. Our Double Rounding method can store the highest bit-width model (e.g., 8-bit) and achieve almost lossless bit-switching, ensuring a stable optimization process. Importantly, this leads to a reduction in training time by approximately 10% [7] compared to separate quantization training.One-shot Mixed-Precision. Previous works mainly utilize costly approaches, such as reinforcement learning [16; 17] and Neural Architecture Search (NAS) [18; 19; 20] , or rely on partial prior knowledge [21; 22] for bit-width allocation, which may not achieve global optimality. In contrast, our proposed one-shot mixed-precision method employs Hessian-Aware optimization to refine a SuperNet via gradient updates, and then obtain the optimal conditional SubNets with less search cost without retraining or fine-tuning. Additionally, Bit-Mixer [9] and MultiQuant [4] implement layer-adaptive mixed-precision models, but Bit-Mixer uses a naive search method to attain a sub-optimal solution, while MultiQuant requires 300 epochs of fine-tuning to achieve ideal performance. Unlike NAS approaches [20] , which focus on altering network architecture (e.g., depth, kernel size, or channels), our method optimizes a once-for-all SuperNet using only quantization techniques without altering the model architecture. Rounding operation, Straight-Through Estimator (STE) [24] is commonly used to handle the nondifferentiability.However, for multi-precision quantization, bit-switching can result in significant accuracy loss, especially when transitioning from higher bit-widths to lower ones, e.g., from 8-bit to 2-bit. To[ \ Storage:32bit /64 2bit 3bit 4bit [ \ Storage:4bit $GD%LWV 2bit 3bit 4bit [ \ Storage:4bit %LWPL[HU 2bit 3bit 4bit [ \ Storage:4bit 'RXEOH5RXQGLQJ 2bit 3bit 4bitFigure 2 : Comparison of four quantization schemes:(from left to right) used in LSQ [2] , AdaBits [3] , Bit-Mixer [9] and Ours Double Rounding. In all cases y = dequant(quant(x)). mitigate this loss, prior works have mainly employed two strategies: one involves bit-switching from a floating-point model (32-bit) to a lower-bit model each time using multiple learnable quantization parameters, and the other substitutes the Rounding operation with the F loor operation, but this results in accuracy decline (especially in 2-bit). In contrast, we propose a nearly lossless bitswitching quantization method called Double Rounding. This method overcomes these limitations by employing a Rounding operation twice. It allows the model to be saved in the highest-bit (e.g., 8-bit) representation instead of full-precision, facilitating seamless switching to other bit-width models. A detailed comparison of Double Rounding with other quantization methods is shown in Figure 2 .Unlike AdaBits, which relies on the Dorefa [1] quantization method where the quantization scale is determined based on the given bit-width, the quantization scale of our Double Rounding is learned online and is not fixed. It only requires a pair of shared quantization parameters, i.e., scale and zero-point. Quantization scales of different precisions adhere to a strict "Power of Two" relationship.Suppose the highest-bit and the target low-bit are denoted as h-bit and l-bit respectively, and the difference between them is ∆ = h -l. The specific formulation of Double Rounding is as follows:EQUATIONEQUATIONEQUATIONwhere the symbol ⌊.⌉ denotes the Rounding function, and clip(x, low, upper) means x is limited to the range between low and upper. Here, W represents the FP32 model's weights, s h ∈ R and z h ∈ Z denote the highest-bit (e.g., 8-bit) quantization scale and zero-point respectively. W h represent the quantized weights of the highest-bit, while W l and W l represent the quantized weights and dequantized weights of the low-bit respectively.Hardware shift operations can efficiently execute the division and multiplication by 2 ∆ . Note that in our Double Rounding, the model can also be saved at full precision by using unshared quantization parameters to run bit-switching and attain higher accuracy. Because we use symmetric quantization scheme, the z h is 0. Please refer to Section A.4 for the gradient formulation of Double Rounding.Unlike fixed weights, activations change online during inference. So, the corresponding scale and zero-point values for different precisions can be learned individually to increase overall accuracy.Suppose X denotes the full precision activation, and X b and X b are the quantized activation and dequantized activation respectively. The quantization process can be formulated as follows:EQUATIONEQUATIONwhere s b ∈ R and z b ∈ Z represent the quantization scale and zero-point of different bit-widths activation respectively. Note that z b is 0 for the ReLU activation function.Question: Does the paper discuss the limitations of the work performed by the authors?Justification: [TODO] Although our proposed methods have achieved comparable results in multi-precision and mixed-precision, this paper has several limitations and improvements. Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.• The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.• The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.• The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.• The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.• If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.• While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best Justification: [TODO] Please refer to the Section A.6 in the appendix.• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).• The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)• The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean.• It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.• For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).• If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
adaptive learning rate scaling for multi-precision: 
 Although our proposed Double Rounding method represents a significant improvement over most previous multi-precision works, the one-shot joint optimization of multiple precisions remains constrained by severe competition between the highest and lowest precisions [10; 4] . Different precisions simultaneously impact each other during joint training, resulting in substantial differences in convergence rates between them, as shown in Figure 3 (c). We experimentally find that this competitive relationship stems from the inconsistent magnitudes of the quantization scale's gradients between high-bit and low-bit quantization during joint training, as shown in Figure 3 EQUATIONη b = 1 × 10 -∆ 2 , if ∆ is even 5 × 10 -( ∆+1 2 ) , if ∆ is odd (7)where the L is the number of layers, clip_grad(.) represents gradient clipping that prevents gradient explosion, max_abs(.) denotes the maximum absolute value of all elements. The ∇s i b denotes the quantization scale's gradients of layer i and η b denotes scaling hyperparameter of different precisions, e.g., 8-bit is 1, 6-bit is 0.1, and 4-bit is 0.01. Note that the ALRS strategy is only used for updating quantization scales. It can adaptively update the learning rates of different precisions and ensure that model can optimize quantization parameters at the same pace, ultimately achieving a minimal convergence gap in higher bits and 2-bit, as shown in Figure 3 (d) .In multi-precision scheme, different precisions share the same model weights during joint training.For conventional multi-precision, the shared weight computes n forward processes at each training iteration, where n is the number of candidate bit-widths. The losses attained from different precisions are then accumulated, and the gradients are computed. Finally, the shared parameters are updated.For detailed implementation please refer to Algorithm A.1 in the appendix. However, we find that if different precision losses separately compute gradients and directly update shared parameters at each forward process, it attains better accuracy when combined with our ALRS training strategy.Additionally, we use dual optimizers to update the weight parameters and quantization parameters simultaneously. We also set the weight-decay of the quantization scales to 0 to achieve stable convergence. For detailed implementation please refer to Algorithm A.2 in the appendix.Unlike multi-precision, where all layers uniformly utilize the same bit-width, mixed-precision SuperNet provides finer-grained adaptive by configuring the bit-width at different layers. Previous methods typically decouple the training and search stages, which need a third stage for retraining or fine-tuning the searched SubNets. These approaches generally incur substantial search costs in selecting the optimal SubNets, often employing methods such as greedy algorithms [26; 9] or genetic algorithms [27; 4] . Considering the fact that the sensitivity [28] , i.e., importance, of each layer is different, we propose a Hessian-Aware Stochastic Bit-switching (HASB) strategy for one-shot mixed-precision training.Specifically, the Hessian Matrix Trace (HMT) is utilized to measure the sensitivity of each layer. We first need to compute the pre-trained model's HMT by around 1000 training images [11] , as shown in Require: Candidate bit-widths set b ∈ B, the HMT of current layer: t l , average HMT: tm; 1: Sample r ∼ U (0, 1] from a uniform distribution; 2: if t l < tm then 3:Compute bit-switching probability of all candidate bi with pi = 1/n; 4:Set s = 0, and i = 0; 5:while s < r do 6:i = i + 1; 7: s = pi + s; 8:end while 9: else 10:Compute bit-switching probability of all candidate bi with pi = bi/∥B∥1; 11:Set s = 0, and i = 0; 12:while s < r do 13:i = i + 1; 14: s = pi + s; 15:end while 16: end if 17: return bi;Note that n and L represent the number of candidate bit-widths and model layers respectively, and ∥ • ∥1 is L1 norm. Setup. In this paper, we mainly focus on ImageNet-1K [30] classification task using both classical networks (ResNet18/50 [14] ) and lightweight networks (MobileNetV2 [31] ), which same as previous works. Experiments cover joint quantization training for multi-precision and mixed precision. We explore two candidate bit configurations, i.e., {8,6,4,2}-bit and {4,3,2}-bit, each number represents the quantization level of the weight and activation layers. Like previous methods, we exclude batch normalization layers from quantization, and the first and last layers are kept at full precision. We initialize the multi-precision models with a pre-trained FP32 model, and initialize the mixed-precision models with a pre-trained multi-precision model. All models use the Adam optimizer [32] with a batch size of 256 for 90 epochs and use a cosine scheduler without warm-up phase. The initial learning rate is 5e-4 and weight decay is 5e-5. Data augmentation uses the standard set of transformations including random cropping, resizing to 224×224 pixels, and random flipping. Images are resized to 256×256 pixels and then center-cropped to 224×224 resolution during evaluation. 
multi-precision: 
 Results. For {8,6,4,2}-bit configuration, the Top-1 validation accuracy is shown in Table 1 . The network weights and the corresponding activations are quantized into w-bit and a-bit respectively.Our double-rounding combined with ALRS training strategy surpasses the previous state-of-the-art (SOTA) methods. For example, in ResNet18, it exceeds Any-Precision [6] by 2.7%(or 2.83%) under w8a8 setting without(or with) using KD technique [15] , and outperforms MultiQuant [4] by 0.63%(or 0.73%) under w4a4 setting without(or with) using KD technique respectively. Additionally, when the candidate bit-list includes 2-bit, the previous methods can't converge on MobileNetV2 during training. So, they use {8,6,4}-bit precision for MobileNetV2 experiments. For consistency, we also test {8,6,4}-bit results, as shown in the "Ours {8,6,4}-bit" rows of Table 1 . Our method achieves 0.25%/0.11%/0.56% higher accuracy than AdaBits [3] under the w8a8/w6a6/w4a4 settings.Notably, our method exhibits the ability to converge but shows a big decline in accuracy on Mo-bileNetV2. On the one hand, the compact model exhibits significant differences in the quantization scale gradients of different channels due to involving DeepWise Convolution [33] . On the other hand, when the bit-list includes 2-bit, it intensifies competition between different precisions during training.To improve the accuracy of compact models, we suggest considering the per-layer or per-channel learning rate scaling techniques in future work. do not achieve optimal performance across all settings. We hypothesize that extending the number of training epochs or combining ILP with other efficient search methods, such as genetic algorithms, may be necessary to achieve optimal results in mixed-precision optimization. A Appendix / supplemental material A.1 OverviewIn this supplementary material, we present more explanations and experimental results.• First, we provide a detailed explanation of the different quantization types under QAT.• We then present a comparison of multi-precision and separate-precision on the ImageNet-1k dataset.• Furthermore, we provide the gradient formulation of Double Rounding.• And, the algorithm implementation of both multi-precision and mixed-precision training approaches.• Finally, we provide more gradient statistics of learnable quantization scales in different networks.In this section, we provide a detailed explanation of the different quantization types during Quantization-Aware Training (QAT), as is shown in Figure 6 . We provide the comparison of Multi-Precision and Separate-Precision on ImageNet-1K dataset.Table 6 shows that our Multi-Precision joint training scheme has comparable accuracy of different precisions compared to Separate-Precision with multiple re-train. This further proves the effectiveness of our proposed One-shot Double Rounding Multi-Precision method. A general formulation for uniform quantization process is as follows:EQUATIONEQUATIONwhere the symbol ⌊.⌉ denotes the Rounding function, clip(x, low, upper) expresses x below low are set to low and above upper are set to upper. b denotes the quantization level (or bit-width), s ∈ R and z ∈ Z represents the quantization scale (or interval) and zero-point associated with each b, respectively. W represents the FP32 model's weights, W signifies the quantized integer weights, and W represents the dequantized floating-point weights.The quantization scale of our Double Rounding is learned online and not fixed. And it only needs a pair of shared quantization parameters, i.e., scale and zero-point. Suppose the highest-bit and the low-bit are denoted as h-bit and l-bit respectively, and the difference between them is ∆ = h -l.The specific formulation is as follows:EQUATIONEQUATIONEQUATION)where s h ∈ R and z h ∈ Z denote the highest-bit quantization scale and zero-point respectively. W h and W l represent the quantized weights of the highest-bit and low-bit respectively. Hardware shift operations can efficiently execute the division and multiplication by 2 ∆ . And the z h is 0 for the weight quantization in this paper. The gradient formulation of Double Rounding for one-shot joint training is represented as follows:EQUATIONEQUATION)where n and p denote the lower and upper bounds of the integer range [N min , N max ] for quantizing the weights or activations respectively. Y represents the FP32 weights or activations, and Y represents the dequantized weights or activations. Unlike weights, activation quantization scale and zero-point of different precisions are learned independently. However, the gradient formulation is the same.This section provides the algorithm implementations of multi-precision, one-shot mixed-precision joint training, and the search stage of SubNets.The multi-precision model with different quantization precisions shares the same model weight(e.g., the highest-bit) during joint training. In conventional multi-precision, the shared weight (e.g., multiprecision model) computes n forward processes at each training iteration, where n is the number of candidate bit-widths. Then, all attained losses of different precisions perform an accumulation, and update the parameters accordingly. For specific implementation details please refer to Algorithm A.1.However, we find that if separate precision loss and parameter updates are performed directly after calculating a precision at each forward process, it will lead to difficulty convergence during training or suboptimal accuracy. In other words, the varying gradient magnitudes of quantization scales of different precisions make it hard to attain stable convergence during joint training. To address this issue, we introduce an adaptive approach (e.g., Adaptive Learning Rate Scaling, ALRS) to alter the learning rate for different precisions during training, aiming to achieve a consistent update pace.This method allows us to directly update the shared parameters after calculating the loss after every forward. We update both the weight parameters and quantization parameters simultaneously using dual optimizers. We also set the weight-decay of the quantization scales to 0 to achieve more stable convergence. For specific implementation details, please refer to Algorithm A.2. for each quantization layer do 7:W b = dequant(quant(W, s b w )) 8: X b = dequant(quant(X, s b x )) 9: O b = Conv( W b , X b ) 10:end for 11:o b = F C(W, O b ) 12:Update BN b layer 13:Compute loss:L b = CE(o b , y) 14:Compute gradients: L b .backward() 15: end for 16: Update weights and scales: optim.step(λ) 17: Clear gradient: optim.zero_grad(); Note that n and L represent the number of candidate bit-widths and model layers respectively. for each quantization layer do 7:W b = dequant(quant(W, s b w )) 8: X b = dequant(quant(X, s b x )) 9: O b = Conv( W b , X b ) 10:end for 11:o b = F C(W, O b ) 12:Update BN b layer 13:Compute loss: L b = CE(o b , y) 14:Compute gradients: L b .backward() 15:Compute learning rate: λ b # please see formula (6) of the main paper 16:Update weights and quantization scales: optim1.step(λ); optim2.step(λ b ) 17:Clear gradient: optim1.zero_grad(); optim2.zero_grad() 18: end for Note that n and L represent the number of candidate bit-widths and model layers respectively.Unlike multi-precision joint quantization, the bit-switching of mixed-precision training is more complicated. In multi-precision training, the bit-widths calculated in each iteration are fixed, e.g., {8,6,4,2}-bit. In mixed-precision training, the bit-widths of different layers are not fixed in each iteration, e.g., {8,random-bit,2}-bit, where "random-bit" is any bits of e.g., {7,6,5,4,3,2}-bit, similar to the sandwich strategy of [39] . Therefore, mixed precision training often requires more training epochs to reach convergence compared to multi-precision training. Bit-mixer [9] conducts the same probability of selecting bit-width for different layers. However, we take the sensitivity of each layer into consideration which uses sensitivity (e.g. Hessian Matrix Trace [11] ) as a metric to identify the selection probability of different layers. For more sensitive layers, preference is given to higher-bit widths, and vice versa. We refer to this training strategy as a Hessian-Aware Stochastic Bit-switching (HASB) strategy for optimizing one-shot mixed-precision SuperNet. Specific implementation details can be found in Algorithm A.3. In additionally, unlike multi-precision joint training, the BN layers are replaced by TBN (Transitional Batch-Norm) [9] , which compensates for the distribution shift between adjacent layers that are quantized to different bit-widths. To achieve the best convergence effect, we propose that the threshold of bit-switching (i.e., σ) also increases as the epoch increases. Justification: [TODO] Please refer to the Abstract Section and Section 1, where related material for the question can be found.• The answer NA means that the abstract and introduction do not include the claims made in the paper.• The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.• The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.• It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Justification: [TODO] Please refer to the Table 5 .• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.• The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.• The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [NA] Justification: [TODO] Due to space limitations, this social impact aspect is not discussed in the main paper. This paper doesn't involve negative societal impacts including potential malicious or unintended uses. Our proposed methods aim to achieve an efficient and effective model compression technique to flexible adaptive different storage and computation requirements, which are beneficial to social advancement.• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.• The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.• If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: [TODO] This paper doesn't have any high risk for misuse.• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.• Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.• We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. • The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL.• The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.• For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.• If this information is not available online, the authors are encouraged to reach out to the asset's creators. • The paper should discuss whether and how consent was obtained from people whose asset is used.• At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.• Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.  
 
 Appendix:  


Decision: The paper proposes algorithms for multi-bit width quantization where parameters can be chosen at different bit-width and trained in a single-shot manner. Double-rounding quantizer helps reduce storage requirements, ALRS dynamically updates learning rate per precision.
All the reviewers highlight limited novelty, missing comparisons to recent literature in multi bit-width quantization, weak results on domain outside of CV (LLMs) and limited ablation studies.
I recommend rejection in its current state, but encourage the authors to address the reviewer comments and strengthen the manuscript for a future publication.

Reviews: review: paper_summary: paper_summary: The authors propose a bit-switching quantization method using Double Rounding, which applies rounding twice to achieve nearly lossless switching without storing a full-precision model. They also introduce Adaptive Learning Rate Scaling (ALRS) to adjust learning rates dynamically across precisions, ensuring consistent quantization updates. Additionally, they develop Hessian-Aware Stochastic Bit-switching (HASB) for one-shot mixed-precision training, optimizing bit-width distribution based on layer sensitivity, thus eliminating retraining stages.
strength_weakness: strength_weakness: **Novelty** is limited and I am not highly motivated that the problem is important.

1.	The main contribution is to not same 32bit weight and different quantization parameters but only the high bidwith using a pretty straightforward idea of double rounding during training
2.	The ALRS is based on observation and heuristic to fix it. It is nice and helps for when trying to use 2 bits as well. Yet, I am not sure it is important for methods that don’t use the double rounding.

**Motivation**

3.	Since we usually don’t switch models based on data I am not sure why this is important. Do we really have edge device that switch on a daily base model precision and thus need to store in small local memory the 32bit model? Can you elaborate why and where multi precision is really important.

4. No results on more recent models (LLMs)
questions: questions: I don't understand why you claim the ALRS method was inspired by LARS. The only similarity seems to be the name. Can you explain the connection?

Can you provide a scenario where your method is particularly important?

You state that “if different precision losses separately compute gradients and directly update shared parameters at each forward process, it attains better accuracy when combined with our ALRS training strategy.” However, this involves updating the gradients four times more frequently, which is inefficient (4x backward pass and optimizer). This seems equivalent to small batch versus large batch training. Have you considered simply increasing the learning rate with the conventional multi-precision training approach? It might achieve the same results.

In Table 4, have you run the same number of forward/backward passes with the {8, 6, 4, 2} and {4, 3, 2} bit-widths? Since you have only three bit-widths in the latter, you might need to run 4/3 more iterations to ensure the total number of updates is the same. Have you tried that?
limitations: limitations: The authors partially discuss limitation
paper_summary: paper_summary: This paper discusses advanced methods in multi-bit model quantization. Specifically, this paper proposes a method for one-shot joint training of multiple precisions. To this end, the authors introduce a double-rounding quantizer that leverages the highest integer precision to achieve nearly lossless bit-switching while reducing storage requirements. Moreover, they also propose an Adaptive Learning Rate Scaling technique that adjusts learning rates dynamically for different precisions. Two proposed techniques mitigate the competitive interference between bit-widths caused by inconsistent gradients of different precisions during biased gradient estimation. They also extend their Double Rounding method to support one-shot mixed precision training and develop a Hessian-aware Bit-witdh sampling strategy. Experimental results on the ImageNet-1K classification task show that their methods outperform state-of-the-art one-shot joint QAT in both multi-precision and mixed-precision scenarios.
strength_weakness: strength_weakness: - More uniquness analysis needed. The using of Hessian information seems a bit trivial, each layer's Hessian is just used to compare with the averaged Hessian trace. Firstly, as shown in recent zero-cost NAS research [1], the architectural proxies will be less effective as the training goes on, I'm not sure the Hessian information obtained on the initial full-precision model will remain useful as the quantization-aware training continues. Moreover, the sampling probability is modified with a simple ascending heuristic, which is not Hessian-aware. 

- Also applies here: the design of the double-rounding quantizer is similar to Bit-Mixer, Adabits, and ABN. Specifically, ABN also uses 

- ALRS needs further ablations. In ALRS, the authors use a fixed scaling ratio to bit-widths, e.g., 8-bit is 1, 6-bit is 0.1, and 4-bit is 0.01, the choice of these scaling factors still requires more ablation studies and discussions. 

- More comparisons needed. Since this paper adopts an ILP-based search algorithm to find optimal subnets, it is better to compare with these ILP-based mixed-precision quantization papers, e.g., [2] and [3]. 



[1] A Deeper Look at Zero-Cost Proxies for Lightweight NAS 
[2] Mixed-precision neural network quantization via learned layer-wise importance, ECCV 2022. 
[3] Hawq-v2: Hessian aware trace-weighted quantization of neural networks, NIPS 2020.
questions: questions: - How do the authors perform KD for the proposed method?  Is an external teacher used or only distilled from the highest precision with the in-place distillation?
limitations: limitations: NA
paper_summary: paper_summary: The paper proposes a QAT scheme to jointly optimize a single model with different precisions. The authors apply their scheme on various CNN-based models on CIFAR-10 and ImageNet datasets.
strength_weakness: strength_weakness: 1. I think the main limitation of the paper is the models and datasets. I believe that the study should be done on larger models (LLMs for example) as a architecture goal. For example, the authors show that they do not save a FP32 master copy of the model in their scheme. However, ResNet style models (or MobileNet) are easy to fit in even moderate GPUs and I don't think FP32 master copy is a big problem in that case (please correct me if I'm wrong).

2. I couldn't find a source-code to reproduce the results of the paper in my side.
questions: questions: 1. The paper assumes a power-of-two relation between different precision. Do you have any theoretical intuition for this assumption? or this is just because of fast HW implementation?

2. What is the cost HMT in different networks? How can we compare it against the e2e runtime?
limitations: limitations: yes.
paper_summary: paper_summary: The paper addresses challenges in model quantization for deep neural networks (DNNs), focusing on optimizing quantization-aware training (QAT) across multiple bit-widths with weight-sharing. To this end, this paper introduces a novel quantization method that exploits the highest integer precision to achieve nearly lossless bit-switching, reducing storage without relying on full precision. Key contributions include: (1) Adaptive Learning Rate Scaling: A technique that dynamically adjusts learning rates for different precisions to address competitive interference and inconsistent gradient issues during one-shot joint training. (2) Double Rounding: An extension for one-step rounding quantizer in fixed-precision quantization to improve accuracy. Experimental results on the ImageNet-1K dataset show that the proposed methods surpass state-of-the-art approaches in both multi-precision and mixed-precision scenarios, achieving higher efficiency and accuracy.
strength_weakness: strength_weakness: - Some analysis is missing. For example, I'm wondering whether the second rounding leads to more quantization errors, as the first rounding is used to produce INT8 weights and second rounding is then performed to quantize lower bit-width, the twice quantization is possible to cause more clipping errors and rounding errors, some analysis could enhance the strength of proposed methods. 
- Some designs should be further clarified, e.g., why ALRS is applied only for the scaling factors? Intuitively, weights of small bit-width is induced large gradient variance by STE, and thus the weights of small bit-width should also benefit from using smaller LR. 
- Fig. 1 is a bit confusing, some colored arrows are not well explained. 
- This works essentially lies in the research of mixed-precision quantization, so I think it is better to compare more MPQ (e.g., HAQ, DNAS, LIMPQ, etc) research in the Sec.4. Moreover, some recent papers on multi bit-width quantization are missed on the , e.g., [1] (PTQ-based) and [2][3] (QAT-based), which could be included into the Related Work. 

[1] Xu, Ke, et al. "PTMQ: Post-training Multi-Bit Quantization of Neural Networks." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 38. No. 14. 2024.

[2] Tang, Chen, et al. "Retraining-free model quantization via one-shot weight-coupling learning." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024. 

[3] Zhong, Yunshan, et al. "MultiQuant: A Novel Multi-Branch Topology Method for Arbitrary Bit-width Network Quantization." arXiv preprint arXiv:2305.08117 (2023).
questions: questions: Please refer to the weaknesses.
limitations: limitations: Please refer to the weaknesses. Overall, this paper currently needs more experiments and analysis to reveal some designs are reasonable.

Comments: title: Answer to the authors rebuttal
Reply
comment: Dear Reviewer T2Zq,

Sorry to bother you again. Thank you for your valuable reply. We have carefully addressed the points raised and have incorporated the suggestions provided to further strengthen the manuscript. Notably, other reviewers have expressed positive evaluations, which we have reflected upon and integrated into the revisions. Given these enhancements and the overall positive reception, we kindly ask if you would consider re-evaluating your score, taking into account the improvements made.

Your thoughtful reassessment would be greatly appreciated and would contribute significantly to the overall quality and impact of our work.

Thank you once again for your time and consideration.

Best regards,

Authors
Dear Reviewer Aa14,

Sorry to bother you again. Thank you for your valuable reply. We have carefully addressed the points raised and have incorporated the suggestions provided to further strengthen the manuscript. Notably, other reviewers have expressed positive evaluations, which we have reflected upon and integrated into the revisions. Given these enhancements and the overall positive reception, we kindly ask if you would consider re-evaluating your score, taking into account the improvements made.

Your thoughtful reassessment would be greatly appreciated and would contribute significantly to the overall quality and impact of our work.

Thank you once again for your time and consideration.

Best regards,

Authors
Dear Reviewer bT7w,

Sorry to bother you again. Thank you for your valuable reply. We have carefully addressed the points raised and have incorporated the suggestions provided to further strengthen the manuscript. Notably, other reviewers have expressed positive evaluations, which we have reflected upon and integrated into the revisions. Given these enhancements and the overall positive reception, we kindly ask if you would consider re-evaluating your score, taking into account the improvements made.

Your thoughtful reassessment would be greatly appreciated and would contribute significantly to the overall quality and impact of our work.

Thank you once again for your time and consideration.

Best regards,

Authors
Dear Reviewer rFvS,

Sorry to bother you again. Thank you for your valuable reply. We have carefully addressed the points raised and have incorporated the suggestions provided to further strengthen the manuscript. Notably, other reviewers have expressed positive evaluations, which we have reflected upon and integrated into the revisions. Given these enhancements and the overall positive reception, we kindly ask if you would consider re-evaluating your score, taking into account the improvements made.

Your thoughtful reassessment would be greatly appreciated and would contribute significantly to the overall quality and impact of our work.

Thank you once again for your time and consideration.

Best regards,

Authors
Dear Reviewer bT7w,

Thanks for your positive comments. We are pleased to address your questions and concerns. We will add the relevant discussions in the final version. We sincerely implore that you can consider raising the final score. In this case, we will be greatly inspired and will contribute to the community permanently. Thanks again for your time and efforts in this paper.

Best regards,

Authors
Thanks for addressing the questions and comments in the previous round. I also read the others' comments and remain positive for the rating.
Dear reviewer rFvS,

Thanks for your positive comments. We are pleased to answer your questions and concerns. We will add the relevant discussions in the final version. We sincerely implore that you can consider raising the final score. In this case, we will be greatly inspired and will contribute to the community permanently. Thanks again for your time and efforts in this paper.

Best regards

Authors
I would like to thank the reviewers for their detailed responses. The authors addressed my questions effectively. Due to their additional experiments and the fact that ALRS has been shown to improve accuracy for other multi-precision methods, I have raised my score to 5.
Dear reviewer Aa14,

Thanks for your positive comments. We are sorry that you can't find the code we provided for some reason. We will forward an anonymous code link to you via AC, and we promise to make the code publicly available on GitHub later. We are pleased to answer your questions and concerns. We will add the relevant discussions in the final version. We sincerely implore that you can consider raising the final score. In this case, we will be greatly inspired and will contribute to the community permanently. Thanks again for your time and efforts in this paper.

Best regards

Authors
Dear reviewer T2Zq,

Thanks for your positive comments. We will add the additional results, discussions, and references in the final revision. We are pleased to answer your questions and concerns. We sincerely implore that you can consider raising the final score. In this case, we will be greatly inspired and will contribute to the community permanently. Thanks again for your time and efforts in this paper.

Best regards

Authors
Thanks for the response. I have carefully read the rebuttal and other reviews, the response addressed my concerns. I will increase my rating while I hope the authors can incorporate these additional results, discussions, and references into the revision. BTW, DNAS refers to the paper entitled "Mixed precision quantization of convnets via differentiable neural architecture search".
Thank you so much for answering my questions and providing more results. For some reason, I couldn't find the code :-)

I will increase my score.