Title: Nearly Lossless Adaptive Bit Switching
Venue: NeurIPS 2024
Accepted: no
Keywords: Model Quantization

One-Shot Mixed-Precision

Multi-Preciison

Quantization-Aware Training
Abstract: Model quantization is widely applied for compressing and accelerating deep neural networks (DNNs). However, conventional Quantization-aware training (QAT) focuses on training DNNs with uniform bit-width. The bit-width settings vary across different hardware and transmission demands, which induces considerable training and storage costs. Hence, the scheme of one-shot joint training multiple precisions is proposed to address this issue. Previous works either store a larger FP32 model to switch between different precision models for higher accuracy or store a smaller INT8 model but compromise accuracy due to using shared quantization parameters. In this paper, we introduce the ${\bf  Double Rounding}$ quantization method, which fully utilizes the quantized representation range to accomplish nearly lossless bit-switching while reducing storage by using the highest integer precision instead of full precision. Furthermore, we observe a competitive interference among different precisions during one-shot joint training, primarily due to inconsistent gradients of quantization scales during backward propagation. To tackle this problem, we propose an Adaptive Learning Rate Scaling (${\bf ALRS}$) technique that dynamically adapts learning rates for various precisions to optimize the training process. Additionally, we extend our \emph{Double Rounding} to one-shot mixed precision training and develop a Hessian-aware Stochastic Bit-switching (${\bf HASB}$) strategy. Experimental results on the ImageNet-1K classification demonstrate that our methods have enough advantages to state-of-the-art one-shot joint QAT in both multi-precision and mixed-precision. Our codes are available at https://anonymous.4open.science/r/Double-Rounding-EF78/README.md.

Abstract

Model quantization is widely applied for compressing and accelerating deep neural networks (DNNs). However, conventional Quantization-Aware Training (QAT) focuses on training DNNs with uniform bit-width. The bit-width settings vary across different hardware and transmission demands, which induces considerable training and storage costs. Hence, the scheme of one-shot joint training multiple precisions is proposed to address this issue. Previous works either store a larger FP32 model to switch between different precision models for higher accuracy or store a smaller INT8 model but compromise accuracy due to using shared quantization parameters. In this paper, we introduce the Double Rounding quantization method, which fully utilizes the quantized representation range to accomplish nearly lossless bit-switching while reducing storage by using the highest integer precision instead of full precision. Furthermore, we observe a competitive interference among different precisions during one-shot joint training, primarily due to inconsistent gradients of quantization scales during backward propagation. To tackle this problem, we propose an Adaptive Learning Rate Scaling ( ALRS ) technique that dynamically adapts learning rates for various precisions to optimize the training process. Additionally, we extend our Double Rounding to one-shot mixed precision training and develop a Hessian-Aware Stochastic Bit-switching ( HASB ) strategy. Experimental results on the ImageNet-1K classification demonstrate that our methods have enough advantages to state-of-the-art one-shot joint QAT in both multi-precision and mixed-precision. Our codes are available at here.

1 Introduction

Recently, with the popularity of mobile and edge devices, more and more researchers have attracted attention to model compression due to the limitation of computing resources and storage. Model quantization [1; 2] has gained significant prominence in the industry. Quantization maps floating-point values to integer values, significantly reducing storage requirements and computational resources without altering the network architecture.

Generally, for a given pre-trained model, the quantization bit-width configuration is predefined for a specific application scenario. The quantized model then undergoes retraining, i.e. , QAT, to mitigate the accuracy decline. However, when the model is deployed across diverse scenarios with different precisions, it often requires repetitive retraining processes for the same model. A lot of computing resources and training costs are wasted. To address this challenge, involving the simultaneous training of multi-precision [3; 4] or one-shot mixed-precision [3; 5] have been proposed. Among these approaches, some involve sharing weight parameters between low-precision and high-precision models, enabling dynamic bit-width switching during inference.

However, bit-switching from high precision (or bit-width) to low precision may introduce significant accuracy degradation due to the

Rounding operation in the quantization process. Additionally, there is

severe competition in the convergence process between higher and lower precisions in multi-precision

FP32

INT8

INT6

INT2

First

Rounding

High bit

Second

Rounding

Low bit

Adaptive

6bit

2bit

4bit

8bit

8bit

6bit

2bit

4bit

Bit Mixing

Selection

Adaptive

Learning Rate

Double

Round Bit

Switch

Multi-Precision

Mixed-Precision

Full-Precision

2bit

4bit

6bit

8bit

layer

layer

8bit

2bit

100%

0%

Probability

Stochastic

Bit-switching

Sensitivity

(a) Saving only 8-bit representation

for various low-precisions

(b) Stabilize learning process

across various precisions

(c) Sensitivity-aware bit selection

for different layers

scheme. In mixed-precision scheme, previous methods often incur vast searching and retraining costs 39

due to decoupling the training and search stages. Due to the above challenges, bit-switching remains 40

a very challenging problem. Our motivation is designing a bit-switching quantization method that 41

doesn't require storing a full-precision model and achieves nearly lossless switching from high-bits to 42

low-bits. Specifically, for different precisions, we propose unified representation, normalized learning 43

steps, and tuned probability distribution so that an efficient and stable learning process is achieved 44

across multiple and mixed precisions, as depicted in Figure 1. 45

To solve the bit-switching problem, prior methods either store the floating-point parameters [6; 7; 4; 8] 46

to avoid accuracy degradation or abandon some integer values by replacing rounding with floor [3; 9] 47

but leading to accuracy decline or training collapse at lower bit-widths. We propose Double Rounding , 48

which applies the rounding operation twice instead of once, as shown in Figure1 (a). This approach 49

ensures nearly lossless bit-switching and allows storing the highest bit-width model instead of the 50

full-precision model. Specifically, the lower precision weight is included in the higher precision 51

weight, reducing storage constraints. 52

Moreover, we empirically find severe competition between higher and lower precisions, particularly 53

in 2-bit precision, as also noted in [10; 4]. There are two reasons for this phenomenon: The optimal 54

quantization interval itself is different for higher and lower precisions. Furthermore, shared weights 55

are used for different precisions during joint training, but the quantization interval gradients for 56

different precisions exhibit distinct magnitudes during training. Therefore, we introduce an Adaptive 57

Learning Rate Scaling (ALRS) method, designed to dynamically adjust the learning rates across 58

different precisions, which ensures consistent update steps of quantization scales corresponding to 59

different precisions, as shown in the Figure 1 (b). 60

Finally, we develop an efficient one-shot mixed-precision quantization approach based on Double 61

Rounding . Prior mixed-precision approaches first train a SuperNet with predefined bit-width lists, 62

then search for optimal candidate SubNets under restrictive conditions, and finally retrain or fine-tune 63

them, which incurs significant time and training costs. However, we use the Hessian Matrix Trace [11] 64

as a sensitivity metric for different layers to optimize the SuperNet and propose a Hessian-Aware 65

Stochastic Bit-switching (HASB) strategy, inspired by the Roulette algorithm [12]. This strategy 66

enables tuned probability distribution of switching bit-width across layers, assigning higher bits to 67

more sensitive layers and lower bits to less sensitive ones, as shown in Figure 1 (c). And, we add the 68

sensitivity to the search stage as a constraint factor. So, our approach can omit the last stage. 69

70

71

72

73

74

75

76

77

78

79

80

81

82

83

84

85

86

87

88

89

90

91

92

93

94

95

96

97

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

In conclusion, our main contributions can be described as:

Double Rounding quantization method for multi-precision is proposed, which stores a single integer weight to enable adaptive precision switching with nearly lossless accuracy.

Adaptive Learning Rate Scaling (ALRS) method for the multi-precision scheme is introduced, which effectively narrows the training convergence gap between high-precision and low-precision, enhancing the accuracy of low-precision models without compromising high-precision model accuracy.

Hessian-Aware Stochastic Bit-switching (HASB) strategy for one-shot mixed-precision SuperNet is applied, where the access probability of bit-width for each layer is determined based on the layer's sensitivity.

Experimental results on the ImageNet1K dataset demonstrate that our proposed methods are comparable to state-of-the-art methods across different mainstream CNN architectures.

2 Related Works

Multi-Precision. Multi-Precision entails a single shared model with multiple precisions by one-shot joint Quantization-Aware Training (QAT). This approach can dynamically adapt uniform bit-switching for the entire model according to computing resources and storage constraints. AdaBits [13] is the first work to consider adaptive bit-switching but encounters convergence issues with 2-bit quantization on ResNet50 [14]. Bit-Mixer [9] addresses this problem by using the LSQ [2] quantization method but discards the lowest state quantized value, resulting in an accuracy decline. Multi-Precision joint QAT can also be viewed as a multi-objective optimization problem. Any-precision [6] and MultiQuant [4] combine knowledge distillation techniques to improve model accuracy. Among these methods, MultiQuant's proposed "Online Adaptive Label" training strategy is essentially a form of self-distillation [15]. Similar to our method, AdaBits and Bit-Mixer can save an 8-bit model, while other methods rely on 32-bit models for bit switching. Our Double Rounding method can store the highest bit-width model (e.g., 8-bit) and achieve almost lossless bit-switching, ensuring a stable optimization process. Importantly, this leads to a reduction in training time by approximately 10% [7] compared to separate quantization training.

One-shot Mixed-Precision. Previous works mainly utilize costly approaches, such as reinforcement learning [16; 17] and Neural Architecture Search (NAS) [18; 19; 20], or rely on partial prior knowledge [21; 22] for bit-width allocation, which may not achieve global optimality. In contrast, our proposed one-shot mixed-precision method employs Hessian-Aware optimization to refine a SuperNet via gradient updates, and then obtain the optimal conditional SubNets with less search cost without retraining or fine-tuning. Additionally, Bit-Mixer [9] and MultiQuant [4] implement layer-adaptive mixed-precision models, but Bit-Mixer uses a naive search method to attain a sub-optimal solution, while MultiQuant requires 300 epochs of fine-tuning to achieve ideal performance. Unlike NAS approaches [20], which focus on altering network architecture (e.g., depth, kernel size, or channels), our method optimizes a once-for-all SuperNet using only quantization techniques without altering the model architecture.

3 Methodology

3.1 Double Rounding

Conventional separate precision quantization using Quantization-Aware Training (QAT) [23] attain a fixed bit-width quantized model under a pre-trained FP32 model. A pseudo-quantization node is inserted into each layer of the model during training. This pseudo-quantization node comprises two operations: the quantization operation quant ( x ) , which maps floating-point (FP32) values to lowerbit integer values, and the dequantization operation dequant ( x ) , which restores the quantized integer value to its original floating-point representation. It can simulate the quantization error incurred when compressing float values into integer values. As quantization involves a non-differentiable Rounding operation, Straight-Through Estimator (STE) [24] is commonly used to handle the nondifferentiability.

However, for multi-precision quantization, bit-switching can result in significant accuracy loss, especially when transitioning from higher bit-widths to lower ones, e.g. , from 8-bit to 2-bit. To

























Storage:32bit



2bit

3bit

4bit

























Storage:4bit



2bit

3bit

4bit

























Storage:4bit



2bit

3bit

4bit

























Storage:4bit



2bit

3bit

4bit

mitigate this loss, prior works have mainly employed two strategies: one involves bit-switching from 121 a floating-point model (32-bit) to a lower-bit model each time using multiple learnable quantization 122 parameters, and the other substitutes the Rounding operation with the Floor operation, but this 123 results in accuracy decline (especially in 2-bit). In contrast, we propose a nearly lossless bit124 switching quantization method called Double Rounding . This method overcomes these limitations by 125 employing a Rounding operation twice. It allows the model to be saved in the highest-bit ( e.g. , 8-bit) 126 representation instead of full-precision, facilitating seamless switching to other bit-width models. A 127 detailed comparison of Double Rounding with other quantization methods is shown in Figure 2. 128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

Unlike AdaBits, which relies on the Dorefa [1] quantization method where the quantization scale is determined based on the given bit-width, the quantization scale of our Double Rounding is learned online and is not fixed. It only requires a pair of shared quantization parameters, i.e. , scale and zero-point . Quantization scales of different precisions adhere to a strict "Power of Two" relationship. Suppose the highest-bit and the target low-bit are denoted as h -bit and l -bit respectively, and the difference between them is ∆ = h -l . The specific formulation of Double Rounding is as follows:

where the symbol ⌊ . ⌉ denotes the Rounding function, and clip ( x, low, upper ) means x is limited to the range between low and upper . Here, W represents the FP32 model's weights, s h ∈ R and z h ∈ Z denote the highest-bit ( e.g. , 8-bit) quantization scale and zero-point respectively. ˜ W h represent the quantized weights of the highest-bit, while ˜ W l and ̂ W l represent the quantized weights and dequantized weights of the low-bit respectively.

Hardware shift operations can efficiently execute the division and multiplication by 2 ∆ . Note that in our Double Rounding , the model can also be saved at full precision by using unshared quantization parameters to run bit-switching and attain higher accuracy. Because we use symmetric quantization scheme, the z h is 0 . Please refer to Section A.4 for the gradient formulation of Double Rounding .

Unlike fixed weights, activations change online during inference. So, the corresponding scale and zero-point values for different precisions can be learned individually to increase overall accuracy. Suppose X denotes the full precision activation, and ˜ X b and ̂ X b are the quantized activation and dequantized activation respectively. The quantization process can be formulated as follows:

where s b ∈ R and z b ∈ Z represent the quantization scale and zero-point of different bit-widths activation respectively. Note that z b is 0 for the ReLU activation function.

3.2 Adaptive Learning Rate Scaling for Multi-Precision

Although our proposed Double Rounding method represents a significant improvement over most 151 previous multi-precision works, the one-shot joint optimization of multiple precisions remains 152 constrained by severe competition between the highest and lowest precisions [10; 4]. Different 153 precisions simultaneously impact each other during joint training, resulting in substantial differences 154

in convergence rates between them, as shown in Figure 3 (c). We experimentally find that this 155 competitive relationship stems from the inconsistent magnitudes of the quantization scale's gradients 156 between high-bit and low-bit quantization during joint training, as shown in Figure 3 (a) and (b). For 157 other models statistical results please refer to Section A.6 in the appendix. 158

         



















         



















1

11

21

31

61

71

81

41

51

Epoch

5

10

15

20

70

65

60

55

50

45

40

35

30

25

Acc1(%)

8bit

6bit

4bit

2bit

gap

1

11

21

31

41

51

61

71

81

Epoch

5

10

15

20

25

30

35

40

45

50

55

60

65

70

Acc1(%)

8bit

6bit

4bit

2bit

(a) 2-bit

(b) 4-bit

(c) w/o ALRS

(d) w. ALRS

Motivated by these observations, we introduce a technique termed Adaptive Learning Rate Scaling 159 (ALRS), which dynamically adjusts learning rates for different precisions to optimize the training 160 process. This technique is inspired by the Layer-wise Adaptive Rate Scaling (LARS) [25] optimizer. 161 Specifically, suppose the current batch iteration's learning rate is λ , we set learning rates λ b of 162 different precisions as follows: 163

where the L is the number of layers, clip_grad ( . ) represents gradient clipping that prevents gradient explosion, max_abs ( . ) denotes the maximum absolute value of all elements. The ∇ s i b denotes the quantization scale's gradients of layer i and η b denotes scaling hyperparameter of different precisions, e.g. , 8-bit is 1 , 6-bit is 0 . 1 , and 4-bit is 0 . 01 . Note that the ALRS strategy is only used for updating quantization scales. It can adaptively update the learning rates of different precisions and ensure that model can optimize quantization parameters at the same pace, ultimately achieving a minimal convergence gap in higher bits and 2-bit, as shown in Figure 3 (d).

In multi-precision scheme, different precisions share the same model weights during joint training. For conventional multi-precision, the shared weight computes n forward processes at each training iteration, where n is the number of candidate bit-widths. The losses attained from different precisions are then accumulated, and the gradients are computed. Finally, the shared parameters are updated. For detailed implementation please refer to Algorithm A.1 in the appendix. However, we find that if different precision losses separately compute gradients and directly update shared parameters at each forward process, it attains better accuracy when combined with our ALRS training strategy. Additionally, we use dual optimizers to update the weight parameters and quantization parameters simultaneously. We also set the weight-decay of the quantization scales to 0 to achieve stable convergence. For detailed implementation please refer to Algorithm A.2 in the appendix.

3.3 One-Shot Mixed-Precision SuperNet

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

Unlike multi-precision, where all layers uniformly utilize the same bit-width, mixed-precision SuperNet provides finer-grained adaptive by configuring the bit-width at different layers. Previous methods typically decouple the training and search stages, which need a third stage for retraining or fine-tuning the searched SubNets. These approaches generally incur substantial search costs in selecting the optimal SubNets, often employing methods such as greedy algorithms [26; 9] or genetic algorithms [27; 4]. Considering the fact that the sensitivity [28], i.e. , importance, of each layer is different, we propose a Hessian-Aware Stochastic Bit-switching (HASB) strategy for one-shot mixed-precision training.

Specifically, the Hessian Matrix Trace (HMT) is utilized to measure the sensitivity of each layer. We 190 first need to compute the pre-trained model's HMT by around 1000 training images [11], as shown in 191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

8bit

2bit

4bit

6bit

Probability

25%

25%

25%

25%

8bit

2bit

4bit

6bit

Probability

25%

25%

25%

25%

8bit

2bit

4bit

6bit

Probability

25%

25%

25%

25%

8bit

2bit

4bit

6bit

Probability

10%

20%

30%

40%

















 















weights

































(a) Unsensitive

(b) Sensitive

(c) Hessian trace

(d) Mixed precision

Figure 4 (c). Then, the HMT of different layers is utilized as the probability metric for bit-switching. Higher bits are priority selected for sensitive layers, while all candidate bits are equally selected for unsensitive layers. Our proposed Roulette algorithm is used for bit-switching processes of different layers during training, as shown in the Algorithm 1. If a layer's HMT exceeds the average HMT of all layers, it is recognized as sensitive, and the probability distribution of Figure 4 (b) is used for bit selection. Conversely, if the HMT is below the average, the probability distribution of Figure 4 (a) is used for selection. Finally, the Integer Linear Programming (ILP) [29] algorithm is employed to find the optimal SubNets. Considering each layer's sensitivity during training and adding this sensitivity to the ILP's constraint factors ( e.g. , model's FLOPs, latency, and parameters), which depend on the actual deployment requirements. We can efficiently attain a set of optimal SubNets during the search stage without retraining, thereby significant reduce the overall costs. All the searched SubNets collectively constitute the Pareto Frontier optimal solution, as shown in Figure 4 (d). For detailed mixed-precision training and searching process ( i.e. , ILP) please refer to the Algorithm A.3 and the Algorithm 2 respectively.

̸

Algorithm 1 Roulette algorithm for bit-switching

Algorithm 2 Our searching process for SubNets

4 Experimental Results

Setup. In this paper, we mainly focus on ImageNet-1K [30] classification task using both classical networks (ResNet18/50 [14]) and lightweight networks (MobileNetV2 [31]), which same as previous works. Experiments cover joint quantization training for multi-precision and mixed precision. We explore two candidate bit configurations, i.e. , {8,6,4,2}-bit and {4,3,2}-bit, each number represents the quantization level of the weight and activation layers. Like previous methods, we exclude batch

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

243

normalization layers from quantization, and the first and last layers are kept at full precision. We initialize the multi-precision models with a pre-trained FP32 model, and initialize the mixed-precision models with a pre-trained multi-precision model. All models use the Adam optimizer [32] with a batch size of 256 for 90 epochs and use a cosine scheduler without warm-up phase. The initial learning rate is 5e-4 and weight decay is 5e-5. Data augmentation uses the standard set of transformations including random cropping, resizing to 224 × 224 pixels, and random flipping. Images are resized to 256 × 256 pixels and then center-cropped to 224 × 224 resolution during evaluation.

4.1 Multi-Precision

Results. For {8,6,4,2}-bit configuration, the Top-1 validation accuracy is shown in Table 1. The network weights and the corresponding activations are quantized into w-bit and a-bit respectively. Our double-rounding combined with ALRS training strategy surpasses the previous state-of-the-art (SOTA) methods. For example, in ResNet18, it exceeds Any-Precision [6] by 2.7%(or 2.83%) under w8a8 setting without(or with) using KD technique [15], and outperforms MultiQuant [4] by 0.63%(or 0.73%) under w4a4 setting without(or with) using KD technique respectively. Additionally, when the candidate bit-list includes 2-bit, the previous methods can't converge on MobileNetV2 during training. So, they use {8,6,4}-bit precision for MobileNetV2 experiments. For consistency, we also test {8,6,4}-bit results, as shown in the "Ours {8,6,4}-bit" rows of Table 1. Our method achieves 0.25%/0.11%/0.56% higher accuracy than AdaBits [3] under the w8a8/w6a6/w4a4 settings.

Notably, our method exhibits the ability to converge but shows a big decline in accuracy on MobileNetV2. On the one hand, the compact model exhibits significant differences in the quantization scale gradients of different channels due to involving DeepWise Convolution [33]. On the other hand, when the bit-list includes 2-bit, it intensifies competition between different precisions during training. To improve the accuracy of compact models, we suggest considering the per-layer or per-channel learning rate scaling techniques in future work.

For {4,3,2}-bit configuration, Table 2 demonstrate that our double-rounding consistently surpasses previous SOTA methods. For instance, in ResNet18, it exceeds Bit-Mixer [9] by 0.63%/0.7%/1.2%(or 0.37%/0.64%/1.02%) under w4a4/w3a3/w2a2 settings without(or with) using KD technique, and outperforms ABN[10] by 0.87%/0.74%/1.12% under w4a4/w3a3/w2a2 settings with using KD technique respectively. In ResNet50, Our method outperforms Bit-Mixer [9] by 0.86%/0.63%/0.1% under w4a4/w3a3/w2a2 settings.

Notably, the overall results of Table 2 are worse than the {8,6,4,2}-bit configuration for joint training. We analyze that this discrepancy arises from information loss in the shared lower precision model

( i.e. , 4-bit) used for bit-switching. In other words, compared with 4-bit, it is easier to directly optimize 244 8-bit quantization parameters to converge to the optimal value. So, we recommend including 8-bit for 245 multi-precision training. Furthermore, independently learning the quantization scales for different 246 precisions, including weights and activations, significantly improves accuracy compared to using 247 shared scales. However, it requires saving the model in 32-bit format, as shown in "Ours*" of Table 2. 248

4.2 Mixed-Precision

Results. We follow previous works to conduct mixed-precision experiments based on the {4,3,2}-bit configuration. Our proposed one-shot mixed-precision joint quantization method with the HASB technique comparable to the previous SOTA methods, as presented in Table 3. For example, in ResNet18, our method exceeds Bit-Mixer [9] by 0.83%/0.72%/0.77%/7.07% under w4a4/w3a3/w2a2/3MP settings and outperforms EQ-Net[5] by 0.2% under 3MP setting. The results demonstrate the effectiveness of one-shot mixed-precision joint training to consider sensitivity with Hessian Matrix Trace when randomly allocating bit-widths for different layers. Additionally, Table 3 reveals that our results do not achieve optimal performance across all settings. We hypothesize that extending the number of training epochs or combining ILP with other efficient search methods, such as genetic algorithms, may be necessary to achieve optimal results in mixed-precision optimization.

4.3 Ablation Studies

ALRS vs. Conventional in Multi-Precision. To verify the effectiveness of our proposed ALRS training strategy, we conduct an ablation experiment without KD, as shown in Table 4, and observe overall accuracy improvements, particularly for the 2bit. Like previous works, where MobileNetV2 can't achieve stable convergence with {4,3,2}-bit, we also opt for {8,6,4}-bit to keep consistent. However, our method can achieve stable convergence with {8,6,4,2}-bit quantization. This demonstrates the superiority of our proposed Double-Rounding and ALRS methods.

Multi-Precision vs. Separate-Precision in Time Cost. Westatistic the results regarding the time cost for multi-precision compared to separate-precision quantization, as shown in Table 5. Multi-precision training costs stay approximate constant as the number of candidate bit-widths.

249

250

251

252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

268

269

276

Pareto Frontier of Different Mixed-Precision Configurations. To verify the effectiveness of our 270 HASB strategy, we conduct ablation experiments on different bit-lists. Figure 5 shows the search 271 results of Mixed-precision SuperNet under {8,6,4,2}-bit, {4,3,2}-bit and {8,4}-bit configurations 272 respectively. Where each point represents a SubNet. These results are obtained directly from ILP 273 sampling without retraining or fine-tuning. As the figure shows, the highest red points are higher than 274 the blue points under the same bit width, indicating that this strategy is effective. 275









































































































(a) {8,6,4,2}-bit

(b) {4,3,2}-bit

(c) {8,4}-bit

5 Conclusion

This paper first introduces Double Rounding quantization method used to address the challenges 277 of multi-precision and mixed-precision joint training. It can store single integer-weight parameters 278 and attain nearly lossless bit-switching. Secondly, we propose an Adaptive Learning Rate Scaling 279 (ALRS) method for multi-precision joint training that narrows the training convergence gap between 280 high-precision and low-precision, enhancing model accuracy of multi-precision. Finally, our proposed 281 Hessian-Aware Stochastic Bit-switching (HASB) strategy for one-shot mixed-precision SuperNet 282 and efficient searching method combined with Integer Linear Programming, achieving approximate 283 Pareto Frontier optimal solution. Our proposed methods aim to achieve a flexible and effective model 284 compression technique for adapting different storage and computation requirements. 285

286

287

288

289

290

291

292

References

S. Zhou, Y. Wu, Z. Ni, X. Zhou, H. Wen, and Y. Zou, 'Dorefa-net: Training low bitwidth convolutional neural networks with low bitwidth gradients,' arXiv preprint arXiv:1606.06160 , 2016.

S. K. Esser, J. L. McKinstry, D. Bablani, R. Appuswamy, and D. S. Modha, 'Learned step size quantization,' arXiv preprint arXiv:1902.08153 , 2019.

Q. Jin, L. Yang, and Z. Liao, 'Adabits: Neural network quantization with adaptive bit-widths,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2020, pp. 2146-2156.

K. Xu, Q. Feng, X. Zhang, and D. Wang, 'Multiquant: Training once for multi-bit 293 quantization of neural networks,' in IJCAI , L. D. Raedt, Ed. International Joint Conferences 294 on Artificial Intelligence Organization, 7 2022, pp. 3629-3635, main Track. [Online]. Available: 295 https://doi.org/10.24963/ijcai.2022/504 296

K. Xu, L. Han, Y. Tian, S. Yang, and X. Zhang, 'Eq-net: Elastic quantization neural networks,' in 297 Proceedings of the IEEE/CVF International Conference on Computer Vision , 2023, pp. 1505-1514. 298

H. Yu, H. Li, H. Shi, T. S. Huang, and G. Hua, 'Any-precision deep neural networks,' in Proceedings of 299 the AAAI Conference on Artificial Intelligence , vol. 35, no. 12, 2021, pp. 10 763-10 771. 300

K. Du, Y. Zhang, and H. Guan, 'From quantized dnns to quantizable dnns,' CoRR , vol. abs/2004.05284, 301 2020. [Online]. Available: https://arxiv.org/abs/2004.05284 302

X. Sun, R. Panda, C.-F. R. Chen, N. Wang, B. Pan, A. Oliva, R. Feris, and K. Saenko, 'Improved techniques 303 for quantizing deep networks with adaptive bit-widths,' in Proceedings of the IEEE/CVF Winter Conference 304 on Applications of Computer Vision , 2024, pp. 957-967. 305

306

307

A. Bulat and G. Tzimiropoulos, 'Bit-mixer: Mixed-precision networks with runtime bit-width selection,' in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2021, pp. 5188-5197.

308

309

310

311

312

313

314

315

316

317

318

319

320

321

322

323

324

325

326

327

328

329

330

331

332

333

334

335

C. Tang, H. Zhai, K. Ouyang, Z. Wang, Y. Zhu, and W. Zhu, 'Arbitrary bit-width network: A joint layer-wise quantization and adaptive inference approach,' 2022. [Online]. Available: https://arxiv.org/abs/2204.09992

Z. Dong, Z. Yao, D. Arfeen, A. Gholami, M. W. Mahoney, and K. Keutzer, 'Hawq-v2: Hessian aware trace-weighted quantization of neural networks,' Advances in neural information processing systems , vol. 33, pp. 18 518-18 529, 2020.

Y. Dong, R. Ni, J. Li, Y. Chen, H. Su, and J. Zhu, 'Stochastic quantization for learning accurate low-bit deep neural networks,' International Journal of Computer Vision , vol. 127, pp. 1629-1642, 2019.

Q. Jin, L. Yang, and Z. Liao, 'Towards efficient training for neural network quantization,' arXiv preprint arXiv:1912.10207 , 2019.

K. He, X. Zhang, S. Ren, and J. Sun, 'Deep residual learning for image recognition,' CoRR , vol. abs/1512.03385, 2015. [Online]. Available: http://arxiv.org/abs/1512.03385

K. Kim, B. Ji, D. Yoon, and S. Hwang, 'Self-knowledge distillation with progressive refinement of targets,' in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2021, pp. 6567-6576.

K. Wang, Z. Liu, Y. Lin, J. Lin, and S. Han, 'Haq: Hardware-aware automated quantization with mixed precision,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2019, pp. 8612-8620.

A. Elthakeb, P. Pilligundla, F. Mireshghallah, A. Yazdanbakhsh, S. Gao, and H. Esmaeilzadeh, 'Releq: an automatic reinforcement learning approach for deep quantization of neural networks,' in NeurIPS ML for Systems workshop, 2018 , 2019.

B. Wu, Y. Wang, P. Zhang, Y. Tian, P. Vajda, and K. Keutzer, 'Mixed precision quantization of convnets via differentiable neural architecture search,' arXiv preprint arXiv:1812.00090 , 2018.

Z. Guo, X. Zhang, H. Mu, W. Heng, Z. Liu, Y. Wei, and J. Sun, 'Single path one-shot neural architecture search with uniform sampling,' in Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XVI 16 . Springer, 2020, pp. 544-560.

M. Shen, F. Liang, R. Gong, Y. Li, C. Li, C. Lin, F. Yu, J. Yan, and W. Ouyang, 'Once quantization-aware training: High performance extremely low-bit architecture search,' in Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , October 2021, pp. 5340-5349.

J. Liu, J. Cai, and B. Zhuang, 'Sharpness-aware quantization for deep neural networks,' arXiv preprint 336 arXiv:2111.12273 , 2021. 337

338

339

340

Z. Yao, Z. Dong, Z. Zheng, A. Gholami, J. Yu, E. Tan, L. Wang, Q. Huang, Y. Wang, M. Mahoney et al. , 'Hawq-v3: Dyadic neural network quantization,' in International Conference on Machine Learning . PMLR, 2021, pp. 11 875-11 886.

341

342

343

344

345

346

347

348

349

350

351

352

353

354

355

356

357

358

359

360

361

362

363

364

365

366

367

368

369

370

371

372

373

374

375

376

377

378

379

380

381

382

B. Jacob, S. Kligys, B. Chen, M. Zhu, M. Tang, A. G. Howard, H. Adam, and D. Kalenichenko, 'Quantization and training of neural networks for efficient integer-arithmetic-only inference,' CoRR , vol. abs/1712.05877, 2017. [Online]. Available: http://arxiv.org/abs/1712.05877

Y. Bengio, N. Léonard, and A. Courville, 'Estimating or propagating gradients through stochastic neurons for conditional computation,' arXiv preprint arXiv:1308.3432 , 2013.

Y. You, I. Gitman, and B. Ginsburg, 'Large batch training of convolutional networks,' arXiv preprint arXiv:1708.03888 , 2017.

Z. Cai and N. Vasconcelos, 'Rethinking differentiable search for mixed-precision neural networks,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2020, pp. 23492358.

Z. Guo, X. Zhang, H. Mu, W. Heng, Z. Liu, Y. Wei, and J. Sun, 'Single path one-shot neural architecture search with uniform sampling,' in European conference on computer vision . Springer, 2020, pp. 544-560.

Z. Dong, Z. Yao, A. Gholami, M. W. Mahoney, and K. Keutzer, 'Hawq: Hessian aware quantization of neural networks with mixed-precision,' in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2019, pp. 293-302.

Y. Ma, T. Jin, X. Zheng, Y. Wang, H. Li, Y. Wu, G. Jiang, W. Zhang, and R. Ji, 'Ompq: Orthogonal mixed precision quantization,' in Proceedings of the AAAI conference on artificial intelligence , vol. 37, no. 7, 2023, pp. 9029-9037.

J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, 'Imagenet: A large-scale hierarchical image database,' in 2009 IEEE conference on computer vision and pattern recognition . Ieee, 2009, pp. 248-255.

M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and L.-C. Chen, 'Mobilenetv2: Inverted residuals and linear bottlenecks,' in Proceedings of the IEEE conference on computer vision and pattern recognition , 2018, pp. 4510-4520.

D. P. Kingma and J. Ba, 'Adam: A method for stochastic optimization,' arXiv preprint arXiv:1412.6980 , 2014.

T. Sheng, C. Feng, S. Zhuo, X. Zhang, L. Shen, and M. Aleksic, 'A quantization-friendly separable convolution for mobilenets,' in 2018 1st Workshop on Energy Efficient Machine Learning and Cognitive Computing for Embedded Applications (EMC2) . IEEE, 2018, pp. 14-18.

Q. Sun, X. Li, Y. Ren, Z. Huang, X. Liu, L. Jiao, and F. Liu, 'One model for all quantization: A quantized network supporting hot-swap bit-width adjustment,' arXiv preprint arXiv:2105.01353 , 2021.

M. Alizadeh, A. Behboodi, M. van Baalen, C. Louizos, T. Blankevoort, and M. Welling, 'Gradient l1 regularization for quantization robustness,' arXiv preprint arXiv:2002.07520 , 2020.

B. Chmiel, R. Banner, G. Shomron, Y. Nahshan, A. Bronstein, U. Weiser et al. , 'Robust quantization: One model to rule them all,' Advances in neural information processing systems , vol. 33, pp. 5308-5317, 2020.

H. Wu, R. He, H. Tan, X. Qi, and K. Huang, 'Vertical layering of quantized neural networks for heterogeneous inference,' IEEE Transactions on Pattern Analysis and Machine Intelligence , vol. 45, no. 12, pp. 15 964-15 978, 2023.

Y. Bhalgat, J. Lee, M. Nagel, T. Blankevoort, and N. Kwak, 'Lsq+: Improving low-bit quantization through learnable offsets and better initialization,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops , 2020, pp. 696-697.

J. Yu, L. Yang, N. Xu, J. Yang, and T. Huang, 'Slimmable neural networks,' arXiv preprint arXiv:1812.08928 , 2018.

A Appendix / supplemental material 383

384

385

386

387

388

389

390

391

A.1 Overview

In this supplementary material, we present more explanations and experimental results.

First, we provide a detailed explanation of the different quantization types under QAT.

We then present a comparison of multi-precision and separate-precision on the ImageNet-1k dataset.

Furthermore, we provide the gradient formulation of Double Rounding.

And, the algorithm implementation of both multi-precision and mixed-precision training approaches.

Finally, we provide more gradient statistics of learnable quantization scales in different networks.

A.2 Different Quantization Types

In this section, we provide a detailed explanation of the different quantization types during 392 Quantization-Aware Training (QAT), as is shown in Figure 6.

(a)

Separate-Precision

:  Each bit-width

requires training a new network with

separate weights by repeating multi-retrain.

(b)

Multi-Precision

: A shared network can be

quantized to any bit-width at runtime without

re-training or finetuning. All layers inside the

network uniformly share the same bit-width.

(c)

Mixed-Precision

: A SuperNet whose

individual layers can be quantized to any

bit-width at runtime, and its searched

subnets without re-training or fine-tuning.

8bit

2bit

𝐿𝐿1

𝐿𝐿2

𝐿𝐿n

32bit

6bit

8bit

2bit

𝐿𝐿1

𝐿𝐿2

𝐿𝐿n

32bit

6bit

8bit

2bit

𝐿𝐿1

𝐿𝐿2

𝐿𝐿n

32bit

6bit

Separate Train

Layer

Input

Bit Mixing

Data Flow

Bit Switching

A.3 Multi-Precision vs. Separate-Precision.

We provide the comparison of Multi-Precision and Separate-Precision on ImageNet-1K dataset. Table 6 shows that our Multi-Precision joint training scheme has comparable accuracy of different precisions compared to Separate-Precision with multiple re-train. This further proves the effectiveness of our proposed One-shot Double Rounding Multi-Precision method.

393

394

395

396

397

398

A.4 The Gradient Formulation of Double Rounding 399

A general formulation for uniform quantization process is as follows: 400

401

402

403

404

405

406

407

408

409

410

411

412

413

414

415

416

417

418

419

420

421

422

423

424

425

426

427

428

429

430

431

432

433

434

435

436

437

where the symbol ⌊ . ⌉ denotes the Rounding function, clip ( x, low, upper ) expresses x below low are set to low and above upper are set to upper . b denotes the quantization level (or bit-width), s ∈ R and z ∈ Z represents the quantization scale (or interval) and zero-point associated with each b , respectively. W represents the FP32 model's weights, ˜ W signifies the quantized integer weights, and ̂ W represents the dequantized floating-point weights.

The quantization scale of our Double Rounding is learned online and not fixed. And it only needs a pair of shared quantization parameters, i.e. , scale and zero-point . Suppose the highest-bit and the low-bit are denoted as h -bit and l -bit respectively, and the difference between them is ∆ = h -l . The specific formulation is as follows:

where s h ∈ R and z h ∈ Z denote the highest-bit quantization scale and zero-point respectively. ˜ W h and ˜ W l represent the quantized weights of the highest-bit and low-bit respectively. Hardware shift operations can efficiently execute the division and multiplication by 2 ∆ . And the z h is 0 for the weight quantization in this paper. The gradient formulation of Double Rounding for one-shot joint training is represented as follows:

where n and p denote the lower and upper bounds of the integer range [ N min , N max ] for quantizing the weights or activations respectively. Y represents the FP32 weights or activations, and ̂ Y represents the dequantized weights or activations. Unlike weights, activation quantization scale and zero-point of different precisions are learned independently. However, the gradient formulation is the same.

A.5 Algorithms

This section provides the algorithm implementations of multi-precision, one-shot mixed-precision joint training, and the search stage of SubNets.

A.5.1 Multi-Precision Joint Training

The multi-precision model with different quantization precisions shares the same model weight( e.g. , the highest-bit) during joint training. In conventional multi-precision, the shared weight ( e.g. , multiprecision model) computes n forward processes at each training iteration, where n is the number of candidate bit-widths. Then, all attained losses of different precisions perform an accumulation, and update the parameters accordingly. For specific implementation details please refer to Algorithm A.1.

However, we find that if separate precision loss and parameter updates are performed directly after calculating a precision at each forward process, it will lead to difficulty convergence during training or suboptimal accuracy. In other words, the varying gradient magnitudes of quantization scales of different precisions make it hard to attain stable convergence during joint training. To address this issue, we introduce an adaptive approach ( e.g. , Adaptive Learning Rate Scaling, ALRS) to alter the learning rate for different precisions during training, aiming to achieve a consistent update pace. This method allows us to directly update the shared parameters after calculating the loss after every forward. We update both the weight parameters and quantization parameters simultaneously using dual optimizers. We also set the weight-decay of the quantization scales to 0 to achieve more stable convergence. For specific implementation details, please refer to Algorithm A.2.

438

439

440

441

442

443

444

445

446

447

448

A.5.2 One-shot Joint Training for Mixed Precision SuperNet

(HASB) strategy for optimizing one-shot mixed-precision SuperNet. Specific implementation details 449 can be found in Algorithm A.3. In additionally, unlike multi-precision joint training, the BN layers 450 are replaced by TBN (Transitional Batch-Norm) [9], which compensates for the distribution shift 451 between adjacent layers that are quantized to different bit-widths. To achieve the best convergence 452 effect, we propose that the threshold of bit-switching ( i.e. , σ ) also increases as the epoch increases. 453

Algorithm A.3 Our one-shot Mixed-precision SuperNet training approach

,

Note that n and L represent the number of candidate bit-widths and model layers respectively.

A.5.3 Efficient one-shot searching for Mixed Precision SuperNet

After training the mixed-precision SuperNet, the next step is to select the appropriate optimal SubNets based on conditions, such as model parameters, latency, and FLOPs, for actual deployment and inference. To achieve optimal allocations for candidate bit-width under given conditions, we employ the Iterative Integer Linear Programming (ILP) approach. Since each ILP run can only provide one solution, we obtain multiple solutions by altering the values of different average bit widths. Specifically, given a trained SuperNet ( e.g. , RestNet18), it takes less than two minutes to solve candidate SubNets. It can be implemented through the Python PULP package. Finally, these searched SubNets only need inference to attain final accuracy, which needs a few hours. This forms a Pareto optimal frontier. From this frontier, we can select the appropriate subnet for deployment. Specific implementation details of the searching process by ILP can be found in Algorithm 2.

A.6 The Gradient Statistics of Learnable Scale of Quantization

In this section, we analyze the changes in gradients of the learnable scale for different models during the training process. Figure 7 and Figure 8 display the gradient statistical results for ResNet20 on CIFAR-10. Similarly, Figure 9 and Figure 10 show the gradient statistical results for ResNet18 on ImageNet-1K, and Figure 11 and Figure 12 present the gradient statistical results for ResNet50 on ImageNet-1K. These figures reveal a similarity in the range of gradient changes between higher-bit quantization and 2-bit quantization. Notably, they illustrate that the value range of 2-bit quantization is noticeably an order of magnitude higher than the value ranges of higher-bit quantization.

454

455

456

457

458

459

460

461

462

463

464

465

466

467

468

469

470

471

472

1

2

3

4

5

6

7

8

9 1011121314151617181920

Layer

6

4

2

0

2

4

The gradient of weight scale

1e

2

8bit

1

2

3

4

5

6

7

8

9 1011121314151617181920

Layer

3

2

1

0

1

2

3

The gradient of weight scale

1e

2

6bit

1

2

3

4

5

6

7

8

9 1011121314151617181920

Layer

3

2

1

0

1

2

3

The gradient of weight scale

1e

2

4bit

1

2

3

4

5

6

7

8

9 1011121314151617181920

Layer

8

6

4

2

0

2

4

6

8

The gradient of weight scale

1e

2

2bit

1

2

3

4

5

6

7

8

9 101112131415161718

Layer

2.5

2.0

1.5

1.0

0.5

0.0

The gradient of activation scale

1e

3

8bit

1

2

3

4

5

6

7

8

9 101112131415161718

Layer

3

2

1

0

1

The gradient of activation scale

1e

4

6bit

1

2

3

4

5

6

7

8

9 101112131415161718

Layer

1.2

1.0

0.8

0.6

0.4

0.2

0.0

0.2

The gradient of activation scale

1e

3

4bit

1

2

3

4

5

6

7

8

9 101112131415161718

Layer

4

3

2

1

0

1

The gradient of activation scale

1e

3

2bit

1

2

3

4

5

6

7

8

9 10111213141516171819

Layer

6

4

2

0

2

4

6

The gradient of weight scale

1e

2

8bit

1

2

3

4

5

6

7

8

9 10111213141516171819

Layer

4

3

2

1

0

1

2

3

The gradient of weight scale

1e

2

6bit

1

2

3

4

5

6

7

8

9 10111213141516171819

Layer

4

2

0

2

4

The gradient of weight scale

1e

2

4bit

1

2

3

4

5

6

7

8

9 10111213141516171819

Layer

2

1

0

1

2

The gradient of weight scale

1e

1

2bit

1

2

3

4

5

6

7

8

9

10 11 12 13 14 15 16

Layer

1.00

0.75

0.50

0.25

0.00

0.25

0.50

0.75

The gradient of activation scale

1e

3

8bit

1

2

3

4

5

6

7

8

9

10 11 12 13 14 15 16

Layer

6

4

2

0

2

4

6

The gradient of activation scale

1e

4

6bit

1

2

3

4

5

6

7

8

9

10 11 12 13 14 15 16

Layer

4

2

0

2

4

The gradient of activation scale

1e

4

4bit

1

2

3

4

5

6

7

8

9

10 11 12 13 14 15 16

Layer

4

2

0

2

4

The gradient of activation scale

1e

4

2bit

1

2

3

4

5

6

7

8

9 10111213141516171819202122232425262728293031323334353637383940414243444546474849505152

Layer

7.5

5.0

2.5

0.0

2.5

5.0

7.5

The gradient of weight scale

8bit

1

2

3

4

5

6

7

8

9 10111213141516171819202122232425262728293031323334353637383940414243444546474849505152

Layer

4

2

0

2

4

The gradient of weight scale

6bit

1

2

3

4

5

6

7

8

9 10111213141516171819202122232425262728293031323334353637383940414243444546474849505152

Layer

6

4

2

0

2

4

6

The gradient of weight scale

4bit

1

2

3

4

5

6

7

8

9 10111213141516171819202122232425262728293031323334353637383940414243444546474849505152

Layer

3

2

1

0

1

2

3

The gradient of weight scale

1e2

2bit

1

2

3

4

5

6

7

8

9 101112131415161718192021222324252627282930313233343536373839404142434445464748

Layer

2

1

0

1

2

The gradient of activation scale

1e

5

8bit

1

2

3

4

5

6

7

8

9 101112131415161718192021222324252627282930313233343536373839404142434445464748

Layer

7.5

5.0

2.5

0.0

2.5

5.0

7.5

The gradient of activation scale

1e

5

6bit

1

2

3

4

5

6

7

8

9 101112131415161718192021222324252627282930313233343536373839404142434445464748

Layer

1.5

1.0

0.5

0.0

0.5

1.0

1.5

The gradient of activation scale

1e

3

4bit

1

2

3

4

5

6

7

8

9 101112131415161718192021222324252627282930313233343536373839404142434445464748

Layer

6

4

2

0

2

4

6

8

The gradient of activation scale

1e

2

2bit

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: [TODO] Please refer to the Abstract Section and Section 1, where related material for the question can be found.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [TODO] [Yes]

Justification: [TODO] Although our proposed methods have achieved comparable results in multi-precision and mixed-precision, this paper has several limitations and improvements. (1) Due to time and computing resource constraints, our methods are only tested on common CNNs-based networks and aren't tested on ViTs-based networks. (2) For multi-precision, compact networks, e.g. , MobileNet, still have a big drop in 2bit. We will try to use per-layer or per-channel adaptive learning rate adjustment in the future. (3) For mixed precision, relying only on one-shot ILP-based SubNets search may yield a suboptimal solution. We further need to combine it with other efficient search methods, e.g. , genetic algorithms, to achieve global optimal.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best

judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: [TODO] We display index numbers wherever formulas and theoretical support are needed. For example, please refer to Section 3.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes] To ensure that our experimental results can be reproduced: (1) we describe the experimental training settings and algorithm pseudocode in detail in Section 4 and Section A.5, and (2) we also provide the code related to all experiments in this paper, allowing the community to improve and conduct further research.

Justification: [TODO]

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: [TODO] Our code are available at here and the data is open source dataset.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: [TODO] Our codes are available here and include all related training and test details.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: [TODO] Please refer to the Section A.6 in the appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: [TODO] Please refer to the Table 5.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: [TODO] We have read the NeurIPS Code of Ethics and conform to it.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: [TODO] Due to space limitations, this social impact aspect is not discussed in the main paper. This paper doesn't involve negative societal impacts including potential malicious or unintended uses. Our proposed methods aim to achieve an efficient and effective model compression technique to flexible adaptive different storage and computation requirements, which are beneficial to social advancement.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: [TODO] This paper doesn't have any high risk for misuse.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: [TODO] We conform to the CC-BY 4.0 license.

Guidelines:

734

735

736

737

738

739

740

741

742

743

744

745

746

747

748

749

750

751

752

753

754

755

756

757

758

759

760

761

762

763

764

765

766

767

768

769

770

771

772

773

774

775

776

777

778

779

780

781

782

783

784

785

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: [TODO] This paper does not release new assets

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: [TODO] This paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: [TODO] This paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses challenges in model quantization for deep neural networks (DNNs), focusing on optimizing quantization-aware training (QAT) across multiple bit-widths with weight-sharing. To this end, this paper introduces a novel quantization method that exploits the highest integer precision to achieve nearly lossless bit-switching, reducing storage without relying on full precision. Key contributions include: (1) Adaptive Learning Rate Scaling: A technique that dynamically adjusts learning rates for different precisions to address competitive interference and inconsistent gradient issues during one-shot joint training. (2) Double Rounding: An extension for one-step rounding quantizer in fixed-precision quantization to improve accuracy. Experimental results on the ImageNet-1K dataset show that the proposed methods surpass state-of-the-art approaches in both multi-precision and mixed-precision scenarios, achieving higher efficiency and accuracy.\n\nSTRENGTHS:\n- This submission is well-written, as well as with good figures in Sec.4.\n- The authors conduct extensive experiments on multiple datasets and multiple networks.\n\nWEAKNESSES:\n- Some analysis is missing. For example, I'm wondering whether the second rounding leads to more quantization errors, as the first rounding is used to produce INT8 weights and second rounding is then performed to quantize lower bit-width, the twice quantization is possible to cause more clipping errors and rounding errors, some analysis could enhance the strength of proposed methods. \n- Some designs should be further clarified, e.g., why ALRS is applied only for the scaling factors? Intuitively, weights of small bit-width is induced large gradient variance by STE, and thus the weights of small bit-width should also benefit from using smaller LR. \n- Fig. 1 is a bit confusing, some colored arrows are not well explained. \n- This works essentially lies in the research of mixed-precision quantization, so I think it is better to compare more MPQ (e.g., HAQ, DNAS, LIMPQ, etc) research in the Sec.4. Moreover, some recent papers on multi bit-width quantization are missed on the , e.g., [1] (PTQ-based) and [2][3] (QAT-based), which could be included into the Related Work. \n\n[1] Xu, Ke, et al. \"PTMQ: Post-training Multi-Bit Quantization of Neural Networks.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 38. No. 14. 2024.\n\n[2] Tang, Chen, et al. \"Retraining-free model quantization via one-shot weight-coupling learning.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024. \n\n[3] Zhong, Yunshan, et al. \"MultiQuant: A Novel Multi-Branch Topology Method for Arbitrary Bit-width Network Quantization.\" arXiv preprint arXiv:2305.08117 (2023).\n\nQUESTIONS:\nPlease refer to the weaknesses.\n\nLIMITATIONS:\nPlease refer to the weaknesses. Overall, this paper currently needs more experiments and analysis to reveal some designs are reasonable.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-05"}

comments: SUMMARY:
The paper proposes a QAT scheme to jointly optimize a single model with different precisions. The authors apply their scheme on various CNN-based models on CIFAR-10 and ImageNet datasets.

STRENGTHS:
1. The paper is well-written
2. The ablation study is strong in my opinion and they evaluate various aspects of their scheme

WEAKNESSES:
1. I think the main limitation of the paper is the models and datasets. I believe that the study should be done on larger models (LLMs for example) as a architecture goal. For example, the authors show that they do not save a FP32 master copy of the model in their scheme. However, ResNet style models (or MobileNet) are easy to fit in even moderate GPUs and I don't think FP32 master copy is a big problem in that case (please correct me if I'm wrong).

2. I couldn't find a source-code to reproduce the results of the paper in my side.

QUESTIONS:
1. The paper assumes a power-of-two relation between different precision. Do you have any theoretical intuition for this assumption? or this is just because of fast HW implementation?

2. What is the cost HMT in different networks? How can we compare it against the e2e runtime?

LIMITATIONS:
yes.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-08

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper discusses advanced methods in multi-bit model quantization. Specifically, this paper proposes a method for one-shot joint training of multiple precisions. To this end, the authors introduce a double-rounding quantizer that leverages the highest integer precision to achieve nearly lossless bit-switching while reducing storage requirements. Moreover, they also propose an Adaptive Learning Rate Scaling technique that adjusts learning rates dynamically for different precisions. Two proposed techniques mitigate the competitive interference between bit-widths caused by inconsistent gradients of different precisions during biased gradient estimation. They also extend their Double Rounding method to support one-shot mixed precision training and develop a Hessian-aware Bit-witdh sampling strategy. Experimental results on the ImageNet-1K classification task show that their methods outperform state-of-the-art one-shot joint QAT in both multi-precision and mixed-precision scenarios.\n\nSTRENGTHS:\n- Eliminating the costs of retraining for mixed-precision quantization is a meaningful and challenging topic.\n\n- The end-to-end experiments are sufficient, and the presentation is good.\n\nWEAKNESSES:\n- More uniquness analysis needed. The using of Hessian information seems a bit trivial, each layer's Hessian is just used to compare with the averaged Hessian trace. Firstly, as shown in recent zero-cost NAS research [1], the architectural proxies will be less effective as the training goes on, I'm not sure the Hessian information obtained on the initial full-precision model will remain useful as the quantization-aware training continues. Moreover, the sampling probability is modified with a simple ascending heuristic, which is not Hessian-aware. \n\n- Also applies here: the design of the double-rounding quantizer is similar to Bit-Mixer, Adabits, and ABN. Specifically, ABN also uses \n\n- ALRS needs further ablations. In ALRS, the authors use a fixed scaling ratio to bit-widths, e.g., 8-bit is 1, 6-bit is 0.1, and 4-bit is 0.01, the choice of these scaling factors still requires more ablation studies and discussions. \n\n- More comparisons needed. Since this paper adopts an ILP-based search algorithm to find optimal subnets, it is better to compare with these ILP-based mixed-precision quantization papers, e.g., [2] and [3]. \n\n\n\n[1] A Deeper Look at Zero-Cost Proxies for Lightweight NAS \n[2] Mixed-precision neural network quantization via learned layer-wise importance, ECCV 2022. \n[3] Hawq-v2: Hessian aware trace-weighted quantization of neural networks, NIPS 2020.\n\nQUESTIONS:\n- How do the authors perform KD for the proposed method?  Is an external teacher used or only distilled from the highest precision with the in-place distillation?\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a bit-switching quantization method using Double Rounding, which applies rounding twice to achieve nearly lossless switching without storing a full-precision model. They also introduce Adaptive Learning Rate Scaling (ALRS) to adjust learning rates dynamically across precisions, ensuring consistent quantization updates. Additionally, they develop Hessian-Aware Stochastic Bit-switching (HASB) for one-shot mixed-precision training, optimizing bit-width distribution based on layer sensitivity, thus eliminating retraining stages.\n\nSTRENGTHS:\n1. ALRS heuristic can help practitioners who wish to train mutli-precision model\n\n2. Authors made extensive experiments on vision models and compare to previse methos\n\n3. Most sections are well written\n\n4. Code is given\n\nWEAKNESSES:\n**Novelty** is limited and I am not highly motivated that the problem is important.\n\n1.\tThe main contribution is to not same 32bit weight and different quantization parameters but only the high bidwith using a pretty straightforward idea of double rounding during training\n2.\tThe ALRS is based on observation and heuristic to fix it. It is nice and helps for when trying to use 2 bits as well. Yet, I am not sure it is important for methods that don’t use the double rounding.\n\n**Motivation**\n\n3.\tSince we usually don’t switch models based on data I am not sure why this is important. Do we really have edge device that switch on a daily base model precision and thus need to store in small local memory the 32bit model? Can you elaborate why and where multi precision is really important.\n\n4. No results on more recent models (LLMs)\n\nQUESTIONS:\nI don't understand why you claim the ALRS method was inspired by LARS. The only similarity seems to be the name. Can you explain the connection?\n\nCan you provide a scenario where your method is particularly important?\n\nYou state that “if different precision losses separately compute gradients and directly update shared parameters at each forward process, it attains better accuracy when combined with our ALRS training strategy.” However, this involves updating the gradients four times more frequently, which is inefficient (4x backward pass and optimizer). This seems equivalent to small batch versus large batch training. Have you considered simply increasing the learning rate with the conventional multi-precision training approach? It might achieve the same results.\n\nIn Table 4, have you run the same number of forward/backward passes with the {8, 6, 4, 2} and {4, 3, 2} bit-widths? Since you have only three bit-widths in the latter, you might need to run 4/3 more iterations to ensure the total number of updates is the same. Have you tried that?\n\nLIMITATIONS:\nThe authors partially discuss limitation", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}