Robust and Efficient Quantization-aware Training via Coreset Selection
Authors: Xijie Huang
Zechun Liu
Shih-Yang Liu
Kwang-Ting Cheng
Venue: Accepted by TMLR ()

Abstract: Quantization-aware training (QAT) is a representative model compression method to reduce redundancy in weights and activations. 
However, most existing QAT methods require end-to-end training on the entire dataset, which suffers from long training time and high energy costs. In addition, the potential label noise in the training data undermines the robustness of QAT. We propose two metrics based on analysis of loss and gradient of quantized weights: error vector score and disagreement score, to quantify the importance of each sample during training. Guided by these two metrics, we proposed a quantization-aware Adaptive Coreset Selection (ACS) method to select the data for the current training epoch. We evaluate our method on various networks (ResNet-18, MobileNetV2, RetinaNet), datasets(CIFAR-10, CIFAR-100, ImageNet-1K, COCO), and under different quantization settings. Specifically, our method can achieve an accuracy of 68.39\% of 4-bit quantized ResNet-18 on the ImageNet-1K dataset with only a 10\% subset, which has an absolute gain of 4.24\% compared to the baseline. Our method can also improve the robustness of QAT by removing noisy samples in the training set.

Published in Transactions on Machine Learning Research (08/2024)
Efficient and Robust Quantization-aware Training via Adaptive Coreset Selection
Xijie Huang', Zechun Lin Shih-yang Lin', Tim Kwang-Ting CHENG!
Hong Kong University of Science and Technology (HKUST), Meta Reality Lab
shuangbs@connnet. hk, zechamliu@metta. com, sliuav@connect, ust. hk, timcheng@ust.hu
Reviewed on OpenReview: https: // openreview. net/ f id= 4c2422226
Abstract
Quantization-aware training (QAT) is representative model compression method to reduce redundancy in weights and activations. However, most existing QAT methods require end- to-end training on the entire dataset, which suffers from long training time and high energy costs. In addition, the potential label noise in the training data undermines the robustness of QAT. In this work, we show that we can improve data efficiency and robustness with a proper data selection strategy designed specifically for QAT. We propose two metrics based analysis of loss and gradient of quantized weights: error vector score and disagroement score, to quantify the importance of each sample during training. Guided by these two metrics, we proposed quantization aware Adaptive Coreset Selection (ACS) method to select the data for the current training epoch. We evaluate our method on varions networks (ResNet-18, MobileNed RetinaNet), MalaastssCIFARRUUS CIFAR-100, ImageNet-1IK COCO), and under different quantization settings. Specifically, our method can achieve an accuracy of 68.39% of 4-bit quantized ResNet-18 on the ImageNet-IK dataset with only 10% subset, which has absolute gain of 4.24% compared to the baseline. Our method can also improve the robustness of QAT by removing noisy samples in the training set..
1
Introduction
92
42
38
Pall
ELIN Score
Test Erron
36
Fargetinng
90
ELIN Scote
Test Error(%)
34
Maderalc
Accuracy
88
ACS (Cun))
36
CCCasterroody
32
ACS (Chrrs)
86
30
30
84
18.5 GPI boun
GPI
28
GPI hours
0.1
0.2
0.3
0.4
0.5
1.0
0.5
0.2
Data Fraction
Data Fraction
Data Fraction
Figure Left: Data scaling curve for 4-bit quantized ResNet-18 on the ImageNet- dataset. Our ACS significantly reduces test error using the same training data fraction compared baselines. Middle: Accuracy of 2/32-bit bit quantized ResNet-18 trained on CIFAR-10 with 10% random label noise. Our ACS is the only method to outperform the full data training performance by effectively removing noisy samples. Right: Test error for quantized ResNet-18 with different data fraction and the same GPU training hours. Under the same training budget, QAT with smaller coreset selected by ACS outperforms full data training.
Deep learning models have achieved remarkable achievements across various applications, including computer vision (Krizhevsky al.] 2012 He et al. 2016 Tan & Le 2019 Kirillov al. 2023 and natural language processing (Kenton & Toutanova] 2019
Yang al. 2019: Conneau & Tample 2019 Touvron et al. 2023

---

Published in Transactions on Machine Learning Research (08/2024)
T,S,E,
(0)
Quantization-awar training on 7 (0)
(R)
Quantization-aware truining T (R) Train for epochs
Selection
Selection
Repeat E/R times
Train for R epochs
QAT with Adaptive Coreset Selection
Selection at iteration
Top 5% sample selected
Hard label
((uf,xx)
devs p(w,,x) y
Quantized Student
Sorting
Weighting coefficient ((L)
Importance
Input Sample x for Selection
d acs for selection
plw,,, x))
dus p((.). P,(1,
Real-value Teacher
Figure 2: An overview of the Adaptive Coreset Selection (ACS) for efficient and robust QAT.
The outstanding performance of these models can attributed to their large mumber of parameters and the availability of largo-scale training data. For example, GPT-3 (Brown et al. 2020) boasts 175 billion parameters and the pre-training is performed on the dataset with more than 410 billion tokens. Neural scaling law Kaplan et al. (2020); Alabdulmohsin et al. (2022) is explanation of this size expansion that empirically demonstrates test loss of a model follows power law to the model size, training dataset size, and amount of computation. While neural scaling law guides the expanding usage of large models, the large model size and training set scale have become the most significant challenges for the training and deployment of deep learning models, especially on edge devices with computation and storage constraints.
Many model compression methods have been proposed recently address these challenges and enable the effective deployment of deep learning models. These model compression techniques include quantization (Zhoul
2016 Choi et al. 2018 Esser et al. 2020 Bhalgat et al. 2020),, pruning (Liu et al.) 2017 2018 Molchanov et al.
et al. 2019 Liu et al. 2019), knowledge distilaation (Hinton et al. 2015 Park et al. 2019 Shen & Xing
2021], and compact network design (Howard et al.) 2017 Pham et al.) 2018). Among the aforementioned
methods, quantization methods have been the most widely adopted because they have the advantage of promising hardware affinity across different architectures (Judd et al. 2016 Jouppi et al. 2017 Sharma et al, 2018). To minimize the performance gap between the quantized and full-precision models, quantizatiin-awwe training (QAT) is often utilized. Although QAT has improved the inference efficiency of the target model, it is computation-intensiin and requires more time than full-procision training. While previous QAT methods assume ideal case in which computation resources unlimited, aim to research more practical case when the cost of QAT should be considered.
Coreset selection techniques aim to mitigate the high training cost and potential negative influence of label noise to improve data efficiency and robustness for full-precision training. Specifically, coreset selection methods leverage the redundancy training datasets and select the most informative data build a coresot. for training. One recent research Sorscher et al. (2022) has pointed out that coreset can be a solution to beat the scaling law Previous methods select data based feature (Agarwal et al., 2020 Sener & Savarese 2018 error Paul et al. 2021) gradient (Killamsetty ct al. 2021I. While those methods can achieve notable efficiency improvement for full-precision training, their effectiveness on low-precision QAT has not. been explored before. To utilize coreset selection met hods to improve the data efficiency and robustness of QAT, the characteristics of quantized weights must considered the design of methods. Existing full-precision methods are not designed for quantization, and severe computation overhead during selection may hinder the application of QAT. Therefore, coresct selection method specifically designed for QAT is required.
al. 2021a). and decision boundary (Margatina
2

---

Published in Transactions on Machine Learning Research (08/2024)
In this work, propose the coreset selection specifically for QAT and find that we can clevate the data efficiency and training robustness. We start by analyzing the impact of removing one specific sample from the training set during QAT and identifying that the error vector score (EVS) is a good theoretical approximation of the importance of cach sample. Based on the common practice of utilizing knowledge distillation during QAT, we also propose the disagreement score (DS) measuring the prediction gap between the quantized model and full-precision models. Based on these metrics, we propose a fully quantizationaavare adaptive coreset selection (ACS) method to select training samples that fit the current training objective. An overview of the proposed method is shown in Fig.
We demonstrate the superiority of our ACS in terms of effectiveness and efficiency on different networks (ResNet-18, MobileNet-Vi V2, RetinaNet), datasets (CIFAR-10, CIFAR-100, ImageNet-IK, COCO), and quanti- zation settings. For 2-bit weights-only quantization of MobileNetV2 on the CIFAR-100 dataset, QAT based on our ACS can achieve a mean accuracy of 67.19% with only 50% training data used for training per epoch. For 4-bit quantization of ResNet--88 on the ImageNet-1K dataset, our ACS can achieve top-1 accuracy of 68.39% compared to the 64.15% of random selection when only 10% training data is selected for the training of every epoch. In summary,
contribution can be summarized as follows:
We are the first to investigate the data efficiency and robustness of quantization-aware training and empirically observe that the importance of different data samples varies during the training process.
We propose two metrics: error vector score and disagreement score, to quantify the importance of each sample based on theoretical analysis of loss gradient.
We propose quantization-awwn Adaptive Coreset Selection (ACS) method, which adaptively selects informative samples that fit current training epochs and prune redundant noisy samples.
We verify ACS on different network architectures, datasets, and quantization settings. Compared with previous methods, ACS can significantly improve both the data efficiency and robustness of quantization aware training shown in Fig. [
2
Related Work
Quantization Quantization methods are powerful tools for improving the efficiency of model inference.
The core insight is replacing full-precision weights and activations with lreer-pre representation. The quantization methods can be classified as cinatiizationawarre training (QAT) (Zhou et al. 2016 Esser et al,
2020 Bhalgat et al.] 2020 Huang et al., 2022 Liu et al. (022cb and post-training quantization (PTQ) (Nagel
et al. 2020 Fang al.] 2020 Wang et al.) 2020 Xiao et al. 2023 Liu al.) 2023b Lee al. 2023 based
on whether to retrain model with quantized weights and activations or start with a pre-trained model and
directly quantize it without extensive training. Based on the characteristics of quantization intervals, these methods can be categorized into uniform and non-uniform quantization. While uniform quantization (Zhou et al. 2016 Choi et al. 2018 Esser et al.] 2020) with uniform interval are more hardware-friendly and efficient, non-unifore quantization [Miyashita et al. 2016 Zhang et al. 2018 Li et al. 2019b Yvinee et al., 2023), due to the flexibility of representation, can minimize the quantization error and achieve better performance than uniform schemes. While previous QAT methods assume an idcal case for unlimited computation resources, we research more practical case to conduct QAT with a fixed computation budget. in this work.
Learning from Noisy Labels The noisy labels are defined as unreliable labels that are corrupted from ground truth. The noise can stem from non-expert labeling malicious label-ftipping attack (Xiao et al. The estimated ratio of noisy labels in real-world datasets ranges from 8.0%-38.5% according to [Song 2022). To improve the robustness of lcarning for noisy labels, the solutions include designing robust architectures that are alluned to label noise (Yao et al.] 2018 Lee et al.) 2019), label correction (Tu et al., Li et al. 2023 2024), robust regularization (Wei et al.) 2021), and robust loss design (Huang et al., One important category that enables robust noiso-tolerant training is sample selection (Paul et al.
2012
ct al.
2023
2023
2021
Xia et al.) 2023a] Park et al. 2024). While the effectiveness of sample selection methods has been verified on full-precision deep learning, their performance on QAT is under-explored.
3

---

Published in Transactions on Machine Learning Research (08/2024)
Coreset Selection Coresct selection targets improving the data efficiency by identifying the informative
training samples. Previous works can be mainly classified into the following categories: Geometry-basee Methods: These methods assume that samples close to each other in the feature space similarly influence the training. These redundant data points should be removed to improve efficioncy. Some representative
works include Contextual diversity (CD) (Agarwal et al. 2020 k-Center-Greed] (Sener & Savarese 2018):
Sorscher et al. (2022), Moderate Coresct (Xia et al., 2023b), and Herding (Welling) 2009). Decision Boundary-based Methods: These methods select samples distributed near the decision boundary. These samples are difficult for the model to separate. Representative works include Adversarial Deepfool (Ducoffe &
Precioso 2018 and Contrastive Active Learning (CAT.) (Margatina et al. 2021). Gradient/
Cradient/Brroo--assee /Error-based Methods: These methods assume the samples are more informative if they contribute more to the error or loss during the training. This calegory includes EL2N (Paul el al. 2021), CRAIG MMrzasoleiman et al., 2020), GradMatch Killamsetty et al. 2021a Forgetting events (Toneva et al. 2019), and AdaCore (Pooladzandi
ct al.
2022 Optimization-basct Methods: These methods formulate the coresct sclection as a. bilevel optimization problem. The outer level objective is the selection of samples and the inner level objective is the optimization of model parameters. The representative works include Borsos et al. 2020 Glister (Killamsetty
et al. 2021b) Zhou et al. (0022), and Retrieve (Killansetty et al.) 2021ch.
These methods perform well in some scenarios of full-precision training(specifee subset fraction, specific outlier distribution, etc.). However, none are verified on quantization settings or consider the requirements for QAT. As shown in Fig. [] the majority of these methods cannot even outperform random sampling in the QAT settings. Morcover, these methods either require early training of the target model for several
epochs (Toneva et al.] 2019; Paul al. 2021 or time-consuming search (Sener & Savarese 2018) and optimization [Killamsotyy ct al. 2021b which leads to heavy computation overhead during QAT.
3 Importance of Each Sample in QAT
One underlying assumption of data scaling law is that all samples in the training set are equally important. However, many previous coreset research (Soschher et al. (2022) have proved that information within different samples is different for full-precision training, and the quantization setting has not been verified yet. In this soction, we will first introduce quantization-awarr training (QAT) and derive the gradient under cross-cntropy and SGD in Sec. 3.1. We then analyze the change of gradient when specific sample is removed from a training batch to investigate the importance of each training sample in Sec. 3.2 We propose approximation of this gradient change considering the prediction error without introducing any memory overhead. We further prove that less training data is required when knowledge distillation is applied to QAT in Sec. 3.3 Another metric quantifying sample importance based on the prediction gap between the quantized student. and full-precision teacher model is also introduced in Sec. 3.3
3.1 Preliminaries of QAT
During QAT, the real-value data v" is converted to b-bit quantized representation e = qv") by the quantizer Φb- Given the scale factor s of the quantizer, the number of positive quantization levels Qp, and the number of negative quantization levels QN we can have the quantizer qo as
v = q((")) = s LcN
QQ.
(1)
where 1.1 is the rounding function that rounds the input to the nearest integer, clip(U, Flow, Fhigh) return v with all value below Flow set to be Flow and all values above Thigh set to be Thigh. For the unsigned quantization, =, Qp = 2d While for the quantization of signed data, QN = 2 Qp = 2 1. To solve the problem that the gradient cannot back-propagate in Equation during QAT, the strough estimator (STE) (Bengio et al. 2013) is utilized to approximate the gradient. In the back-propagation of QAT with STE, the gradient of the loss L with respect to the real-value data of is set to be
0c ac
Ov =
dug 1 1-Q /s5Qp:
(2)
4

---

Published in Transactions on Machine Learning Research (08/2024)
where 1 is the indicator function that outputs 1 within the quantization limit and 0 otherwise. Note that we will apply clipping in the beginning of QAT, the gradient of quantized data can be seen as approximately equal as the gradient of real-value data.
The training set of QAT is denoted as T= { B6) where imput I2 E R are vectors length d and y E {0, 13M are one-hot vectors encoding labels. The neural network to be trained denoted as f'w",, a)), where real-value weights are w". We use cross-entropy loss L(p.y) = -m y(m) logp(m) in the QAT and p(w", . is a probability vector denoting the output of the quantized neural network ((w9,x). Stochastic gradient descent (SGD) used for the optimization. Suppose the real-value weights at iteration are ft and the batch of input samples Ti-1 C T, the weights are updated following
: = wi-1
Σ
gt-1(z,
(3)
(x.
where 1) denotes the learning rate and 9x-1 = V ( the gradient of cross entropy loss.
3.2 Error Vector Score
To look into the importance of each sample {(T), vi),, we measure the difference of the expected magnitude of the loss vector on the training set T and another training set which only removes one specific sample T'= \\ {(x, v)} For simplicity, we approximate all the training dynamics in continuous time. Based on the chain rule and STE of quantization, we have the change of loss L time on sample (ær y) of batch T
dC
= ge((z,)) du? v)
dt ge (x( v) dwf dt
(4)
di (Z.)..T.
According to the discrele-time dynamic of real-value weights in Eq. we have dw? de w - wi-1 -1 (t))) (2, y). To measure the contribution of a specific sample (Ia, 84), we measure the change of loss with and without the sample. For a given data batch T, if the sample (21, 91) & TT, we can ignore the change at (a.)).. For any sample (1z, E T.j F i in the same batch, the importance I(x,, v.) is measured as
dC
dc
I(z., Mi) = dt (2z.w)).T
(5)
dt (1g.))))T)
According to the chain rule, we have
dC dt (a:aa)
(app((/{555))000 duf duf dw? dwf
(6)
According to the characteristics of STE in Eq. Following the updating rule of weights in liq. 3 the gradient
duy
= 1 holds for all input within the clipping range. dw -1 En It-1 .v"). The only
dw;
dt
difference of the training set T and T is the existence of sample (I,, 31). Thus, we have
dC du! (x,,0s))
duf
I(at, 11
dw? dt dL
dt (
gi-1 91-1 vi)
dw?
dC We use duy as a simplification of dC(p(w].xx).as which is only dependant on the current training sample duy
(x), vy) but not dependant the sample (z,, y1) that is removed from batch T. Since learning rate n and the gradient of loss w.r.t. quantized weights de can be seen as constant given T, the importance of the sample dwy
(z, y1) in this batch T is only related to the gradient norm of cross-entropy loss of this sample [1 Bill- The examples with a larger gradient norm expectation have more influence on the supervisod training of other data, which means they are important for QAT and should be included in the coreset. We can select data with high importance by sorting by the norm of gradient, which is also covered in previous works Paul
5

---

Published in Transactions on Machine Learning Research (08/2024)
et al. 2022 However, storing the loss gradient for comparison requires extra memory and is hard to transfer between network architectures. We approximate the norm gradient with the norm of the error vector, which is defined as follows.
Definition 1 (Error Vector Score) defined to be dævs [lp(w] x) yll..
The error vector score of training sample (z,y) at iteration id
For any input x E IRd, gradient norm (19((2,)) is a non-random score. We take its expectation over random minibatch sequence and random initialization to get the expected gradient norm (l]ge y)ll, which can also be indicated as
M
dCo(p(wf, x).v) vi af!mm) df("m)
E (1l
(8)
dwf
where dr(mm) du?
denotes the gradient of m-th logit weights. Since we are using cross-entropy as the loss function, the gradient of loss C on the m-th logit output s!m) follows dLilp(u?.))))) :(w[,, xr) (m))
df!")
Previous works (Fort et al. 2020; Fort & Ganguli 2019) empirically observe that 4f "!!)
dwž are similar across
different logits m E . and training sample (æ, y). Thus, selecting samples based on the error vector score
devs will be more likely to identify the samples with larger gradient norm. Different from the previous method (Paul et al. 2021) also leveraging the error metrics, no early training required, and only use the current quantized model prediction p(wl, æ)) during QAT.
3.3 Disagreement Score and Knowledge Distillation
Intrinsically, a quantized classification network should Iearn an ideal similar mapping f from input sample x to the output logits f(w, x) as a full-precision network, and the gap between the quantized prediction p(w", x) of student model and real-value prediction pr(w", of teacher model needs to be minimized. Based on this insight, knowledge distillation (KD) is widely used during QAT with a full-precision model as the teacher, which can also be seen in previous works (Polino al.) Huang et al.j [022] Mishra & Marr 2018 Liu et al., 2023ah. The loss function is designed to enforce the similarity between the output of the full-precision teacher and the quantized student model as
LKD
(m) PT "w'"", (w", mx)) "o"",;))og(("" log(p! xx))
(9)
N
where the KD loss is defined as the cross-entropy between the output distributions pr of full-precision teacher and quantized student on the same input æ but different weights representation w" and w", 2; is one of the input samples from the training set. m and N denote the classes and total training sample numbers, respectively.
Note that this process can be regarded as the distribution calibration for the student network and one-hot label is not involved during QAT. Since the loss function used for knowledge distillation is still cross-entropy, and we still assume we use SGD for the optimization, most conclusions in Sec. 3.2 still hold by replacing the one-hot ground truth label y with full-precision teacher's prediction pr(w),x), Thus, we propose the disagreement score as follows.
Definition 2 (Disagreement Score) The disagreement score of a training sample (z, v) at iteration t is defined to be dos = lip(u(, .r (w][,)))
The core difference between error vector score devs and disagreement score dps is the target label. While dgvs uses one-hot hard labels, the dps uses the distilled soft labels. We empirically notice that the data nceded for the training reduced when knowledge distillation applicd, which is helpful for our selection with a small data fraction. We further demonstrate the advantage terms of training data requirements using the soft label based on Vapnik- Chenronnnerssssiss theory (Vapnik 1999), which decomposes the classification

---

Published in Transactions on Machine Learning Research (08/2024)
error R(f.) of a classifier f. E Fs as
[FFcc
R(f.) R(F) so
+ Es:
(10)
where (.) denotes the asymptotic approximation and 6, is the approximation error of F, with respect to F. f E FF denotes the real target function. ||c is the VC-Dimension of the function class measuring its capacity. is the number of total training data. 1 S 0g S is indicator measuring the difficulty of the problems. For non-separable and difficult problems, 0t, T. which means the classifier learns at a slow rate of 0(n=)). For separable and easy problems, a, = 1, indicating the classifier learns at a fast rate. In our setting, the quantized model fg Fg diroctly Iearns from the hard labels, the difficulty of the problem high, and we assume 0q 2 we have
(FFAl)
|Fale vi
-(f+)-((F) 0o
E
(11)
where Eg is the approximation error of the quantized model. However, if we first train the full-precision teacher model fr E Fr and then utilize knowledge distillation to learn the representation from the teacher, the difficulty of the learning becomes easier, assuming = 1, we have
R(G) - R() So Frlc
Frlc 10
ter, R(J,) - RO-) SO
+ Eqri
(12)
where the Egr denotes the approximation error of the F, with respect to F and approximation error of the Fg with respect to Fr. respectively. Compared to the direct learning quantized model from the hard label shown in Eq. 11 the knowledge distillation with real-value teacher fr yiclds the classification error as follows:
Frlc + +0
(Fale) n2
|Folc ( (Frlc) )
B(G) - RUS
+
10
+ Er + Egr.
(13)
n
Following previous studies on knowledge distillation (Lopez-Paz et al. soft labels contain more information than hard labels for each sample. Thus, we have E, + Egr S Eq and Combining these two inequalities, we have the inequality vn
2015 Mirzadeh et al. 2020 the
O [Flo+
00
FFlc
o Folc + + + Eqr So 12
|Folc vn
+ Eq:
(14)
which means when the number of training samples n is the same, the upper bound of classification error based on the soft label is lower. When we want to achieve the same upper bound of classification R(fa) R(f) using these two techniques, learning from soft labels requires less data. This is the core reason why we use knowledge distillation and disagreement score dps to select the coreset.
4 Adaptive Coreset Selection for QAT
In Sec. 3 we propose to use dgvs and dps to select the coreset for QAT. While dps could help select those samples that produce large performance gaps between quantized and full-precision models, devs targets more at the error of quantized prediction. These two metrics cover different characteristics of training data, and we need both to improve the diversity of our coreset. For different stages of QAT, different metrics should be considered when selecting samples that fit the current training objective. Previous research (Kin et al. 2019 has shown that QAT should start with the hard label to help better initialization for the quantized model and then use soft labels to guide it to better local minima. In light of this, we propose Adaplive Coreset. Selection (ACS) for QAT to select the important samples considering current training epoch t, dgvs, and dps adaptively.
For the given current training epoch i and the total training epoch E, we propose a cosine annealing weights coefficient B(t) cos( () to consider two metrics simultaneously and balance between them. The final selection metric is a linear combination of devs and dps as follows:
dAcs(t) (t) = ((t)dgvs(() + -=(t))dps(())
(15)
7

---

Published in Transactions on Machine Learning Research (08/2024)
As we have ((0) = 1 and ((E) = 0 in the early stage, the selection is mainly based on dgvs. When the quantized model converges, we focus more on the dps in later epochs. We perform coreset selection every R epoch, where R is determined before training. The pseudo-code for our ACS algorithm is shown in Alg, |
Algorithm 1 Adaptive Coreset Selection for QAT
Training dataset T [[ u:) Real-value network with weights W Coreset data fraction per epoch S, Total training epochs E, Sclection interval R, Initial coresct TAcS (t) = 0
Input:
Output: Quanlized network with weights Wa
Initialize quantized weights W9 from W following Eq.
for E [0..... ..., E 1] do then
if t%R-
((t) cos( (5
for (I,, vi) E T do
dgvs(2,, t) = llp(W: t x,) - Vill2
d 1) = llp(W: t x1) - Pr (W, x1) ||
dacs( t) = (I)) + (1- ((t)
end for
Sort dacs (z,, t); select top 5% samples to replace (tcs(t)
else
(1) + (tcs(t-1)
end if
Train Wa on (t)) following Eq. end for
There are two major advantages of our quantization aware ACS method. The first lies in the adaptation to the training phase when knowledge distillation is applicd. As soft labels retain more information about the target than hard labels, we should encourage the quantized student model to learn sequentially on hard labels first and soft labels then. This implicit learning hierarchy is observed in QKD (Kim et al. 2019) and is named "self- leanning and "tutoring". With the proposed ACS fully aware of this hierarchy, the selected coreset helps stabilize the training and guarantee faster convergence. The second advantage is the diversity of training data. More samples could be covered in the coreset of different epochs, and the coverage of the original full dataset contributes to the convergence of a more robust model. Note that only when the optimal data sequence and high training sample diversity are achieved simultancously, the performance of QAT will be significantly better. We demonstrate in the Appendix. D that even when all data are covered but the order is random, the accuracy of our quantized model will be negatively influenced.
5 Experiments
Datasets and networks The efficiency experiments conducted on CIFAR-100 (Kiizheskyy et al. 2009 and TmageNet-1K3 1K dataset (Deng et al. 2009). We evaluate MobileNetV2 (Howard et al. (017)] on C-FAR-100
and evaluate ResNet-18 (He et al.] 20166) on the ImageNet-1K dataset. The width multiplier is set to be 0.5
for VobileNetV2. We further provide experimental results of quantized RetinaNet Lin et al. (2017) on MS
COCO object detection benchmark (Lin et al. 2014). The robustness experiments are conducted following the setting of Paul al. (2021).
Baselines We choose multiple coreset selection methods from different categories as our baseline. The selected methods include: Random Sampling, EL2N-Score (Paul et al. 2021 Forgetting (Toeva et al.) 2019), Glister (Killamsetty et al. (021b), &CenterGreedy (Sener & Savarese) 2018) Contextual Diversity
(CD) Agarwal et al. 2020), Moderate Coreset (Xia et al. 2023b). For methods involving early training, we
Note that only verify the EL2N-score instcad of Gradient norm (CradN) score Paul
2021 based the
TMLR reproduction from Kirsch 2023) observing that the inconsistency of GraNd at initialization will results sub-optimal performance even compared with random selection.
8

---

Published in Transactions on Machine Learning Research (08/2024)
set training epochs as 5. Note that comparing slaptive-based and non-adaptive-based methods is fair and the common practice of previous research PPooladzandi et al. 2022 Killamsetty et al. 2021c).
Training and selection details For Vo train the network for 200 epochs using learning rate of 0.01, weight decay of 5e-4, batch size of 512, R 20, and SGD optimizer. For ResNet-18 on ImageNet- Ne,, we train the network for 120 epochs using a learning rate of 1.25e-3, no weight decay, batch size of 512, == 10, and Adam optimizer. We use the quantization method and full- pelliprcciioon model following LSQ+ Bhalgat et al. 2020). All the experiments were carried out on 2 NVIDIA RTX 3090 GPUs. As we notice that the results on the ImageNet-1K dataset will not vary significantly across different runs, experiments of each setting are only performed once. For CIFAR-100 experiments, each experiment of different seltings is knowledge distillation with the corresponding full- full-precision model as the teacher in all experiments, regardless of the method and dataset fraction.
repeated 5 times. For fair comparison,
Table 1: Comparison of applying different coreset selection methods 2/32-bit bit quantized MobileNetV2
CIFAR-100 dataset with various subset fractions. When full data selected (S 100%), the mean accuracy and standard deviation is 1..9%. The first and second best accuracy is shown in bold and underlined, and the performance improvement over the previous SOTA is marked in blue.
Mothod/ Mothod/Fraction Fraction
10%
S= 20%
$ 30%
40%
$ 50%
Rondom
62.310..9
64.1110.7
65.2:0.3
65.610.6
66:2:10.9
EL2N Score Paul ct al.
2021
63.020.6
64.011.6
65.6=0.8
64.620.8
65.3:10.4
Forgetting Toneva el
2019
60.7+0.3
63.1+0.6
65.0=0.8
65.3+0.8
66.0+1.22
Glister Killamsetty ef al.
2121b
56.5+11.2
60.4+1.0
61.3+0.6
63.0+1.0
64.8+0.9
GCentyr Sener & Savarese
2018
60.2::0.8
62.74:0.
63.8+0.7
64.840.6
66.3+-1.1
CD (Agarwal al. 2020
60.3+00.8
62.6+00.9
64.0=1.0
64.8=0.4
65.3:00.4
Moderate Xia el
20236
57.9+0.33
60.4+:0.7
62.8=0.6
63.6+0.8
64.5+:1.6
Ours
63.710.88 (10.7) 65.910.77 (t1.8)
66.4-0.8 (?0.8)
66.910.5 (11.3)
67. 07.2:00.5(00.9)
Table 2: Comparison of different methods of 4/4-bit quantized ResNet-18 on ImageNet-IKK When full data are selected (S = 100%), the accuracy is 72.46%. The first and second best accuracy is shown in bold and underlined, and the performance improvement over the previous SOTA is marked in blue.
Method/Fraction Fraacton (%)
10%
S 30%
50%
S 60%
S 70%
80%
Raudom
64.15
08.53
70.49
70.94
71.06
71.96
EL2N Score (Paul sl.
2021
61.71
67.31
70.14
70.89
71.54
71.88
Forgetting (Toeea et
2019
63.09
67.77
70.14
71.00
71.36
71.82
Glister Killasetly el
bb
63.25
08.91
70.92
71.39
71.93
12.22
&Cennerrrredy (Sener & Savirose
2018
62.98
68.56
70.35
71.13
71.59
71.96
CD Agarwal el al 220
63.22
68.74
70.90
71.27
71.78
72.11
Moderate Xin et a.. 2023b
62.39
68.06
70.43
70.62
71.56
71.99
Ous
68.39 (14.24)
71.09 (12.15)
71.59 (10.67)
72.00 (10.61)
72.19 (10.26)
72.31 (10.09)
5.1 Benchmarking Previous Coreset Method
The comparison of the QAT Top-1 accuracy of MobileNetV2 on CIFAR-100 and ResNet-18 on ImageNet-1K is shown in Tab. [ and Tab. 2 We note that most previous methods cannot exceed random selection in our QAT setting, and the few surpass the bascline only on specific data fractions. This trend is also discussed for full-precision training by Kirsch (2023). For example, kCenterGreedy (Sener & Savarese 2018) shows satisfying performance when the subset size is large (50% for CIFAR-10, 70%/80% for ImageNet- 1K) but fails to demonstrate effectiveness on small coreset size. Our method outperforms state-of-the-art methods on all subset fractions S by great margin. Specifically, the RosNot-18 accuracy of 10% subset on ImageNet-1K using our method is 68.39%, achieving an absolute gain of 4.24% compared to the baseline method.
Efficiency analysis The detailed QAT training time of ResNet-18 on ImageNet- IK coreset and NVIDIA RTX 3090 GPUs with different methods is listed in Tab. When full data are selected (S 100%), the

---

Published in Transactions on Machine Learning Research (08/2024)
15
10
110
(a) Full Dataset
(b) Random S = 10%
(c) Ours S = 10%
Figure 3: Visualization of the loss landscape (Li et al. 2018 of 2-bit quantized MobileNet trained on the CIFAR-100 (a) full dataset, (b) 10% random subset, and (c) 10% ACS coreset.
training time is 62.3h. The only method with comparable efficiency to ours is the Moderate Coreset (Xia et al. 2023b), which only needs to perform forwarding on samples and can eliminate optimization or greedy searching. The results prove that our method can effectively reduce training time without incurring computation overhead by the selection algorithm.
Table 3: Training time (hours) of 4/4-bit quantized ResNet-18 on ImageNet-1K3 with various subset fraction.
Method Weethod/Fraction Fraction (%)
10%
30%
50%
60%
70%
80%
EL2N Score
12.7
23.8
39.1
44.1
49.9
56.0
Forgetting
12.7
23.9
39.5
44.7
50.2
56.6
Glister
13.8
23.0
39.9
46.6
58.0
66.8
&CenterGreedy
13.0
25.2
38.4
43.8
50.5
56,9
CD
24.5
37.1
42.5
48.9
55.0
Moderate
11.3
22.3
36.2
42.5
48.2
541.1
Ours (R=10)
11.2
20.7
36.1
42.2
48.0
53.7
Effect of adaptive epochs The choice of selection interval R is vital in our algorithm, as too large R will fail to help adaption and improve data diversity, and too small R will introduce too much computation overhead and undermine the efficiency. We apply grid search on R and empirically prove that R = 10 is optimal for our ImageNet-1K training. The accuracy and training time results are shown in Tab. 4 We can observe from the results that R = 10 achieves similar performance as R = 5 with a shorter training time. As no propaaroongttion of the gradient is involved during the computation of devs and dps, the computation overhead is acceptable under most settings with R > 5 compared with previous methods involving training.
Table 4: Analysis adaptive epochs R.
Table 5: Analysis on strategy B(t).
S=
10%
S=30%
S 5=50%
-0%
Fraction (%)
10%
30%
50%
70%
Fraction
Acc.
Time
Acc.
Time
Acc.
Time
Acc.
Time
fixed
67.95
70.83
71.21
72.01
=
68.8
12.7h
71.05
21.5h
71.47
37.0h
72.15
49.1h
linear
68.35
71.39
72.11
10
68.39
11.3h
71.09
20.7h
71.59
36.1h
72.19
48.0h
sqrt
71.44
72.15
R 20
67.58
11.0h
70.35
20.2h
71.42
35.5h
71.97
47 6h.6h
quadratic
68.
70.95
71.35
72.10
40
60.10
10.7h
69.86
19.8h
71.25
34.9h
72.00
47.2h
deys only
68.06
70.63
71.41
71.96
60
64.96
10.5h
69.27
19.5h
71.05
34.4h
71.93
46.9h
dos only
67.07
70.24
71.43
72.08
>
120
62.82
10.3h
67.62
19.2h
69.98
33.9h
71.37
46.6h
cosine
68.39
71.09
71.59
72.19
Ablation study and effect of annealing strategy As we propose two metrics for coresct selection:
devs and dps, it is important to analyze how should balance between them and the contribution of both
10

---

Published in Transactions on Machine Learning Research (08/2024)
metrics. We try the following different annealing strategies and settings: (1) fixed (3(t) = 0.5); (2) linear (4) quadratic (B(t) 1 ( (5) cosine (5(t) cos( 2 T)); (6) dEVs only (f(t) = 1); (7) dps only (3(t)= - 0): The results of coresct selection of 4-bit quantized ResNet-16 18 on ImageNet-1K are listed in Tab. B. The performance gap with different annealing strategies is close as they all follow the trends to use devs in the early epochs of training and dps in later epochs. Among all these annealing strategics, cosine anncaling is slightly better. When only one metric used for the selection, the performance will drop. We also notice that devs and dps are complementary devs works well with small data fraction and dps has better performance with large data fraction.
(8(t) g) (3) square root (8(t) =1-v -1-vE);; });
}):
5.2 Training with Label Noise
We would like to highlight another application of our method: identifying noisy labels in training data and improving the robustness of the QAT. The nature of advantage of method is to select important training examples for QAT and remove those low-quality redundant examples. This is especially useful when there is label noise in the training set (some examples are not correctly labeled). This application is proposed in previous coreset research Mirzasoleiman et al. (2020), but not verified on QAT. We follow the setting of Mirzasoleiman et al. (2020) to experiment on the QAT of ResNet-18 on CIFAR-10 with 10% randomized labels (labels of 10% of training samples are randomly re-generated). The ResNet-1 18 quantized to 2/32 for weights and activations. When the full dataset selected, the accuracy is 09.911 The comparison of the QAT Top-1 accuracy (%) of 2/32-bit quantized ResNet-18 on CIFAR-10 with 10% randomized labels is shown as follows in Tab. 6 Note that we select E2LN Score and GraNd Score for comparison they are the few coreset selection solutions among the methods we investigate in Tab. 2 claimed with the capacity to remove noise in the training set and improve training robustness.
Table 6: Top-1 Accuracy of 2/32-bit quantized ResNet-18 18 on CIFAR-10 with 10% random label noise
Method/ Method/Praction Fraction (%)
10%
20%
30%
40%
50%
Random
83.14+0.55
84.77+0.80
84.82+0.57
85.09++0.15
85.33+0.60
EL,2N Score
85.80+00.51
86.02+0.555
87.18+0.35
87.51+0.29
88.01+0.45
GraNd Score
85.71+0.30
85.96+0.155
87.10+0.55
87,44+0.20
87.94+0.29
Ours
88.3110.27
89.53:0.62
89.95-10.39
90.21::.39
90.23:10.18
As can be seen from the results in Tab. G our method outperforms all other selection baselines and even performs better than full-data training when coreset size S 2 30%. To quantitatively demonstrate the noisy sample pruming elfectiveness, we report the average recall for this noisy-sample detection problem. Noticeably, while the other pruned samples are in the correct label, they are still mostly redundant and can be pruned without significant performance degradation in QAT. From the results of recall in Tab 7 our method states most of the noisy samples. These results show that our method can successfully prune those samples with incorrect labels. In this case, actually achieve a *lossless acceleration" both the accuracy and efficiency are better than full-data training.
Table 7: The average recall of noisy sample identification on CIFAR-10 with 10% random label noise.
Method Nethod//Fractinn Fraction (%)
10%
20%
30%
40%
50%
EL2N Score
90.7%
80.5%
71.0%
60.2%
47.4%
GraNd Score
92.3%
81.7%
65.4%
41.0%
26.5%
Ours
97.9%
90.2%
84.1%
78.9%
73.5%
5.3 Object Detection Tasks
We further conduct experiments of 4/4-bit QAT of RetinaNet (Lin et al. 2017) with ResNet-18 backbone on the MS COCO object detection dataset (Lin et al.) 2014). We use the same selection metric for the RetinaNet training and use the output for the classification head the probability vector p. When there are multiple
11

---

Published in Transactions on Machine Learning Research (08/2024)
objects in one image, use the mean selection metric dacs of the probability output of all objects. The QAT method used is FQN (Li et al., 2019ah and we perform training on coco-2017-train for 100 epochs. We investigate two coreset fractions: 10% and 50% and use R=110. The results of mAP are listed in Tab. Our method outperforms the random selection baseline and state-of-the-an selection method Modorate Xia et al.
2023b significantly, which proves that our method effect on object detection tasks.
Table 8:
Comparision of performance with different methods of RetinaNct COCO benchmarks.
Method
Fraction
AP
Apts
Ap
Aps
ApM
Apt
Full
100%
28.6
46.9
29.9
14.9
31.2
38.7
Random
10%
21.4
39.8
22.4
7.5
24.3
27.4
EL2N Score
10%
20.7
37.0
19.9
8.4
25.0
28.1
Moderate
10%
22.0
37.8
20.4
8.4
25.0
28.1
Ours
10%
24.4
40.9
25.1
9.9
27.1
31.5
Random
50%
25.4
42.5
26.7
10.7
27.7
32.1
EL2N Score
50%
25.9
42.8
27.0
11.4
28.6
33.8
Moderate
50%
25.0
42.7
25.9
11.2
28.5
33.0
Ours
50%
26.7
44.0
27.8
12.1
30.0
35.1
5.4 Visualization and Analysis
We visualize the loss landscape [Li et al. 2018 of MobileNetV2 training on the full CIFAR-100 dataset, 10%
random subsot of CIFAR-100, and 10% coresct of CIFAR-100 basod on our methods shown in Fig. 3 We can see from the results that QAT on coreset with our method has more centralized and smoother loss compared to the baseline methods, which reflects that our method helps improve the training stability of
QAT. We also visualize the distribution of disagreement scores dps and error vector score dgvs in Fig. 4a and Fig. 4b The setting is the same as the MobileNet V2 experiment listed in Tab. [ We can see from the results that the mean of dps shifts to zero during the QAT, which proves that dps is a useful metric to quantify the importance each sample. The distribution discrepancy between dgvs and dps proves the necessity of considering both metrics to select diverse data into our coresct.
dos Models trained w/ different epochs
dos vs. dEvs in the same epoch (10)
2000
1000
Epoch 10
dos
Epoch 50
dgvs
1500
Fpoch 80
1250
600
1000
750
400
500
200
250
0.25
1.25
0.00
0.25
0.50
0.75
1.00
1.25
(a) dps distribution
(b) dps vs. devs
Figure 4: (a) Distribution of disagreement scores dps on MobileNet V2 for different epochs. (b) Distribution of disagreement scores dps and error vector score devs on MobileNet V2 in the same epoch.
6 Conclusion
This is the first work focusing on the data efficiency and robustness of quantization- training (QAT). By removing samples from the training batches and analyzing the loss gradient, theoretically prove that.
12

---

Published in Transactions on Machine Learning Research (08/2024)
the importance of each sample varies significantly for QAT. The error-vector score and disagreement score are proposed to quantify this importance. Considering the training characteristics of QAT, we propose a fully quantization-awar Adaptive Coreset Selection (ACS) method to better adapt to different training phases and improve training data diversity. Extensive experiments on various datascts, networks, and quantization settings further demonstrate the effectiveness of our method. We also verified the proposed ACS on the noisy label sotting and proved that our method can improve the robustness of QAT.
Acknowledgement
This research supported by HKSAR RGC General Research Fund (GRF) # #16208823.
References
Sharat Agarwal, Himanshu Arora, Saket Anand, and Chetan Arora. Contextual diversity for active learning. In Computer Vision ECCV 2020: 16th European Conference, Glasgow, UK, August 23 28, 2020, Proceedings, Port XVI, pp. 137-153, 2020.
Ibrahim M Alabdulmohsin, Behnam Neyshabur, and Xiaohna Zhai. Revisiting neural scaling laws in language and vision. Advances in Newral Information Processing Systems, 35:22300-2251 2022.
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating propagating gradients through stochastic neurons for conditional computation. arXiv preprint 130:::008.34430 2013.
Yash Bhalgat, Jinwon Lee, Markus Nagel, Tijmen Blankevoort, and Nojun Kwak. Lsq+: Improving low-bit quantization through learnable offsets and better initialization. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pottern Recognition Workshops, pp.
2020.
Zalán Borsos, Mojmir Mutny, and Andreas Krause. Coresets via bilevel optimization for continual learning and streaming. Advances in Neural Information Processing Systems, 33:14879 1879-11890, 2020.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Jungwook Choi, Zhuo Wang, Swagath Venkataramani, Pierce I-Jen Chuang, Vijayalakshmi Srinivasan, and Kailash Gopalakrishnan. Pact: Parameterized clipping activation for quantized neural networks. arXiv preprint Xr:i 06080, 2018.
Alexis Conneau and Guillaume Lample. Cross-lingual language model pretraining. Advances in neural information processing systems, 32, 2019.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image databasc. In 2009 IEEE conference on computer vision and pottern recognátion, pp. 248-255. Icco, 2009.
Alexcy Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image worth 16x16 words: Transformers for image recognition at scale. arXiu preprint arXiv:2010. 1 11929, 2020.
Melanie Ducoffe and Frederic Precioso. Adversarial active learning for deep networks: margin based approach. arXiv preprint NNii:18020088. 2018.
Steven K Esser, Jeffrey L McKinstry, Deepika Bablani, Rathinakumar Appuswamy, and Dharmendra. S Modha. Learned step size quantization. In International Conference Learning Representations, 2020.
Jun Fang, Ali Shafice, Hamzah Abdel-Aziz, David Thorsley, Georgios Georgiadis, and Joseph H Hassoun. fost-traiiing piccewise linear quantization for deep neural networks. In European Conference on Computer Vision, pp. 69 86. Springer, 2020.
13

---

Published in Transactions on Machine Learning Research (08/2024)
Stanislay Fort and Surya Ganguli. Emergent properties of the local geometry of neural loss landscapes. arXiv preprinl 05929, 2019.
Stanislav Fort, Gintare Karolina Dziugaite, Mansheej Paul, Sepideh Kharaghani, Daniel M Roy, and Surya Ganguli. Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the neural langent kernel. Advances in Neural Information Processing Systems, 33:5850 5861, 2020.
Satoru Fujishige. Submodular functions and optimization. Elsevier, 2005.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770 778, 2016.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge a neural network. arXie preprint arXiv:1503, 503.02531, 2015.
Andrew G Howard, Menglong Zhu, Bo Chen, Dmilry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. ai preprint arXiv: 700.0404.....11 2017.
Xijie Huang, Zhiqiang Shen, Shichao Li, Zechun Liu, Hu Xianghong, Jeffry Wicaksana, Eric Xing, and Kwang- Ting Cheng. Sdq: Stochastic differentiable quantization with mixed precision. In International Conference on Machine Learning, pp. 99999 PMLR, 2022.
Zhizhong Huang, Junping Zhang, and Hongming Shan. Twin contrastive learning with noisy labels. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 11661-11670, 2023.
Rishabh Iyer, Ninad Khargoankar, Jeff Bilmes, and Himanshu Asanani. Submodular combinatorial information measures with applications machine learning. arXiv preprint arXiv:2006. 15412, 2020.
Norman Jouppi, Cliff Young, Nishant Patil, David Patterson, Gaurav Agrawal, Raminder Bajwa, Sarah Bates, Suresh Bhatia, Nan Boden, Al Borchers, al. In-datacenter performance analysis of tensor processing unit. In Proceedings of the 44th onnual international symposium computer architecture, pp. 1-12, 2017.
Patrick Judd, Jorge Albericio, Tayler Hetherington, Tor M Aamodt, and Andreas Moshovos. Stripes: Bit- scrial deep neural network computing. In 2016 49th Annual IEEE/ACM International Symposium on Microurchiltecfure (MICRO), pp. 1- IEEE, 2016.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361 2020.
Jacob Devlin Ming- Wei Chang Kenton and Lee Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACI-HILT, pp. 4171-41888 2019.
Krishnateja Killamsetty, Sivasubramanian Durga, Ganesh Ramakrishnan, Abir De, and Rishabh Tyer. Grad- match: Gradient matching based data subset selection for efficient deep model training. In International Conference Machine Learning, pp. 5464-5474. PMLR, 2021a.
Krishnateja Killamsetty, Durga Sivasubramanian, Ganesh Ramakrishnan, and Rishabh Iyer. Glister: General- ization based data subset selection for efficient and robust learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 8110-8118, 2021b.
Krishnateja Killamsotty, Xujiang Zhao, Feng Chen, and Rishabh Iyer. Retrieve: Coreset selection for efficient. and robust semi-supervised learning. Advances in Neural Information Processing Systems, 34:14488-14500 2021c
14

---

Published in Transactions on Machine Learning Research (08/2024)
Jangho Kim, Yash Bhalgat, Jinwon Lee, Chirag Patel, and Nojun Kwak. Qkd: Quantization-aware knowledge distillation. Xiv preprint arXin:1911.1249. 2019.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitchead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. arv preprint arXi::330.006643, 2023.
Andreas Kirsch. Does 'deep learning on a data diet reproduce? overall yes, but grand at initialization does not. Transactions on Machine Learning Research, 2023.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Alex Krizhevsky, Πla Sutskever, and Geoffrey E, Hinton. Imagenet classification with deep convolutional neural networks. Advances in newral information processing systems, 25:1097-1105, 2012.
Jung Hyun Lee, Jeonghoon Kim, Se Jung Kwon, and Dongsoo Lec. Flexround: Learnable rounding based element-wise division for post-training quantization. In International Conference on Machine Learning, pp. 18913 18939. PMLR, 2023.
Kimin Lee, Sukmin Yun, Kibok Lee, Honglak Lee, Bo Li, and Jinwoo Shin. Robust inference via generative classifiers for handling noisy labels. In International conference on machine learning, pp. 2019.
PMT.R,
Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss landscape of neural nets. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, pp. 6391-6401, 2018.
Lin Li, Jun Xiao, Hanrong Shi, Hanwang Zhang, Yi Yang, Wei Liu, and Long Chen. Nicest: Noisy label correction and traiming for robust scene graph generation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.
Rundong Li, Yan Wang, Feng Liang, Hongwei Qin, Junjie Yan, and Rui Fan. Fully quantized network for object detection. In Proceedings of the IEFE/CVE conference on computer vision and pattern recognition, pp. 2810 2819, 2019a.
Yifan Li, Hu Han, Shiguang Shan, and Xilin Chen. Disc: Learning from noisy labels via dynamic instance- specific selection and correction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, pp. 24070-24079, 2023.
Yuhang Li, Xin Dong, and Wei Wang. Additive powers-of-two quantization An efficient non-uniform discretization for neural networks. In International Conference on Learning Representations, 2019b.
Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision ECCV 2014: 13th European Conference, Zurich, Swilzerland, Seplember 6-12, 2014, Proceedings, Part V 13, pp. 740 755. Springer, 2014.
Tsung-Y Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, pp. 2980-2988, 2017.
Shih- Shih-Yang Liu, Zechun Liu, and Kwang- -waang-Ting Cheng. Dacillation-fres quantization for low-bit vision trans- formers. ar preprint arXiv:2302. 02210, 2023a.
Shih-yang Liu, Zechun Liu, Xijie Huang, Pingcheng Dong, and Kwang- Swang-Ting Cheng. Llm-fp4: 4-bit floating- point quantized transformers. In The 2023 Conference on Empirical Methods in Natural Language Processing, 2023b.
Zechun Liu, Haoyuan Mu, Xiangyu Zhang, Zichao Guo, Xin Yang, Kwang-Ting Ting Cheng, and Jian Sun. Metapruning: Meta learning for automatic neural network channel pruning. In Proceedings of the IEEE/CVF international conference on compuler vision, pp. 3296 3305, 2019.
15

---

Published in Transactions on Machine Learning Research (08/2024)
Zechun Liu, Barang-Ting Cheng, Dong Huang, Eric P Xing, and Zhiqiang Shen. Nommiforn-to-umffon quantization: Towards accurate quantization via generalized straight-through estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4942 4952, 2022.
Zechun Liu, Barlas Oguz, Changsheng Zhao, Ernic Chang, Pierre Stock, Yashar Mchdad, Yangyang Shi, Raghuraman Krishnamoorthi, and Vikas Chandra. Llm-qat: Data-[ree quantization aware training for large language models. ar preprint arXiv:2305.17888 2023c.
Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, and Changshui Zhang. Learning efficient. convolutional networks through network slimming. In Proceedings of the IEEE international conference compuler vision, pp. 2736 2744, 2017.
Zhuang Liu, Mingjie Sun, Tinghui Zhou, Gao Huang, and Trevor Darrell. Rethinking the value of network pruning. In International Conference on Learning Representations, 2018.
David Lopez-Paz, Léon Bottou, Bernhard Schölkopf, and Vladimir Vapnik. Unifying distillation and privileged information. arXiv preprint arXiv:1511.08443 2015.
Katerina Margatina, Giorgos Vernikos, Loie Barrault, and Nikolaos Aletras. Active learning by acquiring contrastive examples. In Proceedings of the 2021 Conference Empirical Methods in Natural Language Processing, pp. 650-663, 2021.
Michel Minoux. Accelerated greedy algorithms for maximizing submodular set functions. In Optimization techniques, pp. 234-243. Springer, 1978.
Seyed Iman Mirzadeh, Mehrdad Farajtabar, Aug Li, Nir Levine, Akihiro Matsukawa, and Hassan Ghasemzadeh. Improved knowledge distillation via teacher assistant. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 5191-5198, 2020.
Baharan Mirzasoleiman, Amin Karbasi, Rik Sarkar, and Andreas Krause. Distributed submodular maximiza- tion: Identifying representative elements in massive data. In Advances in Neural Information Processing Systems, pp. 1049-2051 2013.
Baharan Mirzasoleiman, Jeff Bilmes, and Jure Leskovec. Coresets for data-efficient training of machine learning models. In International Conference on Machine Learning, pp. 6950 6960. PMLR, 2020.
Asit Mishra and Debbie Marr. Apprentice: Using knowledge distillation techniques improve low-prccision network accuracy. In International Conference on Learning Representations, 2018.
Daisuke Miyashita, Edward H Lee, and Boris Murmann. Convolutional neural networks using logarithmic data representation. arXie preprint arXiv: 010000 2016.
Pavlo Molchanov, Arun Mallya, Stephen Tyree, luri Frosio, and Jan Kautz. Importance estimation for neural network pruning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, pp. 11264-11272, 2019.
Markus Nagel, Rana Ali Amjad, Mart Van Baalen, Christos Louizos, and Tijmen Blankevoort. Up or down? adaptive rounding for post-training quantization. In International Conference Machine Learning, pp- 7197-7206. PMLR, 2020.
George I, Nemhauser, Laurence A Wolsey, and Marshall I, Fisher. An analysis of approximations for maximizing submodular set functions i. Mathematical programming, 14(1):265-299) 1978.
Dongmin Park, Scola Choi, Doyoung Kim, Hwanjun Song, and Jac-Gil Lee. Robust data pruning under label noise via maximizing re-labeling accuracy. Advances in Neural Information Processing Systems, 36, 2024.
Wonpyo Park, Dongju Kim, Yan Lu, and Minsu Cho. Relational knowledge distillation. In Proceedings of the CCFF/CVF Conference on Computer Vision and Pattern Recogrition,, pp. 3967-3976, 2019.
16

---

Published in Transactions on Machine Learning Research (08/2024)
Mansheej Paul, Surya Ganguli, and Gintare Karolina. Dziugaite. Deep learning on data diet: Finding important examples early in training. Advonces in Neural Information Processing Systems, 34:20596-200 20607, 2021.
Hieu Pham, Melody Guan, Barret Zoph, Quoc Lc, and Jeff Dean. Efficient ncural architecture scarch via parameters sharing. In International Conference on Machine Learning, pp. 4095 4104. PMLR, 2018.
Antonio Polino, Razvan Pascanu, and Dan Alistarh. Model compression via distillation and quantization. In International Conference on Learning Representations.
Omead Pooladzandi, David Davini, and Baharan Mirzasoleiman. Adaptive second order coresets for data- efficient machine Icarning. In International Conference on Machine Learning, pp. 17848-17869. PMLR, 2022.
Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: A core-set approach. International Conference on Learning Representations, 2018.
Hardik Sharma, Jongse Park, Naveen Suda, Liangzhen Lai, Benson Chau, Vikas Chandra, and Hadi Esmaeilzadeh. Bit fusion: Bit-level dynamically composable architecture for accelerating deep neural network. In 2018 ACM/IEEE 45th Annual International Symposium on Computer Architecture (ISCA), pp. 764-775. IEEE, 2018.
Zhiqiang Shen and Eric Xing. A fast knowledge distillation framework for visual recognition. arXiv preprint arXiv:2112.01528, 2021.
Hwanjun Song, Minseok Kim, Dongmin Park, Yooju Shin, and Jae- Jae-Gil Lee. Learning from noisy labels with deep neural networks: A survey. IEEE transactions on neural networks and learning systems, 2022.
Ben Sorscher, Robert Geirhos, Shashank Shekhar, Surya Ganguli, and Ari Morcos. Beyond neural scaling laws: beating power law scaling via data pruning. Advances in Neural Information Processing Systems, 35: 19523 19536, 2022.
Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International Conference on Machine Learning, pp. 6105-6111 6114. PMLR, 2019.
Mariya Toneva, Alessandro Sordoni, Remi Tachet des Combes, Adam Trischler, Yoshua Bengio, and Geoffrey J. Gordon. An empirical study of example forgetting during deep neural network learning. In International Conference on Learning Representations, 2019.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. Xr preprint arXin:2307.09228 09988, 2023.
Yuanpeng Tu, Boshen Zhang, Yuxi Li, Liang Liu, Jian Li, Yabiao Wang, Chengjie Wang, and Cai Rong Zhao. Learning from noisy labels with decoupled meta label purifier. In Proceedingss of the TEEE/CVF Conference on Computer Vision and Pottern Recognition, pp. 19934-19943, 2023.
Vladimir N Vapnik. An overview of statistical learning theory. IEEE transactions on neural networks, 10(5): 088-999,19 1999.
Peisong Wang, Qiang Chen, Xiangyu He, and Jian Cheng. Towards accurate post-training network quantization via bit-split and stitching, In International Conference on Machine Learning, pp.
PMLR, 2020.
Hongxin Wei, Lue Tao, Renchunzi Xie, and Bo An. Open-set label noise can improve robustness against: inherent label noise. Advances in Neural Information Processing Systems, 34:7978 7992, 2021.
Max Welling. Herding dynamical weights learn. In Proceedings of the 26th Annual International Conference Machine Learning, pp- 1121-1128, 2009.
17

---

Published in Transactions on Machine Learning Research (08/2024)
Gert W Wolf. Facility location: concepts, models, algorithms and case studies. series: Contributions to management science, 2011.
Xiaobo Xia, Bo Han, Yibing Zhan, Jun Yu, Mingming Gong, Chen Gong, and Tongliang Liu. Combating noisy labels with sample selection by mining high-discrepancy examples. In Proceedings of the IEFE/CVF International Conference on Computer Vision, pp. 1833-1843 2023a.
Xiaobo Xia, Jiale Liu, Jun Yu, Xu Shen, Bo Han, and Tongliang Liu. Moderate coreset: A universal method of data selection for real-world data-cfficient deep learning. In The Eleuenth International Conference on Learning Represenlations, 2023b.
Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models. In International Conference on Machine Learning, pp. 38087-38099. PMLR, 2023.
Han Xiao, Huang Xiao, and Claudia Eckert. Adversarial label flips attack on support vector machines, In ECAI 2012, pp. 870 875. IOS Press, 2012.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R. Salakhutdinov, and Quoe V Lc. XInet: Generalized auloregressive pretraining for language understanding. Advances in neural informalion processing systems, 32, 2019.
Jiangchao Yao, Jiajie Wang, Ivor W Tsang, Ya Zhang, Jun Sun, Chengqi Zhang, and Rui Zhang. Deep learning from noisy image labels with quality embedding. IEFE Transactions Image Processing, 28(4): 1909 1922, 2018.
Edouard Yvinec, Arnaud Dapogny, Matthieu Cord, and Kevin Bailly. Powerquant: Automorphism search for non-uniform quantization. In The Eleventh International Conference on Learning Representations, 2023.
Dongqing Zhang, Jiaolong Yang, Dongqiangzi Ye, and Gang Hua. Lq-nets: Learned quantization for highly accurate and compact deep neural networks. In Proceedings of the European conference on computer vision (ECCV), pp. 365-382 2018.
Shuchang Zhou, Yuxin Wu, Zekun Ni, Xinyu Zhou, He Wen, and Yuheng Zou. Dorefa-net: Training low bitwidth convolutional neural networks with low bitwidth gradients. arXiv preprint arXiv:1606.0611 2016.
Xiao Zhou, Renjie Pi, Weizhong Zhang, Yong Lin, Zonghao Chen, and Tong Zhang. Probabilistic bilevel coresel selection. In International Conference on Machine Learning, pp. 27287 27302. PMLR, 2022.
Bohan Zhuang, Lingqiao Liu, Mingkui Tan, Chunhua Shen, and Ian Reid. Training quantized neural networks with full-precision auxiliary module. In Proceedings of the IEEE/CVF conference computer vision and pottern recognition, pp. 1488-1499 2020.
18

---

Published in Transactions on Machine Learning Research (08/2024)
Appendix
This appendix includes an additional introduction to related works, training dynamics, efficiency analysis, extended experimental analysis on data coverage, and a discussion of transferability and generalizabllity that is not included in the main text due to space limitations. These contents are organized in separate sections as follows:
Sec. A elaborates the detail of previous representative coreset selection methods and analyzes the reason for failure in quantization-aware training (QAT) scenario.
Sec. B
demonstrate the training dynamics of our ACS with different selection intervals and show how methods help to stabilize the training and accelerate the convergence.
Sec. in Tab. g showing that our coreset selection only incurs minimal efficiency overhead both with and without knowledge distillation.
C includes the real training time composition as supplement of QAT training time comparison
Sec. D provides additional experiments to prove that the improvement of performance with method does not exclusively come from covering more data from the full training set.
Sec. E provides detailed experimental results with and without knowledge distillation (KD). Both the accuracy and real training time are shown.
Sec. analyzes the transferability and generalizability of our coresct from different models.
A Detailed Introduction of Selected Baselines
In this section, we will introduce the selected baseline coreset selection methods in detail and show the defects of these methods when applied to QAT.
EL2N-Score/ Graaer//oaad-oooreee (Paul et al., 2021 The GraNd-score for a given training sample {a, y is defined as (=( = Ev,, 11 (z, v) which the expected magnitude of the loss vector with respect to the weights. The importance of each sample is measured by the expected loss gradient norm, which has a similar intuition to error-vector score devs- However, another assumption from GraNd score is that this approximation only holds when the model has been trained for few epochs. We must perform early training on the current model to have statistics to compute the score. In addition, storing all the gradients incurs significant memory overheads during QAT. The efficiency will be lower than full dataset training with a high subsct fraction. The performance with these metrics is sub-optimal as the converged quantized model in the addition, Based on the reproduction from Kirsch (2023)
later training epochs of QAT is not considered. observing that the inconsistency of GraNd at initialization, the performance of GraNd-score- based coreset selection cannot outperform random selection across various datasets and tasks.
The core contribution of Forgetting is that forgetting and learning events are defined. In the classification setting with a given dataset D = (3). For training example x; obtained after l steps using SGD, the predicted label is demoted as of = arg max p(ya p(yx/xx;; 0²). acc; 1gt= is defined to be a binary indicator encoding whether the classification of the specific example is correct at time step t. Example x; undergoes a forgetting event when acc decreases between two consocutive updates: acc/ acc The example x; is misclassified at step è t+ after having been correctly classified at step Corresponding to the forgetting event, learnóng event is defined to be acc; < acc/+1 With the statistics of forgetting events in early training, more difficult to learn. However, the intuition to select "difficult" samples that incur more misclassification is not always correct. For large-scale datasets (ImageNee-11K, etc.) and datasets with relatively smaller size (such as MNIST, CIFAR-10, etc.). the difficulties of classification vary significantly. Selecting samples that are difficult to learn at the very beginning is not always reasonable. In the quantization-awal training setting, the quantized model will first calibrate the quantization parameters and recover the weight at the early stages. We should not select "difficult samples" for calibration and recovery.
Forgetting (Toneva et al. 2019
select those samples that incur more forgetting events and are
19

---

Published in Transactions on Machine Learning Research (08/2024)
Glister (Killamsetty et al., 2021b) The Glister performs data selection based on the assumption that the inner discrete data selection is an instance of (weakly) submodular optimization. Let V (1,2.....,,) denote a ground set of items (the set of training samples in the setting of coreset selection). Set functions are defined as functions/ f 2 + R that operate on subsets of V. set function f is defined to be a submodular function (Fujishige 2005) if it. satisfies the diminishing returns property that. for subsets S f T S V.J()s) & () - f(S) 2 [((TT). Some natural combinatorial functions (facility location, set cover, concave over modularity, etc.) are submodular functions ę[yer al., 2020). Submodularity is also very appealing because a simple greedy algorithm achieves
I/e constant factor approximation guarantee (Nemhauser et al. 1978) for the problem of maximizing a submodular function subject to a cardinality constraint (which most data selection approaches involve). Moreover, several variants of the greedy algorithm have been proposed, further scaling up submodular maximization to almost linear time complexity (Minoux, 1978 Mirzasoleiman et al.) 2013). However, the greedy algorithm still consumes tremendous time compared other metric-basod selection methods, which only need to perform sorting on the proposed metrics. We also observed that the greedy algorithm fails on QAT for low-bit settings.
kCenterGreedy (Sener & Savarese 2018) As one of the most Straight-forwant geometry-basee coresct selection methods, the intuition simple: we can measure the similarity by the distance of data points and select the center point of data cluster can make use of the redundancy in the dataset. This method aims to solve the minimar facility location problem (Wolf. 2011], which is defined to be selecting h samples as subset S from the full dataset T such that the longest distance between data point in T,S and its closest data point in S is minimized:
min max min D(I;, SCT (
(16)
where -),-)) is the distance measurement function. The problem is NP-hard, and a groedy approximation known as K- K-CENTER GREEDY has been proposed in [eneer & Savarese (2018). Similarly, as the greedy algorithm is involved, the efficiency is negatively influenced, which is not affordable for our QAT setting.
Contextual Diversity (CD) (Agarwal et al., 2020 Specifically designed for coreset selection for deep convolutional neural networks (CNNs), Contextual Diversity (CD) fully leverages the ambiguity in feature representations. The ambiguity is quantified as class-specific confusion as the selection metric. Assume C= C= [1. ] } is the set of classes predicted by a CNN-based model. For a region r within an input image I, let P, P7 T; be the softmax probability vector as predicted by the model 0. The pseudo-label for the region C I is defined as T, arg maxjec PrLi], where the notation PrLi] denotes the jth element of the vector. For a given model 0 over the unlabeled I, the class-specific confusion for class is defined as rgng 1., P. (:0)
- M
with w, 2 0 as the mixing weights. The pairwise contextual diversity,
L.RR Mir
which is the KL-divergence of the metric between two samples I and 12 could be used as distance metric to replace the Euclidean distance in the previous kCenterGreedy (Sener & Savarese] 2018). As this work basically follows kCenterGreedy (Sener & Savarese 2018) to perform coreset selection, the defects also lie in the efficiency of the greedy algorithm and failure with the low-bit sctting.
Moderate Coreset (Xia et al., 2023b) As the most recent work of coreset selection, Moderate has optimal efficiency as no greedy searching or pack-propagation is involved during selection. Previous methods rely on score criterion that only applies to a specific scenario. This work proposes to utilize the score median as proxy of the statistical score distribution and select the data points with scores close the score median into the coresct. The main drawback of this method applied to quantization-aware training is that quantized distribution is not considered.
B QAT Training Dynamics with ACS
In this section, we report the training dynamics, including the training loss and training accuracy of QAT ResNet-18 on ImageNet-1K coreset with a 10% subset fraction. As can be seen from the results in Fig. 5 when the coreset changes adaptively to the current. iterations, the accuracy will drop, and loss will increase
20

---

Published in Transactions on Machine Learning Research (08/2024)
significantly at the specific iteration of updating the subset. However, the quantized model will converge fast on the new subset. The training with an adaptive coreset can effectively help avoid overfitting and improve the final performance.
Tes: Los
Tex Lows
Text Ausuaay
1807
2000
2000
200 1900
(a) Acc-random
(b) R-60
(c) Loss-random
(d) R-60
Toon Loss
Tratt Tans
Thet Los
Text
Thah AsDEEay
Trab Accessy
That Antrray
Text Accesey
190
2014
1491
(e) R=40 40
(f) 2020
R=40
(h) R=20
Toxt
Tiah
Text
(i) R=10 ii) R=5
(k) R=10
(l) R=5
Figure Boiinig/TTestingg loss/accuraae comparison of 4/4-bit quantized ResNet-18 on 10% coreset of ImageNet-1K with different selection interval R.
C Training Time Composition
As a supplement of the detailed QAT training time comparison Tab. 3 time composition of training and selection in Tab. g Since KD will change the loss function of SGD training and is difficult, to break down these two parts, we provide two sets of results with and without KD. The experiments are also conducted quantized ResNet-18 ImageNet-1K coresct with 2 NVIDIA RTX 3090 GPUs.
we also provide the detailed training
The results show that coreset selection only incurs a minimal efficiency overhead both with and without KD. The selection time is a constant and is only related to the full dataset size and the selection intervals R. The training with KD and without KD only influence the backpropagation time of the training and there no backward in the selection phase, which makes the time of selection the same (3.2h) across all settings.
D Coreset Coverage Analysis
The advantages of the proposed quantization-awarc Adaptive Coresct Selection (ACS) algorithms are two-fold: adaption and diversity. In this section, we further demonstrate that improving diversity by covering all the training data in different subsets is not optimal compared to our adaptive coreset. We use the "Full Coverage Split" of the CIFAR-100 datasct as our basoline, which is select non-overlapped samples randomly into the subset of fraction S in the first [1/S] epochs. is guaranteed that all the training samples are included, but the sequence random in this setting. We apply QAT to 2/32-bit quantized MobileNet- V2 the random
21

---

Published in Transactions on Machine Learning Research (08/2024)
Table Composition of training time on QAT of quantized ResNet-18 on ImageNet- 1K with different subset fractions and KD scttings. The bitwidth for quantized ResNet- 18 is 4/4 for weights weights/ /aciivatiovi
Stage/ Fraction (%) Apply KD?
10%
30%
50%
60%
70%
80%
Ours-all
11.3h
20.7h
36.1h
42.2h
48 48.0h
53.8h
Ours-selection
3.2h
3.2h
3.2h
3.2h
3.2h
3.2h
Ours-training
V
8.1h
17.5h
32.9h
39.0h
44.8h
50.6h
Ours-all
9.7h
16.0h
29.5h
34.4h
40.2h
45.4h
Ours-sclection
3.2h
3.2h
3.2h
3.2h
3.2h
3.2h
Ours-training
6.5h
15.8h
26.3h
31.2h
37.0h
42.2h
subset, "Full Coverage Split" subsct, and coresct using our methods. The selection interval R is the same across all settings. When the subset. fraction S is large, the difference in coverage rale with various methods is minor. We then only evaluate the S E {10%, 20%, 30%, 40%, The results are shown in Tab. I0] where we can see that "Full Coverage Split" has limited superiority on performance compared to random baseline, especially when the subset fraction R is large. Our method outperforms the other two settings across all fractions, proving that both adaption and diversity help to improve the performance.
Table 10: Comparision of Top-1 Accuracy with Random vs. Full coverage split vs. Ours
Method/Fraction Fraction (%)
10%
20%
30%
40%
50%
Random
62.25-10.71
6..07=0.47
65.22:40.41
0.85
66.24:10.55
Full Coverage Split
62.55-10.65
64.22=0.81
65.34+1.177
6..69+0.69
66.20
Ours
63.37:0.555
65.9110.40
66.4110.31
66.8611.72
67.19:0.65
E
Detailed Experimental Results without Knowledge Distillation
Knowledge Distillation (KD) is a normal and "default" setting for all previous quantization work (Mishra & Marr 2018 Zhuang et al. 2020 Liu et al.) 2022; Bhalgat et al.] 2020 Huang et al. 2022), including the LSQ quantization (Bhalgat et al. 2020) we use in this paper. For a fair comparison with previous work, we equally utilize KD for all the experiments in this work rogardless of the precision and dataset fraction. The full-data training baseline also involves training with knowledge distillation.
To verify the effectiveness of our method without KD, we remove the knowledge distillation in our method. Since the DS is built on the soft label of the teacher model, we do not use it in our selection w/o KD. Only EVS is applied as the selection metric. We follow the same settings for quantized ResNet-18 on ImageNet- as shown in our paper. The training time and accuracy are shown as follows in Tab. 11 When full data are selected (S 100%), the Top-1 accuracy is 70.21% and training time is 3.1h without KD. From the results see that our method still outperforms previous SOTA without KD and the training
shown in Tab. II efficiency is still optimal.
Table 11: Comparison of Top-1 Accuracy and the training time of QAT of quantized ResNet-18 on ImageNet- IK with different subset fractions without KD settings. The bitwidth for quantized ResNet-18 is 4/4 for seights/activations..
10%
30%
50%
60%
70%
80%
/eahoddFrracion (%)
Acc:
Time
Acc.
'T'me
Acc.
Timme
Anc.
'Time
Acc
'T'me
Acc:
Time
EL2N w/o KD
59.71
10.6h
63.50
17.1h
65.19
31.3h
66.38
35.0h
67.90
42.9h
69.01
48.0h
Glister w/o KD
62.41
11.9h
65.18
22.7h
66.47
34.5h
67.05
41.7h
68.81
50.3h
69.45
56.9h
Ours w/o KD
66.91
9.7h
68.77
16.0h
69.25
29.5h
69.66
34.4h
69.85
40.2h
70.03
45.4h
22

---

Published in Transactions on Machine Learning Research (08/2024)
F Coreset Transferability and Generalizability
The thcorctical analysis of the importance of cach sample in QAT is modcl-specific, which means that data selected using our method is the optimal coreset for the current model. However, if the coreset discovered by a specific pre-trained model is applicable to other models, our method can be a potential solution for the model-agnostic coreset method. addition, if there is a significant coreset overlap across different models, our method can even solve dataset distillation.
We first design experiments to verify the transferability of the coreset using method. We use 2/32-bit. ResNet-18 to generate a coreset CIFAR-100 and apply it to MobileNet-V2. The results are shown in Tab. 12 As the ResNet-18 18 coreset performs better than the random subset on MobileNet-V V2, our coreset has some extent of generalization ability to unseen network structures, but the effectiveness is still worse than the MobileNet coresct.
Table 12: Comparision of Top-1 Accuracy with different coresct of MobileNet-V2 on CIFAR-100.
Method
Coreset Generation Model
10%
20%
30%
40%
50%
Random
6..2
64.07=0.477
65.22+0.41
65.55=0.80
66.24+0..5
Ours
MobileNet
03.5
65.9140040
66.41+40.31
66.86-10.72
67.19:10.65
Ours
ResNet-18
0..94+0.445
64.18+00.73
65.70++.40
65.90+0.51
66.86+00.42
We then design experiments to verify the overlap between the coreset of different models. We analyze the 10% fraction coreset selected using our method in the final epochs of 2/32-bit ResNet-18, ResNet-50, and MobileNet-V2. The percentage of the overlap data is shown in Tab. 13
Table 13: Coresel overlap of different model pairs on CIFAR-100.
Model Pair
Coreset Overlap Percentage
1R MobileNet V2)
37.1% 1%
[ResNet-18, ResNet-500
77.3%
[ResNet-50, MobileNet V2}
29.0%
We empirically find out that there is some overlap between the coreset of different models, and the overlap is more significant when these two models have a similar structure (ResNet-18 and ResNet-50). We then further apply majority voting from the CIFAR-100 coreset of five models (ResNet-18, ResNet-34, ResNet-50, MobileNet-V V2, MobileNet-V3: to generate "general coreset". This "general coreset" of 10% data fraction is applied to 2/32-bit quantized Vision Transformer ViT-B/16 (Dosovitskiy et al.) 2020), where we only got Top-1 accuracy of 72.3%, which is lower than 10% corcset using our method of 78.9% and even lower than 10% random coreset of 74.6%. We conclude that our method is model-specific, which has some extent of generalization ability to unseen network structures but cannot be a general data distillation approach.
23


Reviews:
[{"review_id": "IXjmvdHz7M", "reviewer": "Reviewer_E4To", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength\n\nA computationally efficient and performant novel approach for coreset selection in the QAT setting is proposed.\n\nThis paper empirically compares existing coreset selection methods that are not specialized for the QAT setting and the proposed approach.\n\nThe ablation study reveals the effectiveness of each component (i.e., metric)\n\nWeakness\n\nClarity of Section 3 could be improved (See the requested changes part)\n\nThe comparison is limited between coreset selection methods. It is okay for evaluation of the coreset selection methods. However, it is not clear if the proposed coreset selection method is more effective to achieve a higher accuracy in the QAT setting than other approaches.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a method to subsample training dataset to improve the efficiency of quantization-aware training (QAT). Based on the author’s statement, this work is the first one addressing the coreset selection specialized for QAT. Two metrics are introduced to evaluate the importance of each datum and the data used for training are adaptively selected during the training by using the proposed metrics. Based on these two metrics, a novel approach for adaptive coreset selection is proposed. \n\nExperiments on several datasets are conducted to compare the proposed coreset selection method and other baseline coreset selection methods. These comparisons reveal that the proposed approach achieves higher accuracy under the same sampling ratio than other methods. Moreover, the computational efficiency over the baseline approaches is reported", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "P2. “However, previous methods select data based on feature (Agarwal et al., 2020; Sener & Savarese, 2018), error (Paul et al., 2021), gradient (Killamsetty et al., 2021a), and decision boundary (Margatina et al., 2021) can achieve notable efficiency improvement for full-precision training, but its effectiveness on low-precision QAT has not been explored before. “ The structure “However, …, but …. “ is strange. Please revise it.\n\nP4. The notation is suboptimal. “x” is used for different meanings.\n\nP5. Why does the right-hand side equality in Eq. (4) hold? Because of the quantization in (2), it looks like it does not necessarily hold. Please clarify it.\n\nP5. The authors are mixing up the notation for a standard gradient and STE. Because STE is not a gradient of a Moreover, the authors use a simplified notation for derivative, i.e., df / dw_q to mean df / dw | w = w_q. Combination of above two points makes it hard to verify the correctness of the derivation in Section 3. Please clarify them.\n\nP5. “Since learning rate η and the gradient of loss w.r.t. quantized weights dL dwq t are sample-agnostic, …” Why is the gradient of the loss w.r.t. the quantized weights sample-agnostic? Why does not one need to consider the inner product? Please clarify it.\n\nTable 3, 4, 5. Are these numbers the average values? Please clarify it and provide their STD or IQR to see the significance.\n\nPlease discuss the advantages and disadvantages of the proposed coreset selection method over other approaches to improve QAT. Does the proposed approach more efficient? Can we use the proposed coreset selection with other techniques for improving the final accuracy in the QAT setting?", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "5jAc9fFk6g", "reviewer": "Reviewer_gjZa", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths include:\n- Strong empirical results across various amounts of real data compared to baseline ethods, for example the across-the-board improvement in Figure 1 left.\n- Applied to multiple datasets (CIFAR-10, CIFAR-100, Imagenet-1K, COCO, RetinaNet).\n- The method works best for R=10, which means a new coreset is selected fairly often. This adds a bit of overhead, especially for the 10% case (10h -> 12h). This seems worthwhile though, improving performance from 62.82 to 68.80 for the 10% data case.\n- Faster QAT of vision models is an important field. Efficient training of deployment-ready neural networks can have enormous impact.\n\nWeaknesses include\n- ACS leads to faster convergence, but not convergence to the same point. From Table 1 and 2, we see that the full-data accuracy of MobileNetv2 is 68.1%, but running with S=50% reaches 67.2%. The full accuracy of ResNet18 on Imagenet-1k is 72.46%, but the S-80% method reaches 72.31. Could QAT runs converge to the same point as full data training? If not, is full data QAT always preferred if you care about final performance? With ACS underperforming full data training, who is the intended target audience?\n- A few strong claims are made:\n  - this work being the first to investigate data efficiency is possibly true, but data efficiency of LLM quantization has been investigated previously as well [Dettmers 2023]. \n  - under Definition 1, it is stated that previous works empirically observe a similarity, and therefore that the gradient norm is positively correlated to the error vector. I do believe the gradient norm and the error vector are likely positively correlated, but that would have to be shown. Maybe a less specific claim is warranted here.\n  - in section 4, it is mentioned that ACS helps guarantee faster convergence, but an investigation of convergence behavior is missing (except appendix Figure 5), we are mainly looking at final performance. The model does converge to a better point than random coreset selection methods, but again, i believe the comparison point should be full-data training (without any coreset selection) at the same point in time.\n- In section 5.2, it stated that ACS can succesfully prune samples with incorrect labels, because final performance is better. This is not  shown currently, but it can be. Does the criterion indeed remove noisy labels? Given that criterion E_VS keeps samples with high error, it seems surprising that these noisy samples are removed.\n\n\nAll in all, although the claims are a bit strong, I think the investigation is interesting to those interested in reducing time-to-train during QAT.  However, an important datapoint is missing: full data training under the same time budget. Without that datapoint, I would probably still use full data QAT.\n\n\nReferences\n- [Dettmers 2023]: QLORA: Efficient Finetuning of Quantized LLMs, NeurIPS 2023", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work proposes a coreset selection algorithm, which is used to perform quantization-aware training. Specifically, the authors propose two metrics:\n1. Error vector score, or d_EVS, which is the L2 error between a one-hot encoded label and the model output\n2. Disagreement score, or d_DS, which is the L2 error between the real-weighted model output and the quantized model output.\n\nThey propose to use these metrics to select/filter data, in order to improve QAT performance. The main advantage of this approach is reducing training time, as it allows focusing only on the important samples (the \"coreset\"). They show empirically that this indeed leads to better overall performance compared to baselines, for various coreset sizes.\n\nThey also show that the same procedure can be used to train in scenarios where labels are noisy, as their selection criterion can filter samples with noisy labels. They compare to the EL2N score and GraNd score of previous work, and show improvements across the board.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- If available, a comparison to a full-data run at the same #hours could strengthen this work, in my opinion. At the moment, It is unclear to me why one would use ACS for QAT. What happens if you train a ResNet18 on all data for 12.7 hours? How does it compare to the 10% data run from Table 4? This could show that ACS has a practical advantage.\n- Please indicate how many experiments were performed when showing standard deviation or standard error.\n- Please provide additional support or explanation for some of the stronger claims, see above.\n- Could you add more information on how are the criteria are adapted for detector (retinanet) training? Is this looking only at the classification head of samples? If so, how are images with multiple objects handled?", "broader_impact_concerns": "No concerns on ethical implications. Although coreset selection can increase biases present in training data, this work is far removed from unethical application of models, and the method is largely data-agnostic.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "JG7e0v5Qkp", "reviewer": "Reviewer_BaSd", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**STRENGHTS**\n- The topic of efficient data selection in the context of quantized model training is of clear interests to the audience of this journal\n- The paper is well structured, easy to follow, appropriately cites related works\n- The proposed approximation for sample importance and their combination into the proposed ACS method appears to be novel\n- On the reported CNNs/datasets, comparison against several related techniques of sample-selection is favorable towards ACS in terms of performance (accuracy and training runtime)\n- I commend the authors for reporting in Appendix F on the limited generalization capabilities of coreset selection across diverse models. Negative results are also important\n\n**WEAKNESSES**\n- My main concern is that training dynamics and final performance are dependent on hyperparameter R (selection interval), which is manually selected. In their experiments, the authors applied a grid search on R to determine a good trade-off between accuracy and training time, settling on R=20 for MobileNetv2 and R=10 for ResNet18. I think R is unspecified for RetinaNet for the results reported in Tab.7. However, this is not a practical strategy if the objective is to reduce overall training time. At the same time, it is not clear how a viable strategy to select R on an unexplored model/dataset/quantization combination would look like\n- Limited demonstration of applicability and scalability of this method, as results are reported on a total of 3 models/datasets/quantization combinations (MobileNetV2-CIFAR100 at W2/A32 bits, ResNet18-ImageNet1k at W4A4 bits, RetinaNet[ResNet18 backbone]-COCO at W4A4 bits)\n- In particular for quantization, what drove this selection (weight-only for one model vs. weight & activations for the other)? For a given model and dataset, are results impacted by the choice of quantization (e.g., more aggressive quantization calls for more training data)?\n - The investigation of training with 10% label noise was interesting and given the model performance shown in Tab.6 I agree it is likely that this method, as the authors state, \"can successfully prune those samples with incorrect labels\". However, this statement could have been definitively proven by comparing the pruned labels with the noisy labels, and report what fraction of noisy labels was captured (accuracy / precision / F1 score) and how it changed during training. Was this comparison performed?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposed Adaptive Coreset Selection (ACS), a method to filter data used in Quantization Aware Training (QAT) to reduce training computational budget while preserving performance. \nSelection relies on a linear combination of two metrics: Error Vector Score, that determines the importance of each sample based on its impact on the loss, and Disagreement Score, that accounts for discrepancies between the quantized model being trained and the full-precision teacher. The authors show that ACS can speed up QAT and improve its robustness. Results are provided on CNN for visual tasks (image classification, object detection).", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See previous section.", "broader_impact_concerns": "Not addressed in this manuscript. No concern on my end.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper investigates the coreset selection in the context of quantization-aware training (QAT) to reduce training costs and mitigate the potential negative influence from label noises. Specifically, it proposes two metrics--error vector score and disagreement score--to select a subset of training data during QAT. The effectiveness of these methods is empirically evaluated across a variety of vison models and datasets, demonstrating improved performance compared to other data selection methods.

The paper received recommendations of acceptance from all reviewers. Overall, the paper is well written and easy to follow. The paper explores a new and import topic:  applying efficient data selection for QAT. The proposed methods are sound and intuitive, and the empirical evaluation and ablation studies are reasonally extensive, demonstrating consistent improvement over existing methods.

During the rebuttal, the authors conducted additional experiments, showing its effectiveness of the methods on vision transformers and filtering out noisy samples.  They also provided more clarification on the technical details and addressing concerns related to training dynamics to some extent.

In summary, this paper is technically solid and has the potential to offer benefits in the domain of efficient QAT. Therefore, I recommend acceptance.
Decision: Accept (poster)