Moderate Coreset: A Universal Method of Data Selection for Real-world Data-efficient Deep Learning
Authors: Xiaobo Xia
Jiale Liu
Jun Yu
Xu Shen
Bo Han
Tongliang Liu
Venue: ICLR 2023 poster (2023)

Abstract: Deep learning methods nowadays rely on massive data, resulting in substantial costs of data storage and model training. Data selection is a useful tool to alleviate such costs, where a coreset of massive data is extracted to practically perform on par with full data. Based on carefully-designed score criteria, existing methods first count the score of each data point and then select the data points whose scores lie in a certain range to construct a coreset. These methods work well in their respective preconceived scenarios but are not robust to the change of scenarios, since the optimal range of scores varies as the scenario changes. The issue limits the application of these methods, because realistic scenarios often mismatch preconceived ones, and it is inconvenient or unfeasible to tune the criteria and methods accordingly. In this paper, to address the issue, a concept of the moderate coreset is discussed. Specifically, given any score criterion of data selection, different scenarios prefer data points with scores in different intervals. As the score median is a proxy of the score distribution in statistics, the data points with scores close to the score median can be seen as a proxy of full data and generalize different scenarios, which are used to construct the moderate coreset. As a proof-of-concept, a universal method that inherits the moderate coreset and uses the distance of a data point to its class center as the score criterion, is proposed to meet complex realistic scenarios. Extensive experiments confirm the advance of our method over prior state-of-the-art methods, leading to a strong baseline for future research. The implementation is available at https://github.com/tmllab/Moderate-DS.

Published as a conference paper at ICLR 2023
MODERATE CORESET: A UNIVERSAL METHOD OF DATA SELECTION FOR REAL-WORLD DATA-EFFICIENT
DEEP LEARNING
Xiaobo Xia Jiale Liu Jun Yu² Xu Shen* Bo Han
Tongliang Liu 'f
!Sydney AI Centre, School of Computer Science, The University of Sydney
Schooo of Computer Science and Technology. Xidian University
3 "Department of Automation, University of Science and Technology of China
Alibaba Group *Department of Computer Science, Hong Kong Baptist University
ABSTRACT
Deep learning methods nowadays rely massive data, resulting in substantial costs of data storage and model training. Data selection useful tool to allevi- ate such costs, where a coreset of massive data is extracted to practically perform on par with full data. Based on surefully-designee first count the score of each data point and then select the data points whose scores lie in a certain range to construct a coreset. These methods work well in their re- spective preconceived scenarios but are not robust to the change of scenarios, since the optimal range of scores varies as the scenario changes. The issue limits the application of these methods, because realistic scenarios often mismatch precon- ceived ones, and it is inconvenient or unfeasible to tune the criteria and methods accordingly. In this paper. to address the issue, a concept of the moderate coreset is discussed. Specifically, given any score criterion of data selection, different sce- narios prefer data points with scores in different intervals. As the score median is a proxy of the score distribution in statistics, the data points with scores close to the score median can be seen as a proxy of full data and generalize different sce- narios, which are used construct the moderate coreset. As a proof-of-concepl, a universal method that inherits the moderate coreset and uses the distance of a data point to its class center as the score criterion, is proposed to meet complex realistic scenarios. Extensive experiments confirm the advance of our method over prior state-of-the-art methods, leading strong baseline for future research. The implementation is available at
score criteria, existing methods
INTRODUCTION
Large-scale datasets, comprising millions of examples, are becoming the de-facto standard to achieve state-of-the-art deep learning models (Zhao et al., 2021: Zhao & Bilen, 2021; Yang et al., 2022). Unfortunately, at such scales, both storage costs of the data and computation costs of deep learning model training are tremendous and usually unaffordable by startups non-profit organi- zations (Wang et al., 2018a; Coleman et al., 2020; Sorscher et al., 2022; Pooladzandi et al., 2022), which limits the success of deep learning models specialized equipment and infrastructure (Yang et al., 2023). For instance, the storage needs of ImageNet-22k (Deng et al., 2009) and BDD100K datasets (Yu et al., 2020a) are 1TB and 1.8TB respectively. Training PaLM (Chowdhery al., 2022) once requires a training dataset containing 780 billion high-quality tokens and then takes almost 1,920 TPU years. Additionally, hyper-parameter tuning or network architecture search could further increase the computation costs, which is pessimistic (Strubell et al., 2019; Schwartz et al., 2020).
Data selection came into being to deal with large data and mitigate the above issues for data- eeft--efficient deep learning. More specifically, data selection aims to find the most essential data points and build coreset of large data. Training on the coreset expected to continue the model performance achieved by training on the large data (Huang al., 2021b; Chen et al., 2021). Based on carefully-designee
Corresponding author (tongliang. liu @ sydney.edu.au)

---

Published as a conference paper at ICLR 2023
score criteria, recent works have presented various algorithms of data selection, e.g., in terms of loss values (Han et al., 2018; Jiang et al., 2018), forgetfulness (Toneva et al., 2019; Sorscher et al., 2022), and gradient matching (Paul et al., 2021; Pooladzandi et al., 2022). In respect of procedures, these works first sort the scores achieved by all data points and then simply select the data points with either smaller scores or larger scores, according different scenarios. For instance, for loss-based score criterion, if data is presumed a priori to be perfectly labeled, larger-loss data points are more important and selected (Lei al., 2022). Conversely, data is corrupted by outliers, smaller-loss data points are more critical because of the concern on model robustness (Lyu & Tsang, 2019).
State-of- the-heerr methods on data selection can achieve promising performance as they show. How- ever, they are specially designed for preconceived scenarios. The deliberate characteristic makes them work well under certain situations and demands, not stable or even extremely sensitive to the change of situations or demands, even though the change is slight (the concerns on complex realistic scenarios are detailed Section 2.2). The issue severely restrict the practical applications of the methods, since realistic scenarios cannot always match preconceived ones well, and realistic demands are frequently changed over time (Hendrycks Gimpel, 2017; Wu et al., 2021; Arjovsky al., 2019; Piratla al., 2020; Creager et al., 2021; Shen al., 2021; Li et al., 2022b; Wei et al., 2023; Huang et al., 2023). It is inconvenient, troubled, and often unachievable to tweak our method accordingly (Lu et al., 2018).
In this paper, to address the issue, we discuss a new concept about data selection, i.e., the moderate coreset, which is generic in multiple realistic tasks without any task-specific prior knowledge and adjustments. For the construction of the moderate coreset, given any score criterion of data selec- tion, we characterize the score statistics as a distribution with respect different scenarios. Namely, different scenarios correspond to and require data points with scores in different ranges. The dis- tribution can be generally depicted by the median of scores (James et al., 2013). Accordingly, data points with scores that are close to the score median can be seen as proxy of all data points, which used to build moderate coreset and generalize different scenarios.
As proof-of-concepe we present a universal method of data selection in complex realistic scenarios. Specifically, working with extracted representations extracted by deep models, we implement the distance of data point to its class center score criterion. Data points with scores close to the score median are selected as a coreset for following tasks. Comparing the complicated and time- consuming data selection procedure of many works, e.g., using Hessian calculation (Yang al., 2023), our method is simple and does not need to access model architectures and retrain models. We show that existing state-of-the-ane methods The proposed method is more superior to them in diverse data selection scenarios.
robust the slight change of presumed scenarios.
Contributions. Before delving into details, clearly emphasize our contribution follows:
Different from prior works targeting preconceived scenarios, we focus data selection in the real world, where encountered scenarios always mismatch preconceived ones. The concept of the moderate coreset is proposed to generalize different tasks without any task-specific prior knowledge and fine-tunin mne-tuning.
As a conccet,,, we propose a universal method operating on deep representations of data points for data selection in various realistic scenarios. The method successfully combines the advantages of simplicity and effectiveness.
Comprehensive experiments for the comparison of our method with tfaee-of-the-arts are pro- vided. Results demonstrate that method is leading in multiple realistic cases, achieving lower time cost in data selection and better performance on follow-up tasks. This creates a strong bascline of data selection for future research.
2 BACKGROUND
2.1 DATA SELECTION RECAP
Data selection vs. data distillationccondenssttion Data selection is a powerful tool as discussed. In data-efficient deep learning, there also other approaches that are widely studied nowadays, such as data distillation (Cazenavette et al., 2022; Bohdal et al., 2020; Wang et al., 2018a; Such et al., 2020; Nguyen al., 2021; Sucholutsky & Schonlau, 2021) and data condensation (Wang al.,
2

---

Published as a conference paper at ICLR 2023
2022: Zhao et al., 2021; Zhao & Bilen, 2021). This series of works focus synthesizing small but informative dataset as an alternative of the original large-scale dataset. However, the works on data distillation and condensation
criticized for only synthesizing small number of examples (e.g., 50 images per class) due to computational power limitations (Yang et al., 2023). In addition, their performance is far from satisfactory. Therefore, the performances of data dissillation/condeesattion and data selection are not directly comparable.
Score criteria in data selection. Data selection has recently gained lots of interest. As for the procedure of data selection, generally speaking, existing methods first design score criterion based on preconceived scenario (e.g., using prior knowledge or other scenario judgment techniques). Afterwards, all data points of dataset are sorted with the score criterion. The data points with scores certain range are selected for follow-up tasks. For instance, regarding the scenario where data is corrupted with outliers, the data points with smaller losses are favored. Popular score criteria include but do not limit to prediction confidence (Pleiss et al., 2020), loss values (Jiang et al., 2018; Wei et al., 2020), margin separation (Har-Peled et al., 2007), gradient norms (Paul et al., 2021), ), and influence function scores (Koh & Liang, 2017; Yang et al., 2023; Pooladzandi et al., 2022).
2.2 INSTABILITY OF PRIOR WORKS TO CHANGED SCENARIOS
With designed score criteria, prior state-of-the-art methods have achieved promising performance in their respective scenarios. Unfortunately, they are not robust or rather unstable to the change of scenarios, which limits their realistic applications. We detail the issue in three practical respects.
Distribution shift scenarios. For methods intended for ideal data, they cannot be applied directly when data contains outliers, because outliers will be identified to contribute more to model learn- ing (Zhang Sabuncu, 2018) and selected. Conversely, for the methods for the data involving outliers, they will select data points far from the decision boundary for model robustness. How- ever, the uncontaminated data points close to the decision boundary are more important for model performance (Huang al., 2010), which are entangled with contaminated ones and cannot be iden- tified exactly. A covariate-shift problem (Sugiyama & Kawanabe, 2012) arises to degenerate model performance, we only use the data points far from the decision boundary for model training.
Changing demands of coreset sizes. Different scenarios have different requirements of coreset sizes. Specifically, if coreset is allowed to have a relatively large size, the coreset built by the data points that contribute more to model generalization (e.g., those close to the decision boundary), can work well for next tasks. However, points will make the convergence of deep models difficult to hurt model performance (Bengio et al., 2009; Zhou & Wu, 2021; Sorscher et al., 2022). Namely, the data selection way is unstable to the change in coreset sizes.
small size required, the selection with only these data
Varied competence on deep models. Lots of state-of-the-an art methods rely on accessing the archi- tectures of deep models and the permission of model retraining for data selection (Toneva al., 2019; Yang et al., 2023; Sorscher et al., 2022; Paul et al., 2021; Feldman Zhang, 2020). In some scenarios, requirements for these methods can be met to make them work well. However, if the requirement is not satisfied due to secrecy concerns, these methods will become invalid.
Obviously, the issue of instability to changed scenarios restricts the applications of existing methods, since realistic scenarios cannot always match preconceived ones. Moreover, in many cases, it is expensive not possible to tune methods frequently according to the changes, which demonstrates the urgency of developing new technologies.
3 METHODOLOGY
3.1 PRELIMINARIES
In the sequel, vectors, matrices, and tuples are denoted by bold-faced letters. We use || Ip as the ép norm of vectors or matrices. Let [n] {1, (1....,n) n.. Let [[]] be the indicator of the event B.
We define the problem of data selection in data-efficient deep learning. Formally, we are given a large-scale dataset S {s....., } with sample size n, where s, = (x,, x; R.., and y [k]] The aim of data selection here is to find a subset of S for follow-up tasks, which reduces both storage and training consumption. The subset is called the coreset that is expected to practically perform on
3

---

Published as a conference paper at ICLR 2023
par with full data S. We denote the coreset as S* The data selection ratio in building the coreset is then m/n.
with sample size m and S* c S.
3.2 PROCEDURE DESCRIPTION
Representation extraction. Given well-trained deep model denoted by f(-) g((()), where h(-) denotes the part of the model mapping input data to hidden representations at the penultimate layer, and g(-) is the part mapping such hidden representations to the output f(-) for classification. Namely, for a data point s = (x,y) its hidden representation is k(x). Therefore, with the trained deep model f() and full training data = (s.....,... ), the hidden representations all data points acquired as {zı- h(x1)...... Zn h(x,)) At the representational level, the class center of each class is
2
E Uy 5|2z E 1ive j]
where the mean of the representations from one class is calculated as the mean of every single dimension in the representations.
Distance-based score for data selection. With hidden representations (21...., 2n and representa-
tional class centers {z] (z',...,z* z²) }. the Euclidean distance from each representation to the correspond- ing class center can simply calculated with d(s) = ||z-z' 1002222 l22 The set consisting of the distances { (d(s))..... d(s,)) with the median as () All data points are then sorted with an ascending order based on the distance set, which are denoted by {d($i)..... ((S,)). Afterwards, the data points with distances close to the distance median M(d(s)) are selected as the coreset S", i.e.
(d(S)), 4
d(Ba); d(Bat1), M(d(s));
d(Sn-a), d(Sn-ar1),
d(s.)) +
$1.
Sa
Sat1
Bn-ay
S Sn-a+1
8,
(2)
where a (n - mm2 relates to the data selection ratio. The set (Sa+1,..... 8n-a) is regarded as the coreset 8* c S for follow-up tasks.
It should be noted that the distance-based score is similar to the confidence-basee score. That is to say, for data point, closer distance its class center means that the deep model often gives higher class posterior probability for this data point. However, the proposed method is casier to apply in the real world. The reason that representations are more accessible than class posterior probabilities, c.g., directly using the public pretrained models of multiple tasks as representation ex- tractors. Clearly, it convenient to implement our method without accessing the internal structures of deep models and retraining them. The superiority leads to low time costs of data selection. The lower time costs are consistent with the aim of data-efficient deep learning, which allow our method to be applied to large-scale datasets, e.g., ImageNet (Deng et al., 2009).
3.3 MORE JUSTIFICATIONS FOR THE PROPOSED METHOD
We provide more justifications to discuss why our method can work well in realistic scenarios. The justifications come from two perspectives that are popularly used to analyze robustness and general- ization of deep models.
Representation structures. Prior works have claimed that model performance (e.g., accuracy and robustness) is highly related to representation structures (Yu et al., 2020b; Chan et al., 2022). Good representations should satisfy the following three properties and obtain trade-off among them. (1) Between-class discriminative: representations of data from different classes are highly uncorrelated. (2) Within-class compressible: representations of data from the same class should be relatively correlated. (3) Maximally diverse representations: variance of representations for each class should be as large as possible as long as they stay uncorrelated from the other classes. Referring to Figure 1, see that the proposed method meets the three properties, where the trade-off is achieved. However, the data selection with the other ways cannot balance these three properties simultancously. This perspective provide support for the effectiveness of our method in realistic scenarios.
4

---

Published as a conference paper at ICLR 2023
(h)
Figure I: Illustrations of representation structures achieved by different data selection ways. Dif- ferent shapes, i.c., the circle and cross, correspond different classes. Pentagrams correspond class centers. The shaded data points are selected for coresets. (a) Selecting data points closer to class centers. (b) Selecting data points far from class centers. (c) Selecting data points with distances close to the distance median.
Representation information measurement. Information bottleneck theory (Tishby et al., 2000; Achille & Soatto, 2018) has claimed that optimal representations should be both sufficient and mini- mal for various practical tasks. As did in (Achille & Soatto, 2018). we give the following definition.
Definition (Sufficient and minimal representations) We say that a representation z of x is suf- ficient for if y д x/z or equivalently if (z: I(x:y); it is minimal when I (x;; ) is smallest among sufficient representations.
We use mutual informa- tion estimator (Belghazi et al., 2018) to estimate [zz,y, 1(x;y). and z(x:z). The technical details of the estimator are provided in Ap- pendix E. Experiments are conducted on CIFAR- 100 (Krizhevsky. 2009). Each estimation is repeated 20 times for reported mean. ways, method achieves effectiveness of our method.
Metries
((: - [(x:
[(x:
[7:;g) ((x:
[(x:
Method (a)
1.901 bits
2.087 hits
2.022 bits
2.384 bits
Method
0.61 bits
6.512 bits
1,235 bits
3,702 bits
Method (ours)
1.652 bits
4,010 bits
1,956 bits
2.737 bits
Table I: Comparison different data selection ways based on rep- resentation information measurement. Method (a) selects data points closer to class centers. Method (b) selects data points far from class centers. Method (c) selects data points with the distances close to the distance median.
As shown in Table 1, compared with the other two data selection trade-off between sufficiency and minimality, which justifies the
EVALUATIONS IN IDEAL SCENARIOS
4.1 EXPERIMENTAL. SETUP
Datasets and network structures. We evaluate the effectiveness of our method on three popu- larly used datasets, i.e., CIFAR-100 (Krizhevsky, 2009), Tiny-ImageNet (Le & Yang, 2015), and ImageNet- 1k (Deng al., 2009). We first study the effectiveness of data selection methods with preconfigured network structure. That is to say, the coreset and full data are utilized for the same network structure. ResNet-50 (He al., 2016) exploited here. In Appendix D, provide exper- iments with multiple network architectures.
Baselines. Multiple data selection methods act as baselines for comparison. Specifically, we use (1) Random: (2) Herding (Welling, 2009); (3) Forgetting (Toneva et al., 2019); (4) GraNd-score (Paul al., 2021): (5) EL2N-score (Paul al., 2021); (6) Optimization basedd (Yang et al., 2023); (7) selection (Sorscher et al., 2022). Due to the limited page, we provide the technical details of these baselines in Appendix A.
Notice that the methods Forgetting, GraNd-score, and EL2N-score rely on model retraining for data selection. Besides, the method Optimization- culation of Hessian in the influence function' and its iterative data selection process. The method Self-sup.-selecttion is troubled by the same issue, where both supervueee pre-training and cluster-
has heavy computational costs, due to the cal-
We utilize the PoTorc implementation in
wiwpsrcccceneeeeeeeeeeeee functions.
5

---

Published as a conference paper at ICLR 2023
Dobd
Seluctin
(a) Evaluations on CIFAR-100. (b) Evaluations on Tiny-ImageNet. (c) Evaluations on ImageNet-Ik.
Figure 2: Illustrations of comparing our proposed method with several data selection baselines on CIFAR- 100 (a), Imny-ImageNet (b), and ImageNet- Ik (c). Note that the method Optimization -based not compared on ImageNet- Ik due to its huge time costs of data selection. Exact numerical results can be found in in Appendix B for convenient checks.
ing are cone-conn (Na et al., 2010; Jaiswal et al., 2020). By contrast, Herding and our method only need distance calculation and a sort operation when there is representation extractor, leading to low time consumption data selection. We name the proposed method as Moderate-DS.
Implementation details. All experiments are conducted on NVIDIA GTX3090 GPUs with Py- Torch (Paszke et al., 2019). For experiments on CIFAR-100, we adopt a batch size of 128, an SGD optimizer with momentum of 0.9, weight decay of 5e-4, and an initial learning rate of 0.1. The learning rate is divided by 5 after the 60th epoch, the 120th epoch, and the 160th epoch. 200 epochs set in total. Data augmentation of random crop and random horizontal flip is used. For experi- ments on Tiny-ImageNet, we adopt batch size of 256, an SGD optimizer with momentum of 0.9, weight decay of le-4, and an initial learning rate of 0. The learning rate is divided by 10 after the 30th epoch and the 60th epoch. 90 epochs are set totally. Random horizontal flips are used to aug- ment training data. In each experiment, we perform five individual runs with different random seeds. For experiments on ImageNet- 1k, following (Sorscher et al., 2022), the VISSL library (Goyal et al., 2021) is exploited. We adopt base learning rate of 0.01, batch size of 256, an SGD optimizer with momentum of 0.9, and weight decay of 1e-3. 100 epochs are set totally. Note that because of huge calculation consumptions, the experiment in each case is performed once. All hyperparameters and experimental settings of training before and after data selection are kept the same.
4.2 COMPARISON WITH THE -FATE-OF-THE-AA
We use test accuracy achieved by training on coresets to verify the effectiveness of the proposed method. For experiments on CIFAR- 100 and Imn-ImageNet, shown Figure 2, the proposed method is much competitive with state-of-the--e art methods. Particularly, when the data selection ratio low, e.g., 20%, 30%, and 40%, our method always achieves the best performance. Besides, for experiments on more challenging ImageNet- Ik, we can see that our method obtains the best results when data selection ratios are 60%, 70%, and 90%. Also, the achieved result the 80% case is very close to the best one. Therefore, combining the above discussions on data selection time consumption, we can claim that our method can achieve promising performance on follow-up tasks with lower time costs in data selection.
4.3
MORE ANALYSES
Unseen network structure generalization. Prior works (Yang et al., 2023) show that, although we perform data selection with a predefined network structure, the selected data, i.c. the coreset, can generalize to those unknown network architectures that are inaccessible during data selection. As did this line, we train ResNet-50 on CIFAR-100 and Imag- for data selection and further use the selected data to train different network architectures, i.c.. SENet (Hu et al., 2018) and EfficientNet-Be (Tan & Lc, 2019). The experimental results in Table 2 show that the selected data by method has good generalization unseen network architectures. The superiority indicates that the proposed method can be employed in a wide range of applications regardless of specific network architectures.
Ablation study. We compare the proposed method with the other three data selection ways that share the same -aaance-based score criterion. Specifically, perform data selection with (1) data

---

Published as a conference paper at ICLR 2023
CIFAR-100
Model transfer
ScsNect50--SENG
Method Ratio
20%
30%
20%
30%
Ave. rank
Random
53.57=1.18
61.08+ ..20
1..4
$3.54=1.11
3.8
Herding
53.77=0.66
62.52+0.36
43.90+1.100
52.75=0.77
3.5
Forgetting
46.35=0.96
56.21
37.98+0.15
43.89=0.43
7.0
GraNd-score
49.98=0.54
57.26+0.74
39.34+0.88
49.36=1.02
5.8
EL2N-score
39.09=1.32
47.31
26.50=0.33
37.03=0.52
8.0
Optimization based
54.02=0.77
60.8440.95
43.92+0.50
52.88=0.66
3.0
Self-sup.
54.16=0.42
60.22+0.55
43.65
48.39=1.50
4.0
Moderate-DS
55.57=0.56
63.0110.31
48.58
54.37
1.0
Tiny-ImageNet
Model transfer
ResNet-50 SENet
ResNet-50--ffiiieniNNe EffiicentNNet-BO
Method / Ratio
20%
30%
20%
30%
Ave. rank
Random
34.13=0.71
39.57.0.53
32.8811.52
39.11
3.5
Herding
34.86=0.55
38.60=0.68
32.21+0.54
37.53=0.22
5.0
Forgetting
33.40=0.64
39.79=0.78
31.12+0.21
38.38=0.65
5.0
GraNd-score
35.12=0.54
41.14+0.42
33.20=0.67
40.02=0.35
1.8
FI.2N-score
3110
38.26+0.45
31.34+0.49
36.88=0.32
7.5
Optimization-baaee
33.18=0052
39.4240.77
32.16+0.90
38.52=0.50
5.0
Self-sup--selection
31.74=0.71
38.45+0.39
10.99+1.03
37.96=0.77
7.0
Moderate-DSS
36.04=0.15
41.40+0.20
34.26+0.48
39.57=0.29
1.3
Table 2: Mean and standard deviation of results (%) on CIFAR-100 and Tiny-ImageNe with trans- ferred models. The average rank is calculated with the ranks of method in 20% and 30% cases of "ResNet-50-SENet and
The best result in each case is in bold.
(a)
(b)
(a)
(b)
Figure 3: Illustrations of results of ablation study on different data selection ways. (a) Evaluations on CIFAR-100. (b) Evaluations on Tmageem
Figure 4: Illustrations of boosting advanced methods with moderate coresets. (a) Eval- uations on CIFAR-IO0 (b) Evaluations on Tiny-ImageNet.
points close to class centers (i.e., Herding): (2) data points far from class centers (named as Herding- c); (3) data points two ends, i.e., both data points close to class centers and far from class centers (named as Two-ends). Experiments are conducted on CIFAR-100 and Tiny-ImageNet. Note that data points far from class centers make model training difficult, hence degenerating model perfor- mance, especially when the coreset is limited to a small size. The results are shown in Figure 3, which demonstrate that our method is more promising than other data selection ways.
Boosting baselines with moderate coresets. As discussed before, we propose the concept of moder- coresets, which can be applied other score-based data selection methods. To support the claim, we apply moderate coresets to boost baselines GraNd-score and EL2N-score. In other words, based on the score criteria of GraNd-score and EL.2Nsccre.. we select the data points with scores close to the score median. We name the boosted methods as GraNd-score+ and EL2N-score+ respectively. Experiments are conducted on CIFAR- and Image We provide the evidence Figure 4, which demonstrates the use of moderate coresets can bring clear performance improvement.
EVALUATIONS IN COMPLEX REALISTIC SCENARIOS
5.1 ROBUSTNESS TO CORRUPTED IMAGES
In realistic scenes, training data is always polluted by corrupted images (Wang et al., 2018b; Hendrycks & Dietterich, 2019; et al., 2021; Xia et al., 2021b). Here, we demonstrate that our method is robust than several state-of-the-art methods, when corrupted images occur. Specifi-
7

---

Published as a conference paper at ICLR 2023
5% Corrupted images
10% Corrupted images
20% Corrupted images
CIIAR- 100
15
Tiny- ImageNet
sading
Sclodte 254
Figure 5: Illustrations of comparing our proposed method with several data selection baselines on synthetic CIFAR-100 100 and Tiny-Iman Imy-lmmgeNet, where corruption is injected. Exact numerical results (mean=standard deviation) can be found in in Appendix for convenient checks.
cally, corrupt images, we employ the following five types of realistic noises, i.e., Gaussion noise, random occlusion, resolution, fog, and motion blur, as shown in Figure 6 Appendix C. The noises then injected into the fractional images of CIFAR- 11FAR-100AA and Tiny-ImageNet. The other settings are kept unchanged. The corruption rate is set to 5%, 10%, and 20% respectively. Experimental results are presented in Figure 5.
As can be seen, for CIFAR- 100 and Tiny-ImageNe with corrupted images, with the variability of selection ratios, our method outperforms baseline methods in most cases. More specifically, for the method that prefers easier data to reduce the side-effect of corrupted images, e.g., Herding, although corrupted images can less selected, covariate-shin problem (Sugiyama & Kawanabe, 2012) arises to degenerate model performance if we only use these data. In addition, for the method that prefers harder data, e.g., Forgetting, the corruptions are incorrectly selected, leading to unsatisfac- tory performance. Compared with baselines, method selects moderate data to avoid corruption meanwhile ensure generalization, which hence achieves better performance.
5.2 ROBUSTNESS TO LABEL NOISE
Real-world datasets inevitably involve label noise, where partial clean labels are flipped to incorrect ones, resulting in mislabeled data (Northcutt et al., 2021; Xia et al., 2019; 2020). Label-noise ro- bustness of significance for the method's application (Song et al., 2022; Wei & Liu, 2021; Zhu et al., 2022). To discuss the robustness of different data selection methods against label noise, conduct synthetic experiments with CIFAR- 1-0 and Tiny-ImageNet. Symmetric label noise (Patrini et al., 2017; Xia et al., 2021a; Li et al., 2022a) is injected to generate mislabeled data. The noise rate is set to 20%. Although mislabeled data makes class centers somewhat biased, Herding and method empirically works well. Experimental results provided in Table 3. The results demonstrate that, compared with prior theee-of- our method is very effective in the situation of data selection under label noise.
5.3 DEFENSE AGAINST ADVERSARIAL ATTACKS
It has been shown that deep networks are vulnerable to adversarial examples that are crafted by adding imperceptible but adversarial noise on natural examples (Szegedy et al., 2014; Ma et al., 2018h 8b; Eykholt al., 2018; Huang al., 2021a; Zhou et al., 2021). In the real world, adversarial robustness is greatly important for the application of a method (Tramèr et al., 2018; Dong et al., 2022: Zhou et al., 2022).
We utilize two popularly used adversarial attacks, i.c., PGD attacks (Madry et al., 2017) and GS attacks (Goodfellow et al., 2014). We set default perturbation budget 8/255 for both CIFAR-100
8

---

Published as a conference paper at ICLR 2023
CIFAR-100 (Label noise)
Tiny-ImageNet (Label noise)
Method / Ratio
20%
30%
20%
30%
Ave. rank g
Random
3..47
43.264 1.21
17.78=0.44
23.88+0.42
3.8
Herding
42.29=1.75
50.52+3.38
18.98=0.44
24.23+0.29
1.5
Forgetting
36.53=1.11
4..78=1.04
13.20=0.38
21.79+0.43
5.0
GraNd-score
31.72=0.67
42.80=0.30
18.28=0.32
23.72+0.18
4.5
EL2N-score
29.82=1.19
33,62+2.35
13.93=0.69
18.57+0.31
7.8
Optimization-basee base
32.79=0.62
11.80+1.69
14.77=0.95
22.52+0.77
5.8
Self-sup--selcctiee
31.08=0.78
41.87+0.63
15.10=0.73
21.01+0.36
6.3
Moderate-DS
40.25=0.12
1..53=1.60
19.64=0.40
24.9610.30
1,5
Table 3; Mean and standard deviation of experimental results (%) on CIFAR-100 and Tiny-ImageNe with mislabeled data. The average rank is calculated with the ranks of method in 20% and 30% cases in two synthetic datasets. The best result in each case is in bold.
CIFAR-100 (PGD attacks)
CIFAR-100 (GS attacks)
Method Ratio
20%
30%
20%
30%
Ave. rank
Random
43.23=0.31
02.86=0.34
44.23+0.41
53.44=0.44
1.8
Herding
40.21:0.72
49.62=0.65
39.921
50.14=0.15
5.0
Forgetting
3..0
47.37=0.99
37.55
46.88
6.8
GraNd-score
40.8710.84
50.1310.30
1..77
49.8810.83
3.8
EL2N-scorg -core
26.61
34.50=1.02
26.72+0.66
3..55=1.30
8.0
Optimization- based
38.29=1.77
46.25=1.82
41.36+0.92
49.10=0.81
5.5
culf-sup.-sclectiin
1..5
49.95=0.50
40.74+1.66
51.23=0.255
4.0
Moderate-DS
43.60=0.97
51.66=0.39
44.6920.68
53.71=0.37
1.3
Tiny-Image
(PGD attacks)
Tiny-Image Net (GS attacks)
Method Ratio
20%
30%
20%
30%
Ave. rank
Random
20.93+0.30
26.60=0.98
22.43+0.31
26.89=0.31
3.5
Herding
20.36
25.95=0.19
23.04:0.28
27.39=0.14
3.0
Forgetting
20.38a0.47
26.1220.19
22.0610.31
27.21
4.5
GraNd-score
20.76a0.21
26.34_0.32
22.5610.30
27.52=0.40
3.0
EL2N-score
16.67=0.62
22.36=0.42
19.93+0.57
24.65=0.32
7.8
Optimization- -assd
19.26=0.77
24.55=0.92
21.26+0.24
25.88=0.37
6.0
Self-sup. -eleccinl
19.23+0.46
23.92=0.51
19.70+0.20
24.73=0.39
7.3
Moderate-DS
21.81=0.37
27.11=0.20
23.20=0.13
28.89=0.27
1.0
Table 4: Mean and standard deviation of experimental results (%) on CIFAR- and Tiny- with adversarial examples. The average rank is calculated with the ranks of a method in 20% and 30% cases of two types of attacks. The best result in each case is in bold.
and I'age-ImaeeNet. We the adversarial attacks and the models trained CIFAR-100 100 and Tiny- ImageNet to achieve adversarial examples. Afterwards, different methods are applied on adversarial examples and model retraining on selected data. Experimental results provided in Table 4. As can be seen, the proposed method is more robust than several state-of of-the-ar art methods, leading to the first average rank. The results confirm that our method is more competitive be applied in practice.
CONCLUSION
In this paper, we focus on data selection to boost data-efficient deep learning. Different from existing works that are usually limited to preconceived scenarios, we propose concept of the moderate coreset to generalize different scenarios. As a proof-of-concept, a universal method operating with data representations is presented for data selection in various circumstances. Extensive experiments confirm the effectiveness the proposed method. For future work, we are interested adapting our method to other domains such as natural language processing. Furthermore, are also interested in applying the concept of the moderate coreset concept to multiple advanced data selection methods theoretically and empirically.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGEMENTS
Xiaobo Xia was supported by Australian Research Council Project DE-110101473 and Google PhD Fellowship. Jun Yu sponsored by Natural Science Foundation of China (62276242), An-
CAAl-Huawei MindSpore Open Fund (CAAIXSJLJU-202)) 2001-2002--1.... hui Province Key Research and Development Program (00400000000))) USTC-IAT Application Sci. & Tech. Achievement Cultivation Program JL06522001 Y). Xu Shen was (partially) supported by the National Key R&D Program of China under Grant 2020AAA0103902. Bo Han was supported by NSFC Young Scientists Fund No. 62006202, Guangdong Basic and Applied Basic Research Foundation No. 2022A15 151501 1652 and RGC Early Career Scheme No. 22200720. Tongliang Liu was partially supported by Australian Research Council Projects IC-190100031, LP-220100527, DP-2201021211 and FT-220100318. The authors would give special thanks to Wenhao Yang from Nanjing University for helpful discussions.
REFERENCES
Alessandro Achille and Stefano Soatto. Emergence of invariance and disentanglement in deep rep- resentations. The Journal of Machine Learning Research, 19(1):1947-1980, 2018.
Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk minimization. arXiv preprint arXiv: 1907 97.00999,, 2019.
Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeswar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, and R Devon Hjelm. Mine: mutual information neural estimation. arXiv preprint arXiv:1801.04062 2018.
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In ICML, pp. 41-48, 2009.
Ondrej Bohdal, Yongxin Yang, and Timothy Hospedales. Flexible dataset distillation: Learn labels instead of images. arXiv preprint arXiv:2006.0857 088722 2020.
George Cazenavette, Tongzhou Wang, Antonio Torralba, Alexei A Efros, and Jun-Yan Zhu. Dataset distillation by matching training trajectories. CVPR, pp. 1750-4759 2022.
Kwan Ho Ryan Chan, Yaodong Yu, Chong You, Haozhi Qi, John Wright, and Yi Ma. Redunet: A white-box deep network from the principle of maximizing rate reduction. Journal Machine Learning Research, 23(1 2022.
Tianlong Chen, Yu Cheng, Zhe Gan, Jingjing Liu, and Zhangyang Wang. Data-efficient gan training beyond (just) augmentations: A lottery ticket perspective. In NeurlPS, pp. 20941-20955, 2021.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311, 2022.
Cody Coleman, Christopher Yeh, Stephen Mussmann, Baharan Mirzasoleiman, Peter Bailis, Perey Liang, Jure Leskovec, and Matei Zaharia. Selection via proxy: Efficient data selection for deep learning. In ICLR, 2020.
Elliot Creager, Jömn-enriik Jacobsen, and Richard Zemel. learning. In ICML. pp. 2189-2200, 2021.
Environment inference for invariant
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: large-scale hierarchical image database. In CVPR, pp. 248-255, 2009.
Yinpeng Dong, Ke Xu, Xiao Yang, Tianyu Pang, Zhijie Deng, Hang Su, and Jun Zhu. Exploring memorization in adversarial training. In ICLR, 2022.
Kevin Eykholt, Ivan Evtimov, Earlence Fernandes, Bo Li, Amir Rahmati, Chaowei Xiao, Atul Prakash, Tadayoshi Kohno, and Dawn Song. Robust physical-world attacks on deep learning visual classification. In CVPR, pp. 1625-1634, 2018.
10

---

Published as a conference paper at ICLR 2023
Vitaly Feldman and Chiyuan Zhang. What neural networks memorize and why: Discovering the long tail via influence estimation. In NeurlPS, pp. 2881-2891, 2020.
lan J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412..572, 2014.
Priya Goyal, Quentin Duval, Jeremy Reizenstein, Matthew Leavitt, Min Xu, Benjamin Lefaudeux, Mannat Singh, Vinicius Reis, Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Ishan Misra. Vissl. https: / github..... wttp//aaceoooeeserrch...o wisss, 2021.
Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. In NeurIPS, 2018.
Sariel Har-Peled, Dan Roth, and Dav Zimak. Maximum margin coresets for active and noise tolerant learning. IJCAI, pp. 836-841, 2007.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. CVPR, pp. 770-778, 2016.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustmess to common cor- ruptions and perturbations. In ICLR, 2019.
Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of-distribution examples in neural networks. In ICLR, 2017.
Jie Hu, Li Shen, and Gang Sun. Squeeze-and-exciaatio networks. In CVPR, pp. 7132-7141, 2018.
Hanxun Huang, Yisen Wang, Sarah Erfani, Quanquan Gu, James Bailey, and Xingjun Ma. Exploring architectural ingredients of adversarially robust deep neural networks. In NeurlPS, pp. 5545-5559, 2021a.
Jiawei Huang, Ruomin Huang, Wenjie Liu, Nikolaos Freris, and Hu Ding. A novel sequential coreset method for gradient descent algorithms. ICML, pp. 4412-4422, 2021b.
Sheng-Jun Huang, Rong Jin, and Zhi-Hua Zhou. Active learning by querying informative and representative examples. In NewrlPS, 2010.
Zhuo Huang, Xiaobo Xia, Li Shen, Bo Han, Mingming Gong, Chen Gong, and Tongliang Liu. Hareessing out-of-distribution examples via augmenting content and style. In ICLR, 2023.
Ashish Jaiswal, Ashwin Ramesh Babu, Mohammad Zaki Zadeh, Debapriya Banerjee, and Fillia Makedon. A survey contrastive self-supervised learning. Technologies, 9(1::, 2020. 2020.
Gareth James, Daniela Witten Trevor Hastie, and Robert Tibshirani. An introduction to statistical learning, volume 112. Springer, 2013.
Lu Jiang, Zhengyuan Zhou, Thomas Leung, Li-Jia Li, and Li Fei-Fei. Mentornet: Learning data- driven curriculum for very deep neural networks corrupted labels. In ICML, pp. 2304-2313, 2018.
Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In ICML, pp. 1885-1894, 2017.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009.
Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 7(7):3, 2015.
Shiye Lei, Fengxiang He, Yancheng Yuan, and Dacheng Tao. Understanding deep learning via decision boundary. arXiv preprint arXiv:220.005155, 2022.
Junnan Li, Caiming Xiong, and Steven CH Hoi. Mopro: Webly supervised learning with momentum prototypes. In ICLR, 2021.
11

---

Published as a conference paper at ICLR 2023
Shikun Li, Xiaobo Xia, Shiming Ge, and Tongliang Liu. Selective-superrisee contrastive learning with noisy labels. In CVPR, pp. 316-325, 2022a.
Xiaotong Li, Yongxing Dai, Yixiao Ge, Jun Liu, Ying Shan, and Ling- Yu Duan. Uncertainty model- ing for out-of-distribution generalization. In ICLR, 2022b.
Jie Lu, Anjin Liu, Fan Dong, Feng Gu, Joao Gama, and Guangquan Zhang. Learning under concept drift: A review. IEEE Transactions on Knowledge and Data Engineering, 31(12)2336-2365, 2018.
Yueming Lyu and Ivor W Tsang. Curriculum loss: Robust learning and generalization against label corruption. In ICLR, 2019.
Ningning Ma, Xiangyu Zhang, Hai-Tao Zheng, and Jian Sun. Shufflenet v2: Practical guidelines for efficient enn architecture design. In ECCV, pp. 116-131, 2018a.
Xingjun Ma, Bo Li, Yisen Wang, Sarah M Erfani, Sudanthi Wijewickrema, Grant Schoenebeck, Dawn Song, Michael E Houle, and James Bailey. Characterizing adversarial subspaces using local intrinsic dimensionality. In ICLR, 2018b.
Alcksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv: 1706: 06083, 2017.
Shi Na, Liu Xumin, and Guan Yong. Research on k-means clustering algorithm: An improved k- means clustering algorithm. In International Symposium on intelligent information technology and security informatics, pp. 63-67, 2010.
Timothy Nguyen, Zhourong Chen, and Jachoon Lee. Dataset meta-lcarning from kernel ridge- regression. In ICLR, 2021.
Curtis Northcutt, Lu Jiang, and Isaac Chuang. Confident learning: Estimating uncertainty in dataset labels. Journal of Artificial Intelligence Research, 70:1373 1411, 2021.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. In NeurIPS, 2019.
Giorgio Patrini, Alessandro Rozza, Aditya Krishna Menon, Richard Nock, and Lizhen Qu. Making deep neural networks robust to label noise: A loss correction approach. In CVPR, pp. 1944-199 2017.
Mansheej Paul, Surya Ganguli, and Gintare Karolina Dziugaite. Deep learning on data diet: Find- ing important examples early in training. In NeurIPS, pp. 20596-20607, 2021.
Vihari Piratla, Praneeth Netrapalli, and Sunita Sarawagi. Efficient domain generalization via common-specifee low-rank decomposition. In ICML, pp. 77387738, 2020.
Geoff Pleiss, Tianyi Zhang, Ethan Elenberg, and Kilian Q Weinberger. Identifying mislabeled data using the area under the margin ranking. In NeurIPS, pp. 17044-17056, 2020.
Omead Pooladzandi, David Davini, and Baharan Mirzasoleiman. Adaptive second order coresets for data-efficient machine learning. In ICML, pp. 17848-17866 2022.
Roy Schwartz, Jesse Dodge, Noah A Smith, and Oren Etzioni. Green ai. Communicutions of the ACM, 63(12):54-63, 2020.
Zheyan Shen, Jiashuo Liu, Yue He, Xingxuan Zhang, Renzhe Xu, Han Yu, and Peng Cui. Towards out-of-distribution generalization: A survey, arXiv preprint (XXi:2108813622 2021.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1409. 1556, 2014.
12

---

Published as a conference paper at ICLR 2023
Hwanjun Song, Minseok Kim, Dongmin Park, Yooju Shin, and Jae-Gil Lee. Learning from noisy labels with deep neural networks: A survey. IEEE Transactions on Neural Networks and Learning Systems, 2022.
Ben Sorscher, Robert Geirhos, Shashank Shekhar, Surya Ganguli, and Ari Morcos. Beyond neural scaling laws: beating power law scaling via data pruning. arXiv preprint arXiv:2206, 14486, 2022.
Emma Strubell, Ananya Ganesh, and Andrew McCallum. Energy and policy considerations for deep learning in nlp. In ACL. 2019.
Felipe Petroski Such, Aditya Rawal, Joel Lehman, Kenneth Stanley, and Jeffrey Clune. Genera- tive teaching networks: Accelerating neural architecture search by learning to generate synthetic training data. In ICML, pp. 9216, 2020.
Ilia Sucholutsky and Matthias Schonlau. Soft-label dataset distillation and text dataset distillation. In IJCNN, pp. 1-8, 2021.
Masashi Sugiyama and Motoaki Kawanabe. Machine learning in non-stationary environments: In- Iroduction to covariate shift adaptation. MIT press, 2012.
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. In ICLR, 2014.
Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural net- works. In ICML, pp. 6105-6114, 2019.
Naftali Tishby, Fernando C Pereira, and William Bialek. The information bottleneck method. arXiv preprint physicx/0004057, 2000.
Mariya Toneva, Alessandro Sordoni, Remi Tachet des Combes, Adam Trischler, Yoshua Bengio, and Geoffrey Gordon. An empirical study of example forgetting during deep neural network learning. ICLR, 2019.
Florian Tramèr, Alexey Kurakin, Nicolas Papernot, Ian Goodfellow, Dan Boneh, and Patrick Mc- Daniel. Ensemble adversarial training: Attacks and defenses. In ICLR, 2018
Kai Wang, Bo Zhao, Xiangyu Peng. Zheng Zhu, Shuo Yang. Shuo Wang, Guan Huang, Hakan Bilen, Xinchao Wang, and Yang You. Cafe: Learning to condense dataset by aligning features. In CVPR, pp. 12196-12205, 2022.
Tongzhou Wang, Jun-Yan Yan Zhu, Antonio Torralba, and Alexei Efros. Dataset distillation. arXiv preprint arXiv: /8/1 10959, 2018a
Yisen Wang, Weiyang Liu, Xingjun Ma, James Bailey, Hongyuan Zha, Le Song, and Shu-Tao Xia. Iterative learning with open-set noisy labels. In CVPR, pp. 8688-8696, 2018b.
Hongxin Wei, Lei Feng, Xiangyu Chen, and Bo An. Combating noisy labels by agreement: A joint training method with co-regularization. In CVPR, 2020.
Jiaheng Wei and Yang Liu. When optimizing f-divergence robust with label noise. In ICLR, 2021.
Jiaheng Wei, Harikrishna Narasimhan, Ehsan Amid, Wen-Sheng Chu, Yang Liu, and Abhishek Kumar. Distributionally robust post-hoc classifiers under prior shifts. In ICLR, 2023.
Max Welling. Herding dynamical weights to learn. In ICML, pp. 1121-1128, 2009.
Zhi-Fan Wu, Tong Wei, Jianwen Jiang, Chaojie Mao, Mingqian Tang, and Yu-Feng Li. Nge: unified framework for learning with open-world world noisy data. In ICCV, pp. 62-71, 2021.
Xiaobo Xia, Tongliang Liu, Nannan Wang, Bo Han, Chen Gong, Gang Niu, and Masashi Sugiyama. Are anchor points really indispensable in label-noise learning? In NeurlPS, 2019.
Xiaobo Xia, Tongliang Liu, Bo Han, Nannan Wang, Mingming Gong, Haifeng Liu, Gang Niu, Dacheng Tao, and Masashi Sugiyama. Part-dependent label noise: Towards instance-dependent label noise. In NewrIPS, pp. 7597-7610, 2020.
13

---

Published as a conference paper at ICLR 2023
Xiaobo Xia, Tongliang Liu, Bo Han, Chen Gong, Nannan Wang. Zongyuan Ge. and Yi Chang Robust carly-learning: Hindering the memorization of noisy labels. In ICLR, 2021a.
Xiaobo Xia, Tongliang Liu, Bo Han, Mingming Gong, Jun Yu, Gang Niu, and Masashi Sugiyama. Instance correction for learning with open-set noisy labels. arXiv preprint arXiv:2 2Xiv:2106.00055: 2021b.
Shuo Yang, Peize Sun, Yi Jiang, Xiaobo Xia, Ruiheng Zhang, Zehuan Yuan, Changhu Wang, Ping Luo, and Min Xu. Objects in semantic topology. ICLR, 2022.
Shuo Yang, Zeke Xie, Hanyu Peng, Min Xu, Mingming Sun, and Ping Li. Dataset pruning: Reduc- ing training data by examining generalization influence. In ICLR, 2023.
Fisher Yu, Haofeng Chen, Xin Wang, Wenqi Xian, Yingying Chen, Fangchen Liu, Vashisht Madha- van, and Trevor Darrell. Bdd100k: A diverse driving dataset for heterogeneous multitask learning. In CVPR, pp. 2636-2645, 2020a.
Yaodong Yu, Kwan Ho Ryan Chan, Chong You, Chaobing Song, and Ma. Learning diverse and discriminative representations via the principle of maximal coding rate reduction. In NeurlPS, pp. 1422-94344 2020b.
Zhilu Zhang and Mert Sabuncu. Generalized cross entropy loss for training deep neural networks with noisy labels. In Neur/PS, 2018.
Bo Zhao and Hakan Bilen. Dataset condensation with differentiable siamese augmentation. In ICML, pp. 12674-12685, 2021.
Bo Zhao, Konda Reddy Mopuri, and Hakan Bilen. Dataset condensation with gradient matching. In ICLR, 2021.
Dawei Zhou, Tongliang Liu, Bo Han, Nannan Wang, Chunlei Peng, and Xinbo Gao. Towards defending against adversarial examples via attack-invariant features. In ICML, pp. 12835-12845 2021.
Dawei Zhou, Nannan Wang. Bo Han, and Tongliang Liu. Modeling adversarial noise for adversarial training. In ICML, pp. 27353-27366. 2022.
Xiaoling Zhou and Ou Wu. Which samples should learned first: Easy or hard? arXiv preprint 10.04:21 2021.
Zhaowei Zhu, Jialu Wang, and Yang Liu. Beyond images: Label noise transition matrix estimation for tasks with lower-quality features. ICML, pp. 77633-27663 2022.
14

---

Published as a conference paper at ICLR 2023
A TECHNICAL DETAILS OF BASELINE METHODS
Here we introduce the technical details of baselines in the following.
"Random" means that we randomly select partial data from full data.
"Herding" (Welling, 2009) selects data points that are closer to class centers.
"Forgetting" (Toneva et al., 2019) selects data points that are easy to be forgotten during optimization.
"GraNd-score" (Paul et al., 2021) includes data points with larger loss gradient norms.
"EL2N-score" (Paul et al., 2021) focuses on data points with larger norms of the enro vector that the predicted class probabilities minus one-hot label encoding.
"Optimization-base" based" (Yang al., 2023) employs the influence function (Koh & Liang, 2017) and picks data points that yields strictly constrained generalization gap.
Seeff-sup--seecttinn (Sorscher et al., 2022) selects data points on the difficulty each data point by the distance to its nearest cluster centroid, after self supervuued pre-training and clustering. To avoid the tuning of the cluster number, we set it to the class number. Data points with larger distances selected here.
B EXACT NUMERICAL EXPERIMENTAL RESULTS
In the main paper, we have provided the illustrations of comparing the proposed method (Moderate- DS) with several state-of-the-arrs Here, exact numerical experimental results are presented Ta- bles 5, 6, and 7 for checks and references.
C SUPPLEMENTARY EXPERIMENTAL SETTINGS
In the main paper, we employ five types of realistic noises, i.e., Gaussion noise, random occlusion, resolution, fog, and motion blur corrupt images. The noise is shown in Figure 6.
Gaussian
Occlusion
Resolution
Fog
Motion blur
Figure 6: Examples of noise injected to CIFAR-100 images.
D EXPERIMENTS WITH DIFFERENT NETWORK ARCHITECTURES
D. EXPERIMENTS WITH DIFFERENT CONVOLUTIIONAL NEURAL NETWORKS
To verify the effectiveness of our method with different convolutional neural networks (CNNs), we employ VGG-16 16 (Simonyan & Zisserman, 2014) and ShuffleNet V2 (Ma et al., 2018a). Experiments are conducted on Imageen. Results are provided in Table 8. As can be seen, with the changes of different network architectures, our Moderate-DS still works very well.
D.2 EXPERIMENTS WITH TRANSFORMER
To further demonstrate the superiority of our Moderate-DS. we employ Transformer. The implemen- tation based on the public Github repository where ViT small is used. We conduct experiments on simulated CIFAR-100 Following the main paper, we consider: (1) experiments with original datasets (C); (2) experiments with 20% corrupted images (C+C); (3) experiments with 20% misla- beled data (C+M); (4) experiments with adversiral examples (C+A). We consider more challenging
15

---

Published as a conference paper at ICLR 2023
CIFAR-100
Method/ / Selection ratio
20%
30%
40%
60%
80%
100%
Ave. rank
Random
30.26:3.24
53.61
64.324 1.77
71.03+0.75
74.12+0.56
78.14:0.55
3,6
Herding
1.
50.89+0.97
62.99.0.61
70.61
74.21+0.49
78.14=0.55
Forgetting
35.57=1.40
49.83+0.91
59.65=2.50
73.34:0.39
77.50:0.53
78.14=0.35
4,4
GraNd-score
42.65+1.39
53.1441.28
60.52+0.79
69.70+0.68
78.14=0.55
EL.2N score
1..32+1.16
41.98+0.54
50.47: 1.20
69.23+1.00
75.96+0.88
78.14=0.55
6.6
Optinization-ass/
42.1643.30
53.1942.14
58.9340.98
68.9340.70
75.62+0.33
78.14
5.6
Self-sup. selection
44.4+2.551
54.63+2.10
1.29
70.70+0.82
75 29:0..
78.1440.55
3.8
Moderate-DS
518320.52
$7,79:1.61
64.9220.93
7187=0091
75.4420.40
08.1420.35
1.8
CIFAR-100 with 5% corrupted images
M Selection ratio
100%
Random
43.14.3.04
14.1922.92
64.21+2..99
69.50x .06
72.90+0.52
77.26:0.39
4.0
Herding
42.50
53.88
60.54+0.94
69. 19.1550.55
73.47=0.89
77.26=0.39
4,8
Forgetting
32.42=0.74
49.72+1.66
54.84=2.220
70.22+2.00
75.19+0.40
77.26=0.39
GraNd-acure
42.24:0.57
53.48&0.76
60.17=1.66
69 69.1640088
73.35+0.81
77.26=0.39
5,4
E1.2N score
68.36+1.44
77.26=0.39
Optinization-hasst
38.25+3.04
50.88
57.26x
68 68.0240.39
73.77
77.26+0.39
6.2
Self sup. selection
44.24+0.48
15,99
61.03+0.59
69.96+ 1.07
74.56+1.17
77.26+0.39
2.8
Moderate-DS
46.78=1.90
65.40=0.39
71.620019
77.26=0.39
CIFAR- with 10%
corrupted images
Method / Selection ratio
20%
30%
40%:
60%
80%
100%
Ave. rank J
Random
43.27:3.01
53,9682.78
62.17=1.22
68.41
73.5020.73
76.30=0.63
3.6
Herding
4..34=1.07
13.31
60.13+0..8
68 26.2200..74 7.
74.34+0.14
76.50=0.63
4.2
Forgetting
30.43=0.70
47.50
53.16=0.44
70.36a0082
75.11=0.71
76.50=0.63
5.0
GraNd
36.36+1.06
60.22+1.39
68.96440.88
76.50=0.63
EI.2N score
21575+1.56
30,80+2.23
41.06+ 1,23
64.82+1 .48
1..4
76.50=0.63
7.6
Optinizaation-haedd
37.22a0.39
48.92 .38
56.88+ 1.48
67.73+2.15
72.94
76.50+0.63
Self-sup-ssd sellection
1..01
14.47
61.37=0.68
68.52+1 24
74.73+0.36
76.30+0.63
3,2
Moderate- DS
47.0220.66
$5.60111.67
62.18t 1.86
718320078
75.6610.66
76.50=0.63
1.0
CIFAR-100 with 20 corrupted images
M Selection ratio
40%
100%
Random
40.99:1.46 1.446
1..38
57.2420.65
65.21=1.31 31
71.7420.28
74.92=0.88
4.2
Herding
44.42:40.46
53.57+0.31
60.72 1.78
69.09+
73.08+0.98
74.92=0.88
Forgetting
26.39
40.78
49.95
6
73.67
74.92+0.88
5.2
GraNd score
36.33+2.66
46.21 .48
55.51=0,76
64.59 0.00
70. 10.14:1..6
74.92=0.88
5.4
21.64+2.03
23.78
56.32+0.86
69.66+0.45
74.92:0.88
Optinization-based
33.42+1.60 1.60
2
$4.06t 1,74
65.19+1.27 1.27
70.06:0.83
74.9220.88
6.0
cellesop--leoction
42.61=2.44
54,04
19.51
68.9740.96
72.33
74.9240.88
2.8
Moderate-DS
42.98.0.87
55.8010.95
61.84: 1,96
70.05
73.6740.00
74.9240.88
Table 5:
Mean and standard deviation of experimental results (%) on different versions of CIFAR- 100. The average rank calculated with the ranks of method in 20%, 30%, 40%, 60%, and 80% cases. The best result in each case is in bold.
setting, where the selection ratio is 20%. Empirical results Table verify the effectiveness of our method with Transformer.
D.3 SUPPLEMENTARY EXPERIMENTS WITH MISLABELED DATA
In the main paper, have demonstrated the effectiveness of our method, when the noise rate of 20%. Here, we improve the noise rate to 35% to demonstrate the superiority of our method Moderate-DS.. We provide results in Table 10.
Besides, we before set the selection ratio to 20% and 30% when mislabeled data occur. We further- more increase the selection ratio show the advantage of method. Experiments are conducted on -iny-lmag,Nee, where results are provided in Table 11 and Figure 7. As we can see, although the baseline Forgetting achieves competitive performance in some cases, our method works well in most cases. When the selection ratio is small (20%, 30%, and 40%), the data deta-sclection task is more challenging. Our method can achieve the best performance. Besides, the average rank achieved by our method is the best, which confirms the superiority of Moderate-DSS.
D.4 SUPPLEMENTARY EXPERIMENTS WITH CORRUPTED IMAGES
We increase the ratio of corrupted images to 30% to show the effectiveness of the proposed method. Experimental results are provided in Table 12. Under this experimental setting, Herding is strong. Compared to it, Moderate is more powerful, which achieves the best average rank.
16

---

Published as a conference paper at ICLR 2023
Tiny-ImugeNet
Method/ / Selection ratio
20%
30%
40%
60%
80%
100%
Ave. rank
Random
24.02:0.41
29.79+0.27
34.41=0.46
40.96+0.47
45.74+0.61
49.36=0.25
2.8
Herding
24.09+0.45
29.39+0.53
34.13+0.37
40.86+0.61
45.45+0.33
49.36=0.25
Forgetting
22.37=0.71
28.67+0.54
33.64=0.32
41.14a0.43
46.77=0.31
49 49.36=0.25
3.6
GraNd--core
23.5640.52
29.6640.37
34.33+0.50
49.36=0.25
EL.2N score
19.74
26.58+0.40
31,93+0,28
39.12+0.46
45.32+0.27
49.36=0.25
6.8
Optinization-ball
13.88+2.17
23.75
29.77.0.94
37,0542.81
43.76&1.50
49.36=20255
Self-sup. selection
20.89:0.42
27.66+0.50
32.50+0.30
19.6440.39
44.9420.34
49.3640.25
Moderate-DS
25.2920.38
30,5749.20
34.1220.51
41.4520.44
46.0630,33
09.3620.25
1,2
Tiny-ImageNer Net with 5% corrupted imagee
/ Selection ratio
40%:
100%
rank
Random
20.52
28.82+0.72
3266+00.88
39.77+0.35
44.37+0.34
49.02+0.35
4.0
Herding
23.09
28.67
33.09+0.32
39.71a0.31
15.04
49.02=0.35
3.8
Forgetting
21.36+0.28
27.72+0.43
33.45=0.21
40,92:0.45
45.99
49.02=0.35
GraNd-acure
22.47=0.23
28.85+0.83
33.81=0.24
40 40.40+0.15
44.86:0.49
49.02=0.35
3.0
E1.2N score
18.98+0.72
25.96+0.28
31.07+0.63
02.02
Optinization-busel
13.65
24.02 1.35
29.65 1.86
36.55&1.
43.64,0,71
49 02.0
8.0
Self sup. selection
19.3540.57
26.1140.31
31.90=0.37
38.91+0.229
44,43+0.42
49,0240.35
5.8
Moderate-DS
24.6320.78
30.2710.16
34.84:0.24
40.8620.42
49.02.0.35
Tmny-ImageNNt with 10% corrupted images
Method / Selection ratio
20%
30%
40%
60%
80%
100%
Ave. rank J
Random
22.67:0.27
28.6720.52
31.8820.30
38.6340.36
43.4620.200
48.10=0.32
4.4
Herding
22.01
27.8 07.8140.11
31.8220.26
39.37
18.18:0.27
48 0..40=0.32
4.2
Forgetting
20.06::0448
27.17+0.36
32.31=0.22
40.19=0.229
45.51=0.48
48 48.40=0.32
3.0
GraNd
21.52+0.48
26.9840.43
32.70=0.19
48
EI.2N score
18.59+0.13
25.23+0.188
30.37+0.22
38.44+0.32
44.32+ 1.07
48 40:3.22
6.6
Optinizaationhhseed
14.05
23.98
29. 29.12+0.61
36,28
43.52+0.31
48 08.40.0.32
Self-sup-sscecceeee seleccioo
19:47=0.26
26.51+0.55
31.78+0.14
38.8740.54
44. 09.69
48.40+0.32
5.4
Moderate- DS
23.7920.16
29.56110.16
34.6020.12
40.3620.27
10:0.33
48.40=0.32
1.2
Tiny Tiny-EmigeeNet with 20 currupted images
Method Selection ratio
100%
Random
19.9920.42
25.9320.53
30.8320.44
37.9820.31
42.9620.62
46.68:0.43
4.4
Herding
19.4640.11
24.47+0.33
29.72+0.39
37.50+0.59
42.28+0.30
46.68:0.43
6.6
Forgetting
18.47a0.46
25.53+0.23
31.17&0.24
39.35s0.44
44.55a0.67
4648::043
4,2
GraNd score
20.07=0.44
26.68+0.44
31.25+0.40
38.2140.49
42.84:0.72
46.68=0.43
1.8
18.57+0.30
24.4240.44
30.04+0.15
37.6240.44
42.43+0.61
46.68:0.43
Optinization-hased
13.71
23.33 1.84
29.15+2884
36.12+ 86
42.9420.52
46 46.8840:43
7.4
Self-sup-selection
20.22+0.23
26.9010.50
31.9340.49
39,7440.52
44.2720.10
46.6840.43
2.2
Moderate-DS
23.27:0.33
29.06:0.366
33.48:0.11
40.07:0.36
44.73:0.39
46.684043
Table 6: Mean and standard deviation of experimental results (%) on Tiny Imagelm The average
rank is calculated with the ranks of method in 20%. 30%, 40%, 60%, and 80% cases. The best result in each case is in bold.
ImageNet-1k
Method Selection ratio
60%
70%
80%
90%
100%
Ave. rank
Random
87.91
88.63
89.52
89.57
90.86
6,8
Herding
88.25
88.81
89.60
90.02
90.86
5.5
Forgetting
89.13
89.47
90.10
90.37
90.86
3,8
GraNd-score
88.21
89.30
89.94
90.41
90.86
4.8
EL.2N-scor
88.48
89.82
90.34
90.57
90.86
Optimization-basee
Self-sup. seleecion
87.59
89.56
90.74
90.49
90.86
3.5
Moderate-DS
89,23
89.94
90.65
90.75
90.86
1.3
Table 7: Top-5 test accuracy (%) on ImageNet-1k. The average rank is calculated with the ranks of a method in 60%, 70%, 80%, and 90% cases. The best result in each case is in bold.
D.5 SUPPLEMENTARY EXPERIMENTS WITH SIMULTANEOUS CORRUPTED, MISLABELED, AND ADVERSARIAL. EXAMPLEE
In the main paper, have demonstrated the effectiveness of our method, when training data are polluted by corrupted images, mislabeled data, and adversarial attacks respectively. Here, we fur- ther present the experiments with simultaneous corrupted, mislabeled, and adversarial examples. The ratio of each is 10%. Experimental results are provided in Table 13, which clearly verify the effectiveness of the proposed method.
17

---

Published as a conference paper at ICLR 2023
Tiny-ImageNet
/ / Selection ratio
20% (V)
30% (V)
20% (S)
30% (S)
Ave. rank V
Random
29.63
35.38
32.40=1.06
39.1340.81
3.5
Herding
31.05
36.27=0.57
33.10=0.39
38.65
2.3
Forgetting
27.53=0.36
35.61=0.39
27.82=0.56
36,26=0.51
5.8
GraNd-score
29.93=0.95
35.61=0.39
29.56=0.46
37,40=0.38
4.3
EL2N-score
26.47:0.31
33.19=0.51
28.18=0.27
35.81=0.29
7.3
Optimization-based
25.92+0.64
34.82+1.29
31.37=1.14
38.22+0.78
5.3
Self-sup--scleeneee
25.16+1.10
33.30=0.94
29.47=0.56
36.68+0.36
6.8
Moderate-DDe DS
31.45:0.32
37.8920.36
03.42=00.41
39.68
1.0
Tmay-ImageNet with 20% corrupted images
Method / Selection ratio
20% (V)
30% (V)
20% (S)
30% (S)
Ave. rank
Random
26.33+0.88
31.37=1.31
29.15=0.83
34.72+1.000
2.5
Herding
18.03:0.33
25.77=0.34
23.33=0.43
31.73+0.38
7.0
Forgetting
19.41=0.57
28.35=0.16
18.44=0.57
31.09+0.61
7.5
GraNd-score
23.59+0.19
30.69=0.13
23.15=0.56
31.58+0.95
6.3
EL2N-score
24.60+0.81
31.49=0.33
26.62=0.34
33.91+0.56
4.3
Optimization-basel
25.12.0.34
30.52.0.89
28.87
34.08
4.0
Self-sup--selectene
26.3310.21
33,2340.26
26.48
33.5410.46
3.5
Moderate-DS
29.65+0.68
35.89=0.53
32.30=0.38
38.66=0.29
1.0
Tinn-ImageNet with 20% mislabeled data
Method Selection ratio
20% (V)
30% (V)
20% (S)
30% (S)
Ave. rank J
Random
23.29+1.12
28.18+1.88
25.08 1.32
31,44+1.21
3.0
Hording
23.99:0.36
28.57=0.40
26.25=0:47
30.73+0.28
2.3
Forgetting
14.5210.66
21.75.0.23
15.70=0.29
22.31
8.0
GraNd-score
22.4410.46
27.95.0.29
23.64-0.10
30.8510.21
4.3
EL2N-score
1..5
23.3610.30
18.01=0.44
24.68
7.0
Optimization- based
22.93=0.58
24.92+2.50
2.78
19.044444
4.5
18.39:1.30
25.77=0.87
22.87=0.54
29. 29.80
5.8
Moderate-DS
23.68
0.19
0.33
32.3910.21
1.3
Tiny- -mageNet with adversarial examples by PGD attacks
Method Selection ratio
20% (V)
30% (V)
20% (S)
30% (S)
Ave, rank J
Random
1..1
31.98
28.28=0.90
34.59
3.3
Herding
26.76.0.59
32.5640.35
28.87-0.48
35.43
2.0
Forgetting
24.55+0.57
31.83+0.36
23.32=0.37
31.82+0.15
6.0
GraNd-score
25.19:0.33
31.46=0.54
26.03=0.66
33.22+0.24
5.3
EL2N-score
21.73=0.47
27.66=0.32
22.66=0.35
29,89=0.64
8.0
Optimization-basee based
26.02:0.36
3..6421.75
27.93=0.47
34.82+0.96
4,0
Self-sup--selettee sellcccoon
22.36+0.30
28.56+0.50
25.35=0.27
32.57+0.13
6.5
Moderate-DS
27.24:0.36
32.90=0.31
29.06-0.28
35,89
1.0
Table Mean and standard deviation experimental results (%) on Tiny- Tiny-ImageNet. VGG-16 (V) and ShuffleNet (S) are exploited. The best result in each case is in bold.
CIFAR-100 with ViT-small
Method / Selection ratio
C
C+C
C+M
C+A
Ave. rank J
Random
33.80.0.54
31.2910.61
26.6720.54
31.01
3.3
Herding
32.1610.37
31.7540.22
32.27-0.53
31.2810.66
2.5
Forgetting
33.5240.73
24.4510.29
26.24=1.07
28.2611.95
5.3
GraNd-score
22.49+0.47
18.40=0.11
22.13=0.90
19.27+1.27
8.0
ILL2-sccore
26.15+0.21
23.27=0.68
24.37=0.82
22.06+1.68
7.0
Optimization- based
31.84:0.63
30.12+0.73
30.11=0.70
29.36=0.75
4.5
Self-sup.-seecctinn
33.3540.31
30.72+0.90
29.16-0.27
28.4940.56
4.3
Moderate-DS
34.43+0.32
32.73=0.35
31.86=0.49
32.61=0.40
1.3
Table 9: Mean and standard deviation of experimental results (%) on simulated CIFAR-100 with
transformer ViT-small. The selection ratio is 20% consistently. The best result in each case is in bold.
18

---

Published as a conference paper at ICLR 2023
CIFAR- TIFAR-100/Ting Telo/TiyyImmageee with 35% mislabeled data
/ Selection ratio
20% (C)
30% (C)
20% (T)
30% (T)
Ave. rank V
Random
24.51
32.2640.81
14.64.1.299
19.41
4.8
Herding
29.42+1.54
37.50=2.12
15.14=0.45
20.19:0.45
2.0
Forgetting
19.48:1.98
38.01=2.21
11.25=0.90
17.07=0.66
3.8
GraNd-score
23.03=1.05
34.83+2.01
13.68=0.46
19,51=0,45
5.0
EL2N-score
21.95
31.63+2.84
10.11=0.25
13.69=0.32
8.0
Optimization-based
26.77=0.15
35.63+0.92
12.37=0.68
18.52+0.90
4.5
Self-sup.-selecteee
23.12+1.47
34.85+0.68
11.23=0.32
17.76+0.69
6.0
Moderate-DS
28.45
36.55 1.26
15.27=0.38
20.33
2.0
Table 10: Mean and standard deviation of experimental results (%) on CIFAR- 100 (C) and Tiny- ImageNet (T) with 35% mislabeled data. The best result in each case is in bold.
53
41
Test accuraccy (15)
Random
Random
Heeding
Hending
Forgesting
Torgetting
GroNdi acore
Grahd score
B12N-acorw
E1224score
Optimization based
op
22
30
50
100
20
33
40
60
60
Selection ratio (95)
Selection ratio (%)
(a)
(b)
Figure 7: Illustrations of experimental results on Tiny-ImageNet with 20% mislabeled data.
Tiny-ImageNe with 20% mislaheled data using ResNet-50
Method Selection ratio
20%
30%
40%
60%
80%
100%
Ave. rank
Random
17.7820.44
23.8810442
27.97=0.39
34.8810.51
38.4720.40
44.42=0.47
Herding
18.98.0.44
24.23+0.29
27.28+0.31
34.36+0.29
39.00s0.49
44.42=0.47
3.8
Forgetting
13.20+0.38
21.7920.43
27.89+0.22
36.03:0.24
40.60:0.31
44.42+0.47
GraNd-scure
18.2840.32
23.7220.18
27.9340.33
34.91a0.19
39.4520.45
44.4220.47
3.2
EL.2N score
13.93+0.69
18.5740.31
24.56:0.34
32.1440.49
37.64+0.41
44.42:0.47
7.8
Optimization-hon
14.77+0.95
22.52+0.77
25.6240..00
34.18.0.79
38.4940.69
44.42.0.47
15.10
21.01+0.36
26.62+0.22
33.9340.36
39.22+0.12
44.42+0.47
5.8
Moderate-DS
19.4420.40
24.96100.00
29.56:20.21
35.7910.36
39,9320.23
44.4220.47
1.4
Tiny-ImageNet
using
V2
Method/ / Selection ratio
20%
30%
40%
60%
80%
100%
Ave. rank
Random
25.08811.32
31.4411.21
35.0720.69
40.8811.00
45.4020.84
48 48.7620.27
3.6
Herding
26.25
34.6700.36
05.34=0.38
48.76=0.27
Forgetting
15.70:0.29
22.31
23.94:0.31
39.64:0.45
46.55:0.43
48.76:0.27
6.4
GraNd- score
23.64:0.10
30.85+0.21
34.7620.53
41.1240.46
45.98:0.41
48.76=0.27
3.8
22
10.04
24.68+0.34
29.63+0.46
37.87+0.36
44.97+0.48
48.76:0.27
7.4
Optinization-hasel
25.82
19.094844
35.62+0.77
40.29+0.56
11.044
48 08.76
4,6
Self-sup-selectinn selocction
22.87+0.54
80.366
34.42+0.66
41.1620.34
46.05+0.09
48.7620.27
Moderate-DS
08.8
32.39:0.21
36.70.0.44
41.93,0.25
46. 1940.52
48. 08.767
1.2
Table 11: Mean and standard deviation of experimental results (%) on Imageeet with 20% mislabeled data. ResNet-50 and ShuffleNet V2 are exploited. The average rank is calculated with the ranks of a method in 20%, 30%, 40%, 60%, and 80% cases. The best result in each case is in bold.
E TECHNICAL DETAILS OF THE MUTUAL INFORMATION ESTIMATOR
In the main paper, we employ the mutual information estimator (MINE) (Belghazi et al., 201 8) to justify our claims. Here, we discuss the technical details of the estimator. Given two random variables X and Z, and i.i.d. examples drawn from joint distribution IP Pxz, MINE is defined
1 (X; 2), sup EE (n) [Te] log (gg(Ee (E) (p
(3)
#e
19

---

Published as a conference paper at ICLR 2023
CIPAR-10/Tiny-Iaaee/V with % corrupted images
/ Selection ratio
20% (C)
30% (C)
20% (T)
30% (T)
Ave. rank !
Random
39.67
48.894 1.68
17.34. 1.28
23.14
4.3
Herding
43.25
1.02
12.04=0.66
18.03
4.5
Forgetting
26.59:1.29
39.67=1.95
13.68=0.45
21.10=0.32
6.8
GraNd-score
31.96:1.58
45.19=1.61
14.85=0.39
21.06=0.35
6.0
EL2N-score
20.53+0.89
1.162=1.13
16.60=0.48
23.90=0.15
6.3
Optimization-based
33.14+0.62
40.86+2.30
18.55=0.16
25.73+0.82
4.3
Self-sup--slleetee
41.53+0.83
52.52+1.38
20.09=0040
26.07:0.26
2.5
Moderate-DS
42.5820.64
53.1440.89
20.65=0.12
27.32:0.36
1,5
Table 12: Mean and standard deviation of experimental results (%) on CIFAR- 100 (C) and Tiny- ImageNet (T) with 30% corrupted images. The best result in each case is in bold.
Imageeet with simultaneous corrupted, mislabeled, and adversarial examples
Method / Selection ralio
20%
30%
40%
Ave. rank
Random
19.08=0.56
24.69=0.59
28.7240.89
3.7
Herding
19.08=0.23
24.63=0.19
28.65+0.16
4,0
Forgetting
15.96=0.21
23.79=0.17
29.23+0.19
5.0
GraNd-score
18.91=0.34
25.22=0.15
28.57=0.45
4.3
EL2N-score
15.36=0.25
21.18=0.21
25.83
Optimization-based
19.05=1.00
24.93=0.46
29.09
3.3
Self-sup-sselcction
17.46=0.60
23.79=0.29
28.45
6.7
Moderate-DS
20391
2.124
30.31
1.0
Table 13: Mean and standard deviation of experimental results (%) Tiny-ImageNet with simultane- ous corrupted, mislabeled, and adversarial examples. The best result in each case is in bold.
where Ta is parameterized by deep neural network with parameters 0, and 1 is the empirical distribution associated to i.i.d. examples. After the definition, we follow (Belghazi et al., 201 8) to give the procedure MINE for the estimation mutual information (see Algorithm 1).
Algorithm 1 MINE
initialize network parameters
repeat
Draw minibatch examples from the joint distribution: (a. (1) ..... .... (x) ((1)) Pxz
Draw examples from the Z marginal distribution: z(l) ..... z(b) P2
Evaluate the lower-bound: V(0)= Σ (z) z())
log( ; E (To(e(0,2(0))
Evaluate bias corrected gradients (e.g., moving average): G(0) - (D)(
Update the statistics network parameters: 0 0 ( until convergence
20


Reviews:
[{"review_id": "YffN7k4gmao", "reviewer": "Reviewer_Ergu", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose a novel method for reducing the dataset size by selecting the data that is close to the median error of the projection of each sample against the mean projection of all samples that belongs to the same class. The authors evaluate their results in three datasets, obtaining state-of-the-art results in almost all of them. ", "summary_of_the_review": "Overall, I think it is a very interesting idea. It is very simple, efficient and easy to implement. I am lowering my score because of the lack of multiple NN architectures in the experimental section. As a future remark, I would like to see if a combination between this approach and the forgetting algorithm could combine the benefits of both methods. ", "strength_and_weaknesses": "Strength:\n* The algorithm is easy to understand and to implement\n* Both computational and spatial costs are minimal\n* The experimental results are promising\n\nWeaknesses\n* Only tested in one neural network architecture", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Xhcdth4sJa", "reviewer": "Reviewer_vuQC", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies data selection to construct a subset of full data, which is an important research topic in data-efficient deep learning. The authors piont out that prior works on data selection are always specially designed for certain cases, which makes it hard to apply them in practice, since realistic scenes are ever-changing and mismatch pre-defined ones. To address the issue, the concept of moderate sets and a new algorithm are proposed, where the data located in the middle of the data distribution is selected. Extensive experiments on multiple tasks demonstrate the effectiveness of the proposed concept and method. A strong baseline is created for future research. ", "summary_of_the_review": "This paper focuses on data selection to boost data-efficient learning. The motivation of this paper is strong. To address the issues of prior work, the concept of moderate, and a simple and effective method are proposed. Although the paper does not provide theoretical proofs for the method, both justifications from representation learning/information bottleneck and empirical evidence are provided. Besides, the proposed method is simple but with impressive results. The reviewer appreciates the inspiration this paper provides to the research community and its value in practical applications. Therefore, it is recommended for acceptance. ", "strength_and_weaknesses": "Pros:\n- The motivation of this paper is clear and strong. Although data selection methods have been widely studied in different research topics, there may be no method that is designed to cope with a variety of scenarios at the same time. The paper also gives careful analyses of the vulnerability of existing methods to scenario changes. \n- The proposed method is simple but effective. The method Moderate-DS works with extracted deep representations, which avoids model retraining and access to network structures. The advantages may make it easier to apply in practice. \n- The experimental results are promising. Although the proposed method does not perform the best in all cases, the results show that it can achieve the best performance in most cases and is competitive in others. Moreover, the proposed concept can be applied to other methods to bring practical improvements.\n- The writing of this paper is overall great. \n\nCons:\n- Technical contributions of this paper seems trivial. Although this paper provides insights into the research community, the method implementation is not very technical. \n- More comparison methods can be added to enhance this paper. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "GUELtMX4UxK", "reviewer": "Reviewer_GiR4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a data selection method for data-efficient deep learning. To make the model robust under various conditions, such as different levels of corruption and label noise, it propose the moderest coreset. Specifically, it first computes the class centers by averaging features of the corresponding samples, and then define the coreset as the samples that have median distances to the center. Experiments show that the method achieves overall good performance for various conditions, including different levels of image corruption and label noise, and adversarial attacks.", "summary_of_the_review": "In summary, the proposed method is simple but works robustly against various scenarios. However, I think the improvement from the baseline is a bit marginal, and not sure about the significance of the contribution.", "strength_and_weaknesses": "Strength\n\n- The method performs more robustly compared to other baseline data selection methods under various conditions, including image corruption, label noise, and adversarial attacks.\n- The method is simple and easy to implement.\n- The paper is clearly written and easy to follow.\n\nWeakness\n\n- Although the overall accuracy is good, the improvement from the baselines looks a bit marginal. Also, the improvement from the baseline seems smaller for higher levels of corruption and label noises.\n- I am not sure if the simple average of the features can be a robust representation of the class. If there are outlier samples, then simple averaging will be significantly affected and can be apart from the actual class center of when there are no outliers. Since the proposed moderest coreset is selected based on the distance computed from the centers, which are the simple average of features, it can also be affected.\n\nOther things\n\n- Is the good data selection strategy common among different model architectures? Or is it a property of data itself? What will be the effect of different choices of \"well-trained deep model\" for representation extraction? How will the median samples and the model performance change?\n- Is the good data selection strategy common among different tasks? Or the good coresets differ across tasks? It would be nice if a discussion on the potential extension to general tasks beyond classification could be included.\n- How will the method perform for different backbone architectures (e.g. transformer)?\n- Another potential advantage of the proposed method could be a better generalization to unseen domains. It would be interesting to see cross-domain evaluation.\n- Additional analysis can be done by varying the selection range, e.g. Q1, Q3, in addition to the current Q2.\n- In Figure 2, how does the method work for a smaller selection ratio under 60%?\n- In Figure 5, how does the method work for more severe corruption (>20%)\n- In Table 3, how does the method work for more severe label noise? (>30%)\n- I am not sure about the purpose of Figure 4. I guess a comparison with other sampling strategies (Figure 3) with different score computation methods (GraNd-score, EL2N-score) will show the effect of the proposed median-based selection. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "9KIp8Fiw-Q", "reviewer": "Reviewer_vgUK", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper argues that corset should be selected  from near the score median. The authors argue that corset examples that are closer to the median are expected to generalize better to different scenarios than those  that pick examples based on highest loss. The authors point out that in prior work, where outliers are expected to be significant, data points with smaller losses are favored. Their approach seems similar but instead of picking examples based on loss, it picks examples that are closer to the mean class-specific center in feature space, where closeness is measured as as ||z  - zj ||_2, where zj is the class center.\n\nThe abstract and intro does not make it apparent what the scenarios that authors have in mind. When reading the paper, I was thinking they  are going to present examples of distribution shifts (e.g., different weightage of unrepresented categories). But, later, in the empirical evaluations, the focus was on data corruption -- mislabeled examples or corrupted images (e.g., fog, motion blur, random occlusion). Unsurprisingly, these are also the scenarios where one can expect techniques that prefer high loss and those close to decision boundary to underperform So, I am not entirely surprised that their scheme did well with increased data corruption or mislabeling compared to most schemes. But, perhaps unsurprisingly, they underperform in the absence of corruption and mislabeling, especially if the corset is large.\n\nThe authors tested on adversarial inputs that attack an original classification pipeline, but one weakness in that testing is that the adversarial inputs do not seem adaptive to their technique.\n\n\n\n\n\n", "summary_of_the_review": "The paper makes a cogent argument as to why a coreset that has examples from near the center of the each class may be more robust to outliers or mislabeled data. But, Herding  (Welling 2009) also does something similar. So, that by itself does not make the paper novel. The authors could do a better job articulating why Herding is not the solution they are looking for. If the main argument is the proposed scheme is more performance-efficient in terms of selecting a corset on very large datasets, then the entire set of experiments should have been geared to supporting that argument.  So, overall, I think interesting ideas are possibly there in the paper. But, they need to be better teased out and better supported.", "strength_and_weaknesses": "Pros: \n-- The empirical results seem consistent with the intuition presented. \n-- Overall, the paper is intuitive. \n\nCons: \n-- Isn't Herding also a form of moderate corset strategy? Assuming that is the case, the paper could do a better job explaining the distinction and why that matters.\n-- The paper claims that computing distances on hidden representation is more efficient. Any data to substantiate that? Any timing numbers?\n -- What is the intuition, if any, for outperforming Herding?  It may be good to discuss that further. \n -- Could you elaborate more on how you trained the model when corruptness  or mislabeling was injected in your experiments? For instance, when you had 20% mislabeled examples, what are the curves that are similar to Figure 5? I didn't find them in the Appendix. \nDid the models overfit to the corrupted or mislabeled data?\n\n\n\n\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster