Sample Importance in SGD Training
Authors: Alessio Quercia
Hanno Scharr
Ira Assent
Venue: Submitted to ICLR 2023 (2023)
Keywords: Deep Learning
Sample Importance
Hard Example Mining

Abstract: Deep learning requires increasingly bigger models and datasets to improve generalization on unseen data, where some training data samples may be more informative than others. We investigate this assumption in supervised image classification by biasing SGD (Stochastic Gradient Descent) to sample important samples more often during training of a classifier. In contrast to state-of-the-art, our approach does not require additional training iterations to estimate the sample importance, because it computes estimates once during training using the training prediction probabilities. In experiments, we see that our learning technique converges on par or faster in terms of training iterations and can achieve higher test accuracy compared to state-of-the-art, especially when datasets are not suitably balanced. Results suggest that sample importance has intrinsic balancing properties and that an importance weighted class distribution can converge faster than the usual balanced class distribution. Finally, in contrast to recent work, we find that sample importance is model dependent. Therefore, calculating sample importance during training, rather than in a pre-processing step, may be the only viable way to go.

Under review as a conference paper at ICLR 2023
SAMPLE IMPORTANCE IN SGD TRAINING
Anonymous authors Paper under double-blind review
ABSTRACT
Deep learning requires increasingly bigger models and datasets to improve gener- alization unseen data, where some training data samples may be more informa- tive than others. We investigate this assumption in supervised image classification by biasing SGD (Stochastic Gradient Descent) to sample important samples more often during training of classifier. In contrast to state-of-the-an does not require additional training iterations to estimate the sample importance. because it computes estimates once during training using the training prediction probabilities. In experiments, we see that our learning technique converges on par faster in terms of training iterations and can achieve higher test accuracy compared to state-of-the-ar,, especially when datasets are not suitably balanced. Results suggest that sample importance has intrinsic balancing properties and that an importance weighted class distribution can converge faster than the usual bal- anced class distribution. Finally, in contrast recent work, we find that sample importance is model dependent. Therefore, calculating sample importance during training, rather than in pre-processing step, may be the only viable way to go.
our approach
INTRODUCTION
For many gradient gradient-descent--asee models increasing the model and training data sizes boosts the model performance and their ability generalize. However, the increase in model and training data sizes comes with ever higher computational costs: longer training times and greater energy consumption are required to train model on a given training data. One way to reduce these costs is to optimize the model training procedure. The most common training approach relies on random shuffling without replacement of the data samples during training for a given amount of epochs. As consequence, all the samples are seen by the model the same number of times and are therefore implicitly treated as equally important.
Recent works on hard example mining (Felzenszwalb et al. 2009 Loshchilov & Hutter 2015 Simp.
son| [Alan Alain et al. [2015] Shrivastava et al. [2066] Chang et al. [2017] Katharopoulos
Fleuret, @018|Ariiaa & Valdenegro- Toro| 2020) Pruthi et al. 2020. Lin et al. (017] and coresets selection
Mirzasoleiman et al.) 2020) Killamsetty et al.) 2020;
2021 Yoon et al. [202 Balles et al., 2022)
have shown that the training samples are not equally important to learn a given task (Katharopoulos
& Fleuret| 2018) and that is possible to speed up training by respectively focusing on hard samples or subsets of the training dataset that best approximate the full gradient. These methods estimate sample importance during training in an online fashion and leverage this information to speedup the learning process. However, they often require an additional computational overhead compute the sample importance, which makes them less effective in practice. This is particularly true in the case of coresets selection methods, which are based on conservative estimates (Paul et al. 2021
Another research line has shown that the training samples in dataset can be ranked according to different importance scores (Feldman & Zhang 2020: Feldman 2020 Jiang et al., 2020 Toneva et al. 2018 Paul et al. 2021) and that less important samples can be pruned prior training the model with little to no loss in test accuracy. Since computing sample scores exactly is computationally unfeasible, these methods rely approximations that usually require to fully train at least one model. compute and rank the sample scores, and then select the smallest subset of samples best approximating the test accuracy achieved by model trained on the full training dataset.

---

Under review as a conference paper at ICLR 2023
Our work is in between hard example mining, as assume sample difficulty to be related to sam- ple importance, and data pruning. as we compute sample importance only once after few training epochs, instead of adapting them during training as in hard example mining coresets selection.
Motivation There is increasing interest understanding the importance that training samples have on model generalization to unseen data, in order to boost the model performance by pruning, downweighting or downsampling less important samples. However, we find that existing methods generally face main challenges. On one hand, they impose some computational overhead due to the computation of the samples' importance, On the other hand, they often rely on many parameters select which samples to use during training to boost the model performance. We address these challenges by proposing a simple, yet effective learning method that allows a model to estimate sample importance early in training and to switch focus on more important samples.
Our method is closely related to the data pruning method by Paul et al. (202T). which requires to train 10 model instances for 20 epochs estimate the importance scores that are then used train new model instance from scratch. This method uses four parameters: two compute the sample scores (the number of models and the training epochs) and two to slide a window over the scores to select the subset of samples that best generalizes on the test data (window size and position).
In contrast, our method does not require additional training iterations to compute sample importance, as it is computed only once using the predicted probabilities of single model after a few training epochs. Then training continues by sampling important samples more often. Furthermore, our method has only two parameters: epoch E at the end of which compute the scores and a focusing parameter y (Lin et al. 2017). which we fixed as constant for all the experiments.
Contribution We propose SGD-based method, which pretrains model for few epochs under the assumption that all training samples are equally important for generalization, then it focuses on samples that are more important to learn the given task. In experiments we observe that
our method can identify sample importance with reduced computational cost and converges faster to comparable even better solution compared to state-of-the-au approaches;
sample importance evolves a model-specific way during training;
sample importance can automatically balance the class distributions, and sampling by class im- portance can be more beneficial than balancing the class distributions by number of samples;
our method allows to use multiple augmentations more efficiently than the other baseline methods.
TRAINING OPTIMIZATION
Standard training of deep learning models makes the implicit assumption that each training sample is equally important in a uniform scan through all samples in every epoch. Each sample is used exactly hypothesize that sample can
the number of times (i.e. the number of epochs). In contrast, be more or less important for a given model to learn a given task, and that we can estimate sample importance during training to speed up learning by focusing on more important samples.
Our core idea is to capture sample importance as sample difficulty; the harder a sample the more model can learn from it improve generalization. On the other hand, a sample that can be easily classified can be considered already learned and, therefore, is less informative.
Problem formulation Let S = (z,, Mi) be a training dataset with M samples, V= D1 y:) a test dataset with samples, where x. are images and y. one-hot encoded class labels. Let further fo(.) be model (e.g. a neural network) that can be trained using a gradient descent optimizer (e.g. stochastic gradient descent, SGD), 0 its parameters, So (x:2) the output probabilities after softmax activation. Additionally, let N be the number of training epochs, B the training batch size. T. [M/B] the number of batches (or iterations per epoch) and T = T the overall training iterations (i.e. mini-batch gradient updates). Moreover, let
(1) be the standard training sampler that uniformly scans through all the samples at every epoch without replacement (Chang et al. 2017), where indicator function and S, is the set of samples
S(i] S) 1/([S] - | Ligs.
2

---

Under review as a conference paper at ICLR 2023
already selected during the current epoch. Let L be training loss function (here ((fo(r)). v.) -y Y4 log(fo((z) (2)))) ). i.c the Cross Entropy loss).
The gradient step at iteration t can be computed as
B Σ
1 = B ==1
((e.
(2)
with learning rate 7 and (6) sampled from S using the sampler P(i|Se, 5). Furthermore, let A(V, fo. (.)) be the accuracy, i.e. the fraction of correctly predicted samples, of model fa. after t training iterations on all test samples (z,, y4) E V. Then, let A, = max, (AV,, (V. fe. ())) be the highest test accuracy achieved by the model f.).), and t,, arg max, max, fu, (.)) be the training
iterations needed to reach A, using standard sampling (I). The "speddupp factor' of a training method
yielding model for (-) (note the indicating the other training method) is given fs/bn, if it reaches baseline accuracy As at tn iterations, given tolerance E:
fn
min(t) 15
s.t. A(V, fo..o..))) (.) A.-
Informally, given a training and test set, a model, optimizer, and baseline maximum accuracy, we search for the baseline maximum accuracy with the shortest amount of training iterations. Note that we choose accuracy as a generalization metric, but 33 can be extended other metrics.
SGD WITH SAMPLE IMPORTANCE
We hypothesize that a model can learn given task faster or reliably by focusing important samples in a given dataset. To find sample importance, we leverage the concept of focus during training used for Focal Loss (Lin al. (017), but from a different perspective: instead of regulariz- ing the loss based on sample difficulty (1 fo (2x) (compare (5), we define sampler P where we assign each training sample (I4, 31) weight wn of being selected according to its difficulty:
Piilfor: 8)
W;
with (1-
- fex (1:)y.))
and (1), E S
(4)
Σ/=
P uses replacement and ensures that already learned samples (easy samples) are downsampled, whereas samples that are still to be learned (hard samples) are oversampled. log (2a)9; is the prob- ability that model feg classifies the training sample x7 in the correct class y4. Jög is the model pretrained using the uniform sampler without replacement [ for E epochs. Lastly, y is the focus- ing parameter (Lin et al. 2017). that regulates the focus to put on hard examples. Specifically, 7 emphasizes hard samples, Y < O emphasizes easy samples and y 0 treats all training samples as equally important. Selecting the number of pretraining epochs E appropriately
important, as too high É' reduces efficiency, and small E might lead unreliable sample importance weights wy. To select E we take inspiration from the data pruning Pau Paul et al. (2021). 1), where sample scores are computed after E training epochs. However, differently from their work, we estimate sample importance using a single model instance (instead of 10) and do not train from scratch once the sam- ple importance obtained. By doing so, we ensure that no additional training iterations are required to estimate sample importance.
Furthermore, our method differs from other approaches that use samplers based on sample difficulty (Loshchilov & Hutter 2015): Chang et al., (017)) since we use the sampler described in 4 and only compute sample importance once, after epoch E, then using them until the end of training. Pretraining techniques have already been used in other approaches to start updating the training sampler once the model predictions are stable and is also named burn-in (Chang et al. 2017). warm- start (Killamsetty et al. 2022 2021) 1) or warm-up (Katharopoulos & Fleuret 2018).
We propose an SGD-based method biased towards important samples, which can be summarized as follows:
1. Pretrain a model f on all training samples for < N epochs using the standard sampler (1.
2. At the end of epoch F,, use the model predictions to update the sampler as described in 4
3. Train f for N F epochs using the new sampler, i.e. focusing more important samples.
3

---

Under review as a conference paper at ICLR 2023
4 EXPERIMENTAL EVALUATION
We compare our sampling method against the following baselines and competing methods:
Uniform sampling without replacement The standard deep learning pipeline trains model by uniformly scanning through all the samples at every epoch. Formally, this translates into a training sampler according to IT. We denote this method as SGD-Scan (Chang al. 2017).
Focal Loss One approach to focus training to hard examples is Focal Loss (Ln et al. 2017). which adds a regularization term to the standard Cross Entropy loss
(z.) y.) aa (1 fo (z))))
)" 2 log log(f((
(5)
where Oy is the class weight for the class Yi. (1 - fo (z;)): the regularization term used to em- phasize hard examples. The same term is used as weights w; (4). Note that here the loss of easy samples is downweighted, but all training samples are considered equally important from the point of view of the sampler. Our method does the opposite: it samples hard samples more often than easy samples, but uses the standard Cross Entropy as loss function. Additionally, our method computes sample importance once only (at the end of epoch E), whereas in Focal Loss it is computed in an online fashion (i.e. for every minibatch).
Data pruning with EL2N scores Data pruning emphasizes training samples by ranking them according importance metric, pruming less important ones, and training a sampler (I) uniformly scanning through the remaining ones. We compare with the recent data pruning method by Paul et al. ). here denoted E1.2N. This computes sample scores 8; after F = 20 training epochs as average of k = 10 different model initializations, as 8; = E [I){(;;) yı l2l1. vector. Once the scores ranked, 30% of the worst scores pruned for CIFARI0 10 and 10% for CIFAR100, as these pruning percentages are reported as having best test performance (Paul al. 102T
model using
Le. the expected L2 norm of the
SGD sampled by prediction variance We also compare against a sampling method based on sample prediction variance (Chang al.] (017), which emphasizes samples with high prediction variances with the following training sampler
+ ear (=)))==- 1/2 +
H. S) x
(6)
where prediction variance var( var()o,
estimated on the history of stored prediction proba- bilities H for sample until iteration - 1, e constant that prevents low variance instances from never being selected again. We denote this method SGD-SPV.
SGD sampled by prioritization Lastly, we compare against the two sampling methods described in Schaul et al. (2015). The general sampling equation is P(i),, S) / PG. where 5 are the sample losses, and P; = |8,1 + e for proportional prioritization (here denoted SGD-Prop) and p; =irank((i) for rank-based prioritization (here denoted SGD-Rank), where rank(i) represents the rank of sample sorted according to (5%].
4.1 COMPARATIVE EVALUATION
We compare our method the approaches above using CIFARIO, and CIFAR100, or variations by augmentations or omissions (see below and Appendix A for details). For our method, we set focus parameter y 0.5 for all experiments as preliminary experiments indicate it is stable and suggested in the original work (Li et al. 2017) as best performing (further discussed in Appendix A.1.5).
performant parameter setting. For Focal Loss 5). we use 7
Figure shows test accuracy versus training iterations for ResNet50, trained CIFAR 10 and CI- FAR 100, respectively, for baseline and competitor approaches and our novel scheme. In both cases the spread between the maximum accuracies reached by the different training methods is ss 0.005. Results are summarized in Table For CIFARIO (Figure[), we observe that standard training SGD- Scan performs better than Focal Loss, on par with SGD-SPV and slightly worse than SGD-Propp but
4

---

Under review as a conference paper at ICLR 2023
OFAROO Reshee50
CIMALLOO
0.0550
56C-San
E129
5CD Rank
560-5PV
0.75
E126
560-299
Gum (E-25
0.9525
0.75
0.0300
0.9475
Test
0.9450
Test
0.76
0.9125
0.73
0.900
0.74
75000
50:00
Training
Training iterat
Test accuracy of ResNer50 on CIFARIO (left) and CIFAR100 (right) of baselines, com- petitors and our approach with ? = 0.5 and variable E. Solid lines indicate the mean, shaded regions 16th and 84th percentiles of 4 independent runs (i.e. different random seeds). The horizontal red dashed line average maximum accuracy by standard training (SGD-Scan), the vertical dashed lines the first iteration in which a method on average reaches that maximum accuracy.
Figure 1:
Table Maximum mean test accuracy achieved by all methods and respective standard deviation.
Data
SGD-Scan
EL2N 1
Focal Loss SGD-SPV SGD-Rank SGD-Prop
Ours
CIFARIO 95.1 + 0.1 95.2 + 0.1
94.8 = 0.1 95.0 = 0.1 95.1 1 0.1
95.1 + 0.3
95.2 + 0.2
CIFARTO0 77.8 + 0.2 77.8 + 0.6
77.5 0.6 77.9 + 0.2 77.4 0.4 76.3 0.9 78.1 + 0.4
clearly outperformed by SGD-Rank, EL2N and our approach. EL2N and our method achieve the same maximum performance, but since EL2N induces precomputation overhead, it is considerably slower than our method. For CIFAR 100 the situation different. On average. EL2N cannot improve against SGD-Scan, whereas our method can (in particular with E=1). Moreover, EL2N shows high variance over the different training runs (shaded regions in Figure D.. Consequently, while mean maximum performances of EL2NN and Ours clearly differ, this is not the case for maximum perfor- mances over all runs. conclusion our method reliably delivers on par or better performance than EL2N, and is faster when taking into account the precomputation required by EL2N.
In Figure D we can also see that the average peak accuracy A, of standard training SGD-Scan (red dashed horizontal line) is reached earlier by EL2N (black dashed vertical line) and Ours (blue dashed vertical line) for CIFAR10, and earlier (Ours) or later (EL2N) for CIFAR100. However, Figure does not take into account the additional training iterations required compute the EL2N scores, which are 7800 per model instance, for total of 78000. We take this into account in Figure where investigate the speedup behaviour of the different methods and if it is possible trade slight loss of accuracy for additional speedup. Note that we only add 7800 training iterations to the overall iterations, instead of 78000, as the 10 model runs can be parallelized. However, in terms of energy consumption, those iterations are still additional overhead that our method does not require. In the figure, we show at which iteration each training method reached the maximum accuracy of standard training SGD-Scan minus a tolerance c; and derive the speedup factor as the quotient of the two iteration numbers 4n/44, (see (3). In all cases our method yields top speedup for almost all e, only slightly surpassed by EL2N (dashed), but please recall that our figures ignore the overhead EL2N has for precomputing scores. Including the necessary pre-computations for EL2N in the analysis, method is clearly faster.
4.2 SAMPLE IMPORTANCE MODEL DEPENDENT
Recent works on data pruning (Feldman & Zhang 2020 Feldman 2020 Jiang et al. 2020 Toneva
et al., 2018 Paul et al. 2021 compute descriptive sample importance scores for given dataset according to different metrics. Despite some suggestions model and learning parameters depen- dency, no explicit empirical study is performed on model dependency of such scores. Paul et al.)
It requires 78000 additional iterations excluding hyperparameter tuning (the equivalent of an entire run).
It refers Ours E=l. We omitted Ours F=10 (77.9 + 0.3) for space limitations.
5

---

Under review as a conference paper at ICLR 2023
CIARRO ResNet50
OIFALLOO Restett5)
200-Son
520-ccm prosomp
16
p
Foodl Loes
Feell Loet
1.5
rd0 Pro
5dd Prpp Our (te1)
Dun (Ex10)
1.3
Berations
1.2
11
10"2
10
10
10
10"4
Tolerance
Telerance
Figure 2: Speedup factor to/tn wrt. maximum accuracy achieved on average with SGD-Scan, re- laxed with varying tolerance (from O to 0.01). Results of ResNet50 on CIFARIO (left) and CI- FAR100 (right) for baselines, competitors and our approach with 7 = 0.5 and variable F.
(2021) even suggest that the scores are a property the data, not of the model or task. By contrast, we provide empirical evidence that sample importance is not independent of model architecture (Toneva et al. 2018 Jiang et al. 2020). For assessment, we leverage the concept of class impor- tance wº aggregating sample importance by grouping sample weights wy (see (4) by ground truth class Ve. i.e. th E, We W Li=1 W/-- We model architectures with increasing number of parameters: LeNet (LeCun al. 1989). PyramidNetl 10 (Han et al. (017), GoogleNet (Szegedy et al. 2015). and ResNets 18, 34, 50, 10T eH al. 2016) (more details in Appendix A.1.3) Figure shows that class importance weights change depending model architecture and size. However, importance weights derived from different models are correlated; the overall mean rank correlation 0.77. E.g. we observe that all model architectures agree on the hardest (cat) and easiest (automo- bile) classes to learn, but GoogleNet for which automobile is second easiest after ship. Other classes are ranked differently depending on the model. For instance, the second hardest class dog for PyramidNet1 10 and ResNet101 (and EL2N), bird for the other models. Class importance ranking induced by the scores agrees on the hardest classes, but not on the easiest one, which is truck before pruning and becomes horse after pruning.
Notice that here we report weights computed the best tested epoch E. for each model. In Ap- pendix B.I we report weights computed at epoch E = 10 for every model, further showing that sample importance evolves in a model-specific way during training.
In other words, every model induces its own 'ideal' sample importance when learning new task, but models agree to some extent. Therefore, it seem advisable that each model estimates its respective sample importance during learning and prioritize samples individually. As consequence, pre- computed sample importance scores may only be reliable when adopting the same (or at least a similar) model the used to compute them.
Note that by the same reasoning it is likely that sample importance also depends on the learning hyperparameters (optimizer, learning rate, schedules, etc.) as they shape model learning (Pruthi et al., 2020 Jiang et al., 2020), which we do not explore in this work. However we further discuss sample importance properties in Appendix B among which task and initialization dependency.
4.3 SAMPLE IMPORTANCE HAS INTRINSIC BALANCING PROPERTIES
As mentioned the end of Section 4.1 our method works best when there a difficulty imbalance among the samples of a dataset. To further validate this hypothesis, we create such imbalance on CIFARIOR 100 where 20% of the classes are randomly selected (once for all different runs) and augmented A times (see Appendix I. for details).
In this setting, Figure shows that our method converges faster and to a better solution with respect to the standard baseline. In particular, our method sacrifices little accuracy the augmented classes to focus more on the non-augmented ones, which contain less samples, therefore improving the accuracy them (see also Appendix B.4). This suggests that our method automatically balances the class distributions by sampling more often samples that appear less, considering them more important for training. Note that observe very similar results on ImageNet-1K (see Appendix C

---

Under review as a conference paper at ICLR 2023
pwd.00
Pyrnssseetin,
Redkent& RndMerkt, ReskerS2 poddd Redtvete), pH62
FLOM
0.33
0.2
0.00
Figure 3: Class weights implicitly learned by LeNet, PyramidNet1 10, GoogleNet, ResNet 88, ResNet34, ResNet50 and ResNet101 CIFARIO using method for 4 different initializations (error bars) and class weights induced from EI.2N scores before and after pruning 30% of the sam- ples with lowest scores. Captions on top give average Spearman rank correlation p of sample weights over the initializations for the architectures, whereas those on bottom give the number of parame- ters and the epoch E.
CIARIO0 20% classes) Rester50
CHRRR100
Restie:50
0.8
0.8
0.6
0.6
0.5
0.5
0.3
0.3
5CD Stan
Oura (0-105 Soar
SCD-Saa
Oun (D-20):
0.1
Gar (e-12)
0.1
GU0G0
50000
Training
Training
Figure 4: Test accuracy of all (solid lines), augmented (aug, dashed lines) and non-augmented classes (others, dotted lines) of ResNet50 on CIFARI00 using standard training (SGD-Scan) and our method, when the 20% of the classes are randomly selected and augmented 5 (left) and 20 (right) times.
A balanced class distribution is not necessarily the best option Methods like SMOTE (Chawla al. (2002) try to balance the class distribution tasks where class imbalance negatively affects model performance. We hypothesize that a balanced class distribution is not necessarily the best option and that classes may be weighted according to their importance for learning. In Figure 5 we compare the performance of (i) evenly balanced classes, i.e. a constant number of samples per class (GDD-ccan,, (ii) our sampling method, and (iii) classes sampled according the class importance weights (SGD-SCI, cmp. Figure extracted from the sample weights computed by our method. For both datasets, SGD-SCI on average performs better than the baseline SGD-Sean. For CIFAR 100, it performs slightly better than our per-sample importance sampling method in some iterations, but it converges to comparable results towards the end of training. Therefore, Figure 5 shows that given task by sampling based on class importance.
is possible to improve model performance However, we conclude that assigning weights or changing the number of samples in order to achieve an importance-balancee dataset should not be done on the basis of classes, but considering finer grained measures or statistics, here done per sample. Intuitively this makes sense, when considering that class may consist of multiple complicated subclasses, that need some extra care in comparison to 'simple' classes.
4.4 SAMPLE IMPORTANCE ALLOWS TO RFFICIENTLY USE MULTIPLE DATA AUGMENTATIONS
Lastly, we hypothesize that our method can use multiple augmentations more efficiently than the other baselines, since it should be able oversample the most important samples and downsample all the others. We therefore expect that our method can quickly identify which augmentations are most useful to leam the given task.
We test this by designing an experiment where the original dataset is augmented 5 times, resulting in bigger training dataset (250000 training samples for both CIFAR10 10 and CIFAR 100). Note that
7

---

Under review as a conference paper at ICLR 2023
Reshir50
200-Sann
Gut (C-38)
55950
520-5eeh
Gum (C-33
520.0
0.032
0.780
0.948
0.775
0.946
0.770
0.942
0.910
0.750
Training iterations
Training iterations.
Figure 5: Test accuracy of ResNer50 on CIFARIO (left) and CIFARI 100 (right) using the baseline approach where classes are balanced ISGD-Scam our sample importance approach with variable E and 7 = 0.5, and our class-importance sampled method (SGD-SCI).
0.82
0.965
0.2M
Tecal Loas
500 Bank
CLPT CT 23
0.80
F1.7%
Focal
Gal (E
0.900
0.00
0.055
0.75
0.77
0.78
30000
33400
60000
05000
70000
75005
30000
35000
8000D
81000
70000
75000
Training Iterations
Training iterations
Figure 6: Test accuracy of ResNet50 CIFARIO (left) and CIFAR100 (right) using the baseline approaches and our method with variable E and 7 = 0.5 in presence of multiple augmentations.
still load the original images, but they are selected 5 times with 5 different augmentations (described in Appendix A) during training. We show the learning curves in Figure and summarize the results in T
Comparing Figure with F shows that all methods strongly benefit from the augmentations. However, the spread between the maximum accuracies reached by the different methods approxi- mately doubles. For CIFAR10 our method outperforms the other methods, and on CIFAR100 slightly better than SGD-Scan and SGD-SPV. Speedup is consistent with the results shown in Fig- ure D suggesting that multiple data augmentation yield better results with approximately the same computational effort. Therefore using multiple augmentations can be beneficial it comes at additional cost in terms of iterations, but achieves a higher accuracy.
RELATED WORK
Our work relates to Focal Loss (Lin al., (017)) in the use focus and sample difficulty concepts, and to data pruning (Paul et al. 2021). which also aims estimate sample importance. However, our method differs conceptually: we downsample less important samples instead downweighting their loss, and compute sample importance only once without training new model from scratch.
Table 2:
Maximum mean test accuracy achieved by all methods and respective standard deviation when using multiple augmentations. For the learning curves refer to Figure
Data
SGD- Scan
EL2N
Focal Loss SGD- SPV SGD-Rank SGD-Prop
Ours
CIFARIO 95.7 + 0.2 95.8 + 0.2 95.4 + 0.2 95.8 + 0.1 95.7 + 0.1 95.7 + 0.3 96.2 + 0.2
CIFARI00 79.9 + 0.5 79.0 0.5 79.1 + 0.5 79.9 0.8 78.8
1.2 79.2 + 0.2 80.1 + 0.7
8

---

Under review as a conference paper at ICLR 2023
Other data pruning works suggest focusing high influence and high memorization training sam- ples (Feldman & Zhang 2020, Feldman 2020). Diang et al. (220)) rank samples according to their structural regularities, while Toneva et al. (2018) suggest a forgetting score of the number of forget- ting events a sample underwent during training. All of these methods suffer from the drawback that they have to compute sample importance first, prune accordingly and then train new model from scratch. Morcover, this overhead increases considering that scores are likely model-specific (Toneva ctt al. 2018 Jiang et al.] 2020), which implies that sample scoring needs to be repeated for every model and every dataset. Finally, an issue with these methods is that they require additional training iterations to compute sample importance scores, which our method avoids.
Hard example mining (Felzenszwalb et al., 2009 Loshchilov & Hutter 2015 Simpson 2015 Alain et al. 2013, (hrrivatavaa et al. [016] Chang et al. 201 Katharopoulos & Fleuret 2018 Arriaga & Valdenegro- Toro| 2020] Pruthi et al., 2020 Lin et al. 2017) tries to determine hard samples during training and use them to speedup convergence, under the assumption that hard samples are more informative for generalization. Exact methods to converge to the global optimum using hard (or im- portant) samples are proposed for SVMs (Felzenszwalb et al. 2009) and MLPs (Alain et al. 2015). For deep neural networks, some works emphasize important samples after the backward pass, such as the Focal Loss (Lin et al. 2017), that downweights the loss of easy samples, or Online Hard Ex- ample Mining (OHEM) (Shrivastava et al. 20166) that only computes the backward pass for a subset of samples in a minibatch. Loshchilov & Hutter (2015): [Simpson (2015) sample more often sam- ples with the greatest loss in an online fashion. Other sampling strategies aim reduce the variance of stochastic gradients during training (Katharopoulos & Fleurel 2018) or to sample more often samples with higher prediction variance (Chang et al.] 2017). Similarly, coreset selection methods
Mirzasoleiman et al.] 2020. Killamsetty et al. [2220] 2021 Yoon et al., 2021 Balles et al. 2022]
Killamsetty al. (022) try to find sample subsets during training approximating the gradient of the full training set. Coreset selection is also used to prevent catastrophic forgetting (Yoon et al. 2021
Balles et al. 2022) or to speedup hyperparameter tuning (Killamsetty et al. 2022). Coreset methods
make conservative estimates and are found to be less effective in practice (Paul et al. (02T). Differ- ently from these methods. we compute sample importance only once using ( which emphasizes hard samples without adding the overhead required when estimating the sample importance in an online fashion.
DISCUSSION AND CONCLUSION
We investigated importance weighted sampling of training data in SGD-based learning of classifiers. Our method does not require any additional training iterations overhead to estimate sample im- portance. Over all experiments it proved consistently at least on-par performance with competitors, but can achieve better results with less training effort, when burn-in epochs E well selected. Dependence on this additional parameter is the only limitation we observed.
Our results further show that sample importance weights are at least to some degree model specific. This suggests that alternative methods using pre-computed sample importance scores to prune the data might lead to non-optimal results, especially when the model used train with the pruned data is different from the one used to compute sample importance.
We offer empirical studies on class importance showing class balancing properties of our method and suggesting that unlike commonly assumed, balancing classes by number of samples is not nec- essarily ideal. Our results instead show that model trained with classes sampled according to class importance weights derived by our sample importance weights can outperform a model trained with balanced class weights. Still, we believe that fine- one-grained importance weights (i.e. per sample) are preferable as they allow the sampler to oversample the most important samples, independently from their class. Given these promising results, we hope to encourage further research on sample and class importance, to further understand the learning dynamics and optimize the training procedure.
Lastly, we show that our method makes a more efficient use of multiple augmentations, as it can select important samples from a larger pool of images. Thus, our method allows better results with approximately the same number of iterations as needed to fully train without augmentations. We see interesting opportunities to leverage our method also in applications where there is lack of data and where multiple augmentations could be key to achieve better performances. Here, our method allows the automatic focusing to those augmentations that actually benefit learning.

---

Under review as a conference paper at ICLR 2023
ETHICS STATEMENT
The main effect of our work is to speed up the learning process and automatically balancing classes. Speedup reduces energy consumption and potentially allows for using less powerful hardware for training, a tiny step towards democratizing AI. Balancing classes may help to emphasise underrep- resented samples and thus may raise visibility of minorities in data. It very much depends on the target application if this results in highly desired fair treatment of minorities or an unfair deviation from underlying distributions.
REPRODUCIBILITY STATEMENT
All presented results are reproducible using our code, which will be published upon acceptance.
REFERENCES
AdeelH..
pytorhh-multiccla--fcaal--ssss
https:///ithub. cww.://eeeuccom/edeeeellll 2020.
pytorch-multi oo
Guillaume Alain, Alex Lamb, Chinnadhurai Sankar, Aaron Courville, and Yoshua Bengio. Variance reduction in sgd by distributed importance sampling. arXiv preprint arXiv:1511.0648// 2015.
Octavio Arriaga and Matias Valdenegro-Tone Unsupervised difficulty estimation with action scores. arXiv preprint arXiv:201 1,161,, 2020.
Lukas Balles, Giovanni Zappella, and Cédric Archambeau. Gradient-matching coresets for rehearsal-basee continual learning. arXiv preprint arXiv:2203.14544, 2022.
Haw-Shiuan Chang, Erik Learned- Miller, and Andrew McCallum. Active bias: Training more accurate neural networks by emphasizing high variance samples. Advances Neural Information Processing Systems, 30, 2017.
Nitesh V Chawla, Kevin W Bowyer, Lawrence Hall, and W Philip Kegelmeyer. Smote: synthetic minority over-sampling technique. Journal of artificial intelligence research, 16:321-357, 2002.
Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc V Le. Autoaugment: Learning augmentation strategies from data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 113-123, 2019.
Ekin Cubuk, Barret Zoph, Jonathon Shlens, and Quoe Le. Randaugment: Practical automated data augmentation with reduced search space. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition workshops, pp. 702-703, 2020.
dyhan0920.
Pyramidnet-ppytorchh
https://qithub.
http://11..hb..comm///a/9/2 2018.
William Falcon and The PyTorch Lightning team. PyTorch Lightning. 3 2019. URL https: htt.ttta...cmm//y/oooo........nn
Vitaly Feldman. Does learning require memorization? a short tale about long tail. In Proceedings of the 52nd Annual ACM SICACT Symposium on Theory of Computing, pp. 959, 2020.
Vitaly Feldman and Chiyuan Zhang. What neural networks memorize and why: Discovering the long tail via influence estimation. Advances in Neural Information Processing Systems, 33:2881- 2891, 2020.
Pedro F Felzenszwalb, Ross B Girshick, David McAllester, and Deva Ramanan. Object detection with discriminatively trained bast- models. IEEE transactions on pattern analysis and ma- chine intelligence, 1209):1627-1644 2009.
Dongyoon Han, Jiwhan Kim, and Jummo Kim. Deep pyramidal residual networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5927-5935. 2017.
10

---

Under review as a conference paper at ICLR 2023
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Ziheng Jiang, Chiyuan Zhang, Kunal Talwar, and Michael C Mozer. Characterizing structural regu- larities of labeled data in overparameterized models. arXiv preprint arXiv:2002.03206 2020.
Angelos Katharopoulos and François Fleuret. Not all samples are created equal: Deep learning with importance sampling. In International conference on machine learning, pp. 2525-2534, PMLR, 2018.
Krishnateja Killamsetty, Durga Sivasubramanian. Ganesh Ramakrishnan, and Rishabh Iyer. Glis- ter: Generalization based data subset selection for efficient and robust learning. arXiv preprint arXiv:2012. 10630, 2020.
Krishnateja Killamsetty, S Durga, Ganesh Ramakrishnan, Abir De, and Rishabh lyer. Grad-match: Gradient matching based data subset selection for efficient deep model training. In International Conference Machine Learning, pp. 5464-5474. PMLR, 2021.
Krishnateja Killamsetty, Guttu Sai Abhishek, Alexandre V Evfimievski, Lucian Popa, Ganesh Ra- makrishnan, Rishabh Iyer, et al. Automata: Gradient based data subset selection for compute- efficient hyper-parameter tuning. arXiv preprint arXiv:2203.082. 2022.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Yann LeCun, Berhaard Boser, John S Denker, Donnie Henderson, Richard Howard, Wayne Hub- bard, and Lawrence Jackel. Backpropagation applied handwritten zip code recognition. Neural computation, 1(4):541-551, 1989.
Tsung- Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection, Proceedings of the IEEE international conference on computer vision, pp- 2980-2988, 2017.
Ilya Loshchilov and Frank Hutter. Online batch selection for faster training of neural networks. arXiv preprint 15X 1.06443, 2015.
Baharan Mirzasoleiman, Jeff Bilmes, and Jure Leskovee. Coresets for data-efficient training of machine learning models. In International Conference on Machine Leurning, pp. 6950-6960.
PMLR, 2020.
Jishnu Mukhoti, Viveka Kulharia, Amartya Sanyal, Stuart Golodetz, Philip Torr, and Puneet Doka- nia. Calibrating deep neural networks using focal loss. Advances in Neural Information Process- ing Systems, 13:15888-15299, 2020.
Samuel G Müller and Frank Hutter. Trivialaugment: Tuning-free yet state-of-the-art data augmen- tation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 774- 782, 2021.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary Vito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
Mansheej Paul, Surya Ganguli, and Gintare Karolina Dziugaite. Deep learning on a data diet: Find- ing important examples early in training. Advances in Neural Information Processing Systems, 34, 2021.
Garima Pruthi, Frederick Liu, Satyen Kale, and Mukund Sundararajan. Estimating training data influence by tracing gradient descent. Advances in Neural Information Processing Systems, 33: 19920-19930, 2020.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang. Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3)2111253 2015.
11

---

Under review as a conference paper at ICLR 2023
Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized experience replay. arXiv preprint arXiv:1111.0555,, 2015.
Abhinav Shrivastava, Abhinav Gupta, and Ross Girshick. Training region- based object detectors with online hard example mining. In Proceedings of the IEEE conference computer vision and pattem recognition, pp. 761-769, 2016.
Andrew JR Simpson. oddball sgd": Novelty driven stochastic gradient descent for training deep neural networks. arXiv preprint arXiv:/509.05575. 2015.
soapisnotfar.
pytorch-cifar10. haaeee/re///ddessss/e/eeeeeet. 2017.
httt::://ittubb.om/ snot.fa// TeNet.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Du- mitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1-9, 2015.
Mariya Toneva, Alessandro Sordoni, Remi Tachet des Combes, Adam Trischler, Yoshua Bengio, and Geoffrey Gordon. An empirical study of example forgetting during deep neural network learning. arXiv preprint arXi::
2018.
Ross Wightman, Hugo Touvron, and Hervé Jégou. Resnet strikes back: An improved training procedure timm. arXiv preprint arXiv:2 /0.00://00000466,, 2021.
Omry Yadan. Hydra a framework for elegantly configuring complex applications. Github, 2019.
htRp:tt::/ //g thub. ccom//acceoookrseercchh.ooo
Jachong Yoon, Divyam Madaan, Eunho Yang, and Sung Ju Hwang. Online coreset selection for rehearsal-based continual learning. arXiv preprint arXiv:2106.01083 2021.
A IMPLEMENTATION DETAILS
A.1 TRAINING DETAILS
A.1.1 DATA
We run our experiments on CIFARIO, CIFAR100 (Krizhevsky et al., 2009). all under the MIT licence. We the datasets in the same way as they are used in the work by Paul al. (2021): 50000 samples for training and 10000 for test when using CIFARIO and CIFAR 100. As data augmentation pad all sides by pixels, random crop 32x32 pixels and horizontally flip the image with probability 0.5.
Experiment on multiple data augmentation For the experiment where we adopt multiple data augmentations we use 5 data augmentations: twice the same data augmentations described above, AutoAAgment (Cubuk al.) 2019) with CIFARIO policy, RandAugment (Cubuk et al. 2020 and TrivialAAgmentWide (Müller & Hutter 2021) with default parameters.
Experiment on balaneing properties In the experiment showing the balancing properties, we randomly select 20% of CIFAR 100 classes with constant seed, then use the data augmentations described above when augmenting 5 times, and we repeat those augmentation types 4 times when augmenting 20 times.
1.1 FRAMEWORKS
We implement code using PyTorch (Paszke al. 2017) and Torch Lightning Fallcnn & The
PyTorch Lightning team (2019) as Deep Learning frameworks and Hydra (Yadan 2019) to manage the configuration files. We take additional inspiration how to combine these frameworks from
existing templates
12

---

Under review as a conference paper at ICLR 2023
ResMet50
CHARR100 Rechir50
0.765
0.945
Text
0.940
Test
0.765
0.760
0.935
50th
Cym23
0.755
50T Stor
(ye2)
(y-3)
0.910
0.750
Training iterations
Training iterations.
Figure 7: Different values of in Focal Loss for CIFAR1O (left) and CIFAR100 (right). Note that ? 0.5 leads to unstable results, therefore is not reported.
A.1.3 MODELS
We use ResNet50 (He et al. 2016) as main model. When working on CIFARIO and CIFARI00, we apply the same changes applied by Paul al. (202T): replace the first layers (convolution and max pooling) with single convolution layer with 3x3 kernel and 1x stride. We also use ResNet1 8, ResNet34 and ResNet101 with these changes. Lastly, we use LeNet (LeCun et al. 1989). GoogleNet (Szegeyy et al. 2015), PyramidNet 10 and PyramidNet164 (Han et al. 2017).
Note that we use the models available in torchvision except for LeNet, which we adapt to our imple- mentation from soapisnotfat (2017). and PyramidNet, which we take from dyhan0920 (0018)
A.1.4 HYPERPARAMETERS
We use the same training hyperparameters used by Paul et al. (2021): Stochastic Gradient Descent (SGD) optimizer, initial learning rate = O.1, Nesterov momentum = 0.9, weight decay 0.00005, We use batch size 128 for both CIFARIO and CIFARIO6 We train for 200 epochs, which translates into 78000 training iterations for both CIFARIO 10 and CIFAR 100. We decay the learning rate after 60, 120 and 160 epochs, which for CIFAR10 and CIFAR100 translates 23400, 46800, and 62400 iterations.
A.1.5 BASELINES
We use the Focal Loss (Lin et al. 2017 implementation available at AdeelH (2220). In particular,
we use 7 = 2, as it yields the best results for both CIFARIO and CIFARTO0, as shown in Figure 7 Note that we additionally tested y = 3 as suggested Mukhoti al, (2020), which leads slightly worse results, and
0.5, which leads to unstable convergence downgrading performance. For the EL2N method, we use the pre-computed scores provided to us by Paul et al. (202T) and pruned the data according to the percentages that yield the best performance in their paper. We use our own implementation for the other methods and adapt them to our pipeline. In particular, for SGD-SPV 10 burn-in epochs and 0.2.
A.2 EXPERIMENTAL DETAILS
A.2.1 REPORTING RESULTS
For every result, we report the mean, 16th and 84th percentiles of 4 independent runs (i.e. different random seeds). In the plots, this translates into line (the mean) surrounded by shading area (the percentiles). When showing the test accuracies over training, we zoom at the end of training to highlight the differences among the methods.
A.2.2 TUNING THE METHOD's ' PARAMETERS
Our method relies on 2 parameters: the amount of pre-training epochs E and the focusing parameter
y.
13

---

Under review as a conference paper at ICLR 2023
CIAR20 ResMet50
CHARR30 Rechir50
0.785
0.945
Text
0.940
Test
0.765
0.760
0.935
(B=2)
18a13
Our (8-27)
Our (5-10)
0.755
Das 18w3)
(b=))
Cats (8-26)
Cara (5-16)
(1-23
0.910
0.750
70000
Training iterations
Training iterations
Figure 8: Sensitivity of E when using our method with ResNet50 on CIFAR 1O (left) and CIFAR100 (right) with 7 = 0.5. The blue lines correspond the results shown in the main paper.
630
I-E,p-55
1-3,2:67
1-2,0-67
28 p-85
C- 24.20
C- 100 P464
C- 156
200
6.0
600
Figure 9: Class weights implicitly learned at different epochs by ResNet50 on CIFARIO using our method for 4 different initializations (error bars). Captions on top give the epoch at which weights are computed and the Spearman rank correlation p between the mean sample weights in a chart and the chart on its right (e.g. p between mean E=0 and E=1 weights on the first chart, E=1 and E=2 on the second, and so on).
We start by finding a good value for the epoch at the end of which to compute the weights. We show that fewer than 20 pretraining epochs E' are enough to estimate sample importance and that going further decreases the gencralization performance. We further show that E' depends on the dataset and on the model. In other words, given (initialized) model seems be able quickly learn which samples are most useful to learn the given task, and by switching its focus on these samples can converge faster.
Once a good epoch is found, we tune y. which is used in (Equation 4 to convert sample difficulty to sample weights. We find that a good value for y 0.5 and therefore we use this for all the experiments.
Unless differently specified, we use E 10 for CIFARIO and 1 for CIFAR100. Finally, for the experiments on CIFARIO and CIFAR9 with LeNet, GoogleNet and ResNet18 we use E = 5.
Sensitivity of E Figure & shows that for 0.5 method is slighlty sensitive the number of pretraining epochs E with the uniform random sampler. For CIFARIO we chose E = 10, but values close 10 and between 5 and 20 can also be good candidates, whereas for CIFARI00 only E = 1 leads to better results than the baseline, and other values lead to comparable or slightly worse performance.
In Figure 9 we show how class importance weights (and therefore sample weights) change during training and how subsequent class weights are correlated. Sample weights computed after epoch 100 are less correlated than weights computed earlier in training.
Sensitivity of y In Figure 10 we report the sensitivity analysis of y for the chosen values of E for both CIFAR 10 (E' 10) and CIFARI00 (E = 1). The figure shows that multiple values of y can be used for the same value of E and achieve comparable results. For instance, in CIFAR 10 both = 0.5 and 7 = 1 achieve better performance than the baseline, whereas only y >= lead to drop in performance. For CIFAR100, on the other hand, multiple values of lead to better performance than the baseline (0.1, 0.5. 1.5, 3, whereas the other tested values lead to comparable or slightly worse results.
14

---

Under review as a conference paper at ICLR 2023
ResMet50
CHARR30 Rechir50
0.785
0.030
0.910
0.935
0.710
Text
Test
0.765
0.925
0.760
0.95
Oura (ym0.1)
Dus (pen
Dars 1y
Eyw2)
0.755
Ous (ynt.
Our (ym0.1)
Gum (y=1)
(ye2)
(y-1)
(V-1)
0.910
70000
75005
0.750
70000
Training iterations
Training iterations
Figure 10: Sensitivity of y when using our method with ResNet50 on CIFARIO with F = 10 (left) and CIFARI00 with E (right). The blue lines correspond to the results shown in the main paper.
Pynnisttee10,
Okn wegge
0.35
0.00
Caramp- 2-10 SETATO- 6.9M.-12 parara-11 2- panars 2-10 par panamo- 20 param 42.50-2--
Figure 11: Class weights implicitly learned by LeNet, PyramidNet110, 10, GoogleNet, ResNet18, 18, ResNet34, ResNet50 and ResNet101 on CIFARIO using our method for 4 different initializations (error bars) and class weights induced from EL.2N scores before and after pruning 30% of the sam- ples with lowest scores. Captions on top give average Spearman rank correlation p of sample weights over the 4 initializations for the architectures, whereas those on bottom give the amount of parame- ters and the epoch E.
A.2.3 COMPUTING RESOURCES
We run all experiments single NVIDIA A100-SXM4-40GB. which is provided by an internal cluster. We use approximately 5000 GPU hours for the whole project. The experiments we show in the main paper and Appendix took approximately 700 GPU hours compute.
B FURTHER ANALYSIS ON SAMPLE IMPORTANCE
B. MODEL DEPENDENCY
Figure |I reports class importance weights aggregated from sample importance weights computed by our method with different models and parameters E 10, 7 0.5. Comparing this figure to Bigure3Jshove that class weights from different models get closer when computed at the same epoch (the mean rank correlation of sample weights from all models 0.81), but there are still differences between class weights, implying that sample importance evolves in model-specific way during training.
Figure 12 (left) reports several ablation studies to better show the model dependency of sample importance. In particular, shows that using weights computed by LeNet instead of those computed by our method at epoch E = 10 when training a ResNet50 on CIFAR10 leads to worse results (comparable to the main baseline, SGD-Scaa) than using the weights computed by ResNet50. This further shows that sample weights are modcl-dependent.
Note that using weights computed by ResNet18 at epoch E = instead those computed by our method at epoch E = 10 with ResNet50 leads to slightly better results, whereas using weights computed by ResNet18 at epoch E = 10 leads to slightly worse results. We hypothesize that ResNet18 and ResNet50 share some network similarities and therefore cannot be considered as completely different models, however additional experiments are required to verify this hypothesis well the possibility to use weights computed by smaller ResNet into bigger one.
15

---

Under review as a conference paper at ICLR 2023
FFRLO Reshve50
56c-San
Rester:3 (E-5)
520-5ean Ous (bx10)
Mann
0.0550
Dao (Ex38)
0.032
0.9525
0.0300
0.948
0.9475
0.946
Text
0.9450
0.9125
0.942
8.900
0.910
Training
Training iterations
Figure 12: Test accuracy of ResNer50 on CIFARIO using our method with E = 10, but using other weights instead of computing them, as explained in the following. In both Figures, SGD-Scan and Ours E=10 are respectively the uniform sampling and our method with E 10. (Left) ResNer18 = and ResNer18 E initialization at the specified epochs, LeNer uses weights computed by LeNet at epoch E 5. (Right) Mean uses the average sample weights over the runs computed with ResNet50, 2023 uses sample weights computed with ResNe150 when using seed=2023 (seed used for the last run).
respectively use the weights computed by ResNet18 with the same
LeNet
GoogleNet
ResNet18
ResNet50
0.8
150
0.8
0.8
150
0.8
150
Wtggl
0.6
100
Weight:
0.6
Weigh
0.6
Welght
0.6
0.4
0.4
0.4
0.4
50
50
50
0.2
0.2
0.2
0.2
Rank
x100'
Rank x110
Rank
xx10
Rank x10'
Figure 13: Sample weights W; over four different initializations for four different models trained on CIFAR10. Samples are sorted by their mean assigned weight W, The x-axis is the resulting rank r E (1,.., M) (M = 50k for CIFAR10) 10) from lowest t to highest w. The dashed red line indicates
W. In case plotted points overlap, color changes according the number of overlapping points.
B.2 INITIALIZATION DEPENDENCY
Figure shows initialization dependency with ResNet50 on CIFARIO for 4 runs of our model with different initializations each. For each run, the resulting weights way are averaged w;; Σラ= Wiij where j {L,..., 4] indicates the runs. Samples are sorted in ascending order of Wū with r(i) their resulting rank, i.c. the resulting new index r(i) E {l,..., M). The plots show (r(i), (,()) as dashed red line and dark blue dots are (r(i), (y(0,) with j (1,.... 4]. So for each sample with index r(i) we see the initialization dependence of weights (r()), as vertical spread of the blue points. We observe that for all models the variation in wH()), is small for the hardest samples with r(i) S M and all assigned weights (F((),. Z e, with some acceptable threshold e, say samples are consistently detected as hard and could be identified reliably in pre-processing step. The same is true for the easiest samples r(i) 2 1 and (r()), S but less pronounced for the small networks LeNet and GoogleNet. For medium hard samples, i.c. samples with Wr(s) 0.5 the variation of (,))) is much higher. In many cases the same sample may be assigned Wy(() & 0.2 in one initialization and (r(()) > 0.8 in another initialization. Thus, depending on the initialization, sample may be easy hard for the same model.
0.1. These
Figure 12 (right) shows the test accuracies of ResNe150 on CIFARIO with two additional variants of our method, compared to uniform sampling and our approach with E 10. Both variants start training a ResNet50 as our method until epoch E = 10, then update the sample weights as: the average sample weights (Mean) computed when using runs, and sample weights computed with seed 2023 (fourth run) for all the runs. The figure shows that using the average sample weights leads slightly worse results, whereas using sample weights
approach with ResNet50 on 4 different
16

---

Under review as a conference paper at ICLR 2023
LeNet. p=0.48
GoogleNeet, p=0.62
ResNei18, p=0.69
ResNet50, p=0.66
0.25
welghts
airp
bird
cal
deel
nog
horse
ship
truck
airplana alomaae bid
cal
deer
freo
hots 50l trick
airplance mucomma
bird
cat
dear
froo
horse ship
tuuk
airplatce
aoomabs bard
cal
deer
frog
horsa ship truck
Figure 14: Class weights implicitly learned by ResNet18 and ResNer50 using our method for
different initializations (error bars) CIFAR9, subset of CIFARIO without class dog. As for Fig- ure 3 captions report average Spearman rank correlation p of sample weights over the initializations.
computed by different seed (except for the last run with seed 2023, which uses its original sample weights) can lead to comparable results. This suggests that there is an initialization dependency, but it is not strong.
We conclude, that sample difficulty has a minor dependence on initialization, when computed from model output fe (x:)),. Consequently, it may not necessarily be possible to pre-compute (i) sample difficulty, (ii) an optimal selection of samples, or (iii) an optimal choice of per sample weights. may be necessary that weight assignment done during training, proposed here.
B.3 TASK DEPENDECNY
We show the task dependency by running our method on subset of CIFAR10, where the third most confused class (dog) is removed. We denote this subset CIFAR9. Not surprisingly, the figure shows that the class weights slightly change together with the task, although some general agreement the class importance retained. Of particular interest are the classes bird and cat ResNet18, which in CIFAR9 appear to be very close in difficulty, in contrast to the class weights in CIFAR10 (Figure 3) where is evident that class cat is harder than bird.
Note that for this experiment and therefore we run 70200 iterations. Here the learning rate is decayed after 21060, 42120, 56160 iterations, that is (60, 120, 160 epochs) 351 iterations per epoch.
adjust the training iterations according to the data size (45000)
B.4 CLASS WEIGHTTS IN PRESENCE OF CLASS IMBALANCE
Figure 15 shows the class importance weights computed by our method with E = 10 and y 0.5 using ResNet50 on CIFARIO0 with 20% the classes augmented 5 times (top) and 20 times (bot- tom) (corresponding to Figure 4 left and right). Consistently with assumption
that class weights are reduced for augmented classes, where weights get lower when using more augmenta- tions.
C EVALUATION ON IMAGENET-IK
We additionally evaluate our method on ImageNet-IK (Russakovsky et al., 2015) using a vanilla ResNet50 with a slight modification of the PyTorch procedure described in Wightman et al. (202T), where the learning rate scheduler is decayed after 30, 60 and 75 epochs, because we noticed that de- caying the learning rate once more towards the end slightly improves the final accuracy (i.e. 76.4% instead of 76.1%) 1%). Even though we did not run an extensive search to tune our hyperparameters. Figure [6] shows that using our method with E=10 and 7 = 0.5 achieves slightly worse results towards the end of training, whereas with ς = 0.1 achieves comparable performance to standard training. However, from Figure 17 where augment 20% of the classes (randomly selected once) 20 times, we conclude that our method works best when there a difficulty imbalance the given dataset and that, when samples are (almost) equally difficult, it performs par with vanilla SGD.
17

---

Under review as a conference paper at ICLR 2023
CIFARIOO {5x aug on 20% classes) ResNet50, p=0.64
0.020
Class weights
augmented other
0.015
0.010
0.005
0.000
99
Classes
CIFARLOO (20x aug an 20% classes} + ResNet50, p=0.56
0.020
Class weights
augmented other
0.015
0.010
0.005
0.000
99
Classes
Figure 15: Class weights implicitly learned by ResNet50 on CIFARI00 where 20% of the classes are randomly selected once and augmented 5 times (top) or 20 times (bottom). Error bars show the standard error over 4 different initializations. Caption on top give average Spearman rank correlation p of per sample weights over the 4 initializations. Non-augmented classes are shown as other.
ImageNet I< RecNe550
0.770
560 Sean
Oan (1-12g=033
500 Size
Gam (6-122044
1.03
2.05
Iterations
1.04
1.02
0.745
1.00
0.700
326000 340000
36000: 380000 400000 420000 440000 Training Beratiion
10-4
30
Telerance
Figure 16: Test accuracy (left) and speedup in terms of iterations (right) of ResNet50 on ImageNet- IK using standard training (SGD-Scan) and our method.
The figure shows that using our sample weights leads to better results than using random weights and than focusing on easy samples, computed at the same epoch as described above. Moreover, the disentangled variant achieves slightly worse results than uniform sampling, further suggesting that showing more often samples that are more important using their full gradient leads to an better performance.
Augmentation details For the experiment in Figure 17 we use the following set of different augmentations repeated 4 times, where all sets are preceded by RandomResizeCrop with size 224: RandomHHorizontalFiip. RandomHorizontalFip and ColorJitter, AutoAugment (Cubuk et al. 2019). RandAugment
(5) with parameters suggested in Wightman et al. (2021) for procedure A3, and Triv- ialAugmentWide (Müller & Hutter 202T).
from ht https://gggt/
18

---

Under review as a conference paper at ICLR 2023
ResNee50
Imageleli-K (20x 20% clecoed) BesMleso
20
0.8
Dus (kmd)
1.0
1.6
Ieeatoos
2.4
12
5GD
(B+2):
Dani (Pi-2)
Size
10
Figure 17: (Left) Test accuracy of all (solid lines), augmented (aug, dashed lines) and non- augmented classes (others, dotted lines) of ResNet50 on ImageNet-IK using standard training (SGD-Scan) and our method, when 20% of the classes are randomly selected and augmented 20 times. (Right) Speedup terms of iterations of SGD-Scan and our method with respect to the max- imum accuracy achieved by SGD-Scan.
D VISUAL INSPECTION
In Figure 18 and Figure 19 we visualize respectively the easiest and hardest sample per class (columns) and per seed (rows). These figures reveal different interesting observations:
often the difference between easier and harder is the different orientation of the object inside image (e.g. the frontal cat is considered easy, whereas cat seen from other angles is harder, perhaps because there are less images like so)
it is clear that our method succeeds in identifying easy and hard samples
some of the hardest samples are outliers (e.g. the first images from top of the truck column (right-most).
there is some disagreement in sample importance between the different initializations. This further confirms our initialization dependency claim.
E ADDITIONAL EMPIRICAL EVALUATIONS
Lastly, in the following subsections we show additional ablations in order to give further motivation for our method.
E. COMPUTING WEIGHTS ONLINE AND SGD WITH REPLACEMENT
To the best of our knowledge, we are the first to empirically show that it is possible to estimate sample importance once early in training and achieve better results by continuing training using this estimate as sampling distribution.
In Figure 20 the other hand, show that using our method in an online fashion (i.e. updating the sampling distribution after every iteration) leads to worse results. The figure also shows the results for SGD with replacement.
E.2 IMPORTANCE METRIC CHOICE
We selected 4 importance metric in experiments it proved to be better than other importance metrics (loss, EL2N Paul et (2021) gradient norms), as shown in Figure 21
Note that to compute the gradient norms with current framework minibatch after computing the per-sample losses to compute the par-sample gradient. This procedure took approximately 35 minutes for the single epoch E. Doing this online would be prohibitive.
had to loop each
19

---

Under review as a conference paper at ICLR 2023
Figure 18: Sample with the lowest weight per class (columns) for 4 different initializations (rows) of ResNet50 on CIFARIO with E' 10.
Figure 19: Sample with the highest weight per class (columns) for 4 different initializations (rows) of ResNet50 on CIFAR10 with E = 10.
Nashee50
RamNee:0
0.790
0.955
Oat (E-235
Sul placaaae
CUTS (2-1)
0.9525
0.9500
0.9755
0.790
Tett
09150
0.765
0.925
0.760
0.940
50000
55000
60020
45000
70000
73000
50000
55000
60000
65000
70000
75000
Trining iteations
Training iterations
Figure 20: Test accuracy of ResNet50 CIFARIO (left) and CIFAR100 (right) of SGD-Scan, method, our method with sample importance distribution updated online, and SGD with replace- ment.
E.3 BIASED VS UNBIASED SAMPLING
We propose SGD-based method biased towards important samples, assume that the given dataset biased (i.e. it has an imbalance in sample importance). For comparison, in Figure [2] report results for an unbiased version of method, where weight the gradient steps by the
20

---

Under review as a conference paper at ICLR 2023
FFRROO
56c-San
OurtLart (2-16)
520-5een
Oore Let (E-33
0.0550
(Em38)
Ourrr
Ouz
0.9525
0.760
0.0300
0.775
0.9475
0.770
Text
0.9450
0.9125
8.900
0.750
Training
Training iterations.
Figure 21: Test accuracy of ResNet50 on CIFAR1O (left) and CIFAR100 (right) of SGD-Scan, our method, our method with differnt importance metries: EL2N (without ensembling), loss, and gradient norms (GNorms).
Seidlet5
CIFARLIO RendletS
500 Scar
Oarstrenaaee (P-19)
503 Stze
Char Unbioodd (bw1)
0.75
0.76
0.910
0.76
0.935
0.900
0.74
Iterattons
Training Itorations
Figure 22
inverse of our sample weights (4). The figure shows that using a biased version of our method leads to better results.
E.4 SAMPLING PROPORTIONAL TO EL2N SCORES
Lastly, Figure 23 shows the results for a method that samples proportional to the provided EL2N scores (instead of pruning and using SGD-Scan on the pruned data). The experiment shows that EI.2N scores used as sample weights don't perform as good as the same scores used for data pruning and then training using SGD-Scan the resulting pruned dataset.
CPARLO RaaPwet0
CHARROD ReaNic50
100 90D-
550 SCB
502-9-44 61.2%
0.715
0.760
0.9000
0.775
8.475
88450
09425
0.400
6.755
0.750
Training
Iterations
Figure 23: Test accuracy of ResNet50 on CIFARIO (left) and CIFARI00 (right) of SGD-Scan, our method, and SGD sampled proportional to the provided EL2N scores (without pruning).
21


Reviews:
[{"review_id": "yj8LRZTn_Ff", "reviewer": "Reviewer_cDss", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed a sampling method for SGD that takes into account the importance of training examples. Specifically, the proposed method trains using uniform sampling for the first E epochs, and then compute importance scores for all training examples based on how accurate the current model’s prediction is. The proposed method trains using weighted sampling based on the importance scores for the rest epochs.\n", "summary_of_the_review": "This proposed method looks reasonable, but its idea is not new. In addition, this paper lacks enough analysis and the experiments can be improved with more recent and relevant baselines. ", "strength_and_weaknesses": "Strength\n1. The proposed method is explained clearly and with reasonable motivation.\n2. Extensive experiments are conducted. \n\n\nWeaknesses\n1. The key idea of the proposed method is to use the first E epochs as the warmup stage to obtain importance scores. This idea is quite straightforward and its variants have been considered in several existing works (e.g., Chang et al., 2017; Killamsetty et al., 2022; 2021; Katharopoulos & Fleuret, 2018). It’s not clear why the proposed method is better than these existing methods. Actually, the authors mentioned these methods in the paper but didn’t compare against them in the experiments. \n2. In addition, there are many recent works which take into account example priority for SGD-like methods (for example, see reference [1-4] below). The authors have considered Schaul et al. (2015) in the experiments. But to make the experiments more convincing, more recent priority-based methods should be considered and compared with. Otherwise, readers may feel the baseline is not state-of-the-art.\n4. The paper may benefit from some in-depth analysis about the proposed method. For example, is there any reason why the proposed method should be better than baselines? Do we expect better performance under all scenarios? Other similar works usually contain this kind of analysis (see reference [1-4] below).\n\nReference:  \n[1] Katharopoulos, Angelos, and François Fleuret. \"Not all samples are created equal: Deep learning with importance sampling.\" International conference on machine learning. PMLR, 2018.     \n[2] Hacohen, Guy, and Daphna Weinshall. \"On the power of curriculum learning in training deep networks.\" International Conference on Machine Learning. PMLR, 2019.    \n[3] Liu, Rui, Tianyi Wu, and Barzan Mozafari. \"Adam with bandit sampling for deep learning.\" Advances in Neural Information Processing Systems 33 (2020): 5393-5404.    \n[4] El Hanchi, Ayoub, David Stephens, and Chris Maddison. \"Stochastic Reweighted Gradient Descent.\" International Conference on Machine Learning. PMLR, 2022.   \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "obwuvnzGqEm", "reviewer": "Reviewer_LLpx", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose a new sampling method for training deep neural network. The sampler is based on the sample importance, which is defined as the sample difficulty using the propability. This probability is produced by the early trained model using standard uniform sampling. The following training epoch will use the new sampler thus focus on more important samples. The proposed method is claimed to converge faster and achieve better test accuracy.", "summary_of_the_review": "Overall, I think this paper proposes an interesting sampler idea that requires no additional training iteration, which saves computational cost compare to one exisiting data pruning method. However, the empircal results show marginal benefit compared to other methods, which would limit the ", "strength_and_weaknesses": "[Strength]\n1. The proposed sampler idea is simple and clear to be conducted. It uses the early trained model to produce sample importance and thus no additional training iteration is needed.\n2. The empirical analysis on model dependence and intrinsic balancing properties is interesting. \n\n[Weaknesses]\n1. My  main concern lies in experimental results. \n- The performance on CIFAR-10 by different methods make very little difference.  Numbers in bold should be those are significantly improved without overlap, which is not the case of CIFAR-10. This makes the comparison on CIFAR-10 less interesting.\n- The performance gap on CIFAR-100, besides SGD-Prop, is still very marginal. The \"Ours\" in Table 1 misses the hyperparameter of E=1 or E=10. The Ours with E=10 achieves inferior performance from Figure 2. The option on E seems to make significant difference on the overall performance.\n- Training on CIFAR-like-size dataset would be less insterested regarding saving the training cost. Experiments on larger dataset would be more persuasive. \n\n2. The maximum mean test accuray seem to make a good saling point for the proposed method that it achieves the *best* test performance faster. Whereas this *best* performance could not be defined as the best until all the pre-defined iteration is finished. Thus, this faster speed-up seems to be less attractive in practice.\n\n3. I wonder whether there could be some discussion on extending such sampler with other optimizer (besides the dependence as in sec 4.2), e.g. Adam. This coud make the proposed method more useful to a wider community.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "PMtH3LEtBil", "reviewer": "Reviewer_VNC6", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a data sampling approach to reduce the training time by focusing on difficult points. The main idea is to modify SGD so that it samples points with probability proportional to their sample difficulty, which the authors define using a score that is similar to the EL2N score of Paul et al. (2021), rather than uniformly at random as in standard SGD. The probability of sampling each point is computed once after training the model for E epochs using standard SGD, where E is a small number. Empirical results of the model’s test accuracy as a function of the training iteration, among other metrics, are compared to other baseline methods.", "summary_of_the_review": "The authors tackle a problem that is very relevant and of high significance to the ML community. However, the work is not well placed in the context of prior works that have addressed importance sampling for SGD and fundamental connections are not discussed. The method is not clearly described, seems ad-hoc, and leaves the reader with a lot of questions about its connections to related work (why not sample according to EL2N scores computed over 1 model, which has some theoretical justification, for example?). The empirical evaluations are also not convincing and the purported gains are not statistically significant. In light of these considerations, I recommend rejection.", "strength_and_weaknesses": "## Strengths\n* The problem of speeding up training time of models is highly relevant and significant in the ML community\n* Some components of the method are contextualized with respect to related work (e.g, Focal Loss, Lin et al., 2017)\n\n## Weaknesses\n* There is a significant amount of relevant prior work that is not appropriately discussed or cited. For instance, prior work is rich with theoretically-grounded methods for SGD Importance Sampling [1-3]. In fact, a closed-form solution for the importance sampling distribution that minimizes the sampling variance is known (sample proportional to gradient norms) [1].\n* Unlike the prior works mentioned above, the proposed approach seems to arbitrarily pick the Focal Loss’ sample difficult criterion with $\\gamma = 0.5$ with no compelling justification. In this regard, the work also overlooks deep connections with prior work. For instance, the EL2N score is an approximation of the (expected) gradient norms (see Paul et al., 2021, Definition 2.3 [4]), so sampling proportionally to EL2N scores (with respect to the most-up-to-date model) can be viewed as approximating the closed-form optimal importance sampling distribution [1]. What’s more is that the Focal Loss with $\\gamma = 0.5$ is closely related to EL2N score evaluated for the same model (square root of EL2N score upper bounds Focal Loss with $\\gamma = 0.5$). \n* The authors claim that a main benefit of this work is that the hard points are identified using a model snapshot after training with standard SGD for E epochs. But it is not clear why the updated values for the model’s predictions (even if they are from previous iterations) are not used to adaptively update the importance sampling distribution every once in a while. Additionally, can't the same procedure be applied but with EL2N scores instead?\n* How does the method compare to sampling proportional to EL2N scores instead? It’s not clear whether the choice of the Focal Loss metric is essential for the proposed method. It is also not clear how the reweighting of the selected examples are done. Relatedly, since the samples are no longer picked uniformly at random, each sample has to be reweighted by the sample probability so that the entire data set’s loss is captured in expectation (see [1-3]), but my understanding is that this is not implemented as part of the method.\n* The method uses a previously studied importance metric (Focal Loss) to sample points, without adequate justification or substantial novelty relative to prior work.\n* The empirical evaluations are not statistically significant or compelling. For instance, in Fig. 1 and Table 1 the performance of the proposed approach is well within one standard deviation of the approach based on EL2N scores.\n\n\n[1] https://arxiv.org/pdf/1803.00942.pdf \n\n[2] https://proceedings.neurips.cc/paper/2018/file/967990de5b3eac7b87d49a13c6834978-Paper.pdf \n\n[3] https://arxiv.org/pdf/1401.2753.pdf \n\n[4] https://openreview.net/pdf?id=Uj7pF-D-YvT \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "0zA47gvDHW", "reviewer": "Reviewer_auZ4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose a new importance sampling technique for SGD-based optimizers for training deep neural network models. Empirical studies are carried out on image classification tasks using standard benchmark datasets. ", "summary_of_the_review": "The authors propose an interesting sampling and training scheme. The experiments in the paper seems to suggest that the the weighting scheme is helping the SGD training. However, I believe the experimental setup and analysis don't clearly establish the benefits and issue of the proposed scheme. ", "strength_and_weaknesses": "Pros:\n- The paper is well organized and easy to follow\n- Experiments studies to validate the proposed idea and other follow-up analysis\n\nCons:\nEvaluation:\nThe proposed sample scoring scheme appears to be effective. However, I'm not entirely sure if the evaluation criteria is correct. For starters, the proposed sampler does uniform sampling for E epochs. Given this, its hard to verify how effective the  importance weighting scheme is. \n\nThe method is clearly sensitive to both E and \\gamma. Experiments show that one needs to tune both these parameters for each model and each dataset. Whereas some of the baselines and other methods that are not compared don't need any hyper parameter tuning at all. Therefore claims about the proposed not needing additional training iterations aren't entirely fair in my opinion. \n\nClass imbalance: I don't agree with the arguments in the section about class imbalance. The study there is based on the authors' proposed metric, which has an intuitive notion of \"importance\". Without rigorous theoretical grounding, the claims about class imbalance are not sound. These are however interesting observations and can benefit from thorough analysis. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject