Prioritized training on points that are learnable, worth learning, and not yet learned
Authors: Sören Mindermann
Muhammed Razzak
Mrinank Sharma
Jan M. Brauner
Winnie Xu
Andreas Kirsch
Aidan Gomez
Benedikt Höltgen
Sebastian Farquhar
Yarin Gal
Venue: ICLR 2022 Submitted (2022)
Keywords: Data selection
subset selection
deep learning
active learning

Abstract: We introduce reducible held-out loss selection (RHOLS), a technique for faster model training which selects a sequence of training points that are “just right”. We propose a tractable information-theoretic acquisition function—the reducible heldout loss—to efficiently choose training points that maximize information about a holdout set. We show that the “hard” (e.g. high loss) points usually selected in the optimization literature are typically noisy, leading to deterioration on real-world datasets. At the same time, “easy” (e.g. low noise) samples, often prioritized for curriculum learning, confer less information. In contrast, RHOLS chooses points that are “just right” and trains in fewer steps than the above approaches.

Under review as a conference paper at ICLR 2022
PRIORITIZED TRAINING ON POINTS THAT ARE LEARN- ABLE, WORTH LEARNING, AND NOT YET LEARNED
Anonymous authors Paper under double-blind review
ABSTRACT
We introduce reducible held-out loss selection (RHOLS), a technique for faster model training which selects training points that are "just right". We propose an inforoitiion-hhcorttiic selection function and a tractable, casy-to-implement approximation-the reducible held-out loss-to efficiently choose training points that maximize information about holdout set. We show that "hard" (e-g. high loss) points usually selected in the optimization literature are typically noisy, leading to deterioration on real-world, noisy datasets. At the same time, "easy" (e.g. low noise) samples often prioritized for curriculum learning confer less information. In contrast, RHOLS chooses points that are "just right" and trains in fewer steps than the above approaches.
INTRODUCTION
State-of-the-an models such as GPT-3 (Brown al. 2020), CLIP (Radford al. 2021), and ViT (Dosovitskiy et al.) (02I) achieve remarkable results by training on vast amounts of web scraped data. Especially for large models, training on such datasets requires excessive training times that are measured in weeks or months rather than hours (Radford et al., 2021 He et al., (020). Even practitioners who work with modestly sized models face long development cycles, due to numerous iterations of algorithm design and hyperparameter selection. These considerations highlight that the total time required for training is a core constraint the development of such deep learning models.
To speed up training, practitioners with sufficient resources use many more machines and much larger batches. However, there are rapidly diminishing returns to using larger batches and distributing them across more machines (LeCun et al.) 2012 Keskar et al. 20166), to point where adding machines does not reduce training time (Anil et al. 2018).
At the same time, practitioners realized that not all real-world data samples are equally useful. Many samples are noisy, i.e. mislabelled or inherently ambiguous. For example, the text associated with a web scraped image is rarely accurate description of the image. Other samples are redundant, for example due to the typical overrepresentation of certain object classes in web scraped data (Tian et al.) 202T). Such redundant samples can often be left out without losing performance. In many cases, data is so abundant that state-of-the-an performance is reached in less than single epoch (Brown et al. 2020 Kaplan et al. 2020 With such abundance, it can easily be afforded to skip less useful points.
There are different approaches to selecting the most useful datapoints for training. Some bodies of literature, including curriculum learning, suggest prioritizing easy points with low label noise before training on all points equally (Bengio et al. 2009). While this can improve convergence and generalization, it has no mechanism avoid points that are already learned (redundant). Other works instead suggest to train on points that are hard for the current model. In the optimization literature, online batch selection methods (Loshchilov Hutter 2015) do this by selecting points with high loss or high gradient norm. Such online selection successfully skip points that are already learned and focus on the few points where it is possible to make the greatest learning progress (Figure) I left).
However, we show that prioritising hard examples can be detrimental noisy data. In real- datasets, high loss examples may be mislabelled or inherently ambiguous. Indeed, in controlled experiments, with just 10% uniform label noise, points selected by these methods are overwhelmingly those where the label is corrupted (Figure I, right). Our results show that prioritising them degrades performance severely. Additionally, some samples are hard because they are outliers with unusual

---

Under review as a conference paper at ICLR 2022
Selection Method
Proportion of sum acress barch
which the top 201 of points
100
Proportion of selected points
that had label moise applicd (%)
40
Reducible Loss (Our)
Sampling
are repponible (98)
30-
Frreuccible Loss
Gradient Noem
Loss
20-
40
10-
20
CIFARIO CIFARIOU
CINICIO
CIFARIO CIFARI00
CINICIO
Figure 1: Properties of different training data selection functions: loss, loss, and gradient norm. Left: We score the points in each batch with each selection function and sum the score of the top 20% of points, then divide by the total sum across all points. Across selection functions, the set of top-scoring points is responsible for most of the total; this suggests that many points already learned (redundant) and can be skipped. Right: Proportion of selected points with label noise applied. We added 10% uniform label noise, i.e., we randomly switched each point's s label with 10% probability. Selecting the 10% "hardest" points (high loss or gradient norm) gives batches where > 10% of the labels are mislabeled. We can avoid noisy points by selecting points that are "easy" (low irreducible loss) or have high reducible loss. Results are averaged over 15 epochs.
reducible loss (ours), irreducible
features, far from the other training samples. This type of difficult point can also lie far from the test data, which makes it less relevant for reducing the test loss.
To overcome these limitations, we introduce reducible held-out loss selection (RHOLS). We introduce Information Content (PIC) that quantifies the information each point confers about the labels of holdout set. Approximating this, we derive an intuitive, tractable, and casy-to-implement novel selection functiin: reducible held-out (cross-entropy) loss. By maxi- mizing the information gained about the labels of a held-out set, our method chooses non-redundant, relevant points with low noise-points that are "just right".
a selection
We explore the effects of reducible loss selection on five datasets, with and without additional label noise. We evaluate the reduction in required training steps compared to conventional uniform sampling and recent selection methods. Our method reaches target accuracy in 1.76x-3.822 fewer steps than conventional uniform sampling on CIFAR-10, CIFAR-100, and CINIC-10. In the presence of just 10% label noise, this speedup increases further to 2.34x-5.15s while the speed of state-of-the art prior methods degrades severely. On Clothing 1M, large-scale dataset with noisy, web- web-scraped labels, our method reaches target accuracy in 12.5x fewer steps compared to uniform sampling and 8% higher final accuracy.
2 BACKGROUND: ONLINE BATCH SELECTION
Consider a model p(y) x, 0) trained on data D [2 9:) using stochastic gradient descent. At each training step f, we load a batch b, of size nı from D. In online batch selection (Loshchiloy & Hutter 2015), we pre-sample a larger batch B, of size ng > Thg Then, we construct smaller batch by that consists of the top-ranking n. points in B, ranked by label-aware selection function S(z,. 3.). We perform one gradient step to minimize a mini-batch loss Eier, L( p(ys 2 f)). The following large batch B,+1 is then pre-sampled from D without replacement of the previously sampled points (points are replaced only at the start of the next epoch).
We work with this approach for its simplicity, convergence guarantees, and strong performance in recent work (Kawaguchi & Lu 2020). Further, it could be extended to incorporate stochastic data selection (Jiang et al. 2019) with importance weights (Schaul et al. 2015 Katharopoulos & Fleuret 2018) or to include earlier points I; in B, whose selection score S(x,, y/) has been previously computed (Loshchilov & Hutter 2015). Without importance weights, the simple approach above biases the location of the minimum of the loss. However, this biased selection can improve test performance both in theory and practice (Farquhar al. 2021 Kawaguchi & Lu 2020).
2

---

Under review as a conference paper at ICLR 2022
Notation. For holdout set {u/" {at"), we will write yte, x for brevity. Lower-case variables like y are outcomes and upper-case variables like Y are random variables.
3 REDUCIBLE HELD-OUT LOSS A MEASURE OF PREDICTIVE INFORMATION CONTENT
Online batch selection methods construct batches training time using points that have high loss high gradient norm (Loshchilov & Hutter 2015 Katharopoulos & Fleuret 2018). In this section, we derive and discuss alternative selection function, based on an thoomation--heoreti perspective.
Predictive Information Content (PIC). Our overall objective is to select the point (a, y) E DD for training that minimizes the expected loss (x..) hly x, D, (xr v)] of the model, where
hyy x. D.] == = log log x, D:)
(1)
denotes the cross-entropy loss on (x, y) of a model trained on D.. As proxy, we aim maximize the reduction in the Monte Carlo expectation of the cross-entropy loss a held-out dataset Dho-
hly!
ho
x D. - h(y."") 1 D ( (z,
(2)
(Dhol
iED
For deterministic model (i.e. with fixed parameters), this held-out loss (to a constant factor) equals
hlybo ho Di - h(y" | D (az, v)l.
(PIC-1) (3)
We term this the Predictive Information Content (PIC). It quantifies predictive information: the part of the information gained by observing (x, that is applicable to predicting the holdout labels. It also relates to the Expected Predictive Information Gain (EPIG) recently proposed for active learning (Kirsch et al., 202T). In active learning, one selects points without access to their labels, and therefore Kirsch et al. (2021) must compute the expectation of ( over posterior predictive p(Y xx Dz). However, here we quantify not the expected but the actual information gained when observing (I. We therefore follow naming conventions and term (3) (predictive) information content. See Appendix for more details on the relationship to active learning and EPIG.
Deriving a tractable selection function: the irreducible and reducible losses. The two terms in (3) cannot be computed efficiently; they require first training the model on each potential point (1r, v) and then performing a forward pass with the model on the full holdout set. We derive an alternative, more tractable expression. To make claims precise and our assumptions transparent, we use the language of Baycsian probability theory to formalise our claims. In the Bayesian context, we random variable with prior p(0) and infer posterior (() using
treat model parameters the sequence of already-seen training data D. = bi:t-- The model has predictive distribution p(y|æ, D,) = Jg P(ylæ:, f)) (( dB. When using point estimate of (such an MLE MAP), the predictive distribution can still be written as an integral, but with respect to a Dirac delta at the point estimate an approximation of the true posterior. Applying Bayes rule to p(y"0o y/, ho xr, D,) and using the conditional independence assumption p(y: Iı, 2, D.) = p(us xs, D.), we can reformulate PIC
hly xr, D. hly yh hn D.).
(PIC-2) (4)
moving the holdout set to the conditional. We show this derivation in Appendix
The first term simply the cross-entropy loss on the potential point (ar, y) of a model trained on D. The second term-the cross-entropy loss of model trained on D, and the held-out ser-add, computation as it requires constant updating with increasing D, We can approximate this second term by training model only on the held-out dataset, and refer to it as the irreducible held-out loss:
hly x. ho ho0
(irreducible held-out loss) (5)
This approximation may be reasonable when D. < Drol (e.g. at the start of training) or when small model p④y ho x"") trained on Dno has limited capacity to improve further by training on D, Empirically, we find this approximation does not affect performance (Appendix DD. We refer to
3

---

Under review as a conference paper at ICLR 2022
Eq. (5) the irreducible held-out loss since it is the portion of the loss that could not be reduced by training on the held-out set. Plugging this approximation into Eq. (4). we get the reducible held-out
loss.
hly z, Dj] hly 2,y y =)
(reducible held-out loss) (6)
This provides an easy-to-complls selection function that approximates the Predictive Information Content between Dho and the datapoint (æ, y) for model trained on D,
Understanding reducible loss. We now provide intuition on why the reducible held-out loss selects non-redundant, non-noisy points. The reducible held-out loss contains two terms. The first term, the loss, is high on points difficult for the current model i.e. points that are not yet learned and thus not redundant. Indeed, existing online batch selection methods use loss for selection (Loshchilov & Hutter, 2015). However, not all high loss points are informative-some may be noisy or mislabelled. Crucially, these points cannot be predicted using the holdout set and thus have high irreducible loss. Therefore, these non-in -on-information points have low reducible loss. Indeed, we show that the cross-entropy loss disproportionally selects such noisy points Figure whereas selecting points with high reducible loss (or low irreducible loss) avoids noisy points. In short, RHOLS prioritises points that are not redundant (high loss) and not noisy (low irreducible loss).
Loss based selection has additional pitfalls. The loss is likely higher for outliers- values of x far from the training data, such those a region with low input density Pone (z). Since our aim is to reduce the expected loss P. [h(y x. D.]] under Paue (z,y). we do not want prioritize less relevant outliers. Since such points are also less likely in the held-out data, the irreducible loss can also be expected to be higher. Thus, reducible held-out loss should select points that are more 'relevant' to the holdout set. We find empirical evidence for this below.
Selecting whole batches. We developed the theory assuming the selection of single point (ar, y)- When selecting entire batches, we select the points with the top-rı, scores from a randomly sampled B, of candidate examples. This assumes that each point has little effect on the scores of other points, and is often used as a simplifying assumption for selection in the context of active learning (Kirsch et al. 2019). This a much more reasonable assumption in our case than in active learning, because model predictions are not changed much by single gradient step on one mini-batch.
Algorithm T shows the implementation batch selection using the reducible held-out loss. Note we precompute and store the irreducible losses before training. This way, we avoid recomputing them in every epoch and across different training runs.
Algorithm Reducible held-out loss selection (RHOLS)
1: Input: Learning rate y7. model p(y) x,, Dho)) trained on holdout set Dno. batch size ni, large batch size n > Tb..
2: for (I,, v.) in training set do
3: IrreducibleLose [i] t- hly x4, Dho.)
4: end for
Initialize parameters 80 and f =
5: for t = 1,,,.... ......d..
6: Randomly select large batch B, of size ng.
7: Vi E B, compute Loss [i], the loss of point given parameters Ø*
8:
Vi Be, compute ReducibleLos
+ Loss [i] IrreducibleIoss [1]
9:
b <- top-n; samples in B, in terms of Reducibleioos
10:
Dt mini-batch gradient on b, using parameters 0²
11: gi+1 g¹
Simple parallelized selection. For icage-scale neural network training, practitioners with sufficient resources would use many machines if it further sped up training (Anil al. 2018). However, as more workers are added synchronous or asynchronous gradient descent, the returns diminish to point where adding more workers does not further improve wall clock time (Anil et al. 2018). For
4

---

Under review as a conference paper at ICLR 2022
example, there rapidly diminishing returns for using larger batch sizes distributing given batch across more workers (LeCun et al., 2012. Keskar et al. 2016). The same holds for distributing the model across more workers along its width or depth (Rasley al. 2020: Shoeybi et al., 2019: Huang et al. 2019). However, we can circumvent these diminishing returns by adding a new dimension of paralleiizaiion, namely, for data selection.
Since parallel p forward passes do not suffer from such diminishing returns, one can easily use extra workers for them. The theoretical runtime speedup can be calculated as follows. The computational "y_ times as much as the cost of a training step on 6,- This is because forward
cost of selection is 3 passes require 3x less computation than a full bawkwad pass (Juupp et al. (017). One can accelerate this computation almost arbitrarily by adding more workers that compute losses using a copy of the model being trained. The limit is reached when the time for selection is dominated by the cost of communicating parameter updates to workers. More sophisticated parallelization strategies allow reducing the time overhead even further, see Section 4 To avoid assumptions about the particular strategy used, we report experiment results in terms of the required number of steps.
4 RELATED WORK
Data selection functions. Our reducible loss is best understood an alternative to other selection functions. Such metrics can be categorized by the properties of points they select and by whether they
use information about labels. Both high loss (Loshchilov & Hutter 2015 Kawaguchi & Lu 2020
Jiang et al. 2019) and high prediction uncertainty (Settles) 2009 Li & |2066][2006] Gal et al. 2017
Coleman et al. (0200) select for points that are "difficult" for the current model. But while loss is only
applicable to labelled data, prediction uncertainty is designed for the active learning setting where labels are not available. Despite this difference, due to their similarity they face the same problem: high loss and high uncertainty can be due to an ambiguous ground-trun truth or a noisy label. A related technique prioritizes samples whose labels are easily forgotten during training (Toneva et al. 2018).
Variance reduction methods. Online sample selection methods have also been suggested as a way to reduce the variance of the gradient estimator computed on mini-bbtches (Katharopoulos & Fleuret 2018 2017: Johnson & Guestrin 2018 Allain et al. 2015). Such methods typically use importance sampling, where points with high gradient norm (or approximations thereof) are sampled with high probability. To avoid biasing the gradient estimator, the gradient of bhob-brbbbbbiitty points is then down-weighted. Importance sampling can be effective but also sensitive to hyperparameters (Johnson) (right) shows that the gradient norm is also often highest
& Guestrin 2018). Furthermore, Figure [ for noisy points, which may make it unsuitable for noisy real-world datasets.
Computationally efficient data selection. While we limit our analysis to the choice of selection function and compute the metrics naively, this choice is likely ineffieient in practice. Selection be made cheaper by reusing losses computed in previous epochs (Loshchilov & Hutter] [205]] Diang et al.)
2019) or training small model to predict them (Katharopoulos & Fleuret [20 2017: et al. 2019). Alternatively, core set methods only perform selection once before training (Mirzasoleiman et al.
2020 [Borsos al. 2020). Such methods typically use expensive metrics that are not economical for
large web scraped datasets where training typically only lasts a few epochs.
Time-efficient data selection. To accelerate data selection, prior work uses set of workers that perform forward passes on B, or on the entire dataset asynchronously while the master process trains recently selected data (Alan et al. 2015: Schaul et al.) 2015). This supplements rather than replaces standard data parallelism, a method whose returns diminish as more workers are added be accelerated using low-precision hardware. A forward pass by default already requires roughly 3x less time than a forward-backwarr pass but this speedup can be increased to factor around 10 when using the low precision cores available in modern GPUs and TPUs (Jouppi et al.) 2017 Jiang et al., 2019). While backward passes typically require higher precision, forward passes can tolerate lower precision.
(Anil et al. 2018). Furthermore, forward passes
5

---

Under review as a conference paper at ICLR 2022
MNIST
MNIIS with 10% Label Noisg.
MNIST with Structured Noisg
Ambiguous MNIST
Test Test Accuracy (96)
:00
Test Accuracy (96
Test Accuracy (96)
100
90-
Test Test Accuracy
90-
90-
90
80
70-
70-
70-
70
1500 60
60
500
1000 Steps
500
1000 Steps
1500
500
1500 Steps
1000
500
1000
1500
Steps
Recucible oss [Ours] Uniforrn Sernlling
imeducible loss Gradient North
1
Figure 2: Reducible loss is robust to variety of label noise patterns, while other selection methods degrade. A step corresponds to lines 6 11 in Algorithm 1. Lines correspond to means and shaded areas to minima and maxima across 3 random seeds.
5 EXPERIMENTS
Datasets. We evaluate on five datasets. 1) QMNIST (Yadav & Bottou 2019) augments the MNIST dataset (LeCun et al. 1998) with an additional 50,000 images. We make use of the additional 50,000 images as the holdout set. 2) On CIFAR-10 10 (Krizhevsky & Hinton 2009) we train half of the training set and use the other half as a holdout set used for training the irreducible loss model. 3) CIFAR-100: same CIFAR-10. 4) CINIC-10 (Darlow et al. 2018) has 4.5x more images than CIFAR-100 and includes a holdout set and a test set with 90,000 images each. 5) Clothing-IM (Xiao) al. 2015), which contains over 1 million clothing images divided into 14 classes. The data is web-scraped-a key application area of our work-and thus has noisy labels. The dataset also contains 50k, 14k, and 10k images with clean labels for training, validation, and testing, respectively. We use the 50k clean training samples to train our irreducible loss model, and include the 50k clean training samples in the larger dataset when running the baselines. As web-scraped dataset, Clothing- 1M most representative of the setting where training time rather than data is the bottleneck.
Baselines. The most important baseline is the de facto standard in deep learning, uniform selection (without replacement). We also evaluate against selection functions that have achieved competitive performance recently: the plain cross-entropy loss (as implemented by Kawaguchi & Lu (2020)), gradient norm, and gradient norm with importance sampling (called gradient norm IS in our figures) (Kahharopoulos & Fleuret 2018). We follow prior work by approximating the per-example gradient with the last layer gradient norm for computational tractability. We also compare to the core-set method Selection-via Prox-y (SVP) (Coleman et al.) 2020) that selects data offline before training to probe if maximal performance requires online selection. We report results using maximum entropy SVP and select with the hest-performing model. We compare only to alternative selection functions. Other improvements such predicting reusing losses (Katharopoulos & Fleuret [2077] Zhang ctt al. 2019 Jiang et al. 2019) are orthogonal to our contribution and may be combined with our selection function further accelerate training.
We use a layer MLP for experiments on QMNIST, and a ResNet- 18 for all other datasets. We use the Pytorch default hyperparameters and the AdamW optimiser. Please see Appendix Cor details.
5.1 ROBUSTNESS TO NOISE
We begin by evaluating the performance of different selection methods under a variety of noise patterns on QMNIST and variations thereof. We use this dataset because it has little label noise in its original form, allowing us to test the effect of adding noise. Firstly, we add uniform label noise to 10% of training points. Secondly, we add structured label noise that affects easily confused classes. We follow Rolnick et al. (0077) and flip the labels of the four most frequently confused classes (in the confusion matrix of trained model) with 50% probability. For example, a 2 is often confused with a 5; thus we change the label of all 2s to 5s with 50% probability. Thirdly, we leverage the natural noise distribution of MNIST by using Ambig (Mukhoti et al. 2021 as the training set. AmbiguousMNISTT contains a training set with 60k generated ambiguous digits that have more than one plausible label.

---

Under review as a conference paper at ICLR 2022
While selecting with loss and gradient norm trains accelerates training on the MNIST training set, their performance degrades on all three types of noise distributions (Figure 2 This in line with finding in Figure (right) that they over-select points with noisy labels. in contrast, reducible held-out loss is robust to label noise and trains in fewer steps than all baselines.
Clothing- IM
CIFARIO
CIFARIO Noise)
Steps requirred target accuracy
15:109
Suape Seggen
101
100
200
15-10P
Noise)
Sueps canrrooinn
101
20
40
60
80
Target Accuracy (%)
CINICIO
CINICIO
Selection Method
Staps reggres
Redacitblll (Tlon)
Loss
Usifiom Sangling
SVP
11a
Gradicat
Target Aocaracy
Tirget Accunacy (9)
Figure 3: Gradient steps required achieve given test accuracy (lower better). Left column: large-scale web scraped dataset
The speedup of RHOLS over uniform sampling is greatest with noisy labels (12.5x). Middle column: Speedups are still substantial on clean datasets. Right column: Applying 10% uniform label noise to training data degrades other methods but increases the speedup of our method. A step corresponds to lines 6 in Algorithm Lines correspond to means and shaded areas to minima and maxima across 3 random seeds.
5.2 SPEEDUP IN STEPS REQUIRED
Here, we evaluate how many steps each selection method needs to achieve a given test accuracy.
As focus is evaluating new selection function rather than entire training pipeline, report the number of steps needed to reach target accuracy. How the required steps relate to the wall clock time depends on implementation details that are beyond the scope of this paper. Most importantly, data selection is amenable to parallelization beyond standard data parallelism as discussed in Sections and 4 Furthermore, model trained on a holdout set can be reused for selection during the many runs that are part of training pipeline.
Speedup on clean data. RHOLS reaches the same accuracies in fewer steps than standard uniform selection without replacement across all datasets (Figure [3] top). Furthermore, outperforms all baselines across all datasets. The observed speedup with respect to uniform selection is 3.82x on CIFAR-10, 10, 1.76x on C00,R-100, and 2.88x on CINIC-10.
Speedup on noisy data. We uniformly perturb the labels for 10% of the datapoints. Under label noise, batch selection with reducible held-out loss further increases its speedup over the other methods (Figure bottom). Particularly interesting is that, on noisier data, the speedup over uniform selection also grows. The speedups observe relative uniform selection are 5.15x CIFAR-10 10, 2.34x on CIFAR- 100, and 4.17x on CINIC-10. This aligns with our information-thhorrtiica perspective on reducible held-out loss and with the finding in Figure T (right) that reducible held-out loss avoids noisy points while other methods prefer them.
Speedup on large-scale web-scraped data. On Clothing- 1M, no baseline matches uniform selec- tion, suggesting they are not robust noise. In contrast, RHOLS reaches the accuracy that uniform
7

---

Under review as a conference paper at ICLR 2022
Pereentige of selocted
Steps Roquired
2000
4000
25
5000
1000
Target Acccuracy (%)
Steps
Reducible Loss (Ours)
Ineducible Loss
Loss
Uniform Sampling
Gradient
Figure 4: CINIC-10 with half of the classes absent (irrelevant) the test and holdout sets. Left: steps required to reach given target accuracy on the test set. Right: Proportion of selected points whose labels do appear in the holdout and test sets. RHOLS and irreducible loss select exclusively the relevant classes. ROHLS provides large speedup over existing baselines. Lines correspond to means and shaded areas to standard deviations across random seeds
selection achieves after epoch 50 in just 2 epochs, providing a speedup of 12.5x. Furthermore, RHOLS' reaches a higher final accuracy (74%) than uniform selection (66%).
Reducible held-out loss can select more relevant points. In Section we argued that, in contrast to other methods, RHOLS does not prioritize less relevant points that have low probability under the input density (ar). To investigate this, we subset the CINIC10 dataset; the train set consists of all classes, while the holdout and test set consists of half of the classes. Such setup could also be useful in some real-world scenarios where one would like to avoid certain training points containing e.g. low-quality, biased, toxic information by training the irreducible loss model on clean holdout set.
RHOLS avoids points whose labels do not appear in the validation set (Figure 4). Accordingly, provides a large speedup. Although irreducible loss selection also avoids irrelevant points, RHOLS selects relevant points that are not yet learned, leading to a greater speedup. RHOLS leverages the implicit task relevant description provided by the holdout data accelerate model training.
5.3 CHEAP IRREDUCIBLE LOSS MODELS
While RHOLS reduces the number training of steps required across several datasets, it requires training an additional model on separate holdout set, which poses additional computational cost. Here, we examine how to minimize this cost and amortize across many target model training runs. For practicality, we perform these experiments the clean benchmark datasets, although expect RHOLS to speeds up training more on noisy and redundant web scraped data.
Irreducible loss models can be small and cheap. While Eq. (6) assumes that the target and irreducible loss model architectures identical, we hypothesize that this is not necessary. Across sufficiently similar architectures, the irreducible loss is likely lower for points similar to the holdout data, and higher for points with noisy label (we verify this in Appendix D). None of these require the architectures to be identical. Furthermore, the loss of the target model is used to deprioritize redundant points, and is thus unaffected by the irreducible loss model used.
In Figure we replaced our ResNet-18 18 irreducible loss model with a small CNN reminiscent of LeNet. 21x fewer parameters and requires 29x fewer FLOPs per forward pass than the target ResNet-18. The smaller irreducible loss model accelerates training as much or more than larger model, even though its final accuracy is far lower than the target ResNet-18 (11.5% lower on CIFAR-10, on CIFAR-100, and 8.1% on CINIC-10).
Irreducible loss models require little holdout data In Figure 5 train the irreducible loss model on a small fraction of the available data, 2 4x less data than in our default setting above. To maximize compute savings, still use the small CNN model. In combination, the computational cost
8

---

Under review as a conference paper at ICLR 2022
Dataset CIFARIO CIFARION 100
Default
Small ineducible loss model
Holdout set 25 0% of available data
CINICIO
Holdout set 12.5% available
No holdout set
Architecture transfer
Hyperparameter transter
No spoodap
RHOLS speedup over training
Figure The irreducible loss model can be small, trained on little data, and reused across target models and hyperparameters. The x-axis shows after how many fewer epochs RHOLS reaches the highest accuracy that uniform selection reaches during 100 epochs (with R 0.1). All rows but the first ('Default') use a small CNN as irreducible loss model (in addition the other experiment conditions indicated), Each dot represents an experiment with specific combination of irreducible loss model and target model (multiple seeds per combination shown as separate dots).
ng
reduced 116x compared to experiments in Section 5.2 The speedup reduces when the irreducible loss model is trained less data, but RHOLS still speeds up training on each dataset. Note that for web-scraped data, the dataset is often too large to finish even one epoch, meaning that practitioners can afford larger holdout sets.
Irreducible loss models do not require any holdout data In Figure we train the irreducible loss model without holdout data. We split the training set into two halves and train smaller irreducible loss model on each half. Each model computes the irreducible loss for the half of D that it was not trained on. Although we train two models, each model trained on half the data (assuming equally sized train and validation sets); as such, this approach does not incur additional computational cost.
Irreducible loss models can be reused train different target architectures We find that single small CNN irreducible loss model accelerates the training of target architectures: VGGI] (with batchnorm), GoogleNet, Resnet34, Resnet50, Densenet121, MobileNet-v2. Inception-v3. On CIFAR-10, 10, RHOLS does not accelerate training of VGG11 1 (2 seeds), which is also the architecture on which uniform training performs the worst; i.e. RHOLS empirically does not "miss" a good architecture.
Irreducible loss models can be reused train many targets hyperparameter sweep We find that a single small CNN accelerates the training of all ResNet- 18 target models across hyperparameter grid search. We vary the batch size (160, 320, 960), learning rate (0.0001, 0.001, 0.01), and weight decay coefficient (0.001, 0.01, 0.1). RHOLS speeds up training compared to uniform on nearly all target hyperparameters. The few settings in which doesn't I speed up training are also settings in which uniform training performs very poorly (< 30% accuracy on CIFAR-I00 100, < 80% on CIFAR-10).
DISCUSSION
With ever larger models and datasets, training times keep low-resourced groups from access to deep learning and well-resoureed groups from pushing its frontier. This can already be mitigated by existing methods for parallelization or data selection. However, the benefits are limited by diminishing returns for the former and, we show, problems with noisy data for the latter. We aim alleviate these limits by introducing a novel, theoretically grounded selection function. In controlled experiments, show that its deployment enables substantial speedups curated data and even larger ones under label noise. However, our approach should be combined with methods in Section for cheap and fast selection with maximal speedups.

---

Under review as a conference paper at ICLR 2022
7 ETHICS STATEMENT
It will be important to understand how subset selection might affect performance on data about minority groups. is possible that the selection deprioritizes data points rare groups (since they affect the loss on holdout data less, or prioritizes them late in training once majority groups are already learned.
Since such biases can also stem from the dataset itself (Mehrabi et al. 2021). 1, it should be investigated our method can remove data biases through the use of an unbiased holdout set. By training the irreducible loss model on unbiased data, we can implicitly specify the task to be learned (see Figure 4). This may be useful for specifying that all groups are equally important to learn.
REFERENCES
Guillaume Alain, Alex Lamb, Chinnadhurai Sankar, Aaron Courville, and Yoshua Bengio. Variance reduction in sgd by distributed importance sampling. arXiv preprint arXiv:1511 1.06481, 2015.
Rohan Anil, Gabriel Pereyra, Alexandre Passos, Robert Ormandi, George Dahl, and Geoffrey Hinton. Large scale distributed neural network training through online distillation. arXiv preprint arXiv: 2018
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th annual international conference on machine learning, pp. 41 41-48, 2009.
Zalán Borsos, Mojmír Mutnj, and Andreas Krause. Coresets via bilevel optimization for continual learning and streaming. arXiv preprint arXiv:2006.03885. 2020.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005.14166 2020.
Cody Coleman, Christopher Yeh, Stephen Mussmann, Baharan Mirzasoleiman, Peter Bailis, Percy Liang, Jure Leskovec, and Matei Zaharia. Selection via proxy: Efficient data selection for deep learning. International Conference on Learning Representations, 2020.
Luke N Darlow, Elliot Crowley, Antreas Antoniou, and Amos Storkey. Cinic- is not imagenet or cifar-10. arXiv preprint 1rXiv:1810.03500 2018.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image worth 16x 16 words: Transformers for image recognition scale, 2021.
Sebastian Farquhar, Yarin Gal, and Tom Rainforth. On statistical bias in active learning: How and when to fix it. arXiv preprint arXiv:2101.11665 2021.
Yarin Gal and Zoubin Ghahramani. Dropout bayesian approximation: Representing model uncertainty deep learning. In international conference on machine learning, pp. 1050-1059. PMLR, 2016.
Yarin Gal, Riashat Islam, and Zoubin Ghahramani. Deep bayesian active learning with image data. In International Conference on Machine Learning, pp. 1183-1192, PMLR, 2017.
Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. Deberta: Decoding-enhaneed bert with disentangled attention. arXiv preprint arXiv:2006. 03654,, 2020.
Neil Houlsby, Ferenc Huszár, Zoubin Ghahramani, and Máté Lengyel. Bayesian active learning for classification and preference learning. arXiv preprint arXiv: 112.:574/ 2011.
10

---

Under review as a conference paper at ICLR 2022
Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Chen, HyoukJoong Lee, Jiquan Ngiam, Quoe V Le. Yonghui Wu, et al. Gpipe: Efficient training of giant neural networks using pipeline parallelism. Advances in neural information processing systems, 32: 103-112, 2019.
Angela H Jiang, Daniel L-K Wong, Giulio Zhou, David G Andersen, Jeffrey Dean, Gregory R Ganger, Gauri Joshi, Michael Kaminksy, Michael Kozuch, Zachary C Lipton, al. Accelerating deep learning by focusing on the biggest losers. arXiv preprint arXivv1910.00762, 2019.
Tyler B Johnson and Carlos Guestrin. Training deep models faster with robust, approximate impor- tance sampling. Advances in Neural Information Processing Systems, 31:7265-7275, 2018
Norman P Jouppi, Cliff Young, Nishant Patil, David Patterson, Gaurav Agrawal, Raminder Bajwa, Sarah Bates, Suresh Bhatia, Nan Boden, Al Borchers, et al. In-datacenter performance analysis of a tensor processing unit. In Proceedings of the 44th annual international symposium on computer architecture, pp. 1- 2017.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.0883 000361, 2020.
Angelos Katharopoulos and François Fleuret. Biased importance sampling for deep neural network training. arXiv preprint arXiv: 170.00/706,0043,, 2017.
Angelos Katharopoulos and François Fleuret. Not all samples created equal: Deep learning with importance sampling. In International conference on machine learning, pp. 2525-22 2234. PMLR, 2018.
Kenji Kawaguchi and Haihao Lu. Ordered sgd: A new stochastic optimization framework for empirical risk minimization. In International Conference on Artificial Intelligence and Statistics, pp. 669-679. PMLR, 2020.
Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. arXiv preprint arXiv:1609, 16044466, 2016.
Andreas Kirsch, Joost Van Amersfoort, and Yarin Gal. Batchbald: Efficient and diverse batch acquisition for deep bayesian active learning. arXiv preprint arXiv:1906.08/58. 2019.
Andreas Kirsch, Tom Rainforth, and Yarin Gal. Active learning under pool set distribution shift and noisy data. CoRR, abs/2106. 11719, 2021.
11711 11719
A. Krizhevsky and G. Hinton. Learning multiple layers of features from tiny images. Master's 's thesis, Department of Computer Science, University of Toronto, 2009.
Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11 (6(11))2277-2333 2324, November 1998.
Yann A LeCun, Léon Bottou, Genevieve B Orr, and Klaus-Rober Müller. Efficient backprop. In Neural networks: Tricks of the trade, pp. 9-48. Springer, 2012.
Mingkun Li and Ishwar Sethi. Confidence-baseed active learning. JEEE transactions on pattern analysis and machine intelligence, 28(8): 121-11211111111 2006.
Gaëlle Loosli, Stéphane Canu, and Léon Bottou. Training invariant support vector machines using selective sampling. In Léon Bottou, Olivier Chapelle, Dennis DeCoste, and Jason Weston (eds.). Large Scale Kernel Machines, pp. 301-3000000100000000000000000010 MIT Press, Cambridge, MA., 2007. URL http: //leon.botton
hwtpg//aaprr/aeee loos www.lll-aanuooooo--ooo
Ilya Loshchilov and Frank Hutter. Online batch selection for faster training of neural networks. arXiv preprint C.Vv15511.06445 2015.
Ninarch Mehrabi, Fred Morstatter, Nripsuta Saxena, Kristina Lerman, and Aram Galstyan. A survey on bias and fairness in machine learning. ACM Computing Surveys (CSUR), 54(6):1355, 2021.
11

---

Under review as a conference paper at ICLR 2022
Baharan Mirzasoleiman, Jeff Bilmes, and Jure Leskovec. Coresets for data-efficient training of machine learning models. In International Conference on Machine Learning, pp. 5950-6960. PMLR. 2020.
Jishnu Mukhoti, Andreas Kirsch, Joost van Amersfoort, Philip HS Torr, and Yarin Gal. Deterministic neural networks with appropriate inductive biases capture epistemic and aleatoric uncertainty. arXiv preprint arXiv:2102.1158,, 2021.
Kazuki Osawa, Siddharth Swaroop, Anirudh Jain, Runa Eschenhagen, Richard E Turner, Rio Yokota, and Mohammad Emtiyaz Khan. Practical deep learning with bayesian principles. In Proceedings of the 33rd International Conference on Neural Information Processing Systems, pp. 4287-4299, 2019.
Huy Phan. huyvnphan/pytorrc.ho ciffrr0.. Jan 2021. doi: 10.5281/2eeodo..443100
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision, 2021.
Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. Deepspeed: System optimiza- tions enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery &amp; Data Mining, pp. 3505-3506. 2020.
David Rolnick, Andreas Veit, Serge Belongie, and Nir Shavit. Deep learning is robust to massive label noise. arXiv preprint
2017.
Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized experience replay. arXiv preprint arXiv:151 1.05952. 2015.
Burr Settles. Active learning literature survey, 2009.
Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catan- zaro. Megatron-Im: Training multi-billin parameter language models using model parallelism. arXiv preprint arXiv:1909.0800 08053, 2019.
Yonglong Tian, Olivier Henaff, and Aaron van den Oord. Divide and contrast: Self-supervised learning from uncurated data. arXiv preprint arXiv:2 105. 08054, 2021.
Mariya Toneva, Alessandro Sordoni, Remi Tachet des Combes, Adam Trischler, Yoshua Bengio, and Geoffrey J Gordon. An empirical study of example forgetting during deep neural network learning. arXiv preprint arXiv: 1rXiv:1812.05159, 2018
Tong Xiao, Tian Xia, Yi Yang, Chang Huang, and Xiaogang Wang. Learning from massive noisy labeled data for image classification. In CVPR, 2015.
Chhavi Yadav and Léon Bottou. Cold case: The lost mnist digits. Advances in Neural Information Processing Systems 32, 2019.
Jiong Zhang. Hsiang-Fu Yu, and Inderjit S Dhillon. Autoassist: A framework to accelerate training of deep neural networks. arXiv preprint (rXXiv:9005.0338) 2019.
12


Reviews:
[{"review_id": "zB4IQPU-dxY", "reviewer": "Reviewer_Yofy", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n\nThe paper presents a method for speeding up the training of DNN models. The problem addressed in the paper is important given the long training demand by data-centric applications.\n\n\nWeakness:\n\n1. The claim about training speed-up is evaluated on smaller datasets, whereas the targeted problem appears real while training with huge data. Therefore, it will be of interest to see the performance of the proposed method on bigger datasets.\n\n2. A theoretical justification about training speed up is missing.\n\n3. More reasoning is required for the approximation used in Equation (5). How to ensure that Dt does not cover the relevant (most significant) data points. There may be chances that more decisive data samples are there in Dt  therefore this approximation may not hold true. \n\n4. Training speedup is measured in terms of a number of steps but what does a step consist of, is not clear. More discussion is required about Figure 2.\n\n5. Which parameters are tuned by Algorithm-1 is not clear.\n\n6. It will be of great interest to see the comparative performance of the proposed method in terms of wall-clock time in a constant experimental setting.", "summary_of_the_paper": "The paper presents a method for selecting the most appropriate data points and using them for training to realize the speedup in training. The method proposes the use of reducible loss using a novel selection function named as Predictive Information Content. The paper claims that proposed RHOLS function helps in reaching the benchmark accuracy in fewer steps.", "summary_of_the_review": "It is a good work with good technical contributions on an important problem. \nAuthors need to rectify / address the outlined concerns above. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "lKkvxsH2rWV", "reviewer": "Reviewer_BBTj", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Strengths\n\n1. In general, the paper is well-written, with the main ideas outlined clearly. The approach is well-motivated, backed by information-theoretical reasonings, and seems to be built upon established literature.\n2. One good contribution of the paper is the illustrations of why hard examples may be inappropriate in some application contexts.\n3. The simulation part of the manuscript was carefully constructed with appropriate experiments to support the main findings. The visualizations of the results are very appealing.\n4. The derivation of the PIC (and to some extent, the problem considered in this work) is motivated by developments in active learning but is adapted in a novel way. (This might be either a strength or a weakness, depending on the viewpoints, as well as the perceived importance of the proposed problem).\n\n### Weakness\n\n1. As stated above, the framework of the problem is basically an easier version of a classical active learning problem. In active learning, examples' labels are requested within a budget; in the context of this manuscript, they are already available on both the train set and the hold-out dataset. Aside from the novelty of the applications, other aspects of the approach resemble classical discussions in active learning. These include the contrast between hard/easy samples or aggressive/mellow learning, and the central insight that hard examples might be noisy (which is expected from the active learning literature, many of which are referenced in this work).\n2. The manuscript is a practical demonstration of an idea to prioritize samples for learning and doesn't have strong theoretical supports (that usually accompany active learning methods).", "summary_of_the_paper": "This paper introduces a new technique to select a sequence of training points for faster model training. The newly derived method is tractable, can improve efficiency through selections of examples that are neither too hard nor too easy, and is robust to noise.", "summary_of_the_review": "Overall, my vote for the paper is a (weak) reject. I enjoy reading the paper and think it has some good points and the discussions are meaningful. On the other hand, I think the technical contribution of the work is limited.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "bymQ9B8Xd1W", "reviewer": "Reviewer_NdhY", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "While the proposed is conceptually interesting, I have major concerns with the practicality of the proposed method. By assuming an irreducible loss model trained on hold-out the set, authors need access to an already trained model on a separate IID dataset. This seems like begging the task at hand. In particular, to train a model on some training set, Algorithm 1 (RHOLS) assumes access to a model trained on some hold-out set. Then the proposed criterion uses this irreducible model and the training model to identify/select a batch of examples at every iteration. \n\nOther concerns: \n- Approximation from 4 to 6 might not be accurate if the hold-out set is comparable or smaller than the training data. If the hold-out set is large then why would one not use that for training the model in the first place?  \n- It would be interesting to add some discussion on how is the irreducible model is selected. Some ablation experiments showing the differences from equation 4 to equation 6 can be insightful. \n \nI am open to changing my score if there is any misunderstandings with my interpretation. ", "summary_of_the_paper": "The authors propose a reducible held-out loss selection method for faster model training. Their method selects a sequence of training points that are referred to as \"just right\" for training.  The main methodological contribution of the paper is an information-theoretic selection criterion Predictive Information Content (PIC) to choose training points. Authors emprically show efficacy of their selection criterion over previously proposed baselines.", "summary_of_the_review": "Overall, while the proposed method is novel, there is some serious concern with the practicality of the RHOLS method because it assumes an already trained model on a hold-out set to train another model on training set. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "0eaxDwNXQom", "reviewer": "Reviewer_DHeZ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper proposes a technique to speed up training by another trained model as a proxy for how informative examples are. Specifically, starting with the reduction in loss as a criteria, the authors use Bayes theorem to come up with an approximation called the reducible held-out loss (RHOL). Basically, RHOL compares the differences between the current model trained so far and another model trained a separate holdout dataset. A larger difference indicates that the sample is informative and should be selected in the batch. \n\nOverall, I think the method is neat. However, it defeats the original purpose of the work, which is to present a method for accelerating training in the very large data regime. Suppose one has a large dataset with hundreds of millions of examples (as in CLIP or ViT). Using a model trained on a small holdout dataset as a proxy for informativeness is questionable. On the other hand, if the proxy model is trained on a large hold-out dataset, then training is not really accelerated since one has to train two models now. In the experiments, the authors use small datasets only (of CIFAR10/100 size). In addition, the holdout dataset they use is of a comparable size to the original training set (e.g. half of the images in CIFAR10 are used for holdout). In Page 4, the authors do state that their loss rests on the assumption that the holdout set is large when they argue that approximating h(y | x, holdout, D) with h(y |x, holdout) does not have a big impact.\n\nIn addition, there is the issue of reserving a labeled holdout set. In CIFAR10, for example, half of the examples are reserved. The authors should compare against the baseline method that is trained on the full data (both training and holdout) since that's really the alternative. I don't think that reserving a large subset of labeled examples and training another model on those is a good solution for \"accelerating\" training. \n\nExperimentally, the improvement in speed reported in the paper is around a factor of 2. Since another model is trained and a subset of the labeled data is reserved, I don't find the speedup significant enough for this approach to be useful.  \n\nSome minor comments:\n- The reducible holdout loss criteria is not \"information-theoretic\" as is claimed in the abstract.\n- It is not clear how selecting a subset of examples during training impacts minority groups. It is known, for example, that related effects (e.g. pruning) can have a disproportionate impact on minorities so that should be looked at as well. It is possible that the model selects typical examples first (by learning general rules that apply to them) and focuses on the minority examples later during training (by memorization). In that case, there would be error disparities between both subpopulations.\n\n========= \n\nPost Rebuttal: \n\nI have increased my score based on the new materials added to the paper that addressed some of my concerns. ", "summary_of_the_paper": "The paper proposes a technique to speed up training by using another trained model as a proxy for how informative examples are. The authors conduct experiments on small datasets (e.g. CIFAR10/100) on ResNet-18 to support their claims. ", "summary_of_the_review": "The paper is well-written and easy to read, and the proposed criterion is neat. However, implementing this method in practice for the purpose of speeding up training would require: (1) training another model, and (2) reserving a subset of the labeled data as holdout. I think both requirements constitute serious limitations that defeat the original purpose. \n\nIn addition, the experiments were conducted on small datasets only (at least ImageNet should have been included). In those experiments, the holdout data was of a comparable size to the training set and the improvement in speed is only around a factor of 2, which is not significant given the above concerns. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper provides a method to accelerate training by choosing a subset of points. After the initial submission, the reviewers raised a major concern about the practicality of the method. In the rebuttal phase the authors provided additional experiments on a large datasets that addressed this concern. That being said, the reviews are still quite borderline. The biggest remaining concern is about the quality of writing. Specifically, there are still requests to “fix the narrative” (NdhY, DHeZ). In addition, some details seem to remain vague regarding the positioning of the paper when compared to the active learning literature (BBTj).
Overall, the paper seems to have potential, especially with the new experiments. However, the changes it required when compared to the originally submitted version are simply too extensive to be thoroughly reviewed in a rebuttal phase.
Decision: Reject