Beyond neural scaling laws: beating power law scaling via data pruning
Authors: Ben Sorscher
Robert Geirhos
Shashank Shekhar
Surya Ganguli
Ari S. Morcos
Venue: NeurIPS 2022 Accept (2022)
Keywords: data pruning
neural scaling
data subsetting
replicate theory

Abstract: Widely observed neural scaling laws, in which error falls off as a power of the training set size, model size, or both, have driven substantial performance improvements in deep learning. However, these improvements through scaling alone require considerable costs in compute and energy. Here we focus on the scaling of error with dataset size and show how in theory we can break beyond power law scaling and potentially even reduce it to exponential scaling instead if we have access to a high-quality data pruning metric that ranks the order in which training examples should be discarded to achieve any pruned dataset size. We then test this improved scaling prediction with pruned dataset size empirically, and indeed observe better than power law scaling in practice on ResNets trained on CIFAR-10, SVHN, and ImageNet. Next, given the importance of finding high-quality pruning metrics, we perform the first large-scale benchmarking study of ten different data pruning metrics on ImageNet. We find most existing high performing metrics scale poorly to ImageNet, while the best are computationally intensive and require labels for every image. We therefore developed a new simple, cheap and scalable self-supervised pruning metric that demonstrates comparable performance to the best supervised metrics. Overall, our work suggests that the discovery of good data-pruning metrics may provide a viable path forward to substantially improved neural scaling laws, thereby reducing the resource costs of modern deep learning.

Beyond neural scaling laws: beating power law scaling via data pruning
Ben Sorscher
Robert Geirhos
Shashank Shekhar
Surya Ganguli
Ari W. Morcos Ai
equal contribution
Department of Applied Physics, Stanford University
University Tübingen
3 Meta Al (FAIR)
Joint senior authors
Abstract
Widely observed neural scaling laws, in which error falls off as power of the training set size, model size, or both, have driven substantial performance im- provements in deep learning. However, these improvements through scaling alone require considerable costs in compute and energy. Here we focus on the scaling of error with dataset size and show how in theory can break beyond power law scaling and potentially even reduce it to exponential scaling instead if we have access a high-quality data pruning metric that ranks the order in which training examples should be discarded to achieve any pruned dataset size. We then test this improved scaling prediction with pruned dataset size empirically, and indeed observe better than power law scaling in practice on ResNets trained on CIFAR- 10, SVHN, and ImageNet. Next, given the importance of finding high-quality pruming metrics, we perform the first large-scale benchmarking study of ten different data pruning metries on ImageNet. We find most existing high performing metrics scale poorly ImageNet, while the best are computationally intensive and require labels for every image. We therefore developed a new simple, cheap and scalable self-supervised pruning metric that demonstrates comparable performance to the best supervised metries. Overall, our work suggests that the discovery of good data-pra Mata-pruning metries may provide a viable path forward to substantially improved neural scaling laws, thereby reducing the resource costs of modern deep learning.
Introduction
many domains of machine learning, including vision, language, and speech, demonstrate that test error often falls off power law with either the amount training data, model size, or compute. Such power law scaling has motivated significant societal investments in data collection, compute, and associated energy consumption. However, power law scaling is extremely weak and unsustainable. For example, a drop in error
Empirically observed neural scaling laws 2 5 6
worr done during an internship at Meta Al (FAIR)
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

50-
Perceptron teac truchetstudent wetting
Perceptron tescher-tudentsstting
E
Test accuracy (N))
Theory Simulation
Frac data kags
Keep hard examples
1.0
kept
53
20-
Tost onoe (H6)
Frac dita kept
Japt
100%
10
Pareto frontier
Keep eayy sxamples
-0.
65%
28%
22%
2F
35"2
19
20
60
45
400
11%.
Total examples per parameter (Crot)
Training esamples per (ffgrane)
10%
Training examples per parameter [Pprunal
D
1.-
ResNet18 CIFAR-10
1.00
Information gain per ocample
B
Keep hard esumppii
Keep esamples
1.00
100)
1000 CURP 3018
T
3.00
abundant
Keep evey
Keep eany
when total
examples
-1.00
Tdp Total examples
Trakiing examples per parameter (ffgrame)
Figure 1: Our analytic theory of data pruning predicts that power law scaling of test error with respect to dataset size can be beaten. A: Test error as a function of Oprune = fow with = 0. We observe an excellent match between our analytic theory (solid curves) and numerical simulations (dots) of perceptron learning at parameters N=200 (here: N=200 constant throughout figure). The red curve indicates the Pareto optimal test error achievable from tradeoff between Ohot and f at fixed O'prune- B: We find that when data is abundant (scarce) corresponding to large (small) Cher. the better pruning strategy is to keep the hard (easy) examples. C: Color indicates difference in test error in keeping hard versus easy examples, revealing the change in strategy in (B). D: We tested this prediction on a ResNet18 trained on CIFAR-I0, finding remarkably the same shift in optimal pruning strategy under the EL2N metric. E: Test accuracy as a function of f and Oprone. For every fixed Oprune. there is an optimal fupt (purple curve). F: p( ) for different f.
from 3% to 2% might require an order of magnitude more data, compute, or energy. In language modeling with large transformers, drop in cross entropy loss from about 3.4 to 2.8 nat requires 10 times more training data (Fig. 1 in (2). Also, for large vision transformers, an additional 2 billion pre-training data points (starting from billion) leads to accuracy gain on ImageNet of a few percentage points (Fig. 1 in (7]). Here we ask whether we might be able to do better. For example, can we achieve exponential scaling instead, with a good strategy for selecting training examples? Such vastly superior scaling would mean that we could go from 3% to 2% error by only adding a few carefully chosen training examples, rather than collecting 10x more random ones.
Focusing on scaling of performance with training dataset size, we demonstrate that exponential scaling is possible, both in theory and practice. The key idea is that power law scaling of error with respect to data suggests that many training examples are highly redundant. Thus one should in principle be able to prune training datasets to much smaller sizes and train on the smaller pruned datasets without sacrificing performance. Indeed some recent works I0 have demonstrated this possibility by suggesting various metrics sort training examples in order of their difficulty or importance, ranging from easy or redundant examples to hard or important ones, and pruning datasets by retaining some fraction of the hardest examples. However, these works leave open fundamental theoretical and empirical questions: When and why is successful data pruning possible? What are good metrics and strategies for data pruning? Can such strategies beat power law scaling? Can they scale to ImageNet? Can we leverage large unlabeled datasets to successfully prune labeled datasets? We address these questions through both theory and experiment. Our main contributions are:
1. Employing statistical mechanics, we develop new analytic theory of data pruning in the student-teacher setting for perceptron learning, where examples are pruned based on their teacher margin, with large (small) margins corresponding easy (hard) examples. Our theory quantitatively matches numerical experiments and reveals two striking predictions:
However, note that nats is on a logarithmic scale and and small improvements in nats can lead to large improvements in downstream tasks.

---

(a) The optimal pruning strategy changes depending on the amount of initial data; with abundant (scarce) initial data, one should retain only hard (easy) examples.
(b) Exponential scaling is possible with respect to pruned dataset size provided one chooses an increasing Pareto optimal pruning fraction as function of initial dataset size.
2. We show that the two striking predictions derived from theory hold also in practice in much more general settings. Indeed we empirically demonstrate signatures of exponential scaling of error with respect to pruned dataset size for ResNets trained from scratch on SVHN, CIFAR-IO and ImageNet, and Vision Transformers fine-tuned on CIFAR-10.
3. Motivated by the importance of finding good quality metries for data pruming, we perform a large scale benchmarking study of 10 different data pruning metrics at scale on ImageNet, finding that most perform poorly, with the exception of the most compute intensive metrics.
4. We leveraged self-supervised learning (SSL) to developed new, cheap unsupervised data pruning metric that does not require labels, unlike prior metrics. We show this unsupervised metric performs comparably to the best supervised pruning metries that require labels and much more compute. This result opens the door to the exciting possibility of leveraging pre-trained foundation models to prune new datasets even before they are labeled.
Overall these results shed theoretical and empirical insights into the nature of data in deep learning and our ability to prune it, and suggest our current practice of collecting extremely large datasets may be highly inefficient. Our initial results in beating power law scaling motivate further studies and investments in not just inefficently collecting large amounts of random data, but rather, intelligently collecting much smaller amounts of carefully selected data, potentially leading the creation and dissemination of foundation datasets, in addition to foundation models [12].
2 Background and related work
Our work brings together 3 largely disparate strands of intellectual inquiry in machine learning: (1) explorations of different metrics for quantifying differences between individual training examples; (2) the empirical observation of neural scaling laws; and (3) the statistical mechanics of learning.
2.1 Pruning metrics: not all training examples are created equal
Several recent works have explored various metrics for quantifying individual differences between data points. To describe these metrics in a uniform manner, we will think of all of them as ordering data points by their difficulty, ranging from "casiest" to "hardest." When these metrics have been used for data pruning, the hardest examples are retained, while the casiest ones are pruned away.
EL2N scores. For example [10] trained small ensembles (of about 10) networks for very short time (about 10 epochs) and computed for every training example the average L2 norm of the error vector (EL2N score). Data pruning by retaining only the hardest examples with largest error enabled training from scratch on only 50% and 75% of CIFAR-10 10 and CIFAR-100 respectively without any loss in final test accuracy. However the performance of EL2N ImageNet has not yet been explored.
Forgetting scores and classification margins. 9 noticed that over the entire course of training, some examples are learned early and never forgotten, while others can be learned and unlearned (i.e. forgotten) repeatedly. They developed a forgetting score which measures the degree of forgetting of each example. Intuitively examples with low (high) forgetting scores can be thought of as easy (hard) examples. [9] explored data pruning using these metrics, but not at ImageNet scale.
I defined memorization score for each example, corresponding to how much the probability of predicting the correct label for the example increases when it is present in the training set relative to when is absent; large increase means the example must be memorized (i.e. the remaining training data do not suffice to correctly learn this example). Additionally [3] also considered an influence score that quantifies how much adding a particular example to the training set increases the probability of the correct class label of a test example. Intuitively, low memorization and influence scores correspond to easy examples that are redundant with the rest of the data, while high scores correspond to hard examples that must be individually learned. [13] did not use these
Memorization and influence.
3

---

scores for data pruning as their computation is expensive. We note since memorization explicitly approximates the increase test loss due to removing each individual example, it likely to be good pruning metric (though it does not consider interactions).
Ensemble active learning. Active learning iterates between training model and selecting new inputs to be labeled [14 [5 16, 17 [8]. In contrast, we focus on data pruning: one-shot selection of a data subset sufficient to train to high accuracy from scratch. variety of coreset algorithms (e.g. [19] have been proposed for this, but their computation is expensive, and so data-pruning has been less explored at scale on ImageNet. An early clustering approach (20) allowed training on 90% of ImageNet without sacrificing accuracy. Notably [1] reduced this to 80% by training large ensemble of networks on ImageNet and using ensemble uncertainty to define the difficulty of each example, with low (high) uncertainty corresponding to easy (hard) examples. We will show how to achieve similar pruning performance without labels or the need to train a large ensemble.
Diverse ensembles (DDD). 21 assigned a score to every ImageNet image, given by the number of models in diverse ensemble (10 models) that misclassified the image. Intuitively, low (high) scores correspond to easy (hard) examples. The pruning performance of this metric remains unexplored.
Summary. We note: (1) only one of these metrics has tested well for its efficacy in data pruning at scale on ImageNet: (2) all of these metrics require label information: (3) there is no theory of when and why data pruning is possible for any of these metrics; and (4) none of these works suggest the possibility of exponential scaling. We thus go beyond this prior work by benchmarking the data pruning efficacy of not only these metrics but also a new unsupervised metric we introduce that does not require label information, all at scale on ImageNet. We also develop an analytic theory for data-pruning for the margin metric that predicts not only the possibility of exponential scaling but also the novel finding that retaining easy instead of hard examples is better when data is scarce.
2.2 Neural scaling laws and their potential inefficiency
6 7 g has demonstrated that test loss L often falls off as a power law with different resources like model parameters (N), number of training examples (P), and amount of compute (C). However, the exponents / of these power laws are often close to 0. suggesting potentially inefficient use of resources. For example, for large models with lots of compute, so that the amount of training data constitutes a performance bottleneck, the loss scales as L R P-" Specifically for large transformer based language model, of magnitude increase in training data drops cross-entropy loss by only about 0.6 nats (Fig. 1 in (2). In neural machine translation experiments & varies across language pairs from 0.35 0.48 (Table in (5). Interestingly.
Recent work [
5
order 0.095, which implies
S explored a fixed computation budget C and optimized jointly over model size N and training set size P, revealing that scaling both N and P commensurately as C' increases is compute optimal, and can yield smaller high performing models (trained on more data) than previous work. Nevertheless, for a transformer based language model, a 100x increase in compute, corresponding to 10x increases in both model size and training set size, leads to a drop in cross-entropy loss of only about 0.5 nats (Fig. 2 in X Similar slow scaling holds for large vision transformers where adding 2 billion pre- pre-training images reduces ImageNet performance by a few percentage points (Fig.
77. While all of these results constitute significant improvements in performance, they do come at substantial resource cost whose fundamental origin arises from power law scaling with small exponents. Recent theoretical works [22 23 24) have argued that the power law exponent is governed by the dimension of a data manifold from which training examples are uniformly drawn. Here we explore whether we can beat power law scaling through careful data selection.
2.3 Statistical mechanics of perceptron learning
Statistical mechanies has long played role in analyzing machine learning problems (see e.g. [25 26 27 28 for reviews). One of the most fundamental applications is perceptron learning in the student-teacher setting (29 30 in which random i.i.d. Gaussian inputs are labeled by teacher perceptron to construct a training set. The test error for another student perceptron learning from this training set then scales as a power law with exponent - 1 for such data. Such perceptrons have also been analyzed in an active learning setting where the learner is free to design any new input to be

---

rather than choose from fixed set of inputs, as in data pruning. Recent work [3 has analyzed this scenario but focused on message passing algorithms that are tailored to the case of Gaussian inputs and perceptrons, and are hard to generalize to real world settings. In contrast we analyze margin based pruning algorithms that are used in practice in diverse settings, as in 9 10
labeled [1 32..
3 An analytic theory of data pruning
To better understand data pruning, we employed the replica method from statistical mechanics
34 to
develop an analytic theory of pruning for the perceptron in the student-teacher setting [25] (see App. A
for detailed derivations of all results). Consider training dataset of P examples {x* y I=
where x" E R are i.i.d. zero mean unit variance random Gaussian inputs and yd = sign (T xx)
are labels generated by a teacher perceptron with weight vector T E R We work in the high dimensional statistics limit where N, P - xi but the ratio Onot N of the number of total training examples to parameters remains (1). We then consider a pruning algorithm used [9] 10], namely: (1) train a probe student perceptron for very few epochs on the training data, obtaining weights probes (2) compute the margin m" = J probe xy"x") of each training example, where large (small) margins correspond to easy (hard) examples; (3) construct pruned dataset of size P prunee- fP. where is the fraction of examples kept, by retaining the Prrnne hardest examples, (4) train a new perceptron to
P
completion the smaller dataset with smaller ratio Openne
Prus of examples to parameters.
N
We are interested in the test error E of this final perceptron as a function of Cua. f. and the angle 0 between the probe student J probe and the teacher T. Our theory approximates J prote as simply random Gaussian vector conditioned to have angle with the teacher T. Under this approximation we obtain an analytic theory for 6 (( f, 0) that is asymptotically exact in the high dimensional limit (App.[A) We first examine results when 0 = 0, so we are pruning training examples according to their veridical margins with respect to the teacher (Fig. [T]. We find two striking phenomena, each of which constitute predictions in real-world settings that we will successfully confirm empirically.
The best pruning strategy depends on the amount of initial data. First, we note the test error curve for f = in Fig. [TA corresponding to no pruning. or equivalently to randomly pruning a larger dataset of size Osoo down to size Opprine. exhibits the well known classical perceptron learning power law scaling x Oprime. Interestingly though, for small Cwot keeping the hardest examples performs worse than random pruning (lighter curves above darkest curve for small prune in Fig. [T]. However, for large Oux. keeping the hardest examples performs substantially better than random pruning (lighter curves below darkest curve for large Oprune in Fig. [T]. turns out keeping the easiest rather than hardest examples is better pruning strategy when Ogoo is small (Fig. (C). If one does not have much data to start with, it is better to keep the easiest examples with largest margins (i.e. the blue regions of Fig. (B) to avoid overfitting. The casiest examples provide coarse-grained information about the target function, while the hard examples provide fine- fine-grained information about the target function which can prevent the model from learning if one starts with lots of data. In cases where overfitting is less of an issue, it is best to keep the hardest examples with smallest margin that provide more information about the teacher's 's decision boundary (i.e. the green region of Fig. B). Intuitively, in the limited data regime, it is challenging to model outliers since the basics are not adequately captured; hence, is more important to keep easy examples so that the model can get to moderate error. However, with a larger dataset, the easy examples can be learned without difficulty, making modeling outliers the fundamental challenge.
Fig. IC reveals which pruning strategy is best as a joint function of aso and f. Note the transition between optimal strategies becomes sharper at small fractions f of data kept. This transition between optimal pruning strategies can be viewed as a prediction in more general settings. To test this prediction we trained a ResNet18 on pruned subsets of the CIFAR- 10 dataset (Fig. [P], and observed strikingly similar behavior, indicating the prediction can hold far more generally, beyond perceptron learning. Interestingly, [9 [0] missed this transition, likely because they started pruning from large datasets.
Pareto optimal data pruning can beat power law scaling. A second prediction of our theory is that when keeping a fixed fraction of of the hardest examples as Owt increases (i.e. constant color curves in Fig. [A,, the error initially drops exponentially in O'rmune fawe. but then settles into the universal power law opeine for all fixed f. Thus there is asymptotic advantage to data
5

---

A
B
0
0 10
D
8 20
Jprobs
Simultion
Simulition
Froe dock
End 10
2000
10
10-1
Fmn 4676
K
From 24%
100%
2 3 Training examples parameter (Mppuse)
2 Triining examples per parameter [0pranee
Trainnng eiamples per parameter (Oposse)
Figure 2: Data pruning with an imperfect metric, A: Weight vectors and decision boundaries for a teacher (black) and probe student (red) separated by angle 0. The black point has margin 0 (x) w.r.t. the probe (teacher). B-D: Test error as a function of Oprune for different f and different 8.
pruning at fixed f. However, by pruning more aggressively (smaller /) when given more initial data (larger Oior), one can achieve a Pareto optimal test error as a function of pruned dataset size Openne that remarkably traces out at least exponential scaling law (Fig. [TA, purple curve). Indeed our theory predicts for each prnune a Pareto optimal point in Ckto and f (subject to apenne = fawa). yiclding for every fixed Oprune an optimal fopt. plotted in Fig. T Note fopt decreases with Oprnne indicating more aggressive pruning (smaller fap)) of original datasets of larger size Oia is required to obtain larger Pareto optimal pruned datasets of size Oprune- We will test this striking scaling prediction in Fig. 3
perspective. Classical randomly selected data generates slow power law error scaling because each extra training example provides less new information about the correct decision boundary than the previous example. More formally, let ((ara) denote the typical entropy of the posterior distribution over student perceptron weights consistent with a training set of size Ouo- The information gain I(Caa) due to additional examples beyond Ouo can be defined as the rate at which the posterior entropy is reduced: [(aw) = doue S((wa) In classical perceptron learning / (ato)) decays to zero as power law in Otot reflecting a vanishing amount of information per each new example, leading to the slow power law decay of test error Ohoi However, data pruning can increase the information gained per example by pruning away the uninformative examples. To show this, we generalized the replica calculation of the posterior entropy S and information gain / from random datasets of size Ohar to pruned datasets of size Oprune (App.[A) A).. We plot the resulting information gain I (((pmme) for different ʃ in Fig. T For any fixed f. (CEprune) will eventually decay as a power law as O Oprune- However, by more aggressively pruning (smaller f) datasets of larger size Cuot p(app can converge to finite value [(xc) = 1 nat/example, resulting in larger pruned datasets only adding useful non-redundant information. Since each new example under Pareto optimal data pruning conveys finite information about the target decision boundary, as seen in Fig. D the test error can decay at least exponentially in pruned dataset size as in Fig. TA. Classical results (30) have shown that training examples chosen by maximizing the disagreement of committee of student perceptrons can provide an asymptotically finite information rate, leading to exponential decay in test error. Intriguingly, the Pareto-optimal data pruning strategy we study in this work leads to faster than exponential decay, because includes (partial) information about the target function provided by the probe student (Fig.
Beating power law scaling: an information-theoree
An imperfect pruning metric yields a cross from exponential to power law scaling. We next examine the case of nonzero angle 0 between the probe student J Joobe and the teacher T, such that the ranking of training examples by margin is longer completely accurate (Fig. 2A)). Retaining the hard examples with smallest margin with respect to the probe student will always result in pruned datasets lying near the probe's decision boundary. But if 0 large, such examples might be far from the teacher's ss decision boundary, and therefore could be less informative about the teacher (Fig. 2 As a result our theory, confirmed by simulations, predicts that under nonzero angles 6 the Pareto optimal lower envelope of test error over both aeo and f initially scales exponentially as a function of ppune = Cmm but then crosses over to a power law (Fig. 2BCD). Indeed, at any given nonzero our theory reveals that as Oier (and therefore (pprane) becomes Targe, one cannot decrease test error any further by retaining less than a minimum fraction fmin (0) )) of all available data. For example when 0 = 10 (0- 20") one can do no better asymptotically than pruning down to 24% (46%) of the total

---

Percapiton ttucher-ttdeent wilnig
ResNattl SV-N
Reshect on CIFAR-10
Natsi Imagefeet
Test OF 1N0
33%
70%
33%
427
Pareto frander
Pando
footee
50%
30%
60%
70%
70%
Mando
20%
120%
Training moupples per
102
Trsning examplss
30
Training comples
Training ciamples
Figure 3: Beating power law scaling in practice. A-D: Curves of test error against pruned dataset size in 4 settings. Pruning scores were EL2N [TO] for CIFAR-10 and SVHN and memorization [I3] for ImageNet. See App. B for all pruning/training details and App. Dor for similar ImageNet plots with EL2N. Note solid curves reflect performance with fixed total dataset size; if we prune more aggressively with even larger datasets, scaling could improve further (c.g., dashed lines in A). Error curves with no data pruning (f 1) are labeled with their best-fit power law scaling a= (Note that for SVHN in B an asymptotic constant error E(P + oo) = 1.1% is subtracted from each of the curves to visualize the power law scaling more clearly.)
finetuned CIFA--10
10"-
RooWest0 Frectuned
Figure 4: Data pruning improves transfer learning. A: CIFAR-10 performance of a ViT pre-trained on all of ImageNet2 and fine-tuned on different pruned subsets of CIFAR-10 under the EL2N metric. B: CIFAR-10 performance of ResNet50s pre- trained on different pruned subsets of Ima- geNet1 and fine-tuned on all of CIFAR- 10.
97-
10%
20%
30%
Test enarr
20-1_
75%
95-
100%
0.95
04-
Random Momorizzation
Parato
fronteer
075
125
Fac of Iraggeett uood pretranning
10
103
104
Hne-funing examples
data (Fig. 2CD). As A approaches 0. fmin (0) approaches O, indicating that one can prune extremely aggressively arbitrarily small f while still improving performance, leading at least exponential scaling for arbitrarily large Oprune in Fig. 2.. However, for nonzero 0, the lack of improvement for f < fmin(() (0) at large Oprune renders aggressive pruning ineffective. This result highlights the importance of finding high quality pruning metrics with 0 x 0. Such metrics can delay the cross over from exponential power law scaling as pruned dataset size O'prane increases, by making aggressive pruning with very small f highly effective. Strikingly, in App. Fig. 10 we demonstrate this cross-over in real-world setting by showing that the test error on SVHN is bounded below by a power law when the dataset is pruned by a probe ResNet18 under the EL2N metric, trained for 4 epochs (weak pruning metric) but not a probe ResNet18 trained for 40 epochs (strong pruning metric).
4 Data pruning can beat power law scaling in practice
Our theory of data pruning for the perceptron makes three striking predictions which can be tested in more general settings, such as deep neural networks trained on benchmark datasets: (1) relative to random data pruning, keeping only the hardest examples should help when the initial dataset size large, but hurz when it small; (2) data pruning by retaining a fixed fraction f of the hardest examples should yield power law scaling, with exponent equal to that of random pruning, as the initial dataset size increases; (3) the test error optimized over both initial data set size and fraction of data kept can trace out Pareto optimal lower envelope that beats power law scaling of test error as a function of pruned dataset size, through more aggressive pruning at larger initial dataset size. We verified all three of these predictions on ResNets trained on SVHN, CIFAR-10 10, and ImageNet using varying amounts of initial dataset size and fractions of data kept under data pruning (compare theory in Fig. 3 with deep learning experiments in Fig. Tig-3BCD BCD). In each experimental setting we see better than power law scaling at larger initial data set sizes and more aggressive pruning. Moreover we would likely see even better scaling with even larger initial datasets (as in Fig 3 dashed lines).
7

---

Data pruning improves transfer learning. Modern foundation models are pre-trained on large initial dataset, and then transferred to other downstream tasks by fine-tuning on them. We therefore examined whether data-pruning can be effective for both reducing the amount of fine-tuning data and the amount of pre-training data. To this end, we first analyzed a vision transformer (ViT) pre-trained on ImageNet21 K and then fine-tuned on different pruned subsets of CIFAR-10. Interestingly, pre-trained models allow for far more aggressive data pruning; fine-tuning on only 10% of CIFAR- can match or exceed performance obtained by fine tuning all of CIFAR-I0 (Fig. AA). Furthermore Fig. E provides a new example of beating power law scaling in the setting of fine-tuning. Additionally, we examined the efficacy of pruning pre- training data by pre-training ResNet50s different pruned subsets of ImageNetIK K (exactly as in Fig. 3D) and then fine-tuning them on all of CIFAR-10 10. Fig. 4B demonstrates pre- pre-training on as little as 50% of ImageNet match exceed CIFAR-10 performance obtained by pre-training on all of ImageNet. Thus intriguingly pruning pre-training data different downstream task. Overall
on an upstream task can still maintain high performance these results demonstrate the promise of data pruning in transfer learning for both the pre-training and fine-tuning phases.
A
B
92
3.00
8.00
2.00
3.44
8.00
0.13
6:00
(hiddman
8000
0:34
1.0
8.12
02
42
8:51
0.54
9.51
2:20
21
Geddman
Do poasing (+-2 siD
ectie kerring
3 00
0:35
0:32
2.00
0.53
04
6.12
0.28
3:20
6.12
2.
protonypes
megatting
9:00
0:09
9.20
53
34
65
02
3:50
2:
1.
SHATO
pritetypes
E1.3 (23 model
mode]
8:00
0.33
0:42
0.04
0.34
3.7E
6-4L
madd Poul
0.95
09
0.65
5.
kept
0.5
CLZM
3.00
0.00
0.20
0.12
1.00
teamiing (Chioa 251
100
3.00
0.10
1323
5.10
0114
0.2
5.19
0.39
1:50
8:50
1.
Obinevs 199
Hardeon
well piodlrpes
1.0
3.5
0.
035
Free. data kepp
0.5
037
0.5
06
BLNN
BLN LK
Figure 5: Dataset pruning at ImageNet scale. A: Spearman's 's rank correlation between all pairs of ImageNet metric scores, along with hierarchical clustering (as provided by seaborn. clustermap). B: Benehmarking existing supervised metrics on ImageNet (top-5 validation accuracy). C: Comparing top-5 performance on ImageNet when pruning according to the best existing supervised metric (memorization) and our supervised and self-supervised prototype metrics. all cases, training on 80% of ImageNet approximates training 100%. See App. B for pruning and training details.
5 Benchmarking supervised pruning metrics on ImageNet
We note that the majority of data pruning experiments have been performed on small-scale datasets (i.e. variants of MNIST and CIFAR), while the few pruning metrics proposed for ImageNet have rarely been compared against baselines designed on smaller datasets. Therefore, it is currently unclear how most pruning methods scale to ImageNet and which method is best. Motivated by how strongly the quality of a pruning metric can impact performance in theory (Fig. 2 we decided to fill this knowledge gap by performing a systematic evaluation of 8 different supervised pruning metrics on ImageNet: two variants of influence scores [3]. two variants of EL2N [00, DDD (11). memorization [3], ensemble active learning [T and forgetting 9 See Section 2 for a review of these metrics. Additionally, we include two new prototypicality metrics that we introduce in the next section.
We first asked how consistent the rankings induced by different metrics are by computing the Spearman rank correlation between each pair of metrics 55). Interestingly, we found substantial diversity across metrics, though some (EL2N, DDD, and memorization) were fairly similar with rank correlations above 0.7. However, we observed marked performance differences between metrics: Fig BCC shows test performance when a fraction f of the hardest examples under each metric are kept in the training set. Despite the success of many of these metrics on smaller datasets, only
8

---

few still mathh performance obtained by training on the full dataset, when selecting a significantly smaller training subset (i.e. about 80% of ImageNet). Nonetheless, most metrics continue to beat random pruning, with memorization in particular demonstrating strong performance (Fig.5 C We note that data pruning on ImageNet may be more difficult than data pruning on other datasets, because ImageNet is already carefully curated to filter out uninformative examples.
We found that all pruning metries amplify class imbalance, which results in degraded performance. To solve this we used a simple 50% class balancing ratio for all ImageNet experiments. Further details and baselines without class balancing are shown in App.
Self-supervised data pruning through a prototypicality metric
Fig. Shswoss many data pruning metrics do not scale well to ImageNet, while the few that do require substantial amounts of compute. Furthermore, all these metrics require labels, thereby limiting their ability to prune data for large-scale foundation models trained on massive unlabeled datasets [2] Thus there is clear need for simple, scalable, sulf-supervveed pruning metrics.
To compute self-supervised pruning metric for ImageNet, we perform k-means clustering in the embedding space of an ImageNet pre-trained self-supervised model (here: SWaV (35), D, and define the difficulty of each data point by the Euclidean distance to its nearest cluster centroid, or prototype. Thus easy (hard) examples are the most (least) prototypical. Encouragingly, in Fig. 5FC, find our self supf-vuue prototype metric matches or exceeds the performance of the best supervised metric, memorization, until only 70-80% of the data is kept, despite the fact that our metric does use labels and is much simpler and cheaper to compute than many previously proposed supervised metrics. See App. Fig. for further scaling experiments using the self-supervised metric.
To assess whether the clusters found by our metric align with ImageNet classes, we compared their overlaps in Fig. (A. Interestingly, we found alignment for some but not all classes. For example, class categories such as snakes were largely aligned to a small number of unsupervised clusters, while other classes were dispersed across many such clusters. If class information is available, we can enforce alignment between clusters and classes by simply computing a single prototype for each class (by averaging the embeddings of all examples of this class). While originally intended to be an additional baseline metric (called supervised prototypes, light blue in 5C). this metric remarkably outperforms other supervised metrics and largely matches the performance of memorization, which is prohibitively expensive to compute. Moreover, the performance of the best self-supervised and supervised metries are similar, demonstrating the promise of supe-vuper pruning.
One important choice for the self-supervised prototype metric is the number of clusters k. We found, reassuringly, our results were robust to this choice: & can deviate one order of magnitude more or less than the true number of classes (i.e. 1000 for ImageNet) without affecting performance (App. F))
To better understand example difficulty under various metrics, we visualize extremal images for our self-supervised prototype metric and the memorization metric for one class (Fi Qualitatively, easy examples correspond to highly similar, redundant images, while hard examples look like idiosyncratic outliers. See App. Figs.
15 19 for more classes and metries.
Igg
wingle
self self-sueevised prototyces
memor
qurod bog
Imaguhelccon www.aaoywwwtteeeeeeeeeo hesthe
Figure 6: A: Heat map where each row denotes the probability that images in given cluster come from each ImageNet class. B: The four easiest and hardest images under our self-supervised pruning metric and the best previously published supervised metric (memorization, shown in C) for ImageNet class 100 (black swan).

---

7 Discussion
Summary. We have shown. both in theory and practice, how break beyond slow power law scaling of error versus dataset size to faster exponential scaling, through data pruning. Additionally we have developed a simple self-supervised pruning metric that enables us discard 20% of ImageNet without sacrificing performance, on par with the best and most compute intensive supervised metric.
Limitations. The most notable limitation is that achieving exponential scaling requires a high quality data pruning metric. Since most metrics developed for smaller datasets scale poorly to ImageNet, our results emphasize the importance of future work in identifying high quality, scalable metrics. Our self-supervised metric provides a strong initial baseline. Moreover, a key advantage of data pruning reduced computational cost due to training on smaller dataset for the same number of epochs as the full dataset (see App. Cp. However, we found that performance often increased when training on the pruned dataset for the same number of iterations as on the full dataset, resulting in the same training time, but additional training epochs. However, this performance gain saturated before training time on the pruned dataset approached that on the whole dataset (App. J thereby still yielding computational efficiency gain. Overall this tradeoff between accuracy and training time on pruned data is important to consider in evaluating potential gains due to data pruning. Finally, we found that class- bass-balanning was essential to maintain performance on data subsets (App. HH) Future work will be required to identify ways to effectively select the appropriate amount of class-balancing.
Ethical considerations. A potential negative societal impact could be that data-pruning leads to unfair outcomes for certain groups. We have done a preliminary analysis of how data-pruning affects performance on individual ImageNet classes (App. []. finding no substantial differential effects across classes. However proper fairness tests specific to deployment settings should always be conducted on every model, whether trained on pruned data or not. Additionally, we analyzed the impact of pruning on OOD performance (App. K)
Outlook: Towards foundation datasets. We believe the most promising future direction is the further development of scalable, unsupervised data pruning metrics. Indeed our theory predicts that the application of pruning metrics on larger scale datasets should yield larger gains by allowing more aggressive pruning. This makes data pruning especially exciting for use on the massive unlabeled datasets used to train large foundation models (e.g. 400M image-text pairs for CLIP (36). 3.5B Instagram images (37, 650M images for the DALLE-2 encoder [38] 780B tokens for PALM 39). If highly pruned versions of these datasets can be used to train large number of different models, one can conceive of such carefully chosen data subsets as foundation datasets in which the initial computational cost of data pruning can be amortized across efficiency gains in training many downstream models, just at the initial computational cost of training foundation models is amortized across the efficiency gains of fine-tuning across many downstream tasks. Together, our results demonstrate the promise and potential of data pruning for large-scale training and pretraining.
References
[1] Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad, Md Patwary, Mostofa Ali, Yang Yang. and Yanqi Zhou. Deep learning scaling is predictable, empirically. arXiv preprint arXiv:1712.000
2017.
[2] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown. Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint orXiv:2001.08366 2020.
[3] Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B Brown, Prafulla Dhariwal, Scott Gray, et al. Scaling laws for autoregressive generative modeling. arXiv preprint arXiv:2010. 14.200000.440,, 2020.
[4] Jonathan S. Rosenfeld, Amir Rosenfeld, Yonatan Belinkov, and Nir Shavit. A constructive prediction of the generalization error across scales. International Conference on Learning Representations, 2020.
[5] Mitchell A Gordon, Kevin Duh, and Jared Kaplan. Data and parameter scaling laws for neural machine translation. In Proceedings of the 2021 Conference Empirical Methods in
10

---

Natural Language Processing, pages 3915-5922, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.
[6] Danny Hernandez, Jared Kaplan. Tom Henighan, and Sam McCandlish. Scaling laws for transfer. arXiv preprint (tXii:220220129999 2021.
[7] Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transform- ers. arXiv preprint arXiv:2106. 04560, 2021.
[8] Jordan Hoffmann. Scbastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint arXiv:2203. 15556, 2022.
[9] Mariya Toneva, Alessandro Sordoni, Remi Tachet des Combes, Adam Trischler, Yoshua Bengio, and Geoffrey J Gordon. An empirical study of example forgetting during deep neural network learning. In ICLR, 2019.
[10] Mansheej Paul, Surya Ganguli, and Gintare Karolina Dziugaite. Deep learning on data diet: Finding important examples early in training. Adv. Neural Inf. Process. Syst., 34, December 2021.
[11] Kashyap Chitta, José M Álvarez, Elmar Haussmann, and Clément Farabet. Training data subset search with ensemble active learning. IEEE Trans. Intell. Transp. Syst., pages 1-12, 2021.
[12] Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.0722 2021.
[13] Vitaly Feldman and Chiyuan Zhang. What neural networks memorize and why: Discovering the long tail via influence estimation. Adv. Neural Inf. Process. Syst., 33:2881-289/ 2020.
[14] Burr Settles. Active learning literature survey. Technical Report, 2009.
[15] Antoine Bordes, Seyda Ertekin, Jason Weston, Léon Botton, and Nello Cristianini. Fast kernel classifiers with online and active learning. J. Mach. Learn. Res., 6(9), 2005.
[16] Zeyad Ali Sami Emam, Hong-Min Chu, Ping- Ych Chiang, Wojciech Czaja, Richard Leapman, Micah Goldblum, and Tom Goldstein. Active learning the ImageNet scale. arXiv preprini arXiv:2111. I. /2880, 2021.
[17] Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: A core-set approach. arXiv preprint arXiv: 178. 2017.
[18] Siddharth Karamcheti, Ranjay Krishna, Li Fei-Fei, and Christopher D Manning. Mind your outliers! Investigating the negative impact of outliers on active learning for visual question answering. arXiv preprint arXiv:2107.02331, 2021.
[19] Baharan Mirzasoleiman, Jeff Bilmes, and Jure Leskovec. Coresets for data-efficient training of machine learning models. In Hal Daumé lii and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 6950-6960. PMLR, 2020.
[20] Birodkar, Mobahi, and S Bengio. Semantic redundancies in Ihage-Classification datasets: 2019.
The 10% you don't need. arXiv preprint
[21] Kristof Meding, Luca M. Schulze Buschoff, Robert Geirhos, and Felix A. Wichmann. Trivial or dichotomous data difficulty masks model differences (on ImageNet and beyond). In International Conference on Learning Representations, 2022.
[22] Utkarsh Sharma and Jared Kaplan. Scaling laws from the data manifold dimension. Journal of Machine Learning Research, 23:1111334 2022.
[23] Yasaman Bahri, Ethan Dyer, Jared Kaplan, Jachoon Lec, and Utkarsh Sharma. Explaining neural scaling laws. CoRR, abs/2102.06701, 2021.
[24] Jonathan S. Rosenfeld. Scaling Laws for Deep Learning. PhD thesis, Massachusetts Institute of Technology, USA, 2021.
[25] A Engel and C V den Brocck. Statistical Mechanics of Learning. Cambridge Univ. Press, 2001.
[26] Madhu Advani, Subhaneil Lahiri, and Surya Ganguli. Statistical mechanics of complex neural systems and high dimensional data. J. Stat. Mech: Theory Exp., 2013(03):P03014 2013.
11

---

[27] Yasaman Bahri, Jonathan Kadmon, Jeffrey Pennington, Sam S Schoenholz, Jascha Sohl- Dickstein, and Surya Ganguli. Statistical mechanics of deep learning. Annual Review of Condensed Matter Physics, March 2020.
[28] Lenka Zdeborová and Florent Krzakala. Statistical physics of inference: thresholds and algorithms. Adv. Phys., (555)5533,5,, September 2016.
[29] E Gardner. The space of interactions in neural network models. J. of Physics A, 21:257-270, 1988.
[30] HS Seung, H Sompolinsky, and N Tishby, Statistical mechanies of learning from examples. Phys. Rev. A, 45(8) 1992.
[31] Yoav Freund, H Sebastian Seung, Eli Shamir, and Naftali Tishby. Information, prediction, and query by committee. Adv. Neural Inf. Process. Syst. 5, 1992.
[32] Hai-Jun Zhou. Active online learning in the binary perceptron problem. Commun. Theor: Phys., (2):2443, February 2019.
[33] Hugo Cui, Luca Saglietti, and Lenka Zdehorovà. Large deviations in the perceptron model and consequences for active learning. 2021.
[34] M Mezard, G Parisi, and M A Virasoro. Spin glass theory and beyond. World scientific Singapore, 1987.
[35] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 991 1-94492.. Curran Associates, Inc., 2020.
[36] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning. pages 1748-8765 PMLR, 2021.
[37] Dhruv Mahajan, Ross Girshick, Vignesh Ramanathan, Kaiming He, Manohar Paluri, Yixuan Li, Ashwin Bharambe, and Laurens Van Der Maaten. Exploring the limits of weakly supervised pretraining. In Proceedings of the European conference on computer vision (ECCV), pages 181-196, 2018.
[38] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06/25, 2022.
[39] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.0233: 2022.
[40] Priya Goyal, Quentin Duval, Jeremy Reizenstein, Matthew Leavitt, Min Xu, Benjamin Lefaudeux, Mannat Singh, Vinicius Reis, Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Ishan Misra. VISSL. https: //tts::///ithbb com/ httm//aacbookreeerr///vvss..
2021.
[41] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bemstein, et al. ImageNet large scale visual recognition challenge. International Journal of Computer Vision, 115(3):211-2555 2015.
[42] Kaiyu Yang, Klint Qinami, Li Fei-Fei, Jia Deng, and Olga Russakovsky. Towards fairer datasets: Filtering and balancing the distribution of the people subtree in the ImageNet hierarchy. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency, pages 547-558, 2020.
[43] Yuki M Asano, Christian Rupprecht, Andrew Zisserman, and Andrea Vedaldi. PASS: An ImageNet replacement for self-supervised pretraining without humans. arXiv preprint arXiv:2109.1 13228, 2021.
[44] Ross Wightman. Pytorch image models.
http::///gituub.com///igggg
pytorch- iww---maee-aaa models 2019.
[45] Justin M Johnson and Taghi M Khoshgoftaar. Survey on deep learning with class imbalance.
Journal of Big Data, 6(1):1-54, 2019.
12

---

[46] Robert Geirhos, Kantharaju Narayanappa, Benjamin Mitzkus, Tizian Thieringer, Matthias Bethge, Felix A Wichmann, and Wieland Brendel. Partial success in closing the gap between human and machine vision. Advances in Neural Information Processing Systems, 34:23885- 23899, 2021.
[47] Felix A Wichmann, David HJ Janssen, Robert Geirhos, Guillermo Aguilar, Heiko H Schütt, Marianne Maertens, and Matthias Bethge. Methods and measurements to compare men against machines. Electronic Imaging, Human Vision and Electronic Imaging, 2017(14):36-45, 2017.
[48] Robert Geirhos, Carlos RM Temme, Jonas Rauber, Heiko H Schütt, Matthias Bethge, and Felix A Wichmann. Generalisation in humans and deep neural networks. Advances in Neural Information Processing Systems, 2018.
[49] Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A. Wichmann, and Wieland Brendel. ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness. In International Conference on Learning Representations, 2019.
[50] Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global repre- sentations by penalizing local predictive power. Advances in Neural Information Processing Systems, 32, 2019.
[51] Robert Geirhos, Kristof Meding, and Felix A Wichmann. Beyond accuracy: quantifying trial- by-trial behaviour of CNNs and humans by measuring error consistency. Advances in Neural Information Processing Systems, 33, 2020.
[52] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers Surpassing human-level performance on ImageNet classification. In Proceedings of the IEEE International Conference Computer Vision, pages 1026-1034, 2015.
[53] John P Miller, Rohan Taori, Aditi Raghunathan, Shiori Sagawa, Pang Wei Koh, Vaishaal Shankar, Percy Liang, Yair Carmon, and Ludwig Schmidt. Accuracy on the line: on the strong correlation between out-of-distribution and in-distribution generalization. In International Conference on Machine Learning, pages 7721-7735. PMLR, 2021.
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See discussion section 7
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See discussion section a App.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes) See App.
(b) Did you include complete proofs of all theoretical results? [Yes] See App. A
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main exper- imental results (either in the supplemental material as a URL)? [No] All code for theory plots and numerical perceptron simulations packaged in a reproducible colab notebook.
(b) Did you specify all the training details (c.g.. data splits, hyperparameters, how they were chosen)? [Yes] See App. B
(c) Did you report error bars (e.g.. with respect the random seed after running experi- ments multiple times)? [No] Since we train a very large (order 100) number of models on computationally intensive tasks (e.g. ImageNet). many of our plots contain only results from a single random seed. However, variability due to random seeds can be inferred by the smooth progression of the relevant quantity each plot.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See App. B
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes] See App. B
(c) Did you include any new assets either the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [Yes] See App.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [Yes] See App. B
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "zSWvPjfZ6mA", "reviewer": "Reviewer_ycTR", "summary": "\nMotivated by the unscalable computational costs inherent to the power-law nature governing neural scaling laws in data, parameters and compute, and the insight that this powerlaw implies high redundancy (diminishing informational gain) among increasing amounts of samples the authors systematically evaluate the potential of data pruning theoretical and experimentally through the prism of data scaling properties. \n\nThe authors claim that reducing the generalization functional dependency in data to an $\\textit{exponential}$ dependency is expected both as predicted by an analytic theory they derive and observed in practice through experimentation --- when the data is pruned with a 'high-quality' data pruning metric. \n\n$\\textbf{Theory}$\n\nThe paper establishes a analytic theory of data pruning in the perceptron teacher-student setting --- at the limit of data $P \\rightarrow \\infty$ and perceptron dimension $N \\rightarrow \\infty$ jointly large, with a finite number of samples per dimension $\\alpha_{tot} = \\frac{P}{N}$). The pruning metric is the per-sample margin taken from an 'student-probe' (oracle) preceptron such that a fraction $f$ of the data is retained ranked by the metric, resulting in a smaller (pruned) number of retained samples per dimension $\\alpha_{prune}=f\\alpha_{tot}$. The theory predicts the generalization error of the student (a newly trained perceptron) as a function of number of samples per dimension $\\alpha_{prune}$, and the quality of the oracle, in this case the angle $\\theta$ of deviation of the student-probe perceptron from the teacher (the theory is derived for a general distribution along the probe-student decision boundary normal direction, and analytically solved for the case of ranking by margin, in the case of an originally isotropic Gaussian data distribution, prior to pruning).\n\nThe theory has several insightful and important predictions to the behavior of the generalization error of the student depending on the metric strength:\n\n$\\textit{Optimal pruning: }$\n* When data is 'abundant', restricting the samples to 'hard' (small margin) samples results in superior performance relative to no pruning --- i.e. in the sense that for the same number of samples per dimension, lower generalization error is attainable. When data is 'scarce', such restriction is detrimental in terms of generalization error as a function of training samples.\n\n$\\textit{Scaling functional dependency: }$\n- the error scales as a function of number of samples, for any given (pruned) dataset initially as a better than power-law (claimed exponential) phase which then reverts to a (universal) powerlaw.\n- examining the Pareto of optimal pruning ( as a function of initial dataset size and pruning degree ) better than powerlaw scaling is achievable limited by the quality of the pruning metric. Specifically, there is a lower bound in which the $\\textit{pareto}$ will transition to powerlaw which is associated with the maximal pruning (minimal data fraction $f_{min}$) of the dataset set by the student-probe quality.\n decrease in error \n\n$\\textit{Information theoretic, per sample contribution:}$\n\nThe theory further predicts the information gain per sample added per dimension $I(\\alpha_{prune})$\n\n$\\textbf{Experiments}$\n\nThe theoretical (asymptotic) predictions are shown to hold with excellent agreement in the $\\textit{finite}$ dimension ($N=200$) perceptron learning case trained on data $P_{prune} = N\\alpha_{prune}$ derived as different difficulty subsets of total data sizes of $P_{tot} = P_{prune}/f$ with various oracle qualities (ideal, where the oracle is the teacher $\\theta =0$ and non-ideal $\\theta >0$). \n\nFor real data on CIFAR10, SVHN and Imagenet:\n\n* Hard sample preference in the abundance of data (and vice versa) is shown to hold also in practice (CIFAR10)\n* The 'signatures' of initial steeper than powerlaw pareto frontiers in real world are shown for all datasets, pruned according to previous art and further developed (self supervised) metrics\n* the (lower limit transition to powerlaw and associated maximal pruning) limitation imposed by weak oracle / 'student probe' and how that limitation is pushed with the improvement (further training of the student probe) is shown (with SVHN results in the appendix).\n\n* Transfer learning (CIFAR10): \nAnalysis of pruning the up and downstream datasets is performed with superior performance than full data attained.\n\n$\\textbf{pruning metrics benchmarking and self-supervised metric introduction}$\n\nRecognizing the criticality of the pruning metric (as it sets a lower limit on the transition to powerlaw and max beneficial pruning factor $f$), the authors systematically analyze previous methods at imagent scale. Evaluating both their rank-agreement and relative performance down to $f=0.6$ .\n\nAll above methods require labels and the best performing are computationally intensive. The authors thus construct a self supervised (and supervised version) method and demonstrate its comparable (to best method) performance down to $f=0.8$ (and to memorization-metric-methid down to 70%). The method relies on a (embedding clustering based) self-typicality estimate  from strong pre-trained self-supervised model on Imagenet (SWaV).\n\n\n\n", "questions": "Questions/Requests/Recommendations embedded above, denoted (R#$\\alpha$)", "limitations": "Yes, the authors engage deliberately and adequately with this subject! Specifically uncovering and investigating pruning class imbalance effects --- for both previous methods benchmarked and developed methods --- and addressing it.\n\nThere are additional areas of investigation which are active open questions associated more broadly with the line of research dealing with data pruning and are not specific to this paper --- and should thus be, in this reviewers view, dealt with in other, future, work.\n\nSuch topics include vulnerability implications of training on pruned data. Specifically, it is an open question (to this reviewer's knowledge) how training on pruned data affects robustness to both naturally occurring phenomena (e.g. OOD performance) and bad actor actions (e.g. adversarial attacks, model stealing).\n", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "ֿ$\\textbf{Pre-face:}$\nIt is this reviewer's view that this is a paper of significant insight, methodological implications and multiple important contributions to the field. An exciting paper for its bridge of phenomena (generalization scaling laws, data pruning) and theory drawing and expanding on classical (statistical mechanics perceptron learning) theory, with inroads to practical understanding of implications, role and potential of data pruning, as well as practical contribution at large (Imagenet) scale of pruning (self supervised based) metrics critical to (limiting of) said potential realization.\n\nWith that said, the main improvements recommended are softening/solidifications of some claims - primarily by corresponding with previous art and adding further analysis where merited.\n\nAddressing the below recommendations/requests would improve the paper materially and would likely elevate this reviewer's recommendation further.\n\nDetails below:\n\n$\\textbf{Strengths}$\n\n* the expansion of classical — statistical mechanics perceptron learning — theory, to the data pruning setting is elegant and insightful.\nNotable is the excellent agreement of the $\\textit{asymptotic}$ theory in the $\\textit{finite}$ dimension (N=200) perceptron regime used throughout the paper.\nIt sets the stage for a better understanding, through its predictions which are consistent with observations in real data over both prior art and further introduced pruning methods. \n\nThe predictions are, at least, consistent with observations for practical (residual) networks at large (Imagenet) scale:\n* Easy / hard training advantage — figure 1C (preceptron), 1D (CIFAR10).\n\n* Better than powerlaw / exponential phase of individual constant hardness $f$ as well as resulting pareto — up to the metric limit (upon reverting to powerlaw). Predicted and shown for perceptron 1A, 1F, 3A. \nThis is consistent with observations in figure 3BCD. It is however hard to quantifiably assess the agreement of an actual exponent functional dependence in figure 3BCD or if it is already hitting a powerlaw limit.\n\n* prediction of metric quality on transition to limiting powelaw, and minimal data hardness ($f_{min}$). At real data scale --- consistent with observed $f_{min}$ powerlaw limit and (a little, fortunate) evidence that improved metric via student-probe further training allows to breach said limit Appendix C figure 7.\n\nBeyond theory:\n\n* While the theory does not deal with transfer learning, it is in practice demonstrated that a steep benefit manifests when downstream pruning (notably, adding visual transformer network into the fray).\nIt is worth to dwell on this, as it is different from the results portrayed in the regular pruning scenario in figure 3BCD. In these experiments, as the total dataset size is fixed, and one is only choosing a subset according to hardness ($f$) then no significant performance gain is evident.\nIn contrast, in the downstream Transfer Learning experiment 4A, a noticeable advantage in performance manifests when training on *less*   data (a subset).\n\nImplication strengths:\n\n* Data curation exponential gain potential:\nWhile the authors lead with computation implications, this requires further evidence (see below requests for clarification and concern that pruned data may actually be hard to train on). However, it should be noted, that even if computational gain at training is not available, there is significant value in reducing the burden of data collection and especially reducing data labeling quantities in supervised settings where these are expensive. These benefits are orthogonal to the potential computational savings. \n\n* Practical data curation $\\textit{without labels}$:\nThus, the introduction of an at-Imagenet-scale self-supervised metric which is able to match SOTA supervised data-pruning metrics, is a significant and novel (to the best of this reviewers knowledge) contribution. \n\nThe limitation here is that this assumes a strong (and is limited by its strength) metric (oracle / student-probe). See below elaboration on this in the weakness section.\n\n$\\textbf{Weaknesses}$\n\n$\\textit{Major:}$\n\n1. Computation --- are pruned datasets harder to learn?: \nadd an analysis (theory and experiments) of the computational effect of training on pruned data. numbered (R1a-c) below:\n\nIn order to make claims regarding computational savings which are at the premise of the paper's motivations, these need to be evaluated. There may be reason for concern here, that a pruned dataset does not translate to computational savings. To elucidate this point, hard examples are expected to be iterated upon numerous times (e.g. giving rise to the very notion of the memorization metric []), thus potentially negating the perceived computational benefit of having fewer samples.\nBy way of example and request --- even for the simple perceptron learning cases evaluated --- what were the actual times until convergence. We know the classical upper bound is inversely proportional to the margin squared, i.e. to the hardness (governed by $f$). What does this imply on the expected time to convergence in the statistical-mechanics limit? \nIn other words: $\\textbf{(R1a)}$ what is the theoretical convergence (e.g. relaxation to ground state from high energy initial state or other appropriate expected value) time, and $\\textbf{(R1b)}$ convergence in terms of number of iterations in practice (empirically) of the trained preceptors ? \n\nFor assessing this question in the real data scenarios, there is a need to adopt a well calibrated learning schedule, such that convergence times may be (properly) compared, such that the computational benefit may be assessed.\nWhile the paper mentions the comparison in terms of number of epochs and marginal benefit of further iterations in the real data experiments, consistent with a potential computational advantage, it seems that the training schedules are constant epochs [lines 727, 333]. Thus this does not necessarily demonstrate computational savings --- e.g. a different (shorter) schedule may have marginal (comparable) effect on the full data.\n\nPrevious work has dealt with this optimality question by using adaptive learning schedules and early stopping. \n\n$\\textbf{(R1c)}$ These CIFAR10, Imagenet computation scaling (at early stopping with adaptive learning) should be added to the paper also in order to significantly strengthen the assessment of the computational implications of data pruning (which as noted is one of the core impetuses the authors lead with, and rightfully so, as it is of significant practical implications). \n\nA potential easing of this computationally intensive evaluation (involving the re-training and assessment of multiple models, demanding primarily at imagent scale) may be given by examining the learning curves:\n\nPrevious work [1] showed close relationship between the online and finite data settings under adaptive learning schemes. Since the setting of a constant $f$ falls into the setting investigated by these works, there is potential to verify this finding here on a single $f$ learning curve vs models on the curve (different amounts of samples) and then (assuming it holds) examine just the training curves of the other models on the full data.\n\n2. The practical metric limitation. \n\nThe authors, rightfully, note the limitation imposed by the data pruning metric — effectively requiring a powerful oracle (or method) which sets a lower limit on the maximal pruning possible.\n\nNot that this problem is not one of mere theoretical inconvenience: The exact common scenario where one encounters new data, on which employing a powerful metric for reduction, would be potentially pertinent, is the case where out-of-distribution characteristics limit any practical oracle (regardless of how well in-distribution it was trained).\n\nFor this reviewer, there is no question that amortization (e.g. capitalizing on mutual information between ID, OOD) is a powerful concept.\nHowever a more quantitative analysis of the actual benefit and limitation would be beneficial.\n\nTo make this discussion more quantifiable (and interesting), let us consider the two scenarios of amortization effectively discussed in the paper:\nUse foundation model to imbue a metric, data prune according to the metric, train new model on the pruned model\nUse foundation model in transfer learning (TL) setting — e.g. fine tune on the new data.\n\n$\\textbf(R2)$ Is one superior to the other, in terms of computation? Performance? Quantifiably (theoretically?).\n\n$\\textit{Medium:}$\n\n3. Related work:\n\n$\\textbf{(R3a)}$ Scaling laws --- while the authors mention at the preface notable scaling law works, they miss early work [3] on neural scaling laws, theoretical work [4, 5] as well as pertinent work corresponding with the beyond-powerlaw potential of data-scaling [6]. Also relevant is [7] which explores a resulting pruning pareto — in the complementary setting of parameter pruning, also finding a pareto frontier with varying optimal pruning fraction as a function of original model size (total parameters).\n\n$\\textbf{(R3b)}$ Exponential scaling with data is a known (classical) result when 'high-quality' side information is available --- a correspondence and comparison with query based learning classical expectations of the Pareto frontier is merited (Req):\n\nThe Pareto presented, while methodologically and theoretically insightful, would do well to correspond with the prior art which classically presents expectations for better-than-powerlaw (e.g. exponential) scaling with samples, \\textit{when these are allowed to be chosen}. To put this in the most classical setting (pertinent to the paper) --- in the perceptron learning setting (finding a separating hyperplane), with access to a perfect oracle (i.e. $\\theta=0$), binary search will result in an error scaling exponential with number of samples (queries). \n\nSo with access to additional data (the metric) it is a classical expectation to have, at least, exponential scaling with queries.\nThe explicit addition of this known classical expectation (e.g. binary search), in this reviewer's view, does not erode the novelty of the paper which shows early signs of this prediction in practice, but rather further solidifies its theoretical groundings.\n\n$\\textbf{(R3c)}$ Relatedly, corresponding with curriculum learning in the related work section is advisable --- e.g. as related to the notion of realizing the pareto frontier.\n\n4. Implications of the hard/easy predictions on pareto:\n\nThe authors empirically evaluate a easy (large-margin p(z) specialization — blue in figure 1B) vs hard (small-margin, green in 1B) and show a predicted cross-over.  \nHowever, in the remainder of the paper, derivations and well as experiments, it is the small-margin case only which is considered (to the best of this reviewer's understanding), and in particular as giving rise to the pareto frontier and $f_{opt}(\\alpha_{prune})$.\n$\\textbf{(R4a)}$ - This reviewer could not find the derivation of f_{opt} (figure 1E), please add it or if not derived mention that.\n\nFurther, in light of the crossover, it is this reader’s understanding that the true optimal pareto is upper bounded by a pareto $\\textit{lower} $ than the one used throughout the paper. I.e. the crossover implies that taking the minimum of the large-margin (not explicitly derived) and small-margin pruning policies for any given point ($f, \\alpha_{prune}$) has a section lower than the pareto in the paper, up to that switching point. $\\textbf{(R4b)}$ — it would be good to at least make this point clear in the paper, and to put on a graph like 1A the pareto drawn by this minimum (the derivation, though tedious, would entail the complementary large margin truncated gaussian specialization of $p(z)$, alternatively… just empirically for the trained perceptrons).\n\n\nThe reason that even this, is viewed by this reviewer as an $\\textit{upper bound}$ on the pareto frontier is that there is an expected optimal choice of pruning policy / metric usage $p(z|\\alpha_{prune},f)$. This policy has a margin support as a function of $(\\alpha_{prune},f)$ amenable to the paper derivation framework $\\textbf{(R4c)}$ — is this is correct? If so, can you derive / or at least simulate it? Or, alternatively, show that it is expected to have negligible / non-important effect?). \n\n5. experimental discriminative power of Imagenet pruning methods — need for repetition and clarification:\n\nThis is actually a very important methodological point, it is only making the medium cut, because even if all these methods are actually comparable, the concept of having a self supervised method in the ballpark of supervised, expensive, metrics is in-itself a significant contribution.\nHowever, it would be much preferable if the authors modify the experiment such that more statistical confidence in the claims is readily observable — as it currently stands, the level of noise in the experiment (figure 5C) is high. $\\textbf{(R5)}$ The self supervised experiments as well as the random baseline should have (at least) 3 repetitions, or alternatively, at least a X3 finer pruning grid (‘fractions of data kept’).\nEspecially worrisome is the drop of the proposed self-supervised method below random pruning. \n\nMinor:\n\n6. Clarity of content — general:\n\n$\\textbf{(R6)}$ Please add experimental descriptions where missing experiments and figures, reference to appendix OK. This reader spent significant time divining the experimental setups. Examples: how were the perceptron trained? Is this the classic perceptron learning algorithm? What were the grids of evaluation of all experiments (they vary across figures) etc. \n\n\n7. Typos and minor clarifications:\n\nFigure 1A — garbled $f$ values in legend\n\nCaption of figure 1:\n\n* “The read curve indicates … at fixed $\\alpha_{prune}$ would be better understood if read “The read curve indicates … as function of $\\alpha_{prune}$\n\n* \"E : Test accuracy … f_{opt} (blue curve)” — is it blue ?? looks purple, maybe change to black…\n\nLine 197: “ … exponential scaling law (Fig 1C…” $\\rightarrow$should be “ … exponential scaling law (Fig 1E…” \n\n\n$\\textbf{References:}$\n\n[1] Nakkiran, Preetum, Behnam Neyshabur, and Hanie Sedghi. \"The deep bootstrap framework: Good online learners are good offline generalizers.\" arXiv preprint arXiv:2010.08127 (2020).\n\n[2] Hernandez, Danny, et al. \"Scaling laws for transfer.\" arXiv preprint arXiv:2102.01293 (2021).\n\n[3] A Constructive Prediction of the Generalization Error Across Scales. Jonathan S. Rosenfeld, Amir Rosenfeld, Yonatan Belinkov, Nir Shavit. 2020 International Conference on Learning Representations. https://openreview.net/forum?id=ryenvpEKDr\n\n[4] Scaling Laws from the Data Manifold Dimension. Utkarsh Sharm, Jared Kaplan. 2022 Journal of Machine Learning Research. http://jmlr.org/papers/v23/20-1111.html\n\n[5] Explaining Scaling Laws of Neural Network Generalization.Yasaman Bahri, Ethan Dyer, Jared Kaplan, Jaehoon Lee, Utkarsh Sharma 2021 https://arxiv.org/abs/2102.06701\n\n[6] Scaling Laws for Deep Learning. Jonathan S. Rosenfeld. 2021. https://arxiv.org/abs/2108.07686\n\n[7] Rosenfeld, Jonathan S., et al. \"On the predictability of pruning across scales.\" International Conference on Machine Learning. PMLR, 2021.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "sS31f1D0QD6", "reviewer": "Reviewer_NsH6", "summary": "This paper demonstrates (both theoretically and empirically) that power-law scaling of the error with dataset size can be improved to exponential scaling using high-quality data pruning metrics. First, significant theory work is developed to back this prediction, which is then validated at small scale with ResNet models trained on CIFAR10. The authors notably study the effect of an imperfect \"oracle\" to inform the data pruning metric, showcasing the importance of the oracle quality. Finally, they demonstrate their approach at larger scale with ResNet/ViT models, and study various oracle metrics, introducing their own self-supervised  as well, which performs competitively against costlier metrics.", "questions": "_(Questions/suggestions below follow the numbering from the section above.)_\n\n* **S1**. The paper could benefit from clarifying the fixed multi-epoch setting earlier to streamline reading.\n\nFor the rest of the points mentioned in the section above, I do not expect any direct answer for a rebuttal round, unless the authors have available experiments/justifications to these points. ", "limitations": "The authors discuss the limitations of their approach thoroughly throughout the paper.", "rating": 9, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\nThis is an outstandingly good paper, which benefits from: (1) strong and thorough theoretical backing; (2) great empirical design to validate the theoretical findings and demonstrate them in \"real-world\" conditions; (3) good clarity throughout with high-quality figures and an easy-to-follow line of reasoning. \n\nThe proposed idea is obviously very interesting, as improving over scaling laws could yield to significant gains across the board. Many studies have looked at this under the angle of architectural change, but the data angle taken here is under-explored. \n\n## Weaknesses\n*(All the points below are indexed and linked to questions in the next section, which the authors can answer to clarify/improve the paper. They are ordered by level of importance.)*\n\n* **W1. _(Minor)_ The fixed multi-epoch setting should be clarified and mentioned earlier.** From Appendix B and the discussion l332, the method operates in a \"fixed epoch\" setting. Data is pruned, and then comparisons are made for the same number of training epochs as when training on the full dataset. This is not immediately evident from reading the paper, and clarifying it from the start would help streamline the reading of this work. Similarly, it would be interesting to consider such approches for pretraining (single-epoch) -- but this is beyond the scope of the paper. \n\n* **W2. _(Minor)_ Statement around 10x more training data only reduces loss from 3.4 nets to 2.8 nats feel a bit overstated.** On l27 and l128, the authors indicate that an order of magnitude in data only changes less by 0.6 nats. This is strictly-speaking true, but a bit misleading: cross-entropy loss is an a log-scale, so small changes are actually larger than they appear. Moreover, in large language models, small changes in upstream loss can have dramatic effect on downstream tasks. \n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "gdx2EhiGyY9", "reviewer": "Reviewer_D9bn", "summary": "The paper considers an interesting question of whether we can \"beat\" power-law scalings in data with better pruning strategies.\nThe authors first introduce a theoretical model to study the effect of pruning and its dependence on a few natural parameters (pruning ratio,  initial dataset size), and show that its theoretical analysis matches empirical simulations.\nThen, the authors also observe similar behavior on real datasets.\nThe authors then motivate the importance of a good pruning metric with both theoretical (quantifying the misalignment between student and teacher models) and empirical results (beating power law scaling becomes only possible with a sufficiently good pruning metric).\nThen, the authors carry out a thorough evaluation of existing pruning metrics on large scale datasets, and also develop a new metric which does not require any labels\nOverall, the work demonstrates the possibility of beyond-power law scaling and evaluates many different pruning metrics in large scale settings.", "questions": "Main:\n- Why did the authors not reproduce the scaling experiments for Figure 3 using the various metrics (and particularly the newly developed self-supervised metric?) Was it due to computational budget limitations or some other reason?  Given the overall narrative in the paper, I wanted to see if using better pruning metrics lead to better scaling behavior / better pareto optimal frontier, but I couldn't find an experiment along this line in the paper/Appendix besides Figure 7 (which are based on variations of an existing pruning metric rather than the newly introduced metric). \nIt leaves me wondering if the results were negative, and hence not presented.\nIn any case, I think demonstrating the impact of the quality of various (existing) pruning metrics on the scaling behavior is much more important than the particular metric introduced in the paper.\n\nOthers (some are just curiosity on my part):\n- Is it possible to intuitively gleam the the empirical scaling behavior (for the perceptron model) from the equations given in A.2? Or is numerically solving them the only way?\n- I liked the experiment on the transfer setting (as it is arguably more relevant in practice), but was a bit disappointed it was only covered briefly. Using your theoretical results/framework, is there something interesting that you can say about this setting?\n- From my loose familiarity with the theoretical literature in DL, I understand that people are now capable of giving interesting theoretical results on certain regimes of two layer networks (e.g. NTK). I'm curious, is it possible to extend the perceptron results in the paper to those settings?", "limitations": "Yes (none other than my questions/concerns above).", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "*Strengths*\n- Originality: The work is original in its thesis as well its details: theoretical analysis of pruning on different factors and corresponding empirical demonstration of beyond power law scaling. Related work is well cited.\n I also particularly enjoyed the analysis of the misalignment between student and teacher, and how it impacts optimal pruning strategy.\n- Soundness: The work is overall quite thorough and sound, with theory matching simulation, and a wealth of empirical experiments. That said, there were a few logical/experimental gaps in and leading into the second part of the paper (semi-supervised metric) that I was not sure if addressed properly (see Weaknesses and Questions).\n (I cannot speak much for the theory parts, as I am not familiar with the seemingly standard tools for theoretical analysis on this topic using statistical physics, but I am confident in my evaluation of empirical results.)\n- Clarity: Overall well written and polished. That said, exposition could be improved. In particular, Section 3 is a bit hard to follow due to overuse of mathematical notation and long paragraphs. The logical flow throughout the paper needs some work too (see Weakness 1).\n- Impact: Provides an interesting new perspective (are we limited to power law scaling?), and I believe that the theoretical results as well as empirical results will be valuable to future work in the area.\n\n*Weaknesses*\n- While the paper overall is insightful, it suffers from a suboptimal narrative/focus as well as missing a few important experiments. The paper seems to be sort of two papers stapled into one, unless read very carefully:\nIn the first half, the focus is on scaling behavior, and its theoretical and empirical analysis; the second half focus on a better unsupervised pruning metric for larger scale dataset (ImageNet). \n\nThere is an important logical transition from the first part to the second part, namely in the last paragraph of Section 3 (reference to App. Fig 7 on showing that beyond power law scaling becomes possible for ResNet18 only with a better pruning metric.)\nBut this was easy to miss, and I think this logical transition should be emphasized (perhaps by bringing the Figure into the main paper).\nSo I think this \"weakness\" of the paper is partially addressable with better prose and organization / focusing of results.\n\nThat said, I am left unconvinced by the rigor in which \"beyond power law scaling\" is claimed in the the results in Fig 7b. First, there are only two data points on the scatterplot below the the power law line, and quite close, so it's unclear to me if it's just noise.\nAlso, what prevents me from drawing a slightly steeper power law line that also includes these two data points? (it's possible I missed some justification.)\nIt would be interesting to see if there is some kind of statistical test that can be done to reject the \"null hypothesis\" (power law scaling) with some confidence.\n\n- The benefit of the new semi-supervised metric was not clear to me due to limited experiments on its scaling (see first question). In particular, I would have liked to see a demonstration that the \"pareto optimal\" frontier can be expanded by using this new metric.\nAlso, it is a bit strange to use a semi-supervised model pretrained on ImageNet to prune datasets for ImageNet. \nMore appropriate would be to take a model pretrained on some other dataset (e.g., CLIP) and using that instead.\nGiven these, I felt that the self-supervised metric distracted from the strengths of the rest of the paper rather than adding to them.\n\n- A meta-comment related to above two points. I think the paper would be more impactful if it focused on a subset of its results with a tighter narrative. \n\nI hope to adjust to scores appropriately if my concerns/questions are well addressed.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "g5A94siqbgz", "reviewer": "Reviewer_QtTy", "summary": "This paper explores neural scaling laws and looks at the power law scaling behavior of error with dataset size. While power law scaling has shown that larger dataset sizes can improve accuracy, it comes at a significant cost as power scaling is extremeley weak and unsustainable. This paper demonstrates that exponential scaling is in-fact possible both in thoery and practice, provided we chose the right dataset to train on. Power law scaling indicates that many training examples are hihgly redundant. Thus, one should be able to train on a much smaller sized dataset. This requires data selection or pruning strategies. Prior work has shown this. However, prior work lacks theoretical and empirical rigor. It leaves many questions unanswered like when is data pruning possible, how to do it at scale, can it beat power law scaling and can pruning be done using unlabeled dataset. \n\nThis paper answer these questions by by first unify existing literature on dataset pruning, statistical mechanics of learning and observations of neural scaling law to provide an analytical theory of data pruning. This theory and its associated empirical results help answer many of the questions asked above including showing how to achieve exponential scaling with dataset. They show that their theory is applicable in real world use cases also, like CIFAR10 and SVHN. To scale to ImageNet sized dataset, it is important for the data pruning metric to be cheap to calculate and also not require labeled dataset. In order to achieve this objective, they propose a clustering based unsupervised data pruning metric that performs equally well or better than existing data pruning techniques while still providing exponential scaling capabilities .\n", "questions": "Questions -\n\n1. In Figure 1A, if one draws a vertical line for a fixed alpha_prune, does that represent accuracy/test error at iso-dataset size for various experiments? If yes, how is that possible given that alpha_tot is fixed. If no, \n\n2. Figure 1E’s description suggests a blue curve, 1E does not have any blue curve.\n\n3. Its unclear reading the figure 1 and the description, as to how to achieve exponential scaling. Given a fixed alpha_total and ability to prune by a fraction f, is the idea to decrease f such that for large f (less training data), use easy examples but for small f (more training data) use hard examples?\n\n4. In Figure 4A, a vertical like at x = 10^3 interesects different lines representing different fraction of original dataset. Will all those points be iso-size? If yes, how, given that they all start from the same dataset size.\n\n5. Overall, given that this work assumes a large dataset is always available. Given a large dataset, one can train on a small fraction and achive a better accuracy using intelligent data pruning techniques. This makes it an approach towards efficient training (mentioned in line 336). Is this understanding correct? If yes, this is approach becomes a bit orthogonal to neural scaling law work whose aim is to understand how much resources (data, compute etc) will be needed to improve the accuracy of the model. Neural scaling laws guide investments into various accuracy improvement strategies. ", "limitations": "Have the authors adequately addressed the limitations and potential negative societal impact of their work? - Yes", "rating": 8, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths - \n\n1. The paper targets an important problem of data quality and its impact on accuracy\n\n2. The paper evaluates its claims on a wide range of datasets and backs them with strong theoretical foundation\n\n3. Insights around the kind of data to prefer at what scale were interesting and novel. Preferring easy samples in low resource regime and hard samples in large dataset regime was something that was not quantified before in prior work\n\n4. They show a simple, easy to use and unsupervised data pruning technique to measure the hardness of a sample.  The technique further scales to a larger dataset. The simplicity of the technique makes it a really strong contribution\n\n5. This is the first paper that shows exponential scaling is possible, given a high quality dataset. Where quality refers to how much does a model learn from each sample in the dataset. These results can further fuel more research in dataset creation.\n\n\n\nWeakness - \n\nThe results in the paper are really difficult to parse, making it very difficult to match the figures with the claims in the paper. I do believe that the paper requires some work in its presentation to make it readable to a general audience. Section 3’s Figures which forms the foundation of the claims in this paper are not well explained. Below are some instances\n\n1. Section 3 introduces alpha_tot as a ratio of total training examples (P) and number of parameters in the model (N). alpha_pruning is basically f*alpha_tot where f represents the fraction of the  examples selected. Using this knowledge (as established in lines 159-165) seems incomplete to parse Figure 1. For example, in Figure 1A x axis represents alpha_prune as an increasing variable. That means, f is decreasing. However, the figure also has datapoints associated with different f for the same alpha_prune. Given that alpha_tot is constant, how is that possible? \n\nThere are other similar issues in readability (See questions 1,2,3) that make it really difficult to understand the claims. This makes parsing the next section that shows these results on CIFAR-10 and SVHN equally hard. \n\n2. The technique described in the paper is post-hoc - Given a dataset, how do you prune it to a few high quality samples to reduce the number of samples to train on. This can potentially reduce the training time of the model (line 336). But this aspect was not discussed in detail in the paper. Given that this paper does not discuss how to build a high quality dataset, its impact can be limited. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}]

Author rebuttal:

Decision: Accept