Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 9: Very Strong Accept: Technically flawless paper with groundbreaking impact on at least one area of AI/ML and excellent impact on multiple areas of AI/ML, with flawless evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper demonstrates (both theoretically and empirically) that power-law scaling of the error with dataset size can be improved to exponential scaling using high-quality data pruning metrics. First, significant theory work is developed to back this prediction, which is then validated at small scale with ResNet models trained on CIFAR10. The authors notably study the effect of an imperfect "oracle" to inform the data pruning metric, showcasing the importance of the oracle quality. Finally, they demonstrate their approach at larger scale with ResNet/ViT models, and study various oracle metrics, introducing their own self-supervised  as well, which performs competitively against costlier metrics.
strengths_and_weaknesses: ## Strengths

This is an outstandingly good paper, which benefits from: (1) strong and thorough theoretical backing; (2) great empirical design to validate the theoretical findings and demonstrate them in "real-world" conditions; (3) good clarity throughout with high-quality figures and an easy-to-follow line of reasoning. 

The proposed idea is obviously very interesting, as improving over scaling laws could yield to significant gains across the board. Many studies have looked at this under the angle of architectural change, but the data angle taken here is under-explored. 

## Weaknesses
*(All the points below are indexed and linked to questions in the next section, which the authors can answer to clarify/improve the paper. They are ordered by level of importance.)*

* **W1. _(Minor)_ The fixed multi-epoch setting should be clarified and mentioned earlier.** From Appendix B and the discussion l332, the method operates in a "fixed epoch" setting. Data is pruned, and then comparisons are made for the same number of training epochs as when training on the full dataset. This is not immediately evident from reading the paper, and clarifying it from the start would help streamline the reading of this work. Similarly, it would be interesting to consider such approches for pretraining (single-epoch) -- but this is beyond the scope of the paper. 

* **W2. _(Minor)_ Statement around 10x more training data only reduces loss from 3.4 nets to 2.8 nats feel a bit overstated.** On l27 and l128, the authors indicate that an order of magnitude in data only changes less by 0.6 nats. This is strictly-speaking true, but a bit misleading: cross-entropy loss is an a log-scale, so small changes are actually larger than they appear. Moreover, in large language models, small changes in upstream loss can have dramatic effect on downstream tasks. 


questions: _(Questions/suggestions below follow the numbering from the section above.)_

* **S1**. The paper could benefit from clarifying the fixed multi-epoch setting earlier to streamline reading.

For the rest of the points mentioned in the section above, I do not expect any direct answer for a rebuttal round, unless the authors have available experiments/justifications to these points. 
limitations: The authors discuss the limitations of their approach thoroughly throughout the paper.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: 
Motivated by the unscalable computational costs inherent to the power-law nature governing neural scaling laws in data, parameters and compute, and the insight that this powerlaw implies high redundancy (diminishing informational gain) among increasing amounts of samples the authors systematically evaluate the potential of data pruning theoretical and experimentally through the prism of data scaling properties. 

The authors claim that reducing the generalization functional dependency in data to an $\textit{exponential}$ dependency is expected both as predicted by an analytic theory they derive and observed in practice through experimentation --- when the data is pruned with a 'high-quality' data pruning metric. 

$\textbf{Theory}$

The paper establishes a analytic theory of data pruning in the perceptron teacher-student setting --- at the limit of data $P \rightarrow \infty$ and perceptron dimension $N \rightarrow \infty$ jointly large, with a finite number of samples per dimension $\alpha_{tot} = \frac{P}{N}$). The pruning metric is the per-sample margin taken from an 'student-probe' (oracle) preceptron such that a fraction $f$ of the data is retained ranked by the metric, resulting in a smaller (pruned) number of retained samples per dimension $\alpha_{prune}=f\alpha_{tot}$. The theory predicts the generalization error of the student (a newly trained perceptron) as a function of number of samples per dimension $\alpha_{prune}$, and the quality of the oracle, in this case the angle $\theta$ of deviation of the student-probe perceptron from the teacher (the theory is derived for a general distribution along the probe-student decision boundary normal direction, and analytically solved for the case of ranking by margin, in the case of an originally isotropic Gaussian data distribution, prior to pruning).

The theory has several insightful and important predictions to the behavior of the generalization error of the student depending on the metric strength:

$\textit{Optimal pruning: }$
* When data is 'abundant', restricting the samples to 'hard' (small margin) samples results in superior performance relative to no pruning --- i.e. in the sense that for the same number of samples per dimension, lower generalization error is attainable. When data is 'scarce', such restriction is detrimental in terms of generalization error as a function of training samples.

$\textit{Scaling functional dependency: }$
- the error scales as a function of number of samples, for any given (pruned) dataset initially as a better than power-law (claimed exponential) phase which then reverts to a (universal) powerlaw.
- examining the Pareto of optimal pruning ( as a function of initial dataset size and pruning degree ) better than powerlaw scaling is achievable limited by the quality of the pruning metric. Specifically, there is a lower bound in which the $\textit{pareto}$ will transition to powerlaw which is associated with the maximal pruning (minimal data fraction $f_{min}$) of the dataset set by the student-probe quality.
 decrease in error 

$\textit{Information theoretic, per sample contribution:}$

The theory further predicts the information gain per sample added per dimension $I(\alpha_{prune})$

$\textbf{Experiments}$

The theoretical (asymptotic) predictions are shown to hold with excellent agreement in the $\textit{finite}$ dimension ($N=200$) perceptron learning case trained on data $P_{prune} = N\alpha_{prune}$ derived as different difficulty subsets of total data sizes of $P_{tot} = P_{prune}/f$ with various oracle qualities (ideal, where the oracle is the teacher $\theta =0$ and non-ideal $\theta >0$). 

For real data on CIFAR10, SVHN and Imagenet:

* Hard sample preference in the abundance of data (and vice versa) is shown to hold also in practice (CIFAR10)
* The 'signatures' of initial steeper than powerlaw pareto frontiers in real world are shown for all datasets, pruned according to previous art and further developed (self supervised) metrics
* the (lower limit transition to powerlaw and associated maximal pruning) limitation imposed by weak oracle / 'student probe' and how that limitation is pushed with the improvement (further training of the student probe) is shown (with SVHN results in the appendix).

* Transfer learning (CIFAR10): 
Analysis of pruning the up and downstream datasets is performed with superior performance than full data attained.

$\textbf{pruning metrics benchmarking and self-supervised metric introduction}$

Recognizing the criticality of the pruning metric (as it sets a lower limit on the transition to powerlaw and max beneficial pruning factor $f$), the authors systematically analyze previous methods at imagent scale. Evaluating both their rank-agreement and relative performance down to $f=0.6$ .

All above methods require labels and the best performing are computationally intensive. The authors thus construct a self supervised (and supervised version) method and demonstrate its comparable (to best method) performance down to $f=0.8$ (and to memorization-metric-methid down to 70%). The method relies on a (embedding clustering based) self-typicality estimate  from strong pre-trained self-supervised model on Imagenet (SWaV).




strengths_and_weaknesses: ֿ$\textbf{Pre-face:}$
It is this reviewer's view that this is a paper of significant insight, methodological implications and multiple important contributions to the field. An exciting paper for its bridge of phenomena (generalization scaling laws, data pruning) and theory drawing and expanding on classical (statistical mechanics perceptron learning) theory, with inroads to practical understanding of implications, role and potential of data pruning, as well as practical contribution at large (Imagenet) scale of pruning (self supervised based) metrics critical to (limiting of) said potential realization.

With that said, the main improvements recommended are softening/solidifications of some claims - primarily by corresponding with previous art and adding further analysis where merited.

Addressing the below recommendations/requests would improve the paper materially and would likely elevate this reviewer's recommendation further.

Details below:

$\textbf{Strengths}$

* the expansion of classical — statistical mechanics perceptron learning — theory, to the data pruning setting is elegant and insightful.
Notable is the excellent agreement of the $\textit{asymptotic}$ theory in the $\textit{finite}$ dimension (N=200) perceptron regime used throughout the paper.
It sets the stage for a better understanding, through its predictions which are consistent with observations in real data over both prior art and further introduced pruning methods. 

The predictions are, at least, consistent with observations for practical (residual) networks at large (Imagenet) scale:
* Easy / hard training advantage — figure 1C (preceptron), 1D (CIFAR10).

* Better than powerlaw / exponential phase of individual constant hardness $f$ as well as resulting pareto — up to the metric limit (upon reverting to powerlaw). Predicted and shown for perceptron 1A, 1F, 3A. 
This is consistent with observations in figure 3BCD. It is however hard to quantifiably assess the agreement of an actual exponent functional dependence in figure 3BCD or if it is already hitting a powerlaw limit.

* prediction of metric quality on transition to limiting powelaw, and minimal data hardness ($f_{min}$). At real data scale --- consistent with observed $f_{min}$ powerlaw limit and (a little, fortunate) evidence that improved metric via student-probe further training allows to breach said limit Appendix C figure 7.

Beyond theory:

* While the theory does not deal with transfer learning, it is in practice demonstrated that a steep benefit manifests when downstream pruning (notably, adding visual transformer network into the fray).
It is worth to dwell on this, as it is different from the results portrayed in the regular pruning scenario in figure 3BCD. In these experiments, as the total dataset size is fixed, and one is only choosing a subset according to hardness ($f$) then no significant performance gain is evident.
In contrast, in the downstream Transfer Learning experiment 4A, a noticeable advantage in performance manifests when training on *less*   data (a subset).

Implication strengths:

* Data curation exponential gain potential:
While the authors lead with computation implications, this requires further evidence (see below requests for clarification and concern that pruned data may actually be hard to train on). However, it should be noted, that even if computational gain at training is not available, there is significant value in reducing the burden of data collection and especially reducing data labeling quantities in supervised settings where these are expensive. These benefits are orthogonal to the potential computational savings. 

* Practical data curation $\textit{without labels}$:
Thus, the introduction of an at-Imagenet-scale self-supervised metric which is able to match SOTA supervised data-pruning metrics, is a significant and novel (to the best of this reviewers knowledge) contribution. 

The limitation here is that this assumes a strong (and is limited by its strength) metric (oracle / student-probe). See below elaboration on this in the weakness section.

$\textbf{Weaknesses}$

$\textit{Major:}$

1. Computation --- are pruned datasets harder to learn?: 
add an analysis (theory and experiments) of the computational effect of training on pruned data. numbered (R1a-c) below:

In order to make claims regarding computational savings which are at the premise of the paper's motivations, these need to be evaluated. There may be reason for concern here, that a pruned dataset does not translate to computational savings. To elucidate this point, hard examples are expected to be iterated upon numerous times (e.g. giving rise to the very notion of the memorization metric []), thus potentially negating the perceived computational benefit of having fewer samples.
By way of example and request --- even for the simple perceptron learning cases evaluated --- what were the actual times until convergence. We know the classical upper bound is inversely proportional to the margin squared, i.e. to the hardness (governed by $f$). What does this imply on the expected time to convergence in the statistical-mechanics limit? 
In other words: $\textbf{(R1a)}$ what is the theoretical convergence (e.g. relaxation to ground state from high energy initial state or other appropriate expected value) time, and $\textbf{(R1b)}$ convergence in terms of number of iterations in practice (empirically) of the trained preceptors ? 

For assessing this question in the real data scenarios, there is a need to adopt a well calibrated learning schedule, such that convergence times may be (properly) compared, such that the computational benefit may be assessed.
While the paper mentions the comparison in terms of number of epochs and marginal benefit of further iterations in the real data experiments, consistent with a potential computational advantage, it seems that the training schedules are constant epochs [lines 727, 333]. Thus this does not necessarily demonstrate computational savings --- e.g. a different (shorter) schedule may have marginal (comparable) effect on the full data.

Previous work has dealt with this optimality question by using adaptive learning schedules and early stopping. 

$\textbf{(R1c)}$ These CIFAR10, Imagenet computation scaling (at early stopping with adaptive learning) should be added to the paper also in order to significantly strengthen the assessment of the computational implications of data pruning (which as noted is one of the core impetuses the authors lead with, and rightfully so, as it is of significant practical implications). 

A potential easing of this computationally intensive evaluation (involving the re-training and assessment of multiple models, demanding primarily at imagent scale) may be given by examining the learning curves:

Previous work [1] showed close relationship between the online and finite data settings under adaptive learning schemes. Since the setting of a constant $f$ falls into the setting investigated by these works, there is potential to verify this finding here on a single $f$ learning curve vs models on the curve (different amounts of samples) and then (assuming it holds) examine just the training curves of the other models on the full data.

2. The practical metric limitation. 

The authors, rightfully, note the limitation imposed by the data pruning metric — effectively requiring a powerful oracle (or method) which sets a lower limit on the maximal pruning possible.

Not that this problem is not one of mere theoretical inconvenience: The exact common scenario where one encounters new data, on which employing a powerful metric for reduction, would be potentially pertinent, is the case where out-of-distribution characteristics limit any practical oracle (regardless of how well in-distribution it was trained).

For this reviewer, there is no question that amortization (e.g. capitalizing on mutual information between ID, OOD) is a powerful concept.
However a more quantitative analysis of the actual benefit and limitation would be beneficial.

To make this discussion more quantifiable (and interesting), let us consider the two scenarios of amortization effectively discussed in the paper:
Use foundation model to imbue a metric, data prune according to the metric, train new model on the pruned model
Use foundation model in transfer learning (TL) setting — e.g. fine tune on the new data.

$\textbf(R2)$ Is one superior to the other, in terms of computation? Performance? Quantifiably (theoretically?).

$\textit{Medium:}$

3. Related work:

$\textbf{(R3a)}$ Scaling laws --- while the authors mention at the preface notable scaling law works, they miss early work [3] on neural scaling laws, theoretical work [4, 5] as well as pertinent work corresponding with the beyond-powerlaw potential of data-scaling [6]. Also relevant is [7] which explores a resulting pruning pareto — in the complementary setting of parameter pruning, also finding a pareto frontier with varying optimal pruning fraction as a function of original model size (total parameters).

$\textbf{(R3b)}$ Exponential scaling with data is a known (classical) result when 'high-quality' side information is available --- a correspondence and comparison with query based learning classical expectations of the Pareto frontier is merited (Req):

The Pareto presented, while methodologically and theoretically insightful, would do well to correspond with the prior art which classically presents expectations for better-than-powerlaw (e.g. exponential) scaling with samples, \textit{when these are allowed to be chosen}. To put this in the most classical setting (pertinent to the paper) --- in the perceptron learning setting (finding a separating hyperplane), with access to a perfect oracle (i.e. $\theta=0$), binary search will result in an error scaling exponential with number of samples (queries). 

So with access to additional data (the metric) it is a classical expectation to have, at least, exponential scaling with queries.
The explicit addition of this known classical expectation (e.g. binary search), in this reviewer's view, does not erode the novelty of the paper which shows early signs of this prediction in practice, but rather further solidifies its theoretical groundings.

$\textbf{(R3c)}$ Relatedly, corresponding with curriculum learning in the related work section is advisable --- e.g. as related to the notion of realizing the pareto frontier.

4. Implications of the hard/easy predictions on pareto:

The authors empirically evaluate a easy (large-margin p(z) specialization — blue in figure 1B) vs hard (small-margin, green in 1B) and show a predicted cross-over.  
However, in the remainder of the paper, derivations and well as experiments, it is the small-margin case only which is considered (to the best of this reviewer's understanding), and in particular as giving rise to the pareto frontier and $f_{opt}(\alpha_{prune})$.
$\textbf{(R4a)}$ - This reviewer could not find the derivation of f_{opt} (figure 1E), please add it or if not derived mention that.

Further, in light of the crossover, it is this reader’s understanding that the true optimal pareto is upper bounded by a pareto $\textit{lower} $ than the one used throughout the paper. I.e. the crossover implies that taking the minimum of the large-margin (not explicitly derived) and small-margin pruning policies for any given point ($f, \alpha_{prune}$) has a section lower than the pareto in the paper, up to that switching point. $\textbf{(R4b)}$ — it would be good to at least make this point clear in the paper, and to put on a graph like 1A the pareto drawn by this minimum (the derivation, though tedious, would entail the complementary large margin truncated gaussian specialization of $p(z)$, alternatively… just empirically for the trained perceptrons).


The reason that even this, is viewed by this reviewer as an $\textit{upper bound}$ on the pareto frontier is that there is an expected optimal choice of pruning policy / metric usage $p(z|\alpha_{prune},f)$. This policy has a margin support as a function of $(\alpha_{prune},f)$ amenable to the paper derivation framework $\textbf{(R4c)}$ — is this is correct? If so, can you derive / or at least simulate it? Or, alternatively, show that it is expected to have negligible / non-important effect?). 

5. experimental discriminative power of Imagenet pruning methods — need for repetition and clarification:

This is actually a very important methodological point, it is only making the medium cut, because even if all these methods are actually comparable, the concept of having a self supervised method in the ballpark of supervised, expensive, metrics is in-itself a significant contribution.
However, it would be much preferable if the authors modify the experiment such that more statistical confidence in the claims is readily observable — as it currently stands, the level of noise in the experiment (figure 5C) is high. $\textbf{(R5)}$ The self supervised experiments as well as the random baseline should have (at least) 3 repetitions, or alternatively, at least a X3 finer pruning grid (‘fractions of data kept’).
Especially worrisome is the drop of the proposed self-supervised method below random pruning. 

Minor:

6. Clarity of content — general:

$\textbf{(R6)}$ Please add experimental descriptions where missing experiments and figures, reference to appendix OK. This reader spent significant time divining the experimental setups. Examples: how were the perceptron trained? Is this the classic perceptron learning algorithm? What were the grids of evaluation of all experiments (they vary across figures) etc. 


7. Typos and minor clarifications:

Figure 1A — garbled $f$ values in legend

Caption of figure 1:

* “The read curve indicates … at fixed $\alpha_{prune}$ would be better understood if read “The read curve indicates … as function of $\alpha_{prune}$

* "E : Test accuracy … f_{opt} (blue curve)” — is it blue ?? looks purple, maybe change to black…

Line 197: “ … exponential scaling law (Fig 1C…” $\rightarrow$should be “ … exponential scaling law (Fig 1E…” 


$\textbf{References:}$

[1] Nakkiran, Preetum, Behnam Neyshabur, and Hanie Sedghi. "The deep bootstrap framework: Good online learners are good offline generalizers." arXiv preprint arXiv:2010.08127 (2020).

[2] Hernandez, Danny, et al. "Scaling laws for transfer." arXiv preprint arXiv:2102.01293 (2021).

[3] A Constructive Prediction of the Generalization Error Across Scales. Jonathan S. Rosenfeld, Amir Rosenfeld, Yonatan Belinkov, Nir Shavit. 2020 International Conference on Learning Representations. https://openreview.net/forum?id=ryenvpEKDr

[4] Scaling Laws from the Data Manifold Dimension. Utkarsh Sharm, Jared Kaplan. 2022 Journal of Machine Learning Research. http://jmlr.org/papers/v23/20-1111.html

[5] Explaining Scaling Laws of Neural Network Generalization.Yasaman Bahri, Ethan Dyer, Jared Kaplan, Jaehoon Lee, Utkarsh Sharma 2021 https://arxiv.org/abs/2102.06701

[6] Scaling Laws for Deep Learning. Jonathan S. Rosenfeld. 2021. https://arxiv.org/abs/2108.07686

[7] Rosenfeld, Jonathan S., et al. "On the predictability of pruning across scales." International Conference on Machine Learning. PMLR, 2021.


questions: Questions/Requests/Recommendations embedded above, denoted (R#$\alpha$)
limitations: Yes, the authors engage deliberately and adequately with this subject! Specifically uncovering and investigating pruning class imbalance effects --- for both previous methods benchmarked and developed methods --- and addressing it.

There are additional areas of investigation which are active open questions associated more broadly with the line of research dealing with data pruning and are not specific to this paper --- and should thus be, in this reviewers view, dealt with in other, future, work.

Such topics include vulnerability implications of training on pruned data. Specifically, it is an open question (to this reviewer's knowledge) how training on pruned data affects robustness to both naturally occurring phenomena (e.g. OOD performance) and bad actor actions (e.g. adversarial attacks, model stealing).
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper explores neural scaling laws and looks at the power law scaling behavior of error with dataset size. While power law scaling has shown that larger dataset sizes can improve accuracy, it comes at a significant cost as power scaling is extremeley weak and unsustainable. This paper demonstrates that exponential scaling is in-fact possible both in thoery and practice, provided we chose the right dataset to train on. Power law scaling indicates that many training examples are hihgly redundant. Thus, one should be able to train on a much smaller sized dataset. This requires data selection or pruning strategies. Prior work has shown this. However, prior work lacks theoretical and empirical rigor. It leaves many questions unanswered like when is data pruning possible, how to do it at scale, can it beat power law scaling and can pruning be done using unlabeled dataset. 

This paper answer these questions by by first unify existing literature on dataset pruning, statistical mechanics of learning and observations of neural scaling law to provide an analytical theory of data pruning. This theory and its associated empirical results help answer many of the questions asked above including showing how to achieve exponential scaling with dataset. They show that their theory is applicable in real world use cases also, like CIFAR10 and SVHN. To scale to ImageNet sized dataset, it is important for the data pruning metric to be cheap to calculate and also not require labeled dataset. In order to achieve this objective, they propose a clustering based unsupervised data pruning metric that performs equally well or better than existing data pruning techniques while still providing exponential scaling capabilities .

strengths_and_weaknesses: Strengths - 

1. The paper targets an important problem of data quality and its impact on accuracy

2. The paper evaluates its claims on a wide range of datasets and backs them with strong theoretical foundation

3. Insights around the kind of data to prefer at what scale were interesting and novel. Preferring easy samples in low resource regime and hard samples in large dataset regime was something that was not quantified before in prior work

4. They show a simple, easy to use and unsupervised data pruning technique to measure the hardness of a sample.  The technique further scales to a larger dataset. The simplicity of the technique makes it a really strong contribution

5. This is the first paper that shows exponential scaling is possible, given a high quality dataset. Where quality refers to how much does a model learn from each sample in the dataset. These results can further fuel more research in dataset creation.



Weakness - 

The results in the paper are really difficult to parse, making it very difficult to match the figures with the claims in the paper. I do believe that the paper requires some work in its presentation to make it readable to a general audience. Section 3’s Figures which forms the foundation of the claims in this paper are not well explained. Below are some instances

1. Section 3 introduces alpha_tot as a ratio of total training examples (P) and number of parameters in the model (N). alpha_pruning is basically f*alpha_tot where f represents the fraction of the  examples selected. Using this knowledge (as established in lines 159-165) seems incomplete to parse Figure 1. For example, in Figure 1A x axis represents alpha_prune as an increasing variable. That means, f is decreasing. However, the figure also has datapoints associated with different f for the same alpha_prune. Given that alpha_tot is constant, how is that possible? 

There are other similar issues in readability (See questions 1,2,3) that make it really difficult to understand the claims. This makes parsing the next section that shows these results on CIFAR-10 and SVHN equally hard. 

2. The technique described in the paper is post-hoc - Given a dataset, how do you prune it to a few high quality samples to reduce the number of samples to train on. This can potentially reduce the training time of the model (line 336). But this aspect was not discussed in detail in the paper. Given that this paper does not discuss how to build a high quality dataset, its impact can be limited. 
questions: Questions -

1. In Figure 1A, if one draws a vertical line for a fixed alpha_prune, does that represent accuracy/test error at iso-dataset size for various experiments? If yes, how is that possible given that alpha_tot is fixed. If no, 

2. Figure 1E’s description suggests a blue curve, 1E does not have any blue curve.

3. Its unclear reading the figure 1 and the description, as to how to achieve exponential scaling. Given a fixed alpha_total and ability to prune by a fraction f, is the idea to decrease f such that for large f (less training data), use easy examples but for small f (more training data) use hard examples?

4. In Figure 4A, a vertical like at x = 10^3 interesects different lines representing different fraction of original dataset. Will all those points be iso-size? If yes, how, given that they all start from the same dataset size.

5. Overall, given that this work assumes a large dataset is always available. Given a large dataset, one can train on a small fraction and achive a better accuracy using intelligent data pruning techniques. This makes it an approach towards efficient training (mentioned in line 336). Is this understanding correct? If yes, this is approach becomes a bit orthogonal to neural scaling law work whose aim is to understand how much resources (data, compute etc) will be needed to improve the accuracy of the model. Neural scaling laws guide investments into various accuracy improvement strategies. 
limitations: Have the authors adequately addressed the limitations and potential negative societal impact of their work? - Yes
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper considers an interesting question of whether we can "beat" power-law scalings in data with better pruning strategies.
The authors first introduce a theoretical model to study the effect of pruning and its dependence on a few natural parameters (pruning ratio,  initial dataset size), and show that its theoretical analysis matches empirical simulations.
Then, the authors also observe similar behavior on real datasets.
The authors then motivate the importance of a good pruning metric with both theoretical (quantifying the misalignment between student and teacher models) and empirical results (beating power law scaling becomes only possible with a sufficiently good pruning metric).
Then, the authors carry out a thorough evaluation of existing pruning metrics on large scale datasets, and also develop a new metric which does not require any labels
Overall, the work demonstrates the possibility of beyond-power law scaling and evaluates many different pruning metrics in large scale settings.
strengths_and_weaknesses: *Strengths*
- Originality: The work is original in its thesis as well its details: theoretical analysis of pruning on different factors and corresponding empirical demonstration of beyond power law scaling. Related work is well cited.
 I also particularly enjoyed the analysis of the misalignment between student and teacher, and how it impacts optimal pruning strategy.
- Soundness: The work is overall quite thorough and sound, with theory matching simulation, and a wealth of empirical experiments. That said, there were a few logical/experimental gaps in and leading into the second part of the paper (semi-supervised metric) that I was not sure if addressed properly (see Weaknesses and Questions).
 (I cannot speak much for the theory parts, as I am not familiar with the seemingly standard tools for theoretical analysis on this topic using statistical physics, but I am confident in my evaluation of empirical results.)
- Clarity: Overall well written and polished. That said, exposition could be improved. In particular, Section 3 is a bit hard to follow due to overuse of mathematical notation and long paragraphs. The logical flow throughout the paper needs some work too (see Weakness 1).
- Impact: Provides an interesting new perspective (are we limited to power law scaling?), and I believe that the theoretical results as well as empirical results will be valuable to future work in the area.

*Weaknesses*
- While the paper overall is insightful, it suffers from a suboptimal narrative/focus as well as missing a few important experiments. The paper seems to be sort of two papers stapled into one, unless read very carefully:
In the first half, the focus is on scaling behavior, and its theoretical and empirical analysis; the second half focus on a better unsupervised pruning metric for larger scale dataset (ImageNet). 

There is an important logical transition from the first part to the second part, namely in the last paragraph of Section 3 (reference to App. Fig 7 on showing that beyond power law scaling becomes possible for ResNet18 only with a better pruning metric.)
But this was easy to miss, and I think this logical transition should be emphasized (perhaps by bringing the Figure into the main paper).
So I think this "weakness" of the paper is partially addressable with better prose and organization / focusing of results.

That said, I am left unconvinced by the rigor in which "beyond power law scaling" is claimed in the the results in Fig 7b. First, there are only two data points on the scatterplot below the the power law line, and quite close, so it's unclear to me if it's just noise.
Also, what prevents me from drawing a slightly steeper power law line that also includes these two data points? (it's possible I missed some justification.)
It would be interesting to see if there is some kind of statistical test that can be done to reject the "null hypothesis" (power law scaling) with some confidence.

- The benefit of the new semi-supervised metric was not clear to me due to limited experiments on its scaling (see first question). In particular, I would have liked to see a demonstration that the "pareto optimal" frontier can be expanded by using this new metric.
Also, it is a bit strange to use a semi-supervised model pretrained on ImageNet to prune datasets for ImageNet. 
More appropriate would be to take a model pretrained on some other dataset (e.g., CLIP) and using that instead.
Given these, I felt that the self-supervised metric distracted from the strengths of the rest of the paper rather than adding to them.

- A meta-comment related to above two points. I think the paper would be more impactful if it focused on a subset of its results with a tighter narrative. 

I hope to adjust to scores appropriately if my concerns/questions are well addressed.

questions: Main:
- Why did the authors not reproduce the scaling experiments for Figure 3 using the various metrics (and particularly the newly developed self-supervised metric?) Was it due to computational budget limitations or some other reason?  Given the overall narrative in the paper, I wanted to see if using better pruning metrics lead to better scaling behavior / better pareto optimal frontier, but I couldn't find an experiment along this line in the paper/Appendix besides Figure 7 (which are based on variations of an existing pruning metric rather than the newly introduced metric). 
It leaves me wondering if the results were negative, and hence not presented.
In any case, I think demonstrating the impact of the quality of various (existing) pruning metrics on the scaling behavior is much more important than the particular metric introduced in the paper.

Others (some are just curiosity on my part):
- Is it possible to intuitively gleam the the empirical scaling behavior (for the perceptron model) from the equations given in A.2? Or is numerically solving them the only way?
- I liked the experiment on the transfer setting (as it is arguably more relevant in practice), but was a bit disappointed it was only covered briefly. Using your theoretical results/framework, is there something interesting that you can say about this setting?
- From my loose familiarity with the theoretical literature in DL, I understand that people are now capable of giving interesting theoretical results on certain regimes of two layer networks (e.g. NTK). I'm curious, is it possible to extend the perceptron results in the paper to those settings?
limitations: Yes (none other than my questions/concerns above).
review_title: null

Meta-review: metareview: The paper provides theory and experiments that power law scaling of error with respect to dataset size can be improved to "exponential" scaling by high-quality intelligent data pruning. The paper first provides a theoretical model to study the effect of pruning and show that it matches well in experiments. Beyond this, the paper also observes similar behavior on real datasets. The analysis demonstrates the possibility of beyond-power law scaling by utilizing pruning metrics in large scale settings. 

Reviewers saw many strength on the work: as an “outstandingly good paper” with “thorough theoretical backing” and “great empirical design” `NsH6`; an “exciting paper for its bridge of phenomena and theory” with “significant insight and multiple important contributions to the field” `ycTR`, “interesting and novel insights'' targeting an “important problem” which “evaluates its claims on a wide range of datasets…with strong theoretical foundation” `QtTy` as well as “thorough evaluation” with a “wealth of empirical experiments'' which will be “valuable to future work in the area” `D9bn`. Authors addressed questions and weaknesses raised by the reviewers, during the very active author-reviewer discussion period. The AC encourages the authors to incorporate all the agreed suggested changes in camera ready version. 

AC agrees with the reviewers sentiment that the study and insight will be important to the broad NeurIPS audience where correctly characterizing scaling behavior of neural networks is becoming even more important. 

confidence: Certain
Decision: Accept