Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors propose a reducible held-out loss selection method for faster model training. Their method selects a sequence of training points that are referred to as "just right" for training.  The main methodological contribution of the paper is an information-theoretic selection criterion Predictive Information Content (PIC) to choose training points. Authors emprically show efficacy of their selection criterion over previously proposed baselines.
main_review: While the proposed is conceptually interesting, I have major concerns with the practicality of the proposed method. By assuming an irreducible loss model trained on hold-out the set, authors need access to an already trained model on a separate IID dataset. This seems like begging the task at hand. In particular, to train a model on some training set, Algorithm 1 (RHOLS) assumes access to a model trained on some hold-out set. Then the proposed criterion uses this irreducible model and the training model to identify/select a batch of examples at every iteration. 

Other concerns: 
- Approximation from 4 to 6 might not be accurate if the hold-out set is comparable or smaller than the training data. If the hold-out set is large then why would one not use that for training the model in the first place?  
- It would be interesting to add some discussion on how is the irreducible model is selected. Some ablation experiments showing the differences from equation 4 to equation 6 can be insightful. 
 
I am open to changing my score if there is any misunderstandings with my interpretation. 
summary_of_the_review: Overall, while the proposed method is novel, there is some serious concern with the practicality of the RHOLS method because it assumes an already trained model on a hold-out set to train another model on training set.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper introduces a new technique to select a sequence of training points for faster model training. The newly derived method is tractable, can improve efficiency through selections of examples that are neither too hard nor too easy, and is robust to noise.
main_review: ### Strengths

1. In general, the paper is well-written, with the main ideas outlined clearly. The approach is well-motivated, backed by information-theoretical reasonings, and seems to be built upon established literature.
2. One good contribution of the paper is the illustrations of why hard examples may be inappropriate in some application contexts.
3. The simulation part of the manuscript was carefully constructed with appropriate experiments to support the main findings. The visualizations of the results are very appealing.
4. The derivation of the PIC (and to some extent, the problem considered in this work) is motivated by developments in active learning but is adapted in a novel way. (This might be either a strength or a weakness, depending on the viewpoints, as well as the perceived importance of the proposed problem).

### Weakness

1. As stated above, the framework of the problem is basically an easier version of a classical active learning problem. In active learning, examples' labels are requested within a budget; in the context of this manuscript, they are already available on both the train set and the hold-out dataset. Aside from the novelty of the applications, other aspects of the approach resemble classical discussions in active learning. These include the contrast between hard/easy samples or aggressive/mellow learning, and the central insight that hard examples might be noisy (which is expected from the active learning literature, many of which are referenced in this work).
2. The manuscript is a practical demonstration of an idea to prioritize samples for learning and doesn't have strong theoretical supports (that usually accompany active learning methods).
summary_of_the_review: Overall, my vote for the paper is a (weak) reject. I enjoy reading the paper and think it has some good points and the discussions are meaningful. On the other hand, I think the technical contribution of the work is limited.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper presents a method for selecting the most appropriate data points and using them for training to realize the speedup in training. The method proposes the use of reducible loss using a novel selection function named as Predictive Information Content. The paper claims that proposed RHOLS function helps in reaching the benchmark accuracy in fewer steps.
main_review: Strength:

The paper presents a method for speeding up the training of DNN models. The problem addressed in the paper is important given the long training demand by data-centric applications.


Weakness:

1. The claim about training speed-up is evaluated on smaller datasets, whereas the targeted problem appears real while training with huge data. Therefore, it will be of interest to see the performance of the proposed method on bigger datasets.

2. A theoretical justification about training speed up is missing.

3. More reasoning is required for the approximation used in Equation (5). How to ensure that Dt does not cover the relevant (most significant) data points. There may be chances that more decisive data samples are there in Dt  therefore this approximation may not hold true. 

4. Training speedup is measured in terms of a number of steps but what does a step consist of, is not clear. More discussion is required about Figure 2.

5. Which parameters are tuned by Algorithm-1 is not clear.

6. It will be of great interest to see the comparative performance of the proposed method in terms of wall-clock time in a constant experimental setting.
summary_of_the_review: It is a good work with good technical contributions on an important problem. 
Authors need to rectify / address the outlined concerns above.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a technique to speed up training by using another trained model as a proxy for how informative examples are. The authors conduct experiments on small datasets (e.g. CIFAR10/100) on ResNet-18 to support their claims. 
main_review: The paper proposes a technique to speed up training by another trained model as a proxy for how informative examples are. Specifically, starting with the reduction in loss as a criteria, the authors use Bayes theorem to come up with an approximation called the reducible held-out loss (RHOL). Basically, RHOL compares the differences between the current model trained so far and another model trained a separate holdout dataset. A larger difference indicates that the sample is informative and should be selected in the batch. 

Overall, I think the method is neat. However, it defeats the original purpose of the work, which is to present a method for accelerating training in the very large data regime. Suppose one has a large dataset with hundreds of millions of examples (as in CLIP or ViT). Using a model trained on a small holdout dataset as a proxy for informativeness is questionable. On the other hand, if the proxy model is trained on a large hold-out dataset, then training is not really accelerated since one has to train two models now. In the experiments, the authors use small datasets only (of CIFAR10/100 size). In addition, the holdout dataset they use is of a comparable size to the original training set (e.g. half of the images in CIFAR10 are used for holdout). In Page 4, the authors do state that their loss rests on the assumption that the holdout set is large when they argue that approximating h(y | x, holdout, D) with h(y |x, holdout) does not have a big impact.

In addition, there is the issue of reserving a labeled holdout set. In CIFAR10, for example, half of the examples are reserved. The authors should compare against the baseline method that is trained on the full data (both training and holdout) since that's really the alternative. I don't think that reserving a large subset of labeled examples and training another model on those is a good solution for "accelerating" training. 

Experimentally, the improvement in speed reported in the paper is around a factor of 2. Since another model is trained and a subset of the labeled data is reserved, I don't find the speedup significant enough for this approach to be useful.  

Some minor comments:
- The reducible holdout loss criteria is not "information-theoretic" as is claimed in the abstract.
- It is not clear how selecting a subset of examples during training impacts minority groups. It is known, for example, that related effects (e.g. pruning) can have a disproportionate impact on minorities so that should be looked at as well. It is possible that the model selects typical examples first (by learning general rules that apply to them) and focuses on the minority examples later during training (by memorization). In that case, there would be error disparities between both subpopulations.

========= 

Post Rebuttal: 

I have increased my score based on the new materials added to the paper that addressed some of my concerns. 
summary_of_the_review: The paper is well-written and easy to read, and the proposed criterion is neat. However, implementing this method in practice for the purpose of speeding up training would require: (1) training another model, and (2) reserving a subset of the labeled data as holdout. I think both requirements constitute serious limitations that defeat the original purpose. 

In addition, the experiments were conducted on small datasets only (at least ImageNet should have been included). In those experiments, the holdout data was of a comparable size to the training set and the improvement in speed is only around a factor of 2, which is not significant given the above concerns.
review_title: null

Meta-review: The paper provides a method to accelerate training by choosing a subset of points. After the initial submission, the reviewers raised a major concern about the practicality of the method. In the rebuttal phase the authors provided additional experiments on a large datasets that addressed this concern. That being said, the reviews are still quite borderline. The biggest remaining concern is about the quality of writing. Specifically, there are still requests to “fix the narrative” (NdhY, DHeZ). In addition, some details seem to remain vague regarding the positioning of the paper when compared to the active learning literature (BBTj).
Overall, the paper seems to have potential, especially with the new experiments. However, the changes it required when compared to the originally submitted version are simply too extensive to be thoroughly reviewed in a rebuttal phase.
Decision: Reject