Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes a novel, adaptive strategy to drop classes from the dataset during training based on the f1 score of the model. They show that their approach, Progressive Data Dropout (PDD), is able to achieve similar test/val accuracies while using significantly lower data and time to train on image classification tasks.
strength_and_weaknesses: # Strengths
- The paper is very well written and easy to comprehend.
- The problem tackled by the paper is very interesting and important to the community as it could speed up training significantly.
- The approach presented is novel to my knowledge and explores an interesting direction that I haven't seen before.

# Weaknesses
- The results are difficult to parse. It's really hard to compare the different approaches since they are trained for a different number of rounds using potentially different models. 
- There also seems to be a significant cost in accuracy to using PDD.
- Why do the alternative data-dropping strategies achieve higher test accuracy than even the baseline? It seems like the baseline should be the best approach since it has access to the most data.
- There should be more experiments with different model architectures.
- There are no standard errors in the results.
clarity,_quality,_novelty_and_reproducibility: - The paper was very clear and easy to comprehend.
- The quality of the paper is not very high since the results don't clearly show the advantage of this approach.
- The approach is novel to my knowledge.
- I did not attempt to reproduce any of the experiments in the paper.
summary_of_the_review: While the approach considered in the paper is interesting and novel to my knowledge, the results leave much to be desired. The baseline approach considered seems considerably worse than expected, especially compared to the alternatives from the literature. There also seems to be a significant cost to using PDD in the final test/val accuracy of the model.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper works on the problem of progressive data pruning, in which you attempt to reduce the number of images (in absolute quantity) during training. They do this by subsampling according to class using a dropout score, which is related to the performance of the network on a particular class in the training set. The introduce a hyperparameter (training threshold), which determines when data should be dropped wrt the dropout score. 
strength_and_weaknesses: 
##### Strengths

- Method is intuitive and easy to understand.
- Problem relevance is apparent, reducing training time would have immediate impact.

##### Weaknesses

- Table 3 shows that the author's reimplementation of their baseline was flawed. The results of the existing data dropout paper are reproducible.
- It's very unclear what Tables 1 & 2 are supposed to convey. There are no comparable comparisons in any of these experiments. The authors should fix an architecture and see how many data points it takes to train to the _same accuracy_. As it is, with variations in architecture, final model performance, and # of data points, we can't draw any conclusions from these tables.
clarity,_quality,_novelty_and_reproducibility: **Clarity**

Paper was readable and the method was understandable. However, the experimental setup was quite confusing (covered in weaknesses).

**Novelty**

The method seems novel. 

**Reproducibility**

Reproducibility unclear. The author's seem to have not reproduced their baseline properly, however they did provide code which seems readable. Have not assessed in detail.

summary_of_the_review: I vote to reject.

Their primary experiments are flawed. As is, we cannot to properly assess the efficacy of their method (see Weaknesses for suggestions to improve).
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper proposes a progressive data dropout (PDD) framework that gradually discards the majority of the samples for already-accurate classes, in order to speedup the training. There are other design choices including warmup, residue and swapout. Experiments on MNIST, CIFAR, SVHN and ImageNet are performed.
strength_and_weaknesses: Strengths:
-- The data dropout idea is well-motivated - time or "datapoints" spent on different classes does not have to be equal. Some classes may add little value once they are learned and it makes sense to discard them and save time.

-- The experiments are performed on multiple datasets of different sizes and compared with various previous methods.

Weaknesses:
-- The results on MNIST, CIFAR and SVHN do not convincingly demonstrate the effectiveness. The time spent is drastically reduced but the accuracy is also hurt, sometimes by large margins. The comparison are done usually without controlling the time or accuracy so it is hard to draw useful conclusions. 

-- The baseline ResNet-18 accuracy of CIFAR-10 is at 77%. This should be around 88% instead for ResNet-18 with a reasonable recipe. It is thus hard to judge the reliability of the results given the baseline may be too suboptimal.

-- On the larger scale ImageNet experiments, the saved time is too minimal (less than 1%) when the accuracy is maintained.
clarity,_quality,_novelty_and_reproducibility: The method was simple and presented clearly. The idea to drop almost entire classes seems novel. The code is in the supplement.
summary_of_the_review: Given the insignificance of the results I recommend rejection.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes a technique for removing data points for standard supervised learning problems. Their best performing method partially drops data from a class with a sufficient F1 score, and shows that they can train on ~6M total less data points while only reducing ImageNet accuracy by 1 percentage point. 
strength_and_weaknesses: Strengths: Method is well motivated and an ablation is performed on each of their design decisions.

Weaknesses:
- I found the comparisons to the baselines to be very unclear. For instance, in Table 1 there are various different architectures, values of epochs, and # of datapoints, such that it is not really clear how to compare properly to baselines. So, while the method is sound, it would be extremely useful to see a comparison where the task, architecture, RT, and datapoints are matched to make a comparison.

Comments & Questions:
- May be good to discuss related work such as [1,2].
- How would the method be extended to large datasets which are often unlabelled?

[1] https://arxiv.org/abs/2107.07075
[2] https://arxiv.org/abs/2206.14486
clarity,_quality,_novelty_and_reproducibility: In terms of presentation (clarity + quality) I enjoyed how Section 5 ablates on each element of the method. In terms of novelty I did not understand "we do not compare with those which are so domain-/application-specific as to not be applicable to standard image classification benchmarks" and would appreciate more detail there.
summary_of_the_review: The paper is well motivated and performs careful ablations, however, the comparison to prior methods was extremely hard to follow and discussion of key related work was missing.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposes to use class-level data dropout to accelerate training of neural networks.  Reviewers unanimously move to reject the paper, with a serious concern being experiments that show a nontrivial cost in accuracy to using the proposed method.  The author response does not present any additional results, so these concerns are unresolved and reviewers maintain their rating.  The Area Chair agrees with the reviewer consensus.
justification_for_why_not_higher_score: Reviewers are unanimous in rejecting the paper as experiments are unconvincing with respect to benefits over the baseline.  The author response does not address these concerns.
Decision: Reject