Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper studies data selection to construct a subset of full data, which is an important research topic in data-efficient deep learning. The authors piont out that prior works on data selection are always specially designed for certain cases, which makes it hard to apply them in practice, since realistic scenes are ever-changing and mismatch pre-defined ones. To address the issue, the concept of moderate sets and a new algorithm are proposed, where the data located in the middle of the data distribution is selected. Extensive experiments on multiple tasks demonstrate the effectiveness of the proposed concept and method. A strong baseline is created for future research. 
strength_and_weaknesses: Pros:
- The motivation of this paper is clear and strong. Although data selection methods have been widely studied in different research topics, there may be no method that is designed to cope with a variety of scenarios at the same time. The paper also gives careful analyses of the vulnerability of existing methods to scenario changes. 
- The proposed method is simple but effective. The method Moderate-DS works with extracted deep representations, which avoids model retraining and access to network structures. The advantages may make it easier to apply in practice. 
- The experimental results are promising. Although the proposed method does not perform the best in all cases, the results show that it can achieve the best performance in most cases and is competitive in others. Moreover, the proposed concept can be applied to other methods to bring practical improvements.
- The writing of this paper is overall great. 

Cons:
- Technical contributions of this paper seems trivial. Although this paper provides insights into the research community, the method implementation is not very technical. 
- More comparison methods can be added to enhance this paper. 

clarity,_quality,_novelty_and_reproducibility: Clarity. There are some unclear points in this paper, which need to be addressed carefully. 

(1) The proposed moderate sets exploit the data points with scores close to the score median. It may be more common that we regard the mean/location as a proxy of a distribution. Why did this paper exploit the median? There is no enough discussion on this problem. 

(2) The baseline “Self-sup.-selection” has a similar claim that we should select easy examples for small datasets and hard examples for large datasets, which could be mentioned in the discussions in Section 2.2.

(3) The mutual information estimator is critical for the justification. One suggestion is to provide more technical details about it. 

(4) More baselines in data selection, e.g., [1-3], can be added to make the results more convincing. 

[1] Cody Coleman et al. Selection via Proxy: Efficient Data Selection for Deep Learning. ICLR 2020. 
[2] Kristof Meding et al. Trivial or Impossible—Dichotomous Data Difficulty Masks Model Differences (on ImageNet and Beyond). ICLR 2022. 
[3] Vitaly Feldman and Chiyuan Zhang. What Neural Networks Memorize and Why: Discovering the Long Tail via Influence Estimation. NeurIPS 2020. 

(5) In practice, a dataset always is corrupted by various sources at the same time. Although the paper has presented the proposed method works well in the cases of clean data, corrupted images, and adversarial examples respectively. It will be interesting to see how different methods work when all these factors simultaneously exist. The setting may be more realistic. 

(6) Apart from the baseline Herding, the other baseline methods mainly stress that those “difficult” examples are more helpful. For example, Forgetting selects the examples that are difficult to be memorized. In fact, these methods can also be revised to select “easy” examples. It is interesting to see how these methods work after such revisions. 

(7) Figure 4 shows that the baselines EL2N and Grand can be improved with the concept of moderate sets. Could the paper provide more such evidence to strengthen the contributions of this paper? 

Quality. The quality of this paper is great. Overall, the writing is good. The descriptions of the motivation, research problem, and solutions are clear, following convincing experimental results.

Novelty. The idea is novel to me and is much potential for future research. 

Reproducibility. The descriptions of experimental settings are detailed. The reproducibility is satisfactory.

summary_of_the_review: This paper focuses on data selection to boost data-efficient learning. The motivation of this paper is strong. To address the issues of prior work, the concept of moderate, and a simple and effective method are proposed. Although the paper does not provide theoretical proofs for the method, both justifications from representation learning/information bottleneck and empirical evidence are provided. Besides, the proposed method is simple but with impressive results. The reviewer appreciates the inspiration this paper provides to the research community and its value in practical applications. Therefore, it is recommended for acceptance.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a data selection method for data-efficient deep learning. To make the model robust under various conditions, such as different levels of corruption and label noise, it propose the moderest coreset. Specifically, it first computes the class centers by averaging features of the corresponding samples, and then define the coreset as the samples that have median distances to the center. Experiments show that the method achieves overall good performance for various conditions, including different levels of image corruption and label noise, and adversarial attacks.
strength_and_weaknesses: Strength

- The method performs more robustly compared to other baseline data selection methods under various conditions, including image corruption, label noise, and adversarial attacks.
- The method is simple and easy to implement.
- The paper is clearly written and easy to follow.

Weakness

- Although the overall accuracy is good, the improvement from the baselines looks a bit marginal. Also, the improvement from the baseline seems smaller for higher levels of corruption and label noises.
- I am not sure if the simple average of the features can be a robust representation of the class. If there are outlier samples, then simple averaging will be significantly affected and can be apart from the actual class center of when there are no outliers. Since the proposed moderest coreset is selected based on the distance computed from the centers, which are the simple average of features, it can also be affected.

Other things

- Is the good data selection strategy common among different model architectures? Or is it a property of data itself? What will be the effect of different choices of "well-trained deep model" for representation extraction? How will the median samples and the model performance change?
- Is the good data selection strategy common among different tasks? Or the good coresets differ across tasks? It would be nice if a discussion on the potential extension to general tasks beyond classification could be included.
- How will the method perform for different backbone architectures (e.g. transformer)?
- Another potential advantage of the proposed method could be a better generalization to unseen domains. It would be interesting to see cross-domain evaluation.
- Additional analysis can be done by varying the selection range, e.g. Q1, Q3, in addition to the current Q2.
- In Figure 2, how does the method work for a smaller selection ratio under 60%?
- In Figure 5, how does the method work for more severe corruption (>20%)
- In Table 3, how does the method work for more severe label noise? (>30%)
- I am not sure about the purpose of Figure 4. I guess a comparison with other sampling strategies (Figure 3) with different score computation methods (GraNd-score, EL2N-score) will show the effect of the proposed median-based selection. 

clarity,_quality,_novelty_and_reproducibility: The paper is clearly written. The motivation for the moderate coreset is reasonable, and the proposed method achieves overall good accuracy. As far as I know, the method to select median distances from the class center is novel. As the method is simple and easy to implement, I expect the results can be reproduced.
summary_of_the_review: In summary, the proposed method is simple but works robustly against various scenarios. However, I think the improvement from the baseline is a bit marginal, and not sure about the significance of the contribution.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper argues that corset should be selected  from near the score median. The authors argue that corset examples that are closer to the median are expected to generalize better to different scenarios than those  that pick examples based on highest loss. The authors point out that in prior work, where outliers are expected to be significant, data points with smaller losses are favored. Their approach seems similar but instead of picking examples based on loss, it picks examples that are closer to the mean class-specific center in feature space, where closeness is measured as as ||z  - zj ||_2, where zj is the class center.

The abstract and intro does not make it apparent what the scenarios that authors have in mind. When reading the paper, I was thinking they  are going to present examples of distribution shifts (e.g., different weightage of unrepresented categories). But, later, in the empirical evaluations, the focus was on data corruption -- mislabeled examples or corrupted images (e.g., fog, motion blur, random occlusion). Unsurprisingly, these are also the scenarios where one can expect techniques that prefer high loss and those close to decision boundary to underperform So, I am not entirely surprised that their scheme did well with increased data corruption or mislabeling compared to most schemes. But, perhaps unsurprisingly, they underperform in the absence of corruption and mislabeling, especially if the corset is large.

The authors tested on adversarial inputs that attack an original classification pipeline, but one weakness in that testing is that the adversarial inputs do not seem adaptive to their technique.






strength_and_weaknesses: Pros: 
-- The empirical results seem consistent with the intuition presented. 
-- Overall, the paper is intuitive. 

Cons: 
-- Isn't Herding also a form of moderate corset strategy? Assuming that is the case, the paper could do a better job explaining the distinction and why that matters.
-- The paper claims that computing distances on hidden representation is more efficient. Any data to substantiate that? Any timing numbers?
 -- What is the intuition, if any, for outperforming Herding?  It may be good to discuss that further. 
 -- Could you elaborate more on how you trained the model when corruptness  or mislabeling was injected in your experiments? For instance, when you had 20% mislabeled examples, what are the curves that are similar to Figure 5? I didn't find them in the Appendix. 
Did the models overfit to the corrupted or mislabeled data?







clarity,_quality,_novelty_and_reproducibility: Generally, the paper seems clear.  

The results are not surprising, but that may be OK. I think the novelty over Herding  (and possibly random) needs to be better argued and supported. Downsides need to be better presented. 
summary_of_the_review: The paper makes a cogent argument as to why a coreset that has examples from near the center of the each class may be more robust to outliers or mislabeled data. But, Herding  (Welling 2009) also does something similar. So, that by itself does not make the paper novel. The authors could do a better job articulating why Herding is not the solution they are looking for. If the main argument is the proposed scheme is more performance-efficient in terms of selecting a corset on very large datasets, then the entire set of experiments should have been geared to supporting that argument.  So, overall, I think interesting ideas are possibly there in the paper. But, they need to be better teased out and better supported.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors propose a novel method for reducing the dataset size by selecting the data that is close to the median error of the projection of each sample against the mean projection of all samples that belongs to the same class. The authors evaluate their results in three datasets, obtaining state-of-the-art results in almost all of them. 
strength_and_weaknesses: Strength:
* The algorithm is easy to understand and to implement
* Both computational and spatial costs are minimal
* The experimental results are promising

Weaknesses
* Only tested in one neural network architecture
clarity,_quality,_novelty_and_reproducibility: Clarity:
* Although the idea is easy to follow and to implement, I suggest the authors to encapsulate the idea into one algorithm environment. The intuition behind the idea clearly stated.

Quality:
* The solution is very well presented. The experimental results are quite promising. However, I would like to see how the algorithm behaves when different architectures are provided. The algorithm was only tested using a resnet solution.

Novelty:
* Although the idea is somehow simple, its novelty is clear.

Reproducibility:
* The algorithm is easy to understand and to reproduce. The model hyper-parameters are included in the paper.
summary_of_the_review: Overall, I think it is a very interesting idea. It is very simple, efficient and easy to implement. I am lowering my score because of the lack of multiple NN architectures in the experimental section. As a future remark, I would like to see if a combination between this approach and the forgetting algorithm could combine the benefits of both methods.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: # Summary
This paper focuses on data selection to construct a subset of full data, which is an important research topic in data-efficient deep learning. To address the issue of designing algorithms for certain cases, the concept of moderate sets and a new algorithm are proposed.  Specifically, it first computes the class centers by averaging features of the corresponding samples, and then define the coreset as the samples that have median distances to the center. Extensive experiments on multiple tasks demonstrate the effectiveness of the proposed concept and method. 
# Strengths:
- the core idea is interesting
- the proposed method is easy to implement 
- experiments are overall convincing (Reviewers vuQC and Ergu)
- writing is great 
- a strong baseline is created for future research.
# Weaknesses:
- Technical contributions of this paper seems trivial. Although this paper provides insights into the research community, the method implementation is not very technical.
- Only tested in one neural network architecture

justification_for_why_not_higher_score: The paper shows an interesting and simple model, easy to implement and a strong baseline however a more sophisticated contributions should be necessary for a spotlight.
justification_for_why_not_lower_score: The paper shows an interesting paper that can not be rejected.
Decision: Accept: poster