Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes dataset pruning, a method to remove/reduce training data that has low overall contribution towards model accuracy. The paper uses the concept of influence functions and uses an optimization function to find subsets of data that do not contribute to changes over model parameters. The paper includes theoretical proofs and experiments to validate the claims.



strength_and_weaknesses: Strengths:

+ The paper addresses an important problem given the large datasets in real world, that are often redundant.
+ The validation of the central claim is exhaustive and the paper also evaluates on different architectures.


Weakness

- What is the overall time required for dataset pruning? Do you need to review each data point more than once?


clarity,_quality,_novelty_and_reproducibility: The paper is well written and claims are validated. The papers claims appear to be reproducible.
summary_of_the_review: I recommend accepting this paper because it addresses an important problem, has a clear and practical solution and has been empirically validated.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: **Overall Summary** This paper presents an approach aims to selecting a subset of training data that (1) makes training more computationally efficient while (2) incurring little-to-no loss in accuracy. The proposed approach is analyzed in terms of its generalization performance in addition to computational requirements.

**Methodological Summary** The proposed method uses influence functions to select which datapoints to keep and which to prune in the given training dataset.

**Empirical Summary** The authors perform extensive empirical analysis that compares to baseline methods (herding, forgetting, random), across architectures, and across training settings (standard training, architecture search).
strength_and_weaknesses: This paper presents an interesting, clear, and effectively simple (in a good way) idea for dataset pruning. Here are a summary of strengths and weaknesses

**Strengths**
1. **Effective empirical results** - The authors demonstrate on CIFAR10 & CIFAR100 the effectiveness of their approach. It would appear to have clear improvements over baseline methods, particularly in ~60% pruning.
2. **Simple and well motivated idea** - The proposed idea of using influence functions to select which training points to use is clear and understandable and well suited for the task of reducing training complexity

**Weaknesses and clarifications**

3. **Motivation for Optimization** - I think it would be helpful to justify the need for the constrained optimization problem, compared to say a greedy approximation -- which would select top $m$ examples by $||\mathbb{S}_i||_2$. This is said in "high-gradient-norm samples could be zero if the direction of these two samples’ gradient is opposite", but how does this compare theoretically? empirically? 

4. **Computing Influence Functions (efficiency)** - It would be helpful to clarify the computational cost of computing influence functions in the empirical analysis. 

5. **Computing Influence Functions (during training)** - Is there no benefit to incrementally pruning while training?

6. **Methodological Depth** - I support the simplicity of the approach, however, it is not entirely clear to me whether it would meet the expectations of an ICLR conference paper in terms of the depth to which the authors explore the problem at hand. It seems there are many open questions about (1) class imbalance pruning (2) which parameters are used for influence function computation (3) approximations / exact / efficiency/ tradeoffs  of solution of constrained optimization problem?
clarity,_quality,_novelty_and_reproducibility: Please see weaknesses for clarification questions. 

Please fix cite/citep/citet use.
summary_of_the_review: This paper presents a simple and effective method for dataset pruning using influence functions. There are effective empirical results, but questions about depth of methodological contribution and clarity around some technical details of the approach.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes an optimization-based framework with influence functions for subset selection (or dataset pruning). The authors show in certain cases that the influence based method can be used to prune large datasets with only a small increase in the generalization error. From a practical point of view, this framework can be useful in situations when there is a repeated need for re-training models. 
strength_and_weaknesses: First, I want to highlight that dataset pruning is an important problem and having a robust framework can help in mitigating multiple re-training routines, which can be expensive.  The framework proposed in the paper is new and shows good improvements over other baselines, though the coverage of experiments is slightly limited, which is my main concern. 

Strengths:

	- While both influence function and the optimization algorithm is not new, their combination is relatively new. 

	- The paper is very well written and is coherent.

	- The method has good improvements over the baselines (forgetting and herding), though would like to see some more baselines such as [1] in the main paper. Moreover, showing the generalizability of the framework to other architectures is critical which the authors have shown positive results on. 

Weaknesses:

	- While the paper shows improvements on CIFAR derivatives, it lacks analysis or results on other datasets (e.g., ImageNet derivatives). Verifying the effectiveness of the framework on ImageNet-1k or even ImageNet-100 is important. These results ideally can be presented in the main paper. 

	- The authors should add some details on how to solve the optimization in the main paper.  It's an important piece of information currently lacking in the paper.

	- Some baselines such as [1] are not considered and should be added.


I feel that influence function can be replaced by other influence estimation methods such as datamodels[2] or tracin[3]. It will be beneficial to understand if the updated framework results in better pruning than the baselines. I am assuming it would result in better pruning results, however it would be beneficial to understand which influence based methods are particularly suitable for pruning. 


[1]. https://arxiv.org/pdf/2107.07075

[2]. https://arxiv.org/abs/2202.00622

[3]. https://arxiv.org/abs/2002.08484

clarity,_quality,_novelty_and_reproducibility: - Clarity: The paper is well-written and clear. 

- Novelty: The components of the framework is not novel, however the authors combine two components well to make it work for dataset pruning. 

- Reproducibility: The results could be reproduced with the information given in the paper.
summary_of_the_review: Although the components of the framework is not novel, their combination leads to a generalizable framework (with theoretical guarantees) for dataset pruning.  I will be leaning towards a weak-accept, as I feel that the framework can potentially be used to alleviate the cost of re-training.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper addresses the problem of dataset pruning, which seeks to remove redundant data points from the training set while minimizing model change as much as possible. The method consists of using influence functions to identify the influence of each datapoint, and use a discrete solver to identify which data subset has the least overall influence. The authors derive a rough upper bound for the generalization gap of this method, which are validated experimentally. Empirical results show that the proposed method outperforms competing dataset selection methods such as herding and data forgetting, also show generalization to unseen architectures and improvement in training efficiency.
strength_and_weaknesses: **Strengths**:
- The paper proposes a simple, yet elegant method to perform dataset pruning which is not only a key practical challenge, but can also contribute to a conceptual understanding of the role of particular data points on weights learned by the model. The proposed method seems sound and overcomes limitations of prior methods.

- The broad structure of the paper’s argument is clear and logical. The authors provide an informative presentation of related works and how their work differs from related works, define the problem formulation, present their solution to the problem, and validate their solution empirically, showing that their method outperforms previous methods. The application of this approach to NAS is also very intruiging and presents a strong contribution.

**Weaknesses**: The paper contains typos, awkward sentence structures, and grammatical errors that could benefit from proofreading. Also, the paper does not provide any discussion of the limitations of the method or its assumptions. In addition, there are some questions about the method itself and the interpretation of experimental results (see below). However, these are areas of improvement for the paper, rather than major weaknesses that warrant rejection.

Questions / Suggestions:

- The paper does not provide an understanding of how far off the proposed algorithm is from an oracle algorithm that yields the best data subsets. To this end, it can be helpful to evaluate this approach for small datasets and models, and compare performance against a brute-force oracle which performs an exhaustive search over all data subsets. 

- Is there some analysis on the generalization of the proposed approach from a smaller architecture to a larger one in practice? This might make the proposed approach even more practically relevant at least for the case of NAS.

- Section 6.2, Figure 2
    - The experiment is meant to demonstrate that the generalization gap can be bounded by O(e/n + m/(n^2)) (which is approximately O(e/n)). If this is the case, then the dotted lines should be lower than their corresponding solid lines. However, they both seem to match pointwise, despite the rough nature of the upper bound. Is there a re-scaling of the theoretical results in the graph?

    - What dataset and model are used here? Is it CIFAR10 and ResNet50? It could be helpful to test model architectures of varying orders of complexity to see if empirical results consistently match theoretical results across model architectures or if they deviate more from theoretical results as model complexity increases, as presumably the loss surface becomes less non-linear.

- Section 6.5: Data pruning reduces convergence time, but how much time did it take to perform dataset pruning? It seems that to see the true time savings obtained by the method, the comparison should be [dataset pruning time + NAS training times] vs. [NAS on full training set] (rather than [NAS training time on pruned dataset] vs. [NAS training time on full training set]).

- Regarding this statement in the paper "We compare our method with forgetting Toneva et al. (2019b) because it has the strongest empirical performance compared with Borsos et al. (2020); Paul et al. (2021).", it is still relevant to compare against gradient-norm or loss ranking methods, and as such I would strongly encourage the authors to directly compare against such methods instead of relying on such proxy comparison.
clarity,_quality,_novelty_and_reproducibility: The paper scores high on all four aspects - quality, clarity, novelty and reproducibility. The paper is generally clear, the contributions are novel, the experiments indicate high quality and the algorithm seems simple enough to be reproducible from description.
summary_of_the_review: Accept. I make this recommendation because, overall, the paper addresses a key problem that has both practical and conceptual implications (i.e. how to prune a dataset, which can reduce the computational intensity of model training in practice and help us better understand how models learn from a dataset) in a compelling manner (i.e. proposing a data pruning method that leverages influence functions and discrete optimization).
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper considers a problem of selecting a largest redundant subset in training examples, without which the model parameter changes within $\epsilon$ bound compared to the case of training with the full dataset. The applications of such data selection include the data pruning, where the goal is to identify a set of redundant training samples as many as possible and remove them to reduce the training cost, and neural network search, which aims at searching a network architecture that can achieve the best performance for a specific dataset, with a few training samples as possible. The main idea is to use influence function Koh & Liang (2017a) and formulate the discrete optimization problem to find the largest redundant subset of data with the constraint on the change of the model parameters. By empirical results, the authors show that the proposed optimization can find a subset of training samples effective in data pruning and neural network search. 

strength_and_weaknesses: Strength

- The authors propose a new discrete optimization problem to find a smallest subset of data samples with which a model can be trained with a slight parameter change compared to the model trained with the full dataset. The solution of the proposed optimization can guarantee a small generalization gap. 

- The authors provide simulation results demonstrating that the solution of their optimization (subsets of data samples selected for CIFAR-10 and CIFAR-100, resp.) are indeed effective in applications of data pruning and neural network search. Moreover, the theoretical analysis on generalization gap with respect to the pruning size is shown to be quite close to the empirical generalization gap in Figure 2. 

Weakness

- Computational complexity of the proposed optimization: Even though the proposed discrete optimization does not require re-training of a model, still the optimization itself may require high computational complexity, increasing exponentially in the number of data samples, since the optimization space for (3) or (4) increases exponentially in $n$, e.g. if m=0.5n in (4) (finding 50\% most informative samples) it require ${n\choose n/2}=\Theta(2^n)$ complexity. Can the authors explain the issue with computational complexity?

- Empirical results: In data pruning experiment shown in Fig.1, the authors did not include the most recent SOTA baselines e.g., the Gradient Normed (GraNd) and the Error L2-Norm (EL2N) scores from Paul et al (2021), where the empirical result shows that for CIFAR-10 dataset 50\% training examples can be removed without affecting the test accuracy. The performance of these baselines might be superior than the authors’ method since the authors claim that their method can prune 40\% of examples with 1.3\% test accuracy drop. Can the authors add this comparison as well?

clarity,_quality,_novelty_and_reproducibility: The paper is clearly written and the idea is well addressed with empirical support. The novelty is slightly limited since the influence function has been widely considered in dataset selection. The main idea is on formulating the discrete optimization problem based on the influence function, but the computational cost of this optimization could be the main bottleneck in using this method in practice.
summary_of_the_review: The paper is clearly written and the idea is well addressed with empirical support. The novelty is slightly limited since the influence function has been widely considered in dataset selection. The main idea is on formulating the discrete optimization problem based on the influence function, but the computational cost of this optimization could be the main bottleneck in using this method in practice.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper attempts to improve efficiency of training machine learning models. In this regards, the authors adopt the path of selecting a subset of dataset and propose a discrete optimization using influence functions to select which datapoints to keep and which to prune in the given training dataset. The authors perform empirical analysis that compares to baseline methods (herding, forgetting, random) on CIFAR10 and CIFAR100 datasets. The reviewers appreciated the simplicity of the approach and the empirical success. We thank the authors and reviewers for engaging in discussion towards improving the paper and adding more results like performance in presence of class imbalance. Also please add all reviewer feedback like adding motivation for the optimization problem and details on how to solve the optimization in the main paper as provided during the discussion.
justification_for_why_not_higher_score: - Scalability is a concern
- Need to train a model on entire dataset before pruning
- Not much methodological development
- Evaluation only carried out on simple datasets like CIFAR10 and CIFAR100
justification_for_why_not_lower_score: - Beauty in simplicity which works empirically on CIFAR10 and CIFAR100
- Generalize from dataset selected on small network to larger network as well architecture
Decision: Accept: poster