Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a novel method for computing the derivative of the weights assigned to datapoints in a dataset with respect to the generalization error of a learning task. The core idea being that one can take a pretrained model, and compute its derivative with respect to a fine-tuning task/ dataset and understand which datapoints are important and which datapoints are not important. This can then be used for tasks such as 1) outlier detection, 2) dataset expansion, and 3) dataset reweighting. Comparisons of the proposed work to other previous work reveals that the current work achieves better results on dataset reweighting. 

main_review: Positives
+ The problem formulation and closed form solution to the dataset derivative are potentially interesting and generally useful for deep learning
+ The work achieves good results with respect to other previous works, and experiments on dataset expansion are very interesting

Negatives

My concerns with the work are to do with the methodology and experimental design choices are around comparison with sufficient baselines.

Methodology: The proposed experiments in the paper all use the deep network as a feature extractor and the last linear layer of the network to make decisions on classification. Thus, we are in a pretraining regime (where there is no dataset derivative being used) and a downstream task where we extract the pretrained features and do linear classification. Given this, the experimental / methods section (sec. 3) is very terse and complicated since it talks about linearizing the model etc. where the model actually that is used is infact already linear. I understand it might have been written that way with generality in mind, but the experiments should then also reflect that and demonstrate that the model / approach does indeed work for the deep learning case (say training from raw pixels) or something more general like that. 

Comparison to baselines: A number of important comparisons to baselines / discussion around those approaches seem to be missing in this current work. 

1) The paper mentions that it is difficult to compare to Koh and Liang (2017) which directly addresses the problem of outlier detection and also the problem of dataset reweighting for better genrealization by estimating the influence of upweighting a data sample by epsilon. The paper mentions that the work was not compared to because of computational reasons, but I am struggling to see how that work is much more expensive than the current work or the work of Ren et.al. (2018). For example, given the task of dataset reweighting, it appears to me that the work of Koh and Liang requires the following to be computed: 1) the hessian of the data at the optimum solution on the training set (unweighted), 2) the gradient of the loss with respect to the parameters at the training datapoints, and 3) the gradient of the loss with respect to the parameters at the test datapoints. 1) and3) are required for the proposed approach as well, and 2) is not that expensive to compute for a Mean Square Error Regression. Secondly, unlike the claim in the paper Koh and Liang can be used to do dataset reweighting by computing the influence of each training datapoint on the validation set, and keeping around some top-K training samples and checking how well one does based on them.  (*)

2) Another important paper in this literature which is missing (and should really serve as a baseline) is that of [A]. I think this work should definitely be compared to the current framework. (*)

3) As far as comparison to Ren. et.al. (2018) is concerned, the key difference is that the current work proposes to linearize the classifier function (whereas Ren et.al. (2018)) use a linear approximation to the actual loss function, since they do gradient descent whose variational form is essentially \min_{\theta’) L(\theta’) + || \theta - \theta’||_2^2 which is then approximated as \min_{\theta’} L(\theta) + d/d\theta (L(\theta)) (\theta - \theta’) + ||\theta - theta’||_2^2, solving which gives us the update equation for SGD. The second difference is that while Ren et.al. (2018) solve the update for the dataset in a single step and the inner loop problem in a single step (not fully), the current paper considers a linear approximation to the original problem and solves the two problems exactly. It would be nice to understand given this where exactly the gains of the work over that of Ren. et.al. (2018) come from. This can be achieved in two ways:

1) Solving the current problem sub-optimally by only doing one step of gradient descent and computing inexact solutions in the linear case, even though we know exact solutions (derived in this work exist). This would help to contextualize how much benefit we have from the current approach.
2) Solving a problem that is non-linear (as opposed to fixed feature extraction and linear classification) such that the linear approximation using taylor series actually comes into play in the sense that there are higher-order terms which are being ignored. In this setting it would be important to compare the current approach to Ren. et.al. This would tell us whether one benefits in this problem domain by solving the correct problem approximately (which Ren. et.al. do) or solving an approximate problem correctly (which this paper does) and what the tradeoff is between these modes. (*)

[A]: Lorraine, Jonathan, Paul Vicol, and David Duvenaud. 2019. “Optimizing Millions of Hyperparameters by Implicit Differentiation.” arXiv [cs.LG]. arXiv. http://arxiv.org/abs/1911.02590.
summary_of_the_review: This is a promising method to the well studied problem of identifying the weighting of the datapoints in a learning task that aid generalization on a given task. The current method while promising is missing some vital comparisons which limit the ability of a reader to assess the true impact of the work, along with experiments which make a lot of the methods sections and details in it seem like overly complicated detail. Given this I do not think the current paper is ready for publication. Important points for the rebuttal are marked with a (*).
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors define a data set derivative, which can be used to update weights of individual samples in training data such that a trained model may increase its performance on test data.  In addition to re-weighting samples within a training data set, the authors also describe how the same techniques can be used to select new data points from a pool of potential candidates and used to estimate which samples are likely to be mislabeled in data.
main_review: In general, I think this is a good paper.

I think it addresses a longstanding and open question -- how do we know we are feeding our models the right data during training to best set them up for success?  This paper adds a new tool in the arsenal for figuring out which data is best for a given learning task.  I believe the focus of this paper would be of interest to a broad audience.

The discussion section ends very abruptly.

I would like to see this method extended to apply in cases where the model class in question is not a DNN.  The Dataset Extension, Curation, and Reweighting capabilities would also be of great use for models trained on tabular data.

One thing that feels missing in the paper is a good real-world example of how this method could make positive impact.  i.e. an immediate thought that comes to mind could be a scenario where multiple domain experts label data, but inter-rater reliability may be low - using DIVA to downweight samples which ultimately confuse the model seems useful in a case like this, which often arises in clinical contexts.

A relationship that I would like to see the authors explore in the discussion section is overfitting of the model.  DIVA reduces the weights of hard/confusing images (figure 3.4).  An alternate hypothesis for why the updated model performs better on test data is that the easy/canonical images can be adequately classified by a simpler model.  Side-by-side, the simple model trained on easy data may generalize better than a complex model trained on hard data.  Ruling this out would make DIVA appear stronger, as this would mean it is impossible to improve test set accuracy by simply feeding a model the most canonical images during training.
summary_of_the_review: I think the paper is in good enough shape to warrant publication with only minor changes.  I believe the focus of this paper would be of sheer interest to a broad audience.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper introduces an approach to calculate the derivative of data samples' influence on the validation error. For this, the authors use a recent model linearization method and leverage closed-form leave-one-out loss calculation for linear models. Besides the theoretical derivation, they present consistent improvements in dataset extension, re-weighting, outlier rejection and automatic aggregation of multi-modal data. 
main_review: **List strong and weak points of the paper. Be as comprehensive as possible.**

**Pros:**

* The introduction is well written and motivates the necessity of DIVA and discusses its limitations.
* the related work sectiton does a great job at showing the similarities and differences compared to other approaches.
* Data quality is often a major problem and solved with overparametrized models. Tackling this issue with a closed-form solution is great.
* I think it's an important contribution to raise awareness for the limitations of only one predefined validation set. For computational reasons, deep learning is often not using the previously well established concept of k-fold cross-validation, and this paper offers an attractive alternative for more robust evaluation of neural networks. 

**Cons:**

* There are a few question left open, see questions below.
* The empirical behavior off DIVA is only shown on image data. We do not know how it performs in other domains or types of datasets.
* Minor/Typos:
  * Introduction line 2: "from from"
  * Introduction last paragraph before summary: "LQF" should be defined
  * Figure 5 (right): "Test Error (\\%)" remove "\\"
  * Equation 13: ",." -> only "," should be there?

**Clearly state your recommendation (accept or reject) with one or two key reasons for this choice.**

I think this is a valuable and interesting contribution and I vote for accept. Going beyond a single fixed validation in deep learning set is important and often neglected and DIVA proposes an efficient approximation for a better estimate of the generalization error. 

**Ask questions you would like answered by the authors to help you clarify your understanding of the paper and provide the additional evidence you need to be confident in your assessment.**


* it would be helpful to have maybe a piece of pseudocode describing how the application of DIVA works. 
  * Is it necessary to train the linear layer?
  * Does this layer represent the classification task?
* The paper only shows results for classification tasks. Is it possible to apply DIVA for regression or other tasks in general? Would be helpful to add some clarification and at least mention other tasks.

* the authors mention that 
  
  *For our experiments on dataset optimization we consider datasets that are smaller than the large scale datasets used for pre-training as we believe they reflect more realistic conditions for dataset optimization*

  it seems that computational efficiency could also be an important factor with large data sets, right? 

* the idea of efficient leave-one-out errors is also well known in the kernel machine literature and it seems that some of the results are similar to the results for efficient leave-one-out crossvaliation for e.g. kernel ridge regression or kernel logistic regression (see e.g. [Cawley and Talbot, "Efficient approximate leave-one-out cross-validation
for kernel logistic regression", Machine Learning Journal, 2008](https://link.springer.com/content/pdf/10.1007/s10994-008-5055-9.pdf)). I don't think this similarity weakens the contribution of this submission, but it could be helpful to comment on the relationship between the dual variables in kernel machines and the importance weights in the DIVA approach

* sorry if I missed that, but would it be possible to include a direct comparison of the approximation and the true LOO error? The SGD based optimization seeems a bit unconventional, with the step size and very early stopping. Maybe some of the solutions from the kernel literature or matrix algebra, e.g. using cholesky decompositions, would be helpful for more efficient optimization of DIVA? 

**Provide additional feedback with the aim to improve the paper. Make it clear that these points are here to help, and not necessarily part of your decision assessment.**

A detailed description how to apply DIVA would improve replicability.

Also it would be helpful to see the computational complexity of DIVA as a function of the data set size. 

And I'm not sure I fully understand the argument with DIVA not needing a validation set. Leave-one-out cross-validation (LOO CV) is using validation data, it's just split differently. Ideally every model should report k-fold cv'd results, and the estimate of the generalization error will become better the larger k. The fact that often researchers use a predefined validation set is probably more due to the high computational complexity of doing CV properly. So I think I wouldn't emphasize that aspect too much as an advantage of DIVA, that it's applicable when there is no validation set - k-fold CV/jackknife/LOO would also help in that case.
summary_of_the_review: This paper is a valuable and interesting contribution. It raises awareness for the importance of going beyond a single validation split for evaluation of neural networks and presents an interesting closed form solution for the leave-one-out (LOO) error that is, to the best of my knowledge, novel. The similarities of the proposed approach with efficient LOO cross-validation approaches for kernel machines suggests that there could be synergies that would allow to improve the optimization procedure of DIVA.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a method for dataset optimization that learns sample weights without a separate validation dataset.
By using squared loss and linearization around a pre-trained model, derivative w.r.t. the sample weights can be obtained as closed-form, which realizes efficient end-to-end learning. In addition, this paper uses leave-one-out cross-validation (LOOCV) loss instead of validation loss on a separate dataset, which enables to conduct evaluation without a separate validation dataset.
The experiments are conducted with multiple tasks (Dataset AutoCuration, Dataset extension, detrimental sample detection, multi-modal learning, and data augmentation).
main_review: Pros:
- The paper is well-written and easy to follow.
- Learning sample weights without a separate validation dataset is interesting and useful since it enables us to use the full dataset as training data. Although there are several methods to learn sample weights as described in section 2, they assume a separate validation dataset.
- The authors derive the closed-form expression of the derivative of the loss w.r.t. sample weights and it is attractive in terms of efficiency.

Cons.
- As described in the Introduction, the proposed method assumes pre-trained models to obtain the closed-form expression of the derivative, which might restrict its applicability in some cases. Is it possible for the proposed method to optimize model (neural network) parameters and sample weights simultaneously? 
- Although the authors conducted experiments in multiple tasks, it is not sufficient. For example, comparison methods are a little naive. Comparison with other methods was conducted only in the dataset autocuration experiment.

Other comments and questions:
- Although the authors explicitly derive the Jacobian of $w_{\alpha}$ in eq. (8), is it sufficiently fast by using automatic differentiation of $w_{\alpha}$ (eq. (5))?
- How important is the initialization of sample weights $\alpha$?
- Although the proposed method learns sample weights without additional validation data, is it also possible to tune its hyperparameters such as optimization iteration numbers without the validation data?
- It is better to describe how many trials were performed in each experiment. 
- It is unclear which loss was used ($L_{val}$ and $L_{LOO}$)  in the main paper.

Typo:
- second line of Introduction: how to select from from -> how to select from.
- $z_i$ -> $x_i$ in eq. (13)?
- in Figure 4, the meaning of blue and red lines would be opposite

summary_of_the_review: Although the motivation of this work is interesting and the paper has some technical novelty, experiments can be improved.
Thus, I'm slightly leaning towards the reject side at this moment.

---After rebuttal---
Thanks for the response.
I think there is room for improvement in the experiment (comparison with existing methods for each task), 
but considering the promise of the proposed method in a number of tasks and other reviews, I decide to increase my score to 6.
review_title: null

Meta-review: This paper has been independently reviewed by four expert reviewers. Two of them recommended straight acceptance, one of them assesses this work as marginally acceptable after increasing their score as a result of the author's rebuttal, and the last reviewer considers this paper marginally below the acceptance threshold. While the reviewers agree on the importance of the  targeted problem and relative novelty of the presented work, the main points of criticism involve empirical evaluations - its methodology, experimental design, missing relevant and important comparisons. Since the authors have addressed most of those concerns in their rebuttal, I am leaning towards recommending acceptance of this work for ICLR.
Decision: Accept (Poster)